replacement for comma.ai backend and useradmin dashboard. comes bundled with a modified cabana instance for seamless drive reviewing.

Related tags

Command Line openpilot comma cabana
Overview

retropilot-server

Replacement for comma.ai backend and useradmin dashboard. Bundled with a modified version of comma's cabana to allow viewing & analyzing drives.

If you don't want to host your own instance, check out https://api.retropilot.org/useradmin for a hosted version of the backend, useradmin and cabana.

[Server] Summary

The server consists of 2 node scripts.

server.js is using expressjs and runs the backend (file upload / communication with openpilot) and the useradmin dashboard to manage / view / download drives & logs. worker.js is a background worker that is processing drives (analyzing video files & logs) to prepare drives for playback in cabana and to gather statistics. It automatically terminates itself after 60 minutes to make sure the video/log libraries do not cause memory leaks.

Both scripts can be started with a cronjob each minute, they use locking to make sure they run exclusively.

Attention: Minimum required node version is node 10.

[Server] Installation

npm install
cp config.sample.js config.js
cp database.empty.sqlite database.sqlite
> EDIT config.js

[Server] Running

node -r esm server.js

node -r esm worker.js

[Server] CABANA Support

A compiled version of a custom cabana fork (https://github.com/florianbrede-ayet/retropilot-cabana) is directly bundled in the cabana/ subdirectory and will be served by the express app. After starting server.js, cabana is ready to use.

[Device] Preparation / Enable Custom Server

On the device or in your fork's code, replace all API endpoints with your own server endpoint. This could be executed directly on the device in the shell to use https://api.retropilot.org as backend:

find /data/openpilot -type f -exec sed -i 's/https:\/\/api.commadotai.com/https:\/\/api.retropilot.org/g' {} +

[Device] Swapping Servers (Back)

To switch a device between different servers, you have to remove the old DongleId and reboot:

rm /data/params/d/DongleID
reboot

There is no need to backup the DongleId, as the new server will identify your device based on its imei, serial and public key.

[Device] Raw Drives Not Uploading (fcamera & rlog)

  1. Raw data is only uploaded if the device is sufficiently charged, not connected to an active panda (offroad) and there are no immediate files (boot, crash, qcamera, qlog) remaining.
  2. Your branch might have raw uploads disabled, check Device Settings > Upload Raw Logs.

If that doesn't help or the option is not available, try:

echo "1" > /data/params/d/IsUploadRawEnabled
echo "1" > /data/params/d/UploadRaw
reboot

Current Limitations

OpenPilot before 0.8.3 will not display any statistics or pairing status in the dashboard. The reason is that pre 0.8.3, the offroad.apk with react and comma-api would require recompilation to accept the new endpoints.

The athena websockets interface is not implemented yet, so the comma app and athena specific remote control commands (including "upload on demand") are not functional as of now.

Screenshots

image

image

image

image

Comments
  • Split more logic out of routes, converting HTML generating code to push out json instead

    Split more logic out of routes, converting HTML generating code to push out json instead

    Working on getting clearing down the HTML generating code so we just output pure JSON to allow for a web app/easier API intergrations.

    Split logic out of routes wrote tests for some api endpoints

    opened by AdamSBlack 0
  • removed requirement for imei to register device

    removed requirement for imei to register device

    fr this time

    IMEI should be logged in the database as NULL, we don't reference devices by IMEI's internally anyway. as C3's don't have IMEI's and newer releases of OpenPilot no longer report IMEI's this will fix compatibility with 0.8.3 onwards.

    opened by AdamSBlack 0
  • Connect to RetroPilot

    Connect to RetroPilot

    but cant connect to retropilot device id always N/A changed liek the wrote at github here below: find /data/openpilot -type f -exec sed -i 's/https://api.commadotai.com/https://api.retropilot.org/g' {} +

    also how can connect your branch to : https://github.com/florianbrede-ayet/retropilot-server ?

    tryed with DragonPilot 0.8.9 and 0.8.13 and with a Custom Fork Branch fir Audi

    Get no Dongle ID - Device ID always N/A

    also set at DragonPilot Register at Server etc thank you

    opened by Gamerios 0
  • Docker build error

    Docker build error

    Hi,

    I'm getting an error building an image with the Dockerfile.

    Any ideas ?

    Thanks

    Cloning into 'retropilot-server'... npm WARN deprecated [email protected]: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau npm WARN deprecated [email protected]: this library is no longer supported npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated [email protected]: This package is no longer supported. It's now a built-in Node module. If you've depended on crypto, you should switch to the one that's built-in. npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated [email protected]: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future npm WARN deprecated [email protected]: Please upgrade to v7.0.2+ of superagent. We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing. See the releases tab for more information at https://github.com/visionmedia/superagent/releases. npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap. npm ERR! code 1 npm ERR! path /retropilot-server/node_modules/sqlite3 npm ERR! command failed npm ERR! command sh -c node-pre-gyp install --fallback-to-build npm ERR! Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64 --napi_version=8 --node_abi_napi=napi --napi_build_version=3 --node_napi_label=napi-v3 --python=/usr/bin/python3' (1) npm ERR! node-pre-gyp info it worked if it ends with ok npm ERR! node-pre-gyp info using [email protected] npm ERR! node-pre-gyp info using [email protected] | linux | arm64 npm ERR! node-pre-gyp WARN Using request for node-pre-gyp https download npm ERR! node-pre-gyp info check checked for "/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node" (not found) npm ERR! node-pre-gyp http GET https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz npm ERR! node-pre-gyp http 403 https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz npm ERR! node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz npm ERR! node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v93 ABI, musl) (falling back to source compile with node-gyp) npm ERR! node-pre-gyp http 403 status code downloading tarball https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz npm ERR! gyp info it worked if it ends with ok npm ERR! gyp info using [email protected] npm ERR! gyp info using [email protected] | linux | arm64 npm ERR! gyp info ok npm ERR! gyp info it worked if it ends with ok npm ERR! gyp info using [email protected] npm ERR! gyp info using [email protected] | linux | arm64 npm ERR! gyp ERR! find Python npm ERR! gyp ERR! find Python checking Python explicitly set from command line or npm configuration npm ERR! gyp ERR! find Python - "--python=" or "npm config get python" is "/usr/bin/python3" npm ERR! gyp ERR! find Python - "/usr/bin/python3" is not in PATH or produced an error npm ERR! gyp ERR! find Python Python is not set from environment variable PYTHON npm ERR! gyp ERR! find Python checking if "python3" can be used npm ERR! gyp ERR! find Python - "python3" is not in PATH or produced an error npm ERR! gyp ERR! find Python checking if "python" can be used npm ERR! gyp ERR! find Python - "python" is not in PATH or produced an error npm ERR! gyp ERR! find Python npm ERR! gyp ERR! find Python ********************************************************** npm ERR! gyp ERR! find Python You need to install the latest version of Python. npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not, npm ERR! gyp ERR! find Python you can try one of the following options: npm ERR! gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable" npm ERR! gyp ERR! find Python (accepted by both node-gyp and npm) npm ERR! gyp ERR! find Python - Set the environment variable PYTHON npm ERR! gyp ERR! find Python - Set the npm configuration variable python: npm ERR! gyp ERR! find Python npm config set python "/path/to/pythonexecutable" npm ERR! gyp ERR! find Python For more information consult the documentation at: npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation npm ERR! gyp ERR! find Python ********************************************************** npm ERR! gyp ERR! find Python npm ERR! gyp ERR! configure error npm ERR! gyp ERR! stack Error: Could not find any Python installation to use npm ERR! gyp ERR! stack at PythonFinder.fail (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:330:47) npm ERR! gyp ERR! stack at PythonFinder.runChecks (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:159:21) npm ERR! gyp ERR! stack at PythonFinder. (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:202:16) npm ERR! gyp ERR! stack at PythonFinder.execFileCallback (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:294:16) npm ERR! gyp ERR! stack at exithandler (node:child_process:406:5) npm ERR! gyp ERR! stack at ChildProcess.errorhandler (node:child_process:418:5) npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:520:28) npm ERR! gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:289:12) npm ERR! gyp ERR! stack at onErrorNT (node:internal/child_process:478:16) npm ERR! gyp ERR! stack at processTicksAndRejections (node:internal/process/task_queues:83:21) npm ERR! gyp ERR! System Linux 5.10.17-v8+ npm ERR! gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=3" "--node_napi_label=napi-v3" "--python=/usr/bin/python3" npm ERR! gyp ERR! cwd /retropilot-server/node_modules/sqlite3 npm ERR! gyp ERR! node -v v16.14.0 npm ERR! gyp ERR! node-gyp -v v8.3.0 npm ERR! gyp ERR! not ok npm ERR! node-pre-gyp ERR! build error npm ERR! node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64 --napi_version=8 --node_abi_napi=napi --napi_build_version=3 --node_napi_label=napi-v3 --python=/usr/bin/python3' (1) npm ERR! node-pre-gyp ERR! stack at ChildProcess. (/retropilot-server/node_modules/node-pre-gyp/lib/util/compile.js:83:29) npm ERR! node-pre-gyp ERR! stack at ChildProcess.emit (node:events:520:28) npm ERR! node-pre-gyp ERR! stack at maybeClose (node:internal/child_process:1092:16) npm ERR! node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) npm ERR! node-pre-gyp ERR! System Linux 5.10.17-v8+ npm ERR! node-pre-gyp ERR! command "/usr/bin/node" "/retropilot-server/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" npm ERR! node-pre-gyp ERR! cwd /retropilot-server/node_modules/sqlite3 npm ERR! node-pre-gyp ERR! node -v v16.14.0 npm ERR! node-pre-gyp ERR! node-pre-gyp -v v0.11.0 npm ERR! node-pre-gyp ERR! not ok

    npm ERR! A complete log of this run can be found in: npm ERR! /root/.npm/_logs/2022-03-15T13_29_23_887Z-debug.log

    opened by gomme600 0
  • Error in worker

    Error in worker

    Bzip error and RangeError when running worker.

    start_worker.sh is "/usr/bin/node -r esm worker.js". Node is v10.19.0. bzip2 is v1.0.8.

    $ ./start_worker.sh 
bzip2, a block-sorting file compressor.  Version 1.0.8, 13-Jul-2019.

   usage: bunzip2 [flags and input files in any order]

   -h --help           print this message
   -d --decompress     force decompression
   -z --compress       force compression
   -k --keep           keep (don't delete) input files
   -f --force          overwrite existing output files
   -t --test           test compressed file integrity
   -c --stdout         output to standard out
   -q --quiet          suppress noncritical error messages
   -v --verbose        be verbose (a 2nd -v gives more)
   -L --license        display software version & license
   -V --version        display software version & license
   -s --small          use less memory (at most 2500k)
   -1 .. -9            set block size to 100k .. 900k
   --fast              alias for -1
   --best              alias for -9

   If invoked as `bzip2', default action is to compress.
              as `bunzip2',  default action is to decompress.
              as `bzcat', default action is to decompress to stdout.

   If no file names are given, bzip2 compresses or decompresses
   from standard input to standard output.  You can combine
   short flags, so `-v -4' means the same as -v4 or -4v, &c.

[2022-01-05T09:39:44.867] [INFO] default - STARTING WORKER...
[2022-01-05T09:39:44.871] [INFO] default - Verified storage path /footage/retropilot/realdata
[2022-01-05T09:39:49.908] [INFO] default - processSegmentsRecursive f3b1d6e8 2021-12-21--11-33-39 7
[2022-01-05T09:39:50.955] [INFO] default - processSegmentRLog GpsLocationExternal @ 506702182045: 
[2022-01-05T09:39:50.988] [INFO] default - processSegmentVideo duration: 59.999167s
[2022-01-05T09:39:54.114] [INFO] default - processSegmentRLog GpsLocationExternal @ 524691373913: 
[2022-01-05T09:39:54.277] [INFO] default - processSegmentRLog GpsLocationExternal @ 525693860319: 
[2022-01-05T09:39:54.472] [INFO] default - processSegmentRLog GpsLocationExternal @ 526791222974: 
[2022-01-05T09:39:54.653] [INFO] default - processSegmentRLog GpsLocationExternal @ 527791061776: 
[2022-01-05T09:39:54.849] [INFO] default - processSegmentRLog GpsLocationExternal @ 528790965578: 
[2022-01-05T09:39:55.020] [INFO] default - processSegmentRLog GpsLocationExternal @ 529791151671: 
[2022-01-05T09:39:55.203] [INFO] default - processSegmentRLog GpsLocationExternal @ 530791714014: 
[2022-01-05T09:39:55.370] [INFO] default - processSegmentRLog GpsLocationExternal @ 531791248077: 
[2022-01-05T09:39:55.543] [INFO] default - processSegmentRLog GpsLocationExternal @ 532791311253: 
[2022-01-05T09:39:55.698] [INFO] default - processSegmentRLog GpsLocationExternal @ 533790909326: 
[2022-01-05T09:39:55.864] [INFO] default - processSegmentRLog GpsLocationExternal @ 534790931982: 
[2022-01-05T09:39:56.027] [INFO] default - processSegmentRLog GpsLocationExternal @ 535790971044: 
[2022-01-05T09:39:56.205] [INFO] default - processSegmentRLog GpsLocationExternal @ 536790962710: 
[2022-01-05T09:39:56.373] [INFO] default - processSegmentRLog GpsLocationExternal @ 537790205783: 
[2022-01-05T09:39:56.553] [INFO] default - processSegmentRLog GpsLocationExternal @ 538790947397: 
[2022-01-05T09:39:56.727] [INFO] default - processSegmentRLog GpsLocationExternal @ 539791373021: 
[2022-01-05T09:39:56.897] [INFO] default - processSegmentRLog GpsLocationExternal @ 540794499167: 
[2022-01-05T09:39:57.057] [INFO] default - processSegmentRLog GpsLocationExternal @ 541789592604: 
RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 12. Received 16
    at boundsError (internal/buffer.js:49:9)
    at Buffer.readUInt32LE (internal/buffer.js:105:5)
    at Object.readSize (/footage/retropilot/node_modules/capnp-split/buffer.js:19:23)
    at CapnpStream.readNextMessage (/footage/retropilot/node_modules/capnp-split/stream.js:18:23)
    at CapnpStream._write (/footage/retropilot/node_modules/capnp-split/stream.js:34:15)
    at doWrite (_stream_writable.js:415:12)
    at writeOrBuffer (_stream_writable.js:399:5)
    at CapnpStream.Writable.write (_stream_writable.js:299:11)
    at DestroyableTransform.ondata (/footage/retropilot/node_modules/readable-stream/lib/_stream_readable.js:619:20)
    at DestroyableTransform.emit (events.js:198:13)
    opened by mabnz 0
  • Implementing ORM, JWT for sessions, pair w/o imei

    Implementing ORM, JWT for sessions, pair w/o imei

    Moved logic out of routes, moving SQL queries from direct SQL to Sequelize ORM. Sessions are now stored as a JWT, future expansion will allow users to invaldiate previous sessions.

    opened by AdamSBlack 0
Owner
Florian Brede
Florian Brede
GitHub
Started pack for working with the new GameTest Framework API. Usable in windows, and mobile in worlds and realms!

GameTest FrameWork GameTest FrameWork is a new feature in Minecraft Bedrock Edition. This gives you the ability to script! In this example I will be u

null 40 Dec 24, 2022
A CLI to add and remove local development environments and create HTTPS certificates for them.

A CLI used to create a local https dev environment with the green lock. Setup takes less than 5 minutes and all of your projects will be running locally over HTTPS with custom domain names in no time!

Entrostat (Pty) Ltd 5 Sep 19, 2022
Find and fix problems in your JavaScript code.

ESLint Website | Configuring | Rules | Contributing | Reporting Bugs | Code of Conduct | Twitter | Mailing List | Chat Room ESLint is a tool for ident

ESLint 22k Jan 8, 2023
Infinite Red's cutting edge React Native project boilerplate, along with a CLI, component/model generators, and more!

Ignite - the hottest React Native boilerplate Battle-tested React Native boilerplate The culmination of five years of constant React Native developmen

Infinite Red, Inc. 14.7k Dec 29, 2022
Test your internet connection speed and ping using speedtest.net from the CLI

speed-test Test your internet connection speed and ping using speedtest.net from the CLI Install Ensure you have Node.js version 8+ installed. Then ru

Sindre Sorhus 3.8k Jan 7, 2023
Detect copy-pasted and structurally similar code

Detect copy-pasted and structurally similar JavaScript code. Requires Node.js 6.0+, and supports ES6, JSX as well as Flow. Note: the project has been

Daniel St. Jules 3.5k Dec 26, 2022
Move files and directories to the trash

Move files and folders to the trash Works on macOS (10.12+), Linux, and Windows (8+). Note: The Linux implementation is not very good and not maintain

Sindre Sorhus 2.4k Dec 29, 2022
Pipeable javascript. Quickly filter, map, and reduce from the terminal

Pipeable JavaScript - another utility like sed/awk/wc... but with JS! Quickly filter, map and reduce from the command line. Features a streaming API.

Daniel St. Jules 410 Dec 10, 2022
Kill all Chrome tabs to improve performance, decrease battery usage, and save memory

kill-tabs Kill all Chrome tabs to improve performance, decrease battery usage, and save memory Works on macOS, Linux, Windows. I'm a tab-abuser and I

Sindre Sorhus 371 Jan 8, 2023
📜 Create mutable log lines into the terminal, and give life to your logs!

Because Logging can be pretty and fun Installation $ npm install draftlog What it does It allows you to re-write a line of your log after being writt

Ivan Seidel 1.2k Dec 31, 2022
HMSC (How Much Stuffs CLI) analyst for your files and folders

HMSC ?? About HMSC (How Much Stuffs CLI) analyst for your files and folders ?? Screenshot ?? Requirements Node.js NPM ?? Installation $ npm i -g hmsc

Abdullah Veliyev 26 Jan 10, 2022
Add a hungry turtle to your terminal and feed it every time you mistype 'npm' as 'nom'

Nom Does this ever happen to you? You happily code away on a project, navigating the command line like a pro, testing, error logging, installing packa

Meike Hankewicz 5 Apr 26, 2022
Tasks Management CLI application with Nodejs, Mongodb, inquirer.js, and commander

Tasks CLI Tasks CLI is a program to manage your tasks in a database using terminal or console. This is a sample project for beginners Requirements Nod

Fazt Web 9 Nov 17, 2022
A simple CLI tool to create and manage xhelpers-api projects

A simple CLI tool to create and manage xhelpers-api projects

null 2 Feb 25, 2022
A fullstack(NestJS、React) framework for building efficient and scalable applications

A fullstack(NestJS、React) framework for building efficient and scalable applications. Description The Kunlun CLI is a command-line interface tool that

图灵人工智能研究院前端技术团队 3 Mar 12, 2022
Wordle and Termooo style classic word guessing game for the command line. One new word per day!

Wordle and Termooo style classic word guessing game for the command line. One new word per day!

Anderson Silva 3 Nov 27, 2022
Run a command when a certain file exists, and/or watch files to rerun on changes

Run a command when a certain file exists, and/or watch files to rerun on changes

EGOIST 45 Sep 23, 2022
SFDX Plugin to set Email Deliverability Access Level for an org easily and quickly via command line interface

SFDX Plugin to set Email Deliverability Access Level for an org easily and quickly via command line interface

Max Goldfarb 11 Dec 16, 2022
Keep track of book descriptions on the server-side using MongoDB, Express, and Node.js.

Backend Application A database model/backend for a user directory using Javascript MongoDB, Express, and Node.js. In summary, a backend CRUD model to

Rodrigo Bravo 3 Apr 10, 2022