Replacement for comma.ai backend and useradmin dashboard

Overview

retropilot-server

Uptime Robot status Uptime Robot ratio (7 days) CI workflow

Replacement for comma.ai backend and useradmin dashboard. Bundled with a modified version of comma's cabana to allow viewing & analyzing drives.

If you don't want to host your own instance, check out https://api.retropilot.org/useradmin for a hosted version of the backend, useradmin and cabana.

[Server] Summary

The server consists of 2 node scripts.

  • src/server is using expressjs and runs the backend (file upload / communication with openpilot) and the useradmin dashboard to manage / view / download drives & logs.

  • src/worker is a background worker that is processing drives (analyzing video files & logs) to prepare drives for playback in cabana and to gather statistics. It automatically terminates itself after 60 minutes to make sure the video/log libraries do not cause memory leaks.

Attention: Minimum required node version is node 10.

[Server] Installation

npm install
cp config.sample.js config.js
cp database.empty.sqlite database.sqlite
> EDIT config.js

[Server] Running

node -r esm src/server
node -r esm src/worker

[Server] CABANA Support

A compiled version of a custom cabana fork (https://github.com/florianbrede-ayet/retropilot-cabana) is directly bundled in the cabana/ subdirectory and will be served by the express app. After starting index.js, cabana is ready to use.


[Device] Preparation / Enable Custom Server

On the device or in your fork's code, replace all API endpoints with your own server endpoint. This could be executed directly on the device in the shell to use https://api.retropilot.org as backend:

find /data/openpilot -type f -exec sed -i 's/https:\/\/api.commadotai.com/https:\/\/api.retropilot.org/g' {} +

[Device] Swapping Servers (Back)

To switch a device between different servers, you have to remove the old DongleId and reboot:

rm /data/params/d/DongleID
reboot

There is no need to backup the DongleId, as the new server will identify your device based on its imei, serial and public key.

[Device] Raw Drives Not Uploading (fcamera & rlog)

  1. Raw data is only uploaded if the device is sufficiently charged, not connected to an active panda (offroad) and there are no immediate files (boot, crash, qcamera, qlog) remaining.
  2. Your branch might have raw uploads disabled, check Device Settings > Upload Raw Logs.

If that doesn't help or the option is not available, try:

/data/params/d/IsUploadRawEnabled echo "1" > /data/params/d/UploadRaw reboot">
echo "1" > /data/params/d/IsUploadRawEnabled
echo "1" > /data/params/d/UploadRaw
reboot

Current Limitations

OpenPilot before 0.8.3 will not display any statistics or pairing status in the dashboard. The reason is that pre 0.8.3, the offroad.apk with react and comma-api would require recompilation to accept the new endpoints.

The athena websockets interface is not implemented yet, so the comma app and athena specific remote control commands (including "upload on demand") are not functional as of now.

Screenshots

image

image

image

image

UAT

Launch with:

docker-compose -f docker-compose.yml -f docker-compose.uat.yml up -d
You might also like...

A beautiful, responsive, highly customizable and accessible replacement for JavaScript's popup boxes. Zero dependencies.Alerts ,dialogs

AsgarAlert (v1) for JS Install script defer src="/asgar-alert.js"/script Examples The most basic message: asgar("Hello world!"); A message signali

Dec 20, 2022

Automatic arxiv-ar5iv link replacement in Chrome.

Automatic arxiv-ar5iv link replacement in Chrome. This chrome extension will automatically replace arxiv.org/pdf/* links with ar5iv links for more we

Oct 29, 2022

A Drop-in Jalali Replacement for filament DateTimePicker

Filament Jalali Date Time Picker Field This package is a Drop-in replacement for DatePicker and DateTimePicker field type you just need to replace tho

Dec 3, 2022

Pretty, customisable, cross browser replacement scrollbars

jScrollPane - cross browser custom scroll bars jScrollPane is a jQuery plugin which allows you to replace a browser's default scroll bars (on an eleme

Dec 15, 2022

Lightweight plugin for easy responsive images replacement

Responsive image replacement Check out the example here. resonsive-img.js is a lightweight plugin for fast, clean and easy responsive image replacemen

Sep 20, 2022

An efficient drop-in replacement for JSON.

An efficient drop-in replacement for JSON.

JCOF: JSON-like Compact Object Format A more efficient way to represent JSON-style objects. Status This format isn't nailed down yet. Most changes wil

Nov 26, 2022

A Cypress plugin that generates test scripts from your interactions, a replacement Cypress Studio for Cypress v10 🖱 ⌨

A Cypress plugin that generates test scripts from your interactions, a replacement Cypress Studio for Cypress v10 🖱 ⌨

DeploySentinel Cypress Recorder Plugin Create Cypress tests scripts within the Cypress test browser by simply interacting with your application, simil

Dec 15, 2022

shell script replacement; write shell scripts in js instead of bash, then run them with a single static binary

yavascript YavaScript is a bash-like script runner which is distributed as a single statically-linked binary. Scripts are written in JavaScript. There

Dec 29, 2022

This package is a replacement for superjson to use in your Remix app

This package is a replacement for superjson to use in your Remix app

This package is a replacement for superjson to use in your Remix app. It handles a subset of types that superjson supports, but is faster and smaller.

Jan 3, 2023
Comments
  • WIP work for new server

    WIP work for new server

    New server code from scratch, provides all comma.ai endpoints via NextJS. Connects to the database using prisma.

    GOALS:

    • It should replace the existing code without any issues
    • Reuse datasource and storage
    • Keep old users registered and start to process drives that previously failed

    Work

    • [x] Basic API resolve
    • [x] Connect to the dabase
    • [x] device registration
    • [x] device upload path generation
    • [ ] user registration
    • [ ] link device with user
    opened by jor3l 0
  • User registration potential for duplicates

    User registration potential for duplicates

    There's nothing directly preventing a user to register with the same email other than the "unique" attribute in the database.

        at Query.run (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\sequelize\lib\dialects\postgres\query.js:50:25)
        at C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\sequelize\lib\sequelize.js:313:28
        at processTicksAndRejections (internal/process/task_queues.js:95:5)
        at async PostgresQueryInterface.insert (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\sequelize\lib\dialects\abstract\query-interface.js:297:21)
        at async model.save (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\sequelize\lib\model.js:2417:35)
        at async Function.create (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\sequelize\lib\model.js:1329:12)
        at async C:\Users\adam\Documents\Retropilot\public\retropilot-server\dist\server\router\useradmin.js:174:16 {
      name: 'SequelizeUniqueConstraintError',
      errors: [
        ValidationErrorItem {
          message: 'email must be unique',
          type: 'unique violation',
          path: 'email',
          value: '[email protected]',
          origin: 'DB',
          instance: [accounts],
          validatorKey: 'not_unique',
          validatorName: null,
          validatorArgs: []
        }
      ],
      parent: error: duplicate key value violates unique constraint "accounts_un"
          at Parser.parseErrorMessage (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\parser.js:287:98)
          at Parser.handlePacket (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\parser.js:126:29)
          at Parser.parse (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\parser.js:39:38)
          at Socket.<anonymous> (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\index.js:11:42)
          at Socket.emit (events.js:400:28)
          at addChunk (internal/streams/readable.js:293:12)
          at readableAddChunk (internal/streams/readable.js:267:9)
          at Socket.Readable.push (internal/streams/readable.js:206:10)
          at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
        length: 204,
        severity: 'ERROR',
        code: '23505',
        detail: 'Key (email)=([email protected]) already exists.',
        hint: undefined,
        position: undefined,
        internalPosition: undefined,
        internalQuery: undefined,
        where: undefined,
        schema: 'public',
        table: 'accounts',
        column: undefined,
        dataType: undefined,
        constraint: 'accounts_un',
        file: 'nbtinsert.c',
        line: '670',
        routine: '_bt_check_unique',
        sql: 'INSERT INTO "accounts" ("id","email","password","created","admin") VALUES (DEFAULT,$1,$2,$3,$4) RETURNING "id","email","password","created","last_ping","2fa_token","admin","email_verify_token","g_oauth_sub","two_factor_enabled";',
        parameters: [
          '[email protected]',
          'password_hash***',
          1651328540828,
          false
        ]
      },
      original: error: duplicate key value violates unique constraint "accounts_un"
          at Parser.parseErrorMessage (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\parser.js:287:98)
          at Parser.handlePacket (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\parser.js:126:29)
          at Parser.parse (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\parser.js:39:38)
          at Socket.<anonymous> (C:\Users\adam\Documents\Retropilot\public\retropilot-server\node_modules\pg-protocol\dist\index.js:11:42)
          at Socket.emit (events.js:400:28)
          at addChunk (internal/streams/readable.js:293:12)
          at readableAddChunk (internal/streams/readable.js:267:9)
          at Socket.Readable.push (internal/streams/readable.js:206:10)
          at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
        length: 204,
        severity: 'ERROR',
        code: '23505',
        detail: 'Key (email)=([email protected]) already exists.',
        hint: undefined,
        position: undefined,
        internalPosition: undefined,
        internalQuery: undefined,
        where: undefined,
        schema: 'public',
        table: 'accounts',
        column: undefined,
        dataType: undefined,
        constraint: 'accounts_un',
        file: 'nbtinsert.c',
        line: '670',
        routine: '_bt_check_unique',
        sql: 'INSERT INTO "accounts" ("id","email","password","created","admin") VALUES (DEFAULT,$1,$2,$3,$4) RETURNING "id","email","password","created","last_ping","2fa_token","admin","email_verify_token","g_oauth_sub","two_factor_enabled";',
        parameters: [
          '[email protected]',
          'password_hash*****',
          1651328540828,
          false
        ]
      },
      fields: { email: '[email protected]' },
      sql: 'INSERT INTO "accounts" ("id","email","password","created","admin") VALUES (DEFAULT,$1,$2,$3,$4) RETURNING "id","email","password","created","last_ping","2fa_token","admin","email_verify_token","g_oauth_sub","two_factor_enabled";'
    }
    TypeError: Cannot read property 'dataValues' of undefined
        at C:\Users\adam\Documents\Retropilot\public\retropilot-server\dist\server\router\useradmin.js:181:16
        at processTicksAndRejections (internal/process/task_queues.js:95:5)```
    bug enhancement 
    opened by AdamSBlack 1
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
A monorepo for comma.ai web interfaces and packages

comma webapps This mono-repository contains the web applications and packages for the web UIs of comma.ai Contributing Just pick something and work on

null 5 Sep 27, 2022
no-comma is a javascript library for dealing with inputted numbers that include commas

no-comma no-comma is a javascript library for dealing with inputted numbers that include commas. Nocomma will allow you to check if the number contain

Fatty 3 Jan 27, 2022
Dashboard skeleton Simple and fast dashboard skeleton template

Dashboard skeleton Simple and fast dashboard skeleton template. Installation npm install --save dashboard-skeleton-compostrap Version 1x built on Boo

Compostrap 9 Aug 23, 2022
Venni backend - The backend of the Venni client apps implementing the credit card payments, matching algorithms, bank transfers, trip rating system, and more.

Cloud Functions Description This repository contains the cloud functions used in the Firebase backend of the Venni apps. Local Development Setup For t

Abrantes 1 Jan 3, 2022
Gatsby-Formik-contact-form-with-backend-panel - Full working contact form with backend GUI panel.

Gatsby minimal starter ?? Quick start Create a Gatsby site. Use the Gatsby CLI to create a new site, specifying the minimal starter. # create a new Ga

Bart 1 Jan 2, 2022
A Hackable Markdown Note Application for Programmers. Version control, AI completion, mind map, documents encryption, code snippet running, integrated terminal, chart embedding, HTML applets, plug-in, and macro replacement.

Yank Note A hackable markdown note application for programmers Download | Try it Online >>> Not ecommended English | 中文说明 [toc]{level: [2]} Highlights

洋子 4.3k Dec 31, 2022
A drop in replacement for Hacker News with support for dark mode and more.

Worker News A drop in replacement for Hacker News with support for dark mode, quotes in comments, user identicons and submission favicons. What's cool

Worker Tools 18 Dec 31, 2022