startupDB is an Express middleware function implementing a high-performance in-memory database

Overview

startupDB

startupDB is a database designed to create REST APIs. It is implemented as an Express middleware function and allows for easy implementation of persistent data endpoints. It features protection from dataloss during hardware failure by persisting individual operations in JSON files and offers high performance by serving all data from memory.

Its CRUD operations map directly to POST, GET, UPDATE/PATCH and DELETE methods.

Usage

Example

const express = require('express')
const startupDB = require('startupDB')

const app = express()
app.use("/myDB", startupDB.db)
const server = app.listen(3000)

This will create a database under the myDB directory. Every endpoint that starts with /myDB will translate to a collection with the same name. So localhost:3000/myDB/user will implement POST, GET, PUT, DELETE and PATCH endpoints to create, find, update, delete and change user documents. Data will be persisted to disk in checkpoint/user and oplog/user directories.

API

const startupDB = require('startupDB')

startupDB([options])

Returns the database middleware using the given options

Options

The options object allows you to configure the following options:

Option Name Type Default value Description
dataFiles string './' Path to data directory
validator function undefined Function to validate schema
addTimeStamps function undefined Function to add create/modified timestamps.
opLogArchive string undefined Path to archive directory

Schema validation

A schema validator can be passed using the options.validator function.

Your function should implement the following interface:

/*
 * @param {string} operation: "created" or "modified"
 * @param {object} document: the document to change
 * @return false | array: false or an array with error messages
 */
validator(collection, documents)

Timestamps

startupDB can auto-timestamp your documents using the options.addTimeStamps function.

This function will be called when documents are created or modified. The timestamp function wil be called before your documents will be validated so make sure your schema understand your timestamps.

Your function should implement the following interface:

/*
 * @param {string} operation: "created" or "modified"
 * @param {object} document: the document to change
 * @param {object} oldDocument: the old document (before modify)
 */
function(operation,document,oldDocument)

Example

function (operation, object, oldObject) {
        if (operation == "created") object.__created = new Date().getTime()
        if (operation == "modified") {
            object.__modified = new Date().getTime()
            if (oldObject) object.__created = oldObject.__created
        }
    }

Hooks

startupDB support databasehooks to run endpoint specific code either before or after the CRUD operation. They can be used for everything from authentication to data conversion.

A 'before' hook should implement the following interface:

/*
 * @param {object} req: like in Express
 * @param {object} res: like in Express
 * @param {function} next: like in Express
 * @param {string} collection: the name of the collection
 *
 * @return:  {"statusCode":<HTTP StatusCode>,"data":<response body>,"message":<status message>}
 *
 * return {"statusCode":0} when there are no errors
 *
 */
function(req, res, next){
    return {
        "statusCode":200,
        "data":{
            "name":"value"
        },
        "message":"OK"
        }
}

An 'after' hook should implement the following interface:

/*
 * @param {object} req: like in Express
 * @param {object} response: response object from database
 *
 * @return:  {"error":<HTTP StatusCode>,"data":<response body>,"message":<status message>,"headers":<response headers>}
 *
 * Omit the error property in the response when there are no errors
 */
function(req, response){
    return {
        "data":response.data
        }
}

Commands

startupDB supports several commands that can be executed by sending a POST or GET request to the root.

For example:

curl --header "Content-Type: application/json" \
  --request POST \
  --data "{\"command\":\"purgeOplog\",\"collection\":\"00000/sku\"}" \
  http://127.0.0.1:3000/data
Command Method Function Parameters
GET List all collections.
create POST Create collection, errors when it already exists. collection:"string", storageType:"array" or "object"
drop POST Removes a collection from memory, oplog and checkpoint directories. collection:"string"
ensureCollection POST Create collection if it does not exist, no error if it does. collection:"string", storageType:"array" or "object"
flush POST Create checkpoint and flush oplog. collection:"string"
inspect POST return totalNrObjects in memory.
purgeOplog POST remove all operations from opLog, restoring collection to previous checkpoint. This is usefull for implementating tests. collection:"string"
Comments
  • Server crashes

    Server crashes

    When an operation is performed on a collection that is among the least recently used collection, the GC may decide to delete it causing the application to fail because it expects the data to be available.

    We need to 'touch' the collection before the actual transaction is written to keep the GC from interfering.

    bug 
    opened by bitsplitter 1
  • Error during near-simultaneous access

    Error during near-simultaneous access

    When accessing a resource that is not in memory yet, with multiple requests simultaneously, different errors are thrown, all boiling down to the resource not (completely) loaded in the in-memory cache.

    opened by bitsplitter 1
  • Negative RAM measurement

    Negative RAM measurement

    While testing the RAM usage of the startupDB, I've got in some how a negative RAM usage which is not possible to be. So please fix this bug.

    usedBytesInMemory: -157573965

    bug 
    opened by ahmadshuraih 0
  • Collections are garbage collected too often

    Collections are garbage collected too often

    usedBytesInMemory is updated with a wrong number when a collection loads into memory. As a result, the garbage collection decides to evict collections for no good reason.

    bug 
    opened by bitsplitter 0
  • Server can crash during GC

    Server can crash during GC

    Due to the inaccuracy in memory footprint estimation, GC can draw the wrong conclusion about available memory and evict all collections, even the ones that are in the process of being read form disk.

    bug 
    opened by bitsplitter 0
  • Lack of lastAccessed property on a collection being loaded causes it to be GC'd by startupDBGC

    Lack of lastAccessed property on a collection being loaded causes it to be GC'd by startupDBGC

    When a collection is being loaded (async) startupDB can kick in for another request and decides to remove the new collection since that has no lastAccessed timestamp yet which implies that it is the oldest collection around making it eligible for GC.

    bug 
    opened by bitsplitter 0
  • x-last-oplog-id header returned by HEAD can be invalid

    x-last-oplog-id header returned by HEAD can be invalid

    When using the HEAD method to get the latest update information on a collection, the x-last-oplog-id header will return a wrong id when there are oplog files with different filename lengths. Oplog files are named .json, when retrieving all files from the oplog directory, at least on Windows, files are return sorted, that is, alphabetically. That does not give us the most recent file when not all filenames are the same length.

    bug 
    opened by bitsplitter 0
  • StartupDB throws an error when the updateObjects payload is undefined

    StartupDB throws an error when the updateObjects payload is undefined

    When I call updateObjects without a payload, I get the following error:

    STARTUPDB Error TypeError: Cannot read properties of undefined (reading 'id') at addIdsToItemsThatHaveNone (/Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:117:19) at dbUpdateObjects (/Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:569:5) at async /Users/jeroen/LatestCollectionServer/dist/hooks/beforeGetCampaign.js:24:5 at async processMethod (/Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:692:24) at async /Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:772:20

    As a developer, I would like a more relevant error that helps me pinpoint the issue quicker.

    bug 
    opened by bitsplitter 0
  • Make memory management configurable

    Make memory management configurable

    In order to make startupDB more generic, allow to configure its memory management.

    DB_CACHE_OVERHEAD_RATIO Determines the predicted memory footprint of JSON objects. The default value of 2 seems to work well in existing deployments but in more specific use-cases, another ratio might work better, for example if there are very deeply nested objects that have higher memory overhead.

    DB_CACHE_FRACTION Determines the fraction of memory used for the DB cache. If the NodeJS-Express API that embeds StartupDB needs a larger fraction of available memory allocated to NodeJS via the —-max-old-space-size parameter, this fraction needs to be configurable.

    opened by bitsplitter 0
Owner
Jeroen de Vries
Jeroen de Vries
A simple in-memory time-based cache for both objects and function execution.

What is this? A simple in-memory time-based cache for both objects and function execution. How do I install it? You can install it by using the follow

cadienvan 7 Dec 15, 2022
A simple in-memory key-value cache for function execution, allowing both sync and async operations using the same methods

A simple in-memory key-value cache for function execution, allowing both sync and async operations using the same methods. It provides an invalidation mechanism based both on exact string and regex.

cadienvan 10 Dec 15, 2022
Express middleware for easy OAuth with a variety of providers.

accounted4 Express middleware for easy OAuth2 with a variety of providers. accounted4 is intended to make it easy for developers to add third-party OA

Josh Moore 3 May 7, 2022
express-like middleware system for your remix loaders and actions

remix-middleware Add an express-like middleware stack to your remix loaders and actions! yarn add remix-middleware // ./app/middleware.server.ts expor

Eric Bower 26 Nov 22, 2022
An exercise in building a very minimal (and very stupid) in-memory SQL-like database for educational purposes.

Stupid Database This is an exercise in building a very minimal (and very stupid) in-memory SQL-like database for educational purposes. None of this co

Fabio Akita 196 Dec 20, 2022
In-memory abstract-level database for Node.js and browsers.

memory-level In-memory abstract-level database for Node.js and browsers, backed by a fully persistent red-black tree. The successor to memdown and lev

Level 14 Dec 27, 2022
javascript embeddable / in-memory database

LokiJS The super fast in-memory javascript document oriented database. Enable offline-syncing to your SQL/NoSQL database servers with SyncProxy !! Cod

Joe Minichino 6.4k Jan 9, 2023
ClientDB is an open source in-memory database for enabling real-time web apps.

ClientDB ClientDB is an open source in-memory database for enabling real-time web apps. Build fast, scalable apps that feel silky smooth for users. In

Acapela 540 Dec 24, 2022
ClientDB is an open source in-memory database for enabling real-time web apps.

ClientDB ClientDB is an open-source in-memory database for enabling real-time web apps. Build fast, scalable apps that feel silky smooth for users. In

null 436 Aug 27, 2022
🗃️ An in-memory JS database optimized for large scale storage on the frontend.

BlinkDB is a in-memory JS database optimized for large scale storage on the frontend. import { many } from 'blinkdb'; const items = await many(userTa

BlinkDB 64 Dec 23, 2022
A quickstart AWS Lambda function code generator. Downloads a template function code file, test harness file, sample SAM deffiniation and appropriate file structure.

Welcome to function-stencil ?? A quickstart AWS Lambda function code generator. Downloads a template function code file, test harness file, sample SAM

Ben Smith 21 Jun 20, 2022
Tries to execute sync/async function, returns a specified default value if the function throws

good-try Tries to execute sync/async function, returns a specified default value if the function throws. Why Why not nice-try with it's 70+ million do

Antonio Stoilkov 14 Dec 8, 2022
Wrap a function with bun-livereload to automatically reload any imports inside the function the next time it is called

bun-livereload Wrap a function with bun-livereload to automatically reload any imports inside the function the next time it is called. import liveRelo

Jarred Sumner 19 Dec 19, 2022
🖼️ Tiny JSX helper function for high-DPI (Retina Display) images.

img-srcx Tiny JSX helper function for high-DPI (Retina Display) images. 394 bytes minified and gzipped for ESM (830 bytes for ES5). Installation npm i

Chariz 4 Oct 6, 2022
High performance JSX web views for S.js applications

Surplus const name = S.data("world"), view = <h1>Hello {name()}!</h1>; document.body.appendChild(view); Surplus is a compiler and runtime to all

Adam Haile 587 Dec 30, 2022
AppRun is a JavaScript library for developing high-performance and reliable web applications using the elm inspired architecture, events and components.

AppRun AppRun is a JavaScript library for building reliable, high-performance web applications using the Elm-inspired architecture, events, and compon

Yiyi Sun 1.1k Dec 20, 2022
Crawler Crypto using NodeJS for performance with Elasticsearch DB for high efficiency.

Coin crawler - Coingecko version Crawler using NodeJS for performance with Elasticsearch DB for high efficiency. Requirements For development, you wil

Minh.N.Pham 1 Jan 20, 2022
High performance (de)compression in an 8kB package

fflate High performance (de)compression in an 8kB package Why fflate? fflate (short for fast flate) is the fastest, smallest, and most versatile pure

null 1.4k Dec 28, 2022
high performance、complex interaction table

功能描述 1、高性能、满足复杂交互的编辑表格 2、基于: antd4(https://ant.design/index-cn) ag-grid(https://www.ag-grid.com/) 3、基于原生ag-grid 的API进行封装 一、主要功能 将按下列顺序逐步迭代 1、通用编辑功能 ??

wheel-team 4 Feb 15, 2022