javascript embeddable / in-memory database

Overview

LokiJS

The super fast in-memory javascript document oriented database.

Enable offline-syncing to your SQL/NoSQL database servers with SyncProxy !! Code-free real time syncing, ideal for mobile, electron and web apps.

Join the chat at https://gitter.im/techfort/LokiJS alt CI-badge npm version alt packagequality

Overview

LokiJS is a document oriented database written in javascript, published under MIT License. Its purpose is to store javascript objects as documents in a nosql fashion and retrieve them with a similar mechanism. Runs in node (including cordova/phonegap and node-webkit), nativescript and the browser. LokiJS is ideal for the following scenarios:

  1. client-side in-memory db is ideal (e.g., a session store)
  2. performance critical applications
  3. cordova/phonegap mobile apps where you can leverage the power of javascript and avoid interacting with native databases
  4. data sets loaded into a browser page and synchronised at the end of the work session
  5. node-webkit desktop apps
  6. nativescript mobile apps that mix the power and ubiquity of javascript with native performance and ui

LokiJS supports indexing and views and achieves high-performance through maintaining unique and binary indexes (indices) for data.

Demo

The following demos are available:

  • Sandbox / Playground
  • a node-webkit small demo in the folder demos/desktop_app. You can launch it by running /path/to/nw demos/desktop_app/

Wiki

Example usage can be found on the wiki

Main Features

  1. Fast performance NoSQL in-memory database, collections with unique index (1.1M ops/s) and binary-index (500k ops/s)
  2. Runs in multiple environments (browser, node, nativescript)
  3. Dynamic Views for fast access of data subsets
  4. Built-in persistence adapters, and the ability to support user-defined ones
  5. Changes API
  6. Joins

Current state

LokiJS is at version 1.3 [Eostre].

As LokiJS is written in JavaScript it can be run on any environment supporting JavaScript such as browsers, node.js/node-webkit, nativescript mobile framework and hybrid mobile apps (such as phonegap/cordova).

Made by @techfort, with the precious help of Dave Easterday.

Leave a tip or give us a star if you find LokiJS useful!

Installation

For browser environments you simply need the lokijs.js file contained in src/

You can use bower to install lokijs with bower install lokijs

For node and nativescript environments you can install through npm install lokijs.

Roadmap

  • exactIndex
  • key-value datastore
  • MRU cache
  • MongoDB API compatibility
  • server standalone (tcp and http servers and clients)
  • replication and horizontal scaling

Contact

For help / enquiries contact [email protected]

Commercial Support

For commercial support contact [email protected]

License

Copyright (c) 2015 TechFort [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Comments
  • Joins

    Joins

    At the moment Loki supports in-memory joins e.g.

    var author = authors.insert({name: 'Stephen King'});
    var book = books.insert({title: 'Salems Lot', author: author})
    

    But this will not persist correctly as the author will be serialized as an embedded object, not the original author object. There should be a more solid mechanism to operate joins.

    opened by techfort 67
  • Bulk Inserts and Collection Constraints

    Bulk Inserts and Collection Constraints

    I don't think I'm using indexes correclty.

    When merging 10,000+ records from a remote database into a local database, I have to use the id (not the $loki id) to make sure the document is not already there. (if its there I want to merge instead of update). To do this effeciently, I'm trying to use Loki indices:

    let loki = new Loki('testdb');
    myCollection = loki.addCollection('myCollection', {indices:['id']})
    

    Then on each insert I do

    localRecord = myCollection.findOne({id: remoteRecord.id})
    if( localRecord ) {
      // Merge code
    } else {
      myCollection.insert( remoteRecord );  
    }
    

    When I disable the localRecord check and just insert, things are smoking fast! Only slowed by data fetching/paging.

    • http://screencast.com/t/H3wsetchhQ

    When I enable the findOne check, things slow down a bit:

    • http://screencast.com/t/gBJQ49yBJwW

    When I hack an index to the side that is a big hash of ids mapped to $loki I get really fast again.

    • http://screencast.com/t/V5R0g7hPt

    I think there must be something I don't understand about indexes.

    Please advise :)

    enhancement 
    opened by jrhicks 43
  • Documentation for HTTP wrapper

    Documentation for HTTP wrapper

    I couldn't see the documentation for the HTTP wrapper although it is mentioned in the features list. Would you mind pointing me in the right direction please?

    wontfix 
    opened by dukedougal 38
  • db save circular structure issue

    db save circular structure issue

    Currently I have a problem in the area of loading/saving database. After object update and try to save db the following error occurs:

    stream.js:94
          throw er; // Unhandled stream error in pipe.
                ^
    TypeError: Converting circular structure to JSON
        at Object.stringify (native)
        at LokiEventEmitter.Loki.serialize (/home/teef/leonardo/WebApplications/Leonardo-CommHub/node_modules/lokijs/src/lokijs.js:385:19)
        at LokiEventEmitter.Loki.saveDatabase (/home/teef/leonardo/WebApplications/Leonardo-CommHub/node_modules/lokijs/src/lokijs.js:744:47)
    

    I am using one collection as well as one dynamic view.

    Seems like there are two dynamic views which should not happen.

    DynamicViews: 
       [ { collection: [Circular],
           name: 'aciveDevices',
           persistent: false,
           resultset: [Object],
           resultdata: [],
           resultsdirty: true,
           cachedresultset: null,
           filterPipeline: [],
           sortFunction: null,
           sortCriteria: null,
           sortDirty: false,
           events: [Object] },
         { collection: [Circular],
           name: 'aciveDevices',
           persistent: false,
           resultset: [Object],
           resultdata: [],
           resultsdirty: false,
           cachedresultset: null,
           filterPipeline: [Object],
           sortFunction: null,
           sortCriteria: null,
           sortDirty: false,
           events: [Object] } ],
    

    Database: https://gist.github.com/teef/c3f8f2139a1e13188e09

    opened by teef 35
  • Querying empty collection with binary index throws error

    Querying empty collection with binary index throws error

    If a collection has a binary index set, and the collection is empty, querying it will result in an error. The error happens in the calculateRange function on the following lines:

    var minVal = rcd[index[min]][prop]; var maxVal = rcd[index[max]][prop];

    If the collection is empty, then rcd[index[min/max]] will be undefined. We may want to set default min and max values here, but also, maybe shortcut the find() function to return an empty result set if the data is empty to save some work.

    opened by assafweinberg 32
  • lokiFSAdaptor

    lokiFSAdaptor

    Not an issue as such so sorry but I was wondering if there was an example anywhere of using loki with window.requestFileSystem/window.webkitRequestFileSystem...? My aim is for my DB to persists in a Cordova app by saving it to the FileSystem... I'm developing on Chrome so I could save it to PERSISTENT storage on the browser as well. I've looked all over and it seems as though I'll need to write my own adapter but I was wondering if anyone had already done so so I could cast my eye over their solution?

    opened by annoyingmouse 31
  • Can't get persistent data in Cordova (using cordova-fs-adapter)

    Can't get persistent data in Cordova (using cordova-fs-adapter)

    When I serve the app using the Phonegap Desktop app, Chrome pops up a dialog asking me to allow storing files on this device. So, that's promising... and everything appears to work as expected, but the data isn't persistent. If I refresh the page, all data is lost. This is also the case when I build the app using Phonegap Build and test on my iPhone. If I close the app, data is lost.

    I've noticed an FS adapter console error: Unable to resolve local file system URL, but I'm not sure how to fix that.

    Here's how I'm creating the database:

    var loki = require('lokijs'),
        products,
        prices;
    
    var LokiCordovaFSAdapter = require("loki-cordova-fs-adapter");
    
    var adapter = new LokiCordovaFSAdapter({"prefix": "loki"}),
        db = new loki('pricelog_db', {
            autosave: true,
            autosaveInterval: 1000,
            adapter: adapter
        });
    

    Which is all in a store.js file, which is then imported into my main js file (this is a Vue app). I'm confused about the whole device ready thing, and wonder if this is related to that? Sorry if I'm leaving out any relevant info or have made some glaring mistake. But any advice would be appreciated, thanks!

    opened by chasebank 28
  • Git-js Store

    Git-js Store

    Enable Git style versioning to reduce network overhead on update 'pull'.

    This applies to use cases that may need to receive updates over the network. Loki has the ability to track and apply changes, which pretty much solves the 'push' aspect. However, to create the changes there is currently the necessity of a direct local comparison. This means at some point the client side has to either dump the data or send the entire set over. I think it would be very useful to store version information that a server could use to dynamically diff the objects and return patches or changes that Loki can then apply.

    My thought on the possible solutions were - Tweak the current Loki changes API to dynamically track App Level* version hashes Or Create a JS Git store. https://github.com/creationix/js-git

    (sorry, hope its okay to post potential enhancements)

    enhancement 
    opened by PseudoCorps 27
  • Add upserts

    Add upserts

    Upserts would be nice, and it sounds from #79 like they're on the way.

    I'm not good enough with JS to attempt a patch, but I can share a snippet I'm using to do upserts if anyone wants to use it until LokiJS has them baked in.

    I can't guarantee how fast or robust it is, but it's working for me.

    Note that it requires lodash.

    /**
     * Performs an upsert.
     * This means performing an update if the record exists, or performing an
     * insert if it doesn't.
     * LokiJS (as at version 1.2.5) lacks this function.
     * TODO: Remove this function when LokiJS has built-in support for upserts.
     * @param {object} collection - The target DB collection.
     * @param {string} idField - The field which contains the record's unique ID.
     * @param {object} record - The record to be upserted.
     * @depends lodash
     */
    function upsert(collection, idField, record) {
      var query = {};
      query[idField] = record[idField];
      var existingRecord = collection.findOne(query);
      if (existingRecord) {
        // The record exists. Do an update.
        var updatedRecord = existingRecord;
        // Only update the fields contained in `record`. All fields not contained
        // in `record` remain unchanged.
        _.forEach(record, function(value, key){
          updatedRecord[key] = value;
        });
        collection.update(updatedRecord);
      } else {
        // The record doesn't exist. Do an insert.
        collection.insert(record);
      }
    }
    
    wontfix 
    opened by countermeasure 26
  • Binary indices occasionally corrupt

    Binary indices occasionally corrupt

    (continuation from #47)

    I have users where the lokijs binary indices occasionally corrupt. It's hard to find out how this happens -- it happens infrequently, and a long-running log to catch is would generate very large files for all users. I can't reproduce this at will. Right now I'd be happy to see 1.3.5 drop which has detect and repair functions.

    wontfix 
    opened by retorquere 25
  • IndexedDB massive storage use

    IndexedDB massive storage use

    We are experiencing on some devices that IndexedDB takes up a massive amount of storage, even though the total size of the LokiJS entities are just a few megabytes. Here is a screenshot from a Windows 10 PC with latest version of Chrome (61), as shown in the screenshot the amount of storage reported by .getCatalogSummary() is 3.5 MB. The usage reported by Chrome for IndexedDB, is a massive 2944 MB.

    image

    This is a web app that is also embedded into a Cordova app for Android and iOS. On iOS devices, the used storage is growing at a rapid pace and reporting gigabytes of used storage after some usage of the app.

    LokiJS version: 1.5.0 Angular: 4.3.6

    The app uses Web Sockets between app and services, and receives many updates to entities stored in LokiJS. So the total storage is not a lot, but the entities are updated many times each minute (some of the entities are).

    We use the LokiIndexedAdapter, with autosave and autoload, but .saveDatabase() is also called a lot on individual updates that are received over Web Sockets.

    This happens on just a few desktop PC devices, but appears to be consistent behavior on iOS devices. Same behavior on both iOS 10 and iOS 11.

    There are unfortunately few tools and APIs available to debug what is happening within IndexedDB, does anyone knows any tools or tricks to debug such an issue?

    I'm building a new Cordova app to test a comparison between LokiJS and direct IndexedDB API, to see if many updates will increase the used storage on iOS.

    Additional comment:

    Normally the reported size used by IndexedDB are 10-20MB.

    opened by sondreb 24
  • The unit test of `sotringIndexing` is not correctly implemented

    The unit test of `sotringIndexing` is not correctly implemented

    The variable len is defined here: https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/sortingIndexing.spec.js#L307

    But the value is not replaced here: https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/sortingIndexing.spec.js#L316-L320

    Here: https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/sortingIndexing.spec.js#L323-L327

    The code is testing the same result for three times, which is incorrect.

    opened by Losses 0
  • `compoundEval` is not correctly implemented

    `compoundEval` is not correctly implemented

    Prosperities value is passed to compoundEval here, the type signature of the value is ([keyof Document, isDesc] | keyof Document)[].

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L3364-L3369

    But compoundEval only support [keyof Document, isDesc][]:

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L385

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L394

    But do not support (keyof Document)[].

    opened by Losses 0
  • Unit test of `persistence` is not correctly implemented

    Unit test of `persistence` is not correctly implemented

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/persistence.spec.js#L118-L144

    Since the type of idx is number, so all methods wound not be tested.

    opened by Losses 0
  • `simpleSort` with `useJavascriptSorting` don't support `dotNotation`

    `simpleSort` with `useJavascriptSorting` don't support `dotNotation`

    Here's the implementation:

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L3295-L3301

    But the unit test is trying to use this feature here:

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/ops.spec.js#L251-L253

    opened by Losses 0
  • Incorrect implementation of `inflate`

    Incorrect implementation of `inflate`

    Based on the unit test:

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/immutable.spec.js#L1051-L1059

    Loki should read the returned value of inflater as the returned data, but the implementation didn't do this:

    https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L1741-L1745

    The code should be:

              return function (data) {
                var collObj = new (collOptions.proto)();
                return inflater(data, collObj) ?? collObj;
              };
    
    opened by Losses 0
Owner
Joe Minichino
ML/AI/Big Data, Unix and Heavy Metal. Data Architect @ Teamwork.com
Joe Minichino
Embeddable 3D Rendering Engine with JS, a POC project.

Three.V8 Three.V8 is a proof of concept (POC) of an embedabble 3D rendering engine using JavaScript as user script. Currently, the whole project requi

Fei Yang 24 Nov 29, 2022
An embeddable, programmable order book framework

Verto Flex An embeddable, programmable order book framework Installation npm install @verto/flex or yarn add @verto/flex Prerequisites Your SmartWeave

The Verto Protocol 9 Oct 23, 2022
startupDB is an Express middleware function implementing a high-performance in-memory database

startupDB startupDB is a database designed to create REST APIs. It is implemented as an Express middleware function and allows for easy implementation

Jeroen de Vries 8 Jul 26, 2022
An exercise in building a very minimal (and very stupid) in-memory SQL-like database for educational purposes.

Stupid Database This is an exercise in building a very minimal (and very stupid) in-memory SQL-like database for educational purposes. None of this co

Fabio Akita 196 Dec 20, 2022
In-memory abstract-level database for Node.js and browsers.

memory-level In-memory abstract-level database for Node.js and browsers, backed by a fully persistent red-black tree. The successor to memdown and lev

Level 14 Dec 27, 2022
ClientDB is an open source in-memory database for enabling real-time web apps.

ClientDB ClientDB is an open source in-memory database for enabling real-time web apps. Build fast, scalable apps that feel silky smooth for users. In

Acapela 540 Dec 24, 2022
ClientDB is an open source in-memory database for enabling real-time web apps.

ClientDB ClientDB is an open-source in-memory database for enabling real-time web apps. Build fast, scalable apps that feel silky smooth for users. In

null 436 Aug 27, 2022
🗃️ An in-memory JS database optimized for large scale storage on the frontend.

BlinkDB is a in-memory JS database optimized for large scale storage on the frontend. import { many } from 'blinkdb'; const items = await many(userTa

BlinkDB 64 Dec 23, 2022
Interplanetary Database: A Database built on top of IPFS and made immutable using Ethereum blockchain.

IPDB IPDB (Interplanetary Database) is a key/value store database built on top of IPFS (Interplanetary File System). Project is intended to be an MVP

turinglabs 8 Oct 6, 2022
Visualize, modify, and build your database with dbSpy! An open-source data modeling tool to facilitate relational database development.

Visualize, modify, and build your database with dbSpy! dbSpy is an open-source data modeling tool to facilitate relational database development. Key F

OSLabs 115 Dec 22, 2022
Generate in-memory fake files with custom size

File generator Generate fake in-memory files for varying sizes This package allows you generate fake in-memory files for varying sizes. The generated

Poppinss 19 Nov 4, 2022
Brain wallet using both language and visual memory.

*Check out the big brain on Brett! You're a smart motherfvcker, that's right.* -- Pulp fiction ?? SUPER BRAIN WALLET ?? Use your brain power to the ma

Code's All Right ™ 4 Jun 9, 2022
fix out-of-memory actions at colors@^1.4.1

What's it for? colors@^1.4.1 has action of out-of-memory: https://github.com/Marak/colors.js/commit/5d2d242f656103ac38086d6b26433a09f1c38c75. https://

hoperyy 6 Jan 11, 2022
Horizontally scalable blockchain using STARK's and partitioned transactional memory

quark quark - quick STARK! A decentralized horizontally-scaled state machine that can transfer 1,000,000 unique tokens on Uniswap in a single atomic t

Liam Zebedee 56 Nov 25, 2022
🌌 Fast, in-memory, full-text search engine written in TypeScript. Now in beta.

Installation You can install Lyra using npm, yarn, pnpm: npm i @nearform/lyra yarn add @nearform/lyra pnpm add @nearform/lyra Usage Lyra is quite simp

NearForm 5.1k Dec 30, 2022
Train and test your brain with these memory games

Want to help grow your Hippocampus? Engage your Cerebrum? Try these games! Train and test your brain with these memory games. Each game is designed to help you increase your memory. How much can you memorize with each game? Test yourself.

Christian Schroeder 2 Jun 16, 2022
Memory lane is a website where you can create visual timelines of moments in time.

Memory Lane Moments in Time, All in One Place Description Memory Lane is a web app where users can create visually appealing, minimalistic timelines e

joey 3 Jun 13, 2022
Github action to collect metrics (CPU, memory, I/O, etc ...) from your workflows to help you debug and optimize your CI/CD pipeline

workflow-telemetry-action A GitHub Action to track and monitor the resource metrics of your GitHub Action workflow runs. If the run is triggered via a

Thundra 32 Dec 30, 2022
A flexible, memory compact, fast and dynamic CSG implementation on top of three-mesh-bvh

three-bvh-csg An experimental, in progress, flexible, memory compact, fast and dynamic CSG implementation on top of three-mesh-bvh. More than 100 time

Garrett Johnson 208 Jan 5, 2023