IndexedDB with usability and remote syncing

This is a fork of the awesome idb library, which adds the ability to sync an IndexedDB database with a remote REST API.

The source code for the example above can be found here.

Bundle size: ~3.11 kB brotli'd

Table of content

Features
Disclaimer
Installation
API
1. SyncManager
  1. Options
  2. Methods
2. LiveQuery
  1. Example with Vue.js
Expectations for the REST API
1. Fetching changes
2. Pushing changes
Alternatives

Features

All the usability improvements from the `idb` library

Since it is a fork of the idb library, synceddb shares the same Promise-based API:

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');

const transaction = db.transaction('items', 'readwrite');
await transaction.store.add({ id: 1, label: 'Dagger' });

// short version
await db.add('items', { id: 1, label: 'Dagger' });

More information here.

Sync with a remote REST API

Every change is tracked in a store. The SyncManager then sync these changes with the remote REST API when the connection is available, making it easier to build offline-first applications.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.start();

// will result in the following HTTP request: POST /items
await db.add('items', { id: 1, label: 'Dagger' });

// will result in the following HTTP request: DELETE /items/2
await db.delete('items', 2);

Auto-reloading queries

The LiveQuery provides a way to run a query every time the underlying stores are updated:

import { openDB, LiveQuery } from 'synceddb';

const db = await openDB('my awesome database');

let result;

const query = new LiveQuery(['items'], async () => {
  // result will be updated every time the 'items' store is modified
  result = await db.getAll('items');
});

// trigger the liveQuery
await db.put('items', { id: 2, label: 'Long sword' });

// or manually run it
await query.run();

Inspired from Dexie.js liveQuery.

Disclaimer

out-of-line keys

Entities without keyPath are not currently supported.

no version history

Only the last version of each entity is kept on the client side.

basic conflict management

The last write wins (though you can customize the behavior in the onpusherror handler).

Installation

npm install synceddb

Then:

import { openDB, SyncManager, LiveQuery } from 'synceddb';

async function doDatabaseStuff() {
  const db = await openDB('my awesome database');

  // sync your database with a remote server
  const manager = new SyncManager(db, 'https://example.com');

  manager.start();
  
  // create an auto-reloading query
  let result;
  const query = new LiveQuery(['items'], async () => {
    // result will be updated every time the 'items' store is modified
    result = await db.getAll('items');
  });
}

API

For database-related operations, please see the idb documentation.

SyncManager

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.start();

Options

`fetchOptions`

Additional options for all HTTP requests.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com', {
  fetchOptions: {
    headers: {
      'accept': 'application/json'
    },
    credentials: 'include'
  }
});

manager.start();

Reference: https://developer.mozilla.org/en-US/docs/Web/API/fetch

`fetchInterval`

The number of ms between two fetch requests for a given store.

Default value: 30000

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com', {
  fetchInterval: 10000
});

manager.start();

`buildPath`

A function that allows to override the request path for a given request.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com', {
  buildPath: (operation, storeName, key) => {
    if (storeName === 'my-local-store') {
      if (key) {
        return `/the-remote-store/${key[1]}`;
      } else {
        return '/the-remote-store/';
      }
    }
    // defaults to `/${storeName}/${key}`
  }
});

manager.start();

`buildFetchParams`

A function that allows to override the query params of the fetch requests.

Defaults to ?sort=updated_at:asc&size=100&after=2000-01-01T00:00:00.000Z,123.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com', {
  buildFetchParams: (storeName, offset) => {
    const searchParams = new URLSearchParams({
      sort: '+updatedAt',
      size: '10',
    });
    if (offset) {
      searchParams.append('after', `${offset.updatedAt}+${offset.id}`);
    }
    return searchParams;
  }
});

manager.start();

`updatedAtAttribute`

The name of the attribute that indicates the last updated date of the entity.

Default value: updatedAt

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com', {
  updatedAtAttribute: 'lastUpdateDate'
});

manager.start();

Methods

`start()`

Starts the sync process with the remote server.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.start();

`stop()`

Stops the sync process.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.stop();

`clear()`

Clears the local stores.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.clear();

`hasLocalChanges()`

Returns whether a given entity currently has local changes that are not synced yet.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

await db.put('items', { id: 1 });

const hasLocalChanges = await manager.hasLocalChanges('items', 1); // true

`onfetchsuccess`

Called after some entities are successfully fetched from the remote server.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.onfetchsuccess = (storeName, entities, hasMore) => {
  // ...
}

`onfetcherror`

Called when something goes wrong when fetching the changes from the remote server.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.onfetcherror = (err) => {
  // ...
}

`onpushsuccess`

Called after a change is successfully pushed to the remote server.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.onpushsuccess = ({ operation, storeName, key, value }) => {
  // ...
}

`onpusherror`

Called when something goes wrong when pushing a change to the remote server.

import { openDB, SyncManager } from 'synceddb';

const db = await openDB('my-awesome-database');
const manager = new SyncManager(db, 'https://example.com');

manager.onpusherror = (change, response, retryAfter, discardLocalChange, overrideRemoteChange) => {
  // this is the default implementation
  switch (response.status) {
    case 403:
    case 404:
      return discardLocalChange();
    case 409:
      // last write wins by default
      response.json().then((content) => {
        const version = content[VERSION_ATTRIBUTE];
        change.value[VERSION_ATTRIBUTE] = version + 1;
        overrideRemoteChange(change.value);
      });
      break;
    default:
      return retryAfter(DEFAULT_RETRY_DELAY);
  }
}

LiveQuery

The first argument is an array of stores. Every time one of these stores is updated, the function provided in the 2nd argument will be called.

import { openDB, LiveQuery } from 'synceddb';

const db = await openDB('my awesome database');

let result;

const query = new LiveQuery(['items'], async () => {
  // result will be updated every time the 'items' store is modified
  result = await db.getAll('items');
});

Example with Vue.js

<script>
import { openDB, LiveQuery } from 'synceddb';

export default {
  data() {
    return {
      items: []
    }
  },
  
  async created() {
    const db = await openDB('test', 1, {
      upgrade(db) {
        db.createObjectStore('items', { keyPath: 'id' });
      },
    });
    
    this.query = new LiveQuery(['items'], async () => {
      this.items = await db.getAll('items');
    });
  },
  
  unmounted() {
    // !!! IMPORTANT !!! This ensures the query stops listening to the database updates and does not leak memory.
    this.query.close();
  }
}
script>

Expectations for the REST API

Fetching changes

Changes are fetched from the REST API with GET requests:

GET /?sort=updated_at:asc&size=100&after=2000-01-01T00:00:00.000Z,123

Explanations:

sort=updated_at:asc indicates that we want to sort the entities based on the date of last update
size=100 indicates that we want 100 entities max
after=2000-01-01T00:00:00.000Z,123 indicates the offset (with an update date above 2000-01-01T00:00:00.000Z, excluding the entity 123)

The query parameters can be customized with the buildFetchParams option.

Expected response:

{
  data: [
    {
      id: 1,
      version: 1,
      updatedAt: '2000-01-01T00:00:00.000Z',
      label: 'Dagger'
    },
    {
      id: 2,
      version: 12,
      updatedAt: '2000-01-02T00:00:00.000Z',
      label: 'Long sword'
    },
    {
      id: 3,
      version: -1, // tombstone
      updatedAt: '2000-01-03T00:00:00.000Z',
    }
  ],
  hasMore: true
}

A fetch request will be sent for each store of the database, every X seconds (see the fetchInterval option).

Pushing changes

Each successful readwrite transaction will be translated into an HTTP request, when the connection is available:

Operation	HTTP request	Body
`db.add('items', { id: 1, label: 'Dagger' })`	`POST /items`	`{ id: 1, version: 1, label: 'Dagger' }`
`db.put('items', { id: 2, version: 2, label: 'Long sword' })`	`PUT /items/2`	`{ id: 2, version: 3, label: 'Long sword' }`
`db.delete('items', 3)`	`DELETE /items/3`
`db.clear('items')`	one `DELETE` request per item

Success must be indicated by an HTTP 2xx response. Any other response status means the change was not properly synced. You can customize the error handling behavior with the onpusherror method.

Please see the Express server there for reference.

Alternatives

Here are some alternatives that you might find interesting:

idb: https://github.com/jakearchibald/idb
Dexie.js: https://dexie.org/ (and its ISyncProtocol part)
pouchdb: https://pouchdb.com/
Automerge: https://github.com/automerge/automerge
Yjs: https://github.com/yjs/yjs

an open-source package to make it easy and simple to work with RabbitMQ's RPC ( Remote Procedure Call )

RabbitMQ Easy RPC (Remote Procedure Call ) The Node.js's RabbitMQ Easy RPC Library rabbitmq-easy-RPC is an easy to use npm package for rabbitMQ's RPC

Sep 22, 2022

An obsidian plugin for uploading local images embedded in markdown to remote store and export markdown for publishing to static site.

Obsidian Publish This plugin cloud upload all local images embedded in markdown to specified remote image store (support imgur only, currently) and ex

Dec 13, 2022

Simple and configurable tool to manage daily huddles in a remote team.

Daily Huddle Simple and configurable tool to manage daily huddles in a remote team. See working version. What's this? This repo has been developed as

Oct 2, 2022

open-source implementation of the Turborepo custom remote cache server.

This project is an open-source implementation of the Turborepo custom remote cache server. If Vercel's official cache server isn't a viable option, th

Dec 30, 2022

💻 ssher.vim is interacting remote machines(only linux) through ssh connection

Feb 21, 2022

A custom Chakra UI component that adds ready-made styles for rendering remote HTML content.

Chakra UI Prose Prose is a Chakra UI component that adds a ready-made typography styles when rendering remote HTML. Installation yarn add @nikolovlaza

Jan 3, 2023

Web based application that uses playerctl in it backend to control remotely your audio using the frontend as remote control.

Linux Remote This is a web based application that uses playerctl in it backend to control remotely your audio using the frontend as remote control. Do

Jul 6, 2022

Collection of job openings from Indonesian remote-friendly companies

id-wfa 🇮🇩 This project scrapes job openings from Indonesian companies that have publicly announced that they provide WFA (work-from-anywhere) perks

Dec 25, 2022

A clean-looking, secure, MySQL/MariaDB remote connection terminal made in NodeJS

NodeJS MySQL/MariaDB Terminal NodeJS MySQL/MariaDB Terminal is a remote terminal for MySQL/MariaDB databases, which works in the same way as the offic

Jun 24, 2022

0.0.2(Mar 26, 2022)
Bug Fixes

add missing Content-Type header (6903318)

prevent infinite loop when pushing updates (0a94d53)

Diff: https://github.com/darrachequesne/synceddb/compare/0.0.1...0.0.2
Source code(tar.gz)
Source code(zip)
0.0.1(Mar 26, 2022)
Features

add SyncManager and LiveQuery features (dab36fb)

Source code(tar.gz)
Source code(zip)

IndexedDB with usability and remote syncing

Related tags

Overview

IndexedDB with usability and remote syncing

Features

All the usability improvements from the idb library

Sync with a remote REST API

Auto-reloading queries

Disclaimer

Installation

API

SyncManager

Options

fetchOptions

fetchInterval

buildPath

buildFetchParams

updatedAtAttribute

Methods

start()

stop()

clear()

hasLocalChanges()

onfetchsuccess

onfetcherror

onpushsuccess

onpusherror

LiveQuery

Example with Vue.js

Expectations for the REST API

Fetching changes

Pushing changes

Alternatives

You might also like...

Releases(0.0.2)

0.0.2(Mar 26, 2022)

Bug Fixes

0.0.1(Mar 26, 2022)

Features

Owner

Damien Arrachequesne

Autocompletion, in-code secret peeking 🔎, syncing, and more, for your .env files in VSCode. 👑 From the same people who pioneered dotenv.

This tool allows you to test your chains.json file to see if your chains are available, syncing, or in sync.

Persistent key/value data storage for your Browser and/or PWA, promisified, including file support and service worker support, all with IndexedDB. Perfectly suitable for your next (PWA) app.

fetch and process data in web worker, store in indexedDB.

Demo showcasing information leaks resulting from an IndexedDB same-origin policy violation in WebKit.

PouchDB for Deno, leveraging polyfill for IndexedDB based on SQLite.

❇️ Doxor.js : more comfortable interacting with IndexedDB

A remote nodejs Cache Server, for you to have your perfect MAP Cache Saved and useable remotely. Easy Server and Client Creations, fast, stores the Cache before stopping and restores it again!

fcall, fetch and call any remote hot functions, anywhere, anytime, without installations or configurations.

MerLoc is a live AWS Lambda function development and debugging tool. MerLoc allows you to run AWS Lambda functions on your local while they are still part of a flow in the AWS cloud remote.

All the usability improvements from the `idb` library

`fetchOptions`

`fetchInterval`

`buildPath`

`buildFetchParams`

`updatedAtAttribute`

`start()`

`stop()`

`clear()`

`hasLocalChanges()`

`onfetchsuccess`

`onfetcherror`

`onpushsuccess`

`onpusherror`