many-level
Share an abstract-level database over the network or other kind of stream. The successor to multileveldown. If you are upgrading, please see UPGRADING.md.
📌 Which module should I use? What isabstract-level? Head over to the FAQ.
Usage
Use this module to share an abstract-level database across multiple processes or machines. A host exposes a database of choice over binary streams, using compact Protocol Buffers messages to encode database operations. One or more guests connect to that host and expose it as an abstract-level database, as if it is a regular, local database. They can opt-in to a seamless retry in order to reconnect to a host without aborting any pending database operations.
First create a host and server. The server can be anything that supports binary streams. In this example we'll use a simple TCP server.
const { ManyLevelHost } = require('many-level')
const { Level } = require('level')
const { pipeline } = require('readable-stream')
const { createServer } = require('net')
const db = new Level('./db')
const host = new ManyLevelHost(db)
const server = createServer(function (socket) {
// Pipe socket into host stream and vice versa
pipeline(socket, host.createRpcStream(), socket, () => {
// Disconnected
})
})
server.listen(9000)
Then create some guests:
const { ManyLevelGuest } = require('many-level')
const { pipeline } = require('readable-stream')
const { connect } = require('net')
const db = new ManyLevelGuest()
const socket = connect(9000)
// Pipe socket into guest stream and vice versa
pipeline(socket, db.createRpcStream(), socket, () => {
// Disconnected
})
await db.put('hello', 'world')
console.log(await db.get('hello'))
Encoding options are supported as usual.
Reconnect
To setup reconnect set the retry option to true and reconnect to your server when the connection fails:
const db = new ManyLevelGuest({
retry: true
})
const reconnect = function () {
const socket = connect(9000)
pipeline(socket, db.createRpcStream(), socket, () => {
// Reconnect after 1 second
setTimeout(reconnect, 1000)
})
}
reconnect()
The guest database will now retry your pending operations when you reconnect. If you create an iterator or readable stream and your connection fails halfway through reading that iterator then many-level makes sure to only retry the part of the iterator that you are missing. The downside of retry is that a guest database then cannot provide snapshot guarantees because new iterators (and thus snapshots of the host database) will be created upon reconnect. This is reflected in db.supports.snapshots which will be false if retry is true.
API
Host
host = new ManyLevelHost(db, [options])
Create a new host that exposes the given db, which must be an abstract-level database that supports the 'buffer' encoding (most if not all do). It can also be a sublevel which allows for exposing only a specific section of the database.
The optional options object may contain:
readonly(boolean, defaultfalse): reject write operations likedb.put()preput(function, default none): afunction (key, val, cb) {}to be called beforedb.put()operationspredel(function, default none): afunction (key, cb) {}to be called beforedb.del()operationsprebatch(function, default none): afunction (operations, cb) {}to be called beforedb.batch()operations.
hostStream = host.createRpcStream()
Create a duplex host stream to be piped into a guest stream. One per guest.
Guest
db = new ManyLevelGuest([options])
Create a guest database that reads and writes to the host's database. The ManyLevelGuest class extends AbstractLevel and thus follows the public API of abstract-level with a few additional methods and one additional constructor option. As such, the majority of the API is documented in abstract-level. It supports sublevels, iterator.seek() and every other abstract-level feature, except for the createIfMissing and errorIfExists options which have no effect here. Iterators have built-in end-to-end backpressure regardless of the transport that you use.
The optional options object may contain:
keyEncoding(string or object, default'utf8'): encoding to use for keysvalueEncoding(string or object, default'utf8'): encoding to use for valuesretry(boolean, defaultfalse): if true, resend operations when reconnected. If false, abort operations when disconnected, which means to yield an error on e.g.db.get().
The database opens itself but (unlike other abstract-level implementations) cannot be re-opened once db.close() has been called. Calling db.open() would then yield a LEVEL_NOT_SUPPORTED error.
guestStream = db.createRpcStream([options])
Create a duplex guest stream to be piped into a host stream. Until that's done, operations made on db are queued up in memory. Will throw if createRpcStream() was previously called and that stream has not (yet) closed. The optional options object may contain:
ref(object, defaultnull): an object to only keep the Node.js event loop alive while there are pending database operations. Should have aref()method to be called on a new database operation likedb.get()and anunref()method to be called when all operations have finished (or when the database is closed). A Node.jsnetsocket satisfies that interface. Therefoption is not relevant whenManyLevelGuestis used in a browser environment (which, side note, is not officially supported yet).
guestStream = db.connect()
An alias to createRpcStream() for multileveldown API compatibility.
db.forward(db2)
Instead of talking to a host, forward all database operations to db2 which must be an abstract-level database. This method is used by rave-level and serves a narrow use case. Which is to say, it may not work for anything other than rave-level. Among other things, it assumes that db2 is open and that it uses the same encoding options as the guest database.
db.isFlushed()
Returns true if there are no operations pending to be sent to a host.
Install
With npm do:
npm i many-level
Usage from TypeScript also requires npm install @types/readable-stream.
Contributing
Level/many-level is an OPEN Open Source Project. This means that:
Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
See the Contribution Guide for more details.
Donate
Support us with a monthly donation on Open Collective and help us continue our work.
21 Oct 12, 2022
24 Dec 22, 2022
6 Dec 12, 2022
13 May 31, 2022
12 Jul 7, 2022
62 Dec 30, 2022
6 Apr 25, 2022
176 Dec 26, 2022
107 Dec 26, 2022
3 Dec 28, 2022
3 Nov 25, 2022
15 Dec 5, 2022
67 Jan 2, 2023
16 Jan 6, 2023
18.6k Dec 31, 2022
18.9k Dec 30, 2022
15.4k Dec 30, 2022
11.7k Jan 4, 2023
8.8k Jan 5, 2023