hyperbeedeebee
A MongoDB-like database built on top of Hyperbee with support for indexing
WIP: There may be breaking changes in the indexing before the v1.0.0 release, don't use this for anything you don't mind migrating in the future.
Based on this design
Usage
npm i --save hyperbeedeebee
const Hyperbee = require('hyperbee')
// This module handles networking and storage of hypercores for you
const SDK = require('hyper-sdk')
const {DB} = require('hyperbeedeebee')
const {Hypercore} = await SDK()
// Initialize a hypercore for loading data
const core = new Hypercore('example')
// Initialize the Hyperbee you want to use for storing data and indexes
const bee = new Hyperbee(core)
// Create a new DB
const db = new DB(bee)
// Open up a collection of documents and insert a new document
const doc = await db.collection('example').insert({
hello: 'World!'
})
// doc._id gets set to an ObjectId if you don't specify it
console.log(doc)
// Iterate through data as it's loaded (streaming)
// Usually faster and more memory / CPU efficient
for await (let doc of db.collection('example').find({
clout: {
$gt: 9000
},
})) {
console.log(doc)
}
// Create an index for properties in documents
// This drastically speeds up queries and is necessary for sorting by fields
await db.collection('example').createIndex('createdAt')
// Get all results in an array
// Can skip some results and limit total for pagination
const killbots = await db.collection('example')
.find({type: 'killbot'})
.sort('createdAt', -1)
.skip(30)
.limit(100)
// Get a single document that matches the query
const eggbert = await db.collection('example').findOne({name: 'Eggbert'})
Data Types
HyperbeeDeeBee uses MongoDB's BSON data types for encoding data. You can import the bson library bundled with HyperbeeDeeBee using the following code:
const { BSON } = require('hyperbeedeebee')
From there you can access any of the following data types:
Binary,
Code,
DBRef,
Decimal128,
Double,
Int32,
Long,
UUID,
Map,
MaxKey,
MinKey,
ObjectId,
BSONRegExp,
BSONSymbol,
Timestamp
TODO:
- Sketch up API
- Insert (with BSON encoding)
- Find all docs
- Find by _id
- Find by field eq (no index)
- Find by array field includes
- Find by number field
$gt/$gte/$lt/$lte- Numbers
- Dates
- Find using
$inoperator - Find using
$alloperator - Find using
$existsoperator - Index fields
- Sort by index (with find)
- Indexed find by field
$eq - Flatten array for indexes
- Get field values from index key without getting the doc
- Find on fields that aren't indexed
- Indexed find for
$exists - Indexed find by number field
- Indexed find for
$in - Indexed find for
$all - Hint API (specify index to use)
- Test if iterators clean up properly
- More efficient support for
$gt/$gte/$lt/$lteindexes - More efficient support for
$allindexes - More efficient support for
$inindexes - Detect when data isn't available from peers and emit an error of some sort instead of waiting indefinately.
Important Differences From MongoDB
- There is a single writer for a hyperbee and multiple readers
- The indexing means that readers only need to download small subsets of the full dataset (if you index intelligently)
- No way to do "projections" so keep in mind you're always downloading the full document to disk
- Subset of
find()API is implemented, no Map Reduce API, no$or/$andsince it's difficult to optimize - You can only sort by indexed fields, otherwise there's no difference from loading all the data and sorting in memory
- Fully open source under AGPL-3.0 and with mostly MIT dependencies.
Indexing considerations:
Indexes are super important to make your applications snappy and to reduce the overall CPU/Bandwidth/Storage usage of queries.
- If you do a search by fields that aren't indexed, you'll end up downloading the full collection (this is potentially really slow)
- The order of fields in the index matters, they're used to create an ordered key based on the values
- If you want to sort by a field, make sure it's the first field in an index
- You can have indexed fields before the sorted field if they are only used for $eq operations, this is due to the database's ability to turn them into a prefix to speed up the search.
- If an index cannot be found to satisfy a
sortthe query will fail. - If you're using
$gt/$lt/$gte/$ltein your query, they will perform best if the same considerations as the sort are applied. - If the fields in the index can be used to rule out a document as matching, then you can avoid loading more documents and doing fewer overall comparisons on data.
- If your field is a unicode string which has
0x00bytes in it, then the sorting might break due to the way BSON serializes unicode strings. Proceed with caution!
2k Dec 30, 2022
2 Jul 17, 2022
5 Dec 13, 2022
21 Dec 25, 2022
49 Jan 1, 2023
24 Dec 22, 2022
6 Dec 12, 2022
6 Nov 15, 2022
6.3k Jan 2, 2023
6.8k Jan 3, 2023
9 Nov 20, 2022
6 Apr 25, 2022
7 May 21, 2022
25.2k Dec 30, 2022
9.6k Dec 28, 2022
5.4k Dec 31, 2022
1.4k Nov 30, 2022
570 Dec 14, 2022
96 Nov 21, 2022