ParkyDB - block based, linkable and verifiable document database -- javascript reference implementation

Overview

Ancon ParkyDB

A data mesh database using Web 3.0 technology

Note: Requires Node v17.7.2 and up for development

More about data mesh architecture

ParkyDB (2)

ParkyDB (1)

Block based data layered abstraction

IndexedDB (KV Layer - Layer 0)

Stores key as cid or custom (topic) and values in the layered approached with a schema similar to:

{
   "key": "cid",
   "dag": ...blob...,
   "db": ...blob...,
   "index:": ...blob...,
   "gqls": ...blob...,
   "jsonschema": ...blob...
}

DAG (Linkable and Verifiable Layer - Layer 1)

Stores as an IPLD Multiformats DAG Block. Input must be a JSON payload. Can support either CBOR or JSON. This layer keeps data immutable (no mutations allowed) and uses special directives with query layer.

Document (Document Layer - Layer 2)

Stores as a JSON. Input must be a JSON payload. Used for queries only and represents a snapshot of the immutable data in DAG.

Index (Query and index Layer - Layer 3)

Stores as a Minisearch filter index. Input must be JSON payload. Used for search only and represents the JSON payload index, the @filter GraphQL directive will enable filtering.

GraphQL Schema (Query and index Layer - Layer 3)

Stores a GraphQL Schema. Used with on-demand GraphQL APIs that enables querying the DB and Index layer. Mutations are immutable PUTs in DAG. It also integrates different GraphQL stores using Mesh and appends the data as blocks in the database.

JSON Schema (Verifiable Document Layer - Layer 4)

Stores a JSON Schema. Used to create Verifiable Data Document dapps which might contain or required ERC-721 / Verified Credential compatible schemas. This feature is used for data publishing exclusively.

Protobuf Schema (Messaging Layer - Layer 5)

Stores a Protobuf Schema. Used to integrate data library with Waku and decentralized full nodes

Run tests

We using Ava test framework

npm test

Note: Enable OpenSSL legacy support.

export NODE_OPTIONS="--openssl-legacy-provider"

API v1.0.0-rc.3

Store

import { ParkyDB } from 'parkydb'

// Instantiate new DB instance
const db = new ParkyDB()
await db.initialize()

// Writes a DAG JSON block
const id = await db.putBlock(payload)

// Fetch an existing DAG block
const res = await db.get(id)
// Queries using Dexie
const obs$ = await db.queryBlocks((blocks) => {
    return () => blocks.where({ cid: '' })
});

// Queries with GraphQL a JSON snapshot of the DAG block
const q = await db.query({
    cid: id,
    query: `
    query{
       block(cid: "${id}") {
         network
         key
       }
    }   
    `,
  })

Topics and Subscriptions

import { ParkyDB } from 'parkydb'

// Instantiate new DB instance
const db = new ParkyDB()
// Browsers can only support web sockets connections with Waku v2
const peer =
    '/ip4/0.0.0.0/tcp/8000/wss/p2p/...'
  await this.bob.initialize({
    // wakuconnec options
    wakuconnect: { bootstrap: { peers: [peer] } },
    // Remember these values come from a CLI or UI, DO NOT hardcode when implementing
    withWallet: {
      password: 'zxcvb',
      seed: 'opera offer craft joke defy team prosper tragic reopen street advice moral',
    },
  })
const topic = `/anconprotocol/1/marketplace/ipld-dag-json`

// Writes a DAG JSON block
const id = await db.putBlock({...payload, topic})

// Fetch an existing DAG block
const res = await db.get(id)

const pubsub = await db.createTopicPubsub(topic)

// pubsub methods
//  { 
//   Streams blocks from topic message replies
//   onBlockReply$: pubsub.asObservable(),
//    
//   On demand publishing
//   publish: async (block: BlockValue),
//   
//   Closes any pending subscriptions
//   close: () 
// }

pubsub.onBlockReply$.subscribe((block)=> {

  // GraphQL
  const q = await db.query({
      block,
      query: `
      query{
        block(cid: "${id}") {
          network
          key
        }
      }   
      `,
    })
})  

Wallet

import { ParkyDB } from 'parkydb'

// Instantiate new DB instance
const db = new ParkyDB()
await db.initialize({
  // withWeb3 for interactive usage, eg browsers, smart phones.
  withWeb3: {
    provider: ethers.providers.Web3Provider(windows.ethereum),
    pubkey,
    defaultAddress,
  },
  // withWallet useful for backend use cases (eg NestJS)
  // Remember these values come from a environment variables, CLI or UI, DO NOT hardcode when implementing
  withWallet: {
    password: '',
    // Note: Invented this mnemonic rap, 12 words, as my way to protest #WARINUKRAINE
    seed: 'lumber brown jack house bomb cluster star method guard against war peace',
  }
})

// Where `db.wallet` is metamask keyring controller. See https://github.com/MetaMask/KeyringController
// ParkyDB has an Ed22519 implementation for DID and HPKE use cases
await db.wallet.addNewKeyring('Ed25519', [
  'c87509a1c067bbde78beb793e6fa76530b6382a4c0241e5e4a9ec0a0f44dc0d3',
])

// Wallet is used internally or by setting options to specific usage. See Protocols for how to encrypt and sign.

Protocols (channels)

test('create channel topic, signed and encrypted, cbor as message serialization', async (t) => {
  const { alice, bob }: { alice: ParkyDB; bob: ParkyDB } = t.context as any

  await alice.wallet.addSecp256k1([
    'c87509a1c067bbde78beb793e6fa76530b6382a4c0241e5e4a9ec0a0f44dc0d3',
  ])
  await alice.wallet.submitPassword(`qwerty`)
  let accounts = await alice.wallet.getAccounts()
  t.is(accounts.length, 1)
  const accountA = accounts[0]

  await bob.wallet.submitPassword(`zxcvb`)
  accounts = await bob.wallet.getAccounts()
  t.is(accounts.length, 1)
  const accountB = accounts[0]

  const blockCodec = {
    name: 'cbor',
    code: '0x71',
    encode: (obj: any) => encode(obj),
    decode: (buffer: any) => decode(buffer),
  }
  const topic = `/anconprotocol/1/marketplace/cbor`
  const pubsubAlice = await alice.createChannelPubsub(topic, {
    from: accountA,
    middleware: {
      incoming: [tap()],
      outgoing: [map((v: BlockValue)=> v.document)],
    },
    blockCodec,
  })
  pubsubAlice.onBlockReply$.subscribe(async (block: WakuMessage) => {
    // match topic
    t.is(topic, JSON.parse(block.payloadAsUtf8).topic)
  })
  const pubsubBob = await bob.createChannelPubsub(topic, {
    from: accountB,
    middleware: {
      incoming: [tap()],
      outgoing: [map((v: BlockValue)=> v.document)],
    },
    blockCodec,
  })
  pubsubBob.onBlockReply$.subscribe(async (block: WakuMessage) => {
    // match topic
    t.is(topic, JSON.parse(block.payloadAsUtf8).topic)
    await bob.putBlock(payload, { topic })
  })

  // Say hi
  await alice.putBlock(payload, { topic })
})

DeFi example

test('find multichain tx by sender', async (t) => {
  const {
    alice,
    bob,
    charlie,
    consumer,
  }: {
    alice: ParkyDB
    bob: ParkyDB
    charlie: ParkyDB
    consumer: ParkyDB
  } = t.context as any

  await alice.wallet.submitPassword(`qwerty`)
  let accounts = await alice.wallet.getAccounts()
  const accountA = accounts[0]

  await bob.wallet.submitPassword(`zxcvb`)
  accounts = await bob.wallet.getAccounts()
  const accountB = accounts[0]

  await charlie.wallet.submitPassword(`a1d2f3f4`)
  accounts = await charlie.wallet.getAccounts()
  const accountC = accounts[0]

  await consumer.wallet.submitPassword(`mknjbhvgv`)
  accounts = await consumer.wallet.getAccounts()
  const accountConsumer = accounts[0]

  const blockCodec = {
    name: 'eth-block',
    code: '0x90',
    encode: (obj: any) => encodeDagEth(obj),
    decode: (buffer: any) => decodeDagEth(buffer),
  }

  const topicBSC = `/bsc/1/new_blocks/dageth`
  const topicEthereum = `/ethereum/1/new_blocks/dageth`
  const topicPolygon = `/polygon/1/new_blocks/dageth`

  // Aggregate from BSC, Ethereum and Polygon any Transfer to x address
  // Then pipe calls to Discord channel and an arbitrage bot using a webhook (POST)
  const pubsubAlice = await alice.createChannelPubsub(topicBSC, {
    from: accountA,
    middleware: {
      incoming: [tap()],
      outgoing: [tap()],
    },
    blockCodec,
  })
  const pubsubBob = await bob.createChannelPubsub(topicEthereum, {
    from: accountB,
    middleware: {
      incoming: [tap()],
      outgoing: [tap()],
    },
    blockCodec,
  })
  const pubsubCharlie = await charlie.createChannelPubsub(topicPolygon, {
    from: accountC,
    middleware: {
      incoming: [tap()],
      outgoing: [tap()],
    },
    blockCodec,
  })

  subscribeNewBlocks(
    [
      {
        name: 'bsc',
        chainId: '56',
        rpc: 'wss://somerpc.server',
      },
    ],
    (payload: any) => {
      await alice.putBlock(payload, { topic })
    },
  )

  subscribeNewBlocks(
    [
      {
        name: 'mainnet',
        chainId: '1',
        rpc: 'wss://somerpc.server',
      },
    ],
    (payload: any) => {
      await bob.putBlock(payload, { topic })
    },
  )
  subscribeNewBlocks(
    [
      {
        name: 'polygon',
        chainId: '137',
        rpc: 'wss://somerpc.server',
      },
    ],
    (payload: any) => {
      await charlie.putBlock(payload, { topic })
    },
  )
  const aggregator = await consumer.aggregate(
    [topicBSC, topicEthereum, topicPolygon],
    {
      from: accountConsumer,
      middleware: {
        incoming: [
          filter(
            (v: object) => v.address === '0x...' && v.event === 'Transfer',
          ),
          zip(map(v=>v),reduce((v, init) => (v = new BigNumber(init).add(v)))),
        ],
      },
      blockCodec,
    },
  )

  aggregator.onBlockReply$.subscribe(async (payload: any) => {
    const {v, sum} = payload
    // send to discord or arbitrage bot...
  })
})

Copyright IFESA 2022

You might also like...

An easy-to-read, quick reference for JS best practices, accepted coding standards, and links around the Web

An easy-to-read, quick reference for JS best practices, accepted coding standards, and links around the Web

Feel free to contribute! Where? http://www.jstherightway.org Why? Today we have a bunch of websites running JavaScript. I think we need a place to put

Jan 1, 2023

Modern Spatial Reference System Class. Supports EPSG Codes, PROJ4 String, and Well-Known Text.

spatial-reference-system Modern Spatial Reference System Class. supports EPSG Codes PROJ4 Strings ESRI and OGC Well-Known Text PRJ File install npm in

Jul 22, 2022

Sanity plugin for viewing resources which reference a particular resource.

Sanity plugin for viewing resources which reference a particular resource.

@indent-oss/sanityio-referenced-by Plugin to see which documents reference a particular document referenced-by-sanityio.mov Video Alt Text: Demonstrat

Nov 2, 2022

🎹 Memorize piano scales with ease! A music practice program w/ MIDI support. Consider it an interactive reference manual

🎹 Memorize piano scales with ease! A music practice program w/ MIDI support. Consider it an interactive reference manual

Piano Trainer Learn to play the piano at your own pace through various modes of practice. Watch the video Features MIDI compatible Home row keyboard i

Dec 21, 2022

Stacks Voice is a reference project that builds on the SIP018 signed structured data standard to create an accountless internet forum.

Stacks Voice Stacks Voice is a reference project that builds on the SIP018 signed structured data standard to create an accountless internet forum. Th

Dec 21, 2022

SmartBuilder: A Block-based Visual Programming Framework for Smart Contract Development

SmartBuilder: A Block-based Visual Programming Framework for Smart Contract Development

SmartBuilder A Block-based Visual Programming Framework for Smart Contract Development Technology stack used SmartBuilder Framework - Google Blockly A

Mar 29, 2022

This Photoshop script exports all top-level layers and groups to cropped PNG and JPEG files and creates a file usable in Tumult Hype 4 based on your Photoshop document.

This Photoshop script exports all top-level layers and groups to cropped PNG and JPEG files and creates a file usable in Tumult Hype 4 based on your Photoshop document.

Export To Hype (Photoshop Edition) This Photoshop script exports all top-level layers and groups to cropped PNG and JPEG files and creates a file usab

Nov 9, 2022

Interplanetary Database: A Database built on top of IPFS and made immutable using Ethereum blockchain.

IPDB IPDB (Interplanetary Database) is a key/value store database built on top of IPFS (Interplanetary File System). Project is intended to be an MVP

Oct 6, 2022

Visualize, modify, and build your database with dbSpy! An open-source data modeling tool to facilitate relational database development.

Visualize, modify, and build your database with dbSpy! An open-source data modeling tool to facilitate relational database development.

Visualize, modify, and build your database with dbSpy! dbSpy is an open-source data modeling tool to facilitate relational database development. Key F

Dec 22, 2022
Owner
Ancon Protocol
Industrias de Firmas Electronicas SA - Ancon Protocol
Ancon Protocol
Send encrypted and decrypted messages with verifiable keys and human readable names.

zooko-msg Encrypt and decrypt messages using AES with a preshared ECDH key generated using keys associated with Handshake names. I noticed that there

Publius Federalist 31 Jul 27, 2022
The Next.js reference implementation on how to design better APIs

This is the Next.js reference implementation of my blog bost on How to design better APIs. Getting Started Clone the repository git clone https://gith

Ronald Blüthl 7 Nov 22, 2022
The reference LRB-USDL arbitrage bot implementation, built with neon-js.

README Introduction arby is the reference arbitrage bot for Lyrebird. arby is able to run at a profit because Lyrebird swaps LRB and USDL based on the

null 3 May 9, 2022
Serverless for Web3, which is Immutable and Verifiable✅

Tender Layer 2 for IPFS / API Renderer for Web3 / Serverless for Ethereum Tender is dynamic content serving P2P Protocol for Ethereum. V1 Design A Cod

Hyojun Kim 23 Nov 18, 2022
Non-interactive publicly verifiable distributed key generation and resharing algorithm over BLS12-381

NPVDKG-RS This repository contains a mathematical presentation and some code to demonstrate our developed non-interactive publicly verifiable distribu

NATRIX Official 8 May 19, 2022
WordPress Gutenberg plugin to display the attributes for the currently selected block in the Document sidebar.

Block X-ray Attributes Stable Tag: 1.2.0 Requires at least: 5.5 Tested up to: 5.9 Requires PHP: 7.2 License: GPL v2 or later Tags: block attributes, g

Sal Ferrarello 38 Mar 18, 2022
Reference for How to Write an Open Source JavaScript Library - https://egghead.io/series/how-to-write-an-open-source-javascript-library

Reference for How to Write an Open Source JavaScript Library The purpose of this document is to serve as a reference for: How to Write an Open Source

Sarbbottam Bandyopadhyay 175 Dec 24, 2022
A Complete Javascript Learning Reference Guide.

Learn Javascript A Javascript Reference Repository, which was initially created for my refernce is now open. Any one can Learn and Add more Content to

Abdul Rehman Kalsekar 18 Oct 26, 2022
This document introduces an early implementation of the Node-RED runtime that runs on resource-constrained microcontrollers (MCUs).

Node-RED MCU Edition Copyright 2022, Moddable Tech, Inc. All rights reserved. Peter Hoddie Updated June 25, 2022 Introduction This document introduces

Peter Hoddie 53 Jan 3, 2023
Minimally viable DOM Document implementation for NativeScript.

DOMiNATIVE Minimally viable DOM Document implementation for NativeScript NOTE THIS IS STILL EXPERIMENTAL. Installation Via npm: npm install dominative

SudoMaker 33 Dec 28, 2022