Horizontally scalable blockchain using STARK's and partitioned transactional memory

Overview

quark

quark - quick STARK!

A decentralized horizontally-scaled state machine that can transfer 1,000,000 unique tokens on Uniswap in a single atomic transaction. How?

  • UXTO-like storage. Transactions only lock the parts of state they modify, there is no global state lock like in EVM.
  • ...that's distributed. Instead of every node storing the entire state, we partition the data in the same way Google's Bigtable/Spanner scales to trillions of rows.
  • ...that's decoupled from execution. Self-contained transactions that prove the storage leaves they updated, in O(log N) time. Built using Cairo STARK's.

Install

See the proof-of-concept for more info.

Design.

The blockchain state machine is decoupled into sequencing, execution, and storage layers.

Storage nodes store the persistent state of the blockchain. Storage is modeled as a sorted multidimensional map, where the key is (contract, storage_key), the dimensions are (value, time), and the data is partitioned according to row range.

Execution nodes use the storage network for their memory:

  1. Reads. Every transaction declares the storage leaves it reads from ahead-of-time, and these are fetched in parallel from the storage network at the beginning of tx execution.
  2. Writes. The execution node generates a STARK proof of a transaction's state transition. This is used to prove the storage writes are authorised to the storage nodes.

Transactions are scheduled in parallel. In order to maintain a consistent view of the machine's memory, the sequencer provides the total ordering of transactions to both the execution and storage layers.

The memory model is best described as multiversion concurrency control, as there are no global locks. Like Bitcoin and Solana, transactions can be scheduled in parallel if they don't modify the same state. Unlike Bitcoin and Solana, the cost of verifying transactions is vastly cheaper - atop Cairo proofs, tx verification scales sublinearly in number of VM steps - O(log^2 n).

Storage is partitioned according to row range, which makes it cheap to fetch large numbers of storage slots from a single contract, as this only touches at most a few nodes.

diagram

More information in this system model.

Napkin calculations.

(1) 
Sequencer determines global ordering of txs. Users post txs to the sequencer, which enqueues them. 
[tendermint 400tps = 2.5ms)

(2) 
Executor generates a trace of the state transition. If we can declaratively declare data dependencies, this can be parallelised. Executor writes to the distributed storage journal (a bigtable db).
Assuming good data locality, parallel trace generation.

[100ms to fetch state in parallel from any number of storage nodes]
[10ms to execute transaction in VM]
[1.6s to generate proof]
[100ms to flush writes to storage journal]
[16ms to verify proofs on distributed storage nodes]

Napkin estimate - 2s for a tx, with no hard ceiling on storage or execution steps.

Throughput estimates:

O(log^2 n) proof verification time

Storing state for a popular ERC20 like USDC -

(u256 address, u256 balance)
row size = 64 bytes
1B rows = 64 GB of state
target partition size = 200MB
num partitions = 320
maximum rows per partition = 200MB/64bytes = 3,125,000 rows
You might also like...

Create a maintainable and scalable Node.js GraphQL API with TypeScript, Express, Mongoose and Apollo Server.

Set up and build a Node.js GraphQL API using Typescript, Express, Mongoose with a maintainable and scalable structure

Nov 4, 2022

A framewok for building efficient and scalable, cross-platform bots (WhatsApp, Discord, Telegram and more) in a single codebase

A framewok for building efficient and scalable, cross-platform bots (WhatsApp, Discord, Telegram and more) in a single codebase

project... An over-engineered all-in-one bot framewok for building efficient and scalable bots. Yep that's not a typo u heard it right "scalable bots"

Dec 24, 2022

An exercise in building a very minimal (and very stupid) in-memory SQL-like database for educational purposes.

Stupid Database This is an exercise in building a very minimal (and very stupid) in-memory SQL-like database for educational purposes. None of this co

Dec 20, 2022

In-memory abstract-level database for Node.js and browsers.

memory-level In-memory abstract-level database for Node.js and browsers, backed by a fully persistent red-black tree. The successor to memdown and lev

Dec 27, 2022

Train and test your brain with these memory games

Train and test your brain with these memory games

Want to help grow your Hippocampus? Engage your Cerebrum? Try these games! Train and test your brain with these memory games. Each game is designed to help you increase your memory. How much can you memorize with each game? Test yourself.

Jun 16, 2022

Github action to collect metrics (CPU, memory, I/O, etc ...) from your workflows to help you debug and optimize your CI/CD pipeline

Github action to collect metrics (CPU, memory, I/O, etc ...) from your workflows to help you debug and optimize your CI/CD pipeline

workflow-telemetry-action A GitHub Action to track and monitor the resource metrics of your GitHub Action workflow runs. If the run is triggered via a

Dec 30, 2022

A flexible, memory compact, fast and dynamic CSG implementation on top of three-mesh-bvh

A flexible, memory compact, fast and dynamic CSG implementation on top of three-mesh-bvh

three-bvh-csg An experimental, in progress, flexible, memory compact, fast and dynamic CSG implementation on top of three-mesh-bvh. More than 100 time

Jan 5, 2023

A simple in-memory time-based cache for both objects and function execution.

What is this? A simple in-memory time-based cache for both objects and function execution. How do I install it? You can install it by using the follow

Dec 15, 2022

Inferrd makes Machine Learning deployment easy and scalable.

Inferrd makes Machine Learning deployment easy and scalable.

Introduction ML Deployment made simple, scalable and collaborative The new open-source standard to deploy, monitor and secure Machine Learning service

Dec 16, 2022
Comments
  • Client-side generated STARK proofs

    Client-side generated STARK proofs

    Maybe we could make this even more performant for users by introducing client-side generated proofs.

    This is a blockchain most useful for data-heavy stuff. if you can get the stark proofs down to 0.01ms, you can do a lot more

    If you control all of your storage leaves, then you can generate the proof yourself and include it in the sequencer submission.

    Then optimistically assume that the leaves are the right ones, else pay a sequencing cost.

    This has the benefit of naturally decentralizing the execution layer, since everything can be done client-side.

    idea 
    opened by liamzebedee 0
  • Research / related and prior works.

    Research / related and prior works.

    https://signalsandthreads.com/state-machine-replication-and-why-you-should-care/

    Jane St runs a distributed state machine.

    • Messages come in from all sorts of different data sources.
    • Sequencer produces a canonical order.
    • After which message is disseminated to node for execution/processing.
    • Programs have rudimentary ABI interface that Jane St calls message types (lol).
    • State sharing is interesting.
    • Their sequencer doesn't multicast all messages. Execution nodes subscribe to particular topics - reduce network bandwidth. Publish-subscribe model
    opened by liamzebedee 1
Owner
Liam Zebedee
Blockchain/full-stack, indie R&D, big nerd.
Liam Zebedee
jQuery plugin for horizontally scrolling galleries

jQuery Horiztonal Scroll Gallery This jQuery plugin slides through a horiztonal gallery of items as you scroll down. It makes use of position:sticky s

Jeremy Rue 3 Feb 23, 2022
Transactional Inbox/Outbox pattern for Durable Objects

do-transactional-outbox One of the challenges that many event-driven systems face is the fact that they have to write to the database and send out an

Erwin van der Koogh 5 Sep 27, 2022
HackMIT 2022. 2nd Place in Blockchain for Society sponsored by Jump Crypto. A revolutionary web application that leverages machine learning and blockchain technology to improve the crowdsourcing experience!

?? Wikisafe ?? Wikisafe is a revolutionary new crowdsourcing web application that innovates the process of crowdsourcing information. This application

Benson Liu 5 Dec 8, 2022
It is a very basic implementation of how blockchain works, mainly how the bitcoin blockchain.

How to run this program npm install node core/blockchain.js What is this It is a very basic implementation of how blockchain works, mainly how the bit

Anish Jain 12 May 9, 2022
Set up and build a Node.js REST API using Typescript, Express, Mongoose with a maintainable and scalable structure.

Introduction Create a maintainable and scalable Node.js REST API with TypeScript, Express and Mongoose. The project structure is based on MVC and foll

Adam Khomsi 16 Nov 18, 2022
Brain wallet using both language and visual memory.

*Check out the big brain on Brett! You're a smart motherfvcker, that's right.* -- Pulp fiction ?? SUPER BRAIN WALLET ?? Use your brain power to the ma

Code's All Right ™ 4 Jun 9, 2022
A simple in-memory key-value cache for function execution, allowing both sync and async operations using the same methods

A simple in-memory key-value cache for function execution, allowing both sync and async operations using the same methods. It provides an invalidation mechanism based both on exact string and regex.

cadienvan 10 Dec 15, 2022
Palaemon is an open-source developer tool for monitoring health and resource metrics of Kubernetes clusters and analyzing Out of Memory (OOMKill) errors

Palaemon ?? ?? An Electron based developer tool for Kubernetes cluster monitoring and error analysis Palaemon is a Greek, child sea-god who came to ai

OSLabs Beta 99 Dec 28, 2022
Using decentralized identities with Web 2 to create a no login required website. Built using the Handshake blockchain.

Blending Web 2/3, is this Web .666? ( •̀ᴗ•́ )و ̑̑ Learn more by joining the Handshake Discord Community applause is a platform I built using centraliz

Publius Federalist 13 Mar 3, 2022
Simple shopping cart prototype which shows how React components and Redux can be used to build a friendly user experience with instant visual updates and scalable code in e-commerce applications.

This simple shopping cart prototype shows how React components and Redux can be used to build a friendly user experience with instant visual updates a

Ivan Kuznietsov 3 Feb 8, 2022