Serialization library for data-oriented design structures in JavaScript

Overview

Data-oriented Serialization for SoA/AoA

A zero-dependency serialization library for data-oriented design structures like SoA (Structure of Arrays) and AoA (Array of Arrays).

Features

  • DataWriter: Serializes data from SoA object(s) to binary data provided array of indices to extract data from
  • DataReader: Deserializes from binary data to SoA object(s) to the appropriate indices
  • Binary data packed in an AoS-like format for optimal space efficiency
  • ID mapping included for reading/writing data from/to different sets of indices
  • Includes snapshot and delta modes
    • Snapshot mode serializes the entire state (default)
    • Delta mode only serializes state that has changed since the last serialization call

Planned

  • Support for AoA serialization
  • More flexible API
    • Schemas
    • Ability to pass in ArrayBuffers to write to

Example

import assert from 'assert'
import { createDataWriter, createDataReader } from '@webecs/do-serialization'

/* SoA Object */

const n = 100

const Transform = {
  position: {
    x: new Float32Array(n),
    y: new Float32Array(n),
    z: new Float32Array(n),
  },
  rotation: {
    x: new Float32Array(n),
    y: new Float32Array(n),
    z: new Float32Array(n),
  },
}


/* Config */

// a simple array of SoA objects acts as the configuration
const config = [Transform]


/* Snapshot Mode */

// DataWriter and DataReader must have the same config in order to function correctly
const write = createDataWriter(config)
const read = createDataReader(config)

// initialize SoA state
const e = 0
Transform.position.x[e] = 1
Transform.position.y[e] = 2
Transform.position.z[e] = 3

// serialize
let data = write([e])

// reset SoA state
Transform.position.x[e] = 0
Transform.position.y[e] = 0
Transform.position.z[e] = 0

// deserialize
read(data)

// assert data was deserialized onto SoA state
assert(Transform.position.x[e] === 1) // true
assert(Transform.position.y[e] === 2) // true
assert(Transform.position.z[e] === 3) // true


/* Delta Mode */

// true value for second parameter enables delta mode (needed for both writer & reader)
const writeDeltas = createDataWriter(config, true)
const readDeltas = createDataReader(config, true)

Transform.position.x[e] = 0
Transform.position.y[e] = 0
Transform.position.z[e] = 0

// serialize
data = writeDeltas([e])

assert(data.byteLength === 0) // true, no changes made to the data

// mutate SoA state
Transform.position.x[e] = 1
Transform.position.y[e] = 2
Transform.position.z[e] = 3

// serialize
data = writeDeltas([e])

assert(data.byteLength > 0) // true, changes have been made to the data since the last call

// reset SoA state
Transform.position.x[e] = 0
Transform.position.y[e] = 0
Transform.position.z[e] = 0

// deserialize
readDeltas(data)

// assert changed data was deserialized onto SoA state
assert(Transform.position.x[e] === 1) // true
assert(Transform.position.y[e] === 2) // true
assert(Transform.position.z[e] === 3) // true


/* ID Mapping */

const idMap = new Map([[0,12]])

// this will write index 0 from the SoA object as index 12 in the binary data
data = write([e], idMap)


// this will read index 0 from the binary data to index 12 on the SoA object
read(data, idMap)
You might also like...

🥞Data Structures and Algorithms explained and implemented in JavaScript + eBook

🥞Data Structures and Algorithms explained and implemented in JavaScript + eBook

Data Structures and Algorithms in JavaScript This is the coding implementations of the DSA.js book and the repo for the NPM package. In this repositor

Jan 4, 2023

A collection of all the data structures implemented in javascript code

Data structures A collection of all the data structures implemented in javascript code, interview questions & problems ( with solutions ) Contributors

May 1, 2022

The basics data structures implemented with javascript.

The basics data structures implemented with javascript.

Estrutura de Dados com Javascript Tudo que está escrito e codificado aqui dentro desse repositório foi retirado do livro Estrutura de dados e algoritm

Oct 18, 2022

Ace your next Javascript coding interview by mastering data structures and algorithms.

The Coding Interview: Algorithms + Data Structures Ace your next Javascript coding interview by mastering data structures and algorithms. Problem 1: S

Sep 19, 2022

Bringing an all Open-Source Platform to study Data Structures and Algorithms ⚡

NeoAlgo-Docs Bringing an all Open-Source Platform to study Data Structures and Algorithms ⚡ 🔧 Installation You will need to have NodeJS and Yarn inst

Jun 2, 2022

Code accompanying my illustrated Data structures video series on YouTube

Code accompanying my illustrated Data structures video series on YouTube

Dec 10, 2022

Data Structures

Data Structures

Data-Structures Data structures implementaion using Javascript 🧑‍💻 Data-Structures is a Github repository that contains the implementaion of most of

Sep 14, 2022

Data structures & algorithms implementations and coding problem solutions. Written in Typescript and tested with Jest. Coding problems are pulled from LeetCode and Daily Coding Problem.

technical-interview-prep Data structures & algorithms implementations and coding problem solutions. Written in Typescript and tested with Jest. Coding

Aug 5, 2022

A Typescript companion to the book A Common-Sense Guide to Data Structures and Algorithms by Jay Wengrow

A Typescript companion to the book A Common-Sense Guide to Data Structures and Algorithms by Jay Wengrow

This repository aims to be a companion to the book A Common-Sense Guide to Data Structures and Algorithms by Jay Wengrow. I rewrote most of the data s

Dec 3, 2022
Comments
  • ArrayBuffer argument to data writer function

    ArrayBuffer argument to data writer function

    It would be useful to be able to pass an ArrayBuffer to the data writer function. This allows more flexibility in where the data is written and avoids unnecessary copying and slicing.

    opened by EnderShadow8 0
  • explicit schemas

    explicit schemas

    it may be wise to use explicit schemas instead of using the SoA objects themselves as the schema, and then pass the SoA objects into the reader/writer functions themselves. this would allow reading/writing from/to different SoA objects that have the same schemas

    e.g.

    const n = 100
    
    const Vector3SoA = { x: new Float32Array(n), y: new Float32Array(n), z: new Float32Array(n) }
    
    const Vector3Schema = { x: Float32Array, y: Float32Array, z: Float32Array }
    
    const config = [Vector3Schema]
    
    const objects = [Vector3SoA]
    
    const write = createDataWriter(config)
    const read = createDataRead(config)
    
    const indices = [1,2,3,4]
    const data = write(objects, indices)
    
    read(objects, data)
    
    opened by NateTheGreatt 0
Owner
null
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
A complete guide for learning Object Oriented Programming Pillars, SOLID Principles and Design Patterns with TypeScript!

Object Oriented Programming Expert With TypeScript This repository is a complete guide and tutorial for the principles and techniques of object-orient

Ahmad Jafari 44 Dec 29, 2022
A RESP 'Redis Serialization Protocol' library implementation to generate a server, uses a similar approach to express to define you serer, making it easy and fast.

RESPRESS A RESP 'Redis Serialization Protocol' library implementation to generate a server, uses a similar approach to express to define you serer, ma

Yousef Wadi 9 Aug 29, 2022
Binary-encoded serialization of JavaScript objects with generator-based parser and serializer

YaBSON Schemaless binary-encoded serialization of JavaScript objects with generator-based parser and serializer This library is designed to transfer l

Gildas 11 Aug 9, 2022
superserial provides serialization in any way you can imagine.

superserial After data transfer, when the object needs to be restored, JSON has many limitations. It does not support values such as Infinity and NaN,

DenoStack 24 Dec 23, 2022
Glorious Binary Serialization and Deserialization for TypeScript.

Glorious SerDes for TypeScript The library you can rely on, For binary serialization and deserialization, In Node, Deno, and the Web environment, Whic

Wei 25 Dec 11, 2022
Serialize arbitrary NodeJS closures and customize serialization behavior.

Closure Serializer This is a fork of the Pulumi Closure Serializer. @pulumi/pulumi. Motivation Functionless allows developers to write cloud applicati

null 4 Jul 19, 2022
⛑️ JSON serialization should never fail

⛑️ JSON serialization should never fail. Features Prevent JSON.serialize() from: Throwing Changing types Filtering or transforming values unexpectedly

ehmicky 191 Dec 15, 2022
📝 Algorithms and data structures implemented in JavaScript with explanations and links to further readings

?? Algorithms and data structures implemented in JavaScript with explanations and links to further readings

Oleksii Trekhleb 157.8k Dec 29, 2022