QuestDB Node.js Client

Related tags

Learning resource nodejs-questdb-client
Overview

QuestDB Node.js Client

Installation

npm install @questdb/nodejs-client

Examples

Basic API usage

const { Sender } = require("@questdb/nodejs-client");

async function run() {
    // create a sender with a 4k buffer
    // it is important to size the buffer correctly so messages can fit
    const sender = new Sender({bufferSize: 4096});

    // connect to QuestDB
    // host and port are required in connect options
    await sender.connect({port: 9009, host: "127.0.0.1"});

    // add rows to the buffer of the sender
    sender.table("prices").symbol("instrument", "EURUSD")
        .floatColumn("bid", 1.0195).floatColumn("ask", 1.0221).atNow();
    sender.table("prices").symbol("instrument", "GBPUSD")
        .floatColumn("bid", 1.2076).floatColumn("ask", 1.2082).atNow();

    // flush the buffer of the sender, sending the data to QuestDB
    // the buffer is cleared after the data is sent and the sender is ready to accept new data
    await sender.flush();

    // add rows to the buffer again and send it to the server
    sender.table("prices").symbol("instrument", "EURUSD")
        .floatColumn("bid", 1.0197).floatColumn("ask", 1.0224).atNow();
    await sender.flush();

    // close the connection after all rows ingested
    await sender.close();
    return new Promise(resolve => resolve(0));
}

run().then(value => console.log(value)).catch(err => console.log(err));

Authentication and secure connection

const { Sender } = require("@questdb/nodejs-client");

async function run() {
    // construct a JsonWebKey
    const CLIENT_ID = "testapp";
    const PRIVATE_KEY = "9b9x5WhJywDEuo1KGQWSPNxtX-6X6R2BRCKhYMMY6n8";
    const PUBLIC_KEY = {
        x: "aultdA0PjhD_cWViqKKyL5chm6H1n-BiZBo_48T-uqc",
        y: "__ptaol41JWSpTTL525yVEfzmY8A6Vi_QrW1FjKcHMg"
    };
    const JWK = {
        ...PUBLIC_KEY,
        d: PRIVATE_KEY,
        kid: CLIENT_ID,
        kty: "EC",
        crv: "P-256",
    };

    // pass the JsonWebKey to the sender
    // will use it for authentication
    const sender = new Sender({bufferSize: 4096, jwk: JWK});

    // connect() takes an optional second argument
    // if 'true' passed the connection is secured with TLS encryption
    await sender.connect({port: 9009, host: "127.0.0.1"}, true);

    // send the data over the authenticated and secure connection
    sender.table("prices").symbol("instrument", "EURUSD")
        .floatColumn("bid", 1.0197).floatColumn("ask", 1.0224).atNow();
    await sender.flush();

    // close the connection after all rows ingested
    await sender.close();
    return new Promise(resolve => resolve(0));
}

run().then(value => console.log(value)).catch(err => console.log(err));

TypeScript example

import { Sender } from "@questdb/nodejs-client";

async function run(): Promise<number> {
    // construct a JsonWebKey
    const CLIENT_ID: string = "testapp";
    const PRIVATE_KEY: string = "9b9x5WhJywDEuo1KGQWSPNxtX-6X6R2BRCKhYMMY6n8";
    const PUBLIC_KEY: { x: string, y: string } = {
        x: "aultdA0PjhD_cWViqKKyL5chm6H1n-BiZBo_48T-uqc",
        y: "__ptaol41JWSpTTL525yVEfzmY8A6Vi_QrW1FjKcHMg"
    };
    const JWK: { x: string, y: string, kid: string, kty: string, d: string, crv: string } = {
        ...PUBLIC_KEY,
        d: PRIVATE_KEY,
        kid: CLIENT_ID,
        kty: "EC",
        crv: "P-256",
    };

    // pass the JsonWebKey to the sender
    // will use it for authentication
    const sender: Sender = new Sender({bufferSize: 4096, jwk: JWK});

    // connect() takes an optional second argument
    // if 'true' passed the connection is secured with TLS encryption
    await sender.connect({port: 9009, host: "127.0.0.1"}, true);

    // send the data over the authenticated and secure connection
    sender.table("prices").symbol("instrument", "EURUSD")
        .floatColumn("bid", 1.0197).floatColumn("ask", 1.0224).atNow();
    await sender.flush();

    // close the connection after all rows ingested
    await sender.close();
    return new Promise(resolve => resolve(0));
}

run().then(value => console.log(value)).catch(err => console.log(err));
Comments
  • Added debug option and remove unneded async

    Added debug option and remove unneded async

    Add support to enable/disable the debug to use the console.info messages. By default the debug is false, but in case the user sends true in the options, it will show the console messages.

    Also removed some unneeded async keywords.

    This is to solve #12

    opened by jalamprea 2
  • chore(nodejs): regexp for table and column name validations

    chore(nodejs): regexp for table and column name validations

    Flame diagrams show that validating and writing strings are the hot places in the code, looping through the characters of strings is slow. Replacing it with regular expressions in table and column name validation and character escaping improves performance significantly. However, this is true only when the code using the client does not generate garbage. When the client code is not gc-free overall performance is about the same with regexp. In real life use cases unlikely that we will see gc-free clients so the 200k rows/s ingestion speed remains theoretical but I think it still makes sense to use regexp instead of the for loops. It can lead to performance improvements when the client code does not generate much garbage.

    • Client code is gc-free, regexp is much faster than looping through the characters of the strings.
    for loops, 300k rows, strings constants (no garbage)
	rss:61.39MB;heapTotal:25.31MB;heapUsed:8.94MB;	took: 3.595, rate: 83449.23504867872 rows/s
	rss:62.28MB;heapTotal:25.31MB;heapUsed:14.24MB;	took: 3.663, rate: 81900.0819000819 rows/s
	rss:61.53MB;heapTotal:25.31MB;heapUsed:14.39MB;	took: 3.574, rate: 83939.56351426973 rows/s

regex, 300k rows, strings constants (no garbage)
	rss:58.45MB;heapTotal:23.31MB;heapUsed:7.78MB;	took: 1.561, rate: 192184.49711723256 rows/s
	rss:58.33MB;heapTotal:22.81MB;heapUsed:7.79MB;	took: 1.559, rate: 192431.04554201412 rows/s
	rss:59.81MB;heapTotal:23.56MB;heapUsed:7.8MB;	took: 1.535, rate: 195439.7394136808 rows/s

for loops, 3m rows, strings constants (no garbage)
	rss:84.56MB;heapTotal:48.56MB;heapUsed:16.89MB;	took: 35.096, rate: 85479.82676088443 rows/s
	rss:75.73MB;heapTotal:41.56MB;heapUsed:20.52MB;	took: 35.034, rate: 85631.10121596164 rows/s
	rss:89.25MB;heapTotal:53.06MB;heapUsed:34.04MB;	took: 34.888, rate: 85989.45196055951 rows/s

regex, 3m rows, strings constants (no garbage)
	rss:75.84MB;heapTotal:39.31MB;heapUsed:12.16MB;	took: 14.069, rate: 213234.77148340322 rows/s
	rss:74.95MB;heapTotal:39.81MB;heapUsed:11.31MB;	took: 13.924, rate: 215455.32892846884 rows/s
	rss:74.8MB;heapTotal:39.06MB;heapUsed:17.09MB;	took: 13.785, rate: 217627.8563656148 rows/s
    • Client code generates garbage, overall performance is more or less the same.
    for loops, 300k rows, unique strings (with garbage)
	rss:50.7MB;heapTotal:15.31MB;heapUsed:10.16MB;	took: 27.773, rate: 10801.857919562164 rows/s
	rss:47.31MB;heapTotal:15.06MB;heapUsed:10.05MB;	took: 27.871, rate: 10763.876430698576 rows/s
	rss:49.22MB;heapTotal:15.31MB;heapUsed:7.1MB;	took: 27.846, rate: 10773.540185304892 rows/s

regex, 300k rows, unique strings (with garbage)
	rss:50.02MB;heapTotal:15.06MB;heapUsed:9.15MB;	took: 27.264, rate: 11003.521126760563 rows/s
	rss:49.45MB;heapTotal:15.06MB;heapUsed:8.26MB;	took: 27.721, rate: 10822.120414126475 rows/s
	rss:50MB;heapTotal:15.06MB;heapUsed:6.52MB;	took: 27.493, rate: 10911.868475611975 rows/s

for loops, 500k rows, unique strings (with garbage)
	rss:62.16MB;heapTotal:24.31MB;heapUsed:8.69MB;	took: 102.829, rate: 4862.441529140612 rows/s
	rss:61.53MB;heapTotal:24.31MB;heapUsed:10.96MB;	took: 101.525, rate: 4924.895345973898 rows/s
	rss:61.16MB;heapTotal:24.31MB;heapUsed:12.56MB;	took: 101.551, rate: 4923.634429990842 rows/s

regex, 500k rows, unique strings (with garbage)
	rss:61.55MB;heapTotal:24.56MB;heapUsed:9.03MB;	took: 102.877, rate: 4860.172827745755 rows/s
	rss:60.97MB;heapTotal:24.31MB;heapUsed:12.37MB;	took: 101.236, rate: 4938.95452210676 rows/s
	rss:61.97MB;heapTotal:24.06MB;heapUsed:11.4MB;	took: 100.424, rate: 4978.889508484028 rows/s
    opened by glasstiger 1
  • feat(nodejs): custom logging

    feat(nodejs): custom logging

    Extended the options of the client with a new property: log. Now the host application can pass a logging function to the client.

    log: (level: 'error'|'warn'|'info'|'debug', message: string) => void

    If this property not provided, default logging is used which writes to the console with logging level info.

    enhancement 
    opened by glasstiger 0
  • use `localhost` instead of `127.0.0.1` in examples

    use `localhost` instead of `127.0.0.1` in examples

    in This PR:

    • format example code with prettier
    • use localhost instead of 127.0.0.1 in examples

    The example code is used in other code bases to showcase how this client should be used.

    It is also used with examples.manifest.yaml, which describes examples with interpolatable values.

    In particular, the examples.manifest.yaml has these:

      addr:
    host: localhost
    port: 9009

    when example code like ./examples/basic.js is interpolated with values from examples.manifest.yaml, the host part is missed.

    Thus, we should be consistent and in both examples.manifest.yaml and examples/*.js use as host either:

    1. localhost or
    2. 127.0.0.1

    This PR chooses localhost

    opened by argshook 0
  • chore(nodejs): failproof flush

    chore(nodejs): failproof flush

    Adding a new option, called copyBuffer. If the option is set the client will create a new buffer for each flush() call with the data belongs to the returned promise. This prevents creating duplicate rows if await is missed when calling flush() or if the calls to flush() are not serialised.

    opened by glasstiger 0
  • chore(nodejs): testcontainers

    chore(nodejs): testcontainers

    Test with containerised QuestDB instance. Connect, create table, alter table, ingest data via client, add new column via client. Run selects to assert the data in the test table.

    opened by glasstiger 0
  • Reduce the number of necessary parameters when using ILP auth

    Reduce the number of necessary parameters when using ILP auth

    For connecting via ILP auth, only the key_id and the private_key/d parameter should be needed. However, this client also requests the public key pair, which is annoying as those are not really needed, and the developer need to treat them as secrets.

    Some official clients like Go, JAVA, or C# don't need the extra parameters.

    Once changed, the documentation should be updated to reflect the simplified connection params.

    opened by javier 2
  • Messages in console always visible

    Messages in console always visible

    I want to use this module, but since it uses console.info to show some messages, it is a little bit slow due to the sync code with console. The debug/info messages should be optional and defined as part of the options when the user create the sender

    opened by jalamprea 3
Releases(v1.0.2)

  • v1.0.2(Jan 5, 2023)

    A new option to pass a custom logging function to the client. If no logger specified the client will write log messages to console, default logging level is info.

    Source code(tar.gz)
    Source code(zip)

  • v1.0.1(Sep 21, 2022)

    This release introduces a new option called 'copyBuffer', the option is not set by default. If the option is set the client will create a new buffer for each flush() call with the data belongs to the returned promise. This prevents creating duplicate rows if await is missed when calling flush() or if the calls to flush() are not serialised.

    Example:

    const sender = new Sender({ copyBuffer: true });
    Source code(tar.gz)
    Source code(zip)

  • v1.0.0(Aug 22, 2022)

Owner
QuestDB
QuestDB is the fastest open source time-series database
QuestDB
GitHub
Node 18's node:test, as a node module

node-core-test This is a user-land port of node:test, the experimental test runner introduced in Node.js 18. This module makes it available in Node.js

Julian Gruber 62 Dec 15, 2022
Javascript client for Sanity. Works in node.js and modern browsers (older browsers needs a Promise polyfill).

@sanity/client Javascript client for Sanity. Works in node.js and modern browsers (older browsers needs a Promise polyfill). Requirements Sanity Clien

Sanity 23 Nov 29, 2022
Node.js client for Permify

Permify Node.js Client Use Permify in server-side Node.js Projects. Install npm install @permify/permify-node Usage To get started, create Permify cli

Permify 5 Jan 1, 2023
Official Node.js client library for Devzat plugin API

Devzat plugin API client for Node.js This NPM package allows you to build Devzat plugins/bots with JavaScript/TypeScript. See example/index.ts for a f

Benjamin Smith 2 Apr 26, 2022
A Node.js client & server implementation of the WAMP-like RPC-over-websocket system defined in the OCPP-J protcols.

OCPP-RPC A client & server implementation of the WAMP-like RPC-over-websocket system defined in the OCPP-J protcols (e.g. OCPP1.6J and OCPP2.0.1J). Re

Mikuso 14 Dec 30, 2022
Postgres.js - The Fastest full featured PostgreSQL client for Node.js and Deno

?? Fastest full-featured node & deno client ?? ES6 Tagged Template Strings at the core ??‍♀️ Simple surface API ??️ Dynamic query support ?? Chat and

Rasmus Porsager 4.3k Jan 1, 2023
Node.js implementation of the Socket SDK client

SYNOPSIS A Node.js adapter for the Socket SDK DESCRIPTION Socket SDK uses a simple uri-based protocol for brokering messages between the render proces

Socket Supply Co. 11 Dec 28, 2022
Promise based HTTP client for the browser and node.js

axios Promise based HTTP client for the browser and node.js New axios docs website: click here Table of Contents Features Browser Support Installing E

axios 98k Jan 5, 2023
Lightweight universal Cloudflare API client library for Node.js, Browser, and CF Workers

Cloudflare API Client Lightweight universal HTTP client for Cloudflare API based on Fetch API that works in Node.js, browser, and CF Workers environme

Kriasoft 15 Nov 13, 2022
Yet Another Clickhouse Client for Node.js

yacc-node - Yet Another Clickhouse Client for NodeJS Introduction yacc-node is a zero depencies Clickhouse Client written in Typescript. Installation

Antonio Vizuete 3 Nov 3, 2022
Node-cli-starter - Basic starter kit for building Node CLI applications with TypeScript.

node-cli-starter Minimal starter kit for building Node CLI applications with TypeScript. Getting Started To get started clone repo locally and run npm

Cory Rylan 7 May 17, 2022
Apilytics for Node.js - Easy API analytics for Node backends

apilytics-node Apilytics is a service that lets you analyze operational, performance and security metrics from your APIs without infrastructure-level

Apilytics 9 Sep 2, 2022
This is a vanilla Node.js rest API created to show that it is possible to create a rest API using only vanilla Node.js

This is a vanilla Node.js rest API created to show that it is possible to create a rest API using only vanilla Node.js. But in most cases, I would recommend you to use something like Express in a production project for productivity purposes.

Eduardo Dantas 7 Jul 19, 2022
Inter Process Communication Module for node supporting Unix sockets, TCP, TLS, and UDP. Giving lightning speed on Linux, Mac, and Windows. Neural Networking in Node.JS

Inter Process Communication Module for node supporting Unix sockets, TCP, TLS, and UDP. Giving lightning speed on Linux, Mac, and Windows. Neural Networking in Node.JS

Node IPC 43 Dec 9, 2022
Node js package makes creating node jd dependincies files like Controllers,Entities and Repositories easier by executing a few instructions

Nodejs Studio Node js package makes creating node js project dependincies files like Controllers,Entities and Repositories easier by executing a few i

Syrian Open Source 9 Oct 12, 2022
Spin node create spin api for node

Links Contract api JS api @spinfi/core @spinfi/node @spinfi/node Spin node create spin api for node How to install yarn add @spinfi/node How to init i

Spin 6 Oct 18, 2022
Automatically generated documentation for the Valorant API endpoints the client uses internally.

Valorant API Docs To read documentation and get started, see Docs This is a project designed to automatically document Valorant endpoints based on a J

Techdoodle 223 Dec 25, 2022
Client-Side Prototype Pollution Tools

Client-Side Prototype Pollution Tools Match rules for Burp Software Version Reporter extension Match rules that passively detect vulnerable libraries

Sergey Bobrov 73 Oct 4, 2022
client-side prototype pullution vulnerability scanner

JSPanda JSpanda is client-side prototype pollution vulnerability scanner. It has two key features, scanning vulnerability the supplied URLs and analyz

Red Section 46 Dec 25, 2022