A modern SQLite store for node-cache-manager

Overview

SQLite store for node cache manager

A modern SQlite cache store for node-cache-manager. Featuring:

  • Async SQLite3 using sqlite3
  • async/await support with Promise
  • 100% test coverage and production ready
  • Optimized mset/mget support
  • Supports CBOR for efficient and fast storage (selectable between json or cbor default: json)
  • Support for custom serializers
  • Smart purging support, no configuration required

Why?

The goal was to have a local key value storage built on top of proven technology. While other options like node-cache-manager-fs-binary have similar functionality; they are littered with all sort of problems from race conditions, (multi-process) to corruption. SQLite on the other end has been battle tested, and using WAL allows multiple node processes (forked web servers) to share the same cache across various processes, without the headaches of flat file based systems.

SQLite based storage is ideal for:

  • Faster local storage than filesystem. Yes you heard it right
  • Reslience to corruption and recovery.
  • Multiprocess Node.js processes (typical in server deployments with many cores)
  • Large number of entries.

Installation

npm i cache-manager-sqlite

Requirements

Usage

Single store

const sqliteStore = require('cache-manager-sqlite')
const cacheManager = require('cache-manager')

// SQLite :memory: cache store
const memStoreCache = cacheManager.caching({
    store: sqliteStore,
    options: {
        serializer: 'cbor', // default is 'json'
        ttl: 20 // TTL in seconds
    }
})

// On disk cache on employees table
const cache = cacheManager.caching({
    store: sqliteStore,
    name: 'employees',
    path: '/tmp/cache.db',
    options: {
        serializer: 'cbor'
    }
})


// TTL in seconds
await cache.set('foo', {test: 'bar'}, {ttl: 10})
const value = await cache.get('foo')

Multi-store example:

const cacheManager = require('cache-manager')
const redisStore = require('cache-manager-ioredis')
const sqliteStore = require('cache-manager-sqlite')

const redisCache = cacheManager.caching({ store: redisStore, db: 0, ttl: 600 })
const sqliteCache = cacheManager.caching({ store: sqliteStore, path: '/tmp/cache.db', name: 'users', options: { ttl: 600 } })

const multiCache = cacheManager.multiCaching([sqliteCache, redisCache])

// Basic get/set
await multiCache.set('foo2', 'bar2', { ttl: customTTL })
const v = await multiCache.get('foo2')

// Wrap call example
const userId = 'user-1'

// Optionally pass ttl
await multiCache.wrap(userId, { ttl: customTTL }, async () => {
    console.log("Calling expensive service")
    await getUserFromExpensiveService(userId)
})

// set and get multiple
await multiCache.mset('foo1', 'bar1', 'foo0', 'bar0') //Uses default TTL
await multiCache.mset('foo1', 'bar1', 'foo3', 'bar3', {ttl: customTTL})
await multiCache.mget('foo1', 'foo3')

License

The node-cache-manager-sqlite is licensed under the MIT license.

You might also like...

Modren is a modern store for Linux. It includes support for snaps, flatpaks from Flathub, APT packages and DEBs.

 Modren is a modern store for Linux. It includes support for snaps, flatpaks from Flathub, APT packages and DEBs.

v1.0.0 Made with ❤️ for 🐧 Modren is a modern store for Linux. It includes support for snaps, flatpaks from Flathub, APT packages and DEBs. Download ·

Nov 18, 2022

PouchDB for Deno, leveraging polyfill for IndexedDB based on SQLite.

PouchDB for Deno PouchDB for Deno, leveraging polyfill for IndexedDB based on SQLite. Usage import PouchDB from 'https://deno.land/x/[email protected]

Aug 2, 2022

A Deno ORM for MySQL, SQLite, PostgreSQL, MongoDB, GitHub and serverless service like Deta, InspireCloud, CloudBase, LeanCloud

A Deno ORM for MySQL, SQLite, PostgreSQL, MongoDB, GitHub and serverless service like Deta, InspireCloud, CloudBase, LeanCloud.

Dec 15, 2022

The Remix Blog Stack for deploying to Fly with MDX, SQLite, testing, linting, formatting, etc.

The Remix Blog Stack for deploying to Fly with MDX, SQLite, testing, linting, formatting, etc.

Remix Speed Metal Stack Learn more about Remix Stacks. npx create-remix --template Girish21/speed-metal-stack Remix Blog 📖 This blog starter template

Jan 2, 2023

The Remix Stack for deploying to Fly with SQLite, authentication, testing, linting, formatting, etc.

The Remix Stack for deploying to Fly with SQLite, authentication, testing, linting, formatting, etc.

Remix Indie Stack Learn more about Remix Stacks. npx create-remix --template remix-run/indie-stack What's in the stack Fly app deployment with Docker

Dec 30, 2022

⚡ It is a simplified database module with multiple functions that you can use simultaneously with sqlite, yaml, firebase and json.

Prisma Database Developed with 💙 by Roxza ⚡ An easy, open source database 📦 Installation npm i prisma.db --save yarn add prisma.db 🔮 Importing impo

Jan 3, 2023

Fresh SQLite example on Fly.io.

fresh-sqlite-example Fresh example with SQLite and kysely query builder. See running example on Fly.io. Prerequisites Deno v1.23 or higher SQLite Opti

Nov 25, 2022

A solid create-remix app, that applies best practices into a clean, batteries included template. SQLite version. Deploys to Fly.io

A solid create-remix app, that applies best practices into a clean, batteries included template. SQLite version. Deploys to Fly.io

Remix Barebones Stack npx create-remix --template dev-xo/barebones-stack A solid create-remix app, that follows community guidelines and applies best

Dec 30, 2022

⚡ A powerful, human-friendly database library for JavaScript using SQLite.

⚡ A powerful, human-friendly database library for JavaScript using SQLite.

great.db ⚡ A powerful, human-friendly database library for JavaScript using SQLite. Elegant way to set and retrieve data Queries are executed through

Nov 29, 2022
Comments
  • Introducing serializer support, moving set to use mset, and documentation updates

    Introducing serializer support, moving set to use mset, and documentation updates

    This change achieves various big goals including:

    • Support for serialization protocol (now supports CBOR as serializer)
    • Many documentation fixes and updates including TTL
    • Move TTL from milliseconds to seconds as per cache manager requirements
    • set now uses mset reusing same code
    opened by maxpert 0
  • mget does not return the correct number of returns if values are undefined or the keys are not in alphabetical order

    mget does not return the correct number of returns if values are undefined or the keys are not in alphabetical order

    Hi, huge fan of your library. I went ahead and forked it to meet the following needs:

    • Make it work with cache-manager@5, which simplified the storage contract to be less variadic and more predictable. This doesn't appear to be backwards compatible.
    • Converted it to Typescript so that I can get code completion when passing it to cacheManager.caching.
    • Swapped in better-sqlite3 for sqlite3 as the former is "faster" and the latter uses a node-gyp fork that has bunch of testing required into code that is shipped with the library which breaks builds in an Astro / Vite setup that I'm working in.

    Feel free to port/steal whatever you would like for this library! Open source!

    In doing the porting, I noticed a bug in your implementation of mget. The SELECT statement for this is just an IN query which will return unpredictably if keys do not exist OR the keys are not provided in alphabetical order. In practice, the following have issues from what I can tell:

    await cache.set('a', 'a')
    await cache.set('b', 'b')
    await cache.set('z', 'z')
    
    // This mget has a gap at position 'c' and it should return ['a', 'b', undefined, 'z'] but it returns ['a', 'b', 'z']
    let [a, b, c, z] = await cache.mget('a', 'b', 'c', 'z')
    // this will actually equal 'z'
    assert(c === undefined)
    // this is undefined as the array is only 3 elements long
    assert(z === 'z')
    
    // This mget returns items in reverse order and should be ['z', 'b', 'a'] but it returns ['a', 'b', 'z']
    let [z, b, a] = await cache.mget('z', 'b', 'a')
    // this will actually equal 'a'
    assert(z === 'z')
    // this will actually equal 'z'
    assert(a === 'a')
    

    The way I got around this is by using a varadic CTE that correctly orders and fills in the gaps in the query with almost equal complexity to an IN statement you have.

    > explain query plan WITH getKeys(key) AS (VALUES ("a"), ("b"), ("c"), ("z"))
    SELECT
      getKeys.key,
      val,
      created_at,
      expire_at
    FROM getKeys
    LEFT JOIN kv ON kv.key = getKeys.key;
    
    QUERY PLAN
    |--CO-ROUTINE getKeys
    |  `--SCAN 3 CONSTANT ROWS
    |--SCAN getKeys
    `--SEARCH kv USING INDEX sqlite_autoindex_kv_1 (key=?) LEFT-JOIN
    
    > explain query plan select * from kv where key in ("a", "b", "c", "z");
    
    QUERY PLAN
    `--SEARCH kv USING INDEX sqlite_autoindex_kv_1 (key=?)
    
    opened by eligundry 3
Owner
Zohaib Sibte Hassan
Yet another engineer
Zohaib Sibte Hassan
Grupprojekt för kurserna 'Javascript med Ramverk' och 'Agil Utveckling'

JavaScript-med-Ramverk-Laboration-3 Grupprojektet för kurserna Javascript med Ramverk och Agil Utveckling. Utvecklingsguide För information om hur utv

Svante Jonsson IT-Högskolan 3 May 18, 2022
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
A cache for @azure/msal-node that uses Azure KeyVault as a store

@intility/msal-keyvault-cache A cache for @azure/msal-node that uses Azure KeyVault as a store. Usage Install with npm install @intility/msal-keyvault

Intility 10 Mar 17, 2022
A devtool improve your pakage manager use experience no more care about what package manager is this repo use; one line, try all.

pi A devtool improve your pakage manager use experience no more care about what package manager is this repo use; one line, try all. Stargazers over t

tick 11 Nov 1, 2022
Meogic-tab-manager is an extensible, headless JavaScript tab manager framework.

MeogicTabManager English document MeogicTabManager是一个有可拓展性的、headless的JavaScript标签页管理框架。 MeogicTabManager旨在提供可自由组装页面框架、自定义页面组件、甚至覆盖框架自带事件响应的开发体验。 Meogi

meogic-tech 5 Oct 8, 2022
Node.js library that provide a cache for file metadata or file content.

@file-cache A cache library for file metadata or file content. It is useful for process that work a given series of files and that only need to repeat

azu 16 Aug 6, 2022
A MITM cache between RPCs and a a dAPP. Useful to allow for better performance on a public RPC node

better-cosmos-rpcs A cheaper way to allow for public RPCs as a service WITHOUT scaling issues. No need to rate limit either. How it is done: User GET

Reece Williams 3 Nov 19, 2022
A modern ebook manager and reader with sync and backup capacities for Windows, macOS, Linux and Web

简体中文 | English Koodo Reader A cross-platform ebook reader Download | Preview | Roadmap | Document Preview Feature Format support: EPUB (.epub) Scanned

Troye Guo 8.6k Dec 29, 2022