Easily add filtering, sorting, and pagination to your Node.js REST API through your old friend: the query string!

Overview

QueryQL

npm CircleCI piratepx

QueryQL makes it easy to add filtering, sorting, and pagination to your Node.js REST API through your old friend: the query string! Read our introductory article to learn more about why we wrote it and the problems it solves at Truepic.

QueryQL works with any Node.js web framework (be it Express, Koa, etc.), supports any query builder / ORM through adapters, and allows for custom validators so you can define validation in a familiar way.

Out of the box, QueryQL supports the following:

Installation

$ npm install @truepic/queryql

Getting Started

QueryQL takes a parsed query string (like Express' req.query) and translates it into the appropriate function calls that your query builder / ORM understands to filter, sort, and paginate the records.

Let's consider an example to illustrate:

/images?filter[id][in][]=2&filter[id][in][]=3&filter[status]=open&sort=name&page[size]=10
{
  filter: {
    id: {
      in: [2, 3],
    },
    status: 'open',
  },
  sort: 'name',
  page: {
    size: 10,
  },
}

To support this query, QueryQL only requires you to define (whitelist) what's allowed through what we call a querier. Here's how one might look for the /images endpoint:

const QueryQL = require('@truepic/queryql')

class ImageQuerier extends QueryQL {
  defineSchema(schema) {
    schema.filter('id', 'in')
    schema.filter('status', '=')
    schema.sort('name')
    schema.page()
  }
}

With your querier defined, you can now call it in your router / controller. Here's how it might look in an Express route:

app.get('/images', async (req, res, next) => {
  const querier = new ImageQuerier(req.query, knex('images'))
  let images

  try {
    images = await querier.run()
  } catch (error) {
    // Handle validation error, such as by passing to an Express error handler:
    next(error)
  }

  res.send({ images })
})

Behind-the-scenes, QueryQL takes your initial query builder (knex('images')), and applies the following Knex chain when querier.run() is called:

builder
  .where('id', 'in', [2, 3])
  .where('status', '=', 'open')
  .orderBy('name', 'asc')
  .limit(10)
  .offset(0)

(Remember: While Knex is our default adapter and the query builder used in this example, adapters can be written for any query builder / ORM.)

This is a simple example, but hopefully it illustrates how easy it is to add filtering, sorting, and pagination to your REST API without manually touching your query builder / ORM.

Read the full documentation to learn how to add validation, customize the queries, and more.

Development

Prerequisites

The only prerequisite is a compatible version of Node.js (see engines.node in package.json).

Dependencies

Install dependencies with npm:

$ npm install

Tests

Jest is our testing framework of choice, with file-specific tests contained in the test/src directory. We strive for 100% code coverage.

To run the tests:

$ npm test

During development, it's recommended to run the tests automatically on file change:

$ npm test -- --watch [--notify]

Code Style & Linting

Prettier is setup to enforce a consistent code style. It's highly recommended to add an integration to your editor that automatically formats on save.

ESLint is setup with the "recommended" rules to enforce a level of code quality. It's also highly recommended to add an integration to your editor that automatically formats on save.

To run via the command line:

$ npm run lint

Releasing

After development is done in the development branch and is ready for release, it should be merged into the master branch, where the latest release code lives. Release It! is then used to interactively orchestrate the release process:

$ npm run release
Comments
  • Question: How to handle joins?

    Question: How to handle joins?

    Hello! queryql seems pretty useful, and we're considering using it for a service here at TpT.

    Do you have any examples of what a Querier would look like if it were to support joins? It seems like the examples deal with single tables.

    Say for example our database schema looks like this:

    table foo {
       id int
       the_important_stuff string
    }
    
    table bar {
       id int
       foo_id int references(foo.id)
       type string
    }
    

    And let's say we want to fetch foo by looking at bar.type:

    /foos?filter[bar.type]=APPLE
    

    Is that going to require a custom querier function? Something like this:

      'filter:[bar.type]='(builder, { field, operator, value }) {
        return builder.innerJoin('bar', 'foo.id', 'bar.foo_id').where(field, operator, value);
      }
    
    enhancement 
    opened by nampas 5
  • 'is not' operator always throw validation error

    'is not' operator always throw validation error

    Hi,

    I was trying to create a querier that accepts an 'is not' operator but it seems like there is a validation that prevents the values of the filters to be null however the KnexAdapter stablish that the is not operator needs to be null.

    Minimal example:

    // Querier
    class TestQuerier extends QueryQL {
      defineSchema(schema) {
        schema.filter('test', '=')
        schema.filter('testing', '!=')
        schema.filter('testNull', 'is not')
        schema.sort('test')
        schema.sort('testing')
        schema.page()
      }
    }
    
    // Test
      test('returns `true` if valid', () => {
        const querier = new TestQuerier(
          {
            filter: { test: 123, testNull: { 'is not': null} },
            sort: 'test',
            page: 2,
          },
          knex('test')
        )
    
        expect(querier.validate()).toBe(true)
        // ValidationError: filter:testNull[is not] must be one of [array, boolean, number, string]
      })
    
    // If i try to pass a string
      test('returns `true` if valid', () => {
        const querier = new TestQuerier(
          {
            filter: { test: 123, testNull: { 'is not': 'foo'} },
            sort: 'test',
            page: 2,
          },
          knex('test')
        )
    
        expect(querier.validate()).toBe(true)
        // ValidationError: filter:testNull[is not] must be [null]
      })
    

    I think that it have something to do with the method defineValidation in the FilterParser.

    Thanks in advance

    bug 
    opened by vligas 4
  • "Expected 0 arguments, but got 2" on class which extends QueryQL

    I am trying to get QueryQL to work with my Express + MikroORM app. MikroORM uses knex under the hood so I believe I should be able to get this to work without writing an adapter.

    My Querier class looks like this:

    import QueryQL from '@truepic/queryql';
    
    export class ContentQuerier extends QueryQL {
    
      get pageDefaults() {
        return {
          size: 12
        };
      }
    
    }
    

    and I attempt to use this class in my route:

    const queryBuilder = orm.em.createQueryBuilder(Content);
    const query = req.query;
    const querier = new ContentQuerier(query, queryBuilder);
    

    but I get a TypeScript error on that final line, flagging the query and queryBuilder properties: src/routes/records.ts:19:40 - error TS2554: Expected 0 arguments, but got 2.

    Any idea why TypeScript thinks the QueryQL consturctor doesn't have any parameters? Thanks!

    help wanted 
    opened by johndanek 3
  • Total number of rows with pagination

    Total number of rows with pagination

    With pagination you often have to return two pieces of information. One is the data itself , the second is the total number of rows that would have been returned without pagination. Is there a feature or best practice to be able to return this second piece of information? I guess I can create a second count querier and share the filters with the querier that queries for the data.

    enhancement 
    opened by joris77 3
  • [FEATURE REQUEST] Support SQL AND and OR operators/mixture of operators

    [FEATURE REQUEST] Support SQL AND and OR operators/mixture of operators

    This project is awesome and seems like it would drastically simplify the building of our backend filtering/sorting/pagination, but it falls short in a critical area that I have a requirement for. As the title states, I need to be able to support the AND and OR operators, including nesting of operators. Let's say for example I have this structure:

    const sampleNoteData = {
      id: '1',
      status: 'open',
      authorId: '313'
    }
    

    I want to be able to build a query that looks like this:

    const query = {
      filter: {
        _or: {
          id: {
            in: [1, 3]
          },
          _and: {
            status: 'open',
            authorId: '313'
          }
        }
      }
    }
    

    Nothing would look much different in the QueryQL class:

    class NotesQuerier extends QueryQL {
      defineSchema(schema) {
        schema.filter('id', 'in');
        schema.filter('status', '=');
        schema.filter('authorId', '=');
      }
    }
    

    That would result in a knex query like this:

    knex('notes').whereIn('id', [1, 3]).orWhere(function() {
      this.where('status', 'open').andWhere('authorId', '313')
    });
    
    enhancement 
    opened by baker-travis 3
  • Knex not recognized

    Knex not recognized

    Hello,

    I am trying to setup a simple filter my code looks like this below:

    const QueryQL = require('@truepic/queryql')

    class UsedFlag extends QueryQL { defineSchema(dev_CareRecipients) { dev_CareRecipients.filter('UsedFlag', '=') dev_CareRecipients.page() } }

    router.get('/', async (req, res) => { const querier = new UsedFlag(req.query, knex('dev_CareRecipients')) let dev_CareRecipients

    try {
      dev_CareRecipients = await querier.run()
    } catch (error) {
      // Handle validation error, such as by passing to an Express error handler:
      next(error)
    }
    
    res.send({ dev_CareRecipients })
    

    })

    My url looks like this below

    GET http://vm213062fc:3000/dev_CareRecipients?filter[UsedFlag]=false&page=[size]=10 Could you please let me know if I am going wrong anywhere please

    invalid 
    opened by dwarak123 3
  • Using queryql with Sequelize

    Using queryql with Sequelize

    Hi! This project looks great! I'm trying to work out how to use it with our app, which currently uses the Sequelize ORM. Am I right that Sequelize needs to have a query builder in order to make an adapter for it? Looks like it doesn't yet: https://github.com/sequelize/sequelize/issues/394. If not, any hints you might be able to provide on what SequelizeAdapter might need to look like would be awesome. Here's part of what I've been trying:

    'filter:*'(model, { field, operator, value }) {
            return model.findAll({
                where: {
                    [field]: {
                        [this.operatorMap[operator]]: value,
                    },
                },
            });
        }
    

    But that returns the actual found elements, not a query builder, so I think that's not the right track?

    question 
    opened by twastvedt 2
  • Koa's query doesn't support nested parsing

    Koa's query doesn't support nested parsing

    Hey there, awesome project! I just wanted to point out something that had me stuck for a couple of hours.

    Koa's request object does not support nested parsing of the query string (it uses Node's native querystring).

    I had to install qs to parse the query string properly.

    It could be a good idea to update the docs to make sure people know they need to provide the query object in the correct structure when dealing with nested data.

    documentation 
    opened by Turbotailz 1
  • Validation Problem

    Validation Problem

    Hi Thanks for your queryql

    This is my defineValidation

    defineValidation(schema) { return { 'filter:host[=]': schema.string().required(), 'page:size': schema.number().max(100), } }

    When there is no filter in my request it dose not check filters. It checks only if filter is exists in my request. I have to check it myself

    if (!req.query.filter || !req.query.filter.host) { res.status(400).send(); return; }

    enhancement 
    opened by fdahmadi 1
  • How do you manage long query?

    How do you manage long query?

    Sometimes "batch" selection requires something like:

    /images?filter[id][in][]=229303&filter[id][in][]=393093&filter[id][in][]=102933&filter[id][in][]=3993004&filter[id][in][]=192383....(more and more filters)
    

    And will hit the URL length limitation

    So far, depends on the HTTP method, I prefer using the body to filter (e.g: on PUT { body: { filters: [...] } }) But how you advise to do it for GET method?

    question 
    opened by FPierre 1
  • BREAKING Add `field` option if different than name

    BREAKING Add `field` option if different than name

    For most, this should not be a breaking change.

    It's only breaking if you directly interact with the schema (i.e., querier.schema) and depend on the underlying data structures of filters or sorts. mapFilterFieldsToOperators has also been renamed to mapFilterNamesToOperators.

    opened by jstayton 0
Releases(v0.11.1)
  • v0.11.1(Jan 4, 2023)

  • v0.11.0(Jun 21, 2022)

  • v0.10.3(Feb 28, 2022)

  • v0.10.2(Nov 11, 2021)

  • v0.10.1(Aug 30, 2021)

  • v0.10.0(Jun 22, 2021)

    • Colocate test files with source files (aa42465)
    • Update license year (cda077a)
    • BREAKING Update Joi to latest v17 (c837eeb)
    • Update dependencies (43c2e3e)
    • Drop support for Node.js v10 (no longer maintained) (f12971b)
    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Apr 2, 2021)

    • Update dependencies (ec0001c)
    • Don't coerce string values to numbers or booleans (af6383d)
    • Update dependencies (568a552)
    • Add piratepx (b697c54)
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Dec 3, 2020)

  • v0.7.0(Dec 1, 2020)

    • Update dependencies (e76c7bc)
    • Allow "null"/empty string with Knex is/is not (f73d5f9)
    • BREAKING Use validated values to allow coercion (98cdf38)
    • Update dependencies (4b0f713)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Oct 22, 2020)

  • v0.5.1(Oct 22, 2020)

    • Don't use glob for Prettier files (7cf1c70)
    • "Out-of-the-box" => "out of the box" (ef4350b)
    • Update dependencies (d7c78e5)
    • Don't include "Release" before tag/release name (d482b94)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jul 29, 2020)

    • Update dependencies (662f699)
    • Allow filter:[not ]between strings in Knex (514ea7e)
    • Add function cache to reduce boilerplate (22818ad)
    • Remove unnecessary duplicate parse call (3e04fd3)
    • Add eslint-plugin-node (5086076)
    • Update dependencies (70799c6)
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(May 8, 2020)

    • Permit object filter value as docs state (93c6fa3)
    • Format code with Prettier v2 (b30f6c2)
    • Update dependencies (e64cf10)
    • Drop support for Node.js v8 (no longer maintained) (027eabb)
    • Permit null filter value for is, is not (50b8597)
    • Fix incorrect global config example in docs (838f1d9)
    • Update dependencies (9d9d79a)
    • Add package keywords (7cc7923)
    Source code(tar.gz)
    Source code(zip)
  • v0.3.1(Jan 31, 2020)

    • Run CircleCI jobs on Node.js v8 (oldest supporter) (2cce5a4)
    • Update dependencies (3c464c4)
    • Add link to introductory article in README (c9ab960)
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Nov 20, 2019)

    • Assume Joi returns first error (it's default) (9812873)
    • Upgrade Joi to v16 (bec09a4)
    • Format with Prettier (a621b20)
    • Update dependencies (58df576)
    • Add .prettierignore (f783f56)
    • Update dependencies (9d31b21)
    • Ignore package-lock.js (c02ddd2)
    • Properly link badges in README (a821d99)
    • Publish to npm on release (21f6015)
    • Add npm and CircleCI badges to README (496d76c)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Sep 9, 2019)

    • Update examples with how to require (810bdc6)
    • Namespace package under @truepic (cc4a3fa)
    • Flesh out documentation (e4da8e1)
    • Update dependencies (c3c4b6f)
    • Add additional package.json fields (8630f3d)
    • Add MIT license (4cb2f0c)
    • Move orchestrator parseFlat => parser flatten (5557b03)
    • Rename some variables for clarity (399afb9)
    • Include query key when building key in parsers (c405b57)
    • Update dependencies (245ae28)
    • Run all validation before applying (82d753a)
    • Update dependencies (270150e)
    • Return plain object from defineValidation for Joi (8fe74df)
    • Require adapters to define their operators (05de9b9)
    • Move querier validators into sub-directory (7be35e9)
    • Split parser validation into its own class (293da30)
    • Move apply completely to orchestrator (95238de)
    • Add adapter validation (a0960ef)
    • No need to explicitly parse before validating (ba33293)
    • Add base class for Filterer, Sorter, and Pager (dfb6e93)
    • Add more filter operators to Knex adapter (f215461)
    • Better handle no filtering, sorting, or pagination (ae03749)
    • Centralize Joi ValidationError converter service (63353de)
    • Rename build() => run() (a82be39)
    • Export the Config class for easy access (ef192b7)
    • Update dependencies (a4ebe89)
    • Add Jest for testing (c678f4a)
    • Fix sorts applied in schema order, not query order (f695c30)
    • Fix use of array reduce without initial value (39e834f)
    • Don't merge page options with previous values (5e57821)
    • Fix order value not being returned from sortsFlat (e1a9146)
    • Fix default config resetting for each instance (ddcd0c8)
    • Export adapters, errors, and validators (b657603)
    • Throw NotImplementedError if operator unhandled (4b88637)
    • Comment-out args to methods not implemented (7dac23e)
    • Move query key definition out of parsers (bf66b69)
    • Return or throw from all "abstract" methods (1f19103)
    • Add CircleCI (03f17be)
    • Add ESLint for code quality checks (08fd5ef)
    • Switch from Standard to Prettier format (b191002)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jul 19, 2019)

Owner
Truepic
Restoring authenticity to the internet
Truepic
A framework for real-time applications and REST APIs with JavaScript and TypeScript

A framework for real-time applications and REST APIs with JavaScript and TypeScript Feathers is a lightweight web-framework for creating real-time app

Feathers 14.3k Jan 1, 2023
API Services Made Easy With Node.js

Nodal API Services Made Easy with Node.js View the website at nodaljs.com. Nodal is a web server and opinionated framework for building data manipulat

Keith Horwood 4.5k Dec 26, 2022
A serverless web framework for Node.js on AWS (CloudFormation, CloudFront, API Gateway, Lambda)

---- Sorry, this project is not maintained anymore. ---- dawson is a serverless web framework for Node.js on AWS (CloudFormation, CloudFront, API Gate

dawson 717 Dec 30, 2022
A Node.js express middleware that implements API versioning for route controllers

express-version-route This npm package provides an ExpressJS middleware to load route controllers based on api versions. Implementing API Versioning i

Liran Tal 87 Nov 15, 2022
🍔 A Node.js Serverless Framework for front-end/full-stack developers. Build the application for next decade. Works on AWS, Alibaba Cloud, Tencent Cloud and traditional VM/Container. Super easy integrate with React and Vue. 🌈

Midway - 一个面向未来的云端一体 Node.js 框架 English | 简体中文 ?? 欢迎观看 Midway Serverless 2.0 发布会回放: https://www.bilibili.com/video/BV17A411T7Md 《Midway Serverless 发布

Midway.js 6.3k Jan 8, 2023
A progressive Node.js framework for building efficient, scalable, and enterprise-grade server-side applications on top of TypeScript & JavaScript (ES6, ES7, ES8) 🚀

A progressive Node.js framework for building efficient and scalable server-side applications. Description Nest is a framework for building efficient,

nestjs 53.2k Dec 31, 2022
🥚 Born to build better enterprise frameworks and apps with Node.js & Koa

Features Built-in Process Management Plugin System Framework Customization Lots of plugins Quickstart Follow the commands listed below. $ mkdir showca

egg 18.3k Dec 29, 2022
Fast and low overhead web framework, for Node.js

An efficient server implies a lower cost of the infrastructure, a better responsiveness under load and happy users. How can you efficiently handle the

Fastify 26k Jan 2, 2023
🚀 The Node.js Framework highly focused on developer ergonomics, stability and confidence

Sponsored by FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in Ind

AdonisJS Framework 13.4k Dec 31, 2022
MVC framework making it easy to write realtime, collaborative applications that run in both Node.js and browsers

Derby The Derby MVC framework makes it easy to write realtime, collaborative applications that run in both Node.js and browsers. Derby includes a powe

DerbyJS 4.7k Dec 23, 2022
Marble.js - functional reactive Node.js framework for building server-side applications, based on TypeScript and RxJS.

Functional reactive Node.js framework for building server-side applications, based on TypeScript and RxJS. Ecosystem Name Description @marblejs/core F

Marble.js 2.1k Dec 16, 2022
Actionhero is a realtime multi-transport nodejs API Server with integrated cluster capabilities and delayed tasks

Actionhero The reusable, scalable, and quick node.js API server for stateless and stateful applications NPM | Web Site | Latest Docs | GitHub | Slack

Actionhero 2.3k Jan 4, 2023
Fast, unopinionated, minimalist web framework for node.

Fast, unopinionated, minimalist web framework for node. const express = require('express') const app = express() app.get('/', function (req, res) {

null 59.5k Jan 5, 2023
Expressive middleware for node.js using ES2017 async functions

Expressive HTTP middleware framework for node.js to make web applications and APIs more enjoyable to write. Koa's middleware stack flows in a stack-li

Koa.js 33.5k Jan 4, 2023
Realtime MVC Framework for Node.js

Website Get Started Docs News Submit Issue Sails.js is a web framework that makes it easy to build custom, enterprise-grade Node.js apps. It is design

Balderdash 22.4k Dec 31, 2022
📦🔐A lightweight private proxy registry build in Node.js

Version 6 (Development branch) Looking for Verdaccio 5? Check branch 5.x. Verdaccio is a simple, zero-config-required local private npm registry. No n

Verdaccio 14.3k Dec 31, 2022
Use full ES2015+ features to develop Node.js applications, Support TypeScript.

ThinkJS Use full ES2015+ features to develop Node.js applications, Support TypeScript. 简体中文文档 Installation npm install -g think-cli Create Application

ThinkJS 5.3k Dec 30, 2022
:rocket: Progressive microservices framework for Node.js

Moleculer Moleculer is a fast, modern and powerful microservices framework for Node.js. It helps you to build efficient, reliable & scalable services.

MoleculerJS 5.5k Jan 4, 2023
Node.js framework

Node.js framework Total.js framework is a framework for Node.js platfrom written in pure JavaScript similar to PHP's Laravel or Python's Django or ASP

Total.js 4.2k Jan 2, 2023