๐Ÿ”Ž A simple, tiny and lightweight benchmarking library!

Overview

tinybench

Benchmark your code easily with Tinybench, a simple, tiny and light-weight 7KB (2KB minified and gzipped) benchmarking library! You can run your benchmarks in multiple JavaScript runtimes, Tinybench is completely based on the Web APIs with proper timing using process.hrtime or performance.now.

  • Accurate and precise timing based on the environment
  • Event and EventTarget compatible events
  • Statistically analyzed values
  • Calculated Percentiles
  • Fully detailed results
  • No dependencies

In case you need more tiny libraries like tinypool or tinyspy, please consider submitting an RFC

Installing

$ npm install -D tinybench

Usage

You can start benchmarking by instantiating the Bench class and adding benchmark tasks to it.

const { Bench } = require("tinybench");
const bench = new Bench({ time: 100 });

bench
  .add("foo", () => {
    // code
  })
  .add("bar", async () => {
    // code
  });

await bench.run();

The add method accepts a task name and a task function, so it can benchmark it! This method returns a reference to the Bench instance, so it's possible to use it to create an another task for that instance.

Note that the task name should always be unique in an instance, because Tinybench stores the tasks based on their names in a Map.

Docs

Bench

The Benchmark instance for keeping track of the benchmark tasks and controlling them.

Options:

export type Options = {
  /**
   * time needed for running a benchmark task (milliseconds) @default 500
   */
  time?: number;

  /**
   * number of times that a task should run if even the time option is finished @default 10
   */
  iterations?: number;

  /**
   * function to get the current timestamp in milliseconds
   */
  now?: () => number;

  /**
   * An AbortSignal for aborting the benchmark
   */
  signal?: AbortSignal;

  /**
   * warmup time (milliseconds) @default 100ms
   */
  warmupTime?: number;

  /**
   * warmup iterations @default 5
   */
  warmupIterations?: number;

  /**
   * setup function to run before each benchmark task (cycle)
   */
  setup?: Hook;

  /**
   * teardown function to run after each benchmark task (cycle)
   */
  teardown?: Hook;
};

export type Hook = (task: Task, mode: "warmup" | "run") => void | Promise<void>;
  • async run(): run the added tasks that were registered using the add method
  • async warmup(): warm up the benchmark tasks
  • reset(): reset each task and remove its result
  • add(name: string, fn: Fn): add a benchmark task to the task map
    • Fn: () => any | Promise<any>
  • remove(name: string): remove a benchmark task from the task map
  • get results(): (TaskResult | undefined)[]: (getter) tasks results as an array
  • get tasks(): Task[]: (getter) tasks as an array
  • getTask(name: string): Task | undefined: get a task based on the name

Task

A class that represents each benchmark task in Tinybench. It keeps track of the results, name, Bench instance, the task function and the number of times the task function has been executed.

  • constructor(bench: Bench, name: string, fn: Fn)
  • bench: Bench
  • name: string: task name
  • fn: Fn: the task function
  • runs: number: the number of times the task function has been executed
  • result?: TaskResult: the result object
  • async run(): run the current task and write the results in Task.result object
  • async warmup(): warm up the current task
  • setResult(result: Partial<TaskResult>): change the result object values
  • reset(): reset the task to make the Task.runs a zero-value and remove the Task.result object

TaskResult

the benchmark task result object.

export type TaskResult = {
  /*
   * the last error that was thrown while running the task
   */
  error?: unknown;

  /**
   * The amount of time in milliseconds to run the benchmark task (cycle).
   */
  totalTime: number;

  /**
   * the minimum value in the samples
   */
  min: number;
  /**
   * the maximum value in the samples
   */
  max: number;

  /**
   * the number of operations per second
   */
  hz: number;

  /**
   * how long each operation takes (ms)
   */
  period: number;

  /**
   * task samples of each task iteration time (ms)
   */
  samples: number[];

  /**
   * samples mean/average (estimate of the population mean)
   */
  mean: number;

  /**
   * samples variance (estimate of the population variance)
   */
  variance: number;

  /**
   * samples standard deviation (estimate of the population standard deviation)
   */
  sd: number;

  /**
   * standard error of the mean (a.k.a. the standard deviation of the sampling distribution of the sample mean)
   */
  sem: number;

  /**
   * degrees of freedom
   */
  df: number;

  /**
   * critical value of the samples
   */
  critical: number;

  /**
   * margin of error
   */
  moe: number;

  /**
   * relative margin of error
   */
  rme: number;

  /**
   * p75 percentile
   */
  p75: number;

  /**
   * p99 percentile
   */
  p99: number;

  /**
   * p995 percentile
   */
  p995: number;

  /**
   * p999 percentile
   */
  p999: number;
};

Events

Both the Task and Bench objects extend the EventTarget object, so you can attach listeners to different types of events in each class instance using the universal addEventListener and removeEventListener.

/**
 * Bench events
 */
export type BenchEvents =
  | "abort" // when a signal aborts
  | "complete" // when running a benchmark finishes
  | "error" // when the benchmark task throws
  | "reset" // when the reset function gets called
  | "start" // when running the benchmarks gets started
  | "warmup" // when the benchmarks start getting warmed up (before start)
  | "cycle" // when running each benchmark task gets done (cycle)
  | "add" // when a Task gets added to the Bench
  | "remove"; // when a Task gets removed of the Bench

/**
 * task events
 */
export type TaskEvents =
  | "abort"
  | "complete"
  | "error"
  | "reset"
  | "start"
  | "warmup"
  | "cycle";

For instance:

// runs on each benchmark task's cycle
bench.addEventListener("cycle", (e: BenchEvent) => {
  const task = e.task!;
});

// runs only on this benchmark task's cycle
task.addEventListener("cycle", (e: BenchEvent) => {
  const task = e.task!;
});

BenchEvent

export type BenchEvent = Event & {
  task: Task | null;
};

Prior art

Authors


Mohammad Bagher

Credits


Uzlopak

poyoho

Contributing

Feel free to create issues/discussions and then PRs for the project!

Comments
  • Incorrent timig with for loops

    Incorrent timig with for loops

    This is my code which I am testing

    import { Bench } from "tinybench";
    
    const bench = new Bench({ iterations: 100_000 });
    
    bench
      .add("loop 1", () => {
        let sum = 1;
        for (let index = 0; indexindex < 10000; indexindex++) {
          sum = sum * index;
        }
      })
      .add("loop 2", () => {
        let sum = 1;
        for (let index = 0; indexindex < 10; indexindex++) {
          sum = sum * index;
        }
      });
    
    await bench.run();
    
    console.table(
      bench.tasks.map(({ name, result }) => ({
        "Task Name": name,
        "Time (ps)": result?.totalTime,
        "Variance (ps)": result?.variance * 1000,
      }))
    );
    

    The results are same in both

    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚ (index) โ”‚ Task Name โ”‚      Time (ps)       โ”‚    Variance (ps)     โ”‚
    โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚    0    โ”‚ 'loop 1'  โ”‚ 0.002133705863210011 โ”‚ 0.010551717975465412 โ”‚
    โ”‚    1    โ”‚ 'loop 2'  โ”‚ 0.002180478539965521 โ”‚ 0.030143625437354347 โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
    
    

    I am using node v16.17.1

    opened by ShivamJoker 7
  • Results are incorrect

    Results are incorrect

    I don't know what causes this, but all results seem completely off-bounds.

    Reproduction

    /* eslint-disable no-console */
    import { Bench } from "tinybench";
    
    const suite = new Bench({ iterations: 100_000 });
    
    suite
      .add("10", () => {
        for (let i = 0; i < 10; i++);
      })
      .add("1 000", () => {
        for (let i = 0; i < 1_000; i++);
      })
      .add("100 000", () => {
        for (let i = 0; i < 100_000; i++);
      });
    
    suite.tasks.forEach((task) => {
      task.addEventListener("complete", () => {
        console.log(`${task.name}\t\t${task.result.mean * 1000}`);
      });
    });
    
    await suite.warmup();
    await suite.run();
    
    console.table(suite.tasks.map(({ name, result }) => ({ "Task Name": name, "Average Time (ps)": result.mean * 1000, "Variance (ps)": result.variance * 1000 })));
    

    Output:

    10              33.51586236828007
    1 000           33.56626915780828
    100 000         33.56723708981648
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚ (index) โ”‚ Task Name โ”‚ Average Time (ps) โ”‚    Variance (ps)    โ”‚
    โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚    0    โ”‚   '10'    โ”‚ 33.51586236828007 โ”‚ 0.13611378835207122 โ”‚
    โ”‚    1    โ”‚  '1 000'  โ”‚ 33.56626915780828 โ”‚ 0.12475075068569264 โ”‚
    โ”‚    2    โ”‚ '100 000' โ”‚ 33.56723708981648 โ”‚ 0.13970421827784552 โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
    

    Two things here:

    1. They all seem to take as long, which is not possible.
    2. They all send their "complete" event at the same time.
    opened by ecstrema 5
  • Document how to retrieve results

    Document how to retrieve results

    Add a working example in readme

    The current example does the test, but doesn't log nor output anything to a file. Trying it was very deceiving, although after some research, the lib seems great!

    Original issue

    Reproduction:

    // test.js
    import { Bench } from "tinybench";
    
    const bench = new Bench({ time: 100 });
    
    bench
      .add("Test tinybench", () => {
        let a = 8;
        let b = 1;
        const c = a;
        a = b;
        b = c;
      });
    
    console.log(await bench.run());
    
    node test,js
    

    Output:

    [ m {} ]
    

    Expected output:

    Benchmark results?

    opened by ecstrema 4
  • fix: improve task time calculate logic

    fix: improve task time calculate logic

    Solve Issue #27 and do same task time impovement

    fix:

    • Separate async function and normal function, avoid normal function affected by Promise micro task.
    • Use the sum of samples rather than totalTime, because totalTime include some extra time in event loop.
    • correct hz's calculate logic

    chore:

    • add eslint config for example folder

    Compare

    Before

    image

    After

    image
    opened by kainstar 3
  • Huge difference test result compare with benchmark.js

    Huge difference test result compare with benchmark.js

    When I compare the benchmark result between different libraries, I found a huge difference in 'ops/sec' value, there are my code and console result:

    benchmark.js

    code:

    // import Benchmark, { Suite, map, filter } from "benchmark";
    import { createRequire } from 'node:module'
    
    const require = createRequire(import.meta.url)
    const Benchmark = require('benchmark')
    var suite = new Benchmark.Suite();
    
    suite
      .add("RegExp#test", function () {
        /o/.test("Hello World!");
      })
      .add("String#indexOf", function () {
        "Hello World!".indexOf("o") > -1;
      })
      // add listeners
      .on("cycle", function (event) {
        console.log(String(event.target));
      })
      .on("complete", function (benches) {
        console.log(
          "Fastest is " +
          Benchmark.map(Benchmark.filter(benches, "fastest"), "name")
        );
      })
      // run async
      .run({ async: true });
    

    result:

    image

    tinybench

    code:

    import { Bench } from "tinybench";
    
    const bench = new Bench();
    
    bench
      .add("RegExp#test", () => {
        /o/.test("Hello World!");
      })
      .add("String#indexOf", () => {
        "Hello World!".indexOf("o") > -1;
      });
    
    await bench.run();
    
    console.table(
      bench.tasks.map(({ name, result }) => ({
        "Task Name": name,
        "Period (ms)": result.period,
        "Samples": result.samples.length,
        "ops/sec": result.hz,
        "Average Time (ps)": result?.mean * 1000,
        "Variance (ps)": result?.variance * 1000,
      }))
    );
    

    result:

    image

    You can see the benchmark.js has tens of millions ops/sec but the tinybench only has thousand.

    Maybe tinybench has some extra time cost in executing bench function? Or error time statistics logic? Or there are different explanations for 'ops/sec'?

    opened by kainstar 3
  • Sequential benchmarks should be the default

    Sequential benchmarks should be the default

    This library introduced sequential benches in #18, but there really shouldn't be any other kind of benches. Running multiple benchmarks at the same time means that they are are fighting for the same cpu time, especially since the benches aren't even launched in their own threads.

    Running non-sequential benchmarks is pointless because the results are always going to be wrong unless the benchark is completely synchronous in which case they will run sequentially despite the api used or you are testing some external server and each bench tests different server. I think the examples in readme should use the sequential api and possibly the the non-sequential api should just be removed.

    opened by FINDarkside 1
  • TypeScript error when using with Vitest + Next.js and

    TypeScript error when using with Vitest + Next.js and "skipLibCheck": false

    Next.js uses "target": "es5" due to reasons described here.

    So next build fails due to a type error when tinybench is present via Vitest:

    ../../node_modules/.pnpm/[email protected]/node_modules/tinybench/dist/index.d.ts:254:5
    Type error: Private identifiers are only available when targeting ECMAScript 2015 and higher.
    
      252 |  */
      253 | declare class Bench extends EventTarget {
    > 254 |     #private;
          |     ^
      255 |     signal?: AbortSignal;
      256 |     warmupTime: number;
      257 |     warmupIterations: number;
    

    I have "skipLibCheck": false, so setting "skipLibCheck": true would be a workaround, but I'd lose type safety.

    opened by elliottsj 1
  • Changelog tool as dependency

    Changelog tool as dependency

    Hey :) I think this package should be devDependency as it looks like it is not needed for runtime

    https://github.com/tinylibs/tinybench/blob/7aaa756ba667922b1c0f0abeb372ebaecc5edca1/package.json#L62

    opened by Exelord 1
  • Use JavaScript instead of TypeScript for examples

    Use JavaScript instead of TypeScript for examples

    Having TypeScript syntax in the readme's examples can be potentially confusing for beginners, as it's not valid syntax outside of TypeScript files. I've changed the examples to use plain JavaScript, while preserving the internal types and type documentation.

    opened by nickmccurdy 0
  • fix the sort function

    fix the sort function

    the sort function in https://github.com/tinylibs/tinybench/blob/main/src/task.ts#L76 is not correct, because it only works for strings and as you may know samples are numbers! So fix it to sort numbers. it should be something like sort((a,b) => a - b) I guess.

    opened by Aslemammad 0
Releases(v2.3.1)
Owner
Tinylibs
A place for tiny and minimal libraries
Tinylibs
EggyJS is a Javascript micro Library for simple, lightweight toast popups focused on being dependency-less, lightweight, quick and efficient.

EggyJS EggyJS is a Javascript micro Library for simple, lightweight toast popups. The goal of this library was to create something that meets the foll

Sam 10 Jan 8, 2023
Tiny JavaScript library (1kB) by CurrencyRate.today, providing simple way and advanced number, money and currency formatting and removes all formatting/cruft and returns the raw float value.

Zero dependency tiny JavaScript library (1kB bytes) by CurrencyRate.today, providing simple way and advanced number, money and currency formatting and removes all formatting/cruft and returns the raw float value.

Yurii De 11 Nov 8, 2022
A super tiny Javascript library to make DOM elements draggable and movable. ~500 bytes and no dependencies.

dragmove.js A super tiny Javascript library to make DOM elements draggable and movable. Has touch screen support. Zero dependencies and 500 bytes Gzip

Kailash Nadh 814 Dec 29, 2022
Javascript accordion - tiny and simple.

Javascript Accordion Javascript accordion - tiny and simple. Accordions are useful when you want to toggle between hiding and showing large amount of

Kenan GรผndoฤŸan 4 Dec 8, 2022
Vanilla JavaScript tabs - tiny and simple.

Vanilla JavaScript Tabs Vanilla JavaScript Tabs - simple and awesome. โ€” Inspired by the blazing fast, lightweight, cross-platform and crazy popular Va

Zoltan Toth 56 Dec 22, 2022
A tiny, reactive JavaScript library for structured state and tabular data.

A JavaScript library for structured state. Using plain old JavaScript objects to manage data gets old very quickly. It's error-prone, tricky to track

tinyplex 1.4k Jan 1, 2023
A tiny clock and date, period, or duration math library < 2k (minified/gzipped)

timewave v0.1.4 A tiny time simulation and date/time math library < 3k (minified/gzipped) const clock = Clock(new Date(2022,1,1),{tz:'America/New_York

Simon Y. Blackwell 46 Dec 18, 2022
It's a tiny freeware library on TypeScript to sinhronize page scrolling and navigation meny-bar.

Scroll progress (dual-side-scroll) v1.2.3 Assignment This tiny plugin is designed to show the progress of the page scrolling interactively. There are

Evgeny 4 May 17, 2022
Tiny js library to make DOM elements movable and resizable .

resizedrag.js Tiny js library to make DOM elements movable and resizable . Demo Here . This library has added resizing functionalities to the existing

null 4 Mar 28, 2022
A tiny JavaScript library to easily toggle the state of any HTML element in any contexts, and create UI components in no time.

A tiny JavaScript library to easily toggle the state of any HTML element in any contexts, and create UI components in no time. Dropdown, navigation bu

Matthieu Buรฉ 277 Nov 25, 2022
we try to make a tiny p2p client spec, maybe for sigchain gossip thing, maybe for simple blockchain thing

mininode Mininode is a tiny p2p client for prototyping p2p protocols. It is a specification for a set of interfaces that I made to make it easier to t

Nikolai Mushegian 8 Nov 23, 2022
Dead simple, single file, tiny byte formatter

tiny-byte-size Dead simple, no configuration needed, tiny byte formatter npm install tiny-byte-size Usage const byteSize = require('tiny-byte-size')

Mathias Buus 17 Aug 24, 2022
Tiny API that provide product/library name for a URL

JSer.info Product Name API Tiny API that provide product/library name for a URL. Usage Supported All products. curl https://jser-product-name.deno.dev

JSer.info 6 Oct 21, 2022
A tiny CRDT library.

A tiny CRDT library Manages state synchronisation. License MIT License Copyright (c) 2022 James Addison Permission is hereby granted, free of charge,

James Addison 4 Oct 29, 2022
A tiny, plugin extendable JavaScript utility library with a JQuery-like syntax.

Tiny Friggin' Utility Zapper What is it? A tiny ~7kb extendable JavaScript utility library with a JQuery like syntax for getting work done fast! If yo

Bret 4 Jun 25, 2022
Webview is a tiny cross-platform library to make web-based GUIs for desktop applications.

webview_deno deno bindings for webview Webview is a tiny cross-platform library to make web-based GUIs for desktop applications. โš ๏ธ This project is st

webview 1.2k Jan 2, 2023
Super tiny size multi-touch gestures library for the web. ใ€€ใ€€ใ€€You can touch this โ†’

Preview You can touch this โ†’ http://alloyteam.github.io/AlloyFinger/ Install You can install it via npm: npm install alloyfinger Usage var af = new Al

่…พ่ฎฏ AlloyTeam 3.3k Dec 12, 2022
A tiny, lightning fast jQuery-like library for modern browsers.

Sprint.js Sprint is a high-performance, 5KB (gzipped) DOM library for modern browsers. Sprint notably shines on bandwidth and resource constrained dev

Benjamin De Cock 4.3k Jan 3, 2023
A tiny JavaScript library to enable CSS animations when user scrolls.

A tiny JavaScript library to enable CSS animations when user scrolls. This library allows you to toggle one or more classes on an element when it appe

Matthieu Buรฉ 35 Nov 24, 2022