Why is benchmark.js not getting accurate performance data? I rewrote the test to use the Date object and it's able to determine which method is faster.

• test.js
• test_date.js

Currently to make a benchmark async, you need to set an option defer to true.

Instead of flag approach, why don't we just check if the the function passed to benchmark and saved internally as bench.fn has an argument that is called directly (simple callback) or has a promise method #resolve() that is called when the test is finished?

The following function can detect an async function for the purposes of benchmark.js

// Solution based on this hack to extract function parameters:
// http://stackoverflow.com/questions/1007981/how-to-get-function-parameter-names-values-dynamically-from-javascript
var STRIP_COMMENTS = /((\/\/.*$)|(\/\*[\s\S]*?\*\/))/mg;
function isAsyncFunction(fn) {
  var async = false;
  if (typeof fn === 'function' && fn.length > 0) {
    var fnStr = fn.toString().replace(STRIP_COMMENTS, '');
    var params = fnStr.slice(fnStr.indexOf('(')+1, fnStr.indexOf(')')).match(/([^\s,]+)/g);

    // Should we check if any params are functions or only the last param
    // since that is convention. Or will async benchmarks only ever have
    // exactly one argument, either a callback or a deferred. 
    async = params.some(function(token){
      var withoutParam = fnStr.slice(fnStr.indexOf(token)).slice(token.length);
      var atFirstUsage = withoutParam.slice(withoutParam.indexOf(token));
      var charAfterTokenUsage = atFirstUsage.charAt(token.length);

      if (charAfterTokenUsage === '(') {
        // Should we also check if the function is executed with either 
        // #call() or #apply()? Will a test callback ever be called with
        // either?
        return true;
      }
      if (charAfterTokenUsage === '.' && atFirstUsage.indexOf('resolve()') === token.length + 1) {
        // Should we also check for other methods on the deferred object
        // such as:
        return true;
      }
    });
  }
  return async;
}

This function could be modified to return a string or null instead of a boolean, where the string would be either "promise" or "callback" when the test is async. If you know the user expects a deferred, you can pass the test a deferred. And if the user expects a callback, you can pass the test a callback.

I'm still figuring out the best place to call this function to determine whether or not a test is async, and I'm also trying to figure out how to make it accept a callback.

Benchmark.js on Chromium is buggy because of the UID interpolation in the preprocess method.

RegExp.exec returns an array, so /\d+/.exec("uid123") returns ["123"].

When used with the String.replace function, it usually works fine, however on Chromium ("var r$ = 3").replace(/\$/g, ["123"]) results in "var r[123] = 3" which cannot be compiled and causes Benchmark.js to fail.

Add a Benchmark assert option to pre-run and test that it returns an expected result. This can be useful to ensure snippets being tested at least return the same result.

Firstly, thnx for all the effort you've guys put into benchmark.js - much appreciated.

I'm opening this issue more like a request for comments / question rather than a very precise bug report. My use case is the following one: I'm using benchmark.js with the code that is written using the CommonJS and the code under test needs to be require-ed in the setUp method. But since benchmark.js is "compiling" / "stringifing" benchmark methods I can use the closure scope to get my hand on the require method (or maybe I simply don't know how to do it properly...).

What I'm doing as of today is a work-arround that consists of passing a non-standard option to the benchmark like so:

{
    setup: setUp,
    teardown: function () {
        document.body.removeChild(outDiv);
    },
    require: require
}

and then, in the setUp method I'm doing: var require = this.benchmark._original.options.require;. This way I can access the require variable in both the setup and test method.

While the above works I'm kind of not happy with it and suspect that other people might have a similar use-case of passing a variable defined in the closure scope to benchmark methods. Given the above my questions are the following one:

• is there a better / canonical method of tackling this use-case?
• if not, do you think there is something we could do on the benchmark.js side to support this and similar use-cases?

Would be grateful for any input.

Hello,

I was trying benchmark performance between indexedDB#add and indexedDB#put with the same data sets, but according to this closed issue it is not possible to properly setup with asynchronous function calls in benchmark.js.

IndexedDB is asynchronous only. Opening, creating, CRUD, etc. are all done async in the browser.

Ideally my benchmark would look something like this pseudocode:

setup = function () {
  // open database connection/create db
  // create store
  // populate benchmark data
}

teardown = function () {
  // delete database
  // close db connection
}

test1 = function add () {
  // get transaction request
  // while (data) request.add
}

test1 = function put () {
  // get transaction request
  // while (data) request.put
}

I propose adding support for asynchronous for setup and teardown as more and more modern browser apis are async only.

Benchmark looks great.

Unfortunately, the example on the home page doesn't seem to work out of the box on node v0.8.

I tried making a new file and running it in node

var Benchmark = require('./benchmark.js');

var suite = new Benchmark.Suite;

// add tests
suite.add('RegExp#test', function() {
  /o/.test('Hello World!');
})
.add('String#indexOf', function() {
  'Hello World!'.indexOf('o') > -1;
})
.add('String#match', function() {
  !!'Hello World!'.match(/o/);
})
// add listeners
.on('cycle', function(event) {
  console.log(String(event.target));
})
.on('complete', function() {
  console.log('Fastest is ' + this.filter('fastest').pluck('name'));
})
// run async
.run({ 'async': true });

which will output this:

undefined
undefined
undefined
Fastest is String#indexOf

It looks like the process is working correctly, but the Event objects are not being formed correctly. The cycle event is missing the target property and looks like this:

{ type: 'cycle' }

I can see in the source that event.type is set explicitly (e.g., line 1659 for the cycle event) while event.target is coming from elsewhere (last, bench). Nothing jumps out to me about any of this though, so I was hoping someone else might see something.

Also, I'm running on the brand new node v0.8 (released yesterday), in case that could be related.

I tried this simple example (node v7):

var Benchmark = require("benchmark");

Benchmark.options.onStart = function onStart(evt){
	console.log("onStart",evt.target.name);
};

var main = Benchmark.Suite("main");

main.add("test",function(){
	var x = new Array(1400);
});

main.run();

When I run this on the command line, the output I get is:

1. I'm not sure why onStart is getting called twice?
2. I don't understand why my usage is causing this exception?

Am I missing something?

Webpack emits the following warning when we import benchmark.js

import Benchmark from 'benchmark'

WARNING in ./~/benchmark/benchmark.js
Critical dependency: the request of a dependency is an expression

Moreover, when i'm running the test suite given in this example, i get plenty of those errors even though the test completes with success. :

Uncaught ReferenceError: define is not defined

So i can't simply import this module in my project to compare the performances of two functions. Do I have to create a separate application and to define lodash in global scope, in order to compare 2 functions which are in my project ?

This would be annoying...

I have a quick question regarding benchmark.js.I'm currently building some benchmarks that require some set up before the benchmark itself can be executed and I would like this set up to be placed outside of the timed portion of the code. Is there a utility or option that benchmark.js provides to do this?

I tired onCycle,setup and onStart but none of these seem to execute just before the benchmark itself every time, which is the behavior that I need.

I'm writing a library that needs to be very efficient.

I am often making changes where I need do a benchmark every change.

Benchmark js takes around 85 samples for each item in the suite and ends up taking 30 seconds total.

Even though it would be less accurate, I'm looking for a way of making it much quicker during development. Ideally between 1 and 5 seconds.

I looked through the options and couldn't find one to make it complete faster, even though I tried all the ones that looked promising. Regardless, it takes around 85 samples.

Is there any way to do this?

The Benchmark results typically reported by these runs report a mean number of ops/sec with too many significant digits. Reporting the confidence interval is great, but all the digits reported make it hard to compare results.

I'm trying to remember the math behind this, but if the confidence interval is +/- 1%, it seems there should be no more than 3 significant digits reported, isn't that right?

This can be done fairly easily using the number's toPrecision( digits ) method. The returned value is a string and can be in scientific notation. Under normal circumstances, it shouldn't need commas.

Every test run I get different results on a simple code (for example, a simple for loop)

What settings do I need to specify in order to get stable results for tests and understand that the tests are correct and not a random error?

At first, I thought it was the number of executions of suite, but it was not.

I didn't find the detailed concept of sample on the Internet, benchmark The js source code is not understood.

The approximate guess is:

1. The sample means how many samples have been executed.
2. Each sample contains N executions of suite.
3. Each sample has a fixed execution time

I don't know if the above conjecture is correct. Please give me some advice. Thank you!

I'm trying to use the library:

var Benchmark = require('benchmark');

var str = 'A'.repeat(10000);

var suite = new Benchmark.Suite();

suite.add('partition', function() {
    //$.terminal.partition(str);
}).add('split_equal', function() {
    //$.terminal.split_equal(str, 100);
}).run({ 'async': false });

And got this error in NodeJS 18.

this is the stack trace:

/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:664
          parent = sibling.parentNode,
                           ^

TypeError: Cannot read properties of undefined (reading 'parentNode')
    at runScript (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:664:28)
    at createFunction (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:517:9)
    at createFunction (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:524:44)
    at createCompiled (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:1714:23)
    at clock (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:1608:58)
    at clock (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:1818:20)
    at cycle (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:2007:49)
    at Benchmark.run (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:2114:13)
    at execute (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:860:74)
    at invoke (/home/kuba/projects/jcubic/terminal/repo/node_modules/benchmark/benchmark.js:970:20)

I'm testing a library that uses JSDom to work properly in Node.

This week I was writing a benchmark for some of our tracing code which I'm trying to refactor and speed up.

I stared at this error for far too long:

/Users/jason.marshall/Projects/foo/node_modules/benchmark/benchmark.js:1901
            size = sample.push(clone.times.period),
                          ^

TypeError: Cannot read property 'push' of undefined
    at Array.evaluate (/Users/jason.marshall/Projects/foo/node_modules/benchmark/benchmark.js:1901:27)

After a night's sleep it dawned on me that I might be clobbering internal state:

            event.target.stats = new ...;

The fact that the error doesn't talk about 'stats' kept me from connecting the dots.

If Benchmark is assuming ES6, which I believe it does, it might make sense for the bookkeeping data to be stored under symbols instead of as regular property names. That decreases the likelihood of someone accidentally changing internal state on the object.

Alternatively, are there other ways to do setup for a benchmark? The offending code above I believe was based off of following examples in the docs. If there's a safer way to write an onStart I'd be happy to file a PR to fix the docs.

The Benchmark.js home page shows how to write a test suite, but at no point does it show how to run the test suite and get results.

There is no bin, as you would get with testing frameworks, so my best guess at the correct answer was:

lib/settings/index.bench.js

But this only results in a few lines of console code from the code under test and then nothing. No statistics, no output files.

Stepping through in a debugger, the code is definitely running. It's just not accomplishing anything.