Async utilities for node and the browser

A work in progress to bring modular support and lodash implementations of certain methods.

Previous discussion be found here #984.

Resolves #984

See https://github.com/caolan/async/tree/modularization/build for the generated bundles

TODOS

• [x] script releases:
  • generate modular builds/files
  • copy ES modules to build/es
  • add git commit
  • npm test
  • smoke test build files
  • cd build/
  • publish npm release
  • revert modular build changes, returning async to normal folder structure
  • git tag, git push
• [x] ~~possibly? create npm release for each module~~
• [x] https://github.com/caolan/async/pull/984#issuecomment-168450571
• [x] use baseOrderBy in the implementation of internal/flatten?
• [x] ~~move documentation to jsdoc format (bonus?)~~ save for later
• [x] test package with npm pack before first 2.0 publish

So, I needed the ability to iterate objects (and get the key passed into the iterator) and noticed it was not done because it would break compatibility with the Node.js standard libs. I am proposing this "convention" of adding the Of suffix on iterators that deal with objects. (instead of arrays)

If this entire concept is rejected outright, then I'll stop here. However, if this seems like a good idea/convention I can finish up adding the same concept to the other various collection iterators.

Usage:

async.forEachOf({ a: 1, b: 2 }, function (value, key, callback) {
    // value = 1, 2
    // key   = a, b
}, function (err) {
    // complete
});

This patch adds support for domains to async. The test addition is purely illustrative, i have no idea how to write or run tests the way you've got em and running npm test appears to do nothing.

The style of support:

if (process.domain) callback = process.domain.bind(callback)

Is something we discussed and established at NodeConf SummerCamp. The basic premise is that whoever owns an upper level transaction (like a web framework handling an HTTP request) will create a domain and wrap the handlers it fires for that transaction. The developer writing handlers has no idea there is a domain, and it's up to API authors (async, request, node-redis) to make sure they attach to the domain when it is active. In the case of request, it's all in an EventEmitter so it's automatically added, but for node-redis and async and other pure callback based APIs there will need to be code like this that binds the incoming callback to the domain so that any errors thrown in the handler get picked up by the transaction holder's domain.

I hope that makes sense :)

Still not sure if this is the right take to address this. The main issues is I expect there will be a perf hit as a result of creating a new array ([value, key]) at each step iteration step, instead of just passing keys directly (benchmarks to follow).

Fixes #839

Benchmarks

Comparing v1.3.0 with current on Node v2.3.3
--------------------------------------
each(10) v1.3.0 x 36,160 ops/sec ±3.60% (24 runs sampled), 0.0277ms per run
each(10) current x 35,297 ops/sec ±1.45% (28 runs sampled), 0.0283ms per run
v1.3.0 is faster
--------------------------------------
each(300) v1.3.0 x 6,017 ops/sec ±1.55% (26 runs sampled), 0.166ms per run
each(300) current x 5,883 ops/sec ±1.06% (28 runs sampled), 0.17ms per run
v1.3.0 is faster
--------------------------------------
each(10000) v1.3.0 x 215 ops/sec ±1.33% (28 runs sampled), 4.64ms per run
each(10000) current x 207 ops/sec ±3.90% (28 runs sampled), 4.84ms per run
v1.3.0 is faster
--------------------------------------
eachSeries(10) v1.3.0 x 19,552 ops/sec ±1.27% (27 runs sampled), 0.0511ms per run
eachSeries(10) current x 18,455 ops/sec ±4.19% (26 runs sampled), 0.0542ms per run
v1.3.0 is faster
--------------------------------------
eachSeries(300) v1.3.0 x 1,115 ops/sec ±2.07% (27 runs sampled), 0.897ms per run
eachSeries(300) current x 1,023 ops/sec ±2.77% (25 runs sampled), 0.978ms per run
v1.3.0 is faster
--------------------------------------
eachSeries(10000) v1.3.0 x 35.27 ops/sec ±1.47% (29 runs sampled), 28.4ms per run
eachSeries(10000) current x 31.43 ops/sec ±3.30% (27 runs sampled), 31.8ms per run
v1.3.0 is faster
--------------------------------------
eachLimit(10) v1.3.0 x 25,702 ops/sec ±4.46% (28 runs sampled), 0.0389ms per run
eachLimit(10) current x 25,670 ops/sec ±2.65% (27 runs sampled), 0.039ms per run
Tie
--------------------------------------
eachLimit(300) v1.3.0 x 2,282 ops/sec ±2.68% (28 runs sampled), 0.438ms per run
eachLimit(300) current x 2,321 ops/sec ±3.81% (25 runs sampled), 0.431ms per run
Tie
--------------------------------------
map(10) v1.3.0 x 32,804 ops/sec ±3.23% (25 runs sampled), 0.0305ms per run
map(10) current x 32,166 ops/sec ±1.56% (27 runs sampled), 0.0311ms per run
Tie
--------------------------------------
map(300) v1.3.0 x 5,134 ops/sec ±1.12% (28 runs sampled), 0.195ms per run
map(300) current x 4,830 ops/sec ±2.39% (28 runs sampled), 0.207ms per run
v1.3.0 is faster
--------------------------------------
map(10000) v1.3.0 x 172 ops/sec ±2.41% (28 runs sampled), 5.81ms per run
map(10000) current x 165 ops/sec ±2.19% (28 runs sampled), 6.07ms per run
v1.3.0 is faster
--------------------------------------
mapSeries(10) v1.3.0 x 17,971 ops/sec ±1.44% (27 runs sampled), 0.0556ms per run
mapSeries(10) current x 17,425 ops/sec ±2.19% (27 runs sampled), 0.0574ms per run
v1.3.0 is faster
--------------------------------------
mapSeries(300) v1.3.0 x 969 ops/sec ±3.29% (25 runs sampled), 1.03ms per run
mapSeries(300) current x 962 ops/sec ±2.49% (27 runs sampled), 1.04ms per run
Tie
--------------------------------------
mapSeries(10000) v1.3.0 x 30.75 ops/sec ±4.29% (26 runs sampled), 32.5ms per run
mapSeries(10000) current x 30.44 ops/sec ±5.64% (26 runs sampled), 32.8ms per run
Tie
--------------------------------------
mapLimit(10) v1.3.0 x 24,858 ops/sec ±1.12% (27 runs sampled), 0.0402ms per run
mapLimit(10) current x 25,084 ops/sec ±1.20% (28 runs sampled), 0.0399ms per run
Tie

@aearly @megawac

just a proof of concept where you can run mocha tests in node and in the browser. Works right along side the nodeunit tests, so could be transitioned over one by one.

I think its time. Everything in the milestone is complete, and the docs site is up and running. (except for the during inconsistency issue, which I think is just a docs fix)

I think 2.0.0 is ready for to be released into the wild!

cc @megawac

Add promise support to allow mix different tasks:

async.waterfall([
    function (done) {
        // Do something and call done()
        fs.readFile(filepath, done);
    },
    function(content) {
        // Do something and return Promise
        return mongoose.models.file.create({
            name: filepath,
            content: content
        });
    }
], (err) => {
    if (err) {
        console.error(err);
    }
});

I just forked this project specifically so I could use priorityQueue() and virtually nothing else. It occurs to me that this is a long-term mistake because I'll miss out on future improvements to this function unless I manually import them.

May I suggest splitting async.js into multiple files in a lib sub-directory, to facilitate delivering async's functionality to the browser whilst eschewing unnecessary bytes?

This may be of little benefit for server-side users, but every byte counts in the front-end.

Also, whilst undertaking this pruning, it occurred to me that dependencies between internal and exported methods isn't always clear unless you carefully read the source code. Modularising could assist in making these interdependencies more clear.

What do you think? Should I start a Pull Request for this, or is a bad idea / waste of time?

Merges the implementations of some and every and adds their respective limit functions (with partial short circuiting)

PS I think short circuiting should be done like lodash does it (allowing each to break): if cb resolves with cb(false) you break the loop (in lodash if you return false it breaks the loop)

Resolves #548

I had some case where the second error response was called during the callback of the first error response in an async call before the callback could be replaced with a noop. That resulted in the error case where the callback was called twice with an error message. That was pretty hard to debug. I figured that the same error could occur at other places as well and changed it there too. It would be nice if someone could provide unit tests for these changes, else I will provide them as soon as I have time for it.

Currently the applyEach and applyEachSeries functions have the following type signature:

async.applyEach(fns, ...args?, callback?)

Having a rest parameter in the middle of the arguments list is very difficult to write a type annotation for. It's currently not possible in either Flow or TypeScript, which is why you see stuff like this (I'm writing a similar type definition for Flow right now).

Would it be okay to switch the argument order around and make callback nullable so it is like this?

async.applyEach(fns, callback | null, ...args?)

Or use an array instead of a rest parameter?

async.applyEach(fns, args?: Array<any>, callback?)

What version of async are you using?

3.2.4

Which environment did the issue occur in (Node/browser/Babel/Typescript version)

Node

What did you do? Please include a minimal reproducible case illustrating issue.

Using async.queue I get different execution results when I use promises instead of callbacks.

https://stackblitz.com/edit/node-qeye7d?file=index.js

You can execute the code in this example by typing node index in the stackblitz terminal.

What did you expect to happen?

Results should be the same.

What was the actual result?

Results were different in terms of order of execution.

Here's my code:

const async = require('async');

const myFunc = async () => {
  // CALLBACKS
  console.log();
  console.log('CALLBACKS');

  const worker1 = (job, callback) => {
    console.log('job started', job.name);
    setTimeout(() => callback(null, job), 1000);
  };

  const q1 = async.queue(worker1, 1);

  q1.push({ name: 'job1' }, (err, job) => {
    console.log('job done', job);
  });
  q1.push({ name: 'job2' }, (err, job) => {
    console.log('job done', job);
  });

  await q1.drain();

  // PROMISES
  console.log();
  console.log('PROMISES');

  const worker2 = async (job) => {
    return new Promise((resolve) => {
      console.log('job started', job.name);
      setTimeout(() => resolve(job), 1000);
    });
  };

  const q2 = async.queue(worker2, 1);

  q2.push({ name: 'job1' }).then((job) => {
    console.log('job done', job);
  });
  q2.push({ name: 'job2' }).then((job) => {
    console.log('job done', job);
  });

  await q2.drain();
};

myFunc();

Actual output is:

CALLBACKS job started job1 job done { name: 'job1' } job started job2 job done { name: 'job2' }

PROMISES job started job1 job started job2 job done { name: 'job1' } job done { name: 'job2' }

Expected output is:

CALLBACKS job started job1 job done { name: 'job1' } job started job2 job done { name: 'job2' }

PROMISES job started job1 job done { name: 'job1' } job started job2 job done { name: 'job2' }

I think it would be nice to add adaptive parallelism to existing functions or create new ones.

https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581

For example: There is a maximum, minimum, initial and current limit. There is a callback that accepts an error that occurred during the call (for example, an error related to parallelism or another error) and returns a boolean (whether to reduce the limit or not).

Bumps chai from 4.3.6 to 4.3.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps yargs from 17.5.1 to 17.6.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps jsdoc from 3.6.11 to 4.0.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps fs-extra from 10.1.0 to 11.1.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.