An async control-flow library that makes stepping through logic easy.

Tim Caswell

Last update: Dec 22, 2022

Related tags

Control Flow step

Overview

Step

A simple control-flow library for node.JS that makes parallel execution, serial execution, and error handling painless.

How to install

Simply copy or link the lib/step.js file into your $HOME/.node_libraries folder.

How to use

The step library exports a single function I call Step. It accepts any number of functions as arguments and runs them in serial order using the passed in this context as the callback to the next step.

Step(
  function readSelf() {
    fs.readFile(__filename, this);
  },
  function capitalize(err, text) {
    if (err) throw err;
    return text.toUpperCase();
  },
  function showIt(err, newText) {
    if (err) throw err;
    console.log(newText);
  }
);

Notice that we pass in this as the callback to fs.readFile. When the file read completes, step will send the result as the arguments to the next function in the chain. Then in the capitalize function we're doing synchronous work so we can simple return the new value and Step will route it as if we called the callback.

The first parameter is reserved for errors since this is the node standard. Also any exceptions thrown are caught and passed as the first argument to the next function. As long as you don't nest callback functions inline your main functions this prevents there from ever being any uncaught exceptions. This is very important for long running node.JS servers since a single uncaught exception can bring the whole server down.

Also there is support for parallel actions:

Step(
  // Loads two files in parallel
  function loadStuff() {
    fs.readFile(__filename, this.parallel());
    fs.readFile("/etc/passwd", this.parallel());
  },
  // Show the result when done
  function showStuff(err, code, users) {
    if (err) throw err;
    console.log(code);
    console.log(users);
  }
)

Here we pass this.parallel() instead of this as the callback. It internally keeps track of the number of callbacks issued and preserves their order then giving the result to the next step after all have finished. If there is an error in any of the parallel actions, it will be passed as the first argument to the next step.

Also you can use group with a dynamic number of common tasks.

Step(
  function readDir() {
    fs.readdir(__dirname, this);
  },
  function readFiles(err, results) {
    if (err) throw err;
    // Create a new group
    var group = this.group();
    results.forEach(function (filename) {
      if (/\.js$/.test(filename)) {
        fs.readFile(__dirname + "/" + filename, 'utf8', group());
      }
    });
  },
  function showAll(err , files) {
    if (err) throw err;
    console.dir(files);
  }
);

Note that we both call this.group() and group(). The first reserves a slot in the parameters of the next step, then calling group() generates the individual callbacks and increments the internal counter.

Comments

Serial in a step function ?

I have been looking for a simple way to chain actions when looping over an array but did not found any. It could be great if we can simply do :

Step(
  function() {
    var serial = this.serial;
    arr.forEach(function(el) {
      asyncFunction(el, serial);
    });
  },
  function end() {
    // everything's done
  }
);

but I did not succeeded in hacking the lib so I just wrote this function for the moment:

  function arrayChain(arr, func, callback) {
    var actions = arr.map(function(el) { return function() { func(el) }; });
    actions.push(callback);
    Step.apply(this, actions);
  };

Do you think this feature is useful/possible ?

opened by Tug 11

empty group()s should not complete automatically.
If the steps for a group are defined asynchronous to the this.group() call (i.e. subsequent to the check() call scheduled by next.group), localCallback() ends up being called prematurely since no functions have been added to the group yet.

The specific case where I ran into this was when trying to do the following, where the group() call appears in the callback to an intermediate, async, function:

step( function () { var group = this.group(); LIST_OF_STATES.forEach(function(state) { // intermediate async action request('url_get_counties_by_state', function(err, countiesInfo) { countiesInfo.forEach(function(county) { save_to_db(county, group()); // group here } }); }); }, function (err, counties) { // print out list of all counties in all states } );

(Note: This is slightly problematic in the case where this.group() is called in anticipation of some work being done, but where it's later discovered that there's no actual work needed. However that seems like even more of an edge case then the above, where work is being scheduled asynchronously.)
opened by broofa 9

step ignores wrong javascript code silently

When I execute the following program I expect it to crash or trow errors. However the first function is ignored and 'step 2' is printed on the console.

var step = require ('step');

if (!module.parent)
  step (
    function () {
      asdf // <=== THIS IS WRONG
      console.log ('step 1');
      this ();
    },
    function () {
      console.log ('step 2');
      this ();
    }
  )

opened by lovebug356 5

Sharing a 'data' object between steps

I LOVE step. I use it in ALL my node.js code.

This pull request adds a single line of code to step. A 'data' object that can be used to pass data between steps.

Basically I got the idea from Nue: http://dailyjs.com/2012/02/20/new-flow-control-libraries/

Without this, to use data from step 2 in step 5 you have to declare a variable outside of the overall step()

With this it's as simple as assigning to this.data.foo and then accessing it in a later step.

It's made my code cleaner, and think it would be a good candidate for inclusion into step.

Thoughts?

opened by Sembiance 5
More sophisticated options (could be v0.0.4)
Provide an option where step ignores normal return values. This is necessary when you use step in CoffeeScript where the last expression inside a function is always returned. Usually you do not need to use the sync return value, so you can switch it off by:

step.async()

Many CoffeeScript users will appreciate this change.

Be able to pass the functions' context:

step( contextObject, function (e, next) { // ... next(e, 4) }, function (e, result, next) { doAsync(next.parallel()) doAsync(next.parallel()) }, function (e, r1, r2) { // ... } )

If you omit the contextObject, "next" will still be the context of the calls but is passed as the last argument as well. Convention: The first argument of every step is the error, the number of further arguments is dynamic, the last argument is "next"

Result: There are no significant API changes, so nobody needs to change any code, but you have more sophisticated options to make step more useful.

I could send a patch to integrate this functionality.

Andi
opened by akidee 5
There needs to be a this.destroy

There has to be a this.destroy() that releases all the functions in a step, otherwise there is a memory leak in the case of any error or in the case that a function wants to exit without completing the steps.

Simple leak case:

step(function(){ return; }, function(){ });

2nd function is leaked. The 1st should be allowed to call this.destroy() before returning.

opened by MarkKahn 5

Critical: Step is skipping steps!

Still trying to track down the exact issue, but here's a code snip:

getFileList( dir, dir, cb );
function getFileList( dir, root, cb ){
    var files
      , stats
      , outFiles = [];

    step( function(){
        fs.readdir( dir, this );

    }, function( err, _files ){
        if( err ){ throw err; }

        files = _files;

        var stats = this.group();
        for( var i=0, l=files.length; i<l; i++ ){
            fs.stat( path.join( dir, files[i] ), stats() );
        };

    }, function( err, _stats ){
console.log( 'HERE1' );
console.trace();
    }, function( err, files ){
console.log( 'HERE2' );
console.trace();
        cb( null, files );
    } );
}

from which I get (75% of the time):

HERE1
Trace: 
    at Function.<anonymous> (../docgen/fileList.js:42:9)
    at next (../docgen/node_modules/step/lib/step.js:51:23)
    at check (../docgen/node_modules/step/lib/step.js:73:14)
    at ../docgen/node_modules/step/lib/step.js:86:20
    at check (../docgen/node_modules/step/lib/step.js:101:9)
    at ../docgen/node_modules/step/lib/step.js:118:22
HERE2
Trace: 
    at Function.<anonymous> (../docgen/fileList.js:45:9)
    at next (../docgen/node_modules/step/lib/step.js:51:23)
    at Array.check [as 0] (../docgen/node_modules/step/lib/step.js:73:14)
    at EventEmitter._tickCallback (node.js:126:26)

The other 25% of the time it hangs as expected.

opened by MarkKahn 4

Support all arguments on callback instead of just 1

In the code below, the request module sends the body as the second argument to the callback and i was not getting it passed to me. This makes a change to send all the arguments. So it becomes an array of arrays.

var request = require('request');
var Step = require('step');

var sites = ["www.google.com", "www.yahoo.com", "www.linkedin.com", "www.apple.com", "facebook.com", "palm.com"];


Step(
  function() {
    var group = this.group();
    for (var i=0; i < sites.length; i++) {
      request({uri: "http://" + sites[i]}, group());
    }
   },

  function(err, responses) {
    for (var i = 1; i < responses.length; i++) {
        console.log(responses[i][1]);
        console.log("=======================");
      };
  }
)

opened by phegaro 4

2 major improvements to step
For CoffeeScript users: Be able to ignore returned values

Be able to preserve context, instead use "next" No API changes, so everything is backward compatible.
opened by akidee 4
Remove nextTick check for groups.

Removing this line would be nice. :) https://github.com/creationix/step/blob/master/lib/step.js#L102

This would cause issues when no group() has been called, but it would allow groups to be setup later (in callbacks), which would be pretty cool.

opened by bminer 3
A bug in Step
I think I've found a bug in step, which may cause code executing order to be changed.

This happens when function check in next.parallel executed before the Step sequence finished.

Here is an small example illustrating this:

function C(param, callback) { Step( function D(){ do_some_IO(param, this); }, function E(err, some_data){ this(undefined, some_data); }, callback ); } Step( function A(err) { IO1(data, this.parallel(); IO2(data, this.parallel()); }, function B(err, data1, data2) { C(data, this); }, function F(err, data) { // finish } );

we write this code expecting it running in this order, A->B->C->D->E->F, but in fact, the order may be changed to A->B->C->D->F->E

when A is executing, this.parallel() will set 2 process.nextTick in step, then, when D is performing IO, process.nextTick begins as the the next event loop, and F is shifted from the queue and executed as the next fn in step(in function next), thus F will be executed before E, causing the whole program in error.

So I suggest that we could add a variable telling whether check has been executed by callback, and avoid executing check if check has been invoked by callback. By the way, I think function next.group is suffering from the same bug, but I did not write code to prove it.

I encountered this problem when modifying code of wheat. Thanks a lot for writing such great code.

regards, zTrix
opened by zTrix 3
Alter parallel() and group() functionality

Reasons for this pull request can be seen in issue 58.

One possible issue, up to the author is that I make use of the spread and rest operators (...) which is only available from node version 6.0.0 and onwards.

opened by clethrill 3

Returning multiple parameters in a callback that has been grouped

I have this code on node v6.3.1 with step 1.0.0

var Step = require('step');

Step(
	function one() {
		doThis(0, "0", this);
	},
	function two(err, number, string) {
		console.log(err, number, string);

		var group = this.group();
		for (var i=0; i<2; i++) {
			doThis(i, String(i), group());
		}
	},
	function three(err, numbers, strings) {
		console.log(err, numbers, strings);
		if (err) process.exit(1);
		process.exit(0);
	}
);

function doThis(number, string, callback) {
	callback(null, number, string);
}

I get the following output

null 0 '0'
undefined [ 0, 1 ] undefined

So one calls do this which successfully callbacks with 0 and '0' but then I put it into a loop and use this.group() and it can no longer return the second parameter.

the expected output would be

null 0 '0'
null [ 0, 1 ] [ '0', '1' ]

opened by clethrill 5

Add spm support

A nicer package manager: http://spmjs.io Documentation: http://spmjs.io/documentation

http://spmjs.io/package/step

It is a package manager based on Sea.js which is a popular module loader in China.

spmjs focus in browser side. We supply a complete lifecycle managment of package by using spm.

If you need ownership of step in spmjs, I can add it for your account after signing in http://spmjs.io.

opened by sorrycc 0
group: make 'result-as-the-first-parameter' from EventEmitters available

From http://howtonode.org/control-flow-part-ii:

For cases where there are more than two events and/or they can be called multiple times, then you need the more powerful and flexible EventEmitters.

Using group() for http.get() which is EventEmitter isn't possible with Step's only "The Node.js Callback style"...

opened by olecom 0
nextTick fallback with setImmediate and setTimeout

Uses setImmediate instead of setTimeout as a fallback for nextTick when setImmediate is available. Improves prabirshrestha's browser fallback in IE10+ and makes Step work with setImmediate polyfills. Related to #45 and #32.

opened by shebson 0