Abstracts execution of tasks in parallel using Node.js cluster.

Overview

cluster-map

Travis build status NPM version Canonical Code Style

Abstracts execution of tasks in parallel using Node.js cluster.

It is a high level abstraction around a common pattern used to delegate a list of tasks to the workers.

API

import {
    createClusterMap
} from 'cluster-map';

/**
 * Executes tasks in parallel using Node.js cluster.
 * @typedef {Function} createClusterMap~createMap
 * @param {string[]} tasks An array of unique tasks sent to the workers.
 * @returns {Promise}
 */

/**
 * @typedef {Object} createClusterMap~configuration
 * @property {number} numberOfProcesses Defines number of processes that will be forked (default: number of OS CPUs as determined using https://nodejs.org/api/os.html#os_os_cpus).
 * @property {boolean} log Used to enable logging (https://github.com/gajus/cluster-map#logging) (default: false).
 * @property {number} timeout Used to enable logging of the tasks that take longer than the specified time (in milliseconds) (default: 5000).
 */

/**
 * Used to create a pre-configured instance of `clusterMap`.
 * @param {createClusterMap~cluster} cluster https://nodejs.org/api/cluster.html
 * @param {createClusterMap~configuration} configuration
 * @returns {createMap}
 */
createClusterMap(cluster);

handleTask function is used to receive tasks and respond to the master.

import {
    handleTask
} from 'cluster-map';


/**
 * Handles a task and returns a promise that is resolved with the result of the task.
 * @typedef {Function} handleTask~handler
 * @param {string} task
 * @returns {Promise}
 */

/**
 * @param {handleTask~handler} task
 */
handleTask(cluster);

Example

In this example,

  • Master declares an array of tasks (['task 1', 'task 2', 'task 3']).
  • createClusterMap is used to create an instance of clusterMap.
  • clusterMap is used to task the workers.
  • Workers handle the tasks and reply to the master.
  • clusterMap waits for all tasks to be processed.
  • When all tasks are processes, clusterMap resolves with an array of results.
import cluster from 'cluster';
import {
    createClusterMap,
    handleTask
} from 'cluster-map';

if (cluster.isMaster) {
    let clusterMap,
        tasks;

    tasks = [
        'task 1',
        'task 2',
        'task 3'
    ];

    clusterMap = createClusterMap(cluster);

    clusterMap(tasks)
        .then((results) => {
            console.log(results);
        });
}

if (cluster.isWorker) {
    handleTask((task) => {
        return Promise.resolve(task.slice(-1));
    });
}

Using a separate file

master.js

import cluster from 'cluster';
import {
    createClusterMap
} from 'cluster-map';

let clusterMap,
    tasks;

tasks = [
    'task 1',
    'task 2',
    'task 3'
];

cluster.setupMaster({
    exec: path.resolve(__dirname, 'worker.js')
});

clusterMap = createClusterMap(cluster);

clusterMap(tasks)
    .then((results) => {

    });

worker.js

import {
    handleTask
} from 'cluster-map';

handleTask((task) => {
    return Promise.resolve(task.slice(-1));
});

Logging

Logging is enabled using log configuration.

Logging produces an output that describes:

  • Number of forked processes.
  • Logs time when worker is assigned a task.
  • Logs time when worker responds with a result.
  • Logs time when either of the tasks take longer than the timeout configuration to execute.
[03:36:46] Spawning 8 worker process(es).
[03:36:46] Tasking worker #1. "/bin/babel-external-helpers.js"
[03:36:46] Tasking worker #3. "/bin/babel-node.js"
[03:36:46] Tasking worker #5. "/bin/babel-plugin.js"
[03:36:46] Tasking worker #4. "/bin/babel.js"
[03:36:46] Tasking worker #7. "/index.js"
[03:36:46] Tasking worker #6. "/lib/_babel-node.js"
[03:36:46] Tasking worker #2. "/lib/babel-external-helpers.js"
[03:36:46] Tasking worker #8. "/lib/babel-node.js"
[03:36:48]
Received result from:      /bin/babel-node.js
Tasks in progress (count): 7
Tasks in progress:
  - /bin/babel-external-helpers.js
  - /bin/babel-plugin.js
  - /bin/babel.js
  - /index.js
  - /lib/_babel-node.js
  - /lib/babel-external-helpers.js
  - /lib/babel-node.js
Remaining tasks (count):   495
[03:36:48] Tasking worker #3. /lib/babel-plugin/index.js
[03:36:48]
Received result from:      /bin/babel-external-helpers.js
Tasks in progress (count): 7
Tasks in progress:
  - /bin/babel-plugin.js
  - /bin/babel.js
  - /index.js
  - /lib/_babel-node.js
  - /lib/babel-external-helpers.js
  - /lib/babel-node.js
  - /lib/babel-plugin/index.js
Remaining tasks (count):   494
You might also like...

This is a To-Do List. It shows a minimalist design with the next features: Add new tasks, edit tasks, markup completed tasks, and erase all completed tasks. Built with JavaScript.

Project Name To Do List Built With HTML CSS JavaScript Live Demo To do List Live Demo Link Getting Started This is a To Do List. It shows a minimalist

Jun 9, 2022

This a todo list project that uses webpack. In this project you will find features like adding tasks, deleting tasks, maintaining localstorage, marking tasks completed and clearing all completed tasks.

webpack-Todo list This a todo list project that uses webpack. In this project you will find features like adding tasks, deleting tasks, maintaining lo

Jun 15, 2022

An extension to Async adding better handling of mixed Series / Parallel tasks via object chaining

async-chainable Flow control for NodeJS applications. This builds on the foundations of the Async library while adding better handling of mixed Series

Jul 15, 2019

Actionhero is a realtime multi-transport nodejs API Server with integrated cluster capabilities and delayed tasks

Actionhero is a realtime multi-transport nodejs API Server with integrated cluster capabilities and delayed tasks

Actionhero The reusable, scalable, and quick node.js API server for stateless and stateful applications NPM | Web Site | Latest Docs | GitHub | Slack

Dec 29, 2022

Actionhero is a realtime multi-transport nodejs API Server with integrated cluster capabilities and delayed tasks

Actionhero is a realtime multi-transport nodejs API Server with integrated cluster capabilities and delayed tasks

Actionhero The reusable, scalable, and quick node.js API server for stateless and stateful applications NPM | Web Site | Latest Docs | GitHub | Slack

Jan 4, 2023

To Do list is a small but useful project to make list app , you can add tasks delete tasks and modify tasks, the project built using HTML, CSS, JavaScript

Project Name The To-Do-List app description this is a project in the second week of the second module in microverse. its a useful to do list that save

Jul 25, 2022

Parallel/concurrent async work, optionally using multiple threads or processes

parallel-park Parallel/concurrent async work, optionally using multiple processes Usage parallel-park exports two functions: runJobs and inChildProces

Mar 1, 2022

A to do list where you can add tasks, mark completed tasks and clear all completed tasks.

A to do list where you can add tasks, mark completed tasks and clear all completed tasks.

To Do List A to do list where you can add tasks, mark completed tasks and clear all completed tasks. You can rearrange the order of the tasks by doubl

Mar 4, 2022

This project entails a To-do-List whereby a user can input the tasks they want to do, check the tasks done and also clear all tasks when all of them are completed. It is efficient for a user who want to manage their time and keep track of their day.

This project entails a To-do-List whereby a user can input the tasks they want to do, check the tasks done and also clear all tasks when all of them are completed. It is efficient for a user who want to manage their time and keep track of their day.

Screenshot Here is a screenshot for the project. To-Do-List Project This is a Microverse project that entails a to-do-list which one is able to add an

Jun 16, 2022

A simple to do list app built with HTML, CSS and JavaScript. Users can add daily tasks, edit the tasks, delete the tasks when it's done

To Do List This is a website that allows users to manage the tasks. users can add every task to do and when the task is done users can delete the task

Jul 21, 2022

Resolve parallel promises in key-value pairs whilst maintaining type information

async-kv Resolves promises in key-value pairs maintaining type information. Prerequisites NodeJS 12 or later Installation npm i async-kv yarn add asyn

Feb 17, 2022

Course material for a ~10 hours introductionary course for Julia. Topics: Introduction, Parallel Programming, Data Science

Development We use Franklin.jl to generate the lecture material. To do so, simply activate the environment, use Franklin and run the local server: act

Dec 15, 2022

Create parallel reality of your Substrate network.

Chopsticks Create parallel reality of your Substrate network. Install Make sure you have setup Rust environment (= 1.64). Clone repository with submo

Dec 15, 2022

Business class content management for Node.js (plugins, server cluster management, data-driven pages)

PencilBlue A full featured Node.js CMS and blogging platform (plugins, server cluster management, data-driven pages) First and foremost: If at any poi

Dec 30, 2022

Business class content management for Node.js (plugins, server cluster management, data-driven pages)

PencilBlue A full featured Node.js CMS and blogging platform (plugins, server cluster management, data-driven pages) First and foremost: If at any poi

Dec 30, 2022

A simple in-memory key-value cache for function execution, allowing both sync and async operations using the same methods

A simple in-memory key-value cache for function execution, allowing both sync and async operations using the same methods. It provides an invalidation mechanism based both on exact string and regex.

Dec 15, 2022

🚀 Send a load of requests with nodejs using cluster and with/without Tor for anonymisation 🙈

Accumulator 🚀 Send a load of requests with nodejs using cluster and with/without Tor for anonymisation 🙈 ⚠️ Disclamer, This repo has been created fo

Nov 21, 2022

Limit the execution rate of a function

valvelet This is a small utility to limit the execution rate of a function. It is useful for scenarios such as REST APIs consumption where the amount

Dec 26, 2022

Gulp.js command execution for humans

Gulp.js command execution for humans

Gulp.js command execution for humans. As opposed to similar plugins or to child_process.exec(), this uses Execa which provides: Better Windows support

Dec 14, 2022
Owner
Gajus Kuizinas
Software architect. Passionate about JavaScript, React, GraphQL, Redux. Active open-source contributor.
Gajus Kuizinas
Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

Node Version Manager Table of Contents About Installing and Updating Install & Update Script Additional Notes Troubleshooting on Linux Troubleshooting

nvm.sh 63.8k Jan 9, 2023
Node.js Application Configuration

Configure your Node.js Applications release notes Introduction Node-config organizes hierarchical configurations for your app deployments. It lets you

Loren West 5.9k Jan 4, 2023
Run any command on specific Node.js versions

Run any command on specific Node.js versions. Unlike nvm exec it: can run multiple Node.js versions at once can be run programmatically is 5 times fas

ehmicky 605 Dec 30, 2022
simple metadata scrapper for node.js

meta-fetcher Simple metadata scrapper for node.js. Under the hood it uses isomorphic-unfetch to fetch the metadata, parses it and returns it as json o

Rocktim 137 Nov 6, 2022
Node.js object hash library with properties/arrays sorting to provide constant hashes. It also provides a method that returns sorted object strings that can be used for object comparison without hashes.

node-object-hash Tiny and fast node.js object hash library with properties/arrays sorting to provide constant hashes. It also provides a method that r

Alexander 73 Oct 7, 2022
Node.js CLI tool to visualize an aggregate list of your dependencies' licenses

licenseye Node.js CLI tool to visualize an aggregate list of your project's dependencies' licenses. Install Yarn yarn global add licenseye NPM npm ins

Liran Tal 36 Dec 21, 2022
Clock and task scheduler for node.js applications, providing extensive control of time and callback scheduling in prod and test code

#zeit A node.js clock and scheduler, intended to take place of the global V8 object for manipulation of time and task scheduling which would be handle

David Denton 12 Dec 21, 2021
A simple JS To Do List bundled using Webpack. You can add new tasks, edit existing tasks, check completed tasks, and delete tasks from the list. It is built mainly with Javascript.

To-Do-List A simple HTML list of To Do built using webpack and served by a webpack dev server. Live Link See Demo Built With HTML, CSS, Javascript To

Michael Ugochukwu 3 May 10, 2022