Apply coding rules as described in contributing:

• Use arrow functions in mappers, filters, reducers and combiners
• Use let and const in place of var
• Use array or object destructuring to set variables from array or object: let [a, b] = [1, 2, 3]

https://github.com/skale-me/skale-engine/blob/41f4a09c5d254f5d3b1b8598efaeba5aecb69464/lib/client.js#L339-L340

I just found this for-loop which seems to me as a bug. You are using the variable i as an index to dev, but the i here contains the item itself (or am I missing something?)

Therefore, instead of:

for (var i in dev)
      self.hostId[dev[i].uuid] = dev[i].id;

I would write:

for (const d in dev) {
      self.hostId[d.uuid] = d.id;
}

It seems that there's no way to tell the server not to start any local workers. Using --local=0 was what I expected would turn off any local workers, but that didn't seem to work. Maybe I misunderstood how the command should be used, but if not this will fix that.

$ skale create dd2bis create application dd2bis

[email protected] preinstall /home/felix/skale/cli/dd2bis/node_modules/.staging/skale-engine-1126d865 mkdir -p node_modules && echo "module.exports = require('..');" > node_modules/skale-engine.js

[email protected] /home/felix/skale/cli/dd2bis -- [email protected] +-- [email protected] +-- [email protected] +-- [email protected] +-- [email protected] |-- [email protected] +-- [email protected] +-- [email protected] +-- [email protected] +-- [email protected] | +-- [email protected] | | +-- [email protected] | | | -- [email protected] | | |-- [email protected] | | -- [email protected] | | +-- [email protected] | | +-- [email protected] | | +-- [email protected] | | +-- [email protected] | | +-- [email protected] | |-- [email protected] | +-- [email protected] | +-- [email protected] | | -- [email protected] |-- [email protected] -- [email protected] +-- [email protected]-- [email protected]

Project dd2bis is now ready. Pleas change directory to dd2bis: "cd dd2bis" To run your app: "skale run" To modify your app: edit dd2bis.js

$ cd dd2bis $ skale run /home/felix/skale/cli/dd2bis/node_modules/skale-engine/lib/dataset.js:171 result = combiner(result, res.data); ^ TypeError: Cannot read property 'data' of null at taskDone (/home/felix/skale/cli/dd2bis/node_modules/skale-engine/lib/dataset.js:171:33) at Object. (/home/felix/skale/cli/dd2bis/node_modules/skale-engine/lib/context.js:129:4) at Consumer._transform (/home/felix/skale/cli/dd2bis/node_modules/skale-engine/lib/client.js:120:32) at Consumer.Transform._read (_stream_transform.js:167:10) at Consumer.Transform._write (_stream_transform.js:155:12) at doWrite (_stream_writable.js:301:12) at writeOrBuffer (_stream_writable.js:287:5) at Consumer.Writable.write (_stream_writable.js:215:11) at FromGrid.ondata (_stream_readable.js:536:20) at emitOne (events.js:77:13)

A new skale context method sc.require is added. It specifies a set of modules on user side (master) which have to be deployed in workers for use by callbacks, such as mappers, reducers, etc.

Under the hood, browserify is used on master side to build a bundle which is serialized and sent to workers (as part of task). It is then evaluated in worker global context, and dependencies remain persistent as long as workers live.

This method allows to use in workers any javascript module which can be browserified, so a large number (almost any pure JS package).

The current commit is for local version, not distributed (code will be exactly the same). It is experimental for the moment.

An additional statement sc.bundle could be added as well to inject pre-compiled modules, avoiding the penalty of browserify at each run.

Hi,

For one of my customers, I m interested in making a trial with Skale.

However I need to be able to extract the data from RethinkDB and then process it further. I read #144 but that was not clear if the solution with sc.objectStream would scale.

So, in other words, would you mind writing some how-to instructions on how-to implement the connector on the workers, so I could potentially get this sorted ?

Thank you.

It looks like the nworker parameter is ignored, and instead the internal variable of the same name is set to the value of the local parameter. It seems like nworker should either be removed, or it should be used to set the value of the internal variable instead of local.

I'd be happy to provide a PR if that's desired.

From what I understood meeting with @CedricArtigue , JS was chosen among other things because of the community and that you could have some predictable performance (by reusing objects which prevents triggering GC) and a more reasonable workflow than what's possible with Scala. Another idea for predictable performance would be to use Rust. It has C-like performance (and predictable memory characteristics unlike JS, see #52 ), static typing, awesome community, easy parallelism, actual threads (unlike Node which only has processes), safe memory.

It's possible to expose a Node.js API via a module which code is written in Rust. See https://blog.risingstack.com/how-to-use-rust-with-node-when-performance-matters/ http://calculist.org/blog/2015/12/23/neon-node-rust/

Taking a step back, it's impossible to assess accurately the size of a JavaScript value as it can varies across implementations and according to the context (especially for objects). Also, I'm not sure how much the accuracy of the function matters, so I don't know whether giving feedback is any valuable to you. That said, here is some feedback on https://github.com/skale-me/skale-engine/blob/master/lib/sizeof.js

Boolean

Boolean being 4 bytes seems idiotic. In all reasonable cases, boolean object properties will certainly be 1 byte if not 1 bit (packed in a byte if there are several). At least, it's simple enough a optimization that implementations likely have done it.

Number

For integers, it's very likely JS engines actually store them in 4 bytes instead of 8 as the spec suggests for numbers.

String

I don't know for strings, but I know some time ago, the Mozilla JS team was considering storing ECMAScript strings as UTF-8 (or latin1) and only convert to the spec encoding if that becomes necessary because of subtle string manipulations, so UCS-2 should not be assumed from implementations (I don't know where V8 stands).

Object

obj instanceof Array should certainly be Array.isArray(obj). The array length probably takes 4 bytes as well.

Additionally, you probably want to distinguish Node's Buffer, because they'll likely be used as data and their size is deterministic (that's probably the only object type that is so).

Your object size does not take the "hidden class" size into account. Not sure how much that matters.

No clue how much a Date weighs, but your functions certainly guesses wrong as it does not look for "internal" properties.

Symbols

In Node.js v6:

> typeof Symbol()
'symbol'

This case is not taken into account by your function. 0 will be returned now. Maybe it's fine, but at least the code should be explicit about it.

To reproduce, in terminal 1:

$ cd skale-engine
$ ./bin/server -l 2

In terminal 2:

$ cd skale-engine
$ ./examples/core/parallelize.js
[ 1, 2, 3, 4 ]

assert.js:89
  throw new assert.AssertionError({
  ^
AssertionError: false == true
    at Console.assert (console.js:94:23)
    at /Users/marc/github/skale-engine/examples/core/parallelize.js:9:10
    at /Users/marc/github/skale-engine/node_modules/stream-to-array/index.js:54:9
    at _combinedTickCallback (node.js:376:9)
    at process._tickCallback (node.js:407:11)

The output should be [ 1, 2, 3, 4, 5 ]

Bumps ws from 6.1.2 to 7.4.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.