After doing a fresh git clone of the core repo, there seem to be problems with npm i. See the output below. Removing the package-lock.json fixes the corrupt tarball issue but then that's followed by a No matching version found package error.

Let me know if I've done something dumb and the issues below are related to my environment. btw, I can work around these issues with linked packages but then I'm not so confident about whether my package versions are in sync but at least I can get the code to run.

$ git clone https://github.com/neume-network/core.git
Cloning into 'core'...
remote: Enumerating objects: 457, done.
remote: Counting objects: 100% (74/74), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 457 (delta 45), reused 36 (delta 26), pack-reused 383
Receiving objects: 100% (457/457), 733.78 KiB | 1.59 MiB/s, done.
Resolving deltas: 100% (231/231), done.
robins @ elio ~/dev/web3/neume-network
$ cd core
robins @ elio ~/dev/web3/neume-network/core [main]
$ npm i
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN tarball tarball data for @neume-network/extraction-worker@https://registry.npmjs.org/@neume-network/extraction-worker/-/extraction-worker-0.0.3.tgz (sha512-58nz/lASFuRSsKUi6RYfPHeDElxfCSLBRGNoGdOXQPW16aT5iLdLDdEW6AAMGMmwAemkOYOcTH26Su1YMUNI6g==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for @neume-network/extraction-worker@https://registry.npmjs.org/@neume-network/extraction-worker/-/extraction-worker-0.0.3.tgz (sha512-58nz/lASFuRSsKUi6RYfPHeDElxfCSLBRGNoGdOXQPW16aT5iLdLDdEW6AAMGMmwAemkOYOcTH26Su1YMUNI6g==) seems to be corrupted. Trying again.
npm ERR! code EINTEGRITY
npm ERR! sha512-58nz/lASFuRSsKUi6RYfPHeDElxfCSLBRGNoGdOXQPW16aT5iLdLDdEW6AAMGMmwAemkOYOcTH26Su1YMUNI6g== integrity checksum failed when using sha512: wanted sha512-58nz/lASFuRSsKUi6RYfPHeDElxfCSLBRGNoGdOXQPW16aT5iLdLDdEW6AAMGMmwAemkOYOcTH26Su1YMUNI6g== but got sha512-yQCpSKC7J3aR7/R82ZhAJOGeYpKC/Uzsadour+fv1+6MKho0t4jqot+eiOwio4zCCkZiF+oiEm5lBjKZVWF8tQ==. (41334 bytes)

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/robins/.npm/_logs/2022-06-02T03_55_27_730Z-debug-0.log
robins @ elio ~/dev/web3/neume-network/core [main]
$ rm package-lock.json
robins @ elio ~/dev/web3/neume-network/core [main]
$ npm i
npm ERR! code ETARGET
npm ERR! notarget No matching version found for @neume-network/[email protected].
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/robins/.npm/_logs/2022-06-02T03_56_20_787Z-debug-0.log

Summary

Currently, the lone submodule points to the SSH URL:

https://github.com/neume-network/core/blob/8afe75b6ce1c3ff3ba0096f4a6b8acfa78e81dab/.gitmodules#L3

For new users (like me!), it's possible that we don't have SSH keys set up by default with Git. In that case, Git will produce this funny error:

[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:neume-network/strategies.git' into submodule path '/workspaces/core/src/strategies' failed
Failed to clone 'src/strategies'. Retry scheduled
Cloning into '/workspaces/core/src/strategies'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

My gut tells me that HTTPS on the other hand should "just work" for most configurations without any additional setup (happy to be corrected if that's actually false 😂).

Suggested change

-      url = [email protected]:neume-network/strategies.git
+      url = https://github.com/neume-network/strategies.git

Happy to cut a PR for this, if there are no objections ✌️

If the control flow reaches here then there isn't anything else to execute. But the process won't exit because worker is still running. So, we force close using process.exit().

As the title suggested, added a pull request template for the repo.

For the checklist, I am not how much requirement you would like to see, but I listed 3 for now. This can always improve or adjust as the project is growing.

Let me know if you have anything else to add or feedback.

Close #43

As a part of our internal prototyping, we would be very happy if we could have this dateset updated n times a day with new content to simulate realtime updates.

• this one and others: https://github.com/neume-network/core/blob/83dcf4ca4fd7943935c3c26a7c59569a609c4ebf/crawl_path.mjs#L12
• the first strategy must be call-block-logs followed by logs-to-subgraph

Partially fix #65

This PR adds a test for boot.mjs to check creation of worker. In situations when worker data is incorrect the test will fail.

I couldn't add a test for worker_start.mjs. The main reason is that isMainThread is false due to ava executing the tests in a worker thread.

https://github.com/neume-network/core/blob/472f9ef9c4ccc9555496fa8bc1f2cccf38d9df60/src/worker_start.mjs#L20-L25

It seems that the core src/stategies submodule isn't updating after recent strategies updates. Not quite sure of the root clause of this but it seems that the default branch on clone is a local HEAD branch (which matches the committed core src/strategies subproject commit id). Checking out the 'main' src/strategies branch updates src/strategies to latest and causes an update to the src/strategies subject commit id in the core repo, which then needs to be committed and pushed as an update to github.

It seems that if the core repo is to be kept in-sync with the strategies submodule updates then corresponding core repo updates must made. The updates can diverge but then repo users must make sure to update the submodule each time before use. Sounds like fair practice but it is also an easy step to miss or forget.

Note: submodules are not really my area so caveat the above with this in mind.

We have multiple packages: core, strategies, message-schema, extractor which we are either imported through npm or through git submodule.

The problem with importing packages through npm is that for development I have to modify the code in node_modules. If the code works I have to copy it from node_modules to the actual repo in order to make a pull request. If I made changes across multiple packages. It can cause multiple pull requests for the same feature across multiple repos. Which can be hard to track imo.

The problem with submodules is that they are a little confusing. At least, in my opinion. The problem of multiple PRs across repos for the same feature is a problem with submodules too.

Monorepo architecture is gaining popularity. For example, jest is a monorepo.

Since version 8, npm supports monorepo out of the box so theoretically we won't be introducing any new tools.

This PR tries to implement a basic CLI for neume-network/core.

Usage of CLI

neume --path <crawl-path-file>

Here is a sample crawl path file. The neume-network/core project can still be locally run using npm run dev. The CLI options and commands can be improved in the future as per the needs.

Tests

Even though a new functionality has been added I couldn't think of any unit tests. All the new functionality is pretty straight forward.

• currently, when users download neume-network/core and run it, there's a high chance that bugs in the crawl prevent it from concluding
• generally, although we have many unit tests, the integration between modules can lead to errors that we're currently not tracking. The more we test, the more apparent these errors will become and we can increase the turn-around time to fix them
• hence, there should be an integration tests that runs periodically, e.g. once a day, that downloads the entire neume-network/core codebase, installs all dependencies and starts running a real crawl
• finally, after the crawl has finished, some tolerant health measures should be evaluate (e.g. if all data-dir files of all existing strategies have been created)
• the health measurement shouldn't enforce strict assertions e.g. (assert(file.length, 1234) as we must be conscious of the non-deterministic nature of the crawl

• among other files, at least for the specific /data implementation: https://github.com/neume-network/data/blob/main/scripts/generateBlockLogsCrawlPath seems to be the file that orchestrates the stateful logic of the neume crawl
• now, if we had neume running on ubuntu, then it'd probably be different from the continuous invocation on /data through GitHub actions
• or maybe not and we'd be triggering runs with crontab -e as well?
• but anyhow, we now can use neume/core in a JavaScript context https://github.com/neume-network/core#javascript-usage and so it'd be interesting to have @neume-network/orchestrator that basically takes over whatever the net of scripts in /data does

• although having RPC_HTTP_HOST being slowed down is mostly useful for e.g. infura users, it can actually be really confusing to someone wanting to blast through all block logs fast and wondering why there is no performance whatsoever
• https://github.com/neume-network/core/blob/1cd0595ee009dc109c00855966407561686193cc/config.mjs#L10

• A statement like this will just stop the process after a short while without throwing any noticeable error message
• we want the program to clearly indicate that something is very wrong

  {
      name: "call-block-logs",
      extraction: {
        args: [],
      },
    }

• We're again in npm version hell as @neume-network/schema is now being used everywhere individually
  • https://github.com/neume-network/core/blob/1cd0595ee009dc109c00855966407561686193cc/package.json#L39
  • https://github.com/neume-network/strategies/blob/54631d99cbb1861c9c1381a7b0d5209da7465518/package.json#L50
  • https://github.com/neume-network/extraction-worker/blob/ed02ece7d6487363cc8c3db0207ffd979574f66c/package.json#L62
• We once had the same problem with eth-fun until we just said that we make it a mandatory peer dependency in all downstream packages (strategies, extraction worker) and that we truly require it for @neume-network/core though
• This has worked well IMO and so we should do the same now with @neume-network/schema

• we've deleted all platform specific call-tokenuri strategies in favor of using a generic strategy: https://github.com/neume-network/strategies/commit/08af9f4fecdedad92cf5e8b5c6558f0189b97c12
• we've made that change in /data too: https://github.com/neume-network/data/blob/bfce554e48aa22ab5d4ee2cb5a5306bc4234f355/steps/3.mjs
• but in the readme.md file here and in the exemplary crawl_path.mjs file, we haven't made that change yet
• As soon as we'll start to upgrade the /strategies submodule, beyond the above linked commit tip, we must update the crawl paths