This PR improvers the ponder dev progress bars and ETA calculations. Fixes #18. Also updates the Ponder class to use a type-safe event emitter and renames those events.

Changes:

• Create notSoFastQueue which doesn't hog the event loop quite as much as fastq, and use that for the log handler queue
• Refactors some of the database code to fix a bug where better-sqlite3 errors would cause the process to exit instead of getting caught by the user error handler
• Capture esbuild errors + stack traces and send to user error handler

When create-ponder runs ponder codegen, it should not print the configuration error (due to the RPC).

Couple ways to handle this:

• Silence logs while running ponder codegen
• Refactor the config error handling thing (i don't really like it...)
• Update ponder codegen so it somehow doesn't run the config validations

When an error occurs while running a user handler, the development server should catch the error, stop processing logs, and display the error in the terminal UI. Then, on the next hot reload, Ponder should clear the error and begin processing logs again.

I'm not exactly sure how to do this. Also, the stack trace almost always points to some weird yoga prebuilt thing instead of the actual line with the error. Some errors will originate downstream of user code, like in the entity store (SQLite/Postgres).

Currently the logWorker is silently crashing. This is happening because the stats.sourceStats[source.name] object is uninitialized. When trying to access a field of this undefined object the worker silently fails

The current UI for ponder dev displays the backfill progress, and the log handler progress. It's confusing and not intuitive. "Dev mode" only makes this worse. The handler progress bar should not say "up to date" unless the backfill is also complete.

Ideas

• Remove the backfill progress bar entirely
• Remove the log handler progress bar if its complete
• Change "up to date" to "up to {lastLogProcessedTimestamp}"
• Change 31/31 to 31/???

Changes:

• Create a test dir in packages/core and add a basic test
• Add two options to the ponder CLI
  • rootDir?: string -> path to the root directory of the Ponder project to be ran
  • configFile?: string -> path to the config file to be used
• Change how the Ponder class is configured (to make testing a bit more ergonomic)
• Move some shared async logic into the Ponder.setup() method (migrating the stores, loading handlers) and load the schema in the constructor

Minor changes / notes:

• Fix examples to use latest public APIs
• Replace commander dep with cac
• Fix backfill ETA bug
• Make some of the Ponder methods private

Within event handler functions, the context.contracts object contains an ethers.Contract for each source defined in ponder.config.js. These contracts can be used to make contract calls within event handler functions. These contract objects are connected to a special provider that caches contract calls to reduce the number of RPC requests that need to be made while syncing your project.

While running event handler functions, the ponder runtime injects an ethers.Contract object for each contract defined in ponder.config.js. Originally, I used typechain to generate TypeScript types for these contract objects. Typechain proved to be a bit slow and bulky. The abitype library seems to offer an alternative that avoids generating lots of files. We already have every contracts abi as a JSON file, so it should be easy to use abitype to type these contract objects.

the cache metadata gets updated when the historical sync is complete. however, when fetching "live" logs/blocks/txns, they get added to the cache but the metadata never gets updated.

so, when the app rebuilds, it fetches all logs/blocks/txns since the end of the last historical sync despite the fact that the cache store already has all of the data.

this might require a slight remodel for how the cache metadata table(s). worth doing some research on this.

it might make sense to move graphql and sqlite to separate packages. these packages could be imported and "plugged in" to the @ponder/core package that contains the core indexing code (and probably also the CLI and dev server?).

this would make it easier to write tests against the indexer.

also, separating out the graphql bits as a plugin would make it easier to allow users to customize the Express server that backs the GraphQL API (add authn, logging, etc).

likewise on the database side. i'm sure many users will prefer to use postgres for GP-like experience.

basically, if you have an proxy contract that upgrades its implementation, that basically means its ABI changes over time. a user ran into a bug where the ABI provided for the proxy included only the "latest" implementation contract ABI, and when ponder tried to decode events emitted by an earlier implementation contract, those events weren't found in the new ABI, so ponder threw an error.

it seems like the easiest workaround is to merge the ABIs of all implementation contracts that have ever existing for this proxy. another solution would be to add a source to ponder.config.js for each version of the proxy, and use startBlock and endBlock (NOTE: endBlock doesn't currently exist).

This issue has two related parts.

1. Ponder currently fetches all events emitted by contracts included in ponder.config.js, regardless of which events the user has provided handlers for. We could significantly reduce the backfill load for certain projects by lazily fetching event data, e.g. only fetch events for handlers the user has provided. Example: I want to index the ownership changes (OwnershipTransferred) for an ERC20 contract without indexing every Transfer event.

2. Ponder currently adds all events available for a contract to the indexing queue, regardless of which events the user has provided handlers for. In the indexing task function, if a handler is not found for the given event log, it just does nothing and returns early. Instead, when adding logs to the indexing queue, we could filter to only include logs for events that the user has provided handlers for.

Part 2 is much easier to implement than part 1. The parts can be implemented independently.

When using a cloud platform that uses git deployments (like Render or Railway), a new deployment will be created for each new commit. Then, once the new deployment is deemed healthy by the platform (by responding to the health check path with a 2XX), the platform will start serving traffic to the new deployment and then shut down the old deployment. Cloud platforms normally will wait between 5 and 15 minutes for a healthy response, after which they will consider the deployment failed and shut it down.

Ponder currently sends a healthy response as soon as the GraphQL server starts up, which happens immediately on ponder start. This is problematic because the backfill (and/or the handler processing) will not NOT be done yet, so the entity tables are not up-to-date but the service is saying its healthy.

Solution (for now):

• If the backfill + indexing will take >4 minutes (not sure what heuristic to use for this yet), enter a "backfilling" state, where the service responds as healthy, but requests to /graphql respond with an error message describing that the backfill is in progress.
• If the backfill + indexing will take <4 minutes, enter a "reloading" state, where the service does not respond as healthy until the backfill + indexing are complete. This should enable zero downtime deploys for deployments after the first one.

The reason to use ~4 minutes as the cutoff is to play nice with the common cloud platform behavior of giving new deployments ~5 minutes to become healthy.

Drawbacks with this approach:

• I'm not sure what heuristic to use to estimate the backfill + indexing time.
• If a user makes a change to an existing service that adds a large backfilling load (such as adding a new contract to ponder.config.js), the service will enter the "backfilling" state and the production server will stop responding to requests.

Pseudo example:

// schema
type Terraform @entity {
  mode: String!
}

// query
{
  terraforms (where: { mode_in: ["Daydream", "Origin Daydream"] }) {
    id
  }
}

Throws the error Handler error: SqliteError: no such column: Daydream when resolving the query.

improves experience of writing handler functions because it matches with types coming from ethers event log params. goal is to avoid needing to use BigNumber.from(...) and BigNumber.toString() everywhere