TypeScript framework for deploying distributed indexers on Aleph VMs for Solana.

Overview

Aleph Indexer Framework v0.1

The Aleph Indexer Framework is a high-level abstraction for building multithreaded indexers on Aleph. It is designed to be used by developers who want to build an indexer for a specific Solana program, but don't want to worry about:

  • Fetching blocks from the Solana cluster
  • Parsing transactions
  • Managing the state of the indexer
  • Scaling the indexer to multiple threads
  • Writing the indexer state to disk
  • Reading the indexer state from disk
  • Exposing the indexer state over a GraphQL API

Creating an Indexer

In order to run your own indexer, you need to start by creating a new project:

Running the indexer locally

First, you need to make sure that the framework is running:

npm run start framework

When the services of the framework are ready, you need to run your custom indexer implementation with

npm run start your-indexer-name

Running an indexer on Aleph

We have some examples in this repository with the concrete steps to deploy an Indexer inside Aleph.im network.

Architecture

The framework provides three services:

Fetcher

Each instance tracks accounts of your program and its raw transactions, these accounts are allocated among instances in a balanced manner. During the execution of the service, the whole process (signatures, pending transactions, raw transactions and accounts information) is stored to create a backup, so that in case it crashes during the process, it can start where it ended before. When the signature history of the accounts has been fetched, the service will accept new transaction requests. ie: First fetches pending transactions of the account signatures, then loads new existing requests to the service. Once a transaction is fetched, it is sent directly to the parser through the communication established between services to perform its function.

Parser

It receives the raw transactions directly from the fetcher. It is in charge of processing the transactions, splitting them into the instructions that are part of them. Thus, it obtains the raw instructions and the accounts that participate in each of the instructions.

Indexer

Each instance is an intermediary storage and logging, for backuping the fetcher process, so in case it crashes while processing transactions, it can start where it finished before. Each thread of each indexer instance have a communication channel with the fetchers and parsers instances. All communication between services is done through an event queue for synchronisation between them. In addition, it knows the fetching state of the transaction history of each account, allowing to know the progress of the process of each one of them.

In order to use these services you need to code some custom implementations: TODO

Environment Variables

Here are some of the more important environment variables that you can set to configure the framework:

# For specifying a custom solana RPC node
SOLANA_RPC=https://api.mainnet-beta.solana.com
# How much memory to allocate for the indexer
ALEPH_DEFAULT_VM_MEMORY=512 

The full list of environment variables can be found in the .env.defaults.

Linting

This project uses eslint + prettier for formatting and linting the code with the default recommended rules. Additionally, we use husky and lint-staged for autoconfiguring a pre-commit git hook that will run the linter for the staged files before making the commit. Everything is automatically setted up after running npm install If you are using vscode, install the eslint extension and modify this values in your IDE config for this project to make it work properly:

{
  "editor.formatOnSave": true,
  "[typescript]": {
    "editor.formatOnSave": false
  },
  "[typescriptreact]": {
    "editor.formatOnSave": false
  },
  "[javascript]": {
    "editor.formatOnSave": false
  },
  "[javascriptreact]": {
    "editor.formatOnSave": false
  },
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  }
}
Comments
  • Bump node-fetch from 2.6.1 to 2.6.7

    Bump node-fetch from 2.6.1 to 2.6.7

    Bumps node-fetch from 2.6.1 to 2.6.7.

    Release notes

    Sourced from node-fetch's releases.

    v2.6.7

    Security patch release

    Recommended to upgrade, to not leak sensitive cookie and authentication header information to 3th party host while a redirect occurred

    What's Changed

    Full Changelog: https://github.com/node-fetch/node-fetch/compare/v2.6.6...v2.6.7

    v2.6.6

    What's Changed

    Full Changelog: https://github.com/node-fetch/node-fetch/compare/v2.6.5...v2.6.6

    v2.6.2

    fixed main path in package.json

    Commits
    • 1ef4b56 backport of #1449 (#1453)
    • 8fe5c4e 2.x: Specify encoding as an optional peer dependency in package.json (#1310)
    • f56b0c6 fix(URL): prefer built in URL version when available and fallback to whatwg (...
    • b5417ae fix: import whatwg-url in a way compatible with ESM Node (#1303)
    • 18193c5 fix v2.6.3 that did not sending query params (#1301)
    • ace7536 fix: properly encode url with unicode characters (#1291)
    • 152214c Fix(package.json): Corrected main file path in package.json (#1274)
    • See full diff in compare view
    Maintainer changes

    This version was pushed to npm by endless, a new releaser for node-fetch since your current version.


    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 2
  • Suite of fixes & improvements in stats calculation and Anchor support

    Suite of fixes & improvements in stats calculation and Anchor support

    Problem: Missing type safety made debugging difficult Solution: Added generic typification to AccountTimeSeriesStatsManager

    Problem: Time frame stats would miss entries or break down on singular events, resulting in wrong time series calculation Solution: Fix issues for edge cases in splitDurationIntoIntervals()

    Problem: Anchor-generated indexers would not correctly aggregate accesses to programs Solution: Add "signer" field to all events, denoting the pubkey of the first signer, initiating a transaction

    Problem: Anchor-generated indexers do not work well with current InstructionParser Solution: Created two distinctive InstructionParsers (Anchor- and Spl-) and make a split decision between which to use in the InstructionParserLibrary

    Problem: Certain RPC providers do not accept batch sizes of >100 Solution: Halve batch sizes across the board

    opened by MHHukiewitz 1
  • Bump axios and nx

    Bump axios and nx

    Bumps axios and nx. These dependencies needed to be updated together. Updates axios from 0.21.1 to 1.1.3

    Release notes

    Sourced from axios's releases.

    v1.1.3

    Added

    Added custom params serializer support #5113

    Fixed

    Fixed top-level export to keep them in-line with static properties #5109 Stopped including null values to query string. #5108 Restored proxy config backwards compatibility with 0.x #5097 Added back AxiosHeaders in AxiosHeaderValue #5103 Pin CDN install instructions to a specific version #5060 Handling of array values fixed for AxiosHeaders #5085

    Chores

    docs: match badge style, add link to them #5046 chore: fixing comments typo #5054 chore: update issue template #5061 chore: added progress capturing section to the docs; #5084

    Contributors to this release

    v1.1.2

    Fixed

    • Fixed broken exports for UMD builds.

    Contributors to this release

    v1.1.1

    Fixed

    • Fixed broken exports for common js. This fix breaks a prior fix, I will fix both issues ASAP but the commonJS use is more impactful.

    Contributors to this release

    ... (truncated)

    Changelog

    Sourced from axios's changelog.

    [1.1.3] - 2022-10-15

    Added

    Added custom params serializer support #5113

    Fixed

    Fixed top-level export to keep them in-line with static properties #5109 Stopped including null values to query string. #5108 Restored proxy config backwards compatibility with 0.x #5097 Added back AxiosHeaders in AxiosHeaderValue #5103 Pin CDN install instructions to a specific version #5060 Handling of array values fixed for AxiosHeaders #5085

    Chores

    docs: match badge style, add link to them #5046 chore: fixing comments typo #5054 chore: update issue template #5061 chore: added progress capturing section to the docs; #5084

    Contributors to this release

    Commits
    Maintainer changes

    This version was pushed to npm by jasonsaayman, a new releaser for axios since your current version.


    Updates nx from 14.8.3 to 15.0.0

    Release notes

    Sourced from nx's releases.

    15.0.0

    15.0.0 (2022-10-14)

    Bug Fixes

    • angular: broken test (#12286) (6125dc0)
    • angular: do not use ngcc for new workspaces and projects #12386 (#12504) (80a623e)
    • angular: export generators correctly #12434 (#12460) (32189d5)
    • angular: fix import breaking jasmine-marbles migration (#12415) (e13ad11)
    • angular: ng-add migration should set default project (#12513) (66a0864)
    • angular: switch to using jasmine-marbles for certain symbols (#11896) (3dc72f0)
    • bundling: add missing assets ignore option to @​nrwl/esbuild:esbuild (#12288) (7f76ac5)
    • bundling: correct main field in package.json when using esbuild (#12328) (a14646a)
    • bundling: do not add types field in package.json for esbuild (#12424) (d904160)
    • bundling: fix esbuild build watch (#12448) (e0f7911)
    • bundling: set NODE_ENV correctly to avoid warnings (#12331) (c3e0602)
    • bundling: update error message for esbuild-project generator so it shows correct project name and information (#12323) (20c3951)
    • core: add missing core plugins to list command (#12305) (776b5a0)
    • core: add missing packages (esbuild, webpack, rollup) to nx-migration field (#12285) (37bedce)
    • core: add missing packages to report command (#12306) (6a7a7c1)
    • core: custom generators help should print relevant information (#12258) (f9e57e2)
    • core: daemon should not shut down due to inactivity when there are open connections (759f7ce)
    • core: daemon termination should be ok when client does not need anything (#12519) (8c393b5)
    • core: disable broken git hasher recourse submodules (#12337) (2f0d61f)
    • core: do not fallback to angular cli for ng deploy (#12401) (74e898d)
    • core: do not run postinstall unless it is the main nx package (#12591) (0961cb3)
    • core: drain stdout before exit in print-affected (#12559) (0ff93dd)
    • core: ensure lock file is read from the correct folder (#12294) (a7631e1)
    • core: fix slashes in inputs migrations (#12444) (85a5c89)
    • core: hide leaky error message from git show command in affected (#12521) (f37c334)
    • core: js plugin files should work (#12406) (7ac0dab)
    • core: readRawWorkspaceJson should keep workspace cache up to date (#12528) (ad79453)
    • core: remove trailing slash in outputs that prevents correct caching (#10708) (2889372), closes nrwl/nx#10549
    • core: remove warning for invalid outputs (#12589) (773c0cb)
    • core: set workspaces property when using NPM preset (a947e25)
    • core: swap default method for output capture (#12535) (8ab7f85)
    • core: use node based hasher in git repos with submodules (678f786)
    • devkit: ensure that getProjects works properly without a nx.json (#12295) (40c8859)
    • graph: close tooltips when panning graph (#12468) (305fb2e)
    • js: add outfile to swc compiler in non-watch mode. fixes #12344 (#12345) (469a6d1)
    • js: check if asset create/update event contains a file before copying (#12551) (136e1a8)
    • js: close typescript watch program on SIGINT/SIGTERM (#11722) (f1a24d7)
    • js: inlined buildable libraries original output should not be removed (#12483) (b17893c)
    • linter: check for all .eslintrc.* files when generating a package (#12515) (d4fd7c0)
    • linter: remove leftovers after tslint support removal (#12313) (d91c7ca)
    • misc: buildable libs should not throw when some npm packages are not defined (6775f5d)
    • misc: remove remaining usages of @​nrwl/workspace:run-commands executor (#12299) (9d06b3c)
    • misc: update axios to 1.0.0 (#12437) (c3abe07)
    • nextjs: return correct webpack config for next.js storybook app (#12371) (5ae5f88)

    ... (truncated)

    Commits
    • 840f462 chore(core): update nx.dev/cli/* to nx.dev/nx/* links (#12607)
    • b7d5f1c chore(core): update url for updating Nx (#12606)
    • 26e256e feat(core): add ability not to load dot env files
    • 0961cb3 fix(core): do not run postinstall unless it is the main nx package (#12591)
    • 773c0cb fix(core): remove warning for invalid outputs (#12589)
    • 23de788 chore(repo): update to 15.0.0-rc1 (#12586)
    • 0ff93dd fix(core): drain stdout before exit in print-affected (#12559)
    • 8ab7f85 fix(core): swap default method for output capture (#12535)
    • f9e57e2 fix(core): custom generators help should print relevant information (#12258)
    • 0907ba2 feat(core): add support for npm v1 lock file parsing (#12534)
    • Additional commits viewable in compare view

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Hotfix: Request legacy txn format

    Hotfix: Request legacy txn format

    Problem: The new version of Solana RPC nodes returns the new transaction v0 format by default, which we do not yet support.

    Solution: A workaround explicitly requesting transactions in the 'legacy' format.

    Note to Aleph team: Will not immediately fix requests through the HA proxy.

    EDIT: v0 transactions should be no issue for current indexer framework

    opened by MHHukiewitz 1
  • Fix: Removed existing requests feature and fix account work error

    Fix: Removed existing requests feature and fix account work error

    After the last changes to add an account pending work queue, this feature is not needed and is creating some errors. Also fixed the error on the index account queue to run every minute processing the accounts 100 by 100, instead run only the first time.

    opened by nesitor 0
  • Feat: RPCresponse validation flag & tests, also included some in-code documentation

    Feat: RPCresponse validation flag & tests, also included some in-code documentation

    Used the same library as solana web3 library (superstruct), but changed the method they use create (for example: https://github.com/solana-labs/solana-web3.js/blob/master/src/connection.ts#L4094) for assert, because they apart of validate the response, they process it, to change some strings to publickey directly during the validation (which we dont want to).

    opened by ricardocr987 0
  • Add CoinGecko Integration API Standards into middleware package

    Add CoinGecko Integration API Standards into middleware package

    The CoinGecko Integration API Standards is a description of an API every token or exchange should provide, if they wish to have up-to-date data presented on CoinGecko to its users.

    For exchanges, this is a must, so in case anybody wants to write their own exchange indexer on our framework, it would be beneficial to them if they have already this API specification at hand and only need to link up their indexing with the given API.

    enhancement 
    opened by MHHukiewitz 0
  • Fix accuracy of aggregated time frames

    Fix accuracy of aggregated time frames

    Problem: Sometimes events would be included in the calculation of time frames, they didn't belong to.

    Solution: Use true dividing time frames when aggregating larger time frames, such that their start and end dates line up perfectly.

    opened by MHHukiewitz 1
  • Enforce type safety between interfaces [Debug Flag]

    Enforce type safety between interfaces [Debug Flag]

    Problem: Debugging the indexer framework is unexpectedly hard due to the multi-threaded microservices architecture. Often, the data being sent around may not conform to the types given in the framework's interface definitions. This comes with two issues:

    1. The transaction data being received by the fetcher does not always conform to the types as expected in the framework. Changes in transaction structure can result in faulty data entering the framework and oftentimes leading to cryptic errors deep inside the framework. See #6.
    2. The same is true for data being passed around the microservices. The problem was first encountered when defining JSON objects' fields such as luxon.DateTime in Pull Request #1.

    To improve the debugging process for the indexer framework, I propose the following:

    • adding a flag to enable schema checks on every interface where data is being parsed from JSON.
    • throwing errors every time the data does not conform to the given schema.

    I believe this will help debug current issues on the indexer generator and the above PR, as well as future issues.

    Tasks:

    • [x] Added flag for strict parsing of incoming txn: https://github.com/aleph-im/solana-indexer-framework/pull/9
    • [ ] Enforcing type safety between microservices
    opened by MHHukiewitz 0
  • [Draft] Refactor Time Series Stats: Use Luxon, Generators and allow any kind of Time Frame sizing

    [Draft] Refactor Time Series Stats: Use Luxon, Generators and allow any kind of Time Frame sizing

    Problem: Difficult to read and inefficient custom implementation for time series processing

    Solution:

    • Use entirely luxon-based classes and types (Interval, Duration, DateTime)
    • Removed enum TimeFrame in favor of Duration in API and backend
    • All interval-related functionality in utils/time.ts is now based on generators and executes lazily, where possible
    • The process() function aggregating all stats will now use on average ~10x fewer operations to aggregate all time frame stats
    • Time Frames (or "candles"/"bars") can now be configured to be generated in arbitrary sizes, starting from 1-second, over 5-minute, or even 1,5-days candles.#

    BREAKING CHANGE: timeFrame is now saved to LevelDB as milliseconds of the time frame duration instead of the old enum TimeFrame.

    opened by MHHukiewitz 1
Owner
Aleph.im
Aleph.im
solana-base-app is a base level, including most of the common features and wallet connectivity, try using `npx solana-base-app react my-app`

solana-base-app solana-base-app is for Solana beginners to get them up and running fast. To start run : run npx solana-base-app react my-app change th

UjjwalGupta49 33 Dec 27, 2022
Grupprojekt för kurserna 'Javascript med Ramverk' och 'Agil Utveckling'

JavaScript-med-Ramverk-Laboration-3 Grupprojektet för kurserna Javascript med Ramverk och Agil Utveckling. Utvecklingsguide För information om hur utv

Svante Jonsson IT-Högskolan 3 May 18, 2022
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
Remix Stack for deploying to Vercel with remix-auth, Planetscale, Radix UI, TailwindCSS, formatting, linting etc. Written in Typescript.

Remix Synthwave Stack Learn more about Remix Stacks. npx create-remix --template ilangorajagopal/synthwave-stack What's in the stack Vercel deploymen

Ilango 56 Dec 25, 2022
Deploying Fake Back-End Server & DataBase Using JSON-SERVER, GitHub, and Heroku

Deploying Fake Back-End Server & DataBase Using JSON-SERVER, GitHub, and Heroku. In this article, we will create and host a fake server that we can de

Israel David 0 Sep 5, 2022
The Remix Stack for deploying to AWS with DynamoDB, authentication, testing, linting, formatting, etc.

The Remix Stack for deploying to AWS with DynamoDB, authentication, testing, linting, formatting, etc.

Remix 311 Jan 1, 2023
The Remix Stack for deploying to Fly with Supabase, authentication, testing, linting, formatting, etc.

Remix Supa Fly Stack Learn more about Remix Stacks. npx create-remix --template rphlmr/supa-fly-stack What's in the stack Fly app deployment with Doc

Raphaël Moreau 157 Jan 7, 2023
The Remix Blog Stack for deploying to Fly with MDX, SQLite, testing, linting, formatting, etc.

Remix Speed Metal Stack Learn more about Remix Stacks. npx create-remix --template Girish21/speed-metal-stack Remix Blog ?? This blog starter template

Girish 141 Jan 2, 2023
The Remix Stack for deploying to Fly with SQLite, authentication, testing, linting, formatting, etc.

Remix Indie Stack Learn more about Remix Stacks. npx create-remix --template remix-run/indie-stack What's in the stack Fly app deployment with Docker

Remix 688 Dec 30, 2022
The Remix Stack for deploying to Fly with PostgreSQL, authentication, testing, linting, formatting, etc.

The Remix Stack for deploying to Fly with PostgreSQL, authentication, testing, linting, formatting, etc.

Remix 677 Jan 2, 2023
Yet another library for generating NFT artwork, uploading NFT assets and metadata to IPFS, deploying NFT smart contracts, and minting NFT collections

eznft Yet another library for generating NFT artwork, uploading NFT assets and metadata to IPFS, deploying NFT smart contracts, and minting NFT collec

null 3 Sep 21, 2022
a quick start boilerplate for developing web3 apps and deploying smart contracts.

create-web3 A boilerplate for starting a web3 project. This boilerplate quickly creates a mono repo with 2 environments, a Next JS environment for fro

Eric Roupe 83 Dec 16, 2022
The Remix Stack for deploying to Vercel with testing, linting, formatting, structure and mock for 3rd party API integration.

Remix DnB Stack See it live: https://dnb-stack.vercel.app/ Learn more about Remix Stacks. npx create-remix --template robipop22/dnb-stack What's in th

Robert Pop 61 Dec 13, 2022
Starter repo for quickly deploying a Markdoc app with Next.js

Full Next.js example This is a full-featured boilerplate for a creating a documentation website using Markdoc and Next.js. Setup First, clone this rep

Markdoc 46 Dec 16, 2022
Starter repo for quickly deploying a Markdoc app with Next.js

Full Next.js example This is a full-featured boilerplate for a creating a documentation website using Markdoc and Next.js. Setup First, clone this rep

Markdoc 9 May 29, 2022
Open-source Solana Pay framework

Bedrock Bedrock is a framework for Solana Pay that: Standardizes the methodology used to create transaction requests Provides implementations for many

Bedrock Foundation 11 Nov 26, 2022
Complete Open Source Front End Candy Machine V2 Minter dAPP Built For The Frog Nation NFT Solana Project. Built With React, Candy Machine V2, Typescript

Complete Open Source Front End Candy Machine V2 Minter dAPP Built For The Frog Nation NFT Solana Project. Built With React, Candy Machine V2, Typescript

null 17 Sep 24, 2022
A full-featured Solana Metaplex Candymachine client-side library in Typescript

Candymachine Client SDK A full-featured Solana Metaplex Candymachine client-side library in Typescript Disclaimer: The SDK is currently a pre-alpha ve

Boxfish Studio 36 Nov 10, 2022