Platform data retrieval strategies for music-os.

Overview

music-os-strategies

purpose

The goal of this repository and the overall idea of separating music-os-core's source from its strategies, is to allow any type of music platform to quickly define their on-chain schema - allowing them to re-use music-os-core to index and distribute their and other platform's music ecosystem.

work-in-progress

WARNING: This repository is under active development and APIs aren't stable.

implementing new strategies

To implement a strategy with maximum efficiency, we recommend doing all on-chain and off-chain requests using music-os-core's Extraction Worker API.

Comments
  • EPIC: Integrate the new sound protocol contracts

    EPIC: Integrate the new sound protocol contracts

    • Sound released a new protocol update: https://sound.mirror.xyz/t7GDH8RF6XbHLwbHb_R6QLnDTaln0KdGhExBgul6OXY
    • roundpotatocat reported that hence artists that upgraded don't yet show up in music os: https://github.com/neume-network/data/issues/55
    • e.g. here's a new song contract by latasha for "SHOWTIME" that isn't showing in the feed of music os for now:
      • https://etherscan.io/tx/0xa09cce4737eaab8b5724fb2c85439d4725cca51cdae994f675155a319889c4f4
      • https://www.sound.xyz/callmelatasha/joyride-ii-showtime
    • Here's what we should do
      • for all new soundxyz music NFTs, parse their contracts in a strategy
      • integrate them into music-os accumulator
      • in the process, make sure that there is no name collision between the "old" sound strategy naming and the new "sound protocol" naming
    opened by TimDaub 14
  • something continuously kills lifecycle of long-running strategy

    something continuously kills lifecycle of long-running strategy

    • I've tried multiple times to run https://github.com/neume-network/strategies/blob/main/src/strategies/call-block-logs/extractor.mjs for extended periods (> 2h) and eventually the strategy is just shut down for now apparent reason
    • no helpful related logs are emitted
    2022-08-07T04:35:12.748Z neume-network-extraction-worker:worker Queue stats: {"successRate":0.9997004955950248,"peak":625472,"average":3487264.5626161993,"total":176959}
    2022-08-07T04:35:12.749Z neume-network-extraction-worker:worker Queue stats: {"successRate":0.9997004972875226,"peak":625472,"average":3487291.886499774,"total":176960}
    
    • All events to that point end up being written smoothly but beyond that, it's like the program shut down
    • ps ax | grep node also shows no sign of a node process
    opened by TimDaub 13
  • add generic call-tokenuri strategy

    add generic call-tokenuri strategy

    This PR is part of the effort to improve strategy interface and increase ease of adding new contracts.

    I recommend reviewing individual commits. Combining comments from individual commits here:

    add an outputFilename option for strategies

    Strategies can now optionally define outputFilename in crawlPath. For the following crawl path.

    {
      name: "get-tokenuri",
      extractor: {
        args: [resolve(env.DATA_DIR, "soundxyz-call-tokenuri-transformation")],
        outputFilename: "soundxyz-get-tokenuri"
      },
      transformer: {},
    }
    

    The results for get-tokenuri will be written to soundxyz-get-tokenuri-extraction.

    add a generic call-tokenuri strategy

    Added a generic call-tokenuri strategy to remove the need for diffferent strategies such as soundxyz-call-tokenuri, zora-call-tokenuri, catalog-call-tokenuri etc.

    The benefits include:

    • Less number of files to mantain
    • Easier to refactor
    • Easier to integrate new indie contracts

    This new call-tokenuri strategy has been modified to not need logs-to-subgraph strategy. I have tested this strategy and the output files are the same as before.

    Future scope of work:

    • Remove no longer needed strategies such as *-call-tokenuri.
    • Create more such blueprint strategies. Maybe we can combine all filter-contracts strategies.
    • Remove call-tokenuri strategy factory.
    opened by il3ven 11
  • Quickstart Documentation + Refactor Sample Strategy

    Quickstart Documentation + Refactor Sample Strategy

    • Add quickstart guide, which walks through the steps of defining a web2 strategy and clarifies some of the concepts more thoroughly
    • Refactor the sample strategy get-xkcd in order to follow the documentation more tightly
    • Add schema validation to get-xkcd
    opened by sirnicolaz 10
  • Enumerate through mint songs v2

    Enumerate through mint songs v2

    An example of a new extractor strategy as shown in the neume open office call from july 13 2022

    • [ ] make sure that if source files aren't existent that the strategy fails gracefully and doesn't crash the entire crawl
    • [ ] filter match only NFT ID entries from logs-to-subgraph that correspond to the contract's address
    • [ ] write a metadata retrieval and transformer strategy to normalize the schema into neume-network/schema
    • [ ] potentially replace custom extractor code with strategy factory of call-tokenuri
    opened by TimDaub 10
  • Implement call-soundeditionv1-tokenuri extractor and transformer

    Implement call-soundeditionv1-tokenuri extractor and transformer

    • We need a call-soundeditionv1-tokenuri strategy to get the tokenUri. The result is something like ar://zUTuV0VL7w1z5cuGkBs2b-O3DRY2O2qA_ifSQqjLnzA/1.

    • We can get tokenUri from the sound edition contracts we found in #295. We should track Transfer events where tokenId is 1.

    • You can find detail information on #294

    enhancement good first issue 
    opened by Kunkka0822 9
  • Make tests easier to run

    Make tests easier to run

    I ran into a few hiccups while trying to run the tests in this repository and wanted to document them here with suggested improvements.

    • A section on tests should be added to the README with setup instructions and caveats.
    • Explain .env file setup. I used RPC_HHTP_HOST=https://rpc.ankr.com/eth and IPFS_HTTPS_GATEWAY=https://cloudflare-ipfs.com/ipfs/ (no api key needed).
    • Add empty data directory to repo and hide it's contents or ask user to create it. It wasn't clear until I looked at the .gitignore file that the data directory should be a temp directory in this repo. Also not sure if it needs to be cleaned up between test runs or if that is handled.
    • Tests seem a bit brittle wrt ethereum network state. One tests for soundxyz is failing for me due soundxyz metadata changes.
    opened by neatonk 9
  • upgrade lifecycle handler

    upgrade lifecycle handler

    This is a proposal for an upgrade in lifecycle handler. It solves two main problems.

    • We now know when an extractor has completed and we can then call it's transformer. This should solve problems like #154. I tested it and it does solve. The number of lines in soundxyz-metadata-transformation is the same as soundxyz-metadata-extraction.
    • We now also know when a strategy has ended. This should solve problems like #146.

    Additional benefit:

    • We can now define a path for the crawler to take in advance.

    @TimDaub Have a look at the implementation. I tested it with minimal data and it works. I couldn't test it completely because of no IPFS gateway access right now. I'll try to test more and iron out the bugs.

    opened by il3ven 8
  • Implement sound-protocol-get-tokenuri

    Implement sound-protocol-get-tokenuri

    @TimDaub I have implemented basic feature of get-tokenuri to fetch JSON objects from tokenURI.

    But I didn't filter unique songs yet.

    Still thinking where could be best to do this feature.

    @TimDaub @il3ven If you have any ideas, please let me know

    opened by Kunkka0822 7
  • add try-catch block in `extract` function of lifecycle.mjs

    add try-catch block in `extract` function of lifecycle.mjs

    There can be unhandled errors in strategies. These errors can bubble up and cause premature exit and UnhandledPromiseRejection. To prevent this error and properly reject promise returned by the extract function this commit adds a try-catch block. Also, adds two new test cases for the above described scenario.

    The UnhandledPromiseRejection can be seen in neume-network/data#42 where an error is generated inside the update function of call-block-logs-extractor. https://github.com/neume-network/data/runs/8234616425?check_suite_focus=true#step:10:37

    For this commit to work properly we also need neume-network/core#85. Without it the extract function will reject and it's error will bubble up to neume.mjs but the application still won't close because the worker is still running.

    opened by il3ven 7
  • some call-block-logs-extraction calls don't resolve in downloaded logs

    some call-block-logs-extraction calls don't resolve in downloaded logs

    {
      type: 'json-rpc',
      commissioner: 'call-block-logs',
      method: 'eth_getLogs',
      params: [ { fromBlock: '0xe5a709', toBlock: '0xe5a709' } ],
      version: '0.0.1',
      options: { url: 'http://127.0.0.1:8545', signal: {} },
      results: [],
      error: null
    }
    
    • see this response from our erigon node.
    • https://etherscan.io/block/15403917, but this block contains transactions that emit logs, e.g. this one: https://etherscan.io/tx/0x2e7ade86e34bb40d6c85c8c26c67525c7a2515dddfa7b4647fd384bd44aec966
    • So why was it returned empty from extraction worker?
    bug 
    opened by TimDaub 7
  • Similar as with call-tokenuri, we should make getting an IPFS link a generic extractor strategy too

    Similar as with call-tokenuri, we should make getting an IPFS link a generic extractor strategy too

    • see the https://github.com/neume-network/strategies/tree/main/src/strategies/call-tokenuri strategy and how we're using it in https://github.com/neume-network/data/blob/main/steps/2.mjs
    • But we don't need to have a strategy that implements both extractor and transformer
    • We could also have purely an IPFS extractor strategy instead of relying on the factory pattern
    • take this https://github.com/neume-network/strategies/tree/main/src/strategy-factories/get-ipfs-tokenuri and make it into a configurable strategy that we define in e.g. data/steps/2.mjs

    //cc @il3ven because you've been leading this so far

    opened by TimDaub 0
  • When we misnamed the only or last strategy in a crawl path and there's nothing to execute then the program should crash

    When we misnamed the only or last strategy in a crawl path and there's nothing to execute then the program should crash

    • the strategy is actually called "call-transaction-receipts" (plural) and so it could not been found
    • in that case the program should just crash/end with a warning (because it was the only strategy invoked in the crawl path)
    2022-10-25T14:03:36.670Z neume-network-strategies:lifecycle Starting to execute strategies with the following crawlPath [ [ { name: 'call-transaction-receipt', extractor: { args: [ '/root/data/results/call-block-logs-transformation' ] } } ] ]
    NotFoundError: Failed to find matching strategy for name: "call-transaction-receipt" and type "extraction"
    2022-10-25T14:03:36.767Z neume-network-extraction-worker:worker Starting as worker thread with queue options: "{"queue":{"options":{"concurrent":50}},"endpoints":{"https://metadata.sound.xyz":{"timeout":4000,"requestsPerUnit":6,"unit":"second"},"https://ipfs.io":{"timeout":6000,"requestsPerUnit":50,"unit":"second"}}}"
    
    opened by TimDaub 0
  • We're currently emitting a log ever time a strategies either extractor or transformer isn't found and to a new user that could be rather intimidating or signaling errors

    We're currently emitting a log ever time a strategies either extractor or transformer isn't found and to a new user that could be rather intimidating or signaling errors

    • these logs are completely fine to have, they just state that either an extractor or transformer is missing
    • but that's fine. Maybe we should label them explicitly as warnings
    2022-10-25T14:03:36.661Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/get-minter/extractor.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    2022-10-25T14:03:36.662Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/sound-protocol-filter-contracts/extractor.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    2022-10-25T14:03:36.662Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/soundxyz-filter-contracts/extractor.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    2022-10-25T14:03:36.662Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/zora-drops-filter-contracts/extractor.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    2022-10-25T14:03:36.668Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/call-transaction-receipts/transformer.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    2022-10-25T14:03:36.668Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/get-xkcd/transformer.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    2022-10-25T14:03:36.668Z neume-network-strategies:disc Rejected loading strategy with reason: "Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/root/core/src/strategies/src/strategies/music-os-accumulator/transformer.mjs' imported from /root/core/src/strategies/src/disc.mjs"
    
    opened by TimDaub 0
  • Generate token ownership history from EIP721 event Transfer(from, to, tokenId)

    Generate token ownership history from EIP721 event Transfer(from, to, tokenId)

    • in https://github.com/neume-network/strategies/issues/321 @djfnd and @vaughnmck outline the necessity to understand a token's ownership history
    • with call-block-log-extraction we already download the entire ownership history of music NFTs and then we filter it down with call-block-logs-transformation into mint events
    • we do this by identifying events where from=address(0): https://github.com/neume-network/strategies/blob/54631d99cbb1861c9c1381a7b0d5209da7465518/src/strategies/call-block-logs/transformer.mjs#L46
    • however, did we have a guarantee that call-block-logs-transformation is sorted (which for now btw we don't have) then we could construct an ownership history given the parameters of event Transfer(from, to, tokenId) by tracking the tokenId and from to to a token was transferred over time
    opened by TimDaub 2
  • call-tokenuri should error if no explicit outputFilename is defined

    call-tokenuri should error if no explicit outputFilename is defined

    • It's very unlikely that we'll ever use call-tokenuri without a specific outputFilename
    • hence it should error when outputFilename isn't defined in a crawlpath
    • now we're currently not aware of outputFilename within a strategy and that's part of the problem to solve in this issue
    opened by TimDaub 0
Owner
musicOS
musicOS
I’m a very useful music bot that can make you play a music with a simple command! I have a lot of good commands that you can have a better experience playing your favorites songs!

I’m a very useful music bot that can make you play a music with a simple command! I have a lot of good commands that you can have a better experience playing your favorites songs!

Hugo Kishi 2 Aug 16, 2022
Music-Player - Music player application built with HTML, CSS and vanilla JavaScript

Music-Player Music player application built with HTML, CSS and vanilla JavaScrip

Karthik Umashankar 1 Feb 10, 2022
A mobile app for playing music on Resonate, an open source music streaming co-op.

stream2own Play fair. Website | Twitter | Contributing | Developer Guide ?? Resonate Stream App A mobile app for playing music on Resonate, an open so

Peter Klingelhofer 12 Dec 28, 2022
A decentralized Music Ownership System developed on Truffle Suite and React Bootstrap to allow Music artists to upload their content to the blockchain, that can be streamed by users

A decentralized Music Ownership System developed on Truffle Suite and React Bootstrap to allow Music artists to upload their content to the blockchain, that can be streamed by users.

M. Adil Fayyaz 10 Sep 5, 2022
The #1 cross-platform open source music player

Rhyme The #1 Open-Source Music Player Discord: Matrix: A beautiful looking music player which supports Local music files Create custom playlists view

Rhyme Music Player 51 Apr 13, 2022
JavaScript plugin for playing sounds and music in browsers

JavaScript plugin for playing sounds on user actions and page events. Version: 3.0.7 Project page and demos Download ZIP Support the plugin on GitHub

Denis Ineshin 704 Sep 24, 2022
VexFlow 3 - A JavaScript library for rendering music notation and guitar tablature.

VexFlow 3 - A JavaScript library for rendering music notation and guitar tablature.

Mohit Cheppudira 3.5k Jan 6, 2023
🎼 A sheet music to polylines renderer

?? A sheet music to polylines renderer

Lingdong Huang 218 Jan 1, 2023
A simple package to download music tracks from spotify 🎵

A simple package to download music tracks from spotify ??

Alen Yohannan 26 Dec 25, 2022
⚡ Discord bot with economy, gambling, music, fun, moderation features based on discord.js v12

Crucian Crucian is my discord bot with simple structure based on discord.js Click Here to invite Crucian to your server Author Crucian © Apoo Authored

apoow3b 3 Jul 26, 2021
An adorable lavalink discord music bot that has a lot of features inside it.

Laffey An adorable lavalink discord music bot that has a lot of features inside it. First Project of Weeb-Devs Features: ✓ High quality ✓ Support filt

null 81 Dec 31, 2022
Music bot that uses Lavalink for its audio player

Msv Music ?? A Advanced Discord Music Bot Made With Lavalink And Have Many Premium Features with filters and more willl be add soon About Msv Music ??

M s v 36 Jan 4, 2022
A multi-purpose discord bot, that has 100+ commands. Includes 🎶 Music, 📷 Image Generation, 📊 Leaderboard, and more!

Cleckzie An open-source, multi-purpose discord bot, made with JavaScript. Has useful categories like: ?? Music - filter, seek, queue, volume and more.

null 42 Jul 2, 2022
Discord.js Music Bot with buttons and slash support!

Music Bot Vote for us! (Please) How to use? Download the repo, create a .env file, and add your token into it. You need to change guildOnly to your gu

Garlic Team 59 Nov 24, 2022
A self-hosted, completely private and free music streaming server compatible with Synology Audio Station's web browser interface and smartphone apps.

Open Audio Server Open Audio Server is a music streaming server compatible with Audio Station by Synology. Audio Station creates your own private serv

null 91 Dec 11, 2022
ALi a powerful Discord bot that includes Utility, Fun, Music, Moderation, and much more! (Beta)

ALi ALi a powerful Discord Bot Invite ALi · Report Bug · Request Feature ?? Futures Music Moderation Fun Economy Utility More ?? Installation You can

S Dip 97 Oct 17, 2022
A Discord.js Music Bot, which connects via Lavalink and erela.js

Discord.js Lavalink Music Bot using erela.js A Advance Discord Music Bot Like Hydra Erela.js docs Note this Discord Bot Repository is the Same as for

ᴋᴀʙɪʀ々ꜱɪɴɢʜ 47 Dec 9, 2022
Discord Music Made Simple.

boat-discord Inspiration This project was made because of the shutdown of Groovy and Rhythm. This project was inspired by the connecting two APIs work

Krish Gupta 5 Nov 9, 2022
Web Application that represents a music player using the spotify API, React, JS, CSS, HTML, nodeJS, Firebase, material-ui, JSON and other technologies. Made by Yohan Hmaiti

Web Application that represents a music player using the spotify API, React, JS, CSS, HTML, nodeJS, Firebase, material-ui, JSON and other technologies. Made by Yohan Hmaiti

Yohan Hmaiti 2 Jan 8, 2022