Closed
Conversation
kespinola
force-pushed
the
grpc-ingest
branch
3 times, most recently
from
b5d93b9 to
a31344f
Compare
kespinola
commented
Apr 15, 2024
| run: | | ||
| cargo tree | ||
| git checkout Cargo.lock | ||
| cargo tree --frozen |
Collaborator
Author
There was a problem hiding this comment.
yellowstone crates are referenced using git so require network fetch to retrieve which violates frozen flag.
Collaborator
Author
There was a problem hiding this comment.
https://lib.rs/crates/yellowstone-grpc-client
NVM crate is published
Collaborator
|
Went over it quickly and it looks good. One question though, how would this work if we want to backfill some accounts or run account flushes to reindex some accounts? |
* feat: Add NFT single backfill command (ops) to fetch mint,tokenAcc,metadata for a nft and backfill it to the db * chore: cleanup * chore: optimize
* feat: move backfill to bubblegum crate and verfiy to it * feat: grpc ingest monitor
…le each stream (#168) * feat : separate global stream to multiple streams depending on programs * WIP : add metrics for each program subscription * refactor: add structs with builder for SubscriptionTask and add better shutdown handling * refactor : separate subscription builder to work on each subscription * refactor: change pipeline to handle single stream instead of multiple streams * Update: readme for grpc-ingest, index-readme, ops (#172) * Update: readme for grpc-ingest, index-readme, ops * Update : README * fix: prometheus ports * chore: resolve comments * chore: add comments to config files in grpc-ingest (#173) * change tag name of subscriptions * Remove topograph from grpc2redis --------- Co-authored-by: Kyle Espinola <kyle.s.espinola@gmail.com>
* time the program transformer tasks * add consumer labels to metrics
Co-authored-by: Ahzam Akhtar <ahzamakhtar@Ahzams-MacBook-Pro.local>
Metadata download status metrics --------- Co-authored-by: Kyle Espinola <kyle.s.espinola@gmail.com>
* Update to rust 1.79.0 * feat : add metadata_json backfiller * cleanup * revert metaplex/plerkle-validator image to from ..79 to ..75 --------- Co-authored-by: Kyle Espinola <kyle.s.espinola@gmail.com>
* `wip` concurrent download metadata notifier with channels * `chore` Clean debugging changes * `chore` remove duplicated config files and debug trace * Refactor to download metadata publisher * Time metric for metadata josn publish --------- Co-authored-by: Kyle Espinola <kyle.s.espinola@gmail.com>
* add redis pipes for processing metadata download * resolve comments
Collaborator
Author
|
moved here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Reduce the operational cost and complexity of running DAS by switching to Dragonmouth grpc stream for account and transaction updates.
Upgrade the ingestion engine by refactoring configuration, workers, and metrics.
Approach
Create a new crate named das-grpc-ingest which has 3 primary subcommands.
stream - connects to Dragonmouth and pushes relevant events into a redis queue
ingest - subscribes to redis streams, spawns workers, and processes events using program_transformers crate.
download - fetches metadata json for asset_data replacement to bgtasks
Based on #144
closes #83
Next Steps
Diagram