-
Notifications
You must be signed in to change notification settings - Fork 48
Add das-grpc-ingest crate #240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Nagaprasadvr
wants to merge
83
commits into
metaplex-foundation:main
Choose a base branch
from
rpcpool:grpc-cleanup
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
83 commits
Select commit
Hold shift + click to select a range
ea5379e
fmt
fanatid 8b12316
clippy
fanatid 974c90f
ci: add lock, fmt, clippy checks
fanatid 788973e
include migration to workspace
fanatid 7a870c2
include das_api to workspace
fanatid 7e3f9ef
add global clippy lints
fanatid aa916a4
use workspace
fanatid 033873d
add crate program_transformers
fanatid 1008f86
nft_ingester: use program_transformers crate
fanatid 83b171e
remove not used deps
fanatid 9d8031e
remove AccountInfo
fanatid 096616d
remove plerkle from program_transformers
fanatid ef0b0e9
nft_ingester2: grpc2redis
fanatid 076bffa
add redis streaming for ingester
fanatid d8673cf
create pg pool
fanatid c93adff
parse incoming message from redis
fanatid 53fa481
add force shutdown with signals
fanatid 11b4151
insert download metadata tasks
fanatid 8fca2c2
download-metadata subtask
fanatid 3d5eb18
refactor: rename nft_ingest2
kespinola e50e9ba
fix: applying account and transction filters to grpc subscription req…
kespinola 973a599
refactor: take out multiple connections to dragonmouth and then have …
kespinola dd564e5
refactor: switch to accepting multiple dragonmouth endoints to mimic …
kespinola 5a39d3f
fix: clippy errors
kespinola 139ed01
chor: refactor grpc command to use topograph to clean up the control …
kespinola 0883efc
chore: switch to run_v2 for ingest which handles ingesting accounts a…
kespinola 07b0936
chore: handle metadata json within the ingest command
kespinola dff868b
test: metadata json stream processing
kespinola 8f28a69
feat: set num threads for topograph. report on postgres and redis in …
kespinola 2cf9e57
chore: switch to topograph for thread management. create separate ing…
kespinola 454aa8e
fix: config and ack
kespinola b8249ef
hack: disable pending processing while see why messages arent being d…
kespinola 4211a54
fix: exit early if slot matches metadata json download and reindex is…
kespinola a574695
fix: picking dangling pending messages before reading new
kespinola 078b0f9
fix: ingest stream shutdown and requesting snapshots
kespinola a50ff86
fix: processing pending. add metrics for total workers.
kespinola e6bf732
fix: bubble backfill transform. no op out of download metadata info.
kespinola 103db2f
Support Account Snapshot Flushes (#154)
kespinola 0db4391
fix: keep grpc alive by sending using ping-pong (#160)
Nagaprasadvr 6efad62
Only Time Based Flushes (#161)
kespinola db7ee2d
delete and recreate the consumer (#162)
kespinola b5e9d99
Type out redis message handlers (#163)
kespinola 19b61b8
skip reading new messages if the ack buffer is full. track running ta…
kespinola 2d4e4b9
only push handler job for ingest if capacity (#165)
kespinola 1a75b62
Only throttle based on the task semaphore (#166)
kespinola aea798a
Remove topograph from ingest command (#167)
kespinola fd58811
Configure database connection idle timeout and max lifetime (#169)
kespinola 1bc38e5
Bubblegum verify (#150)
kespinola 794fc32
Separate global stream to multiple Program separated streams and hand…
Nagaprasadvr 3b9e68a
Report proofs as they come (#178)
kespinola 18a6913
Buffer redis messages (#179)
kespinola e53f7eb
Fix tracking ack tasks
kespinola 39fb1b7
fix: add back ack tasks total
kespinola ac053f4
recreate consumer but not groups (#181)
kespinola 51c5ae6
Time Ingest jobs (#182)
kespinola e317912
Use Example Configs (#180)
kespinola 7e302f1
Respect commitment level set in grpc config (#191)
kespinola 2e7efd1
Use tokio join set ingest stream task tracking
kespinola 62f99f7
Add metrics to track metadata_json_download success and failure (#207)
Nagaprasadvr d086bfc
add retry config to download_metadata_json task (#208)
Nagaprasadvr 741032a
Resolve clippy
kespinola df915d0
Use multiple pipelines per subscription (#211)
kespinola 86f223d
Das 106 add metadata json backfiller to das (#210)
Nagaprasadvr e2deabe
Shutdown program on stream error or close (#213)
kespinola a46726d
Global shutdown in the None arm
kespinola 9988bb2
Migrate to channel and worker pool for download notifier (#217)
fernandodeluret 03f5d88
add redis pipes for processing metadata download (#222)
Nagaprasadvr 8c4720d
chore : cleanup
Nagaprasadvr ac28e5e
pull main
Nagaprasadvr 8a9b835
chore: pull main
Nagaprasadvr d7dd0d0
chore: remove bubblegum
Nagaprasadvr 1ad25e5
chore: remove clippy changes
Nagaprasadvr f29c5fe
chore: remove clippy changes
Nagaprasadvr 310047e
remove slot from DownloadMetadataInfo
Nagaprasadvr b8d154f
chore : resolve comments
Nagaprasadvr edf6da7
pull upstream
Nagaprasadvr cb9113f
fix cargo lock files
Nagaprasadvr 55e4320
remove --frozen in github actions for now
Nagaprasadvr 1489bab
drop transaction that have errors
kespinola 146cc5f
pull upstream
Nagaprasadvr 74d0481
pull upstream
Nagaprasadvr 1968bc8
pull upstream
Nagaprasadvr b9e3ebe
Merge branch 'grpc-cleanup' of github.com:rpcpool/digital-asset-rpc-i…
Nagaprasadvr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/metaplex-foundation/digital-asset-rpc-infrastructure/pull/240/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R144
When we update yellowstone-grpc to later package we start getting dependency conflicts. We will be looking at resolving this but would like to do in a follow up.