Add das-grpc-ingest crate - #240

Open

Nagaprasadvr wants to merge 83 commits into

metaplex-foundation:mainfrom

rpcpool:grpc-cleanup

Commits on Apr 16, 2024

fmt

fanatid
authored and
kespinola
committed
ea5379e
clippy

fanatid
authored and
kespinola
committed
8b12316
ci: add lock, fmt, clippy checks

fanatid
authored and
kespinola
committed
974c90f
include migration to workspace

fanatid
authored and
kespinola
committed
788973e
include das_api to workspace

fanatid
authored and
kespinola
committed
7a870c2
add global clippy lints

fanatid
authored and
kespinola
committed
7e3f9ef
use workspace

fanatid
authored and
kespinola
committed
aa916a4
add crate program_transformers

fanatid
authored and
kespinola
committed
033873d
nft_ingester: use program_transformers crate

fanatid
authored and
kespinola
committed
1008f86
remove not used deps

fanatid
authored and
kespinola
committed
83b171e
remove AccountInfo

fanatid
authored and
kespinola
committed
9d8031e
remove plerkle from program_transformers

fanatid
authored and
kespinola
committed
096616d
nft_ingester2: grpc2redis

fanatid
authored and
kespinola
committed
ef0b0e9
add redis streaming for ingester

fanatid
authored and
kespinola
committed
076bffa
create pg pool

fanatid
authored and
kespinola
committed
d8673cf
parse incoming message from redis

fanatid
authored and
kespinola
committed
c93adff
add force shutdown with signals

fanatid
authored and
kespinola
committed
53fa481
insert download metadata tasks

fanatid
authored and
kespinola
committed
11b4151
download-metadata subtask

fanatid
authored and
kespinola
committed
8fca2c2

Commits on Feb 10, 2025

refactor: rename nft_ingest2

kespinola
authored and
Nagaprasadvr
committed
3d5eb18
fix: applying account and transction filters to grpc subscription request

kespinola
authored and
Nagaprasadvr
committed
e50e9ba
refactor: take out multiple connections to dragonmouth and then have the main thread keep a chache of seen events so that it doesnt push to ingest if seen by another connections

kespinola
authored and
Nagaprasadvr
committed
973a599
refactor: switch to accepting multiple dragonmouth endoints to mimic dual validators writing to ingest stream

kespinola
authored and
Nagaprasadvr
committed
dd564e5
fix: clippy errors

kespinola
authored and
Nagaprasadvr
committed
5a39d3f
chor: refactor grpc command to use topograph to clean up the control flow. also pull in backfill to be used in ingest

kespinola
authored and
Nagaprasadvr
committed
139ed01
chore: switch to run_v2 for ingest which handles ingesting accounts and transactions

kespinola
authored and
Nagaprasadvr
committed
0883efc
chore: handle metadata json within the ingest command

kespinola
authored and
Nagaprasadvr
committed
07b0936
test: metadata json stream processing

kespinola
authored and
Nagaprasadvr
committed
dff868b
feat: set num threads for topograph. report on postgres and redis in ingester using dedicate exec

kespinola
authored and
Nagaprasadvr
committed
8f28a69
chore: switch to topograph for thread management. create separate ingest stream which handle reading messages and handling messages.

kespinola
authored and
Nagaprasadvr
committed
2cf9e57
fix: config and ack

kespinola
authored and
Nagaprasadvr
committed
454aa8e
hack: disable pending processing while see why messages arent being deleted

kespinola
authored and
Nagaprasadvr
committed
b8249ef
fix: exit early if slot matches metadata json download and reindex is false.

kespinola
authored and
Nagaprasadvr
committed
4211a54
fix: picking dangling pending messages before reading new

kespinola
authored and
Nagaprasadvr
committed
a574695
fix: ingest stream shutdown and requesting snapshots

kespinola
authored and
Nagaprasadvr
committed
078b0f9
fix: processing pending. add metrics for total workers.

kespinola
authored and
Nagaprasadvr
committed
a50ff86
fix: bubble backfill transform. no op out of download metadata info.

kespinola
authored and
Nagaprasadvr
committed
e6bf732
Support Account Snapshot Flushes (#154)

authored and
Nagaprasadvr
committed
103db2f
fix: keep grpc alive by sending using ping-pong (#160)
Nagaprasadvr
committed
0db4391
Only Time Based Flushes (#161)

kespinola
authored and
Nagaprasadvr
committed
6efad62
delete and recreate the consumer (#162)

kespinola
authored and
Nagaprasadvr
committed
db7ee2d
Type out redis message handlers (#163)

kespinola
authored and
Nagaprasadvr
committed
b5e9d99
skip reading new messages if the ack buffer is full. track running tasks. (#164)

kespinola
authored and
Nagaprasadvr
committed
19b61b8
only push handler job for ingest if capacity (#165)

kespinola
authored and
Nagaprasadvr
committed
2d4e4b9
Only throttle based on the task semaphore (#166)

kespinola
authored and
Nagaprasadvr
committed
1a75b62
Remove topograph from ingest command (#167)

kespinola
authored and
Nagaprasadvr
committed
aea798a
Configure database connection idle timeout and max lifetime (#169)

kespinola
authored and
Nagaprasadvr
committed
fd58811
Bubblegum verify (#150)

kespinola
authored and
Nagaprasadvr
committed
1bc38e5
Separate global stream to multiple Program separated streams and handle each stream (#168)

Nagaprasadvr
and
kespinola
committed
794fc32
Report proofs as they come (#178)

kespinola
authored and
Nagaprasadvr
committed
3b9e68a
Buffer redis messages (#179)

kespinola
authored and
Nagaprasadvr
committed
18a6913
Fix tracking ack tasks

kespinola
authored and
Nagaprasadvr
committed
e53f7eb
fix: add back ack tasks total

kespinola
authored and
Nagaprasadvr
committed
39fb1b7
recreate consumer but not groups (#181)

kespinola
authored and
Nagaprasadvr
committed
ac053f4
Time Ingest jobs (#182)

kespinola
authored and
Nagaprasadvr
committed
51c5ae6
Use Example Configs (#180)

authored and
Nagaprasadvr
committed
e317912
Respect commitment level set in grpc config (#191)

kespinola
authored and
Nagaprasadvr
committed
7e302f1
Use tokio join set ingest stream task tracking

kespinola
authored and
Nagaprasadvr
committed
2e7efd1
Add metrics to track metadata_json_download success and failure (#207)

Nagaprasadvr
and
kespinola
committed
62f99f7
add retry config to download_metadata_json task (#208)
Nagaprasadvr
committed
d086bfc
Resolve clippy

kespinola
authored and
Nagaprasadvr
committed
741032a
Use multiple pipelines per subscription (#211)

kespinola
authored and
Nagaprasadvr
committed
df915d0
Das 106 add metadata json backfiller to das (#210)

Nagaprasadvr
and
kespinola
committed
86f223d
Shutdown program on stream error or close (#213)

kespinola
authored and
Nagaprasadvr
committed
e2deabe
Global shutdown in the None arm

kespinola
authored and
Nagaprasadvr
committed
a46726d
Migrate to channel and worker pool for download notifier (#217)

authored and
Nagaprasadvr
committed
9988bb2
add redis pipes for processing metadata download (#222)
Nagaprasadvr
committed
03f5d88
chore : cleanup
Nagaprasadvr
committed
8c4720d
pull main
Nagaprasadvr
committed
ac28e5e
chore: pull main
Nagaprasadvr
committed
8a9b835
chore: remove bubblegum
Nagaprasadvr
committed
d7dd0d0

Commits on Feb 11, 2025

chore: remove clippy changes
Nagaprasadvr
committed
1ad25e5
chore: remove clippy changes
Nagaprasadvr
committed
f29c5fe

Commits on Feb 12, 2025

remove slot from DownloadMetadataInfo
Nagaprasadvr
committed
310047e

Commits on Feb 13, 2025

chore : resolve comments
Nagaprasadvr
committed
b8d154f

Commits on Feb 17, 2025

pull upstream
Nagaprasadvr
committed
edf6da7
fix cargo lock files
Nagaprasadvr
committed
cb9113f
remove --frozen in github actions for now
Nagaprasadvr
committed
55e4320

Commits on Mar 14, 2025

drop transaction that have errors

kespinola
authored and
Nagaprasadvr
committed
1489bab
pull upstream
Nagaprasadvr
committed
146cc5f

Commits on Apr 25, 2025

pull upstream
Nagaprasadvr
committed
74d0481
pull upstream
Nagaprasadvr
committed
1968bc8
Merge branch 'grpc-cleanup' of github.com:rpcpool/digital-asset-rpc-infrastructure into grpc-cleanup
Nagaprasadvr
committed
b9e3ebe