Skip to content

Implement tracing spans for data columm RPC requests and responses #7831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 70 commits into from
Aug 20, 2025
Merged

Implement tracing spans for data columm RPC requests and responses #7831

merged 70 commits into from
Aug 20, 2025

Conversation

jimmygchen
Copy link
Member

@jimmygchen jimmygchen commented Aug 6, 2025 •
edited
Loading

Issue Addressed

#7830

New root spans

syncing_chain

Span to group events by syncing chain and add context to logs.

outgoing_range_request

Traces a collection of outgoing range requests for a single slot range, tracking the requester (backfill or range sync).

Contains child spans that tracks individual requests to peers:

  • outgoing_blocks_by_range
  • outgoing_columns_by_range
  • outgoing_blobs_by_range
image

outgoing_custody_request

Traces the span of a collection of data column requests for a given block_root, which contains one to many batch_columns_req spans, each mapped to an actual RPC request with a peer_id, client, missing_column_indexes (recorded on response).

image

handle_data_columns_by_root_request

Traces an incoming DataColumnsByRoot request, with fields peer_id, non_custody_indices, client.

handle_data_columns_by_range_request

Traces an incoming DataColumnsByRange request, with fields peer_id, non_custody_indices, client.

NOTE: Outgoing BlocksByRoot and BlobsByRoot not implemented.

jimmygchen added 30 commits July 30, 2025 13:05
@jimmygchen
Add opentelemetry OTLP exporter for spans.
c1a50bb
@jimmygchen
Add workspace_filter to telemetry layer to exclude libp2p traces.
6dba431
@jimmygchen
Squashed commit of the following:
4147599 
commit 52bc831
Author: Michael Sproul <[email protected]>
Date:   Mon Jul 21 13:56:44 2025 +1000

    Remove tracing instrumentation

# Conflicts:
#	beacon_node/network/src/sync/manager.rs
#	beacon_node/network/src/sync/peer_sampling.rs
@jimmygchen
Remove all spans.
04c23d9
@jimmygchen
load_parent test.
03c647a
@jimmygchen
Add gossip block instrumentation.
c0804dc
@jimmygchen
more instrument.
0a8d22f
@jimmygchen
Instrument process_block and import_block.
172d859
@jimmygchen
Add more detailed instruments to import_block
386b425
@jimmygchen
Add more detailed instruments to import_block
4268ae5
@jimmygchen
Add more detailed instruments to import_block
3f55de0
@jimmygchen
More instruments.
26e30c3
@jimmygchen
More instruments.
db4d342
@jimmygchen
More instruments.
62940aa
@jimmygchen
Add service name
e3f6273
@jimmygchen
More instruments
2f6c32b
@jimmygchen
Attempt to fix orphan spans
e7fcb4c
@jimmygchen
A few more
868320a
@jimmygchen
disable resource to test root span not yet received issue
a8055ae
@jimmygchen
Clean ups.
e4e9746
@jimmygchen
Revert potential bad change from 62940aa (resulted in missing root sp…
533ecc0 
…ans)
@jimmygchen
Add instrument back in.
29d65bc
@jimmygchen
Merge branch 'tracing-spans-test' of github.com:jimmygchen/lighthouse…
033861b 
… into tracing-spans-test
@jimmygchen
Make process_block a root span.
3ac075f
@jimmygchen
More fixes
fb3bf31
@jimmygchen
More fixes
333773f
@jimmygchen
Fix span levels and add tracing to gossip/rpc block and blob paths.
e1f584e
@jimmygchen
Add process chain segment
c0991f5
@jimmygchen
Set parent to None on network beacon processors to prevent inheriting…
d340231 
… libp2p spans.
@jimmygchen
Add RPC method instruments.
91a2400
@mergify Mergify
Copy link

mergify bot commented Aug 15, 2025

Some required checks have failed. Could you please take a look @jimmygchen? 🙏

@mergify mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Aug 15, 2025
@jimmygchen jimmygchen added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Aug 19, 2025
@jimmygchen
Copy link
Member Author

I've done some clean up and I think this is now ready for review.

Block lookup is not fully instrumented - so far I've only instrument custody columns by roots, I think eventually it would be nice to group all by roots calls for a single root under a root span. However this PR is getting huge, so i think it would be good to get some feedback and do the rest in a separate PR.

@eserilev would you mind reviewing this please?

@jimmygchen jimmygchen requested a review from eserilev August 19, 2025 08:37
jimmygchen added a commit that referenced this pull request Aug 19, 2025
@jimmygchen
Implement tracing spans for data columm RPC requests and responses (#…
2e3fe9d 
…7831)

Squashed commit of the following:

commit 5ed1c6d
Author: Jimmy Chen <[email protected]>
Date:   Tue Aug 19 18:33:24 2025 +1000

    Code clean up and remove unnecessary span fields.

commit 5aa0b82
Merge: 5467596 34dd1b2
Author: Jimmy Chen <[email protected]>
Date:   Tue Aug 19 17:26:27 2025 +1000

    Merge remote-tracking branch 'origin/unstable' into tracing-spans-rpc

commit 34dd1b2
Author: Jimmy Chen <[email protected]>
Date:   Tue Aug 19 13:48:08 2025 +1000

    Revise data column rpc limits and queue sizes (#7887)

    Revise data column rpc limits and queue sizes. Also removed some outdated TODOs for Fulu / das.

commit 5467596
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 15 17:16:06 2025 +1000

    Remove unused deps

commit 573cbf3
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 15 17:13:57 2025 +1000

    Remove span fields that are not useful

commit 61d8913
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 15 16:53:55 2025 +1000

    Consistent span level

commit 4b90aef
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 15 16:52:38 2025 +1000

    Fix docs

commit 5f18692
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 15 16:48:03 2025 +1000

    Add more context to traces

commit ee50628
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 14 17:20:22 2025 +1000

    Replace syncing chain child spans with events

commit fa3051a
Merge: 9ae161a 921fa75
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 14 16:57:41 2025 +1000

    Merge branch 'tracing-spans-rpc' of github.com:jimmygchen/lighthouse into tracing-spans-rpc

commit 9ae161a
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 14 16:02:49 2025 +1000

    Instrument `SyncingChain`

commit 921fa75
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 14 16:02:49 2025 +1000

    Instrumnet `SyncingChain`

commit ac75779
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 13 18:06:52 2025 +1000

    Remove root span filtering layer, couldn't get it to work.

commit 3e020b9
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 13 18:03:09 2025 +1000

    Add some notes about panic

commit 9132c9e
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 13 17:28:58 2025 +1000

    Add missing instrument on backfill

commit 433010d
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 13 17:24:46 2025 +1000

    Revert Makefile changes and re-arrange imports

commit 7762c16
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 13 16:28:50 2025 +1000

    Skip exporting spans that are not part of / descendants of the allowed root spans list.

commit 5acfd53
Merge: 474d100 522bd9e
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 13 14:13:20 2025 +1000

    Merge remote-tracking branch 'origin/unstable' into tracing-spans-rpc

    # Conflicts:
    #	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
    #	beacon_node/network/src/sync/block_sidecar_coupling.rs
    #	beacon_node/network/src/sync/network_context.rs
    #	beacon_node/network/src/sync/network_context/custody.rs

commit 474d100
Merge: 1d6ee71 40c2fd5
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 8 16:00:54 2025 +1000

    Merge remote-tracking branch 'origin/unstable' into tracing-spans-rpc

    # Conflicts:
    #	beacon_node/beacon_chain/src/block_verification.rs
    #	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
    #	beacon_node/beacon_chain/src/fetch_blobs/mod.rs
    #	beacon_node/lighthouse_network/src/service/mod.rs
    #	beacon_node/network/src/network_beacon_processor/gossip_methods.rs
    #	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
    #	beacon_node/network/src/network_beacon_processor/sync_methods.rs
    #	beacon_node/network/src/sync/network_context.rs
    #	lighthouse/src/main.rs

commit 1d6ee71
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 15:21:13 2025 +1000

    Record client kind properly

commit e2605e3
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 15:10:39 2025 +1000

    Record client kind properly

commit 226b635
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 14:49:02 2025 +1000

    Display client kind instead

commit addf716
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 14:28:01 2025 +1000

    Refactor rabithole to clean things up and only allow for pre-defined root spans to be sent.

commit 57a8d56
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 00:24:09 2025 +1000

    More missing enters..

commit 40b84f3
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 00:18:23 2025 +1000

    Fix spans not entering correctly.

commit e922250
Author: Jimmy Chen <[email protected]>
Date:   Thu Aug 7 00:05:45 2025 +1000

    Rename `outgoing_custody_request` and instrument all rpc methods with peer_id and client.

commit 68c8345
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 23:57:36 2025 +1000

    Track individual range requests.

commit 4c7ef8b
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 22:55:14 2025 +1000

    Close a span properly and remove unnecessary span.

commit cf3a216
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 22:34:43 2025 +1000

    Improve RPC spans.

commit e491c85
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 22:00:47 2025 +1000

    Add missing field declarations.

commit 4dddfc1
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 21:07:18 2025 +1000

    Fix lint

commit a5ce826
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 18:32:53 2025 +1000

    Instrument rpc methods and by root responses. (TODO: by range responses)

commit ec37ad9
Merge: 1cd9683 c06ac81
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 16:12:50 2025 +1000

    Merge branch 'unstable' into tracing-spans-test

commit 1cd9683
Author: Jimmy Chen <[email protected]>
Date:   Wed Aug 6 16:12:37 2025 +1000

    Remove unnecessary `allow` macro usage

commit 4af2855
Author: Jimmy Chen <[email protected]>
Date:   Sat Aug 2 01:29:23 2025 +1000

    Instrument `PendingComponents` set parent to `None` to avoid incorrectly impacting component processing traces

commit b358a1a
Author: Jimmy Chen <[email protected]>
Date:   Sat Aug 2 01:17:13 2025 +1000

    Instrument `PendingComponents`.

commit 61a772f
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 1 17:41:53 2025 +1000

    Fill more instrument gaps in `process_chain_segment`

commit d1751f1
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 1 17:33:59 2025 +1000

    Fill more instrument gaps in `process_chain_segment`

commit 4a3689d
Author: Jimmy Chen <[email protected]>
Date:   Fri Aug 1 17:04:48 2025 +1000

    Remove unnecessary enter()

commit 91a2400
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 23:03:38 2025 +1000

    Add RPC method instruments.

commit d340231
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 21:35:26 2025 +1000

    Set parent to None on network beacon processors to prevent inheriting libp2p spans.

commit c0991f5
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 16:54:21 2025 +1000

    Add process chain segment

commit e1f584e
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 16:14:46 2025 +1000

    Fix span levels and add tracing to gossip/rpc block and blob paths.

commit 333773f
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 14:44:42 2025 +1000

    More fixes

commit fb3bf31
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 14:38:41 2025 +1000

    More fixes

commit 3ac075f
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 13:10:41 2025 +1000

    Make `process_block` a root span.

commit 033861b
Merge: 29d65bc 533ecc0
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 13:10:09 2025 +1000

    Merge branch 'tracing-spans-test' of github.com:jimmygchen/lighthouse into tracing-spans-test

commit 29d65bc
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 13:07:18 2025 +1000

    Add instrument back in.

commit 533ecc0
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 12:44:02 2025 +1000

    Revert potential bad change from 62940aa (resulted in missing root spans)

commit e4e9746
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 11:20:24 2025 +1000

    Clean ups.

commit a8055ae
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 10:30:08 2025 +1000

    disable resource to test root span not yet received issue

commit 868320a
Author: Jimmy Chen <[email protected]>
Date:   Thu Jul 31 10:12:07 2025 +1000

    A few more

commit e7fcb4c
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 23:22:31 2025 +1000

    Attempt to fix orphan spans

commit 2f6c32b
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 20:47:33 2025 +1000

    More instruments

commit e3f6273
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 20:41:24 2025 +1000

    Add service name

commit 62940aa
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 20:21:27 2025 +1000

    More instruments.

commit db4d342
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 18:01:55 2025 +1000

    More instruments.

commit 26e30c3
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 17:40:32 2025 +1000

    More instruments.

commit 3f55de0
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 17:19:01 2025 +1000

    Add more detailed instruments to `import_block`

commit 4268ae5
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 16:57:19 2025 +1000

    Add more detailed instruments to `import_block`

commit 386b425
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 16:48:46 2025 +1000

    Add more detailed instruments to `import_block`

commit 172d859
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 16:27:29 2025 +1000

    Instrument `process_block` and `import_block`.

commit 0a8d22f
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 15:51:34 2025 +1000

    more instrument.

commit c0804dc
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 15:26:00 2025 +1000

    Add gossip block instrumentation.

commit 03c647a
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 14:43:29 2025 +1000

    load_parent test.

commit 04c23d9
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 14:25:40 2025 +1000

    Remove all spans.

commit 4147599
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 14:19:07 2025 +1000

    Squashed commit of the following:

    commit 52bc831
    Author: Michael Sproul <[email protected]>
    Date:   Mon Jul 21 13:56:44 2025 +1000

        Remove tracing instrumentation

    # Conflicts:
    #	beacon_node/network/src/sync/manager.rs
    #	beacon_node/network/src/sync/peer_sampling.rs

commit 6dba431
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 13:08:32 2025 +1000

    Add `workspace_filter` to telemetry layer to exclude libp2p traces.

commit c1a50bb
Author: Jimmy Chen <[email protected]>
Date:   Wed Jul 30 13:05:06 2025 +1000

    Add opentelemetry OTLP exporter for spans.

# Conflicts:
#	beacon_node/beacon_chain/src/validator_custody.rs
#	beacon_node/network/src/sync/network_context/custody.rs
@jimmygchen jimmygchen mentioned this pull request Aug 19, 2025
Closed
2 tasks
@mergify Mergify
Copy link

mergify bot commented Aug 20, 2025

This pull request has merge conflicts. Could you please resolve them @jimmygchen? 🙏

@mergify mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Aug 20, 2025
@jimmygchen
Merge remote-tracking branch 'origin/unstable' into tracing-spans-rpc
6cb4ed5 
# Conflicts:
#	beacon_node/network/src/sync/network_context/custody.rs
@jimmygchen jimmygchen added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Aug 20, 2025
@eserilev eserilev mentioned this pull request Aug 20, 2025
Open
13 tasks
@mergify Mergify
Copy link

mergify bot commented Aug 20, 2025

This pull request has merge conflicts. Could you please resolve them @jimmygchen? 🙏

@mergify mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Aug 20, 2025
@jimmygchen
Merge remote-tracking branch 'origin/unstable' into tracing-spans-rpc
1dd9188 
# Conflicts:
#	beacon_node/beacon_chain/src/validator_custody.rs
@mergify mergify bot added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Aug 20, 2025
eserilev
eserilev approved these changes Aug 20, 2025
View reviewed changes
Copy link
Member

@eserilev eserilev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jimmygchen jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Aug 20, 2025
@mergify mergify bot merged commit f19d4f6 into sigp:unstable Aug 20, 2025
34 checks passed
@jimmygchen jimmygchen mentioned this pull request Aug 21, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eserilev eserilev eserilev approved these changes

@michaelsproul michaelsproul Awaiting requested review from michaelsproul

@jxs jxs Awaiting requested review from jxs jxs is a code owner

Assignees
No one assigned
Labels
ready-for-merge This PR is ready to merge. tracing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jimmygchen @eserilev