Instrument tracing spans for block processing and import

[jimmygchen]

Issue Addressed

#7815

• removes all existing spans, so some span fields that appear in logs like service_name may be lost.
• instruments a few key code paths in the beacon node, starting from root spans named below:

• Gossip block and blobs
  • process_gossip_data_column_sidecar
  • process_gossip_blob
  • process_gossip_block
• Rpc block and blobs
  • process_rpc_block
  • process_rpc_blobs
  • process_rpc_custody_columns
• Rpc blocks (range and backfill)
  • process_chain_segment
• PendingComponents lifecycle
  • pending_components

To test locally:

• Run Grafana and Tempo with Add Tempo tracing backend to collect span data lighthouse-metrics#57
• Run Lighthouse BN with --telemetry-collector-url http://localhost:4317

Some captured traces can be found here: https://hackmd.io/@jimmygchen/r1sLOxPPeg

Removing the old spans seem to have reduced the memory usage quite a lot - i think we were using them on long running tasks and too excessively:

[jimmygchen]

Just reading the doc on spans, I think some of the async functions are instrumented incorrectly, will revisit this on Monday!
https://docs.rs/tracing/latest/tracing/span/struct.Span.html

I've checked all enter() and entered usage and I believe they're good now.

Now seeing good traces in Grafana:

[pawanjay176]

This is brilliant. I'm gonna play around with this later today

[eserilev]

This is great! Just a few minor nits from me, LGTM

[mergify]

Some required checks have failed. Could you please take a look @jimmygchen? 🙏

[jimmygchen]

@eserilev thanks for the review 🙏
I've got some more changes & refactor on another branch, but will merge this one and do a separate PR since this one is getting too big and already reviewed!

[mergify]

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

• ☑️ test-suite-success

You can check the last failing draft PR here: #7842.

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

[mergify]

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

• ☑️ test-suite-success

You can check the last failing draft PR here: #7843.

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.