Skip to content

ref(profiling): implement chunked profile candidate query for flamegraph #95873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 22, 2025

Conversation

viglia
Copy link
Contributor

@viglia viglia commented Jul 18, 2025

This PR introduces an optimized chunked querying strategy for profile flamegraphs that improves performance when querying large time ranges. Instead of querying the entire datetime range at once, the new approach splits queries into exponentially growing chunks, starting with smaller intervals and potentially finding sufficient data without needing to query the full range.

Key Changes:

  • Added feature flag organizations:profiling-flamegraph-use-increased-chunks-query-strategy for gradual rollout
  • Added configurable options for initial chunk size (12h default) and maximum chunk size (48h default)
  • Implemented get_profile_candidates_from_transactions_v2() and get_profile_candidates_from_profiles_v2() methods with chunked querying logic
  • Early termination when sufficient profile candidates are found, reducing unnecessary queries

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 18, 2025
Copy link

codecov bot commented Jul 18, 2025

Codecov Report

Attention: Patch coverage is 7.60870% with 85 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/profiles/flamegraph.py 4.49% 85 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #95873      +/-   ##
==========================================
+ Coverage   86.49%   87.68%   +1.18%     
==========================================
  Files       10532    10583      +51     
  Lines      608735   610037    +1302     
  Branches    23950    23950              
==========================================
+ Hits       526530   534900    +8370     
+ Misses      81912    74844    -7068     
  Partials      293      293              

@viglia viglia changed the title Viglia/ref/implement chunked transaction profile query ref(profiling): implement chunked transaction profile query Jul 18, 2025
@viglia viglia changed the title ref(profiling): implement chunked transaction profile query ref(profiling): implement chunked profile candidate query for flamegraph Jul 18, 2025
@viglia viglia marked this pull request as ready for review July 18, 2025 14:38
@viglia viglia requested a review from a team as a code owner July 18, 2025 14:38
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as resolved.

Copy link
Member

@Zylphrex Zylphrex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fully convinced the profiling-flamegraph-always-use-direct-chunks branch will work as intended but I'm open to giving it a try and iterate.

Comment on lines +710 to +715
# If we still don't have enough continuous profile candidates + transaction profile candidates,
# we'll fall back to directly using the continuous profiling data
if (
len(continuous_profile_candidates) + len(transaction_profile_candidates) < max_profiles
and always_use_direct_chunks
):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I don't remember how well we dedupe profile chunks so this might result in misleading flamegraphs
  2. I'm not fully convinced this condition is enough in higher throughput situations. Though most of the time, the problem we're trying to resolve by directly looking at chunks is in low throughput situations so it may be okay.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. we don't dedupe in vroom so, if we were to send a duplicate in the list it would be used twice for the aggregation, but here we prevent that by keeping track of seen_chunks. So, if we fall in this branch, we'll still check that set and only add chunks that were not previously added and avoid adding duplicates.
  2. agree. I'm mainly worried about:
  • lower throughput situations
  • still keeping the max number of chunks/profiles <= max_profiles

Anyway as you mentioned in another comment, we can give it a try to see how it works and later iterate to improve it.

@viglia viglia merged commit 4ccab01 into master Jul 22, 2025
65 checks passed
@viglia viglia deleted the viglia/ref/implement-chunked-transaction-profile-query branch July 22, 2025 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants