Skip to content

Conversation

@Dimfacion
Copy link
Member

@Dimfacion Dimfacion commented Sep 12, 2025

Proposed changes

  • Batching came back from the dead to haunt us

Testing Instructions

  1. Step-by-step how to test
  2. Environment or config notes

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality
  • For bug fix -> I implemented a test that covers the bug

Further comments

Benchmark

For our benchmark, we used a use case Salt Typhhon like meaning 18 injects having 1000 endpoints throwing each 4 execution trace. All these injects have an output parser. Do note that the endpoints for our tests are fake endpoints. They are just threads on my machine doing post requests with their execution trace.
In total, we are supposed to have 72 000 execution traces ingested.

Current perfs with existing method :
Only 870 traces are pushed in 10mins. The backend can't follow to deal with more and the app is unusable during the benchmark. On average, each trace is taking around 20s to be ingested so around 3 trace ingested by minutes.

Perfs with this PR :
Since we're now using RabbitMQ, we can see the graph of ingestion in it :
Screenshot from 2025-11-28 10-56-54
We can see that all the 72 000 traces are pushed in a minute. They are then ingested in around 6 minutes. The ingestion varies quite a lot but it's around 200 ingestion per second or 12 000 per minute.

@Dimfacion Dimfacion force-pushed the issue/2895_2 branch 3 times, most recently from 7cf31ee to 3dbb8b2 Compare September 16, 2025 16:14
@codecov
Copy link

codecov bot commented Sep 16, 2025

Codecov Report

❌ Patch coverage is 72.98137% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.39%. Comparing base (e11b773) to head (8cfdb14).
⚠️ Report is 1 commits behind head on release/current.

Files with missing lines Patch % Lines
...o/openaev/rest/helper/queue/BatchQueueService.java 70.16% 44 Missing and 10 partials ⚠️
...st/inject/service/BatchingInjectStatusService.java 68.42% 16 Missing and 2 partials ⚠️
.../src/main/java/io/openaev/service/UserService.java 75.00% 3 Missing and 1 partial ⚠️
...rc/main/java/io/openaev/rest/inject/InjectApi.java 81.25% 1 Missing and 2 partials ⚠️
...naev/rest/inject/form/InjectExecutionCallback.java 25.00% 1 Missing and 2 partials ⚠️
...c/main/java/io/openaev/utils/ExpectationUtils.java 25.00% 0 Missing and 3 partials ⚠️
.../openaev/config/ThreadPoolTaskSchedulerConfig.java 90.90% 1 Missing ⚠️
...n/java/io/openaev/rest/finding/FindingService.java 88.88% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@                  Coverage Diff                  @@
##             release/current    #3990      +/-   ##
=====================================================
+ Coverage              50.12%   50.39%   +0.26%     
- Complexity              3683     3721      +38     
=====================================================
  Files                    907      912       +5     
  Lines                  27041    27301     +260     
  Branches                2027     2046      +19     
=====================================================
+ Hits                   13554    13758     +204     
- Misses                 12668    12707      +39     
- Partials                 819      836      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Dimfacion Dimfacion changed the title [backend] Rebasing [backend] Batching injects Sep 22, 2025
@Dimfacion Dimfacion changed the title [backend] Batching injects [backend] feat(engine): Batching injects traces (#2895) Nov 23, 2025
@Dimfacion Dimfacion changed the title [backend] feat(engine): Batching injects traces (#2895) [backend] feat(engine): batching injects traces (#2895) Nov 23, 2025
@Dimfacion Dimfacion changed the title [backend] feat(engine): batching injects traces (#2895) [backend] feat: batching injects traces (#2895) Nov 23, 2025
@Dimfacion Dimfacion changed the title [backend] feat: batching injects traces (#2895) [backend] feat: batching injects traces (#2895) Nov 23, 2025
@Dimfacion Dimfacion force-pushed the issue/2895_2 branch 3 times, most recently from 20364ba to eab9776 Compare November 27, 2025 22:32
@Dimfacion Dimfacion marked this pull request as ready for review November 27, 2025 22:32
@Dimfacion Dimfacion changed the title [backend] feat: batching injects traces (#2895) [backend] feat: batching injects traces (#2830) Nov 27, 2025
@damgouj damgouj self-requested a review December 2, 2025 13:59
Copy link
Member

@damgouj damgouj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some problems during my tests:

  • Can't relaunch an existing atomic testing after the migration
cantLaunchPreviousAtomicTesting - Error in logs during execution of an atomic testing: errorInLogs - Final inject status stays in pending: finalInjectStatusPending

@Dimfacion Dimfacion merged commit 04e2e56 into release/current Dec 4, 2025
11 checks passed
@Dimfacion Dimfacion deleted the issue/2895_2 branch December 4, 2025 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants