Skip to content

feat: Add performance metrics for virtual background TFLite#17182

Open
Aashuti-Tech-Trek wants to merge 1 commit intojitsi:masterfrom
Aashuti-Tech-Trek:feature/virtual-background-metrics-clean
Open

feat: Add performance metrics for virtual background TFLite#17182
Aashuti-Tech-Trek wants to merge 1 commit intojitsi:masterfrom
Aashuti-Tech-Trek:feature/virtual-background-metrics-clean

Conversation

@Aashuti-Tech-Trek
Copy link

This PR introduces performance metrics logging for the virtual background TensorFlow Lite pipeline inside JitsiStreamBackgroundEffect.ts. The goal of this change is to provide measurable performance benchmarks for the current TFLite implementation so that future improvements (such as MediaPipe migration, WebGL compositing, and requestVideoFrameCallback scheduling) can be evaluated against concrete baseline metrics.

The implementation tracks inference time, post-processing time, and achieved frames per second (FPS) over a batch of frames and periodically logs averaged metrics for performance evaluation.

Changes made

  • Added performance metric tracking variables:

    • inferenceTimeSum
    • postProcessingTimeSum
    • frameCount
    • lastReportTime
  • Measured inference execution time around runInference().

  • Measured post-processing execution time around runPostProcessing().

  • Calculated average inference time, average post-processing time, and FPS over a 30-frame interval.

  • Logged performance metrics using the virtual background logger for benchmarking and debugging.

  • Added a TypeScript type guard for FileReader.result in UploadImageButton.tsx to ensure type safety when passing base64 image data to resizeImage().

Why this change

This change provides baseline performance metrics for the existing TensorFlow Lite virtual background pipeline. Since the Virtual Backgrounds project involves migrating to MediaPipe and improving performance using WebGL and requestVideoFrameCallback, having measurable metrics such as inference time, post-processing time, and FPS is important for comparing performance before and after architectural changes.

This helps developers:

  • Benchmark current performance
  • Measure improvements after MediaPipe migration
  • Identify performance bottlenecks
  • Evaluate rendering and inference optimizations

Impact

  • Internal performance instrumentation only.
  • No UI changes.
  • No behavior changes to virtual background rendering.
  • Adds logging for performance benchmarking and debugging.
  • Helps future performance optimization and MediaPipe migration work.

Testing

  • Verified that virtual background still renders correctly.
  • Verified that performance logs appear in the console periodically.
  • Ran npm run tsc:web and confirmed there are no TypeScript errors.

Performance Metrics Preview

Below are sample logs and output while running the virtual background with performance instrumentation enabled.

after

Related Issue Fixes #17174

@jitsi-jenkins
Copy link

Hi, thanks for your contribution!
If you haven't already done so, could you please make sure you sign our CLA (https://jitsi.org/icla for individuals and https://jitsi.org/ccla for corporations)? We would unfortunately be unable to merge your patch unless we have that piece :(.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add baseline performance metrics for virtual background segmentation

2 participants