feat: Add performance metrics for virtual background TFLite#17182
Open
Aashuti-Tech-Trek wants to merge 1 commit intojitsi:masterfrom
Open
feat: Add performance metrics for virtual background TFLite#17182Aashuti-Tech-Trek wants to merge 1 commit intojitsi:masterfrom
Aashuti-Tech-Trek wants to merge 1 commit intojitsi:masterfrom
Conversation
|
Hi, thanks for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces performance metrics logging for the virtual background TensorFlow Lite pipeline inside
JitsiStreamBackgroundEffect.ts. The goal of this change is to provide measurable performance benchmarks for the current TFLite implementation so that future improvements (such as MediaPipe migration, WebGL compositing, andrequestVideoFrameCallbackscheduling) can be evaluated against concrete baseline metrics.The implementation tracks inference time, post-processing time, and achieved frames per second (FPS) over a batch of frames and periodically logs averaged metrics for performance evaluation.
Changes made
Added performance metric tracking variables:
Measured inference execution time around runInference().
Measured post-processing execution time around runPostProcessing().
Calculated average inference time, average post-processing time, and FPS over a 30-frame interval.
Logged performance metrics using the virtual background logger for benchmarking and debugging.
Added a TypeScript type guard for FileReader.result in UploadImageButton.tsx to ensure type safety when passing base64 image data to resizeImage().
Why this change
This change provides baseline performance metrics for the existing TensorFlow Lite virtual background pipeline. Since the Virtual Backgrounds project involves migrating to MediaPipe and improving performance using WebGL and requestVideoFrameCallback, having measurable metrics such as inference time, post-processing time, and FPS is important for comparing performance before and after architectural changes.
This helps developers:
Impact
Testing
Performance Metrics Preview
Below are sample logs and output while running the virtual background with performance instrumentation enabled.
Related Issue Fixes #17174