feat(llc): speech recognition while muted #999
Conversation
WalkthroughThis update introduces a new system to detect when a user is speaking while muted during a call. It adds audio recognition capabilities using WebRTC, exposes new audio processing APIs, implements state management for "speaking while muted" detection, and provides unit tests and supporting mocks for these features. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant CallScreen
participant SpeakingWhileMutedRecognition
participant AudioRecognition (WebRTC)
participant SnackBar
User->>CallScreen: Joins call
CallScreen->>SpeakingWhileMutedRecognition: Initialize with current Call
loop During call
SpeakingWhileMutedRecognition->>Call: Listen for mute state changes
alt User is muted
SpeakingWhileMutedRecognition->>AudioRecognition (WebRTC): Start audio detection
AudioRecognition (WebRTC)-->>SpeakingWhileMutedRecognition: isSpeaking = true
SpeakingWhileMutedRecognition-->>CallScreen: isSpeakingWhileMuted = true
CallScreen->>SnackBar: Show "You are speaking while muted"
else User is unmuted or leaves
SpeakingWhileMutedRecognition->>AudioRecognition (WebRTC): Stop detection
SpeakingWhileMutedRecognition-->>CallScreen: isSpeakingWhileMuted = false
end
end
CallScreen->>SpeakingWhileMutedRecognition: Dispose on call end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15 minutes Assessment against linked issues
Assessment against linked issues: Out-of-scope changesNo out-of-scope changes detected. Suggested reviewers
Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (3)
packages/stream_video/lib/src/audio_processing/audio_recognition.dart (1)
3-11: Add documentation to interface methods.The interface methods lack documentation. Please add doc comments to clarify the expected behavior and lifecycle of each method.
abstract interface class AudioRecognition { + /// Starts audio recognition and invokes [onSoundStateChanged] when sound state changes. + /// Throws if recognition is already started. Future<void> start({ required SoundStateChangedCallback onSoundStateChanged, }); + /// Stops audio recognition. + /// Safe to call multiple times. Future<void> stop(); + /// Disposes of all resources. + /// The instance should not be used after calling this method. Future<void> dispose(); }packages/stream_video/lib/src/audio_processing/speaking_while_muted_recognition.dart (1)
55-61: Consider deferring subscription initialization.Calling
_init()in the constructor starts listening to call state immediately. This could lead to race conditions if the call is not fully initialized. Consider deferring initialization or making it explicit.Consider either:
- Making initialization explicit by exposing an
init()method- Deferring the subscription until the first
start()call- Adding a check for call readiness before subscribing
packages/stream_video/lib/src/audio_processing/audio_recognition_webrtc.dart (1)
13-76: Consider if dual peer connections are necessary.The implementation uses two peer connections to detect audio. This seems overly complex for simple audio level detection. Have you considered using the Web Audio API or directly monitoring the MediaStreamTrack's audio level?
The dual peer connection approach might be inspired by the React Native implementation mentioned in the PR description, but it adds complexity and resource overhead. Consider exploring simpler alternatives:
- Use the Web Audio API's AnalyserNode for direct audio level monitoring
- Monitor the MediaStreamTrack's statistics directly if available
- Document why this approach was chosen if there are specific technical reasons
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
dogfooding/lib/screens/call_screen.dart(1 hunks)packages/stream_video/CHANGELOG.md(1 hunks)packages/stream_video/lib/src/audio_processing/audio_recognition.dart(1 hunks)packages/stream_video/lib/src/audio_processing/audio_recognition_webrtc.dart(1 hunks)packages/stream_video/lib/src/audio_processing/speaking_while_muted_recognition.dart(1 hunks)packages/stream_video/lib/src/call_state.dart(1 hunks)packages/stream_video/lib/stream_video.dart(1 hunks)packages/stream_video/test/src/audio_processing/speaking_while_muted_recognition_test.dart(1 hunks)packages/stream_video/test/src/core/client_state_test.dart(1 hunks)packages/stream_video/test/test_helpers.dart(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (7)
- GitHub Check: analyze
- GitHub Check: stream_video
- GitHub Check: stream_video_push_notification
- GitHub Check: stream_video_noise_cancellation
- GitHub Check: stream_video_flutter
- GitHub Check: stream_video_screen_sharing
- GitHub Check: build
🔇 Additional comments (10)
packages/stream_video/lib/src/call_state.dart (1)
221-221: LGTM - Minor formatting improvementThe blank line addition improves code readability in the
copyWithmethod without affecting functionality.packages/stream_video/test/test_helpers.dart (2)
5-5: Good refactoring - Centralizing mock implementationsMoving
MockCallto the shared test helpers improves reusability across test files and follows good testing practices.
12-12: Well-structured mock for audio recognition testingThe
MockAudioRecognitionmock properly supports testing of the newSpeakingWhileMutedRecognitionfeature and follows the established mocktail patterns in this file.packages/stream_video/test/src/core/client_state_test.dart (1)
7-7: LGTM - Consistent with mock centralizationThe import correctly uses the centralized
MockCallfrom test helpers, eliminating code duplication and improving maintainability.packages/stream_video/CHANGELOG.md (1)
5-5: Well-documented feature additionThe changelog entry clearly describes the new
SpeakingWhileMutedRecognitionfeature and is properly placed in the "Added" section with appropriate formatting.packages/stream_video/lib/stream_video.dart (1)
5-6: Appropriate public API expansionThe new exports correctly expose the audio recognition interfaces and
SpeakingWhileMutedRecognitionclass to the public API, enabling users to implement the speaking-while-muted detection feature as described in the PR objectives.packages/stream_video/lib/src/audio_processing/speaking_while_muted_recognition.dart (1)
9-53: Excellent documentation!The documentation clearly explains the behavior, state transitions, and provides a practical example. This greatly improves the developer experience.
dogfooding/lib/screens/call_screen.dart (1)
57-73: Well-implemented feature integration!The speaking-while-muted recognition is properly integrated with:
- Correct lifecycle management (init in initState, dispose in dispose)
- Safe context usage with mounted check
- Clear user feedback via SnackBar
packages/stream_video/test/src/audio_processing/speaking_while_muted_recognition_test.dart (1)
15-178: Comprehensive test coverage!The test suite thoroughly covers:
- State transitions (muting/unmuting)
- Edge cases (recognition already active)
- Disconnection scenarios
- Proper mock setup and verification
Well-structured and follows testing best practices.
packages/stream_video/lib/src/audio_processing/audio_recognition_webrtc.dart (1)
78-148: Well-implemented audio detection algorithm!The audio level detection logic is well-designed with:
- Smoothing via moving average
- Adaptive baseline noise level
- Proper debouncing for speech detection
- Clear separation of speech onset and offset detection
packages/stream_video/lib/src/audio_processing/speaking_while_muted_recognition.dart
Show resolved
Hide resolved
packages/stream_video/lib/src/audio_processing/speaking_while_muted_recognition.dart
Outdated
Show resolved
Hide resolved
packages/stream_video/lib/src/audio_processing/audio_recognition_webrtc.dart
Outdated
Show resolved
Hide resolved
packages/stream_video/lib/src/audio_processing/audio_recognition_webrtc.dart
Show resolved
Hide resolved
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #999 +/- ##
========================================
+ Coverage 4.75% 4.83% +0.07%
========================================
Files 574 577 +3
Lines 38692 38802 +110
========================================
+ Hits 1841 1875 +34
- Misses 36851 36927 +76 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
packages/stream_video/lib/src/audio_processing/audio_recognition_webrtc.dart
Outdated
Show resolved
Hide resolved
packages/stream_video/lib/src/audio_processing/speaking_while_muted_recognition.dart
Outdated
Show resolved
Hide resolved
Co-authored-by: Maciej Brażewicz <[email protected]>
renefloor
force-pushed
the
feature/speech-recognition-while-muted
branch
from
d7f8c68 to
f65a932
Compare
…ition-while-muted # Conflicts: # packages/stream_video/lib/stream_video.dart # packages/stream_video/test/test_helpers.dart
…ition-while-muted # Conflicts: # packages/stream_video/CHANGELOG.md # packages/stream_video/test/test_helpers.dart
🎯 Goal
Add a notification in dogfooding when the user is talking while muted.
FLU-107
🛠 Implementation details
Added a public
SpeakingWhileMutedRecognitionand an internalAudioRecognitionWebRTCimplementation.The
SpeakingWhileMutedRecognitionstarts and stops the audio recognition when the user gets (un)muted.The
AudioRecognitionWebRTCmakes 2 peer connections to listen to the audio volume, heavily inspired by the react native implementation.It can be used by creating the object and listening to the state stream:
🎨 UI Changes
🧪 Testing
You can test by starting/joining a call, mute yourself and start talking.
☑️Contributor Checklist
General
☑️Reviewer Checklist
Summary by CodeRabbit
Summary by CodeRabbit