Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Driver: Use reorder/jitter buffer for audio decoding #146

Closed
FelixMcFelix opened this issue Aug 8, 2022 · 0 comments
Closed

Driver: Use reorder/jitter buffer for audio decoding #146

FelixMcFelix opened this issue Aug 8, 2022 · 0 comments
Labels
breaking Will either cause existing code to fail to compile, or cause substantial behaviour changes driver Relates to the driver or one of its sub-tasks. events Relates to driver event handling/generation. receive Related to voice receive, decoding, decryption, or events

Comments

@FelixMcFelix
Copy link
Member

For users to correctly handle packet reordering or jitter due to network instability, they currently need to disable our decoding features and reimplement decoding logic themselves -- this gets more complex when handling audio data from several users, which is interspersed and can arrive at different parts of each 20ms window. This makes downmixing particular nasty.

Ideally, we should take this complexity on ourselves. Doing so will require a few changes:

  • Packet events must be separated from decode events. We likely want to write and store audio data into an Arc<[u8]> to minimise copying between these.
  • Decode events must be generated every 20ms, and include audio from every active participant (accessible by SSRC).
  • Each user needs their own resizable jitter/reorder buffer (likely a deque of Arcs). This should have user-configurable target and max depths. If the buffer was allowed to empty (e.g., user disconnects or stops speaking), then it must build back up to its target occupancy before we decode packets again.
  • Packet events should no longer include audio data -- this is to be moved to the decode event.
@FelixMcFelix FelixMcFelix added driver Relates to the driver or one of its sub-tasks. events Relates to driver event handling/generation. breaking Will either cause existing code to fail to compile, or cause substantial behaviour changes receive Related to voice receive, decoding, decryption, or events labels Aug 8, 2022
@FelixMcFelix FelixMcFelix added this to the v0.4.0: Nightingale milestone Aug 8, 2022
FelixMcFelix added a commit that referenced this issue Jan 9, 2023
This PR Introduces a new `VoiceTick` event which collects and reorders all RTP packets to smooth over network instability, as well as to synchronise user audio streams. Raw packet events have been moved to `RtpPacket`, while `SpeakingUpdate`s have been removed as they can be easily computed using the `silent`/`speaking` audio maps included in each event.

Closes #146.
FelixMcFelix added a commit to FelixMcFelix/songbird that referenced this issue Nov 20, 2023
This PR Introduces a new `VoiceTick` event which collects and reorders all RTP packets to smooth over network instability, as well as to synchronise user audio streams. Raw packet events have been moved to `RtpPacket`, while `SpeakingUpdate`s have been removed as they can be easily computed using the `silent`/`speaking` audio maps included in each event.

Closes serenity-rs#146.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Will either cause existing code to fail to compile, or cause substantial behaviour changes driver Relates to the driver or one of its sub-tasks. events Relates to driver event handling/generation. receive Related to voice receive, decoding, decryption, or events
Projects
None yet
Development

No branches or pull requests

1 participant