8cc7a22b0b
This extensive PR rewrites the internal mixing logic of the driver to use symphonia for parsing and decoding audio data, and rubato to resample audio. Existing logic to decode DCA and Opus formats/data have been reworked as plugins for symphonia. The main benefit is that we no longer need to keep yt-dlp and ffmpeg processes alive, saving a lot of memory and CPU: all decoding can be done in Rust! In exchange, we now need to do a lot of the HTTP handling and resumption ourselves, but this is still a huge net positive. `Input`s have been completely reworked such that all default (non-cached) sources are lazy by default, and are no longer covered by a special-case `Restartable`. These now span a gamut from a `Compose` (lazy), to a live source, to a fully `Parsed` source. As mixing is still sync, this includes adapters for `AsyncRead`/`AsyncSeek`, and HTTP streams. `Track`s have been reworked so that they only contain initialisation state for each track. `TrackHandles` are only created once a `Track`/`Input` has been handed over to the driver, replacing `create_player` and related functions. `TrackHandle::action` now acts on a `View` of (im)mutable state, and can request seeks/readying via `Action`. Per-track event handling has also been improved -- we can now determine and propagate the reason behind individual track errors due to the new backend. Some `TrackHandle` commands (seek etc.) benefit from this, and now use internal callbacks to signal completion. Due to associated PRs on felixmcfelix/songbird from avid testers, this includes general clippy tweaks, API additions, and other repo-wide cleanup. Thanks go out to the below co-authors. Co-authored-by: Gnome! <45660393+GnomedDev@users.noreply.github.com> Co-authored-by: Alakh <36898190+alakhpc@users.noreply.github.com>
95 lines
6.0 KiB
Markdown
95 lines
6.0 KiB
Markdown
# Summary
|
|
Songbird defines two main systems:
|
|
* The **gateway**, which communicates with Discord through another client library. This sends voice state updates to join a voice channel, and correlates responses into voice connection info.
|
|
* The **driver**, which uses voice connection info to establish an RTP connection and WS signalling channel to send and receive audio. It then manages audio mixing, audio source management, event tracking, and voice packet reception.
|
|
|
|
Songbird allows users to use one or both of these systems as needed.
|
|
Discord voice connections ultimately require both of these to be handled in some way.
|
|
In many setups for instance, this comes through using a client-specific wrapper in a bot's host language to collect connection information to send to Lavalink/Lavaplayer, hosted on the JVM.
|
|
|
|
# Gateway
|
|
Songbird's **gateway** is an async system, typically managed by a top-level `Songbird` struct, which holds `Arc` pointers to a Discord client library instance (that it can request individual shard references from).
|
|
This maps all `ChannelID`s into `Call` state.
|
|
New `Call`s are created as needed, requesting the shard that each `ChannelID` belongs to from the main client.
|
|
|
|
When asked to join a voice channel, a `Call` communicates with Discord over the shard handle, collates Discord's responses, and produces a `ConnectionInfo` with session information.
|
|
If the driver feature is enabled, then every `Call` is/has an associated `Driver`, and this connection info is passed on to its inner tasks.
|
|
|
|
|
|
|
|
```
|
|
src/manager.rs
|
|
src/handler.rs
|
|
src/serenity.rs
|
|
src/join.rs
|
|
```
|
|
|
|
# Driver
|
|
Songbird's **driver** is a mixed sync/async system for running voice connections.
|
|
Audio processing remains synchronous for the following reasons:
|
|
* Encryption, encoding, and mixing are compute bound tasks which cannot be subdivided cleanly by the Tokio executor. Having these block the scheduler's finite thread count has a significant impact on servicing other tasks.
|
|
* `Read` and `Seek` are considerably more user-friendly to use, implement, and integrate than `AsyncRead`, `AsyncBufRead`, and `AsyncSeek`.
|
|
* Symphonia implements all of its functionality based on synchronous I/O.
|
|
|
|
## Tasks
|
|
Songbird subdivides voice connection handling into several long- and short-lived tasks.
|
|
|
|
* **Core**: Handles and directs commands received from the driver. Responsible for connection/reconnection, and creates network tasks.
|
|
* **Mixer**: Combines audio sources together, Opus encodes the result, and encrypts the built packets every 20ms. Responsible for handling track commands/state. ***Synchronous***.
|
|
* **Thread Pool**: A dynamically sized thread-pool for I/O tasks. Creates lazy tracks using `Compose` if sync creation is needed, otherwise spawns a tokio task. Seek operations always go to the thread pool. ***Synchronous***.
|
|
* **Disposer**: Used by mixer thread to dispose of data with potentially long/blocking `Drop` implementations (i.e., audio sources). ***Synchronous***.
|
|
* **Events**: Stores and runs event handlers, tracks event timing, and handles
|
|
* **Websocket**: *Network task.* Sends speaking status updates and keepalives to Discord, and receives client (dis)connect events.
|
|
* **UDP Tx**: *Network task.* Responsible for transmitting completed voice packets.
|
|
* **UDP Rx**: *Network task.* Decrypts/decodes received voice packets and statistics information.
|
|
|
|
*Note: all tasks are able to message the permanent tasks via a block of interconnecting channels.*
|
|
|
|
|
|
|
|
```
|
|
src/driver/*
|
|
```
|
|
|
|
## Audio handling
|
|
|
|
### Input
|
|
Inputs are audio sources supporting lazy initialisation, being either:
|
|
* **lazy inputs**—a trait object which allows an instructions to create an audio source to be cheaply stored. This will be initialised when needed either synchronously or asynchronously based on what which methods the trait object supports.
|
|
* **live inputs**—a usable audio object implementing `MediaSource: Read + Seek`. `Seek` support may be dummied in, as seek use and support is gated by `MediaSource`. These can be passed in at various stages of processing by symphonia.
|
|
|
|
Several wrappers exist to add `Seek` capabilities to one-way streams via storage or explicitly recreating the struct, `AsyncRead` adapters, and raw audio input adapters.
|
|
|
|
Internally, the mixer uses floating-point audio to prevent clipping and allow more granular volume control.
|
|
Symphonia is used to demux and decode input files in a variety of formats into this floating-point buffer: songbird supports all codecs and containers which are part of the symphonia project, while adding support for Opus decoding and DCA1 container files.
|
|
If a source uses the Opus codec (and is the only source), then it can bypass mixing and re-encoding altogether, saving CPU cycles per server.
|
|
|
|
```
|
|
src/input/*
|
|
```
|
|
|
|
### Tracks
|
|
Tracks hold additional state which is expected to change over the lifetime of a track: position, play state, and modifiers like volume.
|
|
Tracks (and their handles) also allow per-source events to be inserted.
|
|
|
|
Tracks are defined in user code, where they are fully modifiable, before being passed into the driver.
|
|
From this point, all changes and requests are serviced via commands over a `TrackHandle` (so that the audio thread never locks or blocks during user modification).
|
|
|
|
Tracks and Inputs typically exist in a 1:1 relationship, though many Inputs may reference the same backing store.
|
|
|
|
```
|
|
src/tracks/*
|
|
```
|
|
|
|
## Events
|
|
Event handlers are stored on a per-track and global basis, with events being supplied by other tasks in the driver.
|
|
These event handlers are boxed trait objects, each subscribed to an individual event type.
|
|
The event type and data are supplied when this generic handler is called, allowing reuse of event handlers between subscriptions (i.e., via `Arc`).
|
|
|
|
Timed events are driven by "tick" messages sent by the mixer (so that both tasks' view of track state remains in sync), while other event types are set individually (but often fired in batches).
|
|
Global events fire in response to other tasks, or the main "tick".
|
|
|
|
```
|
|
src/events/*
|
|
```
|