servius/songbird
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Driver/Input: Migrate audio backend to Symphonia (#89)

Browse Source 
This extensive PR rewrites the internal mixing logic of the driver to use symphonia for parsing and decoding audio data, and rubato to resample audio. Existing logic to decode DCA and Opus formats/data have been reworked as plugins for symphonia. The main benefit is that we no longer need to keep yt-dlp and ffmpeg processes alive, saving a lot of memory and CPU: all decoding can be done in Rust! In exchange, we now need to do a lot of the HTTP handling and resumption ourselves, but this is still a huge net positive.

`Input`s have been completely reworked such that all default (non-cached) sources are lazy by default, and are no longer covered by a special-case `Restartable`. These now span a gamut from a `Compose` (lazy), to a live source, to a fully `Parsed` source. As mixing is still sync, this includes adapters for `AsyncRead`/`AsyncSeek`, and HTTP streams.

`Track`s have been reworked so that they only contain initialisation state for each track. `TrackHandles` are only created once a `Track`/`Input` has been handed over to the driver, replacing `create_player` and related functions. `TrackHandle::action` now acts on a `View` of (im)mutable state, and can request seeks/readying via `Action`.

Per-track event handling has also been improved -- we can now determine and propagate the reason behind individual track errors due to the new backend. Some `TrackHandle` commands (seek etc.) benefit from this, and now use internal callbacks to signal completion.

Due to associated PRs on felixmcfelix/songbird from avid testers, this includes general clippy tweaks, API additions, and other repo-wide cleanup. Thanks go out to the below co-authors.

Co-authored-by: Gnome! <45660393+GnomedDev@users.noreply.github.com>
Co-authored-by: Alakh <36898190+alakhpc@users.noreply.github.com>
This commit is contained in:
Kyle Simpson
2022-07-23 23:29:02 +01:00
parent 6c6ffa7ca8
commit 8cc7a22b0b
136 changed files with 9761 additions and 4891 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
src/events/context/data/disconnect.rs
View File

@@ -84,21 +84,19 @@ pub enum DisconnectReason {
impl From<&ConnectionError> for DisconnectReason {
fn from(e: &ConnectionError) -> Self {
use ConnectionError::*;
match e {
AttemptDiscarded => Self::AttemptDiscarded,
CryptoModeInvalid
| CryptoModeUnavailable
| EndpointUrl
| ExpectedHandshake
| IllegalDiscoveryResponse
| IllegalIp
| Json(_) => Self::ProtocolViolation,
Io(_) => Self::Io,
Crypto(_) | InterconnectFailure(_) => Self::Internal,
Ws(ws) => ws.into(),
TimedOut => Self::TimedOut,
ConnectionError::AttemptDiscarded => Self::AttemptDiscarded,
ConnectionError::CryptoModeInvalid
| ConnectionError::CryptoModeUnavailable
| ConnectionError::EndpointUrl
| ConnectionError::ExpectedHandshake
| ConnectionError::IllegalDiscoveryResponse
| ConnectionError::IllegalIp
| ConnectionError::Json(_) => Self::ProtocolViolation,
ConnectionError::Io(_) => Self::Io,
ConnectionError::Crypto(_) | ConnectionError::InterconnectFailure(_) => Self::Internal,
ConnectionError::Ws(ws) => ws.into(),
ConnectionError::TimedOut => Self::TimedOut,
}
}
}

39
src/events/context/mod.rs
View File

@@ -62,17 +62,17 @@ pub enum CoreContext {
impl<'a> CoreContext {
pub(crate) fn to_user_context(&'a self) -> EventContext<'a> {
use CoreContext::*;
match self {
SpeakingStateUpdate(evt) => EventContext::SpeakingStateUpdate(*evt),
SpeakingUpdate(evt) => EventContext::SpeakingUpdate(SpeakingUpdateData::from(evt)),
VoicePacket(evt) => EventContext::VoicePacket(VoiceData::from(evt)),
RtcpPacket(evt) => EventContext::RtcpPacket(RtcpData::from(evt)),
ClientDisconnect(evt) => EventContext::ClientDisconnect(*evt),
DriverConnect(evt) => EventContext::DriverConnect(ConnectData::from(evt)),
DriverReconnect(evt) => EventContext::DriverReconnect(ConnectData::from(evt)),
DriverDisconnect(evt) => EventContext::DriverDisconnect(DisconnectData::from(evt)),
Self::SpeakingStateUpdate(evt) => EventContext::SpeakingStateUpdate(*evt),
Self::SpeakingUpdate(evt) =>
EventContext::SpeakingUpdate(SpeakingUpdateData::from(evt)),
Self::VoicePacket(evt) => EventContext::VoicePacket(VoiceData::from(evt)),
Self::RtcpPacket(evt) => EventContext::RtcpPacket(RtcpData::from(evt)),
Self::ClientDisconnect(evt) => EventContext::ClientDisconnect(*evt),
Self::DriverConnect(evt) => EventContext::DriverConnect(ConnectData::from(evt)),
Self::DriverReconnect(evt) => EventContext::DriverReconnect(ConnectData::from(evt)),
Self::DriverDisconnect(evt) =>
EventContext::DriverDisconnect(DisconnectData::from(evt)),
}
}
}
@@ -80,18 +80,17 @@ impl<'a> CoreContext {
impl EventContext<'_> {
/// Retreive the event class for an event (i.e., when matching)
/// an event against the registered listeners.
#[must_use]
pub fn to_core_event(&self) -> Option<CoreEvent> {
use EventContext::*;
match self {
SpeakingStateUpdate(_) => Some(CoreEvent::SpeakingStateUpdate),
SpeakingUpdate(_) => Some(CoreEvent::SpeakingUpdate),
VoicePacket(_) => Some(CoreEvent::VoicePacket),
RtcpPacket(_) => Some(CoreEvent::RtcpPacket),
ClientDisconnect(_) => Some(CoreEvent::ClientDisconnect),
DriverConnect(_) => Some(CoreEvent::DriverConnect),
DriverReconnect(_) => Some(CoreEvent::DriverReconnect),
DriverDisconnect(_) => Some(CoreEvent::DriverDisconnect),
Self::SpeakingStateUpdate(_) => Some(CoreEvent::SpeakingStateUpdate),
Self::SpeakingUpdate(_) => Some(CoreEvent::SpeakingUpdate),
Self::VoicePacket(_) => Some(CoreEvent::VoicePacket),
Self::RtcpPacket(_) => Some(CoreEvent::RtcpPacket),
Self::ClientDisconnect(_) => Some(CoreEvent::ClientDisconnect),
Self::DriverConnect(_) => Some(CoreEvent::DriverConnect),
Self::DriverReconnect(_) => Some(CoreEvent::DriverReconnect),
Self::DriverDisconnect(_) => Some(CoreEvent::DriverDisconnect),
_ => None,
}
}

2
src/events/core.rs
View File

@@ -9,7 +9,7 @@
/// when a client leaves the session ([`ClientDisconnect`]), voice packets ([`VoicePacket`]), and
/// telemetry data ([`RtcpPacket`]). The format of voice packets is described by [`VoiceData`].
///
/// To detect when a user connects, you must correlate gateway (e.g., VoiceStateUpdate) events
/// To detect when a user connects, you must correlate gateway (e.g., `VoiceStateUpdate`) events
/// from the main part of your bot.
///
/// To obtain a user's SSRC, you must use [`SpeakingStateUpdate`] events.

44
src/events/store.rs
View File

@@ -1,7 +1,7 @@
use super::*;
use crate::{
constants::*,
tracks::{PlayMode, TrackHandle, TrackState},
tracks::{ReadyState, TrackHandle, TrackState},
};
use std::{
collections::{BinaryHeap, HashMap},
@@ -24,8 +24,9 @@ pub struct EventStore {
impl EventStore {
/// Creates a new event store to be used globally.
#[must_use]
pub fn new() -> Self {
Default::default()
Self::default()
}
/// Creates a new event store to be used within a [`Track`].
@@ -34,6 +35,7 @@ impl EventStore {
/// a track has been registered.
///
/// [`Track`]: crate::tracks::Track
#[must_use]
pub fn new_local() -> Self {
EventStore {
local_only: true,
@@ -53,21 +55,20 @@ impl EventStore {
return;
}
use Event::*;
match evt.event {
Core(c) => {
Event::Core(c) => {
self.untimed
.entry(c.into())
.or_insert_with(Vec::new)
.push(evt);
},
Track(t) => {
Event::Track(t) => {
self.untimed
.entry(t.into())
.or_insert_with(Vec::new)
.push(evt);
},
Delayed(_) | Periodic(_, _) => {
Event::Delayed(_) | Event::Periodic(_, _) => {
self.timed.push(evt);
},
_ => {
@@ -105,15 +106,12 @@ impl EventStore {
/// Processes all events due up to and including `now`.
pub(crate) fn timed_event_ready(&self, now: Duration) -> bool {
self.timed
.peek()
.map(|evt| {
evt.fire_time
.as_ref()
.expect("Timed event must have a fire_time.")
<= &now
})
.unwrap_or(false)
self.timed.peek().map_or(false, |evt| {
evt.fire_time
.as_ref()
.expect("Timed event must have a fire_time.")
<= &now
})
}
/// Processes all events attached to the given track event.
@@ -183,9 +181,9 @@ impl GlobalEvents {
pub(crate) async fn tick(
&mut self,
events: &mut Vec<EventStore>,
states: &mut Vec<TrackState>,
handles: &mut Vec<TrackHandle>,
events: &mut [EventStore],
states: &mut [TrackState],
handles: &mut [TrackHandle],
) {
// Global timed events
self.time += TIMESTEP_LENGTH;
@@ -199,7 +197,7 @@ impl GlobalEvents {
// Local timed events
for (i, state) in states.iter_mut().enumerate() {
if state.playing == PlayMode::Play {
if state.playing.is_playing() && state.ready == ReadyState::Playable {
state.step_frame();
let event_store = events
@@ -215,7 +213,7 @@ impl GlobalEvents {
}
}
for (evt, indices) in self.awaiting_tick.iter() {
for (evt, indices) in &self.awaiting_tick {
let untimed = (*evt).into();
if !indices.is_empty() {
@@ -223,7 +221,7 @@ impl GlobalEvents {
}
// Local untimed track events.
for &i in indices.iter() {
for &i in indices {
let event_store = events
.get_mut(i)
.expect("Missing store index for Tick (local untimed).");
@@ -261,12 +259,12 @@ impl GlobalEvents {
self.store
.process_untimed(self.time, untimed, EventContext::Track(&global_ctx[..]))
.await
.await;
}
}
// Now drain vecs.
for (_evt, indices) in self.awaiting_tick.iter_mut() {
for indices in self.awaiting_tick.values_mut() {
indices.clear();
}
}

12
src/events/track.rs
View File

@@ -1,3 +1,6 @@
// TODO: Could this be a bitset? Could accelerate lookups,
// allow easy joint subscription & remove Vecs for related evt handling?
/// Track events correspond to certain actions or changes
/// of state, such as a track finishing, looping, or being
/// manually stopped. Voice core events occur on receipt of
@@ -14,7 +17,8 @@ pub enum TrackEvent {
///
/// This event will not fire when a track first starts,
/// but will fire when a track changes from, e.g., paused to playing.
/// This is most relevant for queue users.
/// This is most relevant for queue users: queued tracks placed into a
/// non-empty queue are initlally paused, and are later moved to `Play`.
Play,
/// The attached track has been paused.
Pause,
@@ -22,4 +26,10 @@ pub enum TrackEvent {
End,
/// The attached track has looped.
Loop,
/// The attached track is being readied or recreated.
Preparing,
/// The attached track has become playable.
Playable,
/// The attached track has encountered a runtime or initialisation error.
Error,
}