Repo: Organise and document processes and architecture (#43)

* Add Makefile for common contributor tasks

Narrows down a few commands to automatically format when building, and neatly expose testing/benching.

Empty files added to describe contributor guidelines, overall architecture.

* First draft of contributor guidelines

* Simple architecture diagrams

* Add PNG variants of architecture diagrams

Swapping to these because not having Fira Sans installed on a viewing machine leads to terrible kerning.

* Architecture description.

* MD cross-refs.

ARCHITECTURE.md

@@ -0,0 +1,92 @@
+# Summary
+Songbird defines two main systems:
+* The **gateway**, which communicates with Discord through another client library. This sends voice state updates to join a voice channel, and correlates responses into voice connection info.
+* The **driver**, which uses voice connection info to establish an RTP connection and WS signalling channel to send and receive audio. It then manages audio mixing, audio source management, event tracking, and voice packet reception.
+
+Songbird allows users to use one or both of these systems as needed.
+Discord voice connections ultimately require both of these to be handled in some way.
+In many setups for instance, this comes through using a client-specific wrapper in a bot's host language to collect connection information to send to Lavalink/Lavaplayer, hosted on the JVM.
+
+# Gateway
+Songbird's **gateway** is an async system, typically managed by a top-level `Songbird` struct, which holds `Arc` pointers to a Discord client library instance (that it can request individual shard references from).
+This maps all `ChannelID`s into `Call` state.
+New `Call`s are created as needed, requesting the shard that each `ChannelID` belongs to from the main client.
+
+When asked to join a voice channel, a `Call` communicates with Discord over the shard handle, collates Discord's responses, and produces a `ConnectionInfo` with session information.
+If the driver feature is enabled, then every `Call` is/has an associated `Driver`, and this connection info is passed on to its inner tasks.
+
+![](images/gateway.png)
+
+```
+src/manager.rs
+src/handler.rs
+src/serenity.rs
+```
+
+# Driver
+Songbird's **driver** is a mixed sync/async system for running voice connections.
+Audio processing remains synchronous for the following reasons:
+* Encryption, encoding, and mixing are compute bound tasks which cannot be subdivided cleanly by the Tokio executor. Having these block the scheduler's finite thread count has a significant impact on servicing other tasks.
+* `Read` and `Seek` are considerably more user-friendly to use, implement, and integrate than `AsyncRead`, `AsyncBufRead`, and `AsyncSeek`.
+
+## Tasks
+Songbird subdivides voice connection handling into several long- and short-lived tasks.
+
+* **Core**: Handles and directs commands received from the driver. Responsible for connection/reconnection, and creates network tasks.
+* **Mixer**: Combines audio sources together, Opus encodes the result, and encrypts the built packets every 20ms. Responsible for handling track commands/state. ***Synchronous***.
+* **Disposer**: Used by mixer thread to dispose of data with potentially long/blocking `Drop` implementations (i.e., audio sources). ***Synchronous***.
+* **Events**: Stores and runs event handlers, tracks event timing, and handles 
+* **Websocket**: *Network task.* Sends speaking status updates and keepalives to Discord, and receives client (dis)connect events.
+* **UDP Tx**: *Network task.* Responsible for transmitting completed voice packets.
+* **UDP Rx**: *Network task.* Decrypts/decodes received voice packets and statistics information.
+
+*Note: all tasks are able to message the permanent tasks via a block of interconnecting channels.*
+
+![](images/driver.png)
+
+```
+src/driver/*
+```
+
+## Audio handling
+
+### Input
+Inputs are raw audio sources: composed of a `Reader` (which can be `Read`-only or `Read + Seek`), a framing mechanism, and a codec.
+Several wrappers exist to add `Seek` capabilities to one-way streams via storage or explicitly recreating the struct.
+
+Framing is not always needed (`Raw`), but makes it possible to consume the correct number of bytes needed to decode one audio packet (and/or simplify skipping through the stream).
+Currently, Opus and raw (`i16`/`f32`) audio sources are supported, though only the DCA framing for Opus is implemented.
+At present, the use of the FFmpeg executable allows us to receive raw input, but at heavy memory cost.
+Further implementations are possible in the present framework (e.g., WebM/MKV and Ogg containers, MP3 and linked FFI FFmpeg as codecs).
+
+Internally, the mixer uses floating-point audio to prevent clipping and allow more granular volume control.
+If a source is known to use the Opus codec (and is the only source), then it can bypass mixing altogether.
+
+```
+src/input/*
+```
+
+### Tracks
+Tracks hold additional state which is expected to change over the lifetime of  a track: position, play state, and modifiers like volume.
+Tracks (and their handles) also allow per-source events to be inserted.
+
+Tracks are defined in user code, where they are fully modifiable, before being passed into the driver.
+From this point, all changes and requests are serviced via commands over a `TrackHandle` (so that the audio thread never locks or blocks during user modification).
+
+Tracks and Inputs typically exist in a 1:1 relationship, though many Inputs may reference the same backing store.
+
+```
+src/tracks/*
+```
+
+## Events
+Event handlers are stored on a per-track and global basis, with events being supplied by other tasks in the driver.
+These event handlers are boxed trait objects, each subscribed to an individual event type.
+The event type and data are supplied when this generic handler is called, allowing reuse of event handlers between subscriptions (i.e., via `Arc`).
+
+Timed events are driven by "tick" messages sent by the mixer (so that both tasks' view of track state remains in sync), while other event types are set individually (but often fired in batches).
+Global events fire in response to other tasks, or the main "tick".
+
+```
+src/events/*
+```

CONTRIBUTING.md

@@ -0,0 +1,80 @@
+# Issues
+Thanks for raising a bug or feature request!
+If you're raising a bug or issue, please use the following template:
+
+```md
+**Songbird version:** (version)
+
+**Rust version (`rustc -V`):** (version)
+
+**Serenity/Twilight version:** (version)
+
+**Output of `ffmpeg -version`, `youtube-dl --version` (if relevant):**
+...
+
+**Description:**
+...
+
+**Steps to reproduce:**
+...
+```
+
+Additionally, tag your issue at the same time you create it.
+
+If you're requesting a feature, explain how it will be useful to users or improve the library.
+If you want to implement it yourself, please include a rough explanation of *how* you'll be going about writing it.
+
+# Pull Requests
+Thanks for considering adding new features or fixing bugs in Songbird!
+You might find it helpful to look over [our architecture document] in this repository to understand roughly how components interact at a high level.
+Generally, we ask that PRs have a description that answers the following, under headers or in prose:
+
+* The type of change being made.
+* A high-level description of the changes.
+* Steps taken to test the new feature/fix.
+
+Your PR should also readily compile, pass all tests, and undergo automated formatting *before* it is opened.
+The simplest way to check that your PR is ready is to install [cargo make], and run the following command:
+```sh
+cargo make ready
+```
+
+Merged PRs will be squashed into the repository under a single headline: try to tag your PR correctly, and title it with a single short sentence in the imperative mood to make your work easier to merge.
+*"Driver: Fix missing track state events"* is a good example: it explains what code was modified, the problem that was solved, and would place the description of *how* the problem was solved in the commit/PR body.
+
+If you're adding new features or utilities, please open an issue and/or speak with us on Discord to make sure that you aren't duplicating work, and are in line with the overall system architecture.
+
+At a high level, focus on making new features as clean and usable as possible.
+This extends to directing users away from footguns and functions with surprising effects, at the API level or by documentation.
+Changes that affect or invalidate large areas of the library API will make a lot of users' lives that much harder when new breaking releases are published, so need deeper justification.
+Focus on making sure that new feature additions are general to as many use-cases as possible: for instance, adding some queue-specific state to every audio track forces other users to pay for that additional overhead even when they aren't using this feature.
+
+## Breaking changes
+Breaking changes (in API or API semantics) must be made to target the `"next"` branch.
+Commits here will be released in the next breaking semantic version (i.e., 0.1.7 -> 0.2.0, 1.3.2 -> 2.0.0).
+
+Bugfixes and new features which do not break semantic versioning should target `"current"`.
+Patches will be folded into more incremental patch updates (i.e., 1.3.2 -> 1.3.3) while new features will trigger minor updates (i.e., 1.3.2 -> 1.4.0).
+
+## Documentation and naming
+Doc-comments, comments, and item names should be written in British English where possible.
+All items (`structs`, `enums`, `fn`s, etc.) must be documented in full sentences; these are user-facing, and other developers will naturally rely on them to write correct code and understand what the library can(not) do for them.
+Error conditions, reasons to prefer one method over another, and potential use risks should be explained to help library users write the best code they can.
+
+Code comments should be written similarly – this requirement is not as stringent, but focus on clarity and conciseness.
+Try to focus on explaining *why/what* more confusing code exists/does, rather than *how* it performs that task, to try to prevent comments from aging as the codebase evolves.
+
+## Testing
+Pull requests must not break existing tests, examples, or common feature subsets.
+Where possible, new features should include new tests (particularly around event or input handling).
+
+These steps are included in `cargo make ready`.
+
+## Linting
+Songbird's linting pipeline requires that you have nightly Rust installed.
+Your code must be formatted using `cargo +nightly fmt --all`, and must not add any more Clippy warnings than the base repository already has (as extra lints are added to Clippy over time).
+
+These commands are included in `cargo make ready`.
+
+[cargo make]: https://github.com/sagiegurari/cargo-make
+[our architecture document]: ARCHITECTURE.md

Makefile.toml

@@ -0,0 +1,51 @@
+[tasks.format]
+toolchain = "nightly"
+install_crate = { crate_name = "rustfmt-nightly", rustup_component_name = "rustfmt-preview", binary = "rustfmt", test_arg = "--help" }
+command = "cargo"
+args = ["fmt", "--all"]
+
+[tasks.build]
+args = ["build", "--features", "full-doc"]
+dependencies = ["format"]
+
+[tasks.build-examples]
+args = ["build", "--manifest-path", "./examples/Cargo.toml", "--workspace"]
+command = "cargo"
+dependencies = ["format"]
+
+[tasks.build-gateway]
+args = ["build", "--features", "serenity-rustls"]
+command = "cargo"
+dependencies = ["format"]
+
+[tasks.build-driver]
+args = ["build", "--features", "driver,rustls"]
+command = "cargo"
+dependencies = ["format"]
+
+[tasks.build-old-tokio]
+command = "cargo"
+args = ["build", "--features", "serenity-rustls-tokio-02,driver-tokio-02"]
+dependencies = ["format"]
+
+[tasks.build-variants]
+dependencies = ["build", "build-gateway", "build-driver", "build-old-tokio"]
+
+[tasks.clippy]
+args = ["clippy", "--features", "full-doc", "--", "-D", "warnings"]
+dependencies = ["format"]
+
+[tasks.test]
+args = ["test", "--features", "full-doc"]
+
+[tasks.bench]
+description = "Runs performance benchmarks."
+category = "Test"
+command = "cargo"
+args = ["bench", "--features", "internals,full-doc"]
+
+[tasks.doc]
+args = ["doc", "--features", "full-doc"]
+
+[tasks.ready]
+dependencies = ["format", "test", "build-variants", "build-examples", "doc", "clippy"]

README.md

@@ -20,7 +20,6 @@ The library offers:
 Songbird's gateway functionality requires you to specify the `GUILD_VOICE_STATES` intent.
 
 ## Dependencies
-
 Songbird needs a few system dependencies before you can use it.
 
 - Opus - Audio codec that Discord uses.
@@ -45,15 +44,18 @@ This is an optional dependency. It allows Songbird to download an audio source f
 ## Examples
 Full examples showing various types of functionality and integrations can be found in [this crate's examples directory].
 
-## Attribution
+## Contributing
+If you want to help out or file an issue, please look over [our contributor guidelines]!
 
+## Attribution
 Songbird's logo is based upon the copyright-free image ["Black-Capped Chickadee"] by George Gorgas White.
 
 [serenity]: https://github.com/serenity-rs/serenity
 [twilight]: https://github.com/twilight-rs/twilight
 ["Black-Capped Chickadee"]: https://www.oldbookillustrations.com/illustrations/black-capped-chickadee/
 [lavalink]: https://github.com/Frederikam/Lavalink
 [this crate's examples directory]: https://github.com/serenity-rs/songbird/tree/current/examples
+[our contributor guidelines]: CONTRIBUTING.md
 
 [build badge]: https://img.shields.io/github/workflow/status/serenity-rs/songbird/CI?style=flat-square
 [build]: https://github.com/serenity-rs/songbird/actions