Support serenity simd_json #105

vicky5124 · 2021-12-06T23:31:41Z

modified:   src/shards.rs

…erenity-rs#6) * Depend on Serenity's `next` branch on Songbird's `next` branch * Update the examples to use the `next` branch

Serenity's cache design has changed, this (finally) prevents the examples from failing to compile on CI.

Adds generics to any `Id` types on `Call`. Also includes the overlooked `Songbird::get_or_insert`. Closes serenity-rs#94. Tested using `cargo make ready`.

This matches a recent serenity change where `ClientBuilder` no longer has an explicit lifetime parameter. This was tested using `cargo make ready`.

modified: src/shards.rs

modified: src/handler.rs

modified: src/handler.rs modified: src/shards.rs

FelixMcFelix

Thanks for the PR, just one minor nit regarding import formatting. I had a quick look at the repo and it seems that voice-model is untouched, so I think this is the extent of everything you need to change. :)

src/shards.rs

vicky5124 · 2021-12-07T16:34:48Z

sorry! should have marked this as a draft, as there's still way more files to modify, and things that need to be fixed on serenity first before this can work and be merged.
See serenity-rs/serenity#1613

About the import order, cargo-fmt made it that way.

modified: ../../../Cargo.toml modified: ../../../src/driver/mod.rs modified: ../../../src/input/dca.rs modified: ../../../src/input/error.rs modified: ../../../src/input/ffmpeg_src.rs modified: ../../../src/input/metadata.rs modified: ../../../src/input/ytdl_src.rs modified: ../../../src/ws.rs

modified: Cargo.toml modified: src/driver/mod.rs modified: src/input/dca.rs modified: src/input/error.rs modified: src/input/ffmpeg_src.rs modified: src/input/metadata.rs modified: src/input/ytdl_src.rs modified: src/ws.rs

modified: src/driver/mod.rs modified: src/input/dca.rs modified: src/input/ffmpeg_src.rs modified: src/input/metadata.rs modified: src/input/ytdl_src.rs modified: src/tracks/error.rs modified: src/ws.rs

modified: Cargo.toml modified: examples/serenity/voice_receive/Cargo.toml modified: src/input/metadata.rs modified: src/ws.rs

modified: examples/serenity/voice_receive/Cargo.toml

modified: README.md

modified: .github/workflows/ci.yml

new file: .cargo/config.toml

FelixMcFelix

Currently relies on serenity's simdjson support being fixed before I can end-to-end test this.

Otherwise compiles fine, comments are inline with respect to all driver-related JSON parsing.

FelixMcFelix · 2021-12-13T14:30:40Z

.cargo/config.toml

@@ -0,0 +1,2 @@
+[build]
+rustflags = ["-C", "target-cpu=haswell"]


Why not native?

Can this also be moved to CI such that it's only invoked for the simd-json build task? I'm unsure if the spurious MacOS SIGILL originates from this.

No, unless you do CI with Arm, this target-cpu will work just fine.
Haswell was chosen over native as I've seen a few people mention it's less likely for it to fail CI due to "Illegal Instruction"; MacOS failed because there's still a chance it can fail from that, serenity has had this issue for quite a while now, and it's on Actions for it to be fixed.

Moving this to only the SIMD check would make it more likely to pass, and I'll do it when I have time for it, but it's hard to test CI when it's blocked behind it being accepted due to this being my first PR.

FelixMcFelix · 2021-12-13T14:42:19Z

src/input/dca.rs

+        serenity::json::prelude::from_slice::<DcaMetadata>(raw_json.as_mut_slice())
+            .map_err(DcaError::InvalidMetadata)?
+            .into();
+    #[cfg(not(feature = "serenity"))]


General feedback for all other driver/input uses: wouldn't it make more sense to use the simd-json library ourselves if that feature is selected? That way we can apply this to twilight too, and use serenity::json::* only where we interface with serenity in the gateway portions. Twilight v0.8 now exposes their own JSON type via twilight_gateway::shard::raw_message::Message from what I can see.

I somewhat planned for this, but I have 0 idea how twilight manages SIMD; I will not make a PR of something I'm completely unfamiliar with. If someone else wants to implement the SIMD feature for twilight, go for it.

vicky5124 · 2021-12-25T16:48:08Z

Serenity has fixed it's SIMD issues, this PR is ready to be tested.

FelixMcFelix · 2022-01-08T17:10:03Z

Tried again there, still getting plenty of messages like below after a cargo update:

Jan 08 17:07:25.284  WARN run:recv_event:handle_event{event=Err(SimdJson(Error { index: 0, character: '💩', error: Serde("event GuildCreate: Serde(\"Serde(\\\"Serde(\\\\\\\"invalid type: string \\\\\\\\\\\\\\\"925690819955023872\\\\\\\\\\\\\\\", expected tuple struct MessageId\\\\\\\") at character 0 ('💩')\\\") at character 0 ('💩')\") at character 0 ('💩')") }))}: serenity::gateway::shard: [Shard [0, 1]] Unhandled error: SimdJson(Error { index: 0, character: '�'', error: Serde("event GuildCreate: Serde(\"Serde(\\\"Serde(\\\\\\\"invalid type: string \\\\\\\\\\\\\\\"925690819955023872\\\\\\\\\\\\\\\", expected tuple struct MessageId\\\\\\\") at character 0 ('💩')\\\") at character 0 ('💩')\") at character 0 ('�'')") })

nickelc · 2022-07-05T09:29:02Z

Tried again there, still getting plenty of messages like below after a cargo update:

Jan 08 17:07:25.284  WARN run:recv_event:handle_event{event=Err(SimdJson(Error { index: 0, character: '💩', error: Serde("event GuildCreate: Serde(\"Serde(\\\"Serde(\\\\\\\"invalid type: string \\\\\\\\\\\\\\\"925690819955023872\\\\\\\\\\\\\\\", expected tuple struct MessageId\\\\\\\") at character 0 ('💩')\\\") at character 0 ('💩')\") at character 0 ('💩')") }))}: serenity::gateway::shard: [Shard [0, 1]] Unhandled error: SimdJson(Error { index: 0, character: '�'', error: Serde("event GuildCreate: Serde(\"Serde(\\\"Serde(\\\\\\\"invalid type: string \\\\\\\\\\\\\\\"925690819955023872\\\\\\\\\\\\\\\", expected tuple struct MessageId\\\\\\\") at character 0 ('💩')\\\") at character 0 ('💩')\") at character 0 ('�'')") })

I think that was an issue in simd-json and should be fixed now.

FelixMcFelix · 2022-07-05T10:37:14Z

I'll manually fixup and test after #89 is merged (i.e., after a cleanup pass). Apologies that this has been left for quite so long.

FelixMcFelix · 2022-07-25T12:50:11Z

Okay, I've fixed this up to work without serenity now (and I believe JSON decode in DCA format and YoutubeDL handling should be accelerated too). This might take some fiddling to make sure the CI plays nice -- I've tried to make sure that RUSTFLAGS are only ever set on the Linux/simd target (since MacOS breaks routinely).

This PR adds support for the simd-json library whenever decoding or encoding JSON responses. This may be enabled independently of serenity and twilight support for SIMD acceleration. Co-authored-by: Kyle Simpson <[email protected]>

arqunis and others added 8 commits October 12, 2021 14:43

Deps: Depend on Serenity's next branch on Songbird's next branch (s…

ae69e5e

…erenity-rs#6) * Depend on Serenity's `next` branch on Songbird's `next` branch * Update the examples to use the `next` branch

Deps: Bump streamcatcher version -> 1.0 (serenity-rs#93)

3784490

Examples: Fix serenity-next cache accesses (serenity-rs#99)

94bd290

Serenity's cache design has changed, this (finally) prevents the examples from failing to compile on CI.

Gateway: Add generics to Call methods. (serenity-rs#102)

2129422

Adds generics to any `Id` types on `Call`. Also includes the overlooked `Songbird::get_or_insert`. Closes serenity-rs#94. Tested using `cargo make ready`.

Gateway: Remove lifetime from Serenity setup trait (serenity-rs#103)

7956792

This matches a recent serenity change where `ClientBuilder` no longer has an explicit lifetime parameter. This was tested using `cargo make ready`.

Support serenity simd_json

89de7a1

modified: src/shards.rs

Missed a place

c16c17c

modified: src/handler.rs

fmt

45fb51b

modified: src/handler.rs modified: src/shards.rs

FelixMcFelix requested changes Dec 7, 2021

View reviewed changes

src/shards.rs Outdated Show resolved Hide resolved

vicky5124 marked this pull request as draft December 7, 2021 16:33

vicky5124 added 7 commits December 7, 2021 18:25

Fully replace serde_json with serenity::json

3c83316

modified: Cargo.toml modified: src/driver/mod.rs modified: src/input/dca.rs modified: src/input/error.rs modified: src/input/ffmpeg_src.rs modified: src/input/metadata.rs modified: src/input/ytdl_src.rs modified: src/ws.rs

fmt

ec7d029

modified: src/driver/mod.rs modified: src/input/dca.rs modified: src/input/ffmpeg_src.rs modified: src/input/metadata.rs modified: src/input/ytdl_src.rs modified: src/tracks/error.rs modified: src/ws.rs

Updated serenity to next from path

ab64728

modified: Cargo.toml modified: examples/serenity/voice_receive/Cargo.toml modified: src/input/metadata.rs modified: src/ws.rs

Remove simdjson features

c749f2a

modified: examples/serenity/voice_receive/Cargo.toml

document simdjson feature

86feb99

modified: README.md

Add simdjson to tests

889cbed

modified: .github/workflows/ci.yml

vicky5124 marked this pull request as ready for review December 7, 2021 18:28

vicky5124 requested a review from FelixMcFelix December 7, 2021 18:28

vicky5124 added 2 commits December 7, 2021 20:37

Add serenity's simdjson to test

8ca77e1

modified: .github/workflows/ci.yml

Fix SIMD compilation

4b7e9e9

new file: .cargo/config.toml

FelixMcFelix requested changes Dec 13, 2021

View reviewed changes

FelixMcFelix force-pushed the next branch from 7956792 to 28e370a Compare January 5, 2022 11:14

FelixMcFelix force-pushed the next branch 2 times, most recently from 37a15ec to e0ab49c Compare February 14, 2022 15:06

FelixMcFelix force-pushed the next branch from 49c3346 to 0d66467 Compare July 22, 2022 21:20

FelixMcFelix added 3 commits July 24, 2022 22:46

Merge remote-tracking branch 'upstream/next' into simd_json

91d4c18

Better rustflag handling, and use our own simdjson (not just serenity)

1fe3edf

Migrate metadata handling, all simd_json uses to crate::json

f28fc85

Attempt 2 at CI.

2d85f38

FelixMcFelix merged commit 2774d3e into serenity-rs:next Jul 25, 2022

FelixMcFelix added enhancement New feature or request driver Relates to the driver or one of its sub-tasks. gateway Relates to the gateway feature (Discord's gateway integration). labels Jul 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support serenity simd_json #105

Support serenity simd_json #105

vicky5124 commented Dec 6, 2021

FelixMcFelix left a comment

vicky5124 commented Dec 7, 2021

FelixMcFelix left a comment

FelixMcFelix Dec 13, 2021

vicky5124 Dec 13, 2021

FelixMcFelix Dec 13, 2021 •

edited

Loading

vicky5124 Dec 13, 2021

vicky5124 commented Dec 25, 2021

FelixMcFelix commented Jan 8, 2022

nickelc commented Jul 5, 2022

FelixMcFelix commented Jul 5, 2022

FelixMcFelix commented Jul 25, 2022

		@@ -0,0 +1,2 @@
		[build]
		rustflags = ["-C", "target-cpu=haswell"]

Support serenity simd_json #105

Support serenity simd_json #105

Conversation

vicky5124 commented Dec 6, 2021

FelixMcFelix left a comment

Choose a reason for hiding this comment

vicky5124 commented Dec 7, 2021

FelixMcFelix left a comment

Choose a reason for hiding this comment

FelixMcFelix Dec 13, 2021

Choose a reason for hiding this comment

vicky5124 Dec 13, 2021

Choose a reason for hiding this comment

FelixMcFelix Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

vicky5124 Dec 13, 2021

Choose a reason for hiding this comment

vicky5124 commented Dec 25, 2021

FelixMcFelix commented Jan 8, 2022

nickelc commented Jul 5, 2022

FelixMcFelix commented Jul 5, 2022

FelixMcFelix commented Jul 25, 2022

FelixMcFelix Dec 13, 2021 •

edited

Loading