Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] support processing of (many) more media types #3090

Conversation

NyaaaWhatsUpDoc
Copy link
Member

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc commented Jul 9, 2024

Description

This replaces our media processing pipeline with the previous Go code we were using, to instead use embedded WebAssembly binaries of ffmpeg and ffprobe. As a result we can now easily support a multitude of extra image and video types, and adds audio file support too. This also allows us to ensure all media we processing (minus emojis) gets its metadata stripped, whereas previously it was only select types in select situations.

This also removes media file size enforcement out of the media manager itself, and instead to be handled by callers of the various media processing functions. (while still adding a check to the media manager for whether a "limit" was reached, for nicer error output). This makes the code in the media manager itself a little neater, and as part of this change minimizes the amount to which we wrap http response bodies, to ensure we can rely on Linux's sendfile syscall for optimal performance draining incoming media to /tmp. (as part of this change, i also got rid of the unnecessary separate image / video size config flags, which weren't actually being used, and replaced them with general media max local / remote media sizes).

Performance impact is yet to be measured, but given we're not doing any expensive transcodes it hopefully won't be too big.

Todo

  • update tests to pass given the big overhaul of media processing code
  • add tests for some of the new processable media filetypes
  • finish documenting the code, there's still some placeholder // ... comments

Checklist

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have not leveraged AI to create the proposed changes.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation.
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

"io"
"os"

"codeberg.org/gruf/go-mimetypes"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an embedded map of mimetypes by file extension: https://codeberg.org/gruf/go-mimetypes

@tsmethurst
Copy link
Contributor

Beautiful 😍

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc force-pushed the feature/switch-to-ffmpeg-media-processing branch from da299ba to a91dd7d Compare July 10, 2024 13:38
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc marked this pull request as ready for review July 11, 2024 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants