[feature] audio attachment support

[tsmethurst]

Currently we support images and video, but it would be great to also support audio, in line with other fedi implementations. Todo: define a list of audio file formats, look around for decoders, possibly support transcoding of things like flac and wav into mp3 (preferably v0/vbr).

[tsmethurst]

There's not a very rich field of audio decoding/encoding libraries for Go it seems, but some options seem to be:

• FLAC -- https://github.com/mewkiz/flac (not actively maintained)
• mp3 -- https://github.com/hajimehoshi/go-mp3 (not actively maintained)
• OGG (and various video formats too) -- https://github.com/yapingcat/gomedia
• WAV -- https://github.com/go-audio/wav

[WeebDataHoarder]

For some formats that are ancient (MP3) not active maintenance does not seem that much of an issue.

I saw similar lack of audio decoding/encoding within Go and built this library to handle formats in general https://git.gammaspectra.live/S.O.N.G/Kirika, also with a CGO_ENABLED=0 mode with only a subset supported (or no encoding).

Some external libraries libraries have been forked and improved upon lightly, and released standalone. These are listed on the README file.

</shill>

Format/codec wise, MP3 is supported across all devices and browsers out there, even very old ones, and also all patents are expired. Using it as a "fallback" could work. Also, if interested, I can port shine fixed-point mp3 encoder into Go, so no LAME C library is necessary, but the quality will be worse ofc.

FLAC is supported across all browsers but Internet Explorer (RIP) so that's also an option, but encoders to this are not that common. I have been working in a partial encoder support into FLAC within native Golang, but it's no better than WAV as it's using literal frames.

WAV support tends to be supported also on everything but IE, however, playback can be glitchy or out of sync due to container framing.

Opus is open, but support varies. Within Safari/iOS, Opus is only supported inside a CAF container instead of Ogg.

If you are interested in adding support or just general codecs within Go, CGO or purego, I joined the Matrix room, but I'm probably more reachable at DataHoarder nick in either libera.chat or Rizon IRC.

[tsmethurst]

We have found a way of embedding ffmpeg within the GoToSocial binary to support this, thanks to @NyaaaWhatsUpDoc's incredible work.

So the question now becomes less about "how are we going to do it", and more about "what exactly are we going to do".

In light of that, just posting some discussion from the dev channel here for safekeeping:

Mastodon API response for an MP3 upload (ie., what we should be aiming to mimic):

From kim:

as I have a feeling my metadata clearing probably removes the album cover
so I'd need to figure out args to skip specific metadata types for that

From daenney:

We shouldn't be wiping ID3 tags on audio I think: https://id3.org/id3v2.3.0#Declared_ID3v2_frames. Don't think any of that is privacy sensitive since it's metadata about the audio. Except maybe the ownership frame
Play counter I suppose
https://www.exiftool.org/TagNames/ID3.html has a nicer overview
I don't think anyone's implemented 2.4 much, 2.3 seems to be where it's at today

from f0x:

gps data for field recordings could actually be useful sometimes tho :')

[tsmethurst]

Support for mp3 and ogg has been added in #3090, and the frontend player is updated in #3099 so I think this can be closed now :) We may add support for FLAC in future but that will be a separate issue.