Publishing AV1 over RTSP from FFmpeg does not work (Clock rate not found)

[CounterPillow]

Which version are you using?

git master (e74c458)

Which operating system are you using?

• Linux amd64 standard
• Linux amd64 Docker
• Linux arm64 standard
• Linux arm64 Docker
• Linux arm7 standard
• Linux arm7 Docker
• Linux arm6 standard
• Linux arm6 Docker
• Windows amd64 standard
• Windows amd64 Docker (WSL backend)
• macOS amd64 standard
• macOS amd64 Docker
• Other (please describe)

Describe the issue

Publishing AV1 from FFmpeg over RTSP results in a "400 - Bad Request":

2023/08/28 22:52:21 INF [RTSP] [conn [::1]:33192] closed (invalid SDP: media 1 is invalid: clock rate not found)

Describe how to replicate the issue

1. start the server
2. publish with ffmpeg -stream_loop -1 -re -i Videos/Wanderers.mov -pix_fmt yuv420p -c:v libsvtav1 -preset 8 -b:v 2000k -an -f rtsp rtsp://localhost:8554/pillowstrim

Did you attach the server logs?

yes

2023/08/28 22:56:23 INF MediaMTX v1.0.0-32-ge74c458
2023/08/28 22:56:23 DEB path manager created
2023/08/28 22:56:23 INF [RTSP] listener opened on :8554 (TCP), :18000 (UDP/RTP), :18001 (UDP/RTCP)
2023/08/28 22:56:23 INF [RTSPS] listener opened on :8322 (TCP)
2023/08/28 22:56:23 INF [RTMP] listener opened on :1935
2023/08/28 22:56:23 INF [HLS] listener opened on :18888
2023/08/28 22:56:23 INF [WebRTC] listener opened on :18889 (HTTP), :10808 (ICE/UDP)
2023/08/28 22:56:28 INF [RTSP] [conn [::1]:32810] opened
2023/08/28 22:56:28 DEB [RTSP] [conn [::1]:32810] [c->s] OPTIONS rtsp://localhost:8554/pillowstrim RTSP/1.0
CSeq: 1
User-Agent: Lavf60.3.100


2023/08/28 22:56:28 DEB [RTSP] [conn [::1]:32810] [s->c] RTSP/1.0 200 OK
CSeq: 1
Public: DESCRIBE, ANNOUNCE, SETUP, PLAY, RECORD, PAUSE, GET_PARAMETER, TEARDOWN
Server: gortsplib


2023/08/28 22:56:28 DEB [RTSP] [conn [::1]:32810] [c->s] ANNOUNCE rtsp://localhost:8554/pillowstrim RTSP/1.0
CSeq: 2
Content-Length: 141
Content-Type: application/sdp
User-Agent: Lavf60.3.100

v=0
o=- 0 0 IN IP6 ::1
s=No Name
c=IN IP6 ::1
t=0 0
a=tool:libavformat 60.3.100
m=video 0 RTP/AVP 96
b=AS:2000
a=control:streamid=0

2023/08/28 22:56:28 INF [RTSP] [session 6a6c0a15] created by [::1]:32810
2023/08/28 22:56:28 DEB [RTSP] [conn [::1]:32810] [s->c] RTSP/1.0 400 Bad Request
CSeq: 2
Server: gortsplib


2023/08/28 22:56:28 INF [RTSP] [session 6a6c0a15] destroyed (not in use)
2023/08/28 22:56:28 INF [RTSP] [conn [::1]:32810] closed (invalid SDP: media 1 is invalid: clock rate not found)

Did you attach a network dump?

no

Other info

FFmpeg output:

ffmpeg version n6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.1.1 (GCC) 20230429
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Videos/Wanderers.mov':
  Metadata:
    creation_time   : 2015-05-03T14:31:35.000000Z
  Duration: 00:03:50.00, start: 0.000000, bitrate: 152873 kb/s
  Stream #0:0[0x1](eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(bt709, progressive), 1920x816, 150566 kb/s, SAR 1:1 DAR 40:17, 25 fps, 25 tbr, 25 tbn (default)
    Metadata:
      creation_time   : 2015-05-03T14:31:35.000000Z
      handler_name    : Apple Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : Apple ProRes 422 HQ
      timecode        : 01:00:00:00
  Stream #0:1[0x2](eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, stereo, s32 (24 bit), 2304 kb/s (default)
    Metadata:
      creation_time   : 2015-05-03T14:31:35.000000Z
      handler_name    : Apple Sound Media Handler
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2015-05-03T14:32:40.000000Z
      handler_name    : Time Code Media Handler
      timecode        : 01:00:00:00
Stream mapping:
  Stream #0:0 -> #0:0 (prores (native) -> av1 (libsvtav1))
Press [q] to stop, [?] for help
Svt[info]: -------------------------------------------
Svt[info]: SVT [version]:       SVT-AV1 Encoder Lib v1.7.0-dirty
Svt[info]: SVT [build]  :       GCC 13.2.1 20230801      64 bit
Svt[info]: LIB Build date: Aug 25 2023 13:48:03
Svt[info]: -------------------------------------------
Svt[info]: Number of logical cores available: 32
Svt[info]: Number of PPCS 110
Svt[info]: [asm level on system : up to avx512]
Svt[info]: [asm level selected : up to avx2]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profile   tier (auto)     level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator               : 1920 / 816 / 25 / 1
Svt[info]: SVT [config]: bit-depth / color format                                       : 8 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct                                    : 8 / PSNR / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type                      : 161 / 16 / key frame
Svt[info]: SVT [config]: BRC mode / target bitrate (kbps)                               : VBR / 2000 
Svt[info]: -------------------------------------------
SvtMalloc[info]: SVT Memory Usage:
SvtMalloc[info]:     total allocated memory:       1.55 GB
SvtMalloc[info]:         malloced memory:          510.69 MB
SvtMalloc[info]:         callocated memory:        72.69 MB
SvtMalloc[info]:         allocated aligned memory: 1006.86 MB
SvtMalloc[info]:     mutex count: 20087
SvtMalloc[info]:     semaphore count: 1660
SvtMalloc[info]:     thread count: 378
SvtMalloc[info]:     hash table fulless: 0.094346, hash bucket is healthy
SvtMalloc[info]: top 10 malloced memory locations:
SvtMalloc[info]: (91.58 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Common/Codec/EbBitstreamUnit.c:36
SvtMalloc[info]: (77.36 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbCodingUnit.c:76
SvtMalloc[info]: (64.49 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:116
SvtMalloc[info]: (56.50 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:1181
SvtMalloc[info]: (42.86 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:1376
SvtMalloc[info]: (41.92 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:117
SvtMalloc[info]: (23.81 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:1379
SvtMalloc[info]: (16.76 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:983
SvtMalloc[info]: (13.39 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbModeDecisionProcess.c:232
SvtMalloc[info]: (13.01 MB): /usr/src/debug/svt-av1/SVT-AV1/Source/Lib/Encoder/Codec/EbPictureControlSet.c:488
[out#0/rtsp @ 0x5639b4d22180] Could not write header (incorrect codec parameters ?): Server returned 400 Bad Request
[vost#0:0/libsvtav1 @ 0x5639b4d99040] Error initializing output stream: 
SvtMalloc[info]: you have no memory leak
Conversion failed!

[aler9]

Hello, FFmpeg doesn't support publishing AV1 with RTSP yet. Open an issue in the FFmpeg bug tracker.

[RobotnickIsrael]

Hello
I tried publishing to rtsp with gstreamer and i can't make it work. I have gst 1.22 with the rs av1 plugin. (Latest Mediamtx )
Ive created few av1 files with ffmpeg with different params, like short gop, fast decode, yuv420p.
Then, I send to media mtx using:
Gst-launch-1.0 filesrc location=test.mp4 ! qtdemux ! av1parse ! rtspclientsink location=rtsp://127.0.0.1:8554/test

It connects well, preroll,playing and so on.
Mediamtx logs also ok, saying AV1 stream is published to my path.

When I try to connect to the rtsp address using latest vlc, latest ffplay or gstreamer, I can't get a picture.
I wonder what am I doing wrong?
The only thing I can think of is that the encoded file isn't good for streaming, do you maybe have an example for a working ffmpeg/gstreamer line to create a streamable av1 file? I tried many, all with same results.
I'm also sending the stream to chrome using webrtcbin, which gets the av1 stream and packets, but show nothing.

Any help will be very appreciated
P.S: sdp shows AV1 as expected

[aler9]

@RobotnickIsrael here's a tested GStreamer pipeline that shows how to encode a stream in AV1 and send it with RTSP to the server:

gst-launch-1.0 videotestsrc ! video/x-raw,width=1920,height=1080,format=I420 ! svtav1enc target-bitrate=3000 max-bitrate=3000 preset=8 ! av1parse ! rtspclientsink location=rtspclientsink location=rtsp://localhost:8554/stream

[RobotnickIsrael]

Hi @aler9
I dont have svtav1enc element (Gst 1.22 with rs-plugin), and it doesnt exist on gstreamer website.
But I do have a new ffmpeg with svtav1enc, so I created two files from a short h264 file that I have:
ffmpeg -i h264.mp4 -c:v libsvtav1 -preset 8 -pix_fmt yuv420p -filter:v fps=30 -g 90 -svtav1-params fast-decode=1:tune=1:tbr=3000:mbr=3000:color-format=1 -an av1test.mp4
ffmpeg -i h264.mp4 -c:v libsvtav1 -preset 8 -pix_fmt yuv420p -filter:v fps=30 -g 90 -svtav1-params tbr=3000:mbr=3000:color-format=1 -an av1test2.mp4
As you can see, in both tries I use the same svtav1encoder and same parameters as you do.

Then I take these files and send them to mediamtx:
gst-launch-1.0 uridecodebin uri=http://...av1test.mp4 caps="video/x-av1" ! av1parse ! rtspclientsink location="rtsp://...av1test

gstreamer gives an error after about 10 seconds of flowing pipeline:
rtspclintsink0: Could not read from resource - 400 Bad Request.

When I try to connect to the rtsp server on that path I get nothing, with both original files (tried with ffmpeg+vlc+gstreamer).
If I try to watch the files directly (not the rtsp path but http path) it works well.

mediamtx logs:
[RTSP] conn ... opened
[RTSP] session ... created by ...
[RTSP] session ... is publishing to path áv1test', 1 track (AV1)
After few seconds:
[RTSP] conn ... closed: must be in state [initial], while in state preRecord
[RTSP] session ... destroyed: not in use

Now, If I manage to connect with ffplay before the session was destroyed (why would it stop publish if there are no clients?
Not supposed to), mediamtx first says it publishes to av1test, than that someone is reading from that path.
But ffplay numbers stay on 0, and after a few seconds everything break:
ffplay:
Could not find parameters for stream 0 (Video: none, none): unknown codec
And with it the gstreamer that publishing breaks as well.

Any ideas whats going on?

*I try to pre-encode the file so it will be easier to stream it, no need to encode av1 on runtime.
**gst caps on av1parse - looks good to me
video/x-av1, stream-format=(string)obu-stream, alignment=(strung)tu, codec_date=(buffer)..., width=(int)1280, height=(int)720, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1,chroma-format=(string)4:2:0, but-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)main

[aler9]

@RobotnickIsrael i tested the commands you provided and in my case everything worked perfectly.

First i encoded 40 seconds of a test stream with FFmpeg:

ffmpeg -re -f lavfi -i testsrc=size=1920x1080:rate=30 -pix_fmt yuv420p -c:v libsvtav1 -preset 8 -pix_fmt yuv420p -filter:v fps=30 -g 90 -svtav1-params fast-decode=1:tune=1:tbr=3000:mbr=3000:color-format=1 -an -t 00:40:00 -y av1test.mp4

Then i used GStreamer to parse the file and send its content to the server through RTSP:

gst-launch-1.0 filesrc location=av1test.mp4 ! qtdemux ! av1parse ! rtspclientsink location=rtsp://localhost:8554/stream

result:

2024/04/25 13:48:44 INF [RTSP] [conn [::1]:44124] opened
2024/04/25 13:48:44 INF [RTSP] [session ba53aa64] created by [::1]:44124
2024/04/25 13:48:45 INF [RTSP] [session ba53aa64] is publishing to path 'stream', 1 track (AV1)
2024/04/25 13:48:55 INF [WebRTC] [session 8071b80b] created by [::1]:42080
2024/04/25 13:48:55 INF [WebRTC] [session 8071b80b] peer connection established, local candidate: host/udp/10.42.0.1/8189, remote candidate: host/udp/10.42.0.1/55678
2024/04/25 13:48:55 INF [WebRTC] [session 8071b80b] is reading from path 'stream', 1 track (AV1)

Make sure that the versions of FFmpeg and GStreamer you're using are both up to date. I used GStreamer 1.24.2, built from source:

root@alen:/gst-plugins-rs# gst-launch-1.0 --version
gst-launch-1.0 version 1.24.2
GStreamer 1.24.2
Unknown package origin

and FFmpeg 5.1, built from source:

ale@alen:~/dev/builder-gstreamer$ ffmpeg
ffmpeg version 5.1.git Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)

IMHO ready-to-use packages of GStreamer and FFmpeg cannot be trusted since AV1 is too cutting edge.

If you still have problems, this is an issue that is not related in any way to the server, so you can move the discussion into the community / mailing list of GStreamer or FFmpeg.

[github-actions]

This issue is being locked automatically because it has been closed for more than 6 months.
Please open a new issue in case you encounter a similar problem.