How dose qsv-copy works?

[Hlongyu]

I have two gpu and one video. The video can only decoder by intel grahpics. And my render gpu is nvidia gpu. So i use mpv to play this video with hwdec=auto. It will use qsv-copy to decode it. I want to know technical details.

According to my understanding, this implementation should involve using D3D11 to decode the video and obtain video frames, then using CopySubresourceRegion to copy them to a staging texture, followed by Map to load them into memory, and finally uploading them to an NVIDIA texture.

However, when I implemented this logic myself, GPU-CPU synchronization during copy and map operations caused time-consuming delays. But MPV can play smoothly, and I couldn’t find the relevant technique in its source code. Can anyone help explain this?

[kasper93]

qsv-copy

Any -copy variant is downloading the decoded frame to RAM and proceeds normally as if it comes from swdec.

For the detail, do the data is copied, look at ffmpeg source, for example for qsv-copy it would call qsv_transfer_data_from. For any other API, look at other hwcontext_*.c they all implement HWContextType and transfer_data_to, transfer_data_from functions, if possible.

[Hlongyu]

However, ffmpeg d3d11va translate_data is same as my demo, but mpv use d3d11va-copy can render 16k video with 60fps, which confuse me

[Hlongyu]

Is code here?
Mpv add frame to queue, use av_hwframe_transfer_data until queue full.

Update:
find here

[Hlongyu]

I still can't resolve this issue; I'm not very familiar with GPU and CPU synchronization. Could you provide more guidance on pipeline usage? @kasper93

            AVFrame* ff = av_frame_alloc();
            ret = avcodec_receive_frame(m_decoder, ff);
            if (ret == AVERROR(EAGAIN)) {
                break;
            } else if (ret == AVERROR_EOF) {
                avcodec_flush_buffers(m_decoder);
                break;
            } else if (ret < 0) {
                throw;
            }
            test_frames.push(ff);
            if (test_frames.size() <= 5) {
                continue;
            AVFrame* test_frame = av_frame_alloc();
            auto time1 = xs_gettime_ms();
            av_hwframe_transfer_data(test_frame, test_frames.front(), 0);
            auto time2 = xs_gettime_ms();
            av_frame_free(&test_frames.front());
            qDebug() << "ffmpeg translate time" << time2 - time1;
            av_frame_free(&test_frame);
            test_frames.pop();

ffmpeg translate time 100
fmpeg translate time 130
ffmpeg translate time 119

[Hlongyu]

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "by-gdb",
            "request": "launch",
            "name": "Launch(gdb)",
            "program": ".\\mpv.exe",
            "programArgs": "--hwdec=qsv-copy D:\\ytdl\\16k_1.mkv --log-file=mpv.log",
            "cwd": "C:\\msys64\\home\\uid20843\\mpv-build\\mpv\\build"
        }
    ]
}

int64_t time1 = mp_time_ns();
int res = av_hwframe_transfer_data(dstav, srcav, 0);
int64_t time2 = mp_time_ns();

FILE* file = fopen("mp_image_hw_download.log", "a+");
fprintf(file, "%f\n", (time2-time1)/1000000.0);
fclose(file);

$ cat mpv/build/mp_image_hw_download.log
135.786500
29.555400
25.099200
17.645400
7.633800
32.803100
6.768100
6.398700
6.874500
6.537000
8.273900
8.455300
8.101000
9.284900
8.640000
7.156200
8.245500
9.573300
8.333700
7.674700

Same Video In OBS

uint64_t time1 = os_gettime_ns();
int err = av_hwframe_transfer_data(d->sw_frame, d->hw_frame, 0);
uint64_t time2 = os_gettime_ns();
blog(LOG_INFO, "transfer time is %d", (time2 - time1) / 1000000);

transfer time is 22
transfer time is 18
transfer time is 18
transfer time is 19
transfer time is 20
transfer time is 23
transfer time is 24
transfer time is 15

[Hlongyu]

I originally thought mpv was implemented like this

time	ffmpeg queue	gpu copy frame queue	cpu map frame queue
1	a b c d e
2	b c d e	a ('a' start copy，not finish)
3	c d e	a b
4	d e	a b c
5	e	b c d ('a' copy end)	a
6		c d e	a b
7		d e	a b c
8		e	a b c d
9			a b c d e

but it seems it was actually implemented like this，

time	ffmpeg queue	gpu copy frame queue	cpu map frame queue
1	a b c d e
2	b c d e	a (start copy)	a (map immediately, wait 'a' copy finish)
3	c d e	b	a b
4	d e	c	a b c
5	e	d	a b c d
6		e	a b c d e