low-latency-short-buffer-playback test unclear duration observation

[yanj-github]

All three best devices did not went into waiting status, and I don't have other recordings to check.
However, from reading the spec:

• Observe:
o If the video element transitions into playing state, cancel the timeout.
o If the timeout elapses before the video element has transitioned out of the waiting state, throw an error and terminate the test.
• In case the timeout resolves successfully: Append the rest of the CMAF chunks of CMAF track k.

Are we expecting to see "waiting" in playback and device would display a frame longer when "waiting" then transitions into playing state?
Also not clear what is meant by taking into account "The waiting_duration."?

1. The playback duration shall match the duration of the CMAF Track, i.e. TR[k,S] = TR[k,1] + td[k] taking into account:
   ○ Missing starting and ending frames.
   ○ Potential start frames being rendered before playback.
   ○ Frozen last frame after the playback ended.
   ○ The waiting_duration.

[gitwjr]

For what its worth, when running on a PC I often would often see QR codes flashing regularly, then one QR code would pause for a short period and then they would resume regular flashing. It often resulted in a duration problem or duration with missing frames if I remember correctly. I attributed it to software decode but it sounds similar to this and one or two other issues. I don't remember which tests it occurred on at this point since I tried a lot of different tests and combinations of tests.

[yanj-github]

Thanks for the comment @gitwjr, when I run this on a PC browser I did not see waiting (pause) and resume.
Can you kindly help to double check, if possible please?

Also I have looked at reported recordings for all 8 devices from Plugfest, no longer duration is detected, it seems none of them went into waiting (pause).

I suggest:

• If we can find an example of waiting (pause) and resume in playback, the duration should consider waiting_duration and delay in holding on last frame and starting frame when waiting.
• Otherwise if we cannot find example of waiting (pause), I suggest to remove "The waiting_duration." from the spec.

[gitwjr]

I checked my latest runs (4) from February 29th. These were using my Pixel 6a. The playbacks do not show an obvious pause. 2 failed for missing frames.
"test": "/cfhd_12.5_25_50-local/fullscreen-playback-of-switching-sets__ss1-2.html"
"status": "FAIL",
"message": " Mid frame number tolerance is 10. Following frames are missing in playout 7: 221 229 241 295 334 367 374 394 Following frames are missing in playout 6: 408 428 435 454 528 534 547 554 561 568 Playout 3 starting frame found is 2302, expected to start from 2301. Total of missing frames is 18.",
"name": "[OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order."
"test": "/cfhd_12.5_25_50-local/fullscreen-playback-of-switching-sets__ss1-2.html"
and 2 for duration (72ms and 65 ms).
"status": "FAIL",
"message": " Mid frame number tolerance is 10. Following frames are missing in playout 7: 210 230 242 262 395 Following frames are missing in playout 6: 409 449 455 489 547 569 595 Playout 3 starting frame found is 2302, expected to start from 2301. Total of missing frames is 12.",
"name": "[OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order."

2 failed for duration
"test": "/cfhd_12.5_25_50-local/sequential-track-playback__t1.html"
"status": "FAIL",
"message": "Playback duration 29927.51ms does not match expected duration 30000.0ms +/- tolerance of 50ms. Detected duration is different by 72.49ms. Allowed tolerance is 50ms and duration frame tolerance is 0. Starting missing frame number is 0. Ending missing frame number is 0.",
"name": "[OF] Video: The playback duration shall match the duration of the CMAF Track"
"test": "/cfhd_12.5_25_50-local/sequential-track-playback__t1.html
"status": "FAIL",
"message": "Playback duration 29934.7ms does not match expected duration 30000.0ms +/- tolerance of 50ms. Detected duration is different by 65.3ms. Allowed tolerance is 50ms and duration frame tolerance is 0. Starting missing frame number is 0. Ending missing frame number is 0.",
"name": "[OF] Video: The playback duration shall match the duration of the CMAF Track"

I found several from January that show the pause but the OF was not recording the results back then so I don't have any matching results. These were run using the GoPro4.

[yanj-github]

Thanks @gitwjr. I was looking for test result specifically for this low-latency-short-buffer-playback test.

[gitwjr]

I found one test run that included low-latency-short-buffer-playback (using GoPro4 camera). Playback screen goes blank ~20 seconds into the playback for ~2 seconds then resumes. Although OF results were not recorded (remember that function was not working at the time) the results from TR were a pass for test workflow and video event ended fired. Test was run on 1/5/2024 using latest test suite as of that date.

[jpiesing]

1. I think there's a copy-paste error here, 8.22.4 and 8.22.5 refer to waiting_duration but 8.22.3 defines waiting_timeout. I think 8.22.3 should be changed to waiting_duration. 'waiting_timeout' is not used anywhere else in the document.
2. WIth respect to @yanj-github 's original problem description, I don't think the use of 'waiting' in 'waiting_duration' is related to the HTML video element receiving a 'waiting' event. I think the correct observation is actually under the wrong heading, "Stimulus". I think this should be moved.
3. There are 4 references to 'playing state' and some to 'waiting state' which refer to the figure in 8.22.1. These are not defined and this appears to cause confusion. I think definitions should be added. Here is an example.

In the playing state, the media timeline is advancing and media is being presented. In the waiting state, the media timeline is not advancing and media is not being presented except that, for video, if the start frame is rendered in advance of the media timeline advancing, this is part of the waiting state.

A proposal is implemented in Draft-CTA-5003-Ae-v2.05.

[yanj-github]

1. from https://github.com/cta-wave/dpctf-tests/blob/master/test-config.json "waiting_timeout" is used. If spec changes to use waiting_duration I think the configuration file should changes to match it.
   "low-latency-short-buffer-playback.html": {
   "maximum_forward_buffer": 500,
   "waiting_timeout": 500
   }
   @FritzHeiden is it easy to update the configuration to match the spec change?
2. Observe part in "Stimulus":
   o If the video element transitions into playing state, cancel the timeout.
   o If the timeout elapses before the video element has transitioned out of the waiting state, throw an error and terminate the test.
   Is this observation done is test runner?
3. Thanks for the clarification. Does this mean for the duration check we don't need to consider the "waiting_duration", but just compare detected duration with the duration of the CMAF Track?
   e.g:
   The playback duration shall match the duration of the CMAF Track, i.e. TR[k,S] = TR[k,1] + td[k] taking into account:
   ○ Missing starting and ending frames.
   ○ Potential start frames being rendered before playback.
   ○ Frozen last frame after the playback ended.

A proposal is implemented in Draft-CTA-5003-Ae-v2.05.

Where can I find this draft please? The latest one from https://standards.cta.tech/wg/dpctf/document/33886, I cannot find the changes.

[jpiesing]

A proposal is implemented in Draft-CTA-5003-Ae-v2.05.

Where can I find this draft please? The latest one from https://standards.cta.tech/wg/dpctf/document/33886, I cannot find the changes.

https://standards.cta.tech/wg/dpctf/document/33886?downloadRevision=36523

CTA Causeway is missing one check-box compared to DVB and HbbTV for normal users, the ability to declare that a newly uploaded revision is the active one. The main download button always gives you the active version. To download a non-active version, you need to click on the date / time.