low-latency-short-buffer-playback__t2.html failed on 7 of 8 TV sets in London plugfest

[jpiesing]

low-latency-short-buffer-playback__t2.html failed the "[OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order." observation on 7 of 8 TV sets in the London plugfest.

[jpiesing]

cta-wave/device-playback-task-force#126 applies here.

[yanj-github]

Re; cta-wave/device-playback-task-force#128 and updated spec r2.05:

• from https://github.com/cta-wave/dpctf-tests/blob/master/test-config.json "waiting_timeout" is used. If spec changes to use waiting_duration I think the configuration file should changes to match it.

"low-latency-short-buffer-playback.html": {
"maximum_forward_buffer": 500,
"waiting_timeout": 500
}
@FritzHeiden is it easy to update the configuration to match the spec change?

• Observation: The video element transitions into playing state before the waiting_duration, timeout elapses.
  Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please?
  If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

[FritzHeiden]

@FritzHeiden is it easy to update the configuration to match the spec change?

Yes it is. Should I go ahead and push this to master?

• Observation: The video element transitions into playing state before the waiting_duration, timeout elapses.
  Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please?
  If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

I can add an error event for this

[yanj-github]

@FritzHeiden is it easy to update the configuration to match the spec change?

Yes it is. Should I go ahead and push this to master?

It is up to you. If no regression concerns (it might be used HTML+JS of dpctf-tests) and easy to change please go ahead. If the change has a rick of breaking anything please raise it cta-wave/device-playback-task-force#128 it might be simpler to change the spec now. OF does not use this parameter so we are fine on OF end.

• Observation: The video element transitions into playing state before the waiting_duration, timeout elapses.
  Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please?
  If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

I can add an error event for this

Yes please if this can be done. So when OF detects the specific error event reports failure other wise the test will pass.

[jpiesing]

2024-05-14: We need to decide if this can be closed now and, if not, what needs to be done before it can be closed.

[jpiesing]

2024-05-14: Fraunhofer and Resillion will coordinate.

[yanj-github]

@FritzHeiden @louaybassbouss and @jpiesing re; #153 (comment):

1. Can we update "waiting_timeout" to "waiting_duration" in test-config.json please?
2. or update spec to replace "waiting_duration" with "waiting_timeout"?

Whichever is easier at this stage. All other tests uses "waiting_duration" so option 1 will make the parameter more consistent with others. OF does not read this parameter so no impact on OF end.

[yanj-github]

Also cta-wave/device-observation-framework#82 I have added issue on OF.
We are waiting for error event to be added.

[FritzHeiden]

1. Can we update "waiting_timeout" to "waiting_duration" in test-config.json please?

Done with #180

[jpiesing]

What does this mean for running the OF on recordings from the London plugfest? Will the release version of the OF stop supporting those recordings?

[FritzHeiden]

What does this mean for running the OF on recordings from the London plugfest? Will the release version of the OF stop supporting those recordings?

This parameter is only relevant for executing the test. As Yan already mentioned, the OF ignores this parameter, so this should have no impact on analyzing older recordings.

[yanj-github]

Thanks @FritzHeiden OF does not use this parameter, no OF changes required. and so no impact.

[jpiesing]

• Observation: The video element transitions into playing state before the waiting_duration, timeout elapses.
  Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please?
  If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

I can add an error event for this

@FritzHeiden Was the error event added?

[FritzHeiden]

@FritzHeiden Was the error event added?

This was already implemented as an error event, however, it did not help to understand why the test failed. So I removed it as an error and added a dedicated field in the report with #182

(last entry):