Test Content different codecs for Tests 8.15 and 8.16

[louaybassbouss]

DPCAT Tests 8.15 Source Buffer Re-Initialization (without changeType) and 8.16 Source Buffer Re-Initialization (with changeType) require content in different 2 codecs. For video, we have AVC so we need content in a different codec e.g HEVC and/or AV1. this related comment from @jpiesing related to this issue may help (copied from the google document):

For video, the obvious test content requirement is HEVC / AVC but any HEVC stream would do I think? For audio, the obvious one would be multi-channel to stereo or v.v.. Audio expert confirmation is required.

@FritzHeiden @dsilhavy FYI

[jpiesing]

Would an HEVC equivalent of stream T1 from AVC be good enough? 60s or 30s?

[louaybassbouss]

Would an HEVC equivalent of stream T1 from AVC be good enough?

yes

60s or 30s?

We are fine with both. For 30s content, we can play around 15s of content with codec1 and switch to codec2 for the remaining 30s 15s.

[rbouqueau]

Relates to cta-wave/dpctf-tests#85

[rbouqueau]

We need to review the encoding parameters and validate them: https://github.com/cta-wave/Test-Content-Generation/tree/master/Instructions

[rbouqueau]

Notes from today's call: AP@rbouqueau : @jpiesing suggests to define an equivalent to T1. @haudiobe suggests to validate it at the next DPTCF call next week.

[rbouqueau]

https://github.com/cta-wave/Test-Content-Generation/blob/master/Instructions/chh1.md

There is a table in the v1.39 document with an attempt to define the list of streams. Modifiers are:

• t1: basis (only in 25fps family?)
• t2: same as t1, no default flags / inband signalling (hev1 I guess)
• t3: same as t1, multiple chunks
• t4: same as t1, adding spatial/temporal pairs
• t5: same as t1, 60/1001 fractional frame rate family (?)

Do you feel like t1 to t4 cover all necessary options? For t5, do we want to cover different framerate families this way or generate each t1-t4 for each family (as the main script does) (note:the mezzanine source may be either TearsOfSteel or Croatia depending on the frame rate so mixing frame rate may not be trivial).

Compared to the existing matrix, we don't test the following options, please comment if you think they may be necessary:

• pic timing / vui timing
• different fragment durations
• different frame rate multipliers (e.g. 25 versus 12.5 and 50)

As a test I did add this folder with a t1 encoded in HEVC in all 3 fps families (including the cenc version, since I encoded using the main script). Just add 1,L1_1920x1080,TRUE,FALSE,hvc1,2,regular,duration,1920x1080,0.5,3500,30,chh1 to the Test-Content-Generation/switching_sets_single_track.csv file. This is exactly t1 from cfhd with a lower bitrate.

Can someone test this HEVC content?

[jpiesing]

There is a table in the v1.39 document with an attempt to define the list of streams.

I find the list of streams in 11.3.5 completely disconnected from the list of content options in 11.3.4. From the list of content options, I would have expected a table of comparable complexity to avc not the massively reduced one.

[jpiesing]

Looking further at 11.3.4, there are seem to be 5 content options that have more than one possible value or state.

1. Two options for default flags
   A: default_sample_flags, sample_flags and first_sample_flags neither set in the TrackFragmentHeaderBox nor TrackRunBox
   B: default_sample_flags, sample_flags and first_sample_flags set in the TrackFragmentHeaderBox and TrackRunBox
2. with (W) and without (W/O) picture timing SEI message
3. with (W) and without (W/O) with and without VUI timing information
4. Sample entry, see CMAF clause 9.4.1.2. Two options
   'hvc1' sample entry type (parameter sets within the CMAF Header)
   'hev1' sample entry type (parameter sets within the movie fragment header)
5. Fragments containing one or multiple moof/mdat pairs – two options
   A: Fragment is 1 chunk
   B: Fragment contains multiple chunks (p-frame to p-frame with b-frames)

There are also a number of options that can only have one value or where it's only proposed to test one value.

1. 7.5.17 Track Run Box ('trun') - v1 – combined with SAP type 1
2. 7.5.13 Edit List Box ('elst') - Absent
3. Initialization Constraints - Regular Switching Set, do not apply CMAF clause 7.3.4.2 and 9.2.11.4
4. CMAF Fragment durations - 2 seconds
5. B.2.4 - No SEI messages present.
6. B.3.3.2 - Single VPS.
7. B.3.3.3.2 - colour_description_flag set to 1.
8. B.3.3.3.2 - vui_time_scale and vui_num_units_in_tick set to constant.
9. B.5 - Settings according the requirements of the profile 'chh1' in Table B.1 of ISO/IEC 23000-19.

@Romain: Of the options that have more than one possibility, please can you say which is used for your test stream. Also please could you check that your test stream is compatible with the options where 11.3.4 proposes we only test one value.

[rbouqueau]

please could you check that your test stream is compatible with the options where 11.3.4 proposes we only test one value

I did. Eventually this is something we'll need to add to the WAVE verification script.

7.5.17 Track Run Box ('trun') - v1 – combined with SAP type 1

7.5.13 Edit List Box ('elst') - Absent
Initialization Constraints - Regular Switching Set, do not apply CMAF clause 7.3.4.2 and 9.2.11.4
CMAF Fragment durations - 2 seconds
B.3.3.2 - Single VPS.
B.3.3.3.2 - colour_description_flag set to 1.
B.5 - Settings according the requirements of the profile 'chh1' in Table B.1 of ISO/IEC 23000-19.

All ok.

B.2.4 - No SEI messages present.

Which kind of SEI? I guess from CMAF B.2.4 that we may be talking about payload types 137, 144 and 147 (related to color characteristics). In this sense it is ok.

B.3.3.3.2 - vui_time_scale and vui_num_units_in_tick set to constant.

While investigating these two are not ok (we should play with the encoder settings):

• NOK: vui_parameters_present_flag shall be set to 1
• NOK: is set to "hev1.2.4.L123.B0" or "hvc1.2.4.L123.B0" => current sample exposes hev1.2.4.L123.90

NB: I don't plan to fix the two last issues very soon unless told otherwise.

[rbouqueau]

Let me know when anyone was able to make a test, or if anything's missing to start testing.

[jpiesing]

@louaybassbouss ?

[louaybassbouss]

confirm content HEVC is working. Tested on Safari Mac. More tests on TV devices (HbbTV) planned.

[jpiesing]

While investigating these two are not ok (we should play with the encoder settings):

* NOK: vui_parameters_present_flag shall be set to 1

* NOK:  is set to "hev1.2.4.L123.B0"  or "hvc1.2.4.L123.B0" => current sample exposes hev1.2.4.L123.90

NB: I don't plan to fix the two last issues very soon unless told otherwise.

@nicholas-fr @haudiobe Please can you comment on how serious the two points above are?

[nicholas-fr]

I'm not sure about the first point, the second should probably be fixed as we're providing this as test content. However there are more serious issues that need to be fixed, please see points 3 and 4 below.
I checked the 2023-04-28 chh1 t1 content here:
http://dash-large-files.akamaized.net/WAVE/vectors/chh1_sets/15_30_60/t1/2023-04-28/t1.zip

1. I'm not sure about the first point - was this meant to refer to another VUI flag?
   NOK: vui_parameters_present_flag shall be set to 1
   This IS present and set (=1) in the bitstream, as confirmed with ffmpeg:
   [trace_headers @ 0000025a4e297840] 210 vui_parameters_present_flag 1 = 1

2. The second point relates to one of the constraints:
   general_non_packed_constraint_flag should be =1 but in the stream provided it is =0.
   Based on my understanding of the HEVC spec, this will not cause any decoding issues.
   As we are providing this as test content, I believe it should be corrected.

More importantly, the following need to be fixed:

3. The bitstream seems to be invalid according to ffmpeg:

[trace_headers @ 000001e0a9f88240] rbsp_stop_one_bit out of range: 0, but must be in [1,1].
[trace_headers @ 000001e0a9f88240] Failed to read unit 1 (type 33).
Error initializing bitstream filter: trace_headers

and the JCCP segment validator:

"\u2717 ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 2965 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"\u2717 ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 3219 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",

4. The CMAF brand is missing:

"expected": "chh1",
"detected": "isom,isom,cmf2,iso6,dash"

"\u2717 ### error: ftyp-1 \r###\t\t../src/ValidateAtomList.cpp : 1235 : ### error: ftyp-1 \r###\t\tThe brand 'isom' can only be a compatible, not major, brand",

5. The fragment duration is not 2s but 1.88s.

[rbouqueau]

The brand 'isom' can only be a compatible, not major, brand",

I've never understood where this conformance rule comes from.

1. I'm not sure about the first point - was this meant to refer to another VUI flag?

You are right.

general_non_packed_constraint_flag should be =1 but in the stream provided it is =0

It is 0, and I think (when reading the CMAF spec) that it should be 0. May I ask you to check again and let me know about the reference if I were mistaken?

The bitstream seems to be invalid according to ffmpeg

I couldn't reproduce. Could it come from the FFmpeg bitstream extraction?
@nicholas-fr Would you share your command-line with me?

The CMAF brand is missing

Fixed.

The fragment duration is not 2s but 1.88s.

There was indeed an issue on the first segment. I fixed that.

[nicholas-fr]

I'm not sure about the first point - was this meant to refer to another VUI flag?

You are right.

Would you indicate which VUI flag has an issue then?

general_non_packed_constraint_flag should be =1 but in the stream provided it is =0

It is 0, and I think (when reading the CMAF spec) that it should be 0. May I ask you to check again and let me know about the reference if I were mistaken?

• NOK: is set to "hev1.2.4.L123.B0" or "hvc1.2.4.L123.B0" => current sample exposes hev1.2.4.L123.90

The only difference is general_non_packed_constraint_flag, so if that is set correctly as defined by CMAF (seems it only matters for scalable HEVC), then would you mind clarifying what this issue is about?

The bitstream seems to be invalid according to ffmpeg

I couldn't reproduce. Could it come from the FFmpeg bitstream extraction?
@nicholas-fr Would you share your command-line with me?

The ffmpeg cli used is ffmpeg -i stream.mpd -c copy -bsf:v trace_headers -f null -
The JCCP segment validation seems to raise the same error:

"\u2717 ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 2965 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"\u2717 ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 3219 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",

[rbouqueau]

Thanks for the quick answer.

Would you indicate which VUI flag has an issue then?

@nicholas-fr I'm not sure to understand. I thought vui_parameters_present_flag was at 0 but it was not. All is good here then.

Note to self: I need to check as the gpac invoked command-line seems to involve removal:

-----------------------------------
Command run:
-----------------------------------
/opt/bin/gpac -i "content_files/releases/4/croatia_L1_1920x1080@25_30.mp4":#StartNumber=-2000000:#Representation=1:#IsoBrand=chh1:FID=GEN1  ffsws:osize=1920x1080:SID=GEN1 @ enc:gfloc:c=h265:b=3500k:bf=2:b_strategy=0:fintra=2:gop=2:profile=main:color_primaries=1:color_trc=1:colorspace=1::x265-params="level=41:no-open-gop=1:scenecut=0:nal-hrd=vbr:vbv-bufsize=10500:vbv-maxrate=5250": @ bsrw:novuitiming:FID=V0 --bs_switch=off  -o ./output/chh1_sets/12.5_25_50/t1/2023-05-12//stream.mpd:profile=live:cmaf=cmf2:segdur=2:tpl:template=\$Init=1/init\$\$Segment=1/\$:stl:cdur=2:SID=V0

The only difference is general_non_packed_constraint_flag, so if that is set correctly as defined by CMAF (seems it only matters for scalable HEVC), then would you mind clarifying what this issue is about?

@nicholas-fr I didn't get your remark was related to the codec description. Let me have another look.

The fragment duration is not 2s but 1.88s.
There was indeed an issue on the first segment. I fixed that.

My fix is not definitive though. Investigating for a better fix.

[nicholas-fr]

Thanks Romain.

NOK: is set to "hev1.2.4.L123.B0" or "hvc1.2.4.L123.B0" => current sample exposes hev1.2.4.L123.90

I understand now I think there was a typo in the original comment.
The chh1 CMAF profile requires Main10, but the stream is Main profile 8-bit with a codec string of hvc1.1.6.L123.90.
So this should be fixed so that the stream confirms with the chh1 CMAF profile.

[rbouqueau]

Ok, it seems like the codec description and the ffmpeg's trailing zero, and the vui removal issues may all be related.

[rbouqueau]

Ok so it seems the right codec desc is 'hvc1.2.4.L123.90'. 'hvc1.2.4.L123.BO' would imply 'non_packed_constraint_flag=1' where as CMAF mandates 0.

There is a new version under the 2023-05-12 subfolders of the content that should solve it all. I've also update the database.json.

[nicholas-fr]

Thanks @rbouqueau, I can confirm this resolves those issues.

There are remaining JCCP issues, but I believe all are incorrect (e.g. color primaries, transfer characteristics and matrix coeffs not detected correctly by JCCP).
I'm not sure about the first segment validation issue, I thought that would be resolved now.

"\u2717 ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 2965 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"\u2717 ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 3219 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"! ../src/ValidateAtoms.cpp : 2400 : Warning: In moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found \"btrt\": video sample descriptions would not normally contain this",
"! ../src/ValidateAtoms.cpp : 2400 : Warning: In moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found \"pasp\": video sample descriptions would not normally contain this",
"\u2713 ../src/ValidateAtoms.cpp : 3999 : colr atom of type nclx found, the software does not handle colr atoms of this type. ",
"! ../src/ValidateAtomList.cpp : 2339 : WARNING: In moov-1:udta-1 - unknown/unexpected atom 'meta'",
"\u2717 ### error:  \r###\t\t../src/PostprocessData.cpp : 1147 : ### error:  \r###\t\tAccording to DASH-IF IOP Section 3.2.8 @bandwidth of the Representation (3500000 bps) is set too low given the @minimumBufferTime (2 s), the minimum @bandwidth value required to conform is 4334868 bps.",
"\u2713 "

[rbouqueau]

@nicholas-fr What did you test exactly?

If I go to https://conformance.dashif.org/ and I test https://dash.akamaized.net/WAVE/vectors/chh1_sets/12.5_25_50/t1/2023-05-12/stream.mpd, I get:

! ../src/ValidateAtoms.cpp : 2406 : Warning: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found "hvcC": video sample descriptions would not normally contain this
! ../src/ValidateAtoms.cpp : 2406 : Warning: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found "btrt": video sample descriptions would not normally contain this
! ../src/ValidateAtoms.cpp : 2406 : Warning: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found "pasp": video sample descriptions would not normally contain this
✓ ../src/ValidateAtoms.cpp : 3999 : colr atom of type nclx found, the software does not handle colr atoms of this type.
! ../src/ValidateAtomList.cpp : 2339 : WARNING: In moov-1:udta-1 - unknown/unexpected atom 'meta'
✗ ### error:
### ../src/PostprocessData.cpp : 1147 : ### error:
### According to DASH-IF IOP Section 3.2.8 @bandwidth of the Representation (3500000 bps) is set too low given the @minimumBufferTime (2 s), the minimum @bandwidth value required to conform is 3716606 bps.
✓ 

[jpiesing]

@nicholas-fr ?

[nicholas-fr]

@rbouqueau I tested
php Process_cli.php --cmaf --ctawave --segments https://dash.akamaized.net/WAVE/vectors/chh1_sets/12.5_25_50/t1/2023-05-12/stream.mpd

I see exactly the same result via https://conformance.dashif.org/ at the moment, also when resting with the latest development version (commit ae68be8) on my local machine.

( Side note: I like the formatting improvements :) )

"✗ ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 2965 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"✗ ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 3219 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"! ../src/ValidateAtoms.cpp : 2400 : Warning: In moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found \"btrt\": video sample descriptions would not normally contain this",
"! ../src/ValidateAtoms.cpp : 2400 : Warning: In moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found \"pasp\": video sample descriptions would not normally contain this",
"✓ ../src/ValidateAtoms.cpp : 3999 : colr atom of type nclx found, the software does not handle colr atoms of this type. ",
"! ../src/ValidateAtomList.cpp : 2339 : WARNING: In moov-1:udta-1 - unknown/unexpected atom 'meta'",
"✗ ### error:  \r###\t\t../src/PostprocessData.cpp : 1147 : ### error:  \r###\t\tAccording to DASH-IF IOP Section 3.2.8 @bandwidth of the Representation (3500000 bps) is set too low given the @minimumBufferTime (2 s), the minimum @bandwidth value required to conform is 3716606 bps.",
"✓ "

I'm also confused about this failure:

{
"spec": "CTAWAVE",
"section": "WAVE Content Spec 2018Ed-Section 4.1",
"test": "WAVE content SHALL include one or more Switching Sets conforming to at least one WAVE approved CMAF Media Profile",
"messages": [
  "✓ Switching Set 0 found with only tracks conforming to Media Profile unknown",
  "✗ Video track found, but WAVE video track missing"
],
"state": "FAIL",
"part": {}
}

[rbouqueau]

I'm also confused about this failure:

The conformance tool reported errors w.r.t the WAVE media profile. As a consequence no conforming Switching Set was found (this is what this error says). In general the WAVE section didn't age well.

"✗ ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 2965 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",
"✗ ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t../src/ValidateBitStreams.cpp : 3219 : ### error: moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 \r###\t\t\tValidate_NAL_Unit_HEVC: Trailing zero bits not zero 1 ",

I don't see any issue on my end. I reported it.

"! ../src/ValidateAtoms.cpp : 2400 : Warning: In moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found "btrt": video sample descriptions would not normally contain this",

The btrt box is in the sample entry which seems ok. It seems to be not supported by the ISOSegmentValidator. Reported.

"! ../src/ValidateAtoms.cpp : 2400 : Warning: In moov-1:trak-1:mdia-1:minf-1:stbl-1:stsd-1 - unknown atom found "pasp": video sample descriptions would not normally contain this",

Same, It seems to be not supported by the ISOSegmentValidator. Reported.

"✓ ../src/ValidateAtoms.cpp : 3999 : colr atom of type nclx found, the software does not handle colr atoms of this type. ",

Not supported by the ISOSegmentValidator. Reported.

"! ../src/ValidateAtomList.cpp : 2339 : WARNING: In moov-1:udta-1 - unknown/unexpected atom 'meta'",

Not supported by the ISOSegmentValidator. Note that this box is forwarded from the mezzanine source. Reported.

"✗ ### error: \r###\t\t../src/PostprocessData.cpp : 1147 : ### error: \r###\t\tAccording to DASH-IF IOP Section 3.2.8 @Bandwidth of the Representation (3500000 bps) is set too low given the @minimumBufferTime (2 s), the minimum @Bandwidth value required to conform is 3716606 bps.",

This is useful information. However given its complexity it seems to be wrong most of the time. I've added a low priority note on my end to understand how the computation is made.

"✓ "

I've reported this trailing check.

[gitwjr]

Validator issue that needs to be addressed in Conformance Tool. @FritzHeiden can move forward with the content as it appears to be valid.

[FritzHeiden]

Tests 8.15 and 8.16 using provided HEVC content (chh1) are merged in master. I was not able to test this again, as I don't have access to a browser capable of decoding HEVC content at the moment.

[gitwjr]

@louaybassbouss Fraunhofer will check on an HbbTV terminal after the plugfest.
Deferred until after the Plugfest.

[rbouqueau]

We are after the plugfest :) @louaybassbouss @FritzHeiden  Please let us know when you could test.

[louaybassbouss]

@rbouqueau we will check tomorrow

[louaybassbouss]

@rbouqueau we confirm that chh1 content is valid

[wschidol]

DPCAT Tests 8.15 Source Buffer Re-Initialization (without changeType) and 8.16 Source Buffer Re-Initialization (with changeType) require content in different 2 codecs. For video, we have AVC so we need content in a different codec e.g HEVC and/or AV1. this related comment from @jpiesing related to this issue may help (copied from the google document):

For video, the obvious test content requirement is HEVC / AVC but any HEVC stream would do I think? For audio, the obvious one would be multi-channel to stereo or v.v.. Audio expert confirmation is required.

Not sure whether this issue is already resolved? To be sure, a codec change and a change of channel configuration are different things (even though they tend to go together). To emulate a real-world-ish scenario, we could switch from EAC3 in a 5.1 configuration to AAC in a 2.0 configuration (and/or back).