[Proposal] Add hidden "Dummy media API" experimental feature and add integration tests linked to DRM #1478
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peaBerberian
added
tests
peaBerberian
force-pushed
the
misc/mse-mock
branch
3 times, most recently
from
bb18117 to
7743e95
Compare
peaBerberian
added
the
proposal
peaBerberian
force-pushed
the
misc/mse-mock
branch
5 times, most recently
from
ae80343 to
9527be7
Compare
peaBerberian
force-pushed
the
misc/no-eme-type
branch
2 times, most recently
from
c4aae81 to
a7cc348
Compare
peaBerberian
force-pushed
the
misc/no-eme-type
branch
from
a7cc348 to
2fbc31b
Compare
peaBerberian
force-pushed
the
misc/mse-mock
branch
2 times, most recently
from
b5c4a45 to
418170b
Compare
peaBerberian
force-pushed
the
misc/no-eme-type
branch
3 times, most recently
from
6429671 to
ce097d5
Compare
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
418170b to
9001a12
Compare
peaBerberian
force-pushed
the
misc/no-eme-type
branch
from
ce097d5 to
231f5b1
Compare
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
9001a12 to
46e9cd5
Compare
peaBerberian
added a commit
that referenced
this pull request
Aug 13, 2024
While working on new DRM integration test in #1478, to add tests for the not-yet-released features proposed by #1484, I noticed that some contents anounced HEVC by beginning the codec string with `hev` and others with `hvc`. When comparing family of codecs, we thus had false negative which may mean we would have ended up playing encrypted hevc on devices where it wasn't supported, if it was anounced in the MPD as `hev1...`. I now treat both the same way.
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
46e9cd5 to
e19cc47
Compare
peaBerberian
force-pushed
the
misc/no-eme-type
branch
from
231f5b1 to
d52f7b9
Compare
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
e19cc47 to
5c63db9
Compare
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
d075b0a to
649d15c
Compare
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
649d15c to
25edf27
Compare
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
…nt rule I saw a very unfortunate typo which seems to prevent the RxPlayer from re-using a `MediaKeys` instance between contents sometimes (all the time? to check) since the v4.2.0. I saw that issue while continuing my work on integration tests in the massive #1478 PR. I thought our linter's [strict-boolean-expressions](https://typescript-eslint.io/rules/strict-boolean-expressions/) protected us against those types of typos but it turns out that for some weird reason the persons implementing that rule thought that numbers, strings and nullable object were OK (I just learned that, and I cannot understand why they thought it would be a good idea for a rule called **strict-boolean-expressions** grrrrr). So I fixed that rule, and BIM, multiple typos/missed post-refacto issues were all over the code. Thankfully, it doesn't seem to have created issues other than the aforementioned one.
peaBerberian
force-pushed
the
misc/no-eme-type
branch
from
637ba4f to
3ab41cb
Compare
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
25edf27 to
fbd2163
Compare
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
I noticed yet another bug while continuing to add DRM integration tests thanks to #1478. If the `keySystems` `loadVideo` option was subtly modified between loadVideo calls, yet if we reused the same cached `MediaKeySystemAccess` we risked to rely on the old `keySystems` option, not on the new one. It seems this bug was always there, so it shouldn't impact much applications? But it seems relatively important to me.
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
force-pushed
the
misc/mse-mock
branch
2 times, most recently
from
7d3fdd0 to
f3e9103
Compare
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
f3e9103 to
51c2b4a
Compare
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
added a commit
that referenced
this pull request
Dec 27, 2024
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
51c2b4a to
fed91c3
Compare
This is based on #1478. On #1478, I was mocking MSE+EME + a part of the [HTML5 media spec](https://html.spec.whatwg.org/#media-elements) to allow advanced integration tests without having to create encrypted contents nor having to physically test on all encountered environments. The subset of the MSE and EME API that we use isn't very complex, the harder part was to mock an `HTMLMediaElement`: especially the evolution of the current position, rebuffering, ending, readyState, autoplay etc. A shortcut I consequently took in #1478 was to rely on a potential browser behavior: [the allowed to play](https://html.spec.whatwg.org/#allowed-to-play) concept. By forbidding playback through this mechanism, I could avoid all those complex features while still being spec-compliant. However forbidding playback prevent us from making a lot of useful tests: e.g. on freezing, rebuffering, ending, auto-play etc. So this commit is an attempt to implement that. I relied on the WHATWG HTML spec but I took many shortcuts, especially by removing parts that aren't relied on at all by the RxPlayer. It's possible that I missed some quirks or that it still has some bugs, but it right now seems to following the behavior of a real `HTMLMediaElement` well enough. I also did not implement "directfile" playback for now. For all those missing features, we may ideally throw or indicate in some way that this is not yet implemented.
peaBerberian
force-pushed
the
misc/mse-mock
branch
from
fed91c3 to
fdebe8c
Compare
peaBerberian
added a commit
that referenced
this pull request
Jan 9, 2025
I noticed yet another bug while continuing to add DRM integration tests thanks to #1478. If the `keySystems` `loadVideo` option was subtly modified between loadVideo calls, yet if we reused the same cached `MediaKeySystemAccess` we risked to rely on the old `keySystems` option, not on the new one. It seems this bug was always there, so it shouldn't impact much applications? But it seems relatively important to me.
peaBerberian
added a commit
that referenced
this pull request
Jan 9, 2025
…nt rule I saw a very unfortunate typo which seems to prevent the RxPlayer from re-using a `MediaKeys` instance between contents sometimes (all the time? to check) since the v4.2.0. I saw that issue while continuing my work on integration tests in the massive #1478 PR. I thought our linter's [strict-boolean-expressions](https://typescript-eslint.io/rules/strict-boolean-expressions/) protected us against those types of typos but it turns out that for some weird reason the persons implementing that rule thought that numbers, strings and nullable object were OK (I just learned that, and I cannot understand why they thought it would be a good idea for a rule called **strict-boolean-expressions** grrrrr). So I fixed that rule, and BIM, multiple typos/missed post-refacto issues were all over the code. Thankfully, it doesn't seem to have created issues other than the aforementioned one.
peaBerberian
added a commit
that referenced
this pull request
Jan 9, 2025
Thanks to the PR #1478, that basically allows integration tests on our DRM logic without a real encrypted or decodable contents nor MSE/EME available, I noticed that I very recently brought a minor regression in the rewrite of the `MediaKeySystemAccess` cache reusage rules. We relied on the `keySystems[].type` API to see if the cached one was compatible to the new wanted one, we probably wanted to compare the former with the `type` actually provided as argument to the `navigator.requestMediaKeySystemAccess` API instead (e.g. `keySystems[].type` could be set just to `widevine`, in which case the RxPlayer will probably ask for `com.widevine.alpha` instead). Thankfully, EME defines a [`MediaKeySystemAccess.prototype.keySystem`](https://www.w3.org/TR/encrypted-media-2/#dom-navigator-requestmediakeysystemaccess) property that seems to (according to that recommendation) always be equal to the asked `keySystem`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on #1477
Reason
Today, we can only test the RxPlayer's DRM-related logic either through:
unit tests. Writing those are very time-consuming considering that EME-related code changes often, and thus often lead to unit test rewrites.
To lower the lost time, we added the concept of "global unit tests", which instead of performing tests at the lower function-level, it performed them at the module-level, which has its own API much less subject to change.
Even then, we would prefer more stable and representative tests.
rx-tester: Our closed-source repository that tests real production contents through one of our main application at canal+.This is nice but we have to go through another application that do not use the API at its full extent, so it's not an exhaustive test for us.
It's also closed-source and relying on specific real-life contents where we would prefer the idea of directly running in the RxPlayer CI tests on contents with all kinds of quirks.
manually, which is also time-consuming.
The idea is here to actually run DRM-related integration tests by completely mocking in them:
To be able to run those tests through
npm run test:integrationeither locally or in our CI.Implementation
I decided to add an [incomplete] in-JS HTML5 media element implementation as an "experimental tool" (in
src/experimental/tools/DummyMediaElement).That fake HTMLMediaElement actually implements the
IMediaElementtype defined in #1397 and so should be compatible with what the RxPlayer expects of a media element.It has also those supplementary properties:
FORCED_MEDIA_SOURCE: our implementation of aIMediaSource/ISourceBufferetc. (as defined in [Proposal] Forbid usage of the HTMLMediaElement and MSE TS types #1397) for a MSE implementationFORCED_EME_API: our implementation ofIMediaKeySystemAccess,IMediaKeys,IMediaKeySessionetc. (as defined in [Proposal] Forbid usage of theMediaKeystype and other EME TS types #1477) for an EME implementationWhen those are present, the RxPlayer will rely on them instead of the usual MSE and EME implementations.
I also added:
In the demo page: I added a
Dummy Media APIswitch allowing to rely on our MSE+EME+HTML5 implementation instead of the browser's. This is mainly to be able to debug the RxPlayer logic on contents that cannot normally be played on the demo page, usually because of DRM.With this on, those contents will load - even if with a black screen (though subtitles are displayed) - and the RxPlayer will believe that the content can play.
In integration tests: I added a fake encrypted content with several keys and some tests checking for errors thrown, fallbacks, license request configuration etc.. More tests should be quick to add.
Demo
3.mp4
If the video doesn't work: https://github.com/user-attachments/assets/a737f00d-0279-411a-8af1-bf4c6fe3c385
Video: content playback in the demo page with a mocked media element, where I try different things (seeking in an unbuffered part, in a buffered part, updating the playback rate, rebuffering, ending, play after ending).
The RxPlayer here believes it is playing in a media element but it's just a mock fully written in JavaScript that was provided to it. That mock is able to extract timing information and encryption metadata from media segments but does not actually decode nor decrypt the content.
We can see a delay between the debug information and what is actually happening (e.g. between the spinner being displayed and a the
BUFFERINGstate being shown in the debug element, because the debug element today only updates itself on an interval, this is not linked to this PRWhat's missing
Not the full extent of the MSE, EME nor HTML5 media API is implemented here.
Most notably, I did not yet implement:
persistent licenses. This does not seem hard to do, but I did not take the time yet. My implementation will just gives the standard answer indicating that the feature is not supported
MSE "sequence" mode: this can become complex and we do not need it at all for now in the RxPlayer
Only "mp4" files are handled for now by MSE (over DASH/smooth etc.), as we have to do some container parsing here
What we call "directfile" is not supported
Multiple other minor features