-
Notifications
You must be signed in to change notification settings - Fork 425
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixing up tests and the bugs they exposed #871
Conversation
@@ -233,7 +233,7 @@ private YoutubeParsingHelper() { | |||
* The three digits at the end can be random, but are required. | |||
* </p> | |||
*/ | |||
private static final String CONSENT_COOKIE_VALUE = "PENDING+"; | |||
private static final String CONSENT_COOKIE_VALUE = "YES+"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is to address #820
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, do not change this at all. By using YES+
, you allow Google to track usage of users and so to establish profiles based on what they search, they browse and they watch, using the YouTube requests made by the extractor. The NewPipe projects aim to protect as much as possible users' privacy, so we will not accept this change globally.
Instead, allow only the ability to set the YES+
cookie in YoutubeMixPlaylistExtractor
, but disable it by default. That's I wanted to do in a future PR (because I wanted #863 to be merged first, but I have some changes to do in it first). Based on the changes of this PR in YoutubeMixPlaylistExtractor
, you can find what I did here: https://github.com/AudricV/NewPipeExtractor/commits/consent-cookie-yes%2B-for-yt-mixes-option.
If you want to apply them, you can reuse (partially or not) my changes, but do not add the Content-Length
header in the requests like I did (see #863 (comment) to know why I say this).
Then enable usage of this cookie value in test classes of YoutubeMixPlaylistExtractorTest
, for each test (like I did).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy to apply these changes, although if we only use the YES+
cookie in tests, would that mean the mix extractor won't work for users in the EU?
Btw, any clue how I can test this extractor in NewPipe app? 😅
I've tried searching for playlists, but none of them seem to be youtube mixes.
@@ -89,7 +89,7 @@ public void onFetchPage(@Nonnull final Downloader downloader) | |||
final byte[] body = JsonWriter.string(jsonBody.done()).getBytes(StandardCharsets.UTF_8); | |||
|
|||
final Map<String, List<String>> headers = new HashMap<>(); | |||
addClientInfoHeaders(headers); | |||
addYouTubeHeaders(headers); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addYouTubeHeaders
adds the consent cookie in addition to calling addClientInfoHeaders
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See above.
...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMusicSearchExtractor.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the pull request.
Unfortunately, you will have to change how you fix some tests (see the review comments).
Also, when updating YouTube tests, please test your changes with the mock downloader and, if needed, update the relevant test mocks by running the tests using the recording downloader, then test again the changes with the mock downloader to be sure that everything works properly when using this one.
For more details about how mocks work in the extractor, don't hesitate to take a look at our documentation :)
@@ -233,7 +233,7 @@ private YoutubeParsingHelper() { | |||
* The three digits at the end can be random, but are required. | |||
* </p> | |||
*/ | |||
private static final String CONSENT_COOKIE_VALUE = "PENDING+"; | |||
private static final String CONSENT_COOKIE_VALUE = "YES+"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, do not change this at all. By using YES+
, you allow Google to track usage of users and so to establish profiles based on what they search, they browse and they watch, using the YouTube requests made by the extractor. The NewPipe projects aim to protect as much as possible users' privacy, so we will not accept this change globally.
Instead, allow only the ability to set the YES+
cookie in YoutubeMixPlaylistExtractor
, but disable it by default. That's I wanted to do in a future PR (because I wanted #863 to be merged first, but I have some changes to do in it first). Based on the changes of this PR in YoutubeMixPlaylistExtractor
, you can find what I did here: https://github.com/AudricV/NewPipeExtractor/commits/consent-cookie-yes%2B-for-yt-mixes-option.
If you want to apply them, you can reuse (partially or not) my changes, but do not add the Content-Length
header in the requests like I did (see #863 (comment) to know why I say this).
Then enable usage of this cookie value in test classes of YoutubeMixPlaylistExtractorTest
, for each test (like I did).
@@ -89,7 +89,7 @@ public void onFetchPage(@Nonnull final Downloader downloader) | |||
final byte[] body = JsonWriter.string(jsonBody.done()).getBytes(StandardCharsets.UTF_8); | |||
|
|||
final Map<String, List<String>> headers = new HashMap<>(); | |||
addClientInfoHeaders(headers); | |||
addYouTubeHeaders(headers); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See above.
|
||
return firstContent.has("didYouMeanRenderer") | ||
|| firstContent.has("showingResultsForRenderer"); | ||
return this.getSearchSuggestion() != ""; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you really need the usage of this
? If not, can you remove it, please (not really an issue)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries, will remove.
@@ -83,8 +83,7 @@ public void testAlbumSearch() throws ExtractionException, IOException { | |||
assertEquals("https://c418.bandcamp.com/album/minecraft-volume-alpha", minecraft.getUrl()); | |||
|
|||
// Verify that playlist tracks counts get extracted correctly | |||
assertEquals(24, ((PlaylistInfoItem) minecraft).getStreamCount()); | |||
|
|||
assertTrue(((PlaylistInfoItem) minecraft).getStreamCount() > 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The failure was not on the stream count, according to the latest scheduled tests (that you can see in the Actions
tab of the repository), it was on the first item returned by the search results. The latest run does not have this issue, so I think that's a intermittent issue which can be ignored.
You may revert these changes if you can't reproduce them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like the metadata for this particular video changes back and forth.. When changed the name, the image started to fail, and then there were only 3 streams reported. Perhaps we could just use a different video here?
.../test/java/org/schabi/newpipe/extractor/services/peertube/PeertubePlaylistExtractorTest.java
Show resolved
Hide resolved
.../test/java/org/schabi/newpipe/extractor/services/peertube/PeertubePlaylistExtractorTest.java
Show resolved
Hide resolved
...test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeMixPlaylistExtractorTest.java
Outdated
Show resolved
Hide resolved
@Override public List<MetaInfo> expectedMetaInfo() throws MalformedURLException { | ||
return Collections.singletonList(new MetaInfo( | ||
EMPTY_STRING, | ||
new Description("Funk is a German public broadcast service.", Description.PLAIN_TEXT), | ||
Collections.singletonList(new URL("https://de.wikipedia.org/wiki/Funk_(Medienangebot)?wprov=yicw1")), | ||
Collections.singletonList("Wikipedia (German)") | ||
)); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As the goal of this test is to test if MetaInfo
s are extracted properly from videos metadata, you will have to find a new video with a MetaInfo
and update all the relevant assertions instead of removing the test (and maybe also the test class name and the name of the folder in which mocks are saved).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No other tests in this file seem to be overriding expectedMetaInfo
. I actually suspect the YoutubeStreamExtractor
currently can't extract meta info -- it is contained within the videoPrimaryInfoRenderer
and videoSecondaryInfoRenderer
objects rather than there the current YoutubeParsingHelper.getMetaInfo()
is looking for it. How would we better address this?
Appreciate your timely feedback @AudricV! I'll address the comments in a week's time when I get my laptop back 😀 |
@@ -57,9 +57,9 @@ public void testUploaderName() throws Exception { | |||
@Override public long expectedLength() { return 0; } | |||
@Override public long expectedTimestamp() { return TIMESTAMP; } | |||
@Override public long expectedViewCountAtLeast() { return 0; } | |||
@Nullable @Override public String expectedUploadDate() { return "2020-02-22 00:00:00.000"; } | |||
@Nullable @Override public String expectedTextualUploadDate() { return "2020-02-22"; } | |||
@Override public long expectedLikeCountAtLeast() { return 825000; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They've replaced the stream that was referenced by the test previously. Tests against live streams seem rather fragile -- do we have a better approach to this? Just testing against mocks?
Would be good to still test against the live API.. so perhaps, just find the first stream and check that we can extract non-empty values?
Superseded by #882. Thank you for your effort. |
Attempting to fix CI issues. Tested a local build with NewPipe. Let me know if you believe this should rather be done in a series of separate PRs, or there are better solutions :)