[YouTube] Fix extraction of the Trending kiosk with the new data model returned #920
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AudricV
added
bug
9 tasks
Stypox
requested changes
Sep 11, 2022
.../java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeTrendingExtractor.java
Outdated
Show resolved
Hide resolved
… section This new structure allow us to filter easily Trending shorts and Recently trending sections. On the previous one, this Recently trending section is now filtered, by checking whether sections have a title, which isn't the case for normal trends contrary to the other ones. This makes that the extractor returns now only the real 50 "Now" YouTube trends. Elements inside arrays are now extracted dynamically instead of only the ones of the first index, using Java 8's Stream API. The getInitialPage() method of YoutubeTrendingExtractor can now throw a ParsingException if no selected tab (corresponding to the one of the trends type extracted) has been found. Finally, the licence header has been moved to the top of the file and updated.
AudricV
force-pushed
the
fix-yt-trending-extraction
branch
from
2e71cdf to
d8ddeb5
Compare
Stypox
approved these changes
Sep 13, 2022
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I didn't test the changes on NewPipe but tested multiple times the changes using the
YoutubeTrendingKioskInfoTesttest classThis pull request adds support of the new data model returned for trends, which is A/B tested or deployed by YouTube.
This new structure allow us to filter easily
Trending shortsandRecently trendingsections.On the previous one, this
Recently trendingsection is now filtered (theTrending shortsone uses a different structure, which so was already not extracted), by checking whether sections have a title, which isn't the case for normal trends contrary to the other ones.This makes that the extractor returns now only the real 50
NowYouTube trends, with both data models returned by YouTube.Elements inside arrays are now extracted dynamically instead of only the ones of the first index, using Java 8's Stream API.
Note: The
getInitialPage()method ofYoutubeTrendingExtractorcan now throw aParsingExceptionif no selected tab (corresponding to the one of the trends type extracted) has been found.Finally, the licence header of this class has been moved to the top of the file and updated.
Fixes #899, fixes TeamNewPipe/NewPipe#8835.