Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YouTube] Use the new internal API in NewPipe Extractor #604

Merged
merged 55 commits into from
Aug 3, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
5794eb2
Use the youtubei API for YouTube playlists
AudricV Apr 8, 2021
a12c69d
Use the youtubei API for YouTube channels
AudricV Apr 8, 2021
f461224
Use the youtubei API for YouTube searches + update mocks
AudricV Apr 9, 2021
77c031a
Use the youtubei API for YouTube trends
AudricV Apr 11, 2021
9ab9c66
Use the youtubei.googleapis.com domain for YouTube Music searches + c…
AudricV Apr 11, 2021
991b2c7
Use lightweight requests when getting and checking YouTube API key an…
AudricV Apr 12, 2021
58ce9b0
Fix channel extraction when channel URL is youtube.com/c/username
AudricV Apr 14, 2021
b49ae54
Do some improvements to YoutubeStreamExtractor
AudricV Apr 15, 2021
e075dd5
Update client version, fix some tests, update mocks and do some impro…
AudricV Apr 19, 2021
f46cfb0
Adress reviews and do some improvements
AudricV Apr 25, 2021
4d68283
Fix localization and update client version
AudricV Apr 30, 2021
effcdaa
Update mocks, reenable a test and fix a test
AudricV Apr 30, 2021
f73c923
Don't use the youtubei.googleapis.com but the websites domains + upda…
AudricV May 9, 2021
e7d589e
Use the youtubei API for YouTube videos + update client version
AudricV May 23, 2021
013b902
Use the Android mobile API when there are OTF streams or the content …
AudricV May 29, 2021
3017dde
Fix some typos
AudricV May 29, 2021
14569c4
Readd the return of itags 134 and 136
AudricV May 29, 2021
0f9e9b8
Use the youtubei API for YouTube mixes + update the corresponding tes…
AudricV May 30, 2021
ae5abc0
Fix a typo in YoutubeChannelExtractor
AudricV May 31, 2021
b52732a
Fix the fetch of the playerCode at each time getStsFromPlayerJs is ca…
AudricV May 31, 2021
c32bc6e
Try to don't fetch again the first page when requesting a channel con…
AudricV May 31, 2021
7474049
Update mocks
AudricV May 31, 2021
947baec
Fetch again the desktop player JSON only if the content is protected …
AudricV May 31, 2021
318bc46
Readd the deleted code of views because watching count of livestreams…
AudricV Jun 2, 2021
70927dd
Update client version and mocks
AudricV Jun 2, 2021
6921e80
Try again to don't fetch at each time the JavaScript player
AudricV Jun 2, 2021
a59c2a3
Catch every exception instead of only IOException and ExtractionExcep…
AudricV Jun 2, 2021
54d4551
Adress requested changes in YoutubeParsingHelper and update mobile cl…
AudricV Jun 5, 2021
34a9ccb
Adress requested changes
AudricV Jun 5, 2021
e97a685
Format the YoutubeMusicSearchExtractor class to be in the 100 charact…
AudricV Jun 5, 2021
8c1c728
Real fix of the fetch of the JS player at each the signatureTimestamp…
AudricV Jun 5, 2021
657f165
Update client version and mocks
AudricV Jun 5, 2021
632772d
Adress requested changes in YoutubeParsingHelper
AudricV Jun 6, 2021
accd5dd
Adress requested changes in YoutubeStreamExtractor
AudricV Jun 6, 2021
cc79852
Adress requested changes in other classes
AudricV Jun 6, 2021
81013e5
Add a static method to reset the YouTube deobfuscation code and use i…
AudricV Jun 6, 2021
a6a2c6e
Revert the use of Collections.singletonList instead of Arrays.asList …
AudricV Jun 11, 2021
d8177b5
Loop in all formats to check if the stream has URLs protected by sign…
AudricV Jun 11, 2021
1a6b8da
Annotate YoutubeParsingHelper methods with Nonnull when needed
AudricV Jun 11, 2021
4299d80
Adress changes
AudricV Jun 24, 2021
609919d
Adress again reviews, fix some rebase issues
AudricV Jun 26, 2021
27bd797
Update mocks
AudricV Jun 26, 2021
8aa60d7
Update clients version
AudricV Jul 4, 2021
3adac6a
Workaround for rate limits: always use the Android mobile API
AudricV Jul 5, 2021
629dcd6
Ignore some broken tests
AudricV Jul 5, 2021
2320aec
Update mocks
AudricV Jul 5, 2021
7753556
Adress the last requested changes + update YoutubeCommentsExtractor m…
AudricV Jul 9, 2021
2eeb0a3
Rebase + some code improvements + fix extraction of age-restricted vi…
FireMasterK Jul 28, 2021
d551092
Fix YoutubeStreamExtractorAgeRestrictedTest
AudricV Jul 30, 2021
693f654
Ignore a broken test of YoutubeMixPlaylistExtractorTest
AudricV Jul 30, 2021
286d839
Update mocks (except mocks of YoutubeCommentsExtractorTest)
AudricV Jul 30, 2021
f3e4c9d
Use the youtubei API for YouTube comments
FireMasterK Jul 29, 2021
f8197da
Update YouTube comments and YouTube playlist continuations mocks
AudricV Jul 31, 2021
b74a39c
Reformat some code and don't use the clickTrackingParams in continuat…
AudricV Aug 1, 2021
79cc5c8
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtra…
AudricV Aug 1, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;

Expand All @@ -17,7 +18,7 @@ public CommentsExtractor(final StreamingService service, final ListLinkHandler u
* @apiNote Warning: This method is experimental and may get removed in a future release.
* @return <code>true</code> if the comments are disabled otherwise <code>false</code> (default)
*/
public boolean isCommentsDisabled() {
public boolean isCommentsDisabled() throws ExtractionException {
return false;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,12 @@ public class ItagItem {
/// VIDEO ONLY ////////////////////////////////////////////
// ID Type Format Resolution FPS ///
/////////////////////////////////////////////////////////
// Don't add VideoOnly streams that have normal variants
new ItagItem(160, VIDEO_ONLY, MPEG_4, "144p"),
new ItagItem(133, VIDEO_ONLY, MPEG_4, "240p"),
// new ItagItem(134, VIDEO_ONLY, MPEG_4, "360p"),
new ItagItem(134, VIDEO_ONLY, MPEG_4, "360p"),
new ItagItem(135, VIDEO_ONLY, MPEG_4, "480p"),
new ItagItem(212, VIDEO_ONLY, MPEG_4, "480p"),
// new ItagItem(136, VIDEO_ONLY, MPEG_4, "720p"),
new ItagItem(136, VIDEO_ONLY, MPEG_4, "720p"),
new ItagItem(298, VIDEO_ONLY, MPEG_4, "720p60", 60),
new ItagItem(137, VIDEO_ONLY, MPEG_4, "1080p"),
new ItagItem(299, VIDEO_ONLY, MPEG_4, "1080p60", 60),
Expand All @@ -75,6 +74,7 @@ public class ItagItem {
new ItagItem(313, VIDEO_ONLY, WEBM, "2160p"),
new ItagItem(315, VIDEO_ONLY, WEBM, "2160p60", 60)
};

/*//////////////////////////////////////////////////////////////////////////
// Utils
//////////////////////////////////////////////////////////////////////////*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@ public static String extractJavaScriptCode() throws ParsingException {
return extractJavaScriptCode("d4IGg5dqeO8");
}

/**
* Reset the JavaScript code. It will be fetched again the next time
* {@link #extractJavaScriptCode()} or {@link #extractJavaScriptCode(String)} is called.
*/
public static void resetJavaScriptCode() {
cachedJavaScriptCode = null;
}

private static String extractJavaScriptUrl(final String videoId) throws ParsingException {
try {
final String embedUrl = "https://www.youtube.com/embed/" + videoId;
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,13 @@ private String parseDecodeFunction(final String playerJsCode, final String funct
}
}

@Nonnull
private String parseWithParenthesisMatching(final String playerJsCode, final String functionName) {
final String functionBase = functionName + "=function";
return functionBase + StringUtils.matchToClosingParenthesis(playerJsCode, functionBase) + ";";
}

@Nonnull
private String parseWithRegex(final String playerJsCode, final String functionName) throws Parser.RegexException {
Pattern functionPattern = Pattern.compile(functionName + "=function(.*?}};)\n",
Pattern.DOTALL);
Expand Down
Loading