Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for channel tabs #951

Closed
wants to merge 64 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
ed4559d
fix: support richGridRenderer on channel page
Theta-Dev Oct 12, 2022
8b4b431
feat: add tab support to channel extractor
Theta-Dev Oct 22, 2022
18e3758
feat: add channel tabs
Theta-Dev Oct 23, 2022
78bbbd4
fix: handle unsupported content, hide tab bar with < 2 tabs
Theta-Dev Oct 23, 2022
9a9fae9
feat: prettier channel info page
Theta-Dev Oct 23, 2022
667ab2a
feat: add album tab
Theta-Dev Oct 23, 2022
57865e2
feat: add visitor data config option
Theta-Dev Oct 23, 2022
aed685e
feat: add tab support for Peertube
Theta-Dev Oct 23, 2022
53e772c
feat: add tab support for Soundcloud
Theta-Dev Oct 23, 2022
e6907ca
fix: Peertube playlist urls
Theta-Dev Oct 23, 2022
04c7e46
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev Oct 24, 2022
edaaaac
fix: checkstyle errors
Theta-Dev Oct 24, 2022
1253773
fix: store YouTube visitor data for channel tabs
Theta-Dev Oct 25, 2022
94523ad
feat: add Bandcamp album tab
Theta-Dev Oct 25, 2022
a592c96
test: add channel tab extractor tests
Theta-Dev Oct 25, 2022
f3b064a
fix: change playlist tab parameter to include YTM albums
Theta-Dev Nov 2, 2022
0a458d8
fix: NPE when extracting YT stream items without duration
Theta-Dev Nov 2, 2022
856584f
fix: channel shorts duration parsing
Theta-Dev Nov 2, 2022
7ec6a44
fix: channel short upload date parsing
Theta-Dev Nov 3, 2022
f71fdac
refactor: API changes
Theta-Dev Nov 4, 2022
73c182f
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev Nov 4, 2022
abf0473
fix: support new PlaylistInfoItem interface
Theta-Dev Nov 4, 2022
8a3545c
fix: rename channel tab LIVE to LIVESTREAMS
Theta-Dev Nov 4, 2022
7dba12b
fix: link handler urls for tabs
Theta-Dev Nov 4, 2022
f6d8652
fix: update mock data
Theta-Dev Nov 4, 2022
2245de1
fix: make getTab nonnull
Theta-Dev Nov 16, 2022
8d3bc2b
fix: YoutubeParsingHelper formatting
Theta-Dev Nov 22, 2022
f7e3b71
Merge branch 'dev' into channel-tabs
Theta-Dev Nov 22, 2022
ffd02a4
fix: shorts continuation
Theta-Dev Nov 29, 2022
c156c40
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev Nov 29, 2022
d2c2aca
fix: tests failing to compile
Theta-Dev Nov 29, 2022
8446e20
test: update channel tab mocks
Theta-Dev Nov 29, 2022
9cebcf7
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev Mar 20, 2023
5b63a3e
tests: update mocks, remove special cases for a/b test
Theta-Dev Mar 21, 2023
76052de
fix: YT shorts view count parsing
Theta-Dev Mar 21, 2023
8ecee87
fix: channel extractor tests, docs
Theta-Dev Mar 22, 2023
8cd6439
fix: YT shorts view count parsing (2)
Theta-Dev Mar 22, 2023
c6ee2f3
fix: add checkIfChannelResponseIsValid method
Theta-Dev Mar 22, 2023
97d7ee5
Merge branch 'dev' of github.com:Theta-Dev/NewPipeExtractor into chan…
Theta-Dev Mar 30, 2023
f306db0
refactor: move YT channel utils to YouTubeChannelHelper
Theta-Dev Mar 30, 2023
e57d43f
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev Apr 5, 2023
750f158
add comment to clarify channel tab param
Theta-Dev Apr 5, 2023
e278a2d
refactor: use streams instead of for loops
Theta-Dev Apr 5, 2023
c3651be
Channels are now an Info
Stypox Apr 13, 2023
12ca6a2
fix tests
Theta-Dev Apr 14, 2023
294ffab
Add documentation for ReadyChannelTabLLH
Stypox Apr 14, 2023
bad1238
Fix PeerTube channel tabs content filters
Stypox Apr 14, 2023
6e0ffaf
fix: Peertube channel tab extractor fetching global feed
Theta-Dev Apr 15, 2023
308fc43
fix: make ChannelTabExtractorBuilder serializable
Theta-Dev Apr 16, 2023
6b627f8
feat: fetch YT Shorts using internal playlist
Theta-Dev Apr 16, 2023
2ad496f
refactor: merge YoutubeChannelTabExtractor and YoutubeChannelVideosTa…
Theta-Dev Apr 16, 2023
0c5fdac
fix: remove overridden getId function in PeertubeAccountExtractor
Theta-Dev Apr 18, 2023
6a38811
[Bandcamp] Use same url for tracks and albums channel tabs
Stypox Apr 25, 2023
d47d0f9
fix: add Bandcamp URL suffixes
Theta-Dev Apr 27, 2023
417b797
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev Apr 27, 2023
0e28f2b
tests: add tests for channel tab urls
Theta-Dev Apr 27, 2023
0583515
tests: separate channel/tab tests for Peertube, Bandcamp, Soundcloud
Theta-Dev May 1, 2023
a3f6a7e
tests: add @override to YT channel/tab tests
Theta-Dev May 1, 2023
d868746
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into ch…
Theta-Dev May 1, 2023
2adc2ca
fix: use assertTabs method, rename channelTab mock folder
Theta-Dev May 7, 2023
e8fab3b
fix: add #1050 fix to channel tab name extraction
Theta-Dev May 7, 2023
b1f8905
docs: add docs to ChannelTabInfo
Theta-Dev May 7, 2023
66d8038
refactor: remove getTab() method from ChannelTabExtractor
Theta-Dev May 7, 2023
6c5a225
fix: improve shorts duration parser
Theta-Dev May 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package org.schabi.newpipe.extractor;

import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
Expand Down Expand Up @@ -57,7 +58,8 @@ public static class ServiceInfo {

/**
* Creates a new instance of a ServiceInfo
* @param name the name of the service
*
* @param name the name of the service
* @param mediaCapabilities the type of media this service can handle
*/
public ServiceInfo(final String name, final List<MediaCapability> mediaCapabilities) {
Expand Down Expand Up @@ -97,8 +99,9 @@ public enum LinkType {
* If you Implement one do not set id within your implementation of this extractor, instead
* set the id when you put the extractor into {@link ServiceList}
* All other parameters can be set directly from the overriding constructor.
* @param id the number of the service to identify him within the NewPipe frontend
* @param name the name of the service
*
* @param id the number of the service to identify him within the NewPipe frontend
* @param name the name of the service
* @param capabilities the type of media this service can handle
*/
public StreamingService(final int id,
Expand Down Expand Up @@ -129,29 +132,42 @@ public String toString() {

/**
* Must return a new instance of an implementation of LinkHandlerFactory for streams.
*
* @return an instance of a LinkHandlerFactory for streams
*/
public abstract LinkHandlerFactory getStreamLHFactory();

/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channels.
* If support for channels is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelLHFactory();

/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channel tabs.
* If support for channel tabs is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelTabLHFactory();

/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for playlists.
* If support for playlists is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for playlists or null
*/
public abstract ListLinkHandlerFactory getPlaylistLHFactory();

/**
* Must return an instance of an implementation of SearchQueryHandlerFactory.
*
* @return an instance of a SearchQueryHandlerFactory
*/
public abstract SearchQueryHandlerFactory getSearchQHFactory();

public abstract ListLinkHandlerFactory getCommentsLHFactory();

/*//////////////////////////////////////////////////////////////////////////
Expand All @@ -160,19 +176,22 @@ public String toString() {

/**
* Must create a new instance of a SearchExtractor implementation.
*
* @param queryHandler specifies the keyword lock for, and the filters which should be applied.
* @return a new SearchExtractor instance
*/
public abstract SearchExtractor getSearchExtractor(SearchQueryHandler queryHandler);

/**
* Must create a new instance of a SuggestionExtractor implementation.
*
* @return a new SuggestionExtractor instance
*/
public abstract SuggestionExtractor getSuggestionExtractor();

/**
* Outdated or obsolete. null can be returned.
*
* @return just null
*/
public abstract SubscriptionExtractor getSubscriptionExtractor();
Expand All @@ -192,20 +211,32 @@ public FeedExtractor getFeedExtractor(final String url) throws ExtractionExcepti

/**
* Must create a new instance of a KioskList implementation.
*
* @return a new KioskList instance
*/
public abstract KioskList getKioskList() throws ExtractionException;

/**
* Must create a new instance of a ChannelExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelExtractor
*/
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler)
throws ExtractionException;

/**
* Must create a new instance of a ChannelTabExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelTabExtractor
*/
public abstract ChannelTabExtractor getChannelTabExtractor(ListLinkHandler linkHandler)
throws ExtractionException;

/**
* Must crete a new instance of a PlaylistExtractor implementation.
*
* @param linkHandler is pointing to the playlist which should be handled by this new instance.
* @return a new PlaylistExtractor
*/
Expand All @@ -214,6 +245,7 @@ public abstract PlaylistExtractor getPlaylistExtractor(ListLinkHandler linkHandl

/**
* Must create a new instance of a StreamExtractor implementation.
*
* @param linkHandler is pointing to the stream which should be handled by this new instance.
* @return a new StreamExtractor
*/
Expand Down Expand Up @@ -242,6 +274,19 @@ public ChannelExtractor getChannelExtractor(final String id,
.fromQuery(id, contentFilter, sortFilter));
}

public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), ""));
}

public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab,
final String baseUrl)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), "", baseUrl));
}

public PlaylistExtractor getPlaylistExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
Expand Down Expand Up @@ -284,6 +329,7 @@ public CommentsExtractor getCommentsExtractor(final String url) throws Extractio

/**
* Figures out where the link is pointing to (a channel, a video, a playlist, etc.)
*
* @param url the url on which it should be decided of which link type it is
* @return the link type of url
*/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
package org.schabi.newpipe.extractor.channel;

import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.Extractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;

import java.util.Collections;
import java.util.List;

import javax.annotation.Nonnull;

/*
* Created by Christian Schabesberger on 25.07.16.
Expand All @@ -26,7 +30,7 @@
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/

public abstract class ChannelExtractor extends ListExtractor<StreamInfoItem> {
public abstract class ChannelExtractor extends Extractor {

public static final long UNKNOWN_SUBSCRIBER_COUNT = -1;

Expand All @@ -43,5 +47,10 @@ public ChannelExtractor(final StreamingService service, final ListLinkHandler li
public abstract String getParentChannelUrl() throws ParsingException;
public abstract String getParentChannelAvatarUrl() throws ParsingException;
public abstract boolean isVerified() throws ParsingException;

@Nonnull
public abstract List<ListLinkHandler> getTabs() throws ParsingException;
@Nonnull
public List<String> getTags() throws ParsingException {
return Collections.emptyList();
}
}
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
package org.schabi.newpipe.extractor.channel;

import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.Info;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;

import java.io.IOException;
import java.util.Collections;
import java.util.List;

import javax.annotation.Nonnull;

/*
* Created by Christian Schabesberger on 31.07.16.
Expand All @@ -32,16 +32,14 @@
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/

public class ChannelInfo extends ListInfo<StreamInfoItem> {
public class ChannelInfo extends Info {

public ChannelInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final ListLinkHandler listLinkHandler) {
super(serviceId, id, url, originalUrl, name, listLinkHandler.getContentFilters(),
listLinkHandler.getSortFilter());
final String name) {
super(serviceId, id, url, originalUrl, name);
}

public static ChannelInfo getInfo(final String url) throws IOException, ExtractionException {
Expand All @@ -55,13 +53,6 @@ public static ChannelInfo getInfo(final StreamingService service, final String u
return getInfo(extractor);
}

public static InfoItemsPage<StreamInfoItem> getMoreItems(final StreamingService service,
final String url,
final Page page)
throws IOException, ExtractionException {
return service.getChannelExtractor(url).getPage(page);
}

FireMasterK marked this conversation as resolved.
Show resolved Hide resolved
public static ChannelInfo getInfo(final ChannelExtractor extractor)
throws IOException, ExtractionException {

Expand All @@ -71,8 +62,7 @@ public static ChannelInfo getInfo(final ChannelExtractor extractor)
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();

final ChannelInfo info =
new ChannelInfo(serviceId, id, url, originalUrl, name, extractor.getLinkHandler());
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name);

try {
info.setAvatarUrl(extractor.getAvatarUrl());
Expand All @@ -90,11 +80,6 @@ public static ChannelInfo getInfo(final ChannelExtractor extractor)
info.addError(e);
}

final InfoItemsPage<StreamInfoItem> itemsPage =
ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPage(itemsPage.getNextPage());

try {
info.setSubscriberCount(extractor.getSubscriberCount());
} catch (final Exception e) {
Expand Down Expand Up @@ -130,6 +115,17 @@ public static ChannelInfo getInfo(final ChannelExtractor extractor)
info.addError(e);
}

try {
info.setTabs(extractor.getTabs());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTags(extractor.getTags());
} catch (final Exception e) {
info.addError(e);
}

return info;
}

Expand All @@ -144,6 +140,10 @@ public static ChannelInfo getInfo(final ChannelExtractor extractor)
private String[] donationLinks;
private boolean verified;

private List<ListLinkHandler> tabs = Collections.emptyList();

private List<String> tags = Collections.emptyList();

public String getParentChannelName() {
return parentChannelName;
}
Expand Down Expand Up @@ -223,4 +223,22 @@ public boolean isVerified() {
public void setVerified(final boolean verified) {
this.verified = verified;
}

@Nonnull
public List<ListLinkHandler> getTabs() {
return tabs;
}

public void setTabs(@Nonnull final List<ListLinkHandler> tabs) {
this.tabs = tabs;
}

@Nonnull
public List<String> getTags() {
return tags;
}

public void setTags(@Nonnull final List<String> tags) {
this.tags = tags;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package org.schabi.newpipe.extractor.channel;

import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;

import javax.annotation.Nonnull;

public abstract class ChannelTabExtractor extends ListExtractor<InfoItem> {

public ChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}

@Nonnull
@Override
public String getName() {
return getLinkHandler().getContentFilters().get(0);
}
}
Loading