Change UA to privacy.resistFingerprinting.

[FireMasterK]

• I carefully read the contribution guidelines and agree to them.
• Na - these are tests - I have tested the API against NewPipe.
• Na - these are tests - I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

[TobiGr]

Please also update the user agent in YouTubeCommentsExtractor.

[FireMasterK]

Done.

[FireMasterK]

The comments extractor test seems to fail with the message Please make sure to run the tests with the RecordingDownloader first after changes.

I'm not sure what is to be done to fix this.

[XiangRongLin]

The comments extractor test seems to fail with the message Please make sure to run the tests with the RecordingDownloader first after changes.

@FireMasterK That means that the mocks responses can't be found. Which i guess, i should add to the error message.

As for why it's not finding it. ill have to take a look

[XiangRongLin]

@FireMasterK You'll have to regenerate the mocks for that specific tests, because the request contains the user agent. And every field of the request is used as key to identify which mock response should be given for a given request

[FireMasterK]

@XiangRongLin how would I do that? I'm not sure I understand how these mock tests are generated..

[XiangRongLin]

@FireMasterK Try starting from the doc of DownloaderFactory and maybe take a look at the docs from the specific downloaders too. If something is still unclear, ask, so that i can adjust the doc.

That doc is very "thrown" together

[FireMasterK]

Should be fixed now :D

[XiangRongLin]

Something that would need to be checked is whether the tests now fail against the real websites with the new user agent. since currently they are only running against the mock.

I ran ./gradlew extractor:test --tests=org.schabi.newpipe.extractor.services.youtube.*, which just runs the tests in that package. It completeted with just 4 failures and 840 successe. So those failures should not be related to this.

[FireMasterK]

For me, 840 tests completed, 3 failed, 101 skipped

org.schabi.newpipe.extractor.services.youtube.YoutubeKioskExtractorTest$Trending > testName FAILED
    org.schabi.newpipe.extractor.exceptions.ParsingException: Could not get Trending name
        at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeTrendingExtractor.getName(YoutubeTrendingExtractor.java:84)
        at org.schabi.newpipe.extractor.services.youtube.YoutubeKioskExtractorTest$Trending.testName(YoutubeKioskExtractorTest.java:42)

I've seen this exception in Piped too, which uses the same User-Agent, it doesn't happen all the time, see this paste for a similar trace.

org.schabi.newpipe.extractor.services.youtube.search.YoutubeSearchExtractorTest$MetaInfoTest > testMoreRelatedItems FAILED
    java.lang.AssertionError: Next page have errors
      * Could not get duration

        Caused by:
        org.schabi.newpipe.extractor.exceptions.ParsingException: Could not get duration

org.schabi.newpipe.extractor.services.youtube.search.YoutubeSearchExtractorTest$MetaInfoTest > testMetaInfo FAILED
    java.lang.AssertionError

[B0pol]

• I could not reproduce issue with trending page
• for meta info, it seems like https://www.youtube.com/watch?v=-QAY9ojGWyI is not detected as live stream, and I don't find any other way to know if it's a live stream

[B0pol]

for meta info / live stream, it also fails using the old user agent

[Stypox]

@FireMasterK could you explain why updating the user agent improves privacy and resists fingerprinting?

[TobiGr]

Because we use a really outdated user agent. Thus this user agent is not that common anymore and allows to track users via IP and UA.

[B0pol]

Can you reproduce could not get trending name after #582?

[FireMasterK]

Nothing for the past few hours, so I think it has been resolved.

[TobiGr]

Is it also possible to update the UA in the YouTubeCommentsExtractor?

[FireMasterK]

Is it also possible to update the UA in the YouTubeCommentsExtractor?

I have already done that, you requested that before in an earlier comment 🙃

[TobiGr]

Ups. I did not see that my local branch was not in sync with the upstream 😆