Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[EU countries only] YouTube JSON data cannot be fetched due to new EU cookies agreement #592

Closed
AudricV opened this issue Mar 30, 2021 · 9 comments
Closed

[EU countries only] YouTube JSON data cannot be fetched due to new EU cookies agreement #592

AudricV opened this issue Mar 30, 2021 · 9 comments
Labels
ASAP Issue needs to be fixed as soon as possible youtube service, https://www.youtube.com/

Comments

@AudricV
Copy link
Member

AudricV commented Mar 30, 2021 •
edited
Loading

Google change the way for consent about YouTube cookies agreement in EU countries. Instead of showing a popup in the website, YouTube redirects the user to a new webpage at consent.youtube.com domain. It also affects YouTube JSON fetching data.
Issue seems to be more reproducible now, it's in A/B testing.

Fix for this is to put a cookie CONSENT with YES+ value for every YouTube request, excepting images, video or audios of course.

Or we can also move JSON requests to youtubei.googleapis.com, but all requests are POST (not tested if it works, I am not really sure).

Reported a while ago on Reddit: https://www.reddit.com/r/uBlockOrigin/comments/lz4ibd/new_fullscreen_before_you_continue_to_youtube_page/
@AudricV AudricV added ASAP Issue needs to be fixed as soon as possible youtube service, https://www.youtube.com/ labels Mar 30, 2021
@XiangRongLin
Copy link
Collaborator

@TiA4f8R

  1. Which extractors does this affect?
  2. Can you work on this? For me the app still works as normal, meaning i can't

@XiangRongLin XiangRongLin mentioned this issue Mar 30, 2021
Closed
@AudricV
Copy link
Member Author

AudricV commented Mar 30, 2021

For now, channels and playlists on my side.
But it affects every request to the website, according to some posts in uBlock Origin subreddit.

@AudricV
Copy link
Member Author

AudricV commented Mar 30, 2021

Can you work on this? For me the app still works as normal, meaning i can't

No, I can't, I don't have free time right now.

@TheFrenchGhosty TheFrenchGhosty mentioned this issue Mar 31, 2021
Closed
@TobiGr
Copy link
Contributor

TobiGr commented Apr 1, 2021

I was able to reproduce this. With all choices set to "No", I get this CONSENT cookie YES+cb.20210328-17-p0.de+F+068. The last three digits change randomly.
I think we should add proper cookie handling to the downloader, just like in the DownloaderImpl. I also do not have time for this

@TobiGr TobiGr mentioned this issue Apr 1, 2021
Closed
@B0pol
Copy link
Member

B0pol commented Apr 1, 2021 •
edited
Loading

See ytdl-org/youtube-dl#28628, yt-dlp/yt-dlp#210 for rejecting tracking

@B0pol
Copy link
Member

B0pol commented Apr 2, 2021

Weirdly some people are not affected by this (and living in EU countries) using NewPipe. We probably should only add the CONSENT cookie if needed, i.e. if redirected to consent.youtube.com

@TobiGr
Copy link
Contributor

TobiGr commented Apr 2, 2021

Can you implement that?

@B0pol
Copy link
Member

B0pol commented Apr 2, 2021

no, i'm not affected, it works fine on my newpipe

@AudricV
Copy link
Member Author

AudricV commented Apr 2, 2021

Also, NewPipe already get redirects from consent.youtube.com to youtube.com with a additional param, with a cookie CONSENT which has the PENDING+something value. So it's really strange.

@TobiGr TobiGr mentioned this issue Apr 3, 2021
Merged
3 tasks
return42 pushed a commit to return42/searxng that referenced this issue Jul 25, 2022
@unixfox @return42
[fix] google & youtube - set EU consent cookie
5fb2071 
This change the previous bypass method for Google consent using
``ucbcb=1`` (6face21) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] iv-org/invidious#2207
[2] TeamNewPipe/NewPipeExtractor#592

Closes: searxng#1432
@unixfox unixfox mentioned this issue Jul 25, 2022
Open
kvch pushed a commit to kvch/searx that referenced this issue Jul 30, 2022
@unixfox @kvch
[fix] google & youtube - set EU consent cookie
7123aa1 
This change the previous bypass method for Google consent using
``ucbcb=1`` (6face21) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] iv-org/invidious#2207
[2] TeamNewPipe/NewPipeExtractor#592

Closes: searxng/searxng#1432
kvch added a commit to searx/searx that referenced this issue Jul 30, 2022
@kvch @tiekoetter
@unixfox @return42 @liimee @dalf
pick engine fixes (#3306)
05fe2ee 
* [fix] google engine: results XPath

* [fix] google & youtube - set EU consent cookie

This change the previous bypass method for Google consent using
``ucbcb=1`` (6face21) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] iv-org/invidious#2207
[2] TeamNewPipe/NewPipeExtractor#592

Closes: searxng/searxng#1432

* [fix] sjp engine - convert enginename to a latin1 compliance name

The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: searxng/searxng#1544
Signed-off-by: Markus Heiser <[email protected]>

* [fix] engine tineye: handle 422 response of not supported img format

Closes: searxng/searxng#1449
Signed-off-by: Markus Heiser <[email protected]>

* bypass google consent with ucbcb=1

* [mod] Adds Lingva translate engine

Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.

* openstreetmap engine: return the localized named.

For example: display "Tokyo" instead of "東京都" when the language is English.

* [fix] engines/openstreetmap.py typo: user_langage --> user_language

Signed-off-by: Markus Heiser <[email protected]>

* Wikidata engine: ignore dummy entities

* Wikidata engine: minor change of the SPARQL request

The engine can be slow especially when the query won't return any answer.
See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to

Co-authored-by: Léon Tiekötter <[email protected]>
Co-authored-by: Emilien Devos <[email protected]>
Co-authored-by: Markus Heiser <[email protected]>
Co-authored-by: Emilien Devos <[email protected]>
Co-authored-by: ta <[email protected]>
Co-authored-by: Alexandre Flament <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ASAP Issue needs to be fixed as soon as possible youtube service, https://www.youtube.com/
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@TobiGr @XiangRongLin @B0pol @AudricV