Entire Zooniverse web site blocked from search engines

[eatyourgreens]

Describe the bug

Since launching the new home page, the www.zooniverse.org domain is blocked from appearing in search engines.

To Reproduce

https://www.zooniverse.org/robots.txt disallows indexing of all URLs on the Zooniverse domain.

You can reproduce the problem by searching for Eclipsing Binary Patrol in DuckDuckGo.
https://duckduckgo.com/?q=eclipsing+binary+patrol&t=iphone&ia=web

Google doesn't seem to be affected.
https://www.google.co.uk/search?q=eclipsing+binary+patrol

Expected behavior

Search engines should be allowed to index Zooniverse, so that people can find new projects.

Additional context

• app-root: Dockerize app-root #6010 added a robots.txt file that blocks deploys from being indexed.
• you can use Google Page Speed Insights to check SEO errors: https://pagespeed.web.dev/analysis/https-www-zooniverse-org/zek3cq1vmw

[eatyourgreens]

Google shows links to Zooniverse. Like DuckDuckGo, the description is blocked.

[lcjohnso]

Adding robots.txt originated from wanting to prevent indexing of staging pages (see #2541). Consistent with previous homepage behavior, the restriction should be removed for the main Zooniverse domain.

[eatyourgreens]

Adding robots.txt originated from wanting to prevent indexing of staging pages (see #2541). Consistent with previous homepage behavior, the restriction should be removed for the main Zooniverse domain.

The robots.txt files I added in #2541 don't actually work, as I didn't publish them to their respective domain roots. 😞

https://fe-project.zooniverse.org/robots.txt returns 404, and that domain can be indexed by Google.

The Readme is available at https://fe-project.zooniverse.org/projects/assets/README.md and https://www.zooniverse.org/projects/assets/README.md so the public directory is being published, as expected. 🤔

[eatyourgreens]

I've opened #6335 to publish /robots.txt for the standalone projects app. My mistake in #2541 was that I published the robots file at /projects/robots.txt.

[snblickhan]

A report from Justin Schell re: Mapping Prejudice (project ID: 3877).

Workflow 25524 is appearing in Google search results via an FEM link (frontend.preview), but MP is a PFE project. This has led to one case where someone found this link by searching "washtenaw Zooniverse" and inadvertently submitted FEM classifications to a PFE project, not realizing there was any difference. See screenshot below:

[eatyourgreens]

@snblickhan https://frontend.preview.zooniverse.org/robots.txt blocks search crawlers, but I think it didn’t exist prior to last week.

[eatyourgreens]

#6340 removes /robots.txt from staging too, so frontend.preview would be crawlable.

[goplayoutside3]

@eatyourgreens do you have a suggestion on how to solve the scenarios you described? We do not want www.zooniverse.org/robots.txt, but we do want frontend.preview.zooniverse.org/robots.txt. How to make that happen?

[eatyourgreens]

Can you selectively add that file to the staging deploy, but not to the production deploy? I haven't thought about this for a long time, but I believe that staging and production use different Docker images.

[eatyourgreens]

A very quick search of the app router docs found the answer.

https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots

[goplayoutside3]

the answer

I don't see any mention in the docs link about selective deployment. Unless you're considering using a robots.js file to look for certain env variables 🤔

Was there a different question you're looking for the answer to?

[eatyourgreens]

I hacked something together very quickly in #6341. It seems to work. That's probably as much as I can do for free.

Copilot is great for small jobs like this. It's free for anyone with a .edu or .ac.uk email address.
https://education.github.com/discount_requests/application

[eatyourgreens]

For pages on frontend.preview.zooniverse.org that have already been indexed by Google, you might need to go into Google Search Console (as owners of that subdomain) and explicitly ask Google to remove the frontend.preview subdomain from the search index.

• https://www.google.co.uk/search?q=washtenaw+zooniverse
• https://www.google.co.uk/search?q=zooniverse+alice+sandbox
• https://www.google.co.uk/search?q=site%3Afrontend.preview.zooniverse.org (project pages start to appear somewhere around pages 7 or 8 of this search.)

DuckDuckGo seems to be better about hiding the staging domain in search results, but has also indexed it:

• https://duckduckgo.com/?q=washtenaw+zooniverse
• https://duckduckgo.com/?q=zooniverse+alice+sandbox
• https://duckduckgo.com/?q=site%3Afrontend.preview.zooniverse.org

[eatyourgreens]

You can also add

<meta name="robots" content="noindex nofollow">

to pages that you don't want to be indexed eg. staged projects.
https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

[goplayoutside3]

Via discussion with @zwolf we decided to use the following strategy:

• fem apps all serve the restrictive version of /robots.txt
• host the permissive one in blob storage
• proxy explicit requests to www.zooniverse.org/robots.txt to the permissive version

See zooniverse/static#380 and #6340

[eatyourgreens]

Looks like static.zooniverse.org is already in the Google index, including some data CSVs.

https://www.google.com/gasearch?q=site:static.zooniverse.org

Including the two-page guide to running a Zooniverse project.
https://static.zooniverse.org/www.citizensciencealliance.org/downloads/zooniverse_guide.pdf

[goplayoutside3]

Fixed by zooniverse/static#380 and Google is re-crawling.