Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make robots.txt restrictive by default #3905

Merged
merged 2 commits into from
Mar 19, 2024
Merged

make robots.txt restrictive by default #3905

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ea75b83
make robots.txt restrictive by default
ruslandoga Mar 17, 2024
a0bc05f
add note about cloud robots.txt
ruslandoga Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ All notable changes to this project will be documented in this file.
- Add `CLICKHOUSE_MAX_BUFFER_SIZE_BYTES` env var which defaults to `100000` (100KB)
- Add alternative SMTP adapter plausible/analytics#3654
- Add `EXTRA_CONFIG_PATH` env var to specify extra Elixir config plausible/analytics#3906
- Add restrictive `robots.txt` for self-hosted plausible/analytics#3905

### Removed
- Removed the nested custom event property breakdown UI when filtering by a goal in Goal Conversions
Expand Down
12 changes: 11 additions & 1 deletion lib/plausible_web/endpoint.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,21 @@ defmodule PlausibleWeb.Endpoint do
plug(PlausibleWeb.Tracker)
plug(PlausibleWeb.Favicon)

static_paths = ~w(css js images favicon.ico)

static_paths =
on_full_build do
# NOTE: The Cloud uses custom robots.txt from https://github.com/plausible/website: https://plausible.io/robots.txt
static_paths
Copy link
Contributor Author

@ruslandoga ruslandoga Mar 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://chat.openai.com/share/aa80093e-3f71-43af-9f48-87d7c622311d

I wonder what the solution should be for the Cloud version. Is omitting robots.txt OK or should we COPY different files when building the container image?

Cloud (to keep the previous behavior)

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

Self-hosted

User-agent: *
Disallow: /

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current robots.txt isn't event used in prod because plausible/website serves its own one which the reverse proxy prefers: https://plausible.io/robots.txt

It should be fine to omit. But we should add a comment/doc line for some clarity. We never want CE to be indexed but in Cloud we do want pages like public dashboards and live demo to be indexed. This is controlled via X-Robots-Tag header rather than robots.txt file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then instead of omitting, would it be OK to simply use

User-agent: *
Disallow: /

for both? With a comment, of course.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this c71c114

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the original solution of omitting it. It's quite likely we will move off the Jekyll-based static site in the future. And in that case, if we forget about robots.txt, we might end up in a place where we disallow all indexing for plausible.io which would be bad.

I know omitting adds more code, but it feels safer to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about copying different robots.txt into the container image based on some ARG? So that EE would get

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

and CE would get

User-agent: *
Disallow: /

Copy link
Contributor Author

@ruslandoga ruslandoga Mar 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or having it configured with app env and served by a custom plug.

# in endpoint.ex
plug :serve_robots_txt

on_full_build do
# ...
else
  def robots_txt do
    # or app env lookup
    """
    User-agent: *
    Disallow: /
    """
  end
end

def serve_robots_txt(conn, _opts) do
  case conn.path_info do
    ["robotx.txt"] -> conn |> send_resp(200, robots_txt()) |> halt()
    _other -> conn
  end
end

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think omitting, the way you've done, is good enough for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok! I've added a note about the Cloud's robots.txt in 11409f3

else
static_paths ++ ["robots.txt"]
end

plug(Plug.Static,
at: "/",
from: :plausible,
gzip: false,
only: ~w(css js images favicon.ico robots.txt)
only: static_paths
)

on_full_build do
Expand Down
7 changes: 2 additions & 5 deletions priv/static/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,2 @@
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /
User-agent: *
Disallow: /