Specific URL fails to be downloaded from different galaxy instances

[pcm32]

Describe the bug

The following GTF URL from 10x fails to be downloaded from 3 different galaxy instances (Org, EU, private 22.05 and another private one on a different version). The download works from browsers or wget in the same servers. The file is fairly big (~9 GBs), but the error comes in second, so it doesn't seem to be size related. Or no error message mentions that. I suspect that something in the client definition provided in the HTTP requests by Galaxy might be upsetting the 10x hosting somehow? Or in way at least that wget doesn't upset the same host.

Galaxy Version and/or server at which you observed the bug
Galaxy Version: multiple above 21.x
Commit:

Browser and Operating System
Operating System: macOS and Windows
Browser: Chrome

To Reproduce
Steps to reproduce the behavior:

1. Go to 'upload data'
2. Paste the URL https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz in the Paste/fetch data part

Expected behavior
The file should download correctly, however different errors are shown (HTTP 403, 'fetch_url_allowlist'

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[pcm32]

A colleague here did a small test from the command line with Python and urllib and got 403 Forbidden when trying to fetch
https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz

[pcm32]

And then he tried setting the user-agent to something more common:

>>> import urllib.request
>>> opener = urllib.request.build_opener()
>>> opener.addheaders = [('User-Agent', 'Wget/1.14')]
>>> urllib.request.install_opener(opener)
>>> urllib.request.urlretrieve(
...   "[https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz"](https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz%22),
...   "test.tar.gz")

and that worked. Are we able to influence this somehow without code changes?

[mvdbeek]

Probably not, however blocking the default urllib user agent seems like a bold move from 10x, maybe they're willing to go with something less drastic ? I assume lots of people writing scripts would suffer from this.

#18003 will set the user agent to galaxy/{VERSION}