404 response for various Node.js downloads and pages #5149

twyatt · 2023-03-15T21:20:59Z

URL:

https://nodejs.org/dist/v16.13.0/

Browser Name:

Chrome

Browser Version:

110.0.5481.177

Operating System:

MacOS 12.5.1

How to reproduce the issue:

Attempt to click link to download node-v16.13.0-darwin-x64.tar.gz from https://nodejs.org/dist/v16.13.0/
Expected to download file
Instead, get a 404 page

Using Chome

node-16.13.0-darwin-x64.mov

Using command line

$ wget https://nodejs.org/dist/v16.13.0/node-v16.13.0-darwin-x64.tar.gz

--2023-03-15 14:19:10--  https://nodejs.org/dist/v16.13.0/node-v16.13.0-darwin-x64.tar.gz
Resolving nodejs.org (nodejs.org)... 104.20.23.46, 104.20.22.46
Connecting to nodejs.org (nodejs.org)|104.20.23.46|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-03-15 14:19:10 ERROR 404: Not Found.

The text was updated successfully, but these errors were encountered:

keithlayne · 2023-03-15T21:28:48Z

I only tried one download, but it was a 404 as well. But https://status.nodejs.org/ is all green so I must be HTTPing wrong. 😛

keithlayne · 2023-03-15T21:34:50Z

More seriously, it's the gzip files that seem broken so far. The "download now" link for me (linux x64) uses the xz file...and that link works. Too bad the asdf plugin for node wants the gz version. (Probably should be using xz anyway)

nschonni · 2023-03-15T21:44:28Z

@ovflowd The 404 is returning the new 404 page rather than the old nginx one, so I'm not sure if the new build isn't passing through a request correctly

/cc @nodejs/build

ovflowd · 2023-03-15T21:51:46Z

The only file that is giving 404 is https://nodejs.org/dist/v16.13.0/node-v16.13.0-darwin-x64.tar.gz so far. I've tried all other files of that directory and they're working.

The 404 is returning the new 404 page rather than the old nginx one, so I'm not sure if the new build isn't passing through a request correctly

Afaik we were always for every 404 within the server returning the Website 404. The only change we made was to also add the 404 to non-https requests.

ovflowd · 2023-03-15T21:52:12Z

More seriously, it's the gzip files that seem broken so far. The "download now" link for me (linux x64) uses the xz file...and that link works. Too bad the asdf plugin for node wants the gz version. (Probably should be using xz anyway)

@keithlayne actually no, only one file is 404'ing. All other gz files are working.

ovflowd · 2023-03-15T21:52:34Z

@nodejs/build can we check why only https://nodejs.org/dist/v16.13.0/node-v16.13.0-darwin-x64.tar.gz this file is somehow 404'ing?

blimmer · 2023-03-15T21:52:55Z

I noticed that https://nodejs.org/dist/v18.13.0/node-v18.13.0-linux-arm64.tar.xz is 404ing

but these versions around it are not:

keithlayne · 2023-03-15T21:54:44Z

@ovflowd I was here becasue of this:

 ❯❯❯ asdf install nodejs 18.15.0
Trying to update node-build... ok
Downloading node-v18.15.0-linux-x64.tar.gz...
-> https://nodejs.org/dist/v18.15.0/node-v18.15.0-linux-x64.tar.gz
error: failed to download node-v18.15.0-linux-x64.tar.gz
-> https://nodejs.org/dist/v18.15.0/node-v18.15.0-linux-x64.tar.gz
error: failed to download node-v18.15.0-linux-x64.tar.gz

BUILD FAILED (Pop 22.04 using node-build 4.9.83-92-g23a7606c)

Binary installation failed; try compiling from source with `--compile` flag

So there are more problems than that one file.

ovflowd · 2023-03-15T21:55:59Z

@keithlayne this seems to be a CloudFlare cache issue. You can check by accessing https://direct.nodejs.org/dist/v16.13.0/ which is the direct connection to our server, and all the links there work normally.

I know this doesn't solve your issue, but I'm checking with the team what are our options.

keithlayne · 2023-03-15T21:56:03Z

...and that file just started serving, in the last 15 seconds.

ovflowd · 2023-03-15T21:57:14Z

I can also check that this is being served directly within CloudFlare with curl -v

❯ curl -v https://nodejs.org/dist/v16.13.0/node-v16.13.0-darwin-x64.tar.gz
*   Trying 104.20.22.46:443...
* Connected to nodejs.org (104.20.22.46) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none

As 104.20.22.46 is a CloudFlare IP Address.

ovflowd · 2023-03-15T22:01:20Z

https://nodejs.org/dist/v18.13.0/node-v18.13.0-linux-arm64.tar.xz

https://direct.nodejs.org/dist/v18.13.0/node-v18.13.0-linux-arm64.tar.xz seems to be working fine. So double-down a CloudFlare cache issue.

keithlayne · 2023-03-15T22:05:05Z

@ovflowd That is really good info, thank you! Looks like I can use that url to solve my current pain. Once I started poking around I was starting to see some CF cache headers.

For anyone else who may run into this:

NODEJS_ORG_MIRROR=https://direct.nodejs.org/dist/ asdf install nodejs x.y.z

seems to work - but let's be responsible, node uses a CDN for a reason.

ovflowd · 2023-03-15T22:06:52Z

Please just a friendly reminder, avoid using direct.nodejs.org when possible. Our server sadly was not designed to handle the load that the web usually does to download Node.js 😅

seems to work - but let's be responsible, node uses a CDN for a reason.

Exactly! Thank you for your understanding 🙇

twyatt · 2023-03-15T23:40:54Z

Thank you for the quick response!
At least for me, the Issue appears to have been resolved (I'm now able to download the file I originally reported an issue with).

flash1293 · 2023-03-16T10:54:32Z

Seeing similar errors:

Could not find org.nodejs:node:18.14.2.
Searched in the following locations:
- https://nodejs.org/dist/v18.14.2/node-v18.14.2-linux-x64.tar.gz

rwaite-addland · 2023-03-16T11:46:43Z

This also seems to be affecting the azure devops NodeTool task:

ovflowd · 2023-03-16T12:15:56Z

FYI, we're still monitoring and working on solutions here.

tniessen · 2023-03-16T22:19:32Z

@walec51 @SiWhite I have purged the cache for those files in all physical locations. (But don't actually use those versions, they have long reached end-of-life status and are unsupported. Even 14.x reaches end-of-life next month.)

As I said in my previous comment, some rarely used files, especially those belonging to outdated and unsupported Node.js versions, may be unavailable in some physical locations for a few hours.

ovflowd · 2023-03-17T17:17:19Z

Hey, y'all, as of now, we've done the last fixes that should have handled this incident. We're posting updates on our Status page, and we're going to follow up with a post-mortem afterwards. Thank you!

ashsearle · 2023-03-17T17:18:29Z

Have you changed behaviour of nodejs.org/dist to rate limit and return 403 response code off the back of this incident? (Asking as our CI system is failing due to 403s, not 404s).

Note: 403 response code received for HEAD request.

ovflowd · 2023-03-17T17:21:45Z

I don't think we support HEAD requests for nodejs.org/dist. (Correct me if I'm wrong, @nodejs/build)

ashsearle · 2023-03-17T17:33:44Z

In case of confusion: I meant a HEAD request for a full URL to a specific .tar.gz. AFAIK nothing in our CI system has changed (across multiple different projects)

nschonni · 2023-03-17T17:41:50Z

Is this from Artifactory? There is some filtering being put int place because of problems from a lot of misconfigured instances nodejs/build#3223

ghost · 2023-03-17T18:02:18Z

I'm seeing 403s using Gradle (via faking an Ivy repo) both on CI and on my local development machine trying to hit nodejs.org/dist, and 404s from our Artifactory proxy.

marclefrancois · 2023-03-17T18:05:44Z

+1 for the gradle problem reported by @tridium-gward

ashsearle · 2023-03-17T18:18:20Z

Is this from Artifactory? There is some filtering being put int place because of problems from a lot of misconfigured instances nodejs/build#3223

Our case was from Gradle too (specifically gradle-node-plugin, but I'm guessing the only significance is the plugin leads to a HEAD request, not just a GET.)

ovflowd · 2023-03-17T18:26:53Z

We've just fixed the issue by deploying a new CF Firewall rule.

petty · 2023-03-20T01:19:50Z

I'm still seeing failed builds with respect to Ivy:


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':project-theme-default:nodeSetup'.
> Could not resolve all files for configuration ':project-theme-default:detachedConfiguration1'.
   > Could not resolve org.nodejs:node:16.19.1.
     Required by:
         project :project-theme-default
      > Could not resolve org.nodejs:node:16.19.1.
         > Could not get resource 'https://nodejs.org/dist/v16.19.1/ivy.xml'.
            > Could not GET 'https://nodejs.org/dist/v16.19.1/ivy.xml'. Received status code 403 from server: Forbidden

nschonni · 2023-03-20T01:45:03Z

https://nodejs.org/dist/v16.19.1/ivy.xml

That's a file that has never existed on the Node.js servers. Suggest you open up an issue on the Ivy plugin, as that would be an issue for their plugin

ghost · 2023-03-20T10:33:41Z

@petty Are you creating your own Ivy repo? If so you need to ensure it's configured to only look for artifacts and not metadata (see metadataSources below):

    repositories {
      val nodeRemote = ivy {
        name = "NodeRemote"
        setUrl("https://nodejs.org/dist")
        patternLayout {
          artifact("/[revision]/[module]-[revision](-[classifier])(.[ext])")
        }
        metadataSources {
          // This isn't _actually_ an Ivy repo, so it has no metadata
          artifact()
        }
      }
      // Ensure the repo is a) only used to resolve org.nodejs and b) no other repos can provide
      // any org.nodejs artifacts
      exclusiveContent {
        forRepositories(nodeRemote)
        filter {
          includeGroup("org.nodejs")
        }
      }
    }

tomzi8571 · 2023-03-20T10:42:21Z

Also got the same problem since 17.3.2023:
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':nodeSetup'.
> Could not resolve all files for configuration ':detachedConfiguration1'.
   > Could not resolve org.nodejs:node:12.22.6.
     Required by:
         project :
      > Could not resolve org.nodejs:node:12.22.6.
         > Could not get resource 'https://nodejs.org/dist/v12.22.6/ivy.xml'.
            > Could not GET 'https://nodejs.org/dist/v12.22.6/ivy.xml'. Received status code 403 from server: Forbidden

But also https://nodejs.org/dist/v12.22.6/ returns 500

ovflowd · 2023-03-20T11:45:24Z

@tomzi8571 as explained by @tridium-gward this is not an issue from our side. This file never existed and with the updated configuration it should now be properly handling the requests. So you have a misconfigured setup of your "Ivy"

petty · 2023-03-20T13:16:14Z

@tomzi8571 as explained by @tridium-gward this is not an issue from our side. This file never existed and with the updated configuration it should now be properly handling the requests. So you have a misconfigured setup of your "Ivy"

Ok, but this configuration hasn't changed in years ... and this also broke Friday ...

nschonni · 2023-03-20T13:19:35Z

As a result of triaging this issue, additional blocking rules were setup in nodejs/build#3223. This never worked, but previously didn't hard block as before @petty. This was always a misconfiguration, and won't be rolled back

tniessen · 2023-03-20T13:29:51Z

To add to @nschonni's explanation, this route likely returned 404 before, and your tooling might have silently ignored that particular error code. The new firewall configuration blocks requests from some misconfigured Artifactory/Gradle setups that would otherwise cause unnecessary load on our infrastructure, resulting in status code 403.

petty · 2023-03-20T16:16:15Z

Ok, all good ... I just needed a newer version of the Gradle Node plugin (specifically moving from 1.3.0 to 1.5.3).

-    id 'com.github.node-gradle.node' version '1.3.0'
+    id 'com.github.node-gradle.node' version '1.5.3'

cjiga · 2023-03-20T16:26:25Z

The same problem with version: v10.15.3

`What went wrong:
Execution failed for task ':carelogic-webapp:nodeSetup'.

Could not resolve all dependencies for configuration 'detachedConfiguration1'.
Could not resolve org.nodejs:node:10.15.3.
Required by:
com.qualifacts.carelogic:carelogic-webapp:276.01
> Could not GET 'https://nodejs.org/dist/v10.15.3/ivy.xml'. Received status code 403 from server: Forbidden`

ovflowd · 2023-03-20T17:03:12Z

@cjiga please, read prior comments before posting here.

I'm locking this issue to avoid duplicated comments and pings to the people here.

twyatt added the bug label Mar 15, 2023

ovflowd mentioned this issue Mar 15, 2023

Node 16 Download URL Returns 404 - https://nodejs.org/dist/v16.19.0/node-v16.19.0-linux-x64.tar.gz nodejs/node#47109

Closed

This was referenced Mar 16, 2023

node.lib download links for v19.8.1 (win-x86 and win-x64) are broken #5150

Closed

Download v18.15.tar.xz fails #5151

Closed

ovflowd mentioned this issue Mar 16, 2023

Node some Previous Releases files have disappeared #5152

Closed

This was referenced Mar 16, 2023

Node Dist Index node-v16.17.0-linux-x64.tar.xz 404 nodejs/node#47112

Closed

Azure DevOps Pipeline Task NodeTool@0 Version 18.x results in 404 nodejs/node#47113

Closed

ovflowd mentioned this issue Mar 16, 2023

node 18.10.0 for linux return 404 #5153

Closed

This comment was marked as resolved.

Sign in to view

This was referenced Mar 16, 2023

dist/v14.18.3/node-v14.18.3-linux-x64.tar.gz is inaccessible #5157

Closed

node-v14.19.0-darwin-x64.tar.xz not available for download. 404 not found. nodejs/node#47117

Closed

james64 mentioned this issue Mar 16, 2023

Support install files caching salsita/node_installer#3

Merged

This comment was marked as resolved.

Sign in to view

nschonni mentioned this issue Mar 17, 2023

High number of 404s downloading node in our CI environment nodejs/build#3227

Closed

This was referenced Mar 17, 2023

Create a dedicated staging server for NGINX nodejs/build#3235

Open

Create CI for testing the NGINX config nodejs/build#3236

Open

ovflowd closed this as completed Mar 17, 2023

aSemy mentioned this issue Mar 18, 2023

Fix flaky nodejs dependency Kotlin/dokka#2927

Merged

AbhiPrasad mentioned this issue Mar 20, 2023

can't install node versions on m1 volta-cli/volta#1252

Open

nodejs locked as off-topic and limited conversation to collaborators Mar 20, 2023

ovflowd unpinned this issue Mar 23, 2023

404 response for various Node.js downloads and pages #5149

404 response for various Node.js downloads and pages #5149

Comments

twyatt commented Mar 15, 2023

URL:

Browser Name:

Browser Version:

Operating System:

How to reproduce the issue:

Using Chome

Using command line

keithlayne commented Mar 15, 2023

keithlayne commented Mar 15, 2023

nschonni commented Mar 15, 2023

ovflowd commented Mar 15, 2023

ovflowd commented Mar 15, 2023

ovflowd commented Mar 15, 2023

blimmer commented Mar 15, 2023 • edited Loading

keithlayne commented Mar 15, 2023

ovflowd commented Mar 15, 2023 • edited Loading

keithlayne commented Mar 15, 2023 • edited Loading

ovflowd commented Mar 15, 2023

ovflowd commented Mar 15, 2023

keithlayne commented Mar 15, 2023

ovflowd commented Mar 15, 2023 • edited Loading

twyatt commented Mar 15, 2023 • edited Loading

flash1293 commented Mar 16, 2023

rwaite-addland commented Mar 16, 2023

ovflowd commented Mar 16, 2023

This comment was marked as resolved.

This comment was marked as resolved.

tniessen commented Mar 16, 2023 • edited Loading

ovflowd commented Mar 17, 2023

ashsearle commented Mar 17, 2023 • edited Loading

ovflowd commented Mar 17, 2023 • edited Loading

ashsearle commented Mar 17, 2023

nschonni commented Mar 17, 2023

ghost commented Mar 17, 2023

marclefrancois commented Mar 17, 2023

ashsearle commented Mar 17, 2023

ovflowd commented Mar 17, 2023

petty commented Mar 20, 2023 • edited Loading

nschonni commented Mar 20, 2023

ghost commented Mar 20, 2023

tomzi8571 commented Mar 20, 2023 • edited Loading

ovflowd commented Mar 20, 2023

petty commented Mar 20, 2023

nschonni commented Mar 20, 2023

tniessen commented Mar 20, 2023

petty commented Mar 20, 2023 • edited Loading

cjiga commented Mar 20, 2023

ovflowd commented Mar 20, 2023

blimmer commented Mar 15, 2023 •

edited

Loading

ovflowd commented Mar 15, 2023 •

edited

Loading

keithlayne commented Mar 15, 2023 •

edited

Loading

ovflowd commented Mar 15, 2023 •

edited

Loading

twyatt commented Mar 15, 2023 •

edited

Loading

tniessen commented Mar 16, 2023 •

edited

Loading

ashsearle commented Mar 17, 2023 •

edited

Loading

ovflowd commented Mar 17, 2023 •

edited

Loading

petty commented Mar 20, 2023 •

edited

Loading

tomzi8571 commented Mar 20, 2023 •

edited

Loading

petty commented Mar 20, 2023 •

edited

Loading