Getting error: `501 Not Implemented` #5057

flarco · 2024-10-04T10:24:48Z

Expected behavior

mc find R2/bucket/staging --name "*.csv.gz" should list the files found

Actual behavior

Now getting:
mc: <ERROR> Unable to stat `R2/bucket/staging`. 501 Not Implemented.

Steps to reproduce the behavior

mc find R2/bucket/staging

I confirmed it works in previous version, broken in latest version.

mc --version

mc version RELEASE.2024-10-02T08-27-28Z (commit-id=ce0b4341521de16ae2172d42054bbe054f9c9651)
Runtime: go1.22.8 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

System information

linux and windows give same error.

The text was updated successfully, but these errors were encountered:

harshavardhana · 2024-10-04T10:26:28Z

mc: <ERROR> Unable to stat `R2/bucket/staging`. 501 Not Implemented.

share --debug

flarco · 2024-10-04T10:35:33Z

Here you go

mc: <ERROR> Unable to stat `R2/bucket/staging`. 501 Not Implemented
 (2) find-main.go:200 cmd.checkFindSyntax(..) Tags: [R2/bucket/staging]
 (1) client-url.go:217 cmd.url2Stat(..) Tags: [R2/bucket/staging]
 (0) client-s3.go:1725 cmd.(*S3Client).getObjectStat(..)
 Commit:ce0b4341521d | Release-Tag:RELEASE.2024-10-02T08-27-28Z | Host:vm | OS:linux | Arch:amd64 | Lang:go1.22.8 | Mem:4.4 MiB/18 MiB | Heap:4.4 MiB/12 MiB.

klauspost · 2024-10-04T10:50:50Z

Post the entire --debug output.

flarco · 2024-10-04T11:00:36Z

mc: <DEBUG> GET /bucket/?location= HTTP/1.1
Host: <account>.r2.cloudflarestorage.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-10-02T08-27-28Z
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=3c3fb1573a19ae6c416ff969840c6bc9/20241004/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20241004T103406Z

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cf-Ray: 8cd4847fb80d2063-IAD
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/xml
Date: Fri, 04 Oct 2024 10:34:06 GMT
Server: cloudflare
Vary: Accept-Encoding

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Cloudflare, Inc.
mc: <DEBUG>  >> Expires: 2024-12-31 23:59:59 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: IE
mc: <DEBUG>  >> Organization: Baltimore
mc: <DEBUG>  >> Expires: 2024-12-31 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  125.027405ms

mc: <DEBUG> HEAD /bucket/staging HTTP/1.1
Host: <account>.r2.cloudflarestorage.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-10-02T08-27-28Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20241004/ENAM/s3/aws4_request, SignedHeaders=host;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Checksum-Mode: ENABLED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241004T103406Z

mc: <DEBUG> HTTP/1.1 501 Not Implemented
Connection: close
Cf-Ray: 8cd4848078a62063-IAD
Connection: keep-alive
Content-Type: text/plain;charset=UTF-8
Date: Fri, 04 Oct 2024 10:34:06 GMT
Server: cloudflare

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Cloudflare, Inc.
mc: <DEBUG>  >> Expires: 2024-12-31 23:59:59 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: IE
mc: <DEBUG>  >> Organization: Baltimore
mc: <DEBUG>  >> Expires: 2024-12-31 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  20.72905ms

mc: <ERROR> Unable to stat `R2/bucket/staging`. 501 Not Implemented
 (2) find-main.go:200 cmd.checkFindSyntax(..) Tags: [R2/bucket/staging]
 (1) client-url.go:217 cmd.url2Stat(..) Tags: [R2/bucket/staging]
 (0) client-s3.go:1725 cmd.(*S3Client).getObjectStat(..)
 Commit:ce0b4341521d | Release-Tag:RELEASE.2024-10-02T08-27-28Z | Host:vm | OS:linux | Arch:amd64 | Lang:go1.22.8 | Mem:4.4 MiB/18 MiB | Heap:4.4 MiB/12 MiB.

harshavardhana · 2024-10-04T11:27:06Z

mc: HEAD /bucket/staging HTTP/1.1
Host: .r2.cloudflarestorage.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-10-02T08-27-28Z
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20241004/ENAM/s3/aws4_request, SignedHeaders=host;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date, Signature=REDACTED
X-Amz-Checksum-Mode: ENABLED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241004T103406Z

mc: HTTP/1.1 501 Not Implemented
Connection: close
Cf-Ray: 8cd4848078a62063-IAD
Connection: keep-alive
Content-Type: text/plain;charset=UTF-8
Date: Fri, 04 Oct 2024 10:34:06 GMT
Server: cloudflare

they don't implement HEAD object? or they don't implement x-amz-checksum-mode.

harshavardhana · 2024-10-04T11:28:33Z

You have to ask R2 to implement x-amz-checksum-mode https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

Flowm · 2024-10-09T23:15:36Z

Sad to see this closed as won't fix. Up until now, mc has been a reliable tool for interfacing with a wide variety of S3 compatible services (including Cloudflare R2) besides connecting to the Minio Server.

With the previous release (RELEASE.2024-09-16T17-43-14Z) downloading files from Cloudflare has been working fine, only with RELEASE.2024-10-02T08-27-28Z this stopped working.

Concretely, the request from the new version is rejected by Cloudflare because of the added x-amz-checksum-mode header in the HEAD request before the file is being downloaded. This field has been added in #5043 and as far as I can tell, there is currently no way to disable it. Any chance for an option to disable this new functionality?

RELEASE.2024-09-16T17-43-14Z Authorization Header

Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/20241009/WEUR/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**

RELEASE.2024-10-02T08-27-28Z Authorization Header

Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/20241009/WEUR/s3/aws4_request, SignedHeaders=host;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**

Debug log of RELEASE.2024-09-16T17-43-14Z HEAD request working fine

mc.RELEASE.2024-09-16T17-43-14Z: <DEBUG> HEAD /XXXXXXXXXXXXXXXXXXXXXXXXXXXXX/108fa9433ea4a75d74eee72d58479efe.tm HTTP/1.1
Host: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.r2.cloudflarestorage.com
User-Agent: MinIO (darwin; arm64) minio-go/v7.0.76 mc.RELEASE.2024-09-16T17-43-14Z/RELEASE.2024-09-16T17-43-14Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/20241009/WEUR/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241009T131902Z

mc.RELEASE.2024-09-16T17-43-14Z: <DEBUG> HTTP/1.1 200 OK
Content-Length: 5666458624
Accept-Ranges: bytes
Cf-Ray: 8cfea8f6bdca193f-FRA
Connection: keep-alive
Date: Wed, 09 Oct 2024 13:19:02 GMT
Etag: "2b9bc65548568af391ca63be7a330e99-676"
Last-Modified: Wed, 02 Oct 2024 12:29:30 GMT
Server: cloudflare
X-Amz-Mp-Parts-Count: 676

Debug log of RELEASE.2024-10-02T08-27-28Z HEAD request failing

mc.RELEASE.2024-10-02T08-27-28Z: <DEBUG> HEAD /XXXXXXXXXXXXXXXXXXXXXXXXXXXXX/108fa9433ea4a75d74eee72d58479efe.tm HTTP/1.1
Host: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.r2.cloudflarestorage.com
User-Agent: MinIO (darwin; arm64) minio-go/v7.0.77 mc.RELEASE.2024-10-02T08-27-28Z/RELEASE.2024-10-02T08-27-28Z
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/20241009/WEUR/s3/aws4_request, SignedHeaders=host;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Checksum-Mode: ENABLED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241009T131908Z

mc.RELEASE.2024-10-02T08-27-28Z: <DEBUG> HTTP/1.1 501 Not Implemented
Connection: close
Cf-Ray: 8cfea91ef884db08-FRA
Connection: keep-alive
Content-Type: text/plain;charset=UTF-8
Date: Wed, 09 Oct 2024 13:19:08 GMT
Server: cloudflare

harshavardhana · 2024-10-09T23:29:13Z

With the previous release (RELEASE.2024-09-16T17-43-14Z) downloading files from Cloudflare has been working fine, only with RELEASE.2024-10-02T08-27-28Z this stopped working.

The source is available. You are free to modify the source and use it as you please.

klauspost · 2024-10-10T08:09:59Z

S3 compatible services

They are not "S3 compatible" if they are only partially compatible.... Bug your service provider - we can't keep the world standing still because they provide incompatible solutions.

flarco added community triage labels Oct 4, 2024

harshavardhana closed this as completed Oct 4, 2024

harshavardhana added wontfix working as intended and removed triage labels Oct 4, 2024

minio locked as resolved and limited conversation to collaborators Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting error: `501 Not Implemented` #5057

Getting error: `501 Not Implemented` #5057

flarco commented Oct 4, 2024

harshavardhana commented Oct 4, 2024

flarco commented Oct 4, 2024

klauspost commented Oct 4, 2024 •

edited

Loading

flarco commented Oct 4, 2024

harshavardhana commented Oct 4, 2024

harshavardhana commented Oct 4, 2024

Flowm commented Oct 9, 2024

harshavardhana commented Oct 9, 2024

klauspost commented Oct 10, 2024

Getting error: 501 Not Implemented #5057

Getting error: 501 Not Implemented #5057

Comments

flarco commented Oct 4, 2024

Expected behavior

Actual behavior

Steps to reproduce the behavior

mc --version

System information

harshavardhana commented Oct 4, 2024

flarco commented Oct 4, 2024

klauspost commented Oct 4, 2024 • edited Loading

flarco commented Oct 4, 2024

harshavardhana commented Oct 4, 2024

harshavardhana commented Oct 4, 2024

Flowm commented Oct 9, 2024

harshavardhana commented Oct 9, 2024

klauspost commented Oct 10, 2024

Getting error: `501 Not Implemented` #5057

Getting error: `501 Not Implemented` #5057

klauspost commented Oct 4, 2024 •

edited

Loading