Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

413 Request Entity Too Large with mirror-registry 2.0.3 #179

Open
apodhrad opened this issue Nov 29, 2024 · 10 comments
Open

413 Request Entity Too Large with mirror-registry 2.0.3 #179

apodhrad opened this issue Nov 29, 2024 · 10 comments

Comments

@apodhrad
Copy link

apodhrad commented Nov 29, 2024 •
edited
Loading

When pushing big images to quay installed by mirror-registry 2.0.3 I'm getting this error

413 Request Entity Too Large

Note that this is working with mirror-registry 1.3.5. I will try also other versions.

EDIT: it is working also with 1.3.10
@HammerMeetNail
Copy link
Contributor

Hey @apodhrad, What is the size of the size of the image? Are you able to share it so we can try to recreate?

@LiZhang19817
Copy link

LiZhang19817 commented Dec 2, 2024 •
edited
Loading

@apodhrad We tried to reproduce this issue as you mentioned, but when pushing image with large layer size "9.4GB", can't reproduce this issue, is it possible to share your image to confirm.

$ ./mirror-registry --version

   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers

mirror-registry version v2.0.3

podman push quayomr.qe.devcluster.openshift.com:8443/quay/test123 --tls-verify=false --creds quay:***
Getting image source signatures
Copying blob 2653d992f4ef done   | 
Copying blob 637a3c7a75fa [===>----------------------------------] 1.1GiB / 9.4GiB | 260.1 MiB/s

podman push quayomr.qe.devcluster.openshift.com:8443/quay/test123 --tls-verify=false --creds quay:***
Getting image source signatures
Copying blob 2653d992f4ef done   | 
Copying blob 637a3c7a75fa done   | 
Copying config 76e8b2c5c2 done   | 
Writing manifest to image destination

@LiZhang19817
Copy link

@apodhrad We want to know if you hit the same error as following example.

podman push quayomr.qe.devcluster.openshift.com:8443/quay/ai --tls-verify=false --creds quay:***
Getting image source signatures
Copying blob aa81dcfd55ea [=========================>------------] 20.0GiB / 29.5GiB | 167.9 MiB/s
Copying blob 24b5ce0f1e07 done   | 
Error: writing blob: uploading layer chunked: StatusCode: 413, "<html>\r\n<head><title>413 Request Entity Too Large<..."

@apodhrad
Copy link
Author

apodhrad commented Dec 2, 2024

Hi @LiZhang19817 I hit exactly the same error.
Unfortunately, today with a fresh install I was not able reproduce it :(

@LiZhang19817
Copy link

LiZhang19817 commented Dec 3, 2024 •
edited
Loading

@apodhrad Is it possible to share your test image then we can see if it's the same issue.
Actually there's a config "MAXIMUM_LAYER_SIZE", the default largest layer size is 20GB
https://docs.redhat.com/en/documentation/red_hat_quay/3.13/html-single/configure_red_hat_quay/index#config-fields-storage-fields

@lohbe
Copy link

lohbe commented Feb 14, 2025

I'm getting the same error as well, attempting to mirror OCP 4.17.14 distribution via disk-to-mirror.

2025/02/14 12:43:06  [INFO]   : Mirroring is ongoing. No errors.
 ✓ 97/199 : (19s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50…
 ✗ 98/199 : (23s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b…
 ✗ 99/199 : (23s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75…
 ✗ 100/199 : (23s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a…
101/199 : (23s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ca6b6104dd90…
 ✗ 102/199 : (23s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5…
 ✓ 103/199 : (19s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1…
 ✗ 104/199 : (23s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e…
2025/02/14 12:43:29  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
2025/02/14 12:43:29  [ERROR]  : writing blob: uploading layer chunked: StatusCode: 413, "<html>\r\n<head><title>413 Request Entity Too Large<..."

Using quay from mirror-registry :

$ podman ps -a
CONTAINER ID  IMAGE                                         COMMAND     CREATED       STATUS       PORTS                                       NAMES
8876e15ebc19  registry.access.redhat.com/ubi8/pause:8.10-5  infinity    47 hours ago
  Up 47 hours  0.0.0.0:8443->8443/tcp                      fe354a9444ff-infra
7c512f1d98b3  registry.redhat.io/rhel8/redis-6:1-190        run-redis   47 hours ago  Up 47 hours  0.0.0.0:8443->8443/tcp, 6379/tcp            quay-redis
1d85257f80f0  registry.redhat.io/quay/quay-rhel8:v3.12.5    registry    47 hours ago
  Up 47 hours  0.0.0.0:8443->8443/tcp, 7443/tcp, 8080/tcp  quay-app

imageset-config.yml

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
    - name: stable-4.17 
      minVersion: 4.17.14
      maxVersion: 4.17.14
    graph: true 
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 
      packages: 
       - name: node-observability-operator
  additionalImages: 
   - name: registry.redhat.io/ubi8/ubi:latest
   - name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0

oc-mirror v2 used:

  oc mirror --v2 -c imageset-config.yml file://{{local_cache}}
  oc mirror --v2 -c imageset-config.yml --from file://{{local_cache}} docker://mirror.ocp.lan:8443 --dest-tls-verify=false

@lohbe
Copy link

lohbe commented Feb 14, 2025

adding output of podman logs {{ registry }} | grep -i error

nginx stdout | 2025/02/14 04:43:20 [error] 93#0: *27503 recv() failed (104: Connecti
on reset by peer) while reading response header from upstream, client: 192.168.125.1
, server: _, request: "PATCH /v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/upload
s/8885d0ae-2766-43da-a1c8-af468f3261fd HTTP/1.1", upstream: "http://unix:/tmp/gunico
rn_registry.sock:/v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/uploads/8885d0ae-2
766-43da-a1c8-af468f3261fd", host: "mirror.ocp.lan:8443"
nginx stdout | 2025/02/14 04:43:20 [error] 93#0: *27503 client intended to send too
large body: 11961139 bytes, client: 192.168.125.1, server: _, request: "PATCH /v2/op
enshift-release-dev/ocp-v4.0-art-dev/blobs/uploads/8885d0ae-2766-43da-a1c8-af468f326
1fd HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/openshift-relea
se-dev/ocp-v4.0-art-dev/blobs/uploads/8885d0ae-2766-43da-a1c8-af468f3261fd", host: "
mirror.ocp.lan:8443"
nginx stdout | 2025/02/14 04:43:20 [error] 93#0: *27489 recv() failed (104: Connecti
on reset by peer) while reading response header from upstream, client: 192.168.125.1
, server: _, request: "PATCH /v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/upload
s/5d25eafe-c115-49fc-928c-bef0572f6d69 HTTP/1.1", upstream: "http://unix:/tmp/gunico
rn_registry.sock:/v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/uploads/5d25eafe-c
115-49fc-928c-bef0572f6d69", host: "mirror.ocp.lan:8443"
nginx stdout | 2025/02/14 04:43:20 [error] 93#0: *27489 client intended to send too large body: 97493842 bytes, client: 192.168.125.1, server: _, request: "PATCH /v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/uploads/5d25eafe-c115-49fc-928c-bef0572f6d69 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/uploads/5d25eafe-c115-49fc-928c-bef0572f6d69", host: "mirror.ocp.lan:8443"

@lohbe
Copy link

lohbe commented Feb 14, 2025

Possible oom on the host:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.6Gi       3.5Gi       146Mi        79Mi       250Mi       115Mi

I'll bump to 8GiB instead and see if it makes a difference.

@lohbe
Copy link

lohbe commented Feb 14, 2025

OOM was likely cause of the issue. Issue resolved after providing additional memory to the host.

 ✓ 199/199 : (7s) registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25…
2025/02/14 13:42:56  [INFO]   : === Results ===
2025/02/14 13:42:56  [INFO]   : ✅ 192 / 192 release images mirrored successfully
2025/02/14 13:42:56  [INFO]   : ✅ 5 / 5 operator images mirrored successfully
2025/02/14 13:42:56  [INFO]   : ✅ 2 / 2 additional images mirrored successfully
2025/02/14 13:42:56  [INFO]   : 📄 Generating IDMS file...
2025/02/14 13:42:56  [INFO]   : /home/ben/bin/ocp-4.17/working-dir/cluster-resources
/idms-oc-mirror.yaml file created
2025/02/14 13:42:56  [INFO]   : 📄 Generating ITMS file...
2025/02/14 13:42:56  [INFO]   : /home/ben/bin/ocp-4.17/working-dir/cluster-resources
/itms-oc-mirror.yaml file created
2025/02/14 13:42:56  [INFO]   : 📄 Generating CatalogSource file...
2025/02/14 13:42:56  [INFO]   : /home/ben/bin/ocp-4.17/working-dir/cluster-resources
/cs-redhat-operator-index-v4-17.yaml file created
2025/02/14 13:42:56  [INFO]   : 📄 Generating UpdateService file...
2025/02/14 13:42:56  [INFO]   : /home/ben/bin/ocp-4.17/working-dir/cluster-resources/updateService.yaml file created
2025/02/14 13:42:56  [INFO]   : mirror time     : 7m33.614311242s
2025/02/14 13:42:56  [INFO]   : 👋 Goodbye, thank you for using oc-mirror

@apodhrad - perhaps check memory on the host?

@apodhrad
Copy link
Author

That's a very good point. I will definitely try it. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@HammerMeetNail @apodhrad @lohbe @LiZhang19817