Delete images (repository + data) from registry

[sourav82]

What are the ways to delete the pushed images and repositories from registry?

I guess, Docker does not provide any API to do that operation. Any ways, may be using curl, to delete image data from registry?

[squarism]

I don't have an answer but have a similar question.

I pushed the busybox image to my private 2.0 registry (distribution).
This will work for me. I get a huge json payload.

curl -XGET http://private-host:5000/v2/busybox/manifests/latest
# GET included for effect only  :)

The API doc says that DELETE should work. So I just substitute above and get

curl -XDELETE http://private-host:5000/v2/busybox/manifests/latest                                                                                            
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

So I think that's your answer but it's not working for me.

[dmp42]

Delete will remove the tag, but not the actual payloads.
So, your image will no longer be available, it will still exist on your backend storage.

[dmp42]

Some people have contributed garbage-collector scripts to do the cleanup. You will find them clicking the "delete" label over here.

[squarism]

I think the issue here is that I'm getting a method not supported on the DELETE request not that the blobs are sticking around. If you have an example curl example, that's be great. I could be doing something wrong here. If the OP disagrees, I'll open another issue.

[dmp42]

@squarism so, you are using the golang registry2.

Please head over to https://github.com/docker/distribution and/or irc #docker-distribution

TLDR: delete is not supported yet, in design phase: distribution/distribution#210

[squarism]

Aaah! And I blogged the url to the distribution repo too! I'm sorry, I knew the difference, I failed to see what issue list I was in. 😭 Thanks for the tldr too. 👍

[dmp42]

;)

[alwqx]

+1 with my operation and output:

adolph@geek:docker_registry$ curl -v -X DELETE http://localhost:5000/v2/busybox/manifests/sha256:0fc02bc170932a0d727de2201c65bfe3fe31448dfed6f6bcbc1b37c2668927f5
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> DELETE /v2/busybox/manifests/sha256:0fc02bc170932a0d727de2201c65bfe3fe31448dfed6f6bcbc1b37c2668927f5 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: */*
> 
< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Dec 2015 05:57:29 GMT
< Content-Length: 78
< 
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
* Connection #0 to host localhost left intact
adolph@geek:docker_registry$ curl -v -X DELETE http://localhost:5000/v2/busybox/manifests/list
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> DELETE /v2/busybox/manifests/list HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: */*
> 
< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Dec 2015 05:59:35 GMT
< Content-Length: 78
< 
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
* Connection #0 to host localhost left intact

[CodeJuan]

I sent the same request with @adolphlwq 's request, and got the same response

curl -v -X DELETE http://myregistry/v2/busybox/manifests/sha256:blablabla...

{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

update

I got the solution to delete images

enable delete

set the environment variable REGISTRY_STORAGE_DELETE_ENABLED = True

the API to delete image

1. get the manifest from registry

get v2/<repoName>/manifests/<tagName>

the Docker-Content-Digest is response.Header["Docker-Content-Digest"]
the layerDigests is response.body["fsLayers"]["blobSum"]

1. delete layerDigests

delete v2/<repoName>/blobs/<layerDigests>

1. delete Docker-Content-Digest

delete v2/<repoName>/manifests/<Docker-Content-Digest>

1. then pull the image from registry, the response is invalid character '<' looking for beginning of value

But when I get 'v2/repoName/tags/list', the tag which was been deleted is still exist.......

[mphanikumars]

Hi,

I am very new to Docker.

I have tried like this "curl -X DELETE localhost:5000/v1/repositories/jenkins_ora_jdk/tags/latest" but this works for tag deletion and not the image data for that tag. So i had to write my own script to do that Job. Since i am only the one who tested this script please use with your own risk

This is how i start registry image
mkdir -p /docker/registry/latest && docker run -d -p 5000:5000 -v /docker/registry/latest:/tmp registry

To delete an Image with tag : (This will also cleans the data specific to that tag)

./docker_registry_delete_image -d 5000 "jenkins_ora_jdk:latest"

To see all the Images with tag :

./docker_registry_delete_image -q 5000

: 5000 is Registry PORT

# NOTE

1. Registry service must be running

2. Volume must be mounted locally ( -v /docker/registry/latest:/tmp )

3. This works only in Linux environment

4. This should be executed in where docker registry is running

docker_registry_delete_image.sh.txt

Thanks,
Phani Kumar

[lioncui]

@CodeJuan You must edit registry config.xml with

storage:
    delete:
        enabled: true

reference https://docs.docker.com/registry/spec/api/

[slartibart70]

Hi,
just adding my experiences:

• started registry:2 with my own config (deleting allowed) and own volume mounted storage.
• Pushing images into my registry works ok, my storage contains the data (approx. 100MB)

Now trying to delete:

• curl -v -s -X DELETE "localhost:5000/v2/collctor/blobs/sha256:cbb2acf4..." (see layerDigests above)
• curl -v -s -X DELETE "localhost:5000/v2/collctor/manifests/sha256:66846..." (see Docker-Content-Digest above)

After the above steps, we still have an entry in the catalog:
curl -v -s -X GET "localhost:5000/v2/_catalog" .. which still returns the deleted image
curl -s -X GET localhost:5000/v2//manifests/latest still returns seemingly valid data
but
a docker pull localhost:5000/ does not work any more.
And, the storage space (my ~100MB) are still in use and are not deleted.

Any ideas?

[dmp42]

This repo here is deprecated, and any message here is likely to go unnoticed.
Either report problems on the docker/distribution repo, or on irc (#docker-distribution)

[EugenMayer]

I finally go teached about storage->delete->true in the configuration https://github.com/docker/distribution/blob/master/docs/configuration.md

and implemented it. Still, while fetching the digest https://github.com/EugenMayer/docker_registry_cli/blob/master/DockerRegistryRequest.rb#L92 getting a string like sha256:XXXX

Using this to feed into delete https://github.com/EugenMayer/docker_registry_cli/blob/master/DockerRegistryRequest.rb#L83
i get notfied, that the digest is wrong.

Using the current latest registry:2 (sha256:20f5d95004b71fe14dbe7468eff33f18ee7fa52502423c5d107d4fb0abb05c1d).

Is this a bug / fixed already?

[jshapiro26]

@EugenMayer I've been messing around with this too. It looks like the digest that you return as a header from: get v2/<repoName>/manifests/<tagName> is the wrong digest.

Instead, if you docker pull that image from your registry and grab the Digest it spits out after downloading you should should be able to delete using that with:

curl -u <username:password> -X DELETE https://yourregistry/v2/<repository>/manifests/<newdigest>

If anyone knows of a simple way to get this digest other than pulling the image, please let me know.

Once that is deleted, run garbage collection dry run and then garbage collect on your registry to actually delete the data that you have marked for deletion. On the docker host that is running your registry, run:
docker exec -it <name of container or container ID> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml. It shows that you have N blobs eligible for deletion. Run the command again without --dry-run and it will delete the blobs. Now that they're deleted, if you run the dry-run again you should see that the namespace you deleted has no more blobs.

[EugenMayer]

as requested i created a new issue #1068, maybe you continue work with me there @jshapiro26

[EugenMayer]

ups, used the legacy issue, @jshapiro26 see distribution/distribution#1637 instead

[sergey-tkalych]

My approach how to delete image tag from docker registry (tested on registry version 2.4.0):

1) Update registry configuration file (config.yml) and restart registry

storage:
    delete:
        enabled: true

2) Get tags list

GET ://<registry_host>/v2/<repo_name>/tags/list

3) Get manifest for selected tag

GET ://<registry_host>/v2/<repo_name>/manifests/<tag_name>

4) Copy digest hash from response header

Docker-Content-Digest: <digest_hash>

5) Delete manifest (soft delete). This request only marks image tag as deleted and doesn't delete files from file system. If you want to delete data from file system, run this step and go to the next step

DELETE ://<registry_host>/v2/<repo_name>/manifests/<digest_hash>

Note! You must set headers for request - Accept: application/vnd.docker.distribution.manifest.v2+json

6) Delete image data from file system

Run command from the host machine:

docker exec -it <registry_container_id> bin/registry garbage-collect <path_to_registry_config>

Note! Usually, <path_to_registry_config>=/etc/docker/registry/config.yml

[EugenMayer]

Yes, i already implemented this in my docker_registry_cli:
https://github.com/EugenMayer/docker_registry_cli/blob/master/commands/DockerRegistryCommand.rb#L57

So this works - one of the things i missed was the Accept header: application/vnd.docker.distribution.manifest.v2+json]

But other then that, it works quiet butifully

Hint: dont forget to run the garbage collector to clean up the space

[lvthillo]

There is no API available to trigger the GC so I think it's the only possibility to use docker exec .. ? I'm able to put it in a cron job or something but I don't know how I can get my registry in read-only mode, execute it and disable read-only (we need to push images to the registry, but it has to be forbidden when the GC runs). Or is the only solution to stop the container, start it as read-only, docker exec .., stop container and start it in a normal way? This would take "a lot of" time I assume. (I want to run the GC every time we delete an image)

[EugenMayer]

For, the GC is utterly broken in 2.5 - i would assume the registry being completely useless after more then 2 months - you have to wipe it a redo all images. There are severe issues with blobs, pushed will not work because "the layers are already up to date". Using the GC is completely useless, the only thing actually really kinda works is going to /var/lib/registry/repos.. and delete the folder manually - restart the registry container.

[jshapiro26]

@lorenzvth7 and @EugenMayer, I ended up writing an interactive script for the process I mentioned above and will share it soon and post back here, maybe it'll be useful for someone. The issue i've had is that my team typically just uses the latest tag. The latest tag then starts bloating and there isn't a good way to clean up old versions of latest.

I ended up just parsing the output of the --dry-run garbage collection command and comparing the SHAs against the SHA I get from latest when I run a docker pull. This is because Docker-Content-Digest: <digest_hash> never seems to be correct for the DELETE call to the API.

I noticed your comment about the header: Accept header: application/vnd.docker.distribution.manifest.v2+json and will check that out. Maybe it'll solve all the problems!

[lvthillo]

@jshapiro26 I'm able to get the right sha of my image with tag:latest.
I'm using this command:

curl -k -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://myreg/v2/proj2/base/manifests/latest 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'
sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe

Check in my registry

$ docker exec -it 86deeac4f25f /bin/sh
/ # cd /var/lib/registry/docker/registry/v2/repositories/proj2/base/_manifests/revisions/sha256/52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe/

Hope it helps for your. My issue is the following now:
I delete the sha:
curl -k -v --silent -X DELETE https://myreg/v2/proj2/base/manifests/sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe

This is going fine:

> Accept: */*
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0

I use the GC to clean up:

docker exec -it 86deeac4f25f bin/registry garbage-collect  /etc/docker/registry/config.yml
$ docker exec -it 86deeac4f25f bin/registry garbage-collect /etc/docker/registry/config.yml
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c8/c8144262002cd241e607d7d3ecda450ce4ae8edf7dac8dbf46897d498ac667d8  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/ce/cee0974db2b868f0408f7e3eaba93c11fce3a38f612674477653b04c10369da0  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/06/064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/39/390957b2f4f0cd72b8577795cd8076cdc21d45c7823bbb5c895a494ae6038267  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3d1d7dffe2da492324b1fd62a2939ddc6e50549746cefa9a67684253dfa40b88  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/52/52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/6b/6bc128ff8a43e6b26862384494bd2cbfabc5e9d5277121881cbd0fcbf1442508  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/a5/a52d570eb492f9fb3cebb345c96395fc6156eef00ac42758e1a2d0f4190ecbf6  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392

Check (seems good to me):

/var/lib/registry/docker/registry/v2 # du -sh
63.0K

But now the real issue is repushing:

The push refers to a repository [myreg/proj2/base]
5d3accbaa61c: Layer already exists
3af563f9cad6: Layer already exists
447f88c8358f: Layer already exists
df9a135a6949: Layer already exists
dbaa8ea1faf9: Layer already exists
8a14f84e5837: Layer already exists
latest: digest: sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe size: 1576

Everything seems to exist already in the registry (isn't it cleaned up?).
The sha is the same which seems normal to me (it's the same image).
But checking in the registry?

/var/lib/registry/docker/registry/v2 # du -sh
65.0K

The most of space is taken in /bin/registry

/bin # du -sh registry
26.0M   registry

Something I don't understand.
after deleting the sha and rerunning the GC nothing seems to happen (the tag and _manifest is gone in the registry, but it seems not necessary to delete layers again, the space in / remains the same?).

[jshapiro26]

Thanks for the clarification about sending the header! I'll try that out. It sounds like you likely have an older version of that image that would be untagged now since you've deleted latest. If you remove that image again then ssh to your registry and run docker exec -it <container ID or name> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml does it return any SHAs or blobs under myreg/proj2/base? That might be where the "Layer already exists" is coming from as lots of layers may have been unchanged.

The script i'm cleaning up goes through and deletes all old SHAs for a specified repo, leaving the one tagged as latest in-tact.

[lvthillo]

I pushed the image only with the latest tag. Than I'm able to delete it correctly but after repushing just the same image with just the same tag it goes wrong. Everything seems to exist, All the folders in /var/lib/registry/docker/registry/v2/blobs/sha256/xx are empty. There is a new tag and a new manifest in the registry but it does not take any space. Even pulling the image from the registry does not work..

When I push another image with the same name and same tag (also latest in this case) it's pushing in a fine way (I build it with --no-cache, don't know if it's relevant). I'm able to pull it and everything so this issue only seems to appear when I try to push, delete, repush and pull the exactly same image.

Maybe it's all a bit unclear but to resume:
It seems to work fine for me (pushing, pulling, deleting, clearing) in any case except when I delete image:x and repush image:x (which is the same image, not only the same name). After the "repush" everything still seems to exist but I'm unable to pull.

[zoobab]

@jshapiro26 your ruby script suffers from the same defect as @EugenMayer mentioned. When I repush the same image, I get the same message "Image already exists", and when I try to pull it from the repo, I got the error "Error: image ubuntu not found".

[jshapiro26]

@zoobab What image tag of registry are you using? I am using registry:2.4.1 and have never ran into the issue you're referring to. Also, do you run the garbage collection without --dry-run after you use the script and before pushing again? The script doesn't actually run the garbage collection after all the images and blobs are marked. I solve this by have a cron run the garbage collection from the docker host early in the morning everyday.

To clarify what I wrote the script for, it is used to remove all versions of an image that are not tagged "latest". Therefore if you're pushing an image with a tag that's called "6.2" or something like that, those images will be removed in addition to all the untagged versions of "latest" leaving only the image with the latest tag for the repo you're "cleaning".

I am unable to replicate what you're seeing @zoobab with this script. I use it daily and have an automated build of new images weekly that push successfully. Can you give me some more details so that I might be able to help?

[zoobab]

@jshapiro26 yes, I clean up after without --dry-run so that the disk siez diminishes (which is my goal here). My docker registry version is 2.5.0. When I repush the image after the garbage cleanup, I have this error "Layer already exists".

[jshapiro26]

are you pushing a new build of the image with a different sha? The script leaves the image with the latest tag in-tact on the registry. If you are pushing the same image up it will throw the error you're seeing.

[zoobab]

@jshapiro26 no, I am trying to repush an old image that was deleted, which has the same sha. In your case, you are repushing "latest" which has a new sha because it was freshly built.

[eesprit]

@zoobab did you restart the registry after the garbage collection ?
I have the same behavior as you do if I run the registry GC and don't restart it. After it is restarted, no problem (which is totally understandable as the GC is done outside from the server process, which might have some cache).

[zoobab]

@eesprit I can confirm a restart of the registry solves the "docker push oldimage" problem!

[zoobab]

I thought maybe purging the upload cache would be the root cause of the "docker push oldimage" problem? Is there a way to purge this cache? I have seen some bits in the config.yml file.

maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false

There is no way to purge the cache manually?

[ghost]

I'm trying to delete following your instructions, but get a 403 Forbidden error.

I've set the REGISTRY_STORAGE_DELETE_ENABLED: true in the yaml file that starts up the registry.

[TranceMaster86]

I builded a docker-reg-gc shellscript for that stuff.

https://github.com/TranceMaster86/docker-reg-gc

[lvthillo]

Is there someone who knows what's the best way to delete unused revisions?
As far as I understand the normal way is to delete a revision based on its tag and after that you're able to run the GC which will clean all the rest.

But in DEV we are retagging an image on the same tag (latest). So if we try to use our delete-job (where you have to specify a tag) it only deletes one image (the newest/latest).
All the rest is still there because their revisions aren't deleted.

[TranceMaster86]

Look at my script there https://github.com/TranceMaster86/docker-reg-gc
It removes everything that is older than 150 days....or the count of days you want it to do. So the newst things are left in the registry

[josselinchevalay]

hi do you see that : https://docs.docker.com/registry/spec/api/#/deleting-an-image

[smashingx1]

Ive been using reg (https://github.com/jessfraz/reg/tree/master/vendor) to mark the blobs I want to delete from the registry by doing:

docker run --rm reg/delete -u username -p password --registry 
myregistry.com rm image

I get the output of:

Deleted image@sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

Then I ran the command in my server where the docker registry container is, to run the garbage collector like:

     sudo docker exec -it registry bin/registry garbage-
collect --dry-run /config/config.yml

And I got a lot of output back like:

DEBU[0007] filesystem.Stat("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=15.631837ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=6977da41-fd94-4926-99af-059821e18046 trace.line=137
account-portal: marking manifest sha256:17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee
DEBU[0007] (*manifestStore).Get                          environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry
DEBU[0007] filesystem.GetContent("/docker/registry/v2/repositories/account-portal/_manifests/revisions/sha256/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/link")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=86.1µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=ebd03857-56b6-4957-b791-9c48fdb152e9 trace.line=82
DEBU[0007] filesystem.Stat("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=74.19µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=96520ca1-6ba1-470c-869e-cf0a49846390 trace.line=137
DEBU[0007] filesystem.GetContent("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=20.820776ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=6752e130-1b77-49c2-adda-3b9195aac1df trace.line=82
DEBU[0007] (*schema2ManifestHandler).Unmarshal           environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry
account-portal: marking blob sha256:90769d06ffb7dd1f8d1bc4f7b18e52fb94b651191640b680f028806622b99236
account-portal: marking blob sha256:3b8d91fe75724cafe136236e31d5212cb322d68aa3bfc4bfda97a0a53a55e234
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:dfc6505cfad4d185913ccd772541a0d32e64d4e04d9c249728dd7e194c77340f
account-portal: marking blob sha256:47b142d56ce3cc7766143d5fcd836541dac5006fa66076d4261e9e3d4a24a0cd
account-portal: marking blob sha256:6397456b1db08c3b5fb104497d15eb849a317c35ffeb18b608e6013593b82300
account-portal: marking blob sha256:f1074061d22a0681d732d72a8d11abf0a2f77ee3652ee827c62d696c19697a93
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:7b60f5a7551c3bb15dcd5ecd46a2ec18118cc652f6bdbff23bf16c895a8a1586
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:3b36e36d988def39cf035c9a82076bbff95861bd9092701516d937f7dff24a73
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:283fbeeb0d74113ed38891d145bcbff04a96178d2f1e19bb9dc5564ebd1959ab
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:68ad5d9d0e66645895611177d50dfc46bf1bbf9cd1275e43da416af0d2b2c30a
account-portal: marking blob sha256:6fae118190fe88ba8487a3a2cfac7b1f2d922d313963cd81329fd37b2516ca3d
account-portal: marking blob sha256:9e14db6259dc2696e6fe816ddb106ec1810a4f39075695c66cbeccd58f54aad1

and at the end I get the recap like this:

12530 blobs marked, 0 blobs eligible for deletion

So according to the output no images were removed from my registry. I confirmed by running df -h and yes I still using the same amount of disk space.

So, I don't know what I'm doing wrong, I've followed all the instructions in the docker documentation about the Garbage Collector I don't know what do to do get space back, please help.

[knowit-tuomo]

@messyhead,

I also got 403 responses for the DELETE requests. Turns out I was making the requests through the registry-frontend, which silently overwrote the config variable that should have enabled the DELETE HTTP verb.

The solution was to speak to the registry API directly instead of through the frontend. (ie. port 5000 instead of 80 or 443.)

[ggop]

I followed the steps in 224280919 and 224325910. However it ends up deleting all tags for a repository instead of just one. Garbage collection removes some unused layers as expected and the next push with a new set of tags only shows the new tags set, not the ones from earlier. Furthermore, after I push a image with newer tags, they can't be listed. Let me illustrate, please let me know what I'm doing something wrong here.

/ # /bin/registry --version
/bin/registry github.com/docker/distribution v2.6.1

$ curl -s  http://example.com:5000/v2/myrepo/tags/list | python -m json.tool
{
    "name": "myrepo",
    "tags": [
        "latest",
        "2017-05-16-19-02-latest",
        "2017-05-16-19-04-latest"
    ]
}

$ curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -D - \
  -o /dev/null http://example.com:5000/v2/myrepo/manifests/2017-05-16-19-02-latest 
HTTP/1.1 200 OK
Content-Length: 1970
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b"
X-Content-Type-Options: nosniff
Date: Wed, 17 May 2017 02:24:06 GMT

$ curl -vs -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X DELETE \
  http://example.com:5000/v2/myrepo/manifests/sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b
*   Trying 10.10.10.10...
* TCP_NODELAY set
* Connected to example.com (10.10.10.10) port 5000 (#0)
> DELETE /v2/myrepo/manifests/sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b HTTP/1.1
> Host: example.com:5000
> User-Agent: curl/7.53.1
> Accept: application/vnd.docker.distribution.manifest.v2+json
> 
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Wed, 17 May 2017 02:24:32 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host example.com left intact

# ALL tags are gone, was expecting to see the remaining two
$ curl -s http://example.com:5000/v2/myrepo/tags/list| python -m json.tool
{
    "name": "myrepo",
    "tags": null
}

Now push a new image with some tags. The tags are listed but can't be queried.

$ curl -s  http://example.com:5000/v2/myrepo/tags/list| python -m json.tool
{
    "name": "myrepo",
    "tags": [
        "latest",
        "2017-05-16-19-39-latest"
    ]
}

$ curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -D - -o /dev/null http://example.com:5000/v2/myrepo/manifests/2017-05-16-19-39-latest
HTTP/1.1 404 Not Found
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 17 May 2017 03:04:48 GMT
Content-Length: 184

[ggop]

An addendum to my earlier post, two points:

• The entire process from my previous note worked flawlessly with registry:2.5.1 (in the last step I am able to fetch the digest for the newly uploaded images)
• the reason 3 tags disappeared when I deleted one digest was that they were all referring to the same image. My understanding earlier was that images hang around till all relevant tags are deleted. Turns out it is the other way around, tags go away when the image is deleted.

Anyway it's fine now and I will use 2.5.1 for now

[erikbsap]

what kind of b.s. is this again? this is a basic function of any tool.

[tarasinf]

@sergey-tkalych How to set up the

storage:
    delete:
        enabled: true

if it is gitlab.com?

[tarasinf]

@CodeJuan can you help with deleting?
How to get token with delete rights? As I understand, we need to set sope:pull, push,<some-delete-keyword>, is it?

[ulm0]

@tarasinf gitlab.com is managed by gitlab, they have their settings for the registry. To delete an image simply get to the registry tab in your repo and there's a delete icon in the tag.

[ykyuen]

This oneliner command may help
https://gist.github.com/jaytaylor/86d5efaddda926a25fa68c263830dac1#gistcomment-2662407