Feature Ruest: Keep last N docker images by laste-download date

[tahajahangir]

It's reasonable to keep N last downloaded (or N last created) tags per image.

[donhui]

The rule KeepLatestNFilesInFolder(count=N) may be useful for you.

[allburov]

@tahajahangir did KeepLatestNFilesInFolder help you?

[tahajahangir]

No, the file-based rules don't work with docker images (which have several layers and files). That's why DeleteDockerImagesOlderThan rule exists in addition to DeleteOlderThan.

[donhui]

I have defined the following rules, which can meet my needs. You can refer to them.

CleanupPolicy(
    'docker-demo-cleanup',
    rules.repo('docker-demo'),
    rules.exclude_docker_images(['*:latest', '*:release*']),
    rules.delete_docker_images_older_than(days=30),
    rules.exclude_path("base-tools*"),
    rules.keep_latest_n_file_in_folder(count=3),
)

[allburov]

@tahajahangir do you have any luck with it?
If it works - we can add as a reference to FAQ section

[tahajahangir]

Yes, it works

[afolgado]

As I can see, it won't work. It won't keep N tags per image, but N files per tag. In Docker images, every tag is a folder and every file in that folder is a layer of the image (plus the manifes.json file). This will delete some of the layers of every non-excluded tag, making them useless.

[seth-stvns]

I am seeing the same behavior as @afolgado (sorry didn't mean to send the tag). It is deleting a few layers from every image. Based on artifactory's layout of docker images and other resources (maven, gradle, npm, etc) this filter should be checking the folder directly above the files.

Will dig more into the code to see if I can find the issue when I have some time.

[donhui]

I have defined the following rules, which can meet my needs. You can refer to them.

CleanupPolicy(
    'docker-demo-cleanup',
    rules.repo('docker-demo'),
    rules.exclude_docker_images(['*:latest', '*:release*']),
    rules.delete_docker_images_older_than(days=30),
    rules.exclude_path("base-tools*"),
    rules.keep_latest_n_file_in_folder(count=3),
)

In this case, the rule rules.delete_docker_images_older_than(days=30) is required.
It can filter some docker images to delete.
Then the rule rules.keep_latest_n_file_in_folder(count=3) will filter on this basis.

[zbialik]

FWIW this feature request specifically asks for a method to keep latest by download time, not creation time. Because of this, I don't believe KeepLatestNFilesInFolder achieves the desired request.

I'm now trying to write a custom rule that's basically the same as KeepLatestNFilesInFolder except that it's based on the download time, rather than the creation time. Not sure if anyone else has implemented something similar.

[allburov]

I see, so like KeepLatestNFiles but for docker?
I'm not sure, but may be KeepLatestNFiles could work with docker rules as well, could one try it out?

[allburov]

Sorry, I see the solution for that was accepted
#61 (comment)

[allburov]

@zbialik please share with us the solution when it's ready, I'll be happy to review and accept a PR! 🙏

[zbialik]

@allburov I actually created my own custom rule that's literally a copy of KeepLatestNFiles class definition, except I modified the sort operation.

Specifically, I replace this line:

    artifacts.sort(key=lambda x: x["created"])

With this:

    artifacts.sort(key=lambda x: (
        x["stats"].get("downloaded", "1900-01-01T00:00:00.000Z"),
        x["created"]
    ))

For artifacts without a stats.downloaded field, I default this to a really old date. Then, for artifacts with identical download times (which can happen when there's several with 0 downloads and are defaulted to the old download date), these are sorted by the original created date.