Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MGMT-14453: Fix bugs in installercache #7205

Merged
merged 1 commit into from
Feb 6, 2025

Conversation

paul-maidment
Copy link
Contributor

@paul-maidment paul-maidment commented Jan 20, 2025

This PR is for the purpose of resolving multiple bugs within the installer cache, due to the poor condition of the current cache, it makes sense to fix this in a single PR.

Fixes:

I have implemented fixes for each of the following issues.

  • Mutex was ineffective as not instantiated corrctly, leading to MGMT-14452, MGMT-14453.
  • Naming convention for hardlinks changed to be UUID based to resolve MGMT-14457.
  • Any time we either extract or use a release, the modified time must be updated, not only for cached releases. This was causing premature pruning of hardlinks.
  • LRU cache order updated to be based on microseconds instead of seconds.
  • Eviction checks updated to consider max release size and also cache threshold.
  • We now check there is enough space before writing.
  • During eviction - releases without hard links will be evicted before releases with hard links.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 20, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 20, 2025

@paul-maidment: This pull request references MGMT-14453 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.19.0" version, but no target version was set.

In response to this:

Presently we instantiate a new installercache.Installers instance per execution of the code generator. This renders any mutex handling pointless as we will never have any mutex conflict over a single request.

This PR moves the instantiation into the main and effectively makes it a singleton so that mutexes will function as intended.

This is the root cause behind a couple of reported issues

https://issues.redhat.com/browse/MGMT-14452 - Installer cache removes in-used cached image when out of space https://issues.redhat.com/browse/MGMT-14453 - INSTALLER_CACHE_CAPACITY small value cause to assisted-service crash

Both of these issues are reproducible on master and do not reproduce when running on code based on this PR.

If the cache size is set to a low value such as 1 then the current release will be deleted once the caller has finished with the release.

List all the issues related to this PR

  • New Feature
  • Enhancement
  • Bug fix
  • Tests
  • Documentation
  • CI/CD

What environments does this code impact?

  • Automation (CI, tools, etc)
  • Cloud
  • Operator Managed Deployments
  • None

How was this code tested?

  • assisted-test-infra environment
  • dev-scripts environment
  • Reviewer's test appreciated
  • Waiting for CI to do a full test run
  • Manual (Elaborate on how it was tested)
  • No tests needed

Checklist

  • Title and description added to both, commit and PR.
  • Relevant issues have been associated (see CONTRIBUTING guide)
  • This change does not require a documentation update (docstring, docs, README, etc)
  • Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

  • Are the title and description (in both PR and commit) meaningful and clear?
  • Is there a bug required (and linked) for this change?
  • Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sorry, something went wrong.

@openshift-ci openshift-ci bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 20, 2025
@openshift-ci openshift-ci bot requested review from carbonin and mlorenzofr January 20, 2025 10:50
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 20, 2025
@paul-maidment paul-maidment marked this pull request as draft January 20, 2025 11:12
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 20, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 20, 2025

@paul-maidment: This pull request references MGMT-14453 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.19.0" version, but no target version was set.

In response to this:

Presently we instantiate a new installercache.Installers instance per execution of the code generator. This renders any mutex handling pointless as we will never have any mutex conflict over a single request.

This PR moves the instantiation into the main and makes sure there is only a single instance so that mutexes will function as intended.

This is the root cause behind a couple of reported issues

https://issues.redhat.com/browse/MGMT-14452 - Installer cache removes in-used cached image when out of space https://issues.redhat.com/browse/MGMT-14453 - INSTALLER_CACHE_CAPACITY small value cause to assisted-service crash

Both of these issues are reproducible on master and do not reproduce when running on code based on this PR.

If the cache size is set to a low value such as 1 then the current release will be deleted once the caller has finished with the release.

List all the issues related to this PR

  • New Feature
  • Enhancement
  • Bug fix
  • Tests
  • Documentation
  • CI/CD

What environments does this code impact?

  • Automation (CI, tools, etc)
  • Cloud
  • Operator Managed Deployments
  • None

How was this code tested?

  • assisted-test-infra environment
  • dev-scripts environment
  • Reviewer's test appreciated
  • Waiting for CI to do a full test run
  • Manual (Elaborate on how it was tested)
  • No tests needed

Checklist

  • Title and description added to both, commit and PR.
  • Relevant issues have been associated (see CONTRIBUTING guide)
  • This change does not require a documentation update (docstring, docs, README, etc)
  • Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

  • Are the title and description (in both PR and commit) meaningful and clear?
  • Is there a bug required (and linked) for this change?
  • Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sorry, something went wrong.

@paul-maidment paul-maidment force-pushed the MGMT-14453 branch 4 times, most recently from 0369404 to 715734c Compare January 26, 2025 14:54
@openshift-ci openshift-ci bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 26, 2025
@paul-maidment paul-maidment force-pushed the MGMT-14453 branch 12 times, most recently from fa9cd8d to 27013f9 Compare January 26, 2025 23:43
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 26, 2025
@paul-maidment paul-maidment force-pushed the MGMT-14453 branch 2 times, most recently from 285a8bd to 3a419c2 Compare January 27, 2025 07:26
@paul-maidment paul-maidment force-pushed the MGMT-14453 branch 5 times, most recently from 6938c2f to f4842bf Compare February 4, 2025 23:06
internal/installercache/installercache.go Show resolved Hide resolved
internal/installercache/installercache.go Outdated Show resolved Hide resolved
internal/installercache/installercache.go Outdated Show resolved Hide resolved
internal/installercache/installercache_test.go Outdated Show resolved Hide resolved
openshift/template.yaml Show resolved Hide resolved
@paul-maidment paul-maidment force-pushed the MGMT-14453 branch 3 times, most recently from f6d18ea to a8ff664 Compare February 5, 2025 17:31
Copy link
Member

@carbonin carbonin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is okay to test at this point.
You can remove draft now @paul-maidment

internal/installercache/installercache.go Outdated Show resolved Hide resolved
@carbonin carbonin dismissed their stale review February 5, 2025 19:25

Changes made

@paul-maidment paul-maidment marked this pull request as ready for review February 5, 2025 21:11
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 5, 2025
@openshift-ci openshift-ci bot requested a review from danielerez February 5, 2025 21:12
…in the installer cache

This PR is for the purpose of resolving  multiple bugs within the installer cache, due to the poor condition of the current cache, it makes sense to fix this in a single PR.

* https://issues.redhat.com/browse/MGMT-14452
Installer cache removes in-used cached image when out of space
* https://issues.redhat.com/browse/MGMT-14453
INSTALLER_CACHE_CAPACITY small value cause to assisted-service crash
* https://issues.redhat.com/browse/MGMT-14457
Installer cache - fails to install when running parallel with same version
* Additionally, the cache did not respect limits, so this has been addressed here.

Fixes:

I have implemented fixes for each of the following issues.

* Mutex was ineffective as not instantiated corrctly, leading to [MGMT-14452](https://issues.redhat.com//browse/MGMT-14452), [MGMT-14453](https://issues.redhat.com//browse/MGMT-14453).
* Naming convention for hardlinks changed to be UUID based to resolve [MGMT-14457](https://issues.redhat.com//browse/MGMT-14457).
* Any time we either extract or use a release, the modified time must be updated, not only for cached releases. This was causing premature pruning of hardlinks.
* LRU cache order updated to be based on microseconds instead of seconds.
* Eviction checks updated to consider max release size and also cache threshold.
* We now check there is enough space before writing.
* During eviction - releases without hard links will be evicted before releases with hard links.
Copy link

openshift-ci bot commented Feb 6, 2025

@paul-maidment: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-scos-e2e-aws-ovn 2fc7ba4 link false /test okd-scos-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 6, 2025
Copy link

openshift-ci bot commented Feb 6, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: paul-maidment, rccrdpccl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [paul-maidment,rccrdpccl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit 2fbd268 into openshift:master Feb 6, 2025
16 of 17 checks passed
@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: ose-agent-installer-api-server
This PR has been included in build ose-agent-installer-api-server-container-v4.19.0-202502061139.p0.g2fbd268.assembly.stream.el9.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api-review Categorizes an issue or PR as actively needing an API review. approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants