Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: initContainer restart policy overridden by pod #18481

Conversation

tony84727
Copy link
Contributor

@tony84727 tony84727 commented May 5, 2023

Using main branch version: f4d2d17.

Restart policy of initContainers are overriden by restart policy of the pod.

Reproduce steps:

  1. Create a pod with a initContainer by podman kube play and the following manifest, podman kube play pod.yaml
    pod.yaml
    ---
    apiVersion: apps/v1
    kind: Pod
    metadata:
      name: test
    spec:
      containers:
        - name: nginx
          image: 'docker.io/nginx'
      initContainers:
        - name: first
          image: docker.io/nginx
    Because the initContainer first will start a nginx daemon and will not exit, podman kube play will hang (as expected)
  2. Inspect restart policy of the initContainer by podman inspect test-first --format '{{ .HostConfig.RestartPolicy.Name }}'
  3. Observe the restart policy of the initContainer is not no but always instead

Does this PR introduce a user-facing change?

Init containers created by kubernetes file will not be restarted multiple times.

Todo

  • add a test to check restart policy of initContainers

Restart policy of initContainers should not be overriden by pod and
the restart policy should always be "no".

See containers#16343

Signed-off-by: Tony Duan <[email protected]>
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 5, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 5, 2023

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the do-not-merge/release-note-label-needed Enforce release-note requirement, even if just None label May 5, 2023
Copy link
Member

@vrothberg vrothberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@umohnani8 PTAL

Let's make sure to have similar checks for service and infra containers as well.

@umohnani8
Copy link
Member

@tony84727 would you like to convert this into a PR and add a check for service container as well? If not, I can take over

@tony84727
Copy link
Contributor Author

@umohnani8 I was planning to add some test, at least for initContainers. Feel free to take over, though.

@tony84727
Copy link
Contributor Author

Added a e2e test case for init containers, converting this to an PR.

@tony84727 tony84727 marked this pull request as ready for review May 5, 2023 17:45
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 5, 2023
@tony84727 tony84727 force-pushed the fix/initctr-restart-policy-overridden branch from 419ece6 to 6861158 Compare May 6, 2023 05:03
Comment on lines 188 to 191
checkRestartPolicy := podmanTest.Podman([]string{"inspect", initContainerName, "--format", "{{ .HostConfig.RestartPolicy.Name }}"})
checkRestartPolicy.WaitWithDefaultTimeout()
Expect(checkRestartPolicy).Should(gbytes.Say(define.RestartPolicyNo))
})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test failed in my local environment. It might means init containers added by podman create --init-ctr doesn't have a restart policy name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, looks like restart policy is not added to an init container even when the --restart flag is set.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this test. So that, this PR only fix and test restart policy of init container created by podman kube play. Is it okay to handle this in another PR or issue?

@umohnani8
Copy link
Member

Based on https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#:~:text=If%20a%20Pod's%20init%20container,the%20overall%20Pod%20as%20failed., init containers should have a restart policy of on-failure unless the pod has a restart policy never then the init container should also have a restart policy of never.

@tony84727
Copy link
Contributor Author

Agree. so basically, the restart policy of an init container is inferred from the restart policy of the pod and users are not allowed to modify or override (i.e. through --restart)?

pod restart policy init ctr restart policy
none(missing) on-failure
always on-failure
on-failure on-failure
unless-stopped on-failure
never never

@edsantiago
Copy link
Member

All this policy discussion is way over my head, but I'd just like to chime in and say that this PR fixes #18477 so I'd like to see it proceed.

@tony84727
Copy link
Contributor Author

Maybe I can just remove the failing test and keep restart policy of init containers (created by podman kube play) as no for now? And probably open another issue to continue discuss this?

I'm wondering if there's a spec to follow, regarding to the proper behavior of various restart policies. Or we just look at kubernetes' implementation?

@umohnani8
Copy link
Member

Agree. so basically, the restart policy of an init container is inferred from the restart policy of the pod and users are not allowed to modify or override (i.e. through --restart)?

pod restart policy init ctr restart policy
none(missing) on-failure
always on-failure
on-failure on-failure
unless-stopped on-failure
never never

Yes this looks correct. For the most part, we are following k8s conventions for restart policies. There are currently some minor differences, because we don't want to break existing podman behavior.
I am fine with setting the restart policy for init containers to never all the time and we can open another PR with follow up fixes for the on-failure part.

make the sure restart policy is "no" for init containers created by
`podman kube play`

Signed-off-by: Tony Duan <[email protected]>
@tony84727 tony84727 force-pushed the fix/initctr-restart-policy-overridden branch from 6861158 to 74a5b92 Compare May 10, 2023 15:24
@TomSweeneyRedHat
Copy link
Member

@mheon this one is saying a release note is needed, but it looks like there is one already?
LGTM otherwise, and happy green test buttons.

@rhatdan rhatdan removed the do-not-merge/release-note-label-needed Enforce release-note requirement, even if just None label May 12, 2023
@rhatdan
Copy link
Member

rhatdan commented May 12, 2023

/approve
/lgtm
Thanks @tony84727

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 12, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 12, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhatdan, tony84727

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2023
@openshift-merge-robot openshift-merge-robot merged commit 2783651 into containers:main May 12, 2023
@vrothberg
Copy link
Member

/cherry-pick v4.5

@openshift-cherrypick-robot
Copy link
Collaborator

@vrothberg: #18481 failed to apply on top of branch "v4.5":

Applying: fix: initContainer restart policy overridden by pod
Using index info to reconstruct a base tree...
M	pkg/specgen/generate/container_create.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/specgen/generate/container_create.go
CONFLICT (content): Merge conflict in pkg/specgen/generate/container_create.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 fix: initContainer restart policy overridden by pod
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick v4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants