Sidecar Containers

[Joseph-Irving]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes introduces the built-in support for sidecar containers pattern.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/753-sidecar-containers/README.md
• Discussion Link: Sidecar working group and KEP draft
• Primary contact (assignee): @SergeyKanzhelev
• Responsible SIGs: node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.29
  • Stable release target (x.y): 1.33
• Alpha
  • KEP (k/enhancements) update PR(s):
    • Sidecar containers KEP #3761
    • move sidecar to milestone 1.28 #3968
  • Code (k/k) update PR(s):
    • Add SidecarContainers feature kubernetes#116429
  • Docs (k/website) update PR(s):
    • Add initial (alpha) documentation about sidecar containers website#41802
• Beta
  • KEP (k/enhancements) update PR(s): KEP-753: add PRR questionnaire answers for beta #4255
  • Code (k/k) update PR(s):
    • sidecars: terminate sidecars after main containers kubernetes#120620
    • Promote SidecarContainers feature to beta kubernetes#121579
    • Add cel new validator into Kubernetes kubernetes#121307
  • Docs (k/website) update(s): Document SidecarContainers graduation to beta website#43471

/sig node

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[Joseph-Irving]

@enisoc @dchen1107 @fejta @thockin @kow3ns @derekwaynecarr, opened this tracking issue so that we can discuss.

[kow3ns]

/assign

[Joseph-Irving]

@derekwaynecarr I've done some scoping out of the kubelet changes required for next week's sig-node meeting, I believe that changes are only needed in the kuberuntime package, specifically kuberuntime_manager.go in and kuberuntime_container.go.

In kuberuntime_manager.go you could modify computePodActions to implement the shutdown triggering (kill sidecars when all non-sidecars have permanently exited), and starting up the sidecars first.

In kuberuntime_container.go you could modify killContainersWithSyncResult for terminating the sidecars last and sending the preStop hooks (the preStop hooks bit was a bit debatable, it wasn't settled whether that should be done or not. @thockin had a good point about why you might not want to encourage that behaviour, see comment).

Let me know if you want me to investigate any further.

[resouer]

@kow3ns The discussion makes more sense to me if maybe we can define a full description of containers sequence in Pod spec (sig-app), and how to handle the sequence in kubelet for start, restart and cascading consideration (sig-node). Let's catch the Feb 5 sig-node meeting to give more inputs.

cc @Joseph-Irving

[luksa]

The proposal says that sidecars only run after the init containers run. But what if the use-case requires the sidecar to run while/before the init containers run. For example, if you'd like route the pod's traffic through a proxy running as a sidecar (as in Istio), you probably want that proxy to be in place while the init containers run in case the init container itself does network calls.

[Joseph-Irving]

@luksa I think there's the possibility of looking at having sidecars that run in init phase at some point but currently the proposal is not going to cover that use case. There is currently no way to have concurrent containers running in the init phase so that would be potentially a much larger/messier change than what is being suggested here.

[Joseph-Irving]

Update on this KEP:
I've spoken to both @derekwaynecarr and @dchen1107 from sig-node about this and they did not express any major concerns about the proposal. I will raise a PR to the KEP adding some initial notes around implementation details and clarifying a few points that came up during the discussion.

We still need to agree on the API, it seems there is consensus that a simple way of marking containers as sidecars is prefered over more in depth ordering flags. Having a bool is somewhat limiting though so perhaps something more along the lines of containerLifecycle: Sidecar would be preferable so that we have the option of expanding in the future.

[luksa]

@Joseph-Irving Actually, neither the boolean nor the containerLifecycle: Sidecar are appropriate for proper future extensibility. Instead, containerLifecycle should be an object, just like deployment.spec.strategy, with type: Sidecar. This would allow us to then introduce additional fields. For the "sidecar for the whole lifetime of the pod" solution, it would be expressed along these lines:

containerLifecycle: 
  type: Sidecar
  sidecar:
    scope: CompletePodLifetime

as opposed to

containerLifecycle: 
  type: Sidecar
  sidecar:
    scope: AfterInit

Please forgive my bad naming - I hope the names convey the idea.

But there is one problem with the approach where we introduce containerLifecycle to pod.spec.containers. Namely, it's wrong to have sidecars that run parallel to init containers specified under pod.spec.containers. So if you really want to be able to extend this to init containers eventually, you should find an alternative solution - one that would allow you to mark containers as sidecars at a higher level - i.e. not under pod.spec.containers or pod.spec.initContainers, but something like pod.spec.sidecarContainers, which I believe you already discussed, but dismissed. The init containers problem definitely calls for a solution along these lines.

[Joseph-Irving]

@luksa You could also solve the init problem by just allowing an init container to be marked as a sidecar and have that run alongside the init containers. As I understand it, the problem is that init containers sometimes need sidecars, which is different from needing a container that runs for the entire lifetime of the pod.

The problem with pod.spec.sidecarContainers is that it's a far more complex change, tooling would need to updated and the kubelet would require a lot of modifying to support another set of containers. The current proposal is far more modest, it's only building on what's already there.

[luksa]

@Joseph-Irving We could work with that yes. It's not ideal for the sidecar to shut down after the init containers run and then have the same sidecar start up again, but it's better than not having that option. The bigger problem is that older Kubelets wouldn't handle init-sidecar containers properly (as is the case with main-sidecar containers).

I'd just like you to keep init-sidecars in mind when finalizing the proposal. In essence, you're introducing the concept of "sidecar" into k8s (previously, we basically only had a set of containers that were all equal). Now you're introducing actual sidecars, so IMHO, you really should think this out thoroughly and not dismiss a very important sidecar use-case.

I'd be happy to help with implementing this. Without it, Istio can't provide its features to init containers (actually, in a properly secured Kubernetes cluster running Istio, init containers completely lose the ability to talk to any service).

[Joseph-Irving]

In relation to the implementation discussion in #841, I've opened a WIP PR containing a basic PoC for this proposal kubernetes/kubernetes#75099. It's just a first draft and obviously not perfect but the basic functionality works and gives you an idea of the amount of change required.

cc @enisoc

[Joseph-Irving]

I put together a short video just showing how the PoC currently behaves https://youtu.be/4hC8t6_8bTs. Seeing it in action can be better than reading about it.
Disclaimer: I'm not a pro youtuber.

[Joseph-Irving]

I've opened two new PRs:

• add release checklist, grad criteria, upgrade strategy to sidecar kep #918 is about Graduation critieria, version skew, upgrade/downgrade
• Sidecar KEP API implementation #919 is to decide on the API implementation

Any thoughts or suggestions will be much appreciated.

[kfox1111]

We hit an issue with sidecars being a little undefined. Looking for guidence on what it the right behavior.

When the normal containers exit, the sidecars are shot.

In some cases, they then return exit codes != 0.

At least one k8s client, https://k9scli.io/ considers a completed pod with exit codes other then 0 bad and marks it in its ui as RED.

Its a bit unclear what the expectation around this is. Should it be documented that exit codes != 0 for sidecar containers are normal on main container exit and should be ignored? That would mean k9s's code should be adusted.

If exit code 0 is expected, each sidecar that doesnt return 0 should be adjusted, and perhaps alternate shutdown signals should be supported to the sidecar for more "graceful shutdown" signaling?

[SergeyKanzhelev]

When the normal containers exit, the sidecars are shot.

In some cases, they then return exit codes != 0.

At least one k8s client, https://k9scli.io/ considers a completed pod with exit codes other then 0 bad and marks it in its ui as RED.

From kubernetes perspective, sidecars will receive the SIGTERM following with SIGKILL. If implemented right, they will exit normally and will not mark pod as "red". @kfox1111 what do you see as a Pod status when it is red?

Should it be documented that exit codes != 0 for sidecar containers are normal on main container exit and should be ignored?

Hiding or recommending to hide the bad exit of sidecar may hide real issues with it. We would really hope that sidecars will be implemented correctly and not get into this situation. So I want to make sure there are only a limited set of situations when sidecar will be marked as failed. Any additional comments on how sidecar was implemented, what graceful period was set, and how long container took to terminate, will help understand your use case better

[kfox1111]

phase: Succeeded
initContainerStatuses:
  - containerID: docker://e8fd1c45cd995ef68ad84f9825faf4a35f53797d3308877adb59383a48ddfb3d
    image: ghcr.io/spiffe/spiffe-helper:nightly
    imageID: docker-pullable://ghcr.io/spiffe/spiffe-helper@sha256:6b09786251ae4e3ccef193b98eceae4b3d2bed091f3a8dd92be33a4c207455f2
    lastState: {}
    name: spiffe-helper
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: docker://e8fd1c45cd995ef68ad84f9825faf4a35f53797d3308877adb59383a48ddfb3d
        exitCode: 2
        finishedAt: "2024-09-05T20:29:29Z"
        reason: Error
        startedAt: "2024-09-05T20:29:19Z"
containerStatuses:
  - containerID: docker://eda62dbdb9196efa6723f025194d71aa92cedd3d4dd8c862cc803353d0901f38
    image: busybox:latest
    imageID: docker-pullable://busybox@sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41
    lastState: {}
    name: main
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: docker://eda62dbdb9196efa6723f025194d71aa92cedd3d4dd8c862cc803353d0901f38
        exitCode: 0
        finishedAt: "2024-09-05T20:29:29Z"
        reason: Completed
        startedAt: "2024-09-05T20:29:24Z"

We already decided we're going to update the spiffe-helper to exit 0 on sigterm. it was exiting 2 on sigterm. But may be good to clarify this behavior for other users of the sidecar api so everyone knows what to expect.

[SergeyKanzhelev]

We already decided we're going to update the spiffe-helper to exit 0 on sigterm. it was exiting 2 on sigterm.

It makes sense. Please let us know if there are other cases when it will become failed unexpectedly.

I wonder if this sidecar was used before the feature was implemented. I would think in the past, sidecar (implemented as a regular container) would also receive SIGTERM on pod delete command. So it was suffering from the same issue

[SergeyKanzhelev]

Definitely we can benefit from updating this document mentioning this: https://kubernetes.io/docs/tutorials/configuration/pod-sidecar-containers/

[kannon92]

@SergeyKanzhelev I think you all are deciding to not promote this to stable for 1.32. Should we move this for not for release then?

You all can refine the tests and improve bugs but it seems that due to some of the regressions it may not quite be ready for GA. :(

[matthyx]

@SergeyKanzhelev I think you all are deciding to not promote this to stable for 1.32. Should we move this for not for release then?

You all can refine the tests and improve bugs but it seems that due to some of the regressions it may not quite be ready for GA. :(

Yes this was the decision from this week's meeting.

[SergeyKanzhelev]

As we are getting closer to the GA (not in 1.32 though), we want to concentrate in 1.32 on polishing the feature and fixing known bugs.

Listing issues here to have a list of changes related to this KEP in 1.32:

• Pod status phase misses Pending transition after node reboot kubernetes#126650

  • Fix: fix the issue of losing the pending phase after a node restart. kubernetes#126653

• SyncPod may fail to start a pod with init container if RunContainerError occurs and SidecarContainers is enabled kubernetes#126440

  • Fix: Avoid SidecarContainers code path for non-sidecar pods kubernetes#127162
  • Fix: Restart the init container to not be stuck in created state kubernetes#126543
  • Also cherry-picked to 1.29+

• Sidecar container will not be restarted when image was changed kubernetes#121333

  • Fix: Clarify image change restart behavior for sidecar containers website#47024
  • Fix: Restart sidecar container when the image has changed kubernetes#126525

• Flake in tests

  • Fix: kubelet: delete channel from the terminated after closing it kubernetes#122781

• [Sidecar Containers] Ability to start sidecar and main containers in parallel kubernetes#126471

• [SidecarContainers] improve testing of termination Status of sidecar containers kubernetes#127217

  • Fix: [Sidecar Containers] Expand test coverage for Node E2E tests on pod termination behavior kubernetes#127221

• kubectl shows Pod status Completed if one of the containers exited with an Error kubernetes#121563

• [Sidecar Containers] Eviction message should account for the sidecar containers kubernetes#124938

  • Fix: [Sidecar Containers] Consider init containers in eviction message kubernetes#124947

• Named ports in initContainer sidecars do not work with NetworkPolicies kubernetes#125285

• ​​SideCar startupProbe on 1.28 kubernetes#120848

• Clarify ContainerStatus.Started null value and initialize the value for Init containers kubernetes#115553

• Inconsistent POD status reporting kubernetes#107713

• Desired behavior of a startup failed restartable init container in a pod with restartPolicy set to Never kubernetes#120234

  • Fix: KEP-753: Update documentation to clarify restartPolicy behavior #4880

• SidecarContainers: update the description of InitContainerStatuses type kubernetes#120676

• readinessProbe for initContainer can not be set . but kubectl explain says its possible kubernetes#116760

• The logic of oom_score_adj doesn't work well for sidecar containers kubernetes#111356

• [Sidecar Containers] Pods comparison by maxContainerRestarts should account for sidecar containers kubernetes#124936

  • Fix: [Sidecar Containers] Pods comparison by maxContainerRestarts should account for sidecar containers kubernetes#124952

• Sidecar containers can be interpreted as init containers kubernetes#127701

• [SidecarContainers] Scheduler accounting sidecar resource requests as initContainers kubernetes#127868

  • Fix: [scheduler] calculate pod requests resources with sidecar containers kubernetes#127878

• Named ports specified in sidecar container pod spec are not available to services kubernetes#127958

  • Fix: fix eps named ports does not work in sidecar(initContainer with restartPolicy=Always) kubernetes#127976

• Topology manager should consider restartable init containers kubernetes#119407

• Tutorials:

  • Add tutorial about running Pods with sidecar containers website#46825
  • Document the best practices when implementing mutating webhook website#47465
  • [SidecarContainer] Need to provide sidecar implementation best practices website#47823

[pacoxu]

Some new issues and PR that may be related to this feature:

• Fix named ports of restartable init containers don't propagate to EndpointSlice kubernetes#128850
• Cronjob's stuck job was not marked as failed and blocked new schedules kubernetes#128358
• [Flaking Test] [NodeFeature:SidecarContainers] Containers Lifecycle when A pod with restartable init containers is terminating kubernetes#128113
• [FG:InPlacePodVerticalScaling] Implement resize for sidecar containers kubernetes#128070