do we ever use or need init containers?

Init containers here refer to the runc container used to initialize and deploy the main pod container.

I think this is the best we can do but at the same time AFAIK this is not sufficient to ensure that runc (or crun) never actually run on any isolated CPU.
If, as IIRC, the goal is to test that crun (or runc) never run on isolated CPU, then this test helps but is not sufficient.

One approach to testing is to verify whether runc/crun isolates itself from the isolated CPUs by following the method defined in the OCI specification, which involves generating a config.json and adhering to the specified CPUs based on the applied profile.

Alternatively, a tracing tool like bpftrace can be used to monitor all runc calls and inspect their CPU affinity. However, this method is quite invasive and may be challenging to implement IMO.

One approach to testing is to verify whether runc/crun isolates itself from the isolated CPUs by following the method defined in the OCI specification, which involves generating a config.json and adhering to the specified CPUs based on the applied profile.

Well, does the process which writes config.json run only on reserved CPUs? And does crun (or runc) honor the settings on config.json when setting its own affinity? IOW, who decides and who enforces that runc (or crun) only runs on reserved CPUs?

your test seems fine for workload, but there are open questions for the infra proper

Alternatively, a tracing tool like bpftrace can be used to monitor all runc calls and inspect their CPU affinity. However, this method is quite invasive and may be challenging to implement IMO.

I agree, but the problem here is first and foremost testing the right thing.

@ffromani Do you have any other way in mind?

To ensure the check is robust, I have added another Expect to verify whether it honors the provided CPUs through the profile. You can review the change here: Line The latest commit addresses your concerns.

thanks, I'll defer the decision to @yanirq and @Tal-or

@SargunNarula would it be correct to generalize the problem as whether host's processes are respecting the isolated cpus, when they are allocated exclusively for a Guaranteed pod?

If the answer is yes, maybe we can create a short shell script that will run and prints it's associated cpus, then we can check if it also access to ones that are already allocated by a Guaranteed pod.

Another suggestion I had in mind is to examine if we can add a test for that on crun/runc repo.
Maybe the test's context there would be more suitable for this kind of verification.

@Tal-or Yes, the problem statement is correct. However, I cannot definitively state which other host processes we might consider for this purpose.

Create a short shell script that runs and prints its associated CPUs

The concept of using a shell script was addressed in the original verification process, where we employed a wrapper around the runc process to capture and log its associated CPUs to a separate file. The challenge, however, was that despite replacing the binary with the wrapper, capturing the CPUs remained difficult due to the extremely short-lived nature of the runc process. The original test logic was designed to address that testing approach, as detailed in this link.

Add a test for that in the crun/runc repository

For runc, it might be feasible to design a test. However, verifying its behavior with CPU ranges defined by the performance profile presents a challenge, where this test could be valuable.

For crun, this test would not be applicable due to its distinct workflow. Unlike runc, crun does not rely on init containers. Instead, its workflow involves the following steps:

• Creating a temporary bind mount of the current executable.
• Remounting it as read-only.
• Opening the remounted file.
• Unmounting and deleting the temporary file.
• Using the file descriptor to execute the program.

This difference in implementation makes this testing approach unsuitable for crun.

the challenge, however, was that despite replacing the binary with the wrapper, capturing the CPUs remained difficult due to the extremely short-lived nature of the runc process.

If we're creating our own shell script for test purposes we can make it as longer as needed and the script will print the CPUs. The script doesn't have to be related to runc/crun we only want to make sure it doesn't violate the isolated cpus isolation.

as detailed in this link.

The link is a go file, not sure if that's what you meant.

For runc, it might be feasible to design a test. However, verifying its behavior with CPU ranges defined by the performance profile presents a challenge, where this test could be valuable.

For crun, this test would not be applicable due to its distinct workflow. Unlike runc, crun does not rely on init containers. Instead, its workflow involves the following steps:

Creating a temporary bind mount of the current executable.
Remounting it as read-only.
Opening the remounted file.
Unmounting and deleting the temporary file.
Using the file descriptor to execute the program.
This difference in implementation makes this testing approach unsuitable for crun.

Ok too complex, I get it.

If we aim to check the CPU usage of the shell script, we could extend its runtime. However, this would deviate from the primary objective, which is to verify whether the runc process is violating CPU isolation.

In the future, this might need to be extended to other processes, but as of now, that is not a requirement. The original bug specifically pertains to the runc process utilizing isolated CPUs. Bug link

The Go file contains a single test for this bug, utilizing the shell script approach. Link