Add ability to influence the OS scheduler

[lhstrh]

While our runtime executors run reactions to completion, there is an opportunity (in the presence of an operating system) to leverage the underlying thread scheduler to achieve preemption, assuming that we have multiple workers running per core. As a starting point, to create some experiments, we need to be able to specify things like the scheduling policy, the number of cores, and potentially other things. This will allow us to make use of the functionality implemented in lf-lang/reactor-c#355.

Concrete proposal:

target C {
  cores: <number>
  thread-policy: <normal | realtime-rr | realtime-fifo>
}

[erlingrj]

I think that what we discuss here is mainly relevant to timing enclaves and federations. We therefor need to think about the need to control these parameters both at the global level (as proposed here), but also with the option to override it at the individual federate level.

Both at the global top-level and at the federate/enclave level I think we should expose the following three parameters:

• cores: This is the CPU set that the workers will run on
• thread-policy: The scheduling policy used by the underlying OS
• priority : <number> | <rate-monotonic> | <deadline-monotonic> | <earliest-deadline-first> | <least-laxity-first>

priority here is what we have called "scheduling" in our discussions. I went with priority because, essentially, the scheduling algorithm is about assigning priorities. I propose three categories of priority assignments:

1. Static and user-defined, by passing it just a number
2. Static and derived by our compiler, this would be rate-monotonic (looking at timer periods/phy actions min_spacings) or deadline-monotonic.
3. Dynamic, e.g. EDF or LLF.

Currently, (2) and (3) are just ideas that we are discussing, so we would initially only support (1).

As, mentioned, I think we need to allow setting these parameters both at the global-level and at the enclave/fed level. But that introduces some weird questions e.g.

1. What would it mean to have different federates using different priority assignment schemes? I mostly imagine it to be useful to statically set the priority of some of the federates.
2. Do we really want to let the user run different federates with different OS scheduling policies? It would interfere with the priority assignments. E.g. doesn't make much sense to have a federate be part of an EDF priority assignment scheme but it uses the normal (CFS) if the other federates uses a realtime policy.

Possible strange configuration could be detected by the compiler and a warning could be issued.

[soyerefsane]

I think the first two suggestions you had mainly cover the properties implemented in the retreat, with potentially a minor difference where the core property was referring to the number of cores, not a specific core number. The idea was you might want to use more than one core, so we can enable that. Let's say your system has 4 cores, you set these cores property to 2, the workers target property to 4, and there are 2 enclaves in the system. This would generate the following behavior in the current implementation:

• Each enclave would generate 4 worker threads. These worker threads would get mapped to 2 cores in total starting from the last core. So thread 0 of enc 0 would be mapped to core 3, thread 1 to core 2, thread 2 to core 3 .., thread 0 of enc 1 to core 4 ...

I think in the future it might be useful to add another target property that can be used to map any thread to any core, but the initial goal was to define the number of cores and give the user an inherent way of defining this behavior.

We decided some initial steps to implement the EDF scheduling with priority assignments under the retreat branch with @edwardalee @fra-p. Currently this is a WIP but the idea is to used inferred deadlines to assign priorities to the underlying real-time scheduler. The original details are documented under #2265.

I also had couple of questions for you regarding the priority target parameter you are suggesting.

• You've already mentioned that this is very similar to the scheduler property, what is the advantage of redefining another target parameter for this?
• I can totally see the advantage of a lower cost deadline-monotonic scheduler and not just EDF or NP; however, I'm not sure I fully see the point of the <number> input to this target property. If you define a static number that all reactions can inherit that doesn't necessarily achieve more than what chrt command line argument is achieving. What am I missing here?

For the weird questions, I think it might be easier to take one step at a time -- fedarations might make these discussions way too complicated, specifically due to the rightful questions you have raised. We might want to consider this first just with enclaves (and the default LF programs) before thinking about how can this be extended to fedarated execution.

[erlingrj]

• I think the cpu parameter should enable selecting the set of CPUs to run the program on, not just the number of CPUs.
• Manually assigning a static priority and scheduling policy would, on Linux, result in the same as launching the federates with chrt yes<
• I don't believe the questions I pose change if we limit ourselves to talking only of enclaves

[erlingrj]

It is important to appreciate that what I here call priority is not a renaming of the existing target property scheduler. I would prefer renaming the existing property to executor and what I call here priority to scheduler. But someone has to actually do it...