Uhyve is very slow in Githubs CI

[jounathaen]

Example run:

https://github.com/hermit-os/kernel/actions/runs/11801739625/job/32875789502

• Takes ~2:30 Min with 1 core
• Takes ~11:00 Min with 4 cores

[n0toose]

Uploaded the job-longs.txt for posterity and because GitHub requires you to login to view the logs:

logs_30798294207.zip
job-logs.txt

[n0toose]

After removing the following line on my system (with its fair share of environment variables), I managed to gain a performance boost of 10 seconds (before, it was around ~14) with 20k-27k less hypercalls (depending on the log level).

This probably warrants some further investigation:

uhyve/src/vm.rs

Line 323 in d4bc7da

.envs(env::vars())

[mkroening]

In release mode? This would be most surprising. 🤔

I'll have a look at the kernel to see if anything jumps on me.

[mkroening]

Statistics from running a release-profile rusty_demo on hermit-os/kernel@430da84:

With env in FDT:

VM exits:                       total     cpu0 
  Hypercall(Exit):                  1        1 
  Hypercall(FileWrite):           385      385 
  Hypercall(Uart):              21692    21692 
  PCIRead:                       1043     1043 
  PCIWrite:                      1067     1067 
CPU runtimes:
  cpu 0: 1.127073035s

Without env in FDT:

Run statistics:
VM exits:                       total     cpu0 
  Hypercall(Exit):                  1        1 
  Hypercall(FileWrite):           150      150 
  Hypercall(Uart):               3252     3252 
  PCIRead:                       1043     1043 
  PCIWrite:                      1067     1067 
CPU runtimes:
  cpu 0: 534.278001ms

(@jounathaen it would be nice if the statistics would be sorted automatically, also there is a trailing space)

These numbers do make sense, since the output with the environment is a 21,000 characters longer than without, and we currently do one hypercall per character if I am not mistaken.

This does not explain the slowdown in CI, though, since the CI spends most of the time with number crunching and not with UART output.

Edit: for comparison, QEMU with KVM for rusty_demo without environment variables takes this long:

________________________________________________________
Executed in    1.33 secs      fish           external
   usr time    0.92 millis  916.00 micros    0.00 millis
   sys time  117.51 millis  424.00 micros  117.09 millis

[jounathaen]

We definitely have to implement the buffered serial hypercall in Hermit.

(@jounathaen it would be nice if the statistics would be sorted automatically, also there is a trailing space)

By which measure do you want to sort them? I think alphabetically makes most sense to have the best comparability.

[mkroening]

By which measure do you want to sort them? I think alphabetically makes most sense to have the best comparability.

Yeah, alphabetically is what I was thinking of. I had to sort them manually, the order was non-deterministic.