Improve entropy collection in VMs

[marmarek]

Reported by joanna on 15 Nov 2012 19:03 UTC
While this is only my feeling, I suspect that the entropy collection daemon in our VMs needs some improvements.. This is because of the limited interaction with the physical world of each VM (e.g. mouse events go via vchan instead of via kernel module in a VM).

This can be easily noticed when one tries to generate a new GPG key in a VM -- the gpg would complain about inadequate entropy that is available and will hang until more is produced. One can produce more entropy via various disk activities (e.g. grep through the filesystem), however this:

1. Isn't very convenient
2. It's questionable whether such entropy is of "first-class freshness", or is it somehow inferior to the entropy that could be collected with the help of mouse movements, etc.?

It would probably be desirable to create some entropy producing device that would run in each of the VMs, and to feed this device from Dom0 or other domains exposed to physical hardware (netvm, usbvm?). One should be careful, however, not to distribute the same "entropy bits" to more than one domain, as this would likely compromise domain isolation.

Migrated-From: https://wiki.qubes-os.org/ticket/673

[marmarek]

Comment by joanna on 15 Nov 2012 23:25 UTC
Ok, I see two simple solutions:

1. We run a set of daemons in Dom0 (one for each VM) that essentially do this in a loop:

read_a_chunk_of_bytes (/dev/ranomd);
send_bytes_to_VM(); // via qrexec
sleep (...) // let other read some Dom0's entropy also

Then, in the VM, there is a code that reads the transmitted bytes and sends them into the kernel's rng using the RNDADDENTROPY IOCTL on /dev/random:

http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/char/random.c;h=b86eae9b77dfaeb04dd2d4efefd6ebc01b9e0a93;hb=HEAD#l1265

2. We just enable haveged in each VM (it gathers entopry from measuring internal CPU state):

http://www.issihosts.com/haveged/index.html
http://www.irisa.fr/caps/projects/hipsor/

Note 1: haveged is incredibly fast! Just seem to be a bit TOO fast for me... So, I think I would feel better with the option #1 I think...

Note 2: Dom0 entropy seems pretty reasonable (thanks to mouse and keyboard!), so it's not unreasonable to share it among all the VMs. But perhaps we could allow to manually exclude some VMs from getting the entropy from Dom0 (e.g. those that are not very sensitive). E.g. I have almost 30 domains on my laptop, while there are maybe 4 only that are used for key generation and those are the only ones that need fresh entropy from Dom0.

[marmarek]

Comment by joanna on 21 Nov 2012 10:25 UTC
Some comments are in this thread:

https://groups.google.com/group/qubes-devel/browse_thread/thread/e7023cca06daa219

[marmarek]

Modified by joanna on 8 Feb 2013 12:59 UTC

[marmarek]

Modified by joanna on 30 Aug 2013 17:21 UTC

[marmarek]

Modified by joanna on 20 Apr 2014 17:04 UTC

[marmarek]

Comment by joanna on 3 Jul 2014 12:07 UTC
For now (R2 release) we should just ensure haveged in the default template, I think.

[marmarek]

Comment by marmarek on 4 Jul 2014 02:45 UTC
Why this can't be the final solution? I don't believe we ever implement any other solution for this...

Also - do we want to cover by this fix also updates to template (which would mean hard dependency on haveged from qubes-core-vm)? Or installing it in new templates would be enough (so on R2 ISO)?

[marmarek]

Comment by joanna on 4 Jul 2014 09:37 UTC
I think just the new template, no need to issue updates. This is no a security problem, rather a usability -- i.e. if read() on /dev/random hangs, it's an annoyance to the user.

I agree to closing this ticket with haveged.

[marmarek]

Comment by marmarek on 4 Jul 2014 10:02 UTC
http://git.qubes-os.org/?p=marmarek/linux-template-builder.git;a=commit;h=e416c1a5b3b5a97525881d38efc0eef39659b48c

[adrelanos]

From https://wiki.archlinux.org/index.php/Haveged#Virtual_Machines

As discussed at Is it appropriate to use haveged as a source of entropy on virtual machines?, it can be contested whether haveged provides quality entropy within a virtual environment.

---

It's not as simple as writing to /dev/random. From man random(4)

This differs from writing to /dev/random or /dev/urandom, which only adds some data but does not increment the entropy count. The following structure is used:

---

Looks like this could be implemented by reading from /dev/random and forwarding that entropy qrexec, ioctl(2), RNDADDENTROPY to VMs.

[marmarek]

Program attached there returns numbers between 10 and 100 (30-50 on my
system), so theoretically it means that VMs have access to real rdtsc.
This is on R3rc1 (Xen 4.4.2), not sure how about R2 (Xen 4.1.2). I'll
check Xen documentation later - I think I've seen some option for this.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

https://github.com/mirage/xentropyd

[jpouellet]

xref: https://groups.google.com/d/topic/qubes-devel/6qAR1lTBzn8/discussion

[adrelanos]

haveged is discouraged in VMs by Andre Seznec, one of haveged's main authors. Source:
BetterCrypto/Applied-Crypto-Hardening@cf7cef7#commitcomment-23006392

Entropy is needed before systemd / systemd-random-seed.service / haveged by the kernel (and possibly others, which is not researched). (Wrote a bit about that here #2704 (comment).)

Please reopen. @andrewdavidwong

[smuellerDD]

Am Sonntag, 10. Oktober 2021, 03:00:48 CEST schrieb Cliff Maceyak: Hi Cliff,

> In short: `/dev/random` gives random output directly from hardware. > `/dev/urandom` gives random output from a crypto algorithm initialized > and sometimes refreshed with output from `/dev/random`. [This is wholly inaccurate and not how the Linux kernel CSPRNG works.](https://www.2uo.de/myths-about-urandom/#structure)

Correct. Both are completely identical with the exception that /dev/random blocks during boot until the kernel thinks it received 128 bits of entropy. From that point on there is no difference any more. Ciao Stephan

[smuellerDD]

Am Sonntag, 10. Oktober 2021, 03:02:55 CEST schrieb Cliff Maceyak: Hi Cliff,

> This is right. This is why the jitterentropy-rngd (and after I notified > Neil Hormann from rng-tools) inject data into the kernel after the elapse > of some time irrespective whether the kernel asks for the data or not. This doesn't make sense; you don't run out of entropy when you read from the random pool.

There is no way that the kernel tells you that data has been pulled. Therefore this strategy of injecting data once in a while is considered appropriate. Besides, this operation once every 10 minutes is cheap and is not considered to cause concerns. Ciao Stephan

[qua3k]

There is no way that the kernel tells you that data has been pulled. Therefore this strategy of injecting data once in a while is considered appropriate. Besides, this operation once every 10 minutes is cheap and is not considered to cause concerns.

If the pool has been initialized you don't need to "inject data"; there is no reason to do this.

[3hhh]

In short: /dev/random gives random output directly from hardware. /dev/urandom gives random output from a crypto algorithm initialized and sometimes refreshed with output from /dev/random.

This is wholly inaccurate and not how the Linux kernel CSPRNG works.

Thanks for the clarification (nice pictures!) - looks like I fell for the myth.

Then let me retry my "in short" statement (feel free to correct me again):
Currently (kernel 4.8 - 5.5):

• /dev/random: Always seeded CSPRNG output, blocking when it believes it doesn't have enough entropy anymore to reseed.
• /dev/urandom: Direct CSPRNG output, even when not seeded.

Future (kernel 5.6+):

• /dev/random: Always seeded CSPRNG output, never blocks after seeded once.
• /dev/urandom: Direct CSPRNG output, even when not seeded.

Ironically that makes me conclude to only use /dev/random from 5.6 onwards for everything - quite contrary to the link conclusion.

For 4.8 - 5.5 it's reasonable to use /dev/urandom after having checked that the CSPRNG was initially seeded e.g. via getrandom.

Btw reseeding may make sense for cloned VM snapshots etc. or if you don't want to 100% trust the initial seed. If you wouldn't need reseeding at all, the whole entropy collection structure would be pointless / only serve to obtain the initial seed.

[brendanhoar]

Stephan Müller's recent work on LRNG (code not yet(?)* accepted in the kernel tree ... replacing almost everything I thought I knew about randomness in Linux, heh) includes a good write up of his jitter-based entropy source.

That write up can be found here: https://www.chronox.de/jent/doc/CPU-Jitter-NPTRNG.pdf .

In particular it discusses why this jitter implementation is a valid source in virtual environments.

I'd propose utilizing that as a credited entropy source for boot time entropy in VMs, esp. if the CPU HWRND (via RDRAND) is no longer credited by compilation flag (Qubes maintainer decisions) and/or many non-RDRAND systems are experiencing long boot times due to a lack of credited entropy.

The credit rate for jitter can be assigned at boot time as per: https://www.chronox.de/lrng/doc/lrng.pdf .

There are also power-up noise source tests which are configurable at kernel compile time, which might be useful for Whonix templates, for example.

In addition to the new LRNG code availability on recent kernels, backports to many older kernels are available up through the current patchset (v42).

B

• corrected from "now accepted", due to my misreading a phoronix post.

[qua3k]

For 4.8 - 5.5 it's reasonable to use /dev/urandom after having checked that the CSPRNG was initially seeded e.g. via getrandom.

It's not. Developers should be using getrandom(...), not /dev/(u)random. Older kernels without getrandom(...) don't have any properly designed APIs for obtaining random numbers, and using /dev/random means that it will continue to block unnecessarily.

[DemiMarie]

For 4.8 - 5.5 it's reasonable to use /dev/urandom after having checked that the CSPRNG was initially seeded e.g. via getrandom.

It's not. Developers should be using getrandom(...), not /dev/(u)random. Older kernels without getrandom(...) don't have any properly designed APIs for obtaining random numbers, and using /dev/random means that it will continue to block unnecessarily.

It’s okay to use /dev/urandom after checking that the CSPRNG has been seeded. That is necessary when using cryptsetup, for example.

[jirka-h]

But upstream does not seem to listen - dead silence for years now. The person allegedly "responsible" for it just seems to ignore what I am doing. Yet ignoring it does not make things better.

Oh, that's bad. I wish I could help you. Keeping my fingers crossed!

I am looking for obscure systems: very small scale CPUs for IoTs, SPARC or other rare systems. I have coverage for Intel x86 (old and contemporary), AMD, ARM 32/64 bit, POWER LE/BE, IBM Z, RISC-V, MIPS.

I have access to some not-so-common systems like ARM 64 bit and IBM Power LE. For x86_64, I have access to large (multi-socket, max CPU count) systems. If you happen to need help with testing on any of these, please let me know.

Viele Grüße!
Jirka

[smuellerDD]

Am Sonntag, 10. Oktober 2021, 22:16:23 CEST schrieb jirka-h: Hi jirka,

> But upstream does not seem to listen - dead silence for years now. The > person allegedly "responsible" for it just seems to ignore what I am > doing. Yet ignoring it does not make things better. Oh, that's bad. I wish I could help you. Keeping my fingers crossed! > I am looking for obscure systems: very small scale CPUs for IoTs, SPARC or > other rare systems. I have coverage for Intel x86 (old and contemporary), > AMD, ARM 32/64 bit, POWER LE/BE, IBM Z, RISC-V, MIPS. I have access to some not-so-common systems like ARM 64 bit and IBM Power LE. For x86_64, I have access to large (multi-socket, max CPU count) systems. If you happen to need help with testing on any of these, please let me know.

Thanks a lot. I had access to ARM64 and POWER LE. But I do not have access to a multi-socket system. If you would give it a try, that would be great. We can take that offline - but simply: 1. apply the patch series v42 - enable all options for LRNG - enable the DRBG (CONFIG_LRNG_DRBG and the KCAPI DRNG (CONFIG_LRNG_KCAPI) as modules 2. reboot 3. execute the stress test as root: test/swap_stress.sh - there should be no issue, no BUG/WARN/panic - it is a loop with 5000 iterations 4. Obtain the raw entropy data: cd test ./getrawentropy -s 1000000 > /dev/shm/lrng_raw_noise.data 5. Send the file to me If you happen to be fancy and you happen to have another spare cycle, execute the automated tests. You need the steps outlined in tests/automation/README.md Thanks a lot! Stephan

[jirka-h]

Hi Stephen,

I have successfully built a kernel with your patch and stored it as rpm. The tests on the single-socket server have passed. I will run the tests on a multi-socket server when there are some spare HW cycles later this week and send it to you via email.

Cheers,
Jirka

[smuellerDD]

Am Dienstag, 12. Oktober 2021, 10:24:49 CEST schrieb jirka-h: Hi jirka,

Hi Stephen, I have successfully built a kernel with your patch and stored it as rpm. The tests on the single-socket server have passed.

That is great!

I will run the tests on a multi-socket server when there are some spare HW cycles later this week and send it to you via email.

Thank you very much.

Cheers, Jirka

Ciao Stephan

[jirka-h]

Hi Stephan,

what is the expected runtime for

./getrawentropy -s 1000000 > /dev/shm/4x_Xeon-E5-4610_v2_lrng_raw_noise.data

It's running on an old 4 socket (4x_Xeon-E5-4610_v2) server for 6 hours now, and so far, only 200 KiB of data has been collected. It's around 10Bytes per second.

I want to make sure if this is expected. I plan to run on more servers, but I want to make sure everything is all right.

Thanks!

Jirka

$ cat /proc/lrng_type 
DRNG name: ChaCha20 DRNG
Hash for reading entropy pool: SHA-256
Hash for operating aux entropy pool: SHA-256
LRNG security strength in bits: 256
per-CPU interrupt collection size: 1024
number of DRNG instances: 4
Standards compliance: 
High-resolution timer: true
LRNG minimally seeded: true
LRNG fully seeded: true
Continuous compression: true

[smuellerDD]

Am Donnerstag, dem 14.10.2021 um 03:39 -0700 schrieb jirka-h:

Hi Stephan, what is the expected runtime for ``` ./getrawentropy -s 1000000 > /dev/shm/4x_Xeon-E5-4610_v2_lrng_raw_noise.data ``` It's running on an old 4 socket (4x_Xeon-E5-4610_v2) server for 6 hours now, and so far, only 200 KiB of data has been collected. It's around 10Bytes per second. I want to make sure if this is expected. I plan to run on more servers, but I want to make sure everything is all right.

That is the amount of events trickling in. If you send, say, a ping flood to it or do another way of causing interrupts like dd if=$DISK of=file count=10000 oflag=direct then you get your results in a few minutes. Thanks a lot Stephan

…

Thanks! Jirka ``` $ cat /proc/lrng_type DRNG name: ChaCha20 DRNG Hash for reading entropy pool: SHA-256 Hash for operating aux entropy pool: SHA-256 LRNG security strength in bits: 256 per-CPU interrupt collection size: 1024 number of DRNG instances: 4 Standards compliance: High-resolution timer: true LRNG minimally seeded: true LRNG fully seeded: true Continuous compression: true ```

[jpouellet]

On Wed, Oct 6, 2021 Demi Marie Obenour wrote: 2. Increase the amount of random seed provided to VMs from dom0 via QubesDB.

Assuming you mean a single static value and not somehow a different one provided dynamically on every read: Not great. QubesDB values are readable by non-root userspace. Having the same kernel entropy seed exposed to subsequent (potentially lesser-trusted) code seems a bad idea. If not otherwise combined with sufficient additional other entropy, then there is a risk that prng state could be deduced from a subsequent read of the qubesdb random seed. A mechanism which uses a channel whose content does not stick around after the fact (and certainly not also made available to less-trusted code) seems a better fit for the kernel entropy-seeding use case.

[marmarek]

FWIW the initial seed is removed from QubesDB after reading it (and wiped from the process memory too). It happens rather early in the boot process. But I agree there could be a better way. I consider researching how bootloader can provide random see to the kernel (there is such option in theory) and maybe use that.

…

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab

[jirka-h]

That is the amount of events trickling in. If you send, say, a ping flood to it or do another way of causing interrupts like dd if=$DISK of=file count=10000 oflag=direct then you get your results in a few minutes. Thanks a lot Stephan

This has helped, especially on systems with NVMe storage.

I have processed the results with ent utility (git clone https://github.com/ProhtMeyhet/ent-random-entropy-test.git), see results below. Entropy estimation is 3.122 bits per byte regardless of the test system used. Is this expected?

intel-E5-4627-v2-4s/4x_Xeon_E5-4627_v2_lrng_raw_noise.data.ent:Entropy = 3.122261 bits per byte.
intel-gold-6126-4s/4x_Xeon_Gold_6126_lrng_raw_noise.data.ent:Entropy = 3.122186 bits per byte.
intel-E5-4607-4s/4x_Xeon_E5-4607_lrng_raw_noise.data.ent:Entropy = 3.122283 bits per byte.
amd-epyc3-milan-7713-2s/2x_EPYC_7713__lrng_raw_noise.data.ent:Entropy = 3.122571 bits per byte.
intel-E5-4610-v2-4s/4x_Xeon-E5-4610_v2_lrng_raw_noise.data.ent:Entropy = 3.122024 bits per byte.
intel-gold-6126-2s/2x_Xeon_Gold_6126_lrng_raw_noise.data.ent:Entropy = 3.122105 bits per byte.
amd-epyc3-milan-7313-2s/2x_EPYC_7313_lrng_raw_noise.data.ent:Entropy = 3.122709 bits per byte.
intel-E7-4870-8s/8x_Xeon-E7-4870_lrng_raw_noise.data.ent:Entropy = 3.122159 bits per byte.

I will send you a link to download the full results via email.

Thanks
Jirka

[smuellerDD]

Am Samstag, 16. Oktober 2021, 14:17:50 CEST schrieb jirka-h: Hi jirka,

> That is the amount of events trickling in. If you send, say, a ping flood > to it or do another way of causing interrupts like dd if=$DISK of=file > count=10000 oflag=direct then you get your results in a few minutes. > Thanks a lot Stephan This has helped, especially on systems with NVMe storage. I have processed the results with `ent` utility (git clone https://github.com/ProhtMeyhet/ent-random-entropy-test.git), see results below. Entropy estimation is 3.122 bits per byte regardless of the test system used. Is this expected?

This is fully expected, because the data you obtained is ASCII numbers and not binary data. The ASCII numbers are simply an export of the kernel time stamps obtained per interrupt. A user space tool at [1] is used to conver it into a binary string. This string could be processed with ent, but better with the SP800-90B tool set as the data is non-IID. The tool set at [2] exactly does that. With the application of these tools, the data you obtained looks very good: it returns about 7 bits of entropy per time stamp - the LRNG heuristic applies 1 bit of entropy - see [2] section "Interpretation of Results" [1] https://github.com/smuellerDD/lrng/blob/master/test/sp80090b/validation-runtime/extractlsb.c#L198 [2] https://github.com/smuellerDD/lrng/tree/master/test/sp80090b/validation-runtime Thanks a lot for your support. Ciao Stephan

[jirka-h]

Hallo Stephan,

thanks for checking the results!

the data you obtained is ASCII numbers

Oh, my bad! I was assuming the data are binary without actually checking the values.

I have now rerun the analysis based on the above instructions and the results are indeed excellent!

intel-E5-4607-4s/4x_Xeon_E5-4607_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 7.468554
intel-gold-6126-4s/4x_Xeon_Gold_6126_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 6.904956
intel-gold-6126-2s/2x_Xeon_Gold_6126_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 6.915690
intel-E5-4627-v2-4s/4x_Xeon_E5-4627_v2_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 7.268956
amd-epyc3-milan-7313-2s/2x_EPYC_7313_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 7.207621
intel-E7-4870-8s/8x_Xeon-E7-4870_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 7.287790
intel-E5-4610-v2-4s/4x_Xeon-E5-4610_v2_lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 7.264641
amd-epyc3-milan-7713-2s/2x_EPYC_7713__lrng_raw_noise.data.minentropy_FF_8bits.txt:min(H_original, 8 X H_bitstring): 7.007947

Thanks a lot!
Jirka

[3hhh]

IIRC this one can be closed with 4.1 as the VM kernel now does the job itself?

[andrewdavidwong]

IIRC this one can be closed with 4.1 as the VM kernel now does the job itself?

Closing as done. If anyone believes this issue is not yet done, please leave a comment, and we'll be happy to reopen it. Thank you.

[adrelanos]

created just now to ask Stephan Mueller:

• Should Xen, VirtualBox implement something similar to QEMU / KVM's virtio-rng? Keep feeding the host's /dev/random into the VM? smuellerDD/jitterentropy-rngd#28