Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BouncyCastleFipsJsse application hangs on systems which do not have enough entropy #26736

Closed
fedinskiy opened this issue Jul 14, 2022 · 7 comments
Closed

BouncyCastleFipsJsse application hangs on systems which do not have enough entropy #26736

fedinskiy opened this issue Jul 14, 2022 · 7 comments
Labels
area/securepipeline issues related to ensure Quarkus can be used in a secure pipeline setups like FIPS or similar kind/bug Something isn't working

Comments

@fedinskiy
Copy link
Contributor

Describe the bug

I have an application with BouncyCastleFipsJsse endpoint. When I start the application on a freshly created virtual machine, it hangs on start up with messages "Thread blocked". After looking into code from stacktraces, I found, that the code blocks, while reading from /dev/random which does not have enough entropy yet(cat /proc/sys/kernel/random/entropy_avail returns 35). This problem does not affect BouncyCastleFips without JSSE.

Expected behavior

The application should not hang on any system.

Actual behavior

The application hangs and puts these stacktraces into cli:

2022-07-14 14:53:04,347 WARN  [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-2,5,main] has been blocked for 7933 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked
	at io.vertx.core.net.impl.TCPServerBase.listen(TCPServerBase.java:125)
	at io.vertx.core.net.impl.TCPServerBase.bind(TCPServerBase.java:100)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:217)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:149)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:154)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$WebDeploymentVerticle.setupTcpHttpServer(VertxHttpRecorder.java:1069)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$WebDeploymentVerticle.start(VertxHttpRecorder.java:1011)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:196)
	at io.vertx.core.impl.DeploymentManager$$Lambda$715/0x00000001005f7440.handle(Unknown Source)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63)
	at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:38)
	at io.vertx.core.impl.EventLoopContext$$Lambda$716/0x00000001005f6040.run(Unknown Source)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at [email protected]/java.lang.Thread.run(Thread.java:829)

and

2022-07-14 14:53:04,345 WARN  [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-7,5,main] has been blocked for 7932 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked
	at [email protected]/java.io.FileInputStream.readBytes(Native Method)
	at [email protected]/java.io.FileInputStream.read(FileInputStream.java:279)
	at [email protected]/java.io.FilterInputStream.read(FilterInputStream.java:133)
	at [email protected]/sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424) # here it blocks
	at [email protected]/sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:441)
	at [email protected]/sun.security.provider.NativePRNG$Blocking.engineGenerateSeed(NativePRNG.java:274)
	at [email protected]/java.security.SecureRandom.generateSeed(SecureRandom.java:857)
	at org.bouncycastle.crypto.util.BasicEntropySourceProvider$1.getEntropy(Unknown Source)
	at org.bouncycastle.crypto.fips.ContinuousTestingEntropySource.getEntropy(Unknown Source)
	at org.bouncycastle.crypto.fips.HashSP800DRBG.getEntropy(Unknown Source)
	at org.bouncycastle.crypto.fips.HashSP800DRBG.init(Unknown Source)
	at org.bouncycastle.crypto.fips.HashSP800DRBG.<init>(Unknown Source)
	at org.bouncycastle.crypto.fips.FipsDRBG$HashDRBGProvider.get(Unknown Source)
	at org.bouncycastle.crypto.fips.DRBGPseudoRandom.lazyInitDRBG(Unknown Source)
	at org.bouncycastle.crypto.fips.DRBGPseudoRandom.getSecurityStrength(Unknown Source)
	at org.bouncycastle.crypto.fips.FipsSecureRandom.getSecurityStrength(Unknown Source)
	at org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider.getDefaultSecureRandom(Unknown Source)
	at org.bouncycastle.jcajce.provider.ProvRandom$1.createInstance(Unknown Source)
	at org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider$BcService.newInstance(Unknown Source)
	at [email protected]/java.security.SecureRandom.getDefaultPRNG(SecureRandom.java:290)
	at [email protected]/java.security.SecureRandom.<init>(SecureRandom.java:219)
	at [email protected]/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:80)
	at [email protected]/javax.crypto.Mac.getInstance(Mac.java:273)
	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.calculateMac(Unknown Source)
	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.verifyMac(Unknown Source)
	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source)
	at [email protected]/java.security.KeyStore.load(KeyStore.java:1479)
	at io.vertx.core.net.impl.KeyStoreHelper.loadKeyStore(KeyStoreHelper.java:216)
	at io.vertx.core.net.KeyStoreOptionsBase.getHelper(KeyStoreOptionsBase.java:187)
	at io.vertx.core.net.KeyStoreOptionsBase.getTrustManagerFactory(KeyStoreOptionsBase.java:217)
	at io.vertx.core.net.impl.SSLHelper.getTrustMgrFactory(SSLHelper.java:314)
	at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:465)
	at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:456)
	at io.vertx.core.net.impl.SSLHelper.validate(SSLHelper.java:494)
	at io.vertx.core.net.impl.TCPServerBase.listen(TCPServerBase.java:151)
	at io.vertx.core.net.impl.TCPServerBase.bind(TCPServerBase.java:100)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:217)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:149)
	at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:154)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$WebDeploymentVerticle.setupTcpHttpServer(VertxHttpRecorder.java:1069)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder$WebDeploymentVerticle.start(VertxHttpRecorder.java:1023)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:196)
	at io.vertx.core.impl.DeploymentManager$$Lambda$715/0x00000001005f7440.handle(Unknown Source)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63)
	at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:38)
	at io.vertx.core.impl.EventLoopContext$$Lambda$716/0x00000001005f6040.run(Unknown Source)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at [email protected]/java.lang.Thread.run(Thread.java:829)

How to Reproduce?

  1. make sure, that your system(preferably on a VM) doesn't have much entropy: cat /proc/sys/kernel/random/entropy_avail should return a small value, I reproduced this robustly, with results between 35 to 851.
  2. git clone -b 2.7 [email protected]:quarkus-qe/quarkus-test-suite.git tests
  3. cd tests/security/bouncycastle-fips/bcFipsJsse
  4. mvn quarkus:dev # be careful! You will not be able to stop this process with Ctrl+c, you will have to use Ctrl+z, jobs -l and kill

For debugging, do this instead of step 4:

  1. mvn clean package -DskipTests -DskipITs
  2. java -Djava.security.debug=provider -jar target/quarkus-app/quarkus-run.jar > debug.txt 2>&1 and Ctrl+C after several seconds.
  3. grep "NativePRNG" debug.txt

Output of uname -a or ver

4.18.0-372.9.1.el8.x86_64

Output of java -version

11.0.13, vendor: Red Hat, Inc.

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.7.6.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)

Additional information

No response
@fedinskiy fedinskiy added the kind/bug Something isn't working label Jul 14, 2022
@quarkus-bot quarkus-bot bot added the area/securepipeline issues related to ensure Quarkus can be used in a secure pipeline setups like FIPS or similar label Jul 14, 2022
@quarkus-bot
Copy link

quarkus-bot bot commented Jul 14, 2022

/cc @Karm, @jerboaa

@sberyozkin
Copy link
Member

Interesting, also CC @cescoffier, I guess the only option is to tune setupTcpHttpServer logic a bit to fail early after a timeout ?

@cescoffier
Copy link
Member

I would need to check. The vertx stack generally indicates a problem when netty computes the machine id (most of the time because of dns). However if I remember directly there is a random object that may be used.

If that's the case, one approach would be to pass a machine id, and avoid the computation.

@cescoffier cescoffier self-assigned this Jul 17, 2022
@cescoffier
Copy link
Member

My bad, it's not the machine id in this case. It's when loading the key store, it requires a random generator that blocks because of the lack of entropy.

@sberyozkin can we try to switch to SHA1PRNG?

@cescoffier cescoffier removed their assignment Jul 18, 2022
@sberyozkin
Copy link
Member

sberyozkin commented Jul 18, 2022 •
edited
Loading

Hi @cescoffier I think if it were to be done then it could only be done on the test case level only, however, this would probably make the test not really working in a FIPS mode as SHA1PRNG is not FIPS compliant.

I still think though the problem should be addressed at the Vert.x HTTP level - something has gone wrong at the low level while doing setupTcpHttpServer but it feels like VertxHTTP should be able to shutdown in this case.

@cescoffier
Copy link
Member

I agree, but here is the thing: if you do not have enough entropy, you will be stuck forever (until the entropy reaches a certain level).
Maybe we can prevent the issue by creating entropy on startup (not sure how to do that). Vert.x is following this approach: https://github.com/vert-x3/vertx-auth/blob/master/vertx-auth-common/src/main/java/io/vertx/ext/auth/PRNG.java

@sberyozkin
Copy link
Member

This one must've been fixed by #40665, @fedinskiy, please reopen if you will get a chance to double check and confirm it is still an issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/securepipeline issues related to ensure Quarkus can be used in a secure pipeline setups like FIPS or similar kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cescoffier @sberyozkin @fedinskiy