"Can not connect to Ryuk" on macOS with Docker for Mac 2.4.0.0

[sazzer]

I've seen this exact error come and go a fair bit on here, but it's just today started happening for me.

I'm using testcontainers-java 1.12.0, but updating to the latest version has no impact at all.

When running my test, I get the following:

2020-08-28 17:19:18.778,INFO ,[main]:,o.t.d.DockerClientProviderStrategy,,,,,Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2020-08-28 17:19:18.833,INFO ,[main]:,o.t.d.DockerClientProviderStrategy,,,,,Will use 'okhttp' transport
2020-08-28 17:19:19.490,INFO ,[main]:,o.t.d.UnixSocketClientProviderStrategy,,,,,Accessing docker with local Unix socket
2020-08-28 17:19:19.491,INFO ,[main]:,o.t.d.DockerClientProviderStrategy,,,,,Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
2020-08-28 17:19:19.492,INFO ,[main]:,o.t.DockerClientFactory,,,,,Docker host IP address is localhost
2020-08-28 17:19:19.680,INFO ,[main]:,o.t.DockerClientFactory,,,,,Connected to docker:
  Server Version: 19.03.13-beta2
  API Version: 1.40
  Operating System: Docker Desktop
  Total Memory: 2996 MB
2020-08-28 17:19:19.988,INFO ,[main]:,o.t.u.RegistryAuthLocator,,,,,Credential helper/store (docker-credential-desktop) does not have credentials for quay.io

And then a seemingly infinite amount of the following:

2020-08-28 17:19:20.733,WARN ,[testcontainers-ryuk]:,o.t.u.ResourceReaper,,,,,Can not connect to Ryuk at localhost:32781
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:113)
	at java.lang.Thread.run(Thread.java:748)

This is running with Java 1.8.0_222 from Amazon Corretto, running on macOS 10.15.6.

This definitely worked yesterday. I did have an update to the Docker Engine today, and I'm assuming that's relevant, but everything else I'm doing with Docker is all working fine - including a different project that uses the Rust version of TestContainers. (Though I notice that doesn't use Ryuk, which might be relevant)

Cheers

[mpschaeuble]

@sazzer Have you found a solution in the meantime? I have the same issue after updating Docker Engine today.

Edit: "Downgraded" from Docker Desktop Edge to Docker Desktop Stable and the issue has gone. Looks like a problem in combination with Docker 2.3.6.0.

[gesellix]

For the recent Docker Desktop 2.4.x Release on macOS, see #3159.

[BHSDuncan]

I'm on MacOS 10.14.6 (Mojave).

Both TestContainers v1.12.3 and the latest (v1.14.3) have ceased to work with the latest update of Docker that I received today. The exact same symptoms as the OP happen.

Version 2.4.0.0 (48506) from the "stable" Channel.

Engine is: 19.03.13

Anyone found this and/or have a workaround, short of downgrading/rolling back the version of Docker Desktop (Community)?

[benlei-gfm]

I'm also experiencing the same issue

[rnorth]

Docker for Mac 2.4.0.0 was released yesterday, pushing this issue into the stable version.

We released 1.15.0-rc2 which includes a fix (#3159) for compatibility - please upgrade.

[telapo]

Docker for Mac 2.4.0.0 was released yesterday, pushing this issue into the stable version.

We released 1.15.0-rc2 which includes a fix (#3159) for compatibility - please upgrade.

No-one has asked me, but I confirm that 1.15.0-rc2 works ;-)

[tmohme]

Same here: 1.15.0-rc2 solves the problem.
Thanks for the fast reaction 👍

[jkroepke]

I have the same issue.

Using 1.15.0-rc2 could be a workaround. It's available in maven central.

https://mvnrepository.com/artifact/org.testcontainers/testcontainers/1.15.0-rc2

[schophil]

1.15.0-rc2 works for me to.

I do have some questions regarding this topic:
When will 1.15.0 be released?
What is the impact of not using ryuk?

[jkroepke]

What is the impact of not using ryuk?

Containers will not terminated.

[bsideup]

NoClassDefFoundError (com/fasterxml/jackson/annotation/JsonMerge)

@tonymurphy please make sure that you're not forcing an old version of jackson-annotations.

[rnorth]

@schophil

When will 1.15.0 be released?

Very soon - we mainly want to get #3102 over the line and into the 1.15.0 final release, as part of mitigation for the upcoming Docker Hub rate limits (#3099)

What is the impact of not using ryuk?

Ryuk does final cleanup of containers that might have escaped being killed. Testcontainers code tries its best, but in case of a sudden JVM termination or a bug there's a limit to what the Java code can do. Ryuk is external to the JVM process and provides a more surefire method of cleaning up. Having Ryuk enabled is something we strongly recommend.

[bsideup]

@vinay-thakkar you can:

1. use Testcontainers 1.15.0-rc2 that includes the fix
2. Disable gRPC FUSE in Docker for Mac 2.4.0
3. Downgrade to Docker for Mac 2.3.x

[vinay-thakkar]

thanks will try those out.

[vinay-thakkar]

Ok just to confirm the flick of a gRPC switch has worked for now until all the libraries come back to their synergy.

[unalsurmeli]

@vinay-thakkar The same problem came to me when I updated the docker version. Does this problem occur only for the desktop?

[vinay-thakkar]

@vinay-thakkar The same problem came to me when I updated the docker version. Does this problem occur only for the desktop?

Not sure about other env. For me it occurred only for 'Docker for Mac'. Our circle CI pipelines are pinned to older versions of Docker so no idea there. And don't have windows machine in team to confirm.

[avaz]

Just an update.

Updating to 1.5.0-rc2 solved the issue for me. However running JUnit5 tests in a gradle project from IntelliJ end up in no container being cleaned.

[bsideup]

@astubbs we will close it when 1.15.0 is out :)

@avaz perhaps you disabled Ryuk to workaround the issue and forgot to enable it back?

[avaz]

@bsideup you were right, my mistake, sorry for the inconvenience. Thanks!

[aaborschenko]

Disable gRPC FUSE in Docker for Mac 2.4.0 until fix their deprecated usages of com.github.dockerjava.core as was advised quite some time ago

Worx perfectly

[bsideup]

This issue is about Docker for Mac 2.4.0.0 only. The error message ("can not connect to <...>") itself does not mean that you're having the same issue.

Consider creating a new issue if you're having issues with 1.15.0-rc2.

For those who are searching for a fix, see this comment:
#3166 (comment)

[rnorth]

One further update: 1.15.0 was released today including the fix (the same as the previous RC builds)