Failing to create docker test-containers in bitbucket pipelines using 1.15.0-rc2 #3296
Comments
|
Ergh - there's not much to go on, but this post (final para) suggests that there's an incompatibility between docker-java and Bitbucket pipelines. I wonder why we've not heard about this previously. It's pretty odd that this issue is apparently only occurring with Bitbucket Pipelines, but I think it's pretty clear that a change would be needed in docker-java to reinstate(?) the content-length header if it's missing. WDYT @bsideup ? |
|
Hi, reporting the issue to the docker-java project. Updated the initial post to include the full stack-trace from the initial failures. All the other integration tests fail as well with this stack-trace: |
|
Hello, Unfortunately we cant just add support for Transfer-Encoding: Chunked as the issue itself isnt with the transfer-encoding directly. We run an authorization plugin attached to every steps docker in docker daemon that inspects the payloads of all the requests to the docker daemon for potentially malicious arguments that would allow users to subvert the security of containers (such as --privileged containers) as we are running a multi tenanted environment we dont want users to be able to subvert this. How this plugins work is that when the docker daemon receives a request from a client (either the docker-cli, curl, some client in your favourite language etc) it first forwards that request (both the request line e.g. uri and query params and request body) to any authorization plugins which than perform inspections of them and return either an allow or deny (with an optional message) response. see for further information on authorization plugins: https://docs.docker.com/engine/extend/plugins_authorization/ Now unfortunately due to this line of code in the docker daemon https://github.com/moby/moby/blob/master/pkg/authorization/authz.go#L58
Now unfortunately if your using transfer-encoding: chunked the content-length that most docker clients send through is 0 which is valid for that transfer encoding so that means the request body fails this request and the authorization plugin denies the request as we cant validate your request body doesnt contain any potentially malicious arguments. Well behaving clients may set the content-length but than there is still and issue in that the body isnt forwarded to the authorization plugin as the docker daemon doesnt first buffer in the request body so if we were to read it it would drain the request body and than the docker daemon would fail the request as it couldnt read the body of the drained stream. Really docker should improve as this affects not just pipelines but anyone using authorization plugins we are lucky enough to know of this and this can be exploited in other auth plugins who dont perform these extra validations of content-length. So for now this is the reason why we cant support transfer-encoding: chunked. Now as it seems your using https://github.com/docker-java/docker-java It has multiple client backends (okhttp, jersey, netty) and I know one of them defaults to using transfer-encoding: chunked have you recently changed the backend used in a newer version of test-containers? |
|
Does anyone know where we stand on this? It's been 3 weeks since the last comment, and this is still an ongoing issue. Just curious if anyone has found a valid resolution. |
|
@pconrey it requires a change (workaround) in docker-java, see the discussion there docker-java/docker-java#1467 |
|
Thanks, @bsideup, but it looks like that one still isn't resolved either. Just trying to get a sense of when this will be fixed. For now I've downgraded to TestContainers v1.14.3, and downgraded Docker Desktop for Mac to 2.3.0.5. This combination works with bitbucket cloud, so I'll stick with that until this is fixed. |
|
@pconrey as a FOSS project, we have the least resources in this combo (Docker, BitBucket, us) to work on this, sorry 😅 FYI you can upgrade your Docker for Mac and disable gRPC-FUSE, see #3166 (comment) |
|
@bsideup, I totally understand. Wasn't complaining, just making a note of my resolution so others could follow suit if necessary. |
|
Hey, is this issue supposed to be solved in 1.15.0? I'm getting a following error: |
|
I had the same issue with group: 'org.testcontainers', name: 'mysql', version: '1.15.0-rc2'
it seems that when RYUK is enabled , and version is '1.15.0-rc2' time="2020-11-16T21:18:09.412624553Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system" |
|
There is no need to disable Ryuk when on Docker for Mac, just disable gRPC-FUSE in Docker for Mac's settings. Since the fix is in docker-java, I will lock this issue and close once Testcontainers is released with a new version of docker-java. |
|
FYI I just merged #3498 that includes a workaround that should cover most of the cases. |
|
Released in 1.15.1 |
Recently updated to 1.15.0-rc2 from 1.14.3 due to #3166, however now our bitbucket pipelines we are getting the following errors:
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3, imagePullPolicy=DefaultPullPolicy()) Caused by: com.github.dockerjava.api.exception.DockerException: Status 403: {"message":"authorization denied by plugin pipelines: Invalid content length provided"}Seems the issue is in the ElasticSearchContainer. Unsure what it is.
We have set
export TESTCONTAINERS_RYUK_DISABLED=trueEntire stack trace:
The text was updated successfully, but these errors were encountered: