network kind is ambiguous #1596
Comments
|
For now you're going to need to serialize creating the first cluster. I'm not sure if there's a non-racy way to do this in docker. |
|
xref: moby/moby#20648 docker-compose has this same issue :/ |
|
I don't think docker gives us sufficient tools to avoid a race condition here coordinating a docker network between multiple processes, unless we do our own out-of-band multi-process locking. ... that's not something I'm super excited to add right now and full of potential problems, would it be acceptable instead if we developed sufficient tooling to allow kind to natively create multiple clusters in one command? I've been sketching out a design for that functionality anyhow. |
|
thanks, but that would not solve my problem. our CI jobs are independent
and don't coordinate with each other
…On Wed, May 20, 2020 at 3:29 PM Benjamin Elder ***@***.***> wrote:
I don't think docker gives us sufficient tools to avoid a race condition
here coordinating a docker network between multiple processes, unless we do
our own out-of-band multi-process locking.
... that's not something I'm super excited to add right now and full of
potential problems, would it be acceptable instead if we developed
sufficient tooling to allow kind to natively create multiple clusters in
one command? I've been sketching out a design for that functionality anyhow.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1596 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAR5KLGIH3UKUMECFWYVPVTRSQVRZANCNFSM4NBGCXNQ>
.
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
|
|
er, but they're using the same docker instance?
are they on the same filesystem even ...?
EDIT: I ask because if not (it sounds like perhaps not) even something like deciding on our own file lock path would not work.
|
|
Currently workarounds are:
|
BenTheElder
added
the
priority/important-soon
|
I built a test bed with https://godoc.org/github.com/docker/docker/api/types#NetworkCreate package main
import (
"context"
"fmt"
"sync"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
cli, err := client.NewClientWithOpts(client.FromEnv)
if err != nil {
panic(err)
}
networkName := "test"
createNetwork := func() {
r, e := cli.NetworkCreate(context.Background(), networkName, types.NetworkCreate{
CheckDuplicate: true,
Driver: "bridge",
})
fmt.Println(r, e)
}
deleteNetwork := func() {
fmt.Println(cli.NetworkRemove(context.Background(), networkName))
}
var wg sync.WaitGroup
wg.Add(2)
go func() {
createNetwork()
wg.Done()
}()
go func() {
createNetwork()
wg.Done()
}()
wg.Wait()
deleteNetwork()
}results (always the same, except the random IDs): $ go run .
{8d6b80658e72d596f19c35bd90226171056dc9f93610aec3c2b55b20ad55ff4e } <nil>
{ad09baf925e2a213132c1b9072ec54bc70aaaa0e558a771cc3de2b509d72e948 } <nil>
Error response from daemon: network test is ambiguous (2 matches found based on name) |
|
I've got a pretty good idea how we can hack a working solution but it's going to be ... a hack. |
|
Wrote up a detailed outline of the hack I'm considering (Shared with members of [email protected], our standard SIG Testing group. I can't open documents to the entire internet by automated policy, but I can share with groups. This group is open to join, join it for access, this is common for Kubernetes documents) |
BenTheElder
added
the
lifecycle/active
|
This should be mitigated in 0.9.0 (just released, this was the last blocking issue), please let us know if you still encounter issues. |
|
FYI @howardjohn @JeremyOT it should be safe to do concurrent multi-cluster bringup in CI in v0.9.0 without any workarounds. |
|
We're hitting this again in 0.16.0-- have there been changes made that might affect the previous fix? |
|
Not that I'm aware of. If you have a reliable reproducer or any other new info, please file a new bug. We haven't had any reports of this issue in the 2+ years since the fix. |
|
This will be patched in docker going forward and we shouldn't have any more duplicate networks on future releases moby/moby#20648 (comment) We'll need to keep the workaround we shipped for now but it should be fixed in docker going forward 🎉 |
What happened:
Kicked off multiple builds in our CI environment, some tests use KIND to spin up clusters. Saw s bunch of failures:
What you expected to happen:
No interference between tests, as we experienced w/ KIND 0.7.x
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
kind version):kind v0.8.1 go1.13.9 linux/amd64kubectl version):docker info):/etc/os-release):The text was updated successfully, but these errors were encountered: