Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use --bootstrap to wait for remote to become active #1103

Merged
merged 1 commit into from
May 12, 2022
Merged

Use --bootstrap to wait for remote to become active #1103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 2 additions & 17 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,28 +139,13 @@ jobs:
name: Launch remote buildkitd
if: matrix.driver == 'remote'
run: |
docker run -d --privileged \
docker run -d \
--privileged \
--name=remote-buildkit \
-p 1234:1234 \
--health-cmd "buildctl debug workers" \
--health-interval 1s \
${{ matrix.buildkit }} \
--addr unix:///run/buildkit/buildkitd.sock \
--addr tcp://0.0.0.0:1234
-
name: Check remote buildkitd
if: matrix.driver == 'remote'
run: |
try=0
max=10
until [ "$(docker container inspect remote-buildkit --format '{{ .State.Health.Status }}')" = "healthy" ]; do
if [ $try -gt $max ]; then
echo >&2 "healthcheck failed after $max trials"
exit 1
fi
sleep $(awk "BEGIN{print (100 + $try * 20) * 0.002}")
try=$(expr $try + 1)
done
-
name: Test
run: |
Expand Down
30 changes: 26 additions & 4 deletions driver/remote/driver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ package remote

import (
"context"
"time"

"github.com/docker/buildx/driver"
"github.com/docker/buildx/util/progress"
"github.com/moby/buildkit/client"
"github.com/pkg/errors"
)

type Driver struct {
Expand All @@ -23,17 +23,39 @@ type tlsOpts struct {
}

func (d *Driver) Bootstrap(ctx context.Context, l progress.Logger) error {
return nil
for i := 0; ; i++ {
info, err := d.Info(ctx)
if err != nil {
return err
}
if info.Status != driver.Inactive {
return nil
}

select {
case <-ctx.Done():
return ctx.Err()
default:
if i > 10 {
i = 10
}
time.Sleep(time.Duration(i) * time.Second)
}
}
}

func (d *Driver) Info(ctx context.Context) (*driver.Info, error) {
c, err := d.Client(ctx)
if err != nil {
return nil, errors.Wrapf(driver.ErrNotConnecting, err.Error())
return &driver.Info{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should still return error and we should check for "network error" on the caller side.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree about preserving the error to be able to display - I'm not sure about how much to change the caller side. Info gets called before Boot, which requires some fiddly logic changes in generic code to ignore network errors in that case.

I think there's two scenarios for errors here:

  • Failure to connect to the endpoint providing the driver (can't connect to kubernetes, can't connect to docker daemon, etc) - this kind of failure means it's not worth trying to bootstrap, something is wrong.
  • Failure to connect to the individual driver node (as in this case) - bootstrapping here is fine

Could we maybe rework status to be a struct and add an Err field to it? Then we could express a status of a node that's inactive because of an err. It currently feels like the error returned from here is quite tied to it's status, and is a little inconsistent across the drivers.

Status: driver.Inactive,
}, nil
}

if _, err := c.ListWorkers(ctx); err != nil {
return nil, errors.Wrapf(driver.ErrNotConnecting, err.Error())
return &driver.Info{
Status: driver.Inactive,
}, nil
}

return &driver.Info{
Expand Down
2 changes: 2 additions & 0 deletions hack/test-driver
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ if [ "$DRIVER" != "docker" ]; then
createFlags="$createFlags --append"
fi
buildxCmd create ${createFlags} \
--bootstrap \
--name="${builderName}" \
--node="${builderName}-${platform/\//-}" \
--platform="${platform}" \
Expand All @@ -69,6 +70,7 @@ if [ "$DRIVER" != "docker" ]; then
createFlags="$createFlags --config=${BUILDKIT_CFG}"
fi
buildxCmd create ${createFlags} \
--bootstrap \
--name="${builderName}" \
--platform="${PLATFORMS}" \
--driver="${DRIVER}" \
Expand Down