Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[abctl] minimum requirements not specified in documentation, gives error when try to use less resources than required #40231

Closed
techieshark opened this issue Jun 24, 2024 · 8 comments
Closed

[abctl] minimum requirements not specified in documentation, gives error when try to use less resources than required #40231

techieshark opened this issue Jun 24, 2024 · 8 comments
Labels
area/abctl Issues with the abctl quickstart cli community proj/abctl team/deployments type/bug Something isn't working

Comments

@techieshark
Copy link
Contributor

techieshark commented Jun 24, 2024 •
edited
Loading

What happened?

I was reading https://docs.airbyte.com/deploying-airbyte/quickstart#setup--launch-airbyte

I ran

brew tap airbytehq/tap
brew install abctl # and after this failed I tried via `go install github.com/airbytehq/abctl@latest` but same problems
abctl local install

and it failed:

  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /Users/username/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 25.0.3
 SUCCESS  Port 8000 appears to be available
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created
 SUCCESS  Cluster 'airbyte-abctl' created
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 2
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 3
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1dda97ce0b
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1dda97ce0b
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 2
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 4
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1dda97ce0b
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 3
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 5
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 6
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd24d9fe1c39
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": dial tcp 10.244.0.13:8006: connect: connection refused
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 7
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd24d9fe1c39
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": dial tcp 10.244.0.13:8006: connect: connection refused
            Count: 2
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 8
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd297f96ebc3
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.13:8006/health": dial tcp 10.244.0.13:8006: connect: connection refused
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd24d9fe1c39
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": dial tcp 10.244.0.13:8006: connect: connection refused
            Count: 3
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1dda97ce0b
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 4
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 9
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd2e2e22f766
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd2e2e4c0da1
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.13:8006/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1dda97ce0b
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 5
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 10
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1dda97ce0b
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 6
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-689bbc977c-gqs98.17dbdd1932933fc7
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.11:8001/api/v1/health": dial tcp 10.244.0.11:8001: connect: connection refused
            Count: 11
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd2e2e22f766
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
            Count: 2
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd2e2e4c0da1
            Reason: Unhealthy
            Message: Liveness probe failed: Get "http://10.244.0.13:8006/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
            Count: 2
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd326aafea50
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": read tcp 10.244.0.1:52584->10.244.0.13:8006: read: connection reset by peer
            Count: 1
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-airbyte-api-server-b5b8c99f-dbpmf.17dbdd24d9fe1c39
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.13:8006/health": dial tcp 10.244.0.13:8006: connect: connection refused
            Count: 4
  ERROR   Failed to install airbyte/airbyte Helm Chart
  ERROR   Unable to install Airbyte locally
  ERROR   could not install airbyte chart: could not install helm: Get "https://127.0.0.1:56964/api/v1/namespaces/airbyte-abctl/services/airbyte-abctl-airbyte-webapp-svc": http2: client connection lost

What did you expect to happen?

I expected it not to fail and to show me some URL to go to to see locally hosted airbyte

Abctl Version

$ abctl version
version: v0.5.0

Docker Version

$ docker version
Client:
 Version:           20.10.21-rd
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        ac29474
 Built:             Tue Nov 22 22:21:43 2022
 OS/Arch:           darwin/arm64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS Version

# On Mac:
$ uname -a
Darwin machine.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64
@techieshark techieshark added area/abctl Issues with the abctl quickstart cli needs-triage type/bug Something isn't working labels Jun 24, 2024
@marcosmarxm marcosmarxm changed the title unable to follow quick start, install locally on mac [abctl] unable to follow quick start, install locally on mac Jun 26, 2024
@marcosmarxm
Copy link
Member

@techieshark sorry the lack of updates here. Are you still experiencing the issue or found a solution?

@whanata
Copy link

whanata commented Jul 15, 2024

@marcosmarxm I'm experiencing this issue, but in an EC2 Box on Amazon Linux 2023.

@wkeifenheim
Copy link

I also experienced this issue, on ARM-based MacOS, and figured I would try upping the resources available (from 2 cpu 2gb memory) to (4 cpu 12gb memory). The additional resources got me past the readiness probe messages, however I am still blocked by another reported issue #41063 and possibly #40641.

I saw a commit in the repo for abctl about setting minimum resource requests which clued me into providing more resources to the docker machine. Documenting resource requirements would be helpful.

@marcosmarxm
Copy link
Member

@airbytehq/platform-deployments can someone take a look into this issue ?

@cgardens
Copy link
Contributor

We plan to add a minimum resources requirement to the docs shortly.

@jackonawalk
Copy link

@cgardens to confirm, is the minimum resource requirements are something like 4 CPU, 12GB memory? thx!

@whanata
Copy link

whanata commented Aug 13, 2024

Using kind directly in a EC2 Machine (had issues using ABCTL), I had to use T4G XXLARGE to be able to run 1 extract and load (from Mongo to Snowflake).

I believe it's due to the amount of resource it locks in (kubernetes will specify a specific amount of resource it will need to spin up a pod), but it doesn't use all of it which means we needed a big enough box to accomodate for that.

I tried tweaking the setting for how much resource the Extract + Load pods needed, but it didn't really change (I may have missed it from the documentation).

@jackonawalk
Copy link

I was able to resolve this issue by using a container with 8GB RAM instead of 4GB

@marcosmarxm marcosmarxm changed the title [abctl] unable to follow quick start, install locally on mac [abctl] minimum requirements not specified in documentation, gives error when try to use less resources than required Aug 13, 2024
@bgroff bgroff closed this as not planned Won't fix, can't repro, duplicate, stale Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/abctl Issues with the abctl quickstart cli community proj/abctl team/deployments type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@bgroff @wkeifenheim @techieshark @marcosmarxm @whanata @jackonawalk @cgardens @nataliekwong @octavia-squidington-iii