Provider does not connect to kubernetes cluster

[mvgijssel]

The kustomization provider does not seem to connect to my kubernetes cluster. After doing apply it always fails after 21 retries. I've tried updating the KUBE_CONFIG and KUBECONFIG environment variables to correct and incorrect configurations, but the output is always the same. Same for the kubeconfig_path, kubeconfig_raw and context variables, setting them to either correct or incorrect settings yields the same not-working result.

The setup is pretty basic:

terraform {
  required_version = ">= 0.13.0"

  required_providers {
    kustomization = {
      source  = "local/setup/kustomization"
      version = "0.0.1"
    }
  }
}

provider "kustomization" {
  kubeconfig_path = var.kube_config
}

data "kustomization" "radarr" {
  path = "${path.module}/manifests"
}

resource "kustomization_resource" "radarr" {
  for_each = data.kustomization.radarr.ids

  manifest = data.kustomization.radarr.manifests[each.value]
}

terraform apply output

data.kustomization.radarr: Refreshing state...
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Creating...
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Creating...
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [10s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [10s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [20s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [20s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [30s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [30s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [40s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [40s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [50s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [50s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [1m0s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [1m0s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [1m10s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [1m10s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [1m20s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [1m20s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [1m30s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [1m30s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [1m40s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [1m40s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [1m50s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [1m50s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [2m0s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [2m0s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [2m10s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [2m10s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [2m20s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [2m20s elapsed]
kustomization_resource.radarr["apps_v1_Deployment|default|radarr"]: Still creating... [2m30s elapsed]
kustomization_resource.radarr["~G_v1_Service|default|radarr"]: Still creating... [2m30s elapsed]

Error: ResourceCreate: GroupVersionKind '/v1, Kind=Service' couldn't find resource (21 retries)

  on main.tf line 20, in resource "kustomization_resource" "radarr":
  20: resource "kustomization_resource" "radarr" {



Error: ResourceCreate: GroupVersionKind 'apps/v1, Kind=Deployment' couldn't find resource (21 retries)

  on main.tf line 20, in resource "kustomization_resource" "radarr":
  20: resource "kustomization_resource" "radarr" {

Terraform version: v0.13.0
terraform-provider-kustomize version: 0.2.0-beta
kubectl version: 1.17.0
kubernetes version: 1.18.8
OS: macos 10.15.6 (Catalina)

[pst]

I haven't debugged this yet and haven't tried 0.13 either. But I've seen this error before when the namespace did not exist. Also, unlike kubectl the provider does not default to default as the namespace if no namespace is explicitly set in the customization.

Let me know if any of this fixed it for you, if so, next step would probably be to see if the error message can be improved. If not, we're back to square one and I need to do some debugging.

[mvgijssel]

Thanks for the reply @pst! But to answer my own question, I suffer from hashicorp/terraform#3536. The terraform-provider-kustomize works perfect when I do the following:

git clone https://github.com/kbst/terraform-provider-kustomize.git
cd terraform-provider-kustomize
export CGO_ENABLED=1
make release-binaries
extract and copy the plugin to ~/.terraform.d/plugins/local/setup/kustomization/0.0.1/darwin_amd64/terraform-provider-kustomization

I'm not very familiar with Go and don't know the implications of setting the CGO_ENABLED flag for the created binaries (besides it fixing DNS issues on macos), but would you consider compiling the release binaries for macos with this flag?

[pst]

I'm on linux and have no access to a Mac. If I export this, the darwin build fails with:

$ export CGO_ENABLED=1
$ make release-binaries
GOOS=linux GOARCH=amd64 go build -o terraform.d/plugins/linux_amd64/terraform-provider-kustomization_v0.2.0-beta.0
tar -caf terraform-provider-kustomization-linux-amd64_v0.2.0-beta.0.tgz terraform.d/plugins/linux_amd64/terraform-provider-kustomization_v0.2.0-beta.0
GOOS=darwin GOARCH=amd64 go build -o terraform.d/plugins/darwin_amd64/terraform-provider-kustomization_v0.2.0-beta.0
# plugin
/usr/bin/ld: $WORK/b696/_x002.o: in function `pluginLookup':
/usr/local/go/src/plugin/plugin_dlopen.go:27: undefined reference to `dlsym'
/usr/bin/ld: /usr/local/go/src/plugin/plugin_dlopen.go:29: undefined reference to `dlerror'
/usr/bin/ld: $WORK/b696/_x002.o: in function `pluginOpen':
/usr/local/go/src/plugin/plugin_dlopen.go:19: undefined reference to `dlopen'
/usr/bin/ld: /usr/local/go/src/plugin/plugin_dlopen.go:21: undefined reference to `dlerror'
collect2: error: ld returned 1 exit status
# net
/usr/local/go/src/net/cgo_bsd.go:15:72: could not determine kind of name for C.AI_MASK
# os/user
/usr/local/go/src/os/user/getgrouplist_darwin.go: In function ‘mygetgrouplist’:
/usr/local/go/src/os/user/getgrouplist_darwin.go:16:11: warning: implicit declaration of function ‘getgrouplist’; did you mean ‘mygetgrouplist’? [-Wimplicit-function-declaration]
   16 |  int rv = getgrouplist(user, (int) group, buf, ngroups);
      |           ^~~~~~~~~~~~
      |           mygetgrouplist
make: *** [Makefile:13: release-binaries] Error 2

I'm no Go expert either, but it seems to be related to cross-compiling.

[pst]

One idea could be to build using Github Actions, which if I am not mistaken does support Linux, Mac and Windows nodes.

[mvgijssel]

One idea could be to build using Github Actions, which if I am not mistaken does support Linux, Mac and Windows nodes.

Correct! I'm currently looking into https://hub.docker.com/r/dockercore/golang-cross and https://github.com/elastic/golang-crossbuild to use a special docker container for cross compilation with CGO enabled. That might be a little bit easier as it removes the need for a macos / windows machine :).

I'll let you know if that approach works!

[mvgijssel]

I've had success with the following

git clone https://github.com/kbst/terraform-provider-kustomize.git
go get github.com/crazy-max/xgo
xgo --targets=darwin/amd64 terraform-provider-kustomize

It produces a binary with CGO included fixing macos dns issues 🎉.

The xgo binary is a helper around a docker container which includes the SDKs for windows, darwin and linux for all processor architectures https://github.com/crazy-max/xgo.

[pst]

I'll look into changing the pipeline to use this or the image.

[pst]

A related fix, do not retry for minutes if the root cause is a connection error. #49

[pst]

Regarding the CGO thing, I have been approved to publish this provider on the Terraform registry. Working towards this, Hashicorp suggests to use Goreleaser with the following config which explicitly disables CGO and states it doesn't work with Goreleaser or Terraform Cloud.

Not knowing enough about Go and CGO I'll stick with the Hashicorp defaults for this, which unfortunately means I can't provide releases that work for your use-case.

How do you use other providers, e.g. the AWS provider?

[mvgijssel]

How do you use other providers, e.g. the AWS provider?

Re-compiling all the providers with CGO_ENABLED=1 😓, which is not great.

So I stumbled on this tool https://github.com/greenboxal/dns-heaven which fixes the problem of DNS resolution on macos for golang programs for me 🎉. Really nice as I can now finally use providers from the Terraform registry!