Flux v2 support does not work on Windows

[wat-usmdon]

What were you trying to accomplish?

I'm trying to deploy an EKS cluster with Flux v2 from a Windows environment.

What happened?

Cluster creation works fine, but whenever it gets to the part where Flux is installed or whenever I run eksctl enable flux I get an error about needing to run aws configure.

This happens only with eksctl. If I run flux check --pre directly I pass all prerequisite checks.

How to reproduce it?

You should be able to reproduce this with any configuration for Flux v2 as long as you are on Windows because it doesn't actually get to the part where it tries to apply the configuration. The examples should be fine. I don't have anything special in my config except my private repository info.

Logs

2021-03-17 16:17:22 [ℹ]  running pre-flight checks
► checking prerequisites
✔ kubectl 1.20.4 >=1.18.0-0

Unable to locate credentials. You can configure credentials by running "aws configure".
✗ Kubernetes API call failed: Get "https://00000000000000000000000000000000.gr7.us-east-2.eks.amazonaws.com/version?timeout=32s": getting credentials: exec: executable aws failed with exit code 253
Error: running Flux pre-flight checks: exit status 1

Anything else we need to know?

eksctl 0.40.0 downloaded by Scoop on Windows 10. AWS credentials are a token saved in ~/.aws/credentials.

Versions

> eksctl version
0.40.0
> kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Apparent cause

I can see that eksctl is launching flux which is launching kubectl which is launching aws eks get-token. Running any of these commands without eksctl works perfectly.

If I have procmon running when I run these commands I can see that aws eks get-token is invoked with a very different environment when launched from eksctl. Specifically, the successful invocations have all my normal environment variables but the unsuccessful notifications only have GITHUB_TOKEN (blank) PATH HOME AWS_ACCESS_KEY_ID (blank) AWS_SECRET_ACCESS_KEY (also blank) SYSTEMROOT AWS_STS_REGIONAL_ENDPOINTS KUBERNETES_EXEC_INFO. If I start removing variables from my shell and running aws eks get-token myself, I start getting the same behavior as eksctl after deleting the variable HOMEPATH. I guess Python uses this variable for finding ~/.aws.

In executor.go there is code that creates a command and copies environment variables into it, but only the specified environment variables are copied. Looking at the Go documentation apparently what this means is that if executor.go wants to pass any environment variables to a child process than those environment variables become the only environment variables passed to the child process (except SYSTEMROOT). I think this code needs to start by populating the child process environment with the current process environment first. On Linux, Python must be locating the home directory by reading /etc/passwd or something, at least as a fallback.

[Callisto13]

Thanks for reporting this @wat-usmdon , and for providing lots of detail 😄 . Flux v2 support is still experimental and we are glad to see someone is attempting to use it: we weren't really sure before 😅 .

We will look into this issue asap and hopefully come up with a resolution quickly.

[wat-usmdon]

If I'm setting up a new cluster today should I still be using Flux v1 despite #2963?

[Callisto13]

So Flux v2 itself is not experimental. All eksctl is doing is using config to run the new flux for you after your cluster is ready. You therefore have the option to do something like:

eksctl create cluster ...
flux bootstrap ...

with each command separately which is 100% the same thing as what eksctl is doing (just not with a handy 2-in-1 config).

Flux docs.

[Callisto13]

I would definitely recommend switching to Flux 2 if the option to run separate commands is possible.

[wat-usmdon]

This fixes it for me

diff --git a/pkg/executor/executor.go b/pkg/executor/executor.go
index 7a7fec10..1945bda2 100644
--- a/pkg/executor/executor.go
+++ b/pkg/executor/executor.go
@@ -45,6 +45,7 @@ func (e ShellExecutor) ExecInDir(command string, dir string, args ...string) err
 func (e ShellExecutor) buildCmd(command string, args ...string) *exec.Cmd {
        cmd := exec.Command(command, args...)

+       cmd.Env = os.Environ()
        for k, v := range e.envVars {
                cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", k, v))
        }

I'll probably use the separate commands for now anyway.

[Callisto13]

Cool, and thanks for the fix tip 😄

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[artyom-p]

Same issue for me, solving that by running flux bootstrap manually after cluster is provisioned, but it would be really nice to make it work through eksctl.