Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hot-reloading on windows doesn't appear to work natively #2129

Closed
wyrie-zz opened this issue Nov 5, 2020 · 9 comments · Fixed by #2218
Closed

Hot-reloading on windows doesn't appear to work natively #2129

wyrie-zz opened this issue Nov 5, 2020 · 9 comments · Fixed by #2218
Assignees
@thsig
Labels
bug needs-more-info priority:medium Medium priority issue or feature

Comments

@wyrie-zz
Copy link

wyrie-zz commented Nov 5, 2020

Bug

Windows users running garden outside of WSL on various shells cannot use hot-reloading. The issue we are seeing relates to how paths are handled on windows.

Current Behavior

A windows user, in our configuration, trying to launch a hot-reload session would get a Error: bind EADDRINUSE message. This comes from garden wanting to establish a port forwarding session but fails.

The scenario above occurs in a bash shell and is due to the fact that the path to the kubectl binary is resolved to a windows style path (i.e. C:\...), which the bash shell does not understand and cannot therefore establish a port-forwarding session.

Switching to powershell we get past the error above but then hit a different path issue when it comes to rsync. Rsync resolves paths to /cygdrive/c/ which then is not understood in powershell.

Expected behavior

Hot reloading should work in bash and powershell on windows, with the path handling considered and resolved correctly.

Reproducible example

Key elements to our environment are:

  • Windows 10
  • git bash for windows
  • powershell
  • a garden project with:
    • hot reloading module
    • another deployed with a helm module that is a dependent

Workaround

None

Suggested solution(s)

This all looks entirely related to how paths are handled. Making that more robust should squash this one.

Additional context

https://kubernetes.slack.com/archives/CKM7CP8P9/p1604396117244300

Your environment

  • OS: Windows
  • How I'm running Kubernetes: EKS, docker-desktop

garden version
0.12.9
@eysi09
Copy link
Collaborator

eysi09 commented Nov 6, 2020

Thanks for submitting the issue. For what it's worth, we test every release manually on a Windows machine in Powershell and the cygdrive paths haven't caused any issues, but perhaps that's particular to our setup. And admittedly we haven't tested the releases on git bash.

Assigning this to @thsig who's most familiar with the hot reloading.

@eysi09 eysi09 added the bug label Nov 6, 2020
@wyrie-zz
Copy link
Author

wyrie-zz commented Nov 7, 2020

@eysi09 that’s good to know. So does that mean if cygwin is installed it’ll probably work in the meantime?

@eysi09 eysi09 added the priority:high High priority issue or feature label Nov 13, 2020
@eysi09
Copy link
Collaborator

eysi09 commented Nov 13, 2020

Apologies for the late reply, but yes, it should. At least that's been the case on our end. And we're actively looking into this alongside other changes we're making to better handle paths on Windows.

edvald added a commit that referenced this issue Nov 30, 2020
@edvald
fix(core): fix failing port forwards on Windows in certain scenarios
c2828e3 
Relates to #2129
@edvald edvald mentioned this issue Nov 30, 2020
Merged
edvald added a commit that referenced this issue Dec 1, 2020
@edvald
fix(core): fix failing port forwards on Windows in certain scenarios
358efa3 
Relates to #2129
@eysi09 eysi09 added priority:medium Medium priority issue or feature needs-more-info and removed priority:high High priority issue or feature labels Dec 11, 2020
@eysi09
Copy link
Collaborator

eysi09 commented Dec 11, 2020

@wyrie, are you still bumping into this? AFAICT, we don't have Cygwin installed on our testing machine, and the assumption was that Powershell actually handles the /cygdrive/ paths. At least that's been our experience so far.

Is there anything in your setup that you can think of that would explain the difference?

@MarvDann
Copy link

Hi Guys, I can confirm after upgrading to version 0.12.11, we are still running into hot deploy issues.
I'm running commands from git bash like this
garden deploy --hot=ai-ui --env=martin-aub-6648.remote

Error we are seeing is
⏳ Processing... Error: bind EADDRINUSE 127.10.0.70:25672 at listenOnMasterHandle (net.js:1380:18) at shared (internal/cluster/child.js:126:3) at Worker.<anonymous> (internal/cluster/child.js:97:7) at process.onInternalMessage (internal/cluster/utils.js:47:8) at process.emit (events.js:327:22) at process.<anonymous> (C:\snapshot\project\tmp\pkg\cli\node_modules\source-map-support\source-map-support.js:495:21) at processEmit [as emit] (C:\snapshot\project\tmp\pkg\cli\node_modules\signal-exit\index.js:161:32) at emit (internal/child_process.js:876:12) at processTicksAndRejections (internal/process/task_queues.js:85:21) ✨ Done in 146.29s.

I'll give it a go with PowerShell

@edvald
Copy link
Collaborator

edvald commented Dec 15, 2020

Thanks for the update @MarvDann. Still trying to track this down because we've not been able to reproduce the error. I suspect it has something to do with differences in our Windows setup.

Perhaps we could ask you to try one thing: Run garden deploy --watch --env=martin-aub-6648.remote --log-level=debug (adding the log level and using --watch instead of with the --hot flag), see if that causes the same problem, and if so paste the logs surrounding the error message. I have a hunch, and this would help either confirm or reject it :)

@MarvDann
Copy link

Interesting update: - I deleted my environment, restarted my machine and tried the same command i tried previously (with the hot flag) and it worked.
It seems to be an issue if I have already ran a deploy command previously and then I run a subsequent deploy command with the hot flag over the top of it without first deleting my environment.

I'll try the command you suggested next to see how that works out.

@edvald
Copy link
Collaborator

edvald commented Dec 15, 2020

Ah, interesting. That's still a bug we need fix ofc, but helps narrow this down.

@MarvDann
Copy link

MarvDann commented Dec 15, 2020 •
edited
Loading

@edvald I ran the command you suggested with the --watch flag and increased log level.

Here are the logs I'm getting....

Deploy �


Retrieved client auth token from local config db
�  Garden dashboard running at http://localhost:9777
i providers                 → Getting status...
   Execing 'C:\Users\martin\.garden\tools\kubectl\ae11aaa356d92add\kubectl.exe --context=ec-dev-k8s-eks-cluster config view --raw' in C:\Users\martin\.garden\tools\kubectl\ae11aaa356d92add  
Something went wrong while checking for the latest Garden version.
   i garden system             → Initializing...
   Checking Garden system service status for plugin "kubernetes": current version 0.12.11, version in cluster: 0.12.9, oldest permitted version: 0.9.0, up to date: true
   Kubernetes: Getting API resource info for group v1
   Getting currently deployed resource statuses...
   Comparing expected and deployed resources...
   All resources match.
i providers                 → Getting status...
√ providers                 → Getting status... → Done
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system status garden-build-sync --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system status garden-registry-proxy --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system status garden-docker-registry --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system status garden-util-daemon --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get values garden-util-daemon --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get values garden-registry-proxy --output 
json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get values garden-build-sync --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get values garden-docker-registry --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get manifest garden-util-daemon' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get manifest garden-build-sync' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get manifest garden-registry-proxy' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace garden-system get manifest garden-docker-registry' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
   √ garden system             → Initializing...
√ providers                 → Getting status... → Cached
   i Run with --force-refresh to force a refresh of provider statuses.
i Service docs is disabled

⏳  Processing...
i rabbitmq                  → Syncing module sources (44 files)...
i postgres                  → Syncing module sources (45 files)...
i minio                     → Syncing module sources (17 files)...
i base-chart                → Syncing module sources (8 files)...
i python-base               → Syncing module sources (1 file)...
i ruby-base                 → Syncing module sources (1 file)...
i node-base                 → Syncing module sources (1 file)...
i nginx-base                → Syncing module sources (2 files)...
Kubernetes: Getting API resource info for group networking.k8s.io/v1beta1
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 status postgres --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Syncing 0 files from ..\..\cygdrive\c\Projects\dai\services to ..\..\cygdrive\c\Projects\dai\.garden\build\ingress-services (with delete)
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 status minio --output json' in C:\U s
ers\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
   Syncing 45 files from ..\..\cygdrive\c\Projects\dai\services\postgresql to ..\..\cygdrive\c\Projects\dai\.garden\build\postgres (with delete)
   Syncing 44 files from ..\..\cygdrive\c\Projects\dai\services\rabbitmq to ..\..\cygdrive\c\Projects\dai\.garden\build\rabbitmq (with delete)
   Syncing 17 files from ..\..\cygdrive\c\Projects\dai\services\minio to ..\..\cygdrive\c\Projects\dai\.garden\build\minio (with delete)
   Syncing 1 files from ..\..\cygdrive\c\Projects\dai to ..\..\cygdrive\c\Projects\dai\.garden\build\python-base (with delete)
   Syncing 1 files from ..\..\cygdrive\c\Projects\dai to ..\..\cygdrive\c\Projects\dai\.garden\build\node-base (with delete)
   Syncing 2 files from ..\..\cygdrive\c\Projects\dai to ..\..\cygdrive\c\Projects\dai\.garden\build\nginx-base (with delete)
   Syncing 1 files from ..\..\cygdrive\c\Projects\dai to ..\..\cygdrive\c\Projects\dai\.garden\build\ruby-base (with delete)
   Syncing 8 files from ..\..\cygdrive\c\Projects\dai\services\base-chart to ..\..\cygdrive\c\Projects\dai\.garden\build\base-chart (with delete)
Getting currently deployed resource statuses...
Comparing expected and deployed resources...
All resources match.
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 status rabbitmq --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 get values postgres --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 get values minio --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
√ rabbitmq                  → Syncing module sources (44 files)... → Done (took 5.8 sec)
i rabbitmq                  → Getting build status for v-fe19dd5db5...
i rabbitmq                  → Building version v-fe19dd5db5...
√ rabbitmq                  → Building version v-fe19dd5db5... → Done (took 0.1 sec)
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 get values rabbitmq --output json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
√ postgres                  → Syncing module sources (45 files)... → Done (took 6.5 sec)
i postgres                  → Getting build status for v-494951100e...
i postgres                  → Building version v-494951100e...
√ postgres                  → Building version v-494951100e... → Done (took 0 sec)
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 get manifest minio' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 get manifest postgres' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
√ minio                     → Syncing module sources (17 files)... → Done (took 11.7 sec)
i minio                     → Getting build status for v-580b7266cb...
i minio                     → Building version v-580b7266cb...
√ minio                     → Building version v-580b7266cb... → Done (took 0 sec)
i minio                     → Deploying version v-580b7266cb...
√ minio                     → Deploying version v-580b7266cb... → Already deployed
   Starting proxy to minio/minio:9000 on 127.10.0.67:9000
   → Forward: 127.10.0.67:9000 → minio:9000 (minio)
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 get manifest rabbitmq' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
i postgres                  → Deploying version v-494951100e...
√ postgres                  → Deploying version v-494951100e... → Already deployed
   Starting proxy to postgres/postgres-headless:5432 on 127.10.0.68:5432
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 status sut-service --output json' i n
 C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 status task-scheduler-service --out p
ut json' in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
   Starting proxy to postgres/postgres:5432 on 127.10.0.69:51046
Execing 'C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64\helm.exe --kube-context ec-dev-k8s-eks-cluster --namespace dai-martin-aub-6648 status asset-manager --output json' 
in C:\Users\martin\.garden\tools\helm\b953dd95f12c100f\windows-amd64
   → Forward: postgres://127.10.0.68:5432 → postgres-headless:5432 (tcp-postgresql)
   → Forward: postgres://127.10.0.69:51046 → postgres:5432 (tcp-postgresql)
Error: bind EADDRINUSE 127.10.0.69:51046
    at listenOnMasterHandle (net.js:1380:18)
    at shared (internal/cluster/child.js:126:3)
    at Worker.<anonymous> (internal/cluster/child.js:97:7)
    at process.onInternalMessage (internal/cluster/utils.js:47:8)
    at process.emit (events.js:327:22)
    at process.<anonymous> (C:\snapshot\project\tmp\pkg\cli\node_modules\source-map-support\source-map-support.js:495:21)
    at processEmit [as emit] (C:\snapshot\project\tmp\pkg\cli\node_modules\signal-exit\index.js:161:32)
    at emit (internal/child_process.js:876:12)
    at processTicksAndRejections (internal/process/task_queues.js:85:21)
✨  Done in 167.15s.```

edvald added a commit that referenced this issue Jan 20, 2021
@edvald
fix(k8s): hanging port forward processes on Windows
b3233bd 
Fixes #2129
@edvald edvald mentioned this issue Jan 20, 2021
Merged
thsig pushed a commit that referenced this issue Jan 20, 2021
@edvald @thsig
fix(k8s): hanging port forward processes on Windows
d6dcd45 
Fixes #2129
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thsig thsig

Labels
bug needs-more-info priority:medium Medium priority issue or feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix port forward issues on Windows garden-io/garden
5 participants
@edvald @thsig @eysi09 @MarvDann @wyrie-zz