Unexpected upload network traffic while tailing K8s logs

[mattpolzin]

Bug

Current Behavior

While running tasks or building new Docker images, upload bandwidth spikes to 1Mbps (the max for my slow internet connection) and it stays maxed out for the duration of the task execution (and usually for up to a minute after the task container has "completed" as seen via k9s). The Garden CLI does not report the task has finished when it actually finishes (as noted, it will sometimes succeed or even report a failure a minute after the task has finished successfully in the remote container) because it is apparently bogged down with the network requests it is making. This is not traffic related to sending build context to the cluster, that traffic actually stays closer to 300KB/s upload. As a side effect of bandwidth being maxed out, I also see failed connections frequently which results in Garden reporting a failure to me even though things were working fine in the cluster (the problem was merely communication with the cluster).

I was able to confirm that for some reason this upload traffic is entirely related to tailing logs by commenting out the following code in a local build of Garden:

diff --git a/core/src/plugins/kubernetes/logs.ts b/core/src/plugins/kubernetes/logs.ts
index 34e235e17..a32ed241b 100644
--- a/core/src/plugins/kubernetes/logs.ts
+++ b/core/src/plugins/kubernetes/logs.ts
@@ -230,9 +230,9 @@ export class K8sLogFollower<T> {
   public async followLogs(opts: LogOpts) {
     await this.createConnections(opts)
 
-    this.intervalId = setInterval(async () => {
-      await this.createConnections(opts)
-    }, this.retryIntervalMs)
+//    this.intervalId = setInterval(async () => {
+//      await this.createConnections(opts)
+//    }, this.retryIntervalMs)
 
     return new Promise((resolve, _reject) => {
       this.resolve = resolve

When the above code has been commented out, upload bandwidth stays at almost 0 during task execution rather than constantly maxed out at 1Mbps. The problem of failed connections also goes away because network contention is no longer a problem.

Expected behavior

I expect the Garden process of running a task that is entirely contained in the cluster (no transmission of data for the operation of the task once the Docker image has been built) to result in very little upload network bandwidth usage.

Workaround

Build a custom version of Garden with log tailing commented out. This is what I am working with currently because the experience is so drastically better.

Your environment

• OS: macOS 13.0.1
• How I'm running Kubernetes: Azure

garden version: 0.12.48

[to266]

Oh, we've also observed that!

[mcsteele8]

This is also a big blocker for me. I have seen this same issue where a task has completed but the garden-cli does not proceed because it is stuck waiting for response from remote resource. Big uptick in network consumption is usually aways involved and block other applications from being able to work. Example: slack, chrome, k9s all stop working unit the garden
cmd is killed

[eysi09]

Thanks for flagging this!

We are aware and addressing this is top priority.

[stefreak]

@mattpolzin @to266 @mcsteele8 the fix now landed in main! 🥳 Sorry that it took so long; we wanted to get this one right.

Can you run garden self-update edge and see if it works with the latest edge version?

[mattpolzin]

@stefreak in my very preliminary testing, this makes all the difference!

[stefreak]

Awesome, glad to hear that!

[mcsteele8]

@eysi09 Do you know when the next garden release will be that will include this change?

[stefreak]

@mcsteele8 @mattpolzin @to266 hey everyone, I just wanted to let you know that this has been released as part of 0.12.53

[mattpolzin]

Thanks for putting all of that work into fixing this!

[stefreak]

Honestly, it was a pleasure :) Thank you so much for the high quality report on this, and your patience.