fix(k8s): more stable & performant log streaming (#3730)

* fix(k8s): more stable & performant log streaming 

Fixes an issue with very high upload bandwidth use when running Kubernetes based tests/tasks that produce a lot of log output.

co-written by @stefreak and @thsig

- Improved connection management and pod lifecycle logic, including more robust connection timeout enforcement.
- Removed keepalive logic, since it doesn't work on all operating systems.
- Improved deduplication logic to generate fewer false positives (and eliminate false-negatives).
- Use sinceTime when fetching logs on retry to make sure we don't fetch any unnecessary logs.
- When a runner pod terminates, we make sure to wait until the final logs have been fetched.
- Default to using the tail option in conjunction with a "max log lines in memory" setting instead of limitBytes to avoid clipping / incomplete log lines while also avoiding the loading of too much log data into memory.
- Only start one connection attempt at a time, to prevent multiple connections to the same container at once.
- Make sure that we only call createConnections once it has finished, so that there is only one concurrent instance of the method running per LogFollower at a time.

Fixes #3586.

co-authored-by: thsig <[email protected]>
co-authored-by: Steffen Neubauer <[email protected]>

* fix(k8s): make sure LogFollower only connects once

I noticed the following log message when I increased latency & packet
loss in Network Link Conditioner:

```
[silly] <Not connected to container vault in Pod vault-0. Connection status is connecting>
```

This means the connection is not established yet, but the LogFollower is
connecting yet again (which causes a vicious cycle and makes the
internet connection even worse).

This is probably the root cause for the issue described in #3586.

With this bug fixed, I am 100% certain this PR Fixes #3586

Co-authored-by: Thorarinn Sigurdsson <[email protected]>
co-authored-by: Steffen Neubauer <[email protected]>

* improvement(k8s): cap age of logs on retry attempt in `garden logs`

When streaming logs from the k8s api using `garden logs`, we do not want
to stream old log messages as the user might have been disconnected for
a long time (e.g. when the laptop went to sleep)

Co-authored-by: Eyþór Magnússon <[email protected]>
co-authored-by: Steffen Neubauer <[email protected]>

---------

Co-authored-by: Steffen Neubauer <[email protected]>
Co-authored-by: thsig <[email protected]>
Co-authored-by: Eyþór Magnússon <[email protected]>

core/src/commands/logs.ts

@@ -42,7 +42,9 @@ const logsOpts = {
     `,
   }),
   "follow": new BooleanParameter({
-    help: "Continuously stream new logs from the service(s).",
+    help: deline`
+    Continuously stream new logs from the service(s).
+    When the \`--follow\` option is set, we default to \`--since 1m\`.`,
     alias: "f",
   }),
   "tail": new IntegerParameter({
@@ -104,8 +106,8 @@ export class LogsCommand extends Command<Args, Opts> {
   help = "Retrieves the most recent logs for the specified service(s)."
 
   description = dedent`
-    Outputs logs for all or specified services, and optionally waits for news logs to come in. Defaults
-    to getting logs from the last minute when in \`--follow\` mode. You can change this with the \`--since\` option.
+    Outputs logs for all or specified services, and optionally waits for news logs to come in. Defaults to getting logs
+    from the last minute when in \`--follow\` mode. You can change this with the \`--since\` or \`--tail\` options.
 
     Examples: