Add ability to execute scripts via Kubernetes Jobs #690

APErebus · 2023-11-27T00:11:17Z

Background

This is the big one 😱

This adds support for executing scripts supplied to ScriptServiceV3Alpha by Kubernetes Jobs.

Results

Adds a new KubernetesJobScriptExecutor that is used when the environment variable OCTOPUS__TENTACLE__K8SUSEJOBS is set to True

At a high level, the new RunningKubernetesJobScript creates a new k8s Job and executes the Bootstrap script sent from server in the octopuslabs/k8-workertools container image. It then starts two background processes, one to monitor the status of the job itself (so it knows when it's finished) and another to read the streamed log files (stdout.log and stderr.log) and write them into the Output.log. For more info on this process, see below

The tag of the worker tools image used is linked to the version of k8s the tentacle is running in, which we query via the API.

It's implied that there is also a volume called tentacle-home that has access to the tentacle's work directory so it can execute the script. This is supplied as raw yaml in the OCTOPUS__TENTACLE__K8SJOBVOLUMEYAML environment variable.

Script Logs

As Tentacle needs to write the Output.log in a thread-safe manner (so logs don't get munged), we execute the bootstrap script using a bootstrapRunner.sh script which redirects the stdout and stderr streams from the Bootstrap script into two files in the work directory, stdout.log and stderr.log. Then there is a process in the KubernetesJobOutputStreamWriter class that periodically (every 250ms) tail observes those files for new log entries. When there are, they are then written to the tentacle's Output.log under the same locking mechanism as the rest of the code.

Once the job is finished, we perform one last read of those files just in case there was final log items flushed.

Testing

Currently there is no automated tests for this code. I'm still working on refining it so manual testing will suffice for now until the codebase stabilises.

I was able to successfully run a health check for the the Kubernetes Tentacle in my local server.

Shortcut story: [sc-59580]

After

How to review this PR

Quality ✔️

Pre-requisites

I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
I have considered informing or consulting the right people, according to the ownership map.
I have considered appropriate testing for my change.

APErebus · 2023-11-27T02:29:12Z

source/Octopus.Tentacle/Kubernetes/InClusterKubernetesClientConfigProvider.cs

+    {
+        public KubernetesClientConfiguration Get()
+        {
+            return KubernetesClientConfiguration.InClusterConfig();


When we are running in a cluster, this will just use the ambient service account auth

APErebus · 2023-11-27T02:29:44Z

source/Octopus.Tentacle/Kubernetes/KubernetesClusterService.cs

+            {
+                var versionInfo = await Client.Version.GetCodeAsync();
+
+                return new ClusterVersion(int.Parse(versionInfo.Major), int.Parse(versionInfo.Minor));


the patch number of a k8s cluster is not provided by the cluster itself (and isn't important for what we need it for)

APErebus · 2023-11-27T02:30:31Z

source/Octopus.Tentacle/Kubernetes/KubernetesJobService.cs

+            }
+        }
+
+        public async Task Watch(ScriptTicket scriptTicket, Func<V1Job, bool> onChange, Action<Exception> onError, CancellationToken cancellationToken)


Rather than polling the TryGet endpoint continually, this creates an IAsyncEnumerable which yields when there is a change

APErebus · 2023-11-27T02:31:07Z

source/Octopus.Tentacle/Kubernetes/bootstrapRunner.sh

+
+format() {
+	now=$(date -u +"%Y-%m-%dT%H:%M:%S.%N%z")
+	echo "$now|$2" | tee -a "$1"


we use tee so the logs are written to the job/pod logs as well as the log files

APErebus · 2023-11-27T02:31:53Z

source/Octopus.Tentacle/Kubernetes/bootstrapRunner.sh

+# This ungodly hack is to stop the pod from being killed before the last log has been flushed
+sleep 0.250 #250ms


It seems like the pod is destroyed before the file logs are flushed. This little hack just gives the pod time to flush the last redirected output to the files

shortcut-integration · 2023-11-27T02:35:44Z

This pull request has been linked to Shortcut Story #59580: Add ability for Tentacle to schedule new k8s jobs when running scripts/deployments.

zentron

Nice work, It took a bunch of time work to get to this point but it looks fairly clean. And nice that it was done in conjunction with the H&E group.

Just a couple questions here and there, I dont think anything particularly structural.

source/Octopus.Tentacle/ExternalInit.cs

zentron · 2023-11-28T20:35:15Z

source/Octopus.Tentacle/Kubernetes/KubernetesConfig.cs

+{
+    public static class KubernetesConfig
+    {
+        public static string Namespace => Environment.GetEnvironmentVariable("OCTOPUS__K8STENTACLE__NAMESPACE")


Minor suggestion, might be a little cleaner with a local helper function since there are a bunch of environment variable lookups with this pattern.
Given its only 3 its borderline that its needed

I chose to pull out a common method in a2a48b7

source/Octopus.Tentacle/Kubernetes/KubernetesJobService.cs

zentron · 2023-11-28T20:41:15Z

source/Octopus.Tentacle/Kubernetes/KubernetesJobService.cs

+                timeoutSeconds: 1800, //same as the TTL of the job itself
+                cancellationToken: cancellationToken);
+
+            await foreach (var (type, job) in response.WatchAsync<V1Job, V1JobList>(onError, cancellationToken: cancellationToken))


Interesting pattern 👍

zentron · 2023-11-29T00:44:15Z

source/Octopus.Tentacle/Scripts/Kubernetes/RunningKubernetesJobScript.cs

+                using var writer = ScriptLog.CreateWriter();
+                try
+                {
+                    using (ScriptIsolationMutex.Acquire(workspace.IsolationLevel,


This presumably works because when running one instance, the mutex is accessable by that one Tentacle process.
If/when we allow multiple instances to run within a cluster we may need to answer if this isolationlevel check is still relevant.

Correct. When/if we get to HA, then we'll need a different approach to how to do the script isolation mutex or, as you say, if it's still relevant

zentron · 2023-11-29T00:45:49Z

source/Octopus.Tentacle/Scripts/Kubernetes/RunningKubernetesJobScript.cs

+                            },
+                            ServiceAccountName = KubernetesConfig.JobServiceAccountName,
+                            RestartPolicy = "Never",
+                            Volumes = volumes


Nice to see that worked :)

zentron · 2023-11-29T00:48:26Z

source/Octopus.Tentacle/Scripts/Kubernetes/RunningKubernetesJobScript.cs

+                                    },
+                                    Env = new List<V1EnvVar>
+                                    {
+                                        new(EnvironmentVariables.TentacleHome, $"/data/tentacle-home/{instanceName}"),


Some of those other environment variables sound like they could be relevant for Calamari. TentacleJournal or TentacleInstanceName?

hrm, good call. I'll investigate. I added these ones because they were needed for the health-check

Yep, I missed TentacleJournal and TentacleInstanceName. Added in a2a48b7

zentron · 2023-11-29T00:51:02Z

source/Octopus.Tentacle/Scripts/Kubernetes/RunningKubernetesJobScript.cs

+            new(1, 28, 2),
+        };
+
+        async Task<string> GetDefaultContainer()


Minor, consider pulling some parts of this class such as this worker tool container resolution into its own class that can be tested seperately. ALso makes this class easier to grok.

Done. Created a new KubernetesJobContainerResolver in a2a48b7

zentron · 2023-11-29T00:52:42Z

source/Octopus.Tentacle/Scripts/ScriptExecutorFactory.cs

        }

        public IScriptExecutor GetExecutor()
        {
-            return shellScriptExecutor.Value;
+            return KubernetesConfig.UseJobs switch


Would it be simpler to push as much of the "if k8s do this" into being container registration concern?

yeah, fair call. I'll move this into the module and only register the appropriate executor based on the env var

Done in a2a48b7

There is no longer a factory and the script executors are now conditionaly registered in the Autofac container

Co-authored-by: Rob E <[email protected]>

APErebus · 2023-11-29T02:19:09Z

@zentron I've updated the PR with the changes suggested. I did end up removing the IScriptExecutorFactory and moving it into the module.

I also pulled the container stuff into a new class, IKubernetesJobContainerResolver

zentron

Nice one, this is a huge piece of the puzzle 👍

APErebus force-pushed the ap/k8s-jobs-script-execution branch from 562aea6 to 58762ac Compare November 27, 2023 00:34

APErebus mentioned this pull request Nov 27, 2023

Add support for running scripts as kubernetes jobs #672

Closed

3 tasks

APErebus added 20 commits November 27, 2023 13:08

Add Kubernetes Client SDK

4666cec

Support executing scripts via Kubernetes Job

8e74d6e

Continued changes

b8d66f4

Add ScriptStateStore support to k8s jobs

548af9d

Add support for supplying volume info via env vars

4b2cc93

Set up for running in WSL

7f998e8

Adjust run config

4d8b21d

Fix a number of issues with running k8s scripts

4a4670f

Update script runner with new code

ecb14b0

Try building the scriptrunner as a self-contained app

282580e

Build ScriptRunner as self-contained single file package

0b2ad67

Run the script runner as the entry point

21d66e6

Run bash script to handle output and error logs

6fa5a4c

Read from job streams and write to Output.log

a057ba7

Remove Kubernetes.ScriptRunner

af2b333

Self-cleanup

966c6c2

Cleanup after self-review

f579f1c

More minor cleanup

8843207

Remove ScriptRunner configuration

f7cb8a8

Fix solution

5069e7a

APErebus force-pushed the ap/k8s-jobs-script-execution branch from b6234e2 to 5069e7a Compare November 27, 2023 02:09

APErebus added 4 commits November 27, 2023 13:10

Fix rebase issue

b524284

Revert global.json

571df71

Remove duplicate package references

6d2b9e5

Remove run configuration

cd625dd

APErebus marked this pull request as ready for review November 27, 2023 02:28

APErebus requested a review from a team as a code owner November 27, 2023 02:28

APErebus commented Nov 27, 2023

View reviewed changes

APErebus mentioned this pull request Nov 27, 2023

Add KubernetesJobScriptExecutionContext #693

Merged

3 tasks

APErebus and others added 3 commits November 27, 2023 15:18

Merge branch 'main' into ap/k8s-jobs-script-execution

5092c07

Update environment variables

96ad11b

Merge branch 'main' into ap/k8s-jobs-script-execution

4a20af7

APErebus mentioned this pull request Nov 28, 2023

Add kubernetes tentacle docker build #699

Merged

3 tasks

zentron requested changes Nov 29, 2023

View reviewed changes

APErebus and others added 3 commits November 29, 2023 12:45

Change to exception filter

97f639b

Co-authored-by: Rob E <[email protected]>

PR feedback

a2a48b7

Merge branch 'main' into ap/k8s-jobs-script-execution

e5f2f63

APErebus requested a review from zentron November 29, 2023 02:17

zentron approved these changes Nov 29, 2023

View reviewed changes

APErebus merged commit be796e5 into main Nov 29, 2023
2 checks passed

APErebus deleted the ap/k8s-jobs-script-execution branch November 29, 2023 04:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to execute scripts via Kubernetes Jobs #690

Add ability to execute scripts via Kubernetes Jobs #690

APErebus commented Nov 27, 2023 •

edited

Loading

APErebus Nov 27, 2023

APErebus Nov 27, 2023

APErebus Nov 27, 2023

APErebus Nov 27, 2023

APErebus Nov 27, 2023

shortcut-integration bot commented Nov 27, 2023

zentron left a comment

zentron Nov 28, 2023

APErebus Nov 29, 2023

zentron Nov 28, 2023

zentron Nov 29, 2023

APErebus Nov 29, 2023

zentron Nov 29, 2023

zentron Nov 29, 2023

APErebus Nov 29, 2023

APErebus Nov 29, 2023 •

edited

Loading

zentron Nov 29, 2023

APErebus Nov 29, 2023

zentron Nov 29, 2023

APErebus Nov 29, 2023

APErebus Nov 29, 2023

APErebus commented Nov 29, 2023

zentron left a comment

		# This ungodly hack is to stop the pod from being killed before the last log has been flushed
		sleep 0.250 #250ms

Add ability to execute scripts via Kubernetes Jobs #690

Add ability to execute scripts via Kubernetes Jobs #690

Conversation

APErebus commented Nov 27, 2023 • edited Loading

Background

Results

Script Logs

Testing

After

How to review this PR

Pre-requisites

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shortcut-integration bot commented Nov 27, 2023

zentron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

APErebus Nov 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

APErebus commented Nov 29, 2023

zentron left a comment

Choose a reason for hiding this comment

APErebus commented Nov 27, 2023 •

edited

Loading

APErebus Nov 29, 2023 •

edited

Loading