chore: deprecating container_runtime config, agentrm supporting singularity,podman, and apptainer #9516

ShreyaLnuHpe · 2024-06-13T18:03:02Z

Ticket

Description

Deprecate all container runtimes config except for Docker for AgentRM

Potential Impact: Can only run AgentRM with Docker container runtime. Can no longer run with Apptainer/Singularity or podman.

AgentRM: Support for Singluarity, Podman, and Apptainer has been deprecated in 0.33.0 and is now
removed. Docker is the only container runtime supported by Agent resource manager (AgentRM). It
is still possible to use podman with AgentRM by using the podman emulation layer. For detailed
instructions, follow steps in the link: Emulating Docker CLI with Podman. You
might need to also configure checkpoint_storage in experiment or master configurations: Master
Config Reference

In enterprise edition, Slurm resource manager still supports singularity, podman, or apptainer use.
For detailed instructions, visit :ref:deploy-on-slurm-pbs.

Test Plan

Pass CircleCI + Manual testing by setting the below code in tools/devcluster.yaml file
happy case: container_runtime: docker
unhappy case: container_runtime: podman

example:

- agent:
      pre:
        - sh: make -C agent build
      cmdline:
        - agent/build/determined-agent
        - run
        - --config-file
        - :config

      # config_file is just an agent.yaml
      config_file:
        master_host: 127.0.0.1
        master_port: 8080
        container_master_host: $DOCKER_LOCALHOST
        log:
          level: trace
        container_runtime: docker

command to run- % devcluster -c tools/devcluster.yaml
experiment command-
% cd examples/tutorials/mnist_pytorch
% det experiment create const.yaml .

Checklist

Changes have been manually QA'd
User-facing API changes need the "User-facing API Change" label.
Release notes should be added as a separate file under docs/release-notes/.
See Release Note for details.
Licenses should be included for new code which was copied and/or modified from any external code.

netlify · 2024-06-13T18:03:21Z

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	`b6c4e44`
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/669174b782a450000804251b

ShreyaLnuHpe · 2024-06-13T18:07:13Z

agent/internal/agent.go

+	}()
+	docker := docker.NewClient(dcl)
+
+	// a.log.Tracef("setting up %s runtime", a.opts.ContainerRuntime)


Commented a few parts of code instead of completely removing it as we are unclear if we want to deprecate whole of container_runtime (configuration option/variable) or just overloaded use of "container runtime" (container "providers", e.g. docker, podman, apptainer).

I will make corrections as needed in future commits

I think the only valid option for container_runtime should be docker

so if I had a config file that specified container_runtime: docker I would expect that to not error after this change

every other container runtime I would expect the agent to return an error message saying it can't run

NicholasBlaskey · 2024-06-13T19:31:33Z

agent/cmd/determined-agent/init.go

@@ -152,6 +152,6 @@ func registerAgentConfig() {
 	registerInt(flags, name("agent-reconnect-backoff"), defaults.AgentReconnectBackoff,
 		"Time between agent reconnect attempts")

-	registerString(flags, name("container-runtime"), defaults.ContainerRuntime,
-		"The container runtime to use")
+	registerString(flags, name("docker-container-runtime"), defaults.DockerContainerRuntime,


What is this change?

change is to make config definition for /etc/determined/agent.yaml file.
My changes will be undone here after your suggestions below.

NicholasBlaskey · 2024-06-13T19:33:28Z

agent/internal/agent.go

+	}()
+	docker := docker.NewClient(dcl)
+
+	// a.log.Tracef("setting up %s runtime", a.opts.ContainerRuntime)


I think the only valid option for container_runtime should be docker

so if I had a config file that specified container_runtime: docker I would expect that to not error after this change

NicholasBlaskey · 2024-06-13T19:33:53Z

agent/internal/agent.go

+	}()
+	docker := docker.NewClient(dcl)
+
+	// a.log.Tracef("setting up %s runtime", a.opts.ContainerRuntime)


every other container runtime I would expect the agent to return an error message saying it can't run

NicholasBlaskey · 2024-06-13T19:36:25Z

agent/internal/container/container_runtime.go

-)
-
-// ContainerRuntime is our interface for interacting with runtimes like Docker or Singularity.
-type ContainerRuntime interface {


I kinda like the interface. I think its a lot harder to go the other way (that is put code behind a clean interface when it isn't) than it is to go from a clean interface to not having one.

So I think my preference would be to just leave this in here for now.

NicholasBlaskey · 2024-06-13T19:37:30Z

agent/cmd/determined-agent/init.go

@@ -152,6 +152,6 @@ func registerAgentConfig() {
 	registerInt(flags, name("agent-reconnect-backoff"), defaults.AgentReconnectBackoff,


I think we are going to need a lot of changes to the circleci file to no longer run agent tests

ShreyaLnuHpe · 2024-06-14T19:29:07Z

agent/internal/options/options.go

-	// requires root or a suid installation with /etc/subuid --fakeroot.
-	AllowNetworkCreation bool `json:"allow_network_creation"`
-}
-
 // PodmanOptions configures how we interact with podman.
 type PodmanOptions struct {


I am currently leaving this out. Would remove it if not needed in the next subtask

codecov · 2024-06-14T19:29:43Z

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 53.07%. Comparing base (0a57cde) to head (b6c4e44).
Report is 23 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9516      +/-   ##
==========================================
+ Coverage   52.91%   53.07%   +0.16%     
==========================================
  Files        1255     1253       -2     
  Lines      153086   152507     -579     
  Branches     3230     3229       -1     
==========================================
- Hits        81004    80950      -54     
+ Misses      71931    71406     -525     
  Partials      151      151

Flag	Coverage Δ
backend	`44.47% <0.00%> (+0.38%)`	⬆️
harness	`72.76% <ø> (ø)`
web	`51.30% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
agent/cmd/determined-agent/init.go	`100.00% <ø> (ø)`
agent/internal/options/options.go	`29.09% <ø> (ø)`
agent/internal/agent.go	`0.00% <0.00%> (ø)`

... and 6 files with indirect coverage changes

kkunapuli · 2024-06-14T20:18:07Z

agent/internal/agent.go

@@ -124,29 +122,14 @@ func (a *Agent) run(ctx context.Context) error {
 	var cruntime container.ContainerRuntime
 	switch a.opts.ContainerRuntime {
 	case options.PodmanContainerRuntime:


I think we also want to remove this switch statement; no one should even be using the configuration option.

I think we should simply set cruntime to options.DockerContainerRuntime (or whatever makes sense in the code). We can check if a.opts.ContainerRuntime has a value and is not docker if we want to print a generic warning ... but I think we can simplify this section of the code.

Yes, earlier i removed the switch statement all together and had only Docker configs, but @NicholasBlaskey suggested we throw error in here

I agree to both cases, maybe now we can return an error, and later just have a case default which handles with a generic error message

This configuration option shouldn't be known to users; we do not want them using it even if they have discovered it.

It looks like docker is the default for container runtime configuration. I think this because there's no default clause in the original switch statement, but you should confirm this is true.

I think we can have a much simpler check. In pseudo-code, something like:

if a.opts.ContainerRuntime != "" and a.opts.ContainerRuntime != "Docker" { // print warning // error and exit } // continue to set up ContainerRuntime interface using Docker

Let me know if this is still confusing and we can go over the code together! (I might be able to explain it better in a call :)

kkunapuli · 2024-06-14T20:19:32Z

agent/internal/options/options.go

-	SingularityOptions SingularityOptions `json:"singularity_options"`
-	PodmanOptions      PodmanOptions      `json:"podman_options"`
+	ContainerRuntime string        `json:"container_runtime"`
+	PodmanOptions    PodmanOptions `json:"podman_options"`


We don't want users to be using these configuration options. I'm ok with removing them in the next set of work though if that's what you prefer (I saw one of your comments).

I was thinking of having contianer_runtime configuration as is, but will try PodmanOption in later iterations.

I see Nick likes the ContainerRuntime interface and wants to keep it. I think that makes a lot of sense.

I'm saying, we don't need to keep the user-facing container_runtime configuration option. Either way, yes we should definitely remove PodmanOptions in the next PR if it's not done in this one.

I see what you mean here. That's possible and makes sense.

I think there may be some confusion since there's multiple ContainerRuntimes in the code.

looking at this snippet ....

var cruntime container.ContainerRuntime // ContainerRuntime is an interface, defined in agent/internal/container_runtime.go switch a.opts.ContainerRuntime { // ContainerRuntime is a string, populated by user configuration

The interface is good to keep. The configuration option one is up for debate.

NicholasBlaskey · 2024-06-24T15:06:05Z

.circleci/real_config.yml

@@ -2840,7 +2840,7 @@ jobs:
                  PROJECT=$(terraform -chdir=tools/slurm/terraform output --raw project)
                  gcloud compute scp agent/build/determined-agent "$INSTANCE_NAME":~ --zone $ZONE
                  gcloud compute ssh --zone "$ZONE" "$INSTANCE_NAME" --project "$PROJECT" -- \
-                  srun determined-agent --master-host=<<parameters.master-host>> --master-port=<<parameters.master-port>> --resource-pool=default --container-runtime=<<parameters.container-run-type>>
+                  srun determined-agent --master-host=<<parameters.master-host>> --master-port=<<parameters.master-port>> --resource-pool=default


We can delete the agent-use parameter and all steps that use it since it should be unused

So I would completely delete this when block and the when block below this

Yes I understand your comment here and will update it.
We would need to remove the test-e2e-slurm case as well, right?

jobs: test-e2e-slurm: parameters: - when: condition: equal: ["-A", <<parameters.agent-use>>]

NicholasBlaskey · 2024-06-24T15:07:20Z

agent/cmd/determined-agent/init.go

@@ -151,7 +151,4 @@ func registerAgentConfig() {
 		"Max attempts agent has to reconnect")
 	registerInt(flags, name("agent-reconnect-backoff"), defaults.AgentReconnectBackoff,
 		"Time between agent reconnect attempts")
-
-	registerString(flags, name("container-runtime"), defaults.ContainerRuntime,


I think we should leave this flag in but only allow docker as the container-runtime

If for some reason someone has determined-agent --container-runtime=docker I don't think we should break that use case

NicholasBlaskey · 2024-06-24T15:08:45Z

agent/internal/options/options.go

@@ -81,10 +80,6 @@ type Options struct {
 	// master config.
 	AgentReconnectBackoff int `json:"agent_reconnect_backoff"`

-	ContainerRuntime   string             `json:"container_runtime"`


Same comment about leaving this option in but checking that is it always docker

So if someone has container_runtime: docker in their agent config I think we shouldn't break them

According to our previous iterations in the PR, I thought we agreed on having to remove ContainerRuntime config all-together. I see what you are saying here and agree, whereas earlier I assumed because this config was never documented, it was not actually used by any customers.

@kkunapuli - Do you agree to the same?

@ShreyaLnuHpe Yes, that was my understanding from talking with Bradley (that container_runtime configuration option was never documented).

As you've dug into the task more thoroughly, it seems you've uncovered an additional use case that was documented: using agents with slurm. I'm ok with leaving the configuration option in, with non-Docker values returning an error. I though that extra config options would be ignored, not result in errors.

@NicholasBlaskey @kkunapuli
Shall we now have a documentation of container_runtime config, and mentioning that it only intakes docker value in case of agentRM?

No, I think don't document it. We won't block someone from having container_runtime: docker but we don't want anyone adding it either.

NicholasBlaskey · 2024-06-24T15:10:54Z

tools/slurm/scripts/slurmcluster.sh

@@ -76,7 +76,7 @@ while [[ $# -gt 0 ]]; do
            echo '  -A '


I think the -A flag is the one we want to remove

Do we need any code changes to the scripts here to no longer support -A beyond changing the usage description?

NicholasBlaskey · 2024-06-24T15:11:48Z

tools/slurm/scripts/slurmcluster.sh

@@ -76,7 +76,7 @@ while [[ $# -gt 0 ]]; do
            echo '  -A '
            echo "           Description: Invokes a slurmcluster that uses agents instead of the launcher."
            echo "           Example: $0 -A"
-            echo '  -c {enroot|podman|singularity}'
+            echo '  -c {docker}'


I think enroot|podman|singularity are the correct options, since we don't support docker with launcher and we still will support the other runtimes for launcher

NicholasBlaskey · 2024-06-24T15:15:24Z

agent/internal/agent.go

@@ -120,47 +118,18 @@ func (a *Agent) run(ctx context.Context) error {
 		return fmt.Errorf("failed to detect devices: %v", devices)
 	}

-	a.log.Tracef("setting up %s runtime", a.opts.ContainerRuntime)


do we have a release note?

I will add a release note, thanks

NicholasBlaskey · 2024-06-24T15:19:36Z

docs/setup-cluster/slurm/hpc-with-agent.rst

-file. Below is a minimal example using a resource pool named for the user (``$USER``) and
-``singularity`` as the container runtime platform. If configured using variables such as ``$HOME``,
-a single ``agent.yaml`` could be shared by all users.
+file. Below is a minimal example using a resource pool named for the user (``$USER``). ``docker``


I don't think this will work

I think in slurm environments it is very likely that the launched jobs won't have access to docker so the agent method won't be supported

I think this whole hpc-with-agent workload is no longer going to be supported, so I think we can remove all these docs

would this entail removing the page?: https://docs.determined.ai/latest/setup-cluster/slurm/hpc-with-agent.html

if so, please create a redirect. there is a guide here on using the docs redirect tool https://hpe-aiatscale.atlassian.net/wiki/spaces/DOC/pages/1338310660/How+to+Use+the+Docs+Redirect+Tool

let me know how i can help

@tara-det-ai - While using redirect to remove a .rst file, what do we update after the colon ':' in this file?
"setup-cluster/slurm/hpc-with-agent.rst" : "_index.html",

[Edited] The above code gives the below error-

broken redirect detected: Link(src='setup-cluster/deploy-cluster/slurm/hpc-with-agent', dst='setup-cluster/slurm/hpc-with-agent')
check failed; the following previously-published urls seem to have been dropped and should be assigned redirects:
setup-cluster/slurm/hpc-with-agent

Redirect removing file doc
It mentions about if we have moving a file from one directory location to other, but doesn't mention anything about when we remove the file completely?

Remove the page with git rm

run git rm setup-cluster/slurm/hpc-with-agent.rst

Remove the toctree element since the page is gone

Edit the docs/setup-cluster/slurm/_index.rst page to remove the toctree element hpc-with-agent

Create a new redirect by editing an existing redirect

modify the docs/.redirects/redirects.json file, replacing line 16
"setup-cluster/deploy-cluster/slurm/hpc-with-agent": "../../slurm/hpc-with-agent.html",
with
"setup-cluster/deploy-cluster/slurm/hpc-with-agent": "../../slurm/_index.html",

Should build with no errors after that.

@tara-det-ai -

Error occurred while building after making the above suggested change-

% make -C docs check
git ls-files -z '*.rst' | xargs -0 rstfmt -w 100 --check
python3 redirects.py check
check failed; the following previously-published urls seem to have been dropped and should be assigned redirects:
setup-cluster/slurm/hpc-with-agent
make: *** [check] Error 1

The above was solved when I added another line (reference to setup-cluster/slurm/hpc-with-agent in docs/.redirects/redirects.json file.
"setup-cluster/slurm/hpc-with-agent": "../slurm/_index.html",

Let me know if this looks right.

NicholasBlaskey · 2024-06-24T15:37:30Z

tools/slurm/README.md

@@ -5,7 +5,7 @@
 1. Install Terraform following [these instructions](https://developer.hashicorp.com/terraform/downloads).
 2. Download the [GCP CLI](https://cloud.google.com/sdk/docs/install-sdk) and run `gcloud auth application-default login` to get credentials.
 3. Run `make slurmcluster` from the root of the repo and wait (up to 10 minutes) for it to start.
-   - To specify which container runtime environment to use, pass in `FLAGS="-c {container_run_type}"` to `make slurmcluster`. Choose from either `singularity` (default), `podman`, or `enroot`.
+   - To specify which container runtime environment to use, pass in `FLAGS="-c {container_run_type}"` to `make slurmcluster`. Choose from either `singularity` (default), `podman`, or `enroot`. If utilizing Slurmcluster with Determined Agents, `docker` container runtime environment is the sole available option.


I don't think we support Slurmcluster with docker

For clarity, the intent is to no longer give customers the option to use slurm with determined agents.

At least, that's what I understood from slack thread: https://hpe-aiatscale.slack.com/archives/C06GMG83ZE0/p1718224380888699

Corrected it. Let me know mentioning it straight out makes sense or not.

I think we want to remove the entire last sentence, If utilizing Slurmcluster with Determined Agents, docker container runtime environment is the sole available option.

We don't want customers to use Slurmcluster with Determiend Agents. At least, that's my understanding.

NicholasBlaskey · 2024-06-24T15:39:10Z

agent/internal/options/options.go

@@ -81,10 +80,6 @@ type Options struct {
 	// master config.
 	AgentReconnectBackoff int `json:"agent_reconnect_backoff"`

-	ContainerRuntime   string             `json:"container_runtime"`


what about image_root?

https://docs.determined.ai/latest/reference/deploy/agent-config-reference.html#image-root

docs this have any affect on Docker?

I am not very clear of your question here.

If you mean that removing Singularity/Podman/Apptainer as container_runtime for AgentRM might affect with setting those image cache in image_root parameter - Looks unlikely as we know we have these options as is for when deploying Slurm though Launcher or any non-agent way shown here

Or if you mean that as AgentRM won't be supporting Singularity/Podman/Apptainer, hence it would have to make corrections in here. I do not think docker will be affected even in this case.

Kindly correct if my understanding is not right

I think there is two image_root config options.

One lives in master config with resource_manager.agent_root while the other is in agent config as image_root

For the agent config option, I think that Docker does not use the image_root option given it was added for other runtimes. So I think now that we have removed the other runtimes I think we can also remove image_root if Docker does not use it.

tara-det-ai

viewed hpc-with-agent

dannysauer

I'm OK with the infra-relevant portion of this PR, pending the stuff Nick suggested.

NicholasBlaskey

looks pretty good

Does the slurm CI still pass on this branch?

NicholasBlaskey · 2024-07-02T16:43:52Z

agent/internal/agent.go

@@ -121,46 +119,24 @@ func (a *Agent) run(ctx context.Context) error {
 	}

 	a.log.Tracef("setting up %s runtime", a.opts.ContainerRuntime)
+	if a.opts.ContainerRuntime != options.DockerContainerRuntime {
+		a.log.Error("%w creation is not supported, please update agent container runtime config to use Docker instead.",


do we mean container runtime instead of creation?

NicholasBlaskey · 2024-07-02T16:45:12Z

agent/internal/agent.go

+			a.opts.ContainerRuntime)
+		return fmt.Errorf("%s creation not available", a.opts.ContainerRuntime)
+	}
+
 	var cruntime container.ContainerRuntime


I think we don't need to declare cruntime ahead of time

var cruntime container.ContainerRuntime

and can just do this on line 139

cruntime := docker.NewClient(dcl)

NicholasBlaskey · 2024-07-02T16:50:10Z

docs/release-notes/deprecate-containerRuntime.rst

+
+-  Singularity, Podman, and Apptainer Container runtimes for AgentRM: Launching a Singluarity/Podman/Apptainer container runtimes for Agent is no longer supported. Docker is the only option that is supported.
+
+-  Determined Agent on Slurm/PBS: Slurmcluster with Determined Agents is not supported any more. For detailed instructions on existing ways to deploy, visit :ref:deploy-on-slurm-pbs. This change was announced in version 0.33.0.


I think we can remove the second note since slurmcluster is only used internally

NicholasBlaskey · 2024-07-02T16:52:52Z

docs/release-notes/deprecate-containerRuntime.rst

+
+**Deprecations**
+
+-  Singularity, Podman, and Apptainer Container runtimes for AgentRM: Launching a Singluarity/Podman/Apptainer container runtimes for Agent is no longer supported. Docker is the only option that is supported.


To avoid repeating here I think I would start the note with

AgentRM: ...

I would add a note that reminds users
"If you want to use singularity, podman, or apptainer the Determined master enterprise edition still supports this "

I think I would also clarify that
"This change only affects when container_runtime is set to podman, using a podman emulation layer is unchanged"

Thanks for suggesting! Its far better approach!

NicholasBlaskey · 2024-07-02T16:54:37Z

tools/slurm/scripts/slurmcluster.sh

@@ -214,16 +206,6 @@ if [[ $OPT_CONTAINER_RUN_TYPE == "enroot" ]]; then
    fi
 fi

-TEMPYAML=$TEMPDIR/slurmcluster.yaml


I think on line 143 there is another reference to $DETERMINED_AGENT

I didn't remove the line in 143 because the description of those lines meant grabbing token for launcher service, which I think is still valid.
Though when we do not set DETERMINED_AGENT=1 anymore, hence those set of lines will never be executed.

I am inclined towards removing the line 143 if [[ -z $DETERMINED_AGENT ]]; then if condition and bring the lines within line 144 - 150 outside the if parameter.

What do you think?

I think since we don't set DETERMINED_AGENT

the if [[ -z $DETERMINED_AGENT ]]; then checks that DETERMINED_AGENT isn't set so the if will always be true. So I think your suggestion of removing the if statement and keeping the rest of the code is right

NicholasBlaskey · 2024-07-02T16:56:25Z

tools/slurm/scripts/slurmcluster.sh

@@ -214,16 +206,6 @@ if [[ $OPT_CONTAINER_RUN_TYPE == "enroot" ]]; then
    fi
 fi

-TEMPYAML=$TEMPDIR/slurmcluster.yaml
-envsubst <$PARENT_PATH/slurmcluster.yaml >$TEMPYAML
-if [[ -n $DETERMINED_AGENT ]]; then


I'm curious here what deploys the agents? As in is there another step which launches the agents? I'm not super familiar with this code

I am myself not very familiar with this shell script, but according to these lines I understand that - If $DETERMINED_AGENT is set, this command deletes lines between resource_manager and resource_manager_end in $TEMPYAML.
But we do not need to generate a devcluster file deployed with agent anymore.

ah ok, looks like agents are deployed manually so we don't need to do anything else

NicholasBlaskey · 2024-07-02T16:58:32Z

agent/internal/options/options.go

@@ -81,10 +80,6 @@ type Options struct {
 	// master config.
 	AgentReconnectBackoff int `json:"agent_reconnect_backoff"`

-	ContainerRuntime   string             `json:"container_runtime"`


I think there is two image_root config options.

One lives in master config with resource_manager.agent_root while the other is in agent config as image_root

For the agent config option, I think that Docker does not use the image_root option given it was added for other runtimes. So I think now that we have removed the other runtimes I think we can also remove image_root if Docker does not use it.

tara-det-ai · 2024-07-02T18:27:25Z

tools/slurm/README.md

@@ -5,7 +5,7 @@
 1. Install Terraform following [these instructions](https://developer.hashicorp.com/terraform/downloads).
 2. Download the [GCP CLI](https://cloud.google.com/sdk/docs/install-sdk) and run `gcloud auth application-default login` to get credentials.
 3. Run `make slurmcluster` from the root of the repo and wait (up to 10 minutes) for it to start.
-   - To specify which container runtime environment to use, pass in `FLAGS="-c {container_run_type}"` to `make slurmcluster`. Choose from either `singularity` (default), `podman`, or `enroot`.
+   - To specify which container runtime environment to use, pass in `FLAGS="-c {container_run_type}"` to `make slurmcluster`. Choose from either `singularity` (default), `podman`, or `enroot`. If utilizing Slurmcluster with Determined Agents, `docker` container runtime environment is the sole available option.


Suggested change

- To specify which container runtime environment to use, pass in `FLAGS="-c {container_run_type}"` to `make slurmcluster`. Choose from either `singularity` (default), `podman`, or `enroot`. If utilizing Slurmcluster with Determined Agents, `docker` container runtime environment is the sole available option.

- To specify which container runtime environment to use, pass ``FLAGS="-c {container_run_type}"`` to make slurmcluster. You can choose from ``singularity`` (default), ``podman``, or ``enroot``. If you are using Slurmcluster with Determined Agents, docker is the only available container runtime environment.

sandrews-hpe · 2024-07-09T21:52:11Z

docs/release-notes/deprecate-containerRuntime.rst

+-  AgentRM: Launching a Singluarity/Podman/Apptainer container runtimes for Agent is no longer
+   supported. Docker is the only option that is supported. This change only affects when
+   container_runtime is set to podman, using a podman emulation layer is unchanged. If you want to
+   use singularity, podman, or apptainer the Determined master enterprise edition still supports it.


Docker is the only option that is supported. This change only affects when
container_runtime is set to podman, using a podman emulation layer is unchanged.

This is confusing to me. Maybe something like this would be clearer: "Docker and Podman using the emulation layer are still supported."

If you want to
use singularity, podman, or apptainer the Determined master enterprise edition still supports it.

Is this statement accurate? Are you able to run AgentRM with singularity, podman, or apptainer in EE? If so, then wouldn't it be the same amount of work to support all containers in both EE and OSS? Why bother deprecating it?

If you want to
use singularity, podman, or apptainer the Determined master enterprise edition still supports it.

This came from a comment from Nick: #9516 (comment)

I'm not exactly sure what he means. I think he was referring to how the other container runtimes (podman/singularity/apptainer) still work with slurm (and slurm is ee only). If I recall correctly, podman also works with kubernetesrm though ...

If it's accurate, I think I'd rather say something like "Kubernetes and Slurm resource managers still support singularity, podman, or apptainer use."

ShreyaLnuHpe · 2024-07-10T22:50:30Z

.circleci/real_config.yml

@@ -5614,39 +5450,6 @@ workflows:
              ld_library_path:
            security:
              initial_user_password: ${INITIAL_USER_PASSWORD}
-      - test-e2e-slurm:
-            name: test-e2e-slurm-agent-singularity-znode


@NicholasBlaskey / @kkunapuli - Please confirm if we no longer be needing this test as well.

My understanding is, that we should not test any agent related singularity znode changes, but I am not sure, so need an opinion.

Ongoing discussion on the failed state of the test- https://hpe-aiatscale.slack.com/archives/C04C9JXB1C2/p1720448458815989

is this testing EE + Slurm + Singularity which remains a supported usecase if I'm reading the PR description correctly?
update this is testing: https://docs.determined.ai/latest/setup-cluster/slurm/hpc-with-agent.html
make sense to treat these the same so remove?

4524: name: [test-e2e-slurm-agent-podman-gcp] 5427: name: [test-e2e-slurm-agent-podman-gcp] 5620: name: test-e2e-slurm-agent-singularity-znode

All the znodes tests should start getting scheduled and running after the backlog is cleared let's check on that in the channel

I agree that EE + Slurm + Singularity is a supported use case, but the name of the test test-e2e-slurm-agent-singularity-znode confuses me a little as we are deprecating all AGENT related singularity/podman/apptainer use cases.

Yes, for this PR, we have removed Agent on Slurm/PBS document and removed the below test runs (you can see above in this PR) as well:

4524: name: [test-e2e-slurm-agent-podman-gcp] 5427: name: [test-e2e-slurm-agent-podman-gcp]

kkunapuli

Great work!

I left a couple minor suggestions. As a reminder, please don't merge until after user acceptance testing.

kkunapuli · 2024-07-11T19:40:46Z

agent/internal/agent.go

+	if a.opts.ContainerRuntime != options.DockerContainerRuntime {
+		a.log.Error(a.opts.ContainerRuntime,
+			" Container Runtime is not supported, please update runtime config to use Docker instead.")
+		return fmt.Errorf("%s Container Runtime not available", a.opts.ContainerRuntime)


nit: I think it's more idiomatic to format the error with the %s at the end, so container runtime not available: %s

I think Go requires errors to be all lowercase?

kkunapuli · 2024-07-11T19:48:05Z

docs/release-notes/deprecate-containerRuntime.rst

+-  AgentRM: Support for Singluarity, Podman, and Apptainer has been deprecated in 0.33.0 and is now
+   removed. Docker is the only container runtime supported by Agent resource manager (AgentRM). It
+   is still possible to use podman with AgentRM by using the podman emulation layer. For detailed
+   instructions, follow steps in the link: `Emulating Docker CLI with Podman


I think to be more consistent with docs style, we should use "visit" instead of "follow steps in the link".

e.g.,

For detailed instructions, visit `Emulating Docker CLI with Podman <https://podman-desktop.io/docs/migrating-from-docker/emulating-docker-cli-with-podman>`

kkunapuli · 2024-07-11T19:53:53Z

tools/slurm/README.md

-Now, you can launch jobs like normal using the Determined CLI. You can check the status of the allocated resources using `det slot list`.
-
-If you encounter an issue with jobs failing due to `ModuleNotFoundError: No module named 'determined'` run `make clean all` to rebuild determined. 
+### Note: We no longer support Slurmcluster with Determined Agents


nit: I think it's more consistent with docs style to say:

Slurmcluster with Determined Agents is no longer supported.

sandrews-hpe · 2024-07-11T20:03:57Z

docs/release-notes/deprecate-containerRuntime.rst

+   Config Reference
+   https://docs.determined.ai/latest/reference/deploy/master-config-reference.html#checkpoint-storage`
+
+In enterprise edition, Slurm resource manager still supports singularity, podman, or apptainer use.


Update capitalization for Singularity, Podman, and Apptainer

I'm not confident, but I think we want either "In the enterprise edition, ..." or "In Enterprise Edition, ..." Maybe more of a question for Tara.

Let's discuss this during our user acceptance call with Caetano.

hamidzr · 2024-07-11T22:26:40Z

docs/.redirects/redirects.json

@@ -13,7 +13,8 @@
    "architecture/system-architecture": "../get-started/architecture/system-architecture.html",
    "architecture/introduction": "../get-started/architecture/introduction.html",
    "setup-cluster/deploy-cluster/slurm/install-on-slurm": "../../slurm/install-on-slurm.html",
-    "setup-cluster/deploy-cluster/slurm/hpc-with-agent": "../../slurm/hpc-with-agent.html",
+    "setup-cluster/deploy-cluster/slurm/hpc-with-agent": "../../slurm/_index.html", 
+    "setup-cluster/slurm/hpc-with-agent": "../slurm/_index.html",


didn't know we have this, nice.

hamidzr · 2024-07-12T16:02:52Z

tools/slurm/scripts/slurmcluster.sh

-    # the master waits for agents to connect and provide resources.
-    sed -i -e '/resource_manager/,/resource_manager_end/d' $TEMPYAML
-fi
-echo "Generated devcluster file: $TEMPYAML"


https://app.circleci.com/pipelines/github/determined-ai/determined/58038/workflows/b30764bf-8db9-4b01-acba-ad1d92c0b0e5/jobs/2757463?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

is this still getting generated? the failure might be relate to this

NicholasBlaskey

A lot of people have reviewed this since I last looked, so I don't feel the need to review it. Let me know if you would like me to look.

tara-det-ai · 2024-07-16T18:07:27Z

docs/release-notes/deprecate-containerRuntime.rst

+   https://docs.determined.ai/latest/reference/deploy/master-config-reference.html#checkpoint-storage`
+
+In enterprise edition, Slurm resource manager still supports Singularity, Podman, or Apptainer use.
+For detailed instructions, visit :ref:deploy-on-slurm-pbs.


:orphan:

Deprecations

AgentRM: As of version 0.33.0, support for Singularity, Podman, and Apptainer has been deprecated and is now officially removed. Docker is the only container runtime supported by Agent resource manager (AgentRM). However, you can still use Podman with AgentRM by utilizing the Podman emulation layer. For instructions, visit the Podman Desktop documentation and search for "Emulating Docker CLI with Podman". Additionally, you may need to configure checkpoint_storage in your experiment configuration or :ref:master configuration <master-config-reference>.

In the enterprise edition, the Slurm Resource Manager continues to support Singularity, Podman, and Apptainer.

the only restructured text anchor that i could find for slurm/pbs is this one

.. _install-on-slurm:

so maybe you meant to use the following:

:ref:install-on-slurm

in any case, this one does not exist and it would need proper formatting if it did exist: :ref:deploy-on-slurm-pbs.

@ShreyaLnuHpe PR 9662 should fix these

ShreyaLnuHpe requested a review from a team as a code owner June 13, 2024 18:03

ShreyaLnuHpe requested a review from hamidzr June 13, 2024 18:03

cla-bot bot added the cla-signed label Jun 13, 2024

ShreyaLnuHpe requested a review from kkunapuli June 13, 2024 18:03

ShreyaLnuHpe commented Jun 13, 2024

View reviewed changes

ShreyaLnuHpe changed the title ~~chore: deprecating container_runtime~~ chore: draft deprecating container_runtime Jun 13, 2024

ShreyaLnuHpe requested a review from NicholasBlaskey June 13, 2024 19:26

NicholasBlaskey reviewed Jun 13, 2024

View reviewed changes

ShreyaLnuHpe force-pushed the shreya/deprecateContainerRuntime branch from dd9734f to cd0e626 Compare June 14, 2024 19:26

ShreyaLnuHpe commented Jun 14, 2024

View reviewed changes

ShreyaLnuHpe changed the title ~~chore: draft deprecating container_runtime~~ chore: deprecating container_runtime Jun 14, 2024

kkunapuli reviewed Jun 14, 2024

View reviewed changes

ShreyaLnuHpe requested review from NicholasBlaskey and kkunapuli June 18, 2024 17:37

ShreyaLnuHpe requested a review from a team as a code owner June 18, 2024 19:13

ShreyaLnuHpe requested a review from dannysauer June 18, 2024 19:13

determined-ci added the documentation Improvements or additions to documentation label Jun 18, 2024

determined-ci requested a review from a team June 18, 2024 19:13

ShreyaLnuHpe changed the title ~~chore: deprecating container_runtime~~ chore: deprecating container_runtime config, AgentRM supporting singularity,podman, and apptainer Jun 18, 2024

NicholasBlaskey reviewed Jun 24, 2024

View reviewed changes

tara-det-ai reviewed Jun 24, 2024

View reviewed changes

dannysauer approved these changes Jun 27, 2024

View reviewed changes

determined-ci requested a review from a team July 1, 2024 18:05

ShreyaLnuHpe requested a review from NicholasBlaskey July 2, 2024 16:42

NicholasBlaskey reviewed Jul 2, 2024

View reviewed changes

ShreyaLnuHpe requested a review from tara-det-ai July 2, 2024 17:44

tara-det-ai reviewed Jul 2, 2024

View reviewed changes

ShreyaLnuHpe added 8 commits July 5, 2024 12:18

removed slurm-agent tests

0f92a48

remove podman

768c735

based on comments

039c695

corrections in doc

e2da94b

nit

7b65ba7

as per comments

c79c8d3

fmt checks

0e07272

nit

13516a6

ShreyaLnuHpe force-pushed the shreya/deprecateContainerRuntime branch from 8234d62 to 13516a6 Compare July 5, 2024 20:28

remove error

a4ec1e8

sandrews-hpe reviewed Jul 9, 2024

View reviewed changes

correction after testing

17abb7c

ShreyaLnuHpe commented Jul 10, 2024

View reviewed changes

ShreyaLnuHpe added 2 commits July 11, 2024 10:22

add info in release note

11b5446

nit fmt

7d16c32

ShreyaLnuHpe requested review from sandrews-hpe and NicholasBlaskey July 11, 2024 18:39

kkunapuli approved these changes Jul 11, 2024

View reviewed changes

sandrews-hpe reviewed Jul 11, 2024

View reviewed changes

minor wording corrections

5febf09

ShreyaLnuHpe changed the title ~~chore: deprecating container_runtime config, AgentRM supporting singularity,podman, and apptainer~~ chore: deprecating container_runtime config, agentrm supporting singularity,podman, and apptainer Jul 11, 2024

hamidzr reviewed Jul 12, 2024

View reviewed changes

ShreyaLnuHpe added 2 commits July 12, 2024 11:16

correction of circleci failure

e5748fc

nit: remove space

b6c4e44

ShreyaLnuHpe requested a review from hamidzr July 12, 2024 19:30

NicholasBlaskey reviewed Jul 15, 2024

View reviewed changes

ShreyaLnuHpe merged commit c3e0a41 into main Jul 15, 2024
118 of 121 checks passed

ShreyaLnuHpe deleted the shreya/deprecateContainerRuntime branch July 15, 2024 16:15

ShreyaLnuHpe mentioned this pull request Jul 16, 2024

chore: remove singularity agent znode nightly test #9659

Merged

5 tasks

tara-det-ai reviewed Jul 16, 2024

View reviewed changes

		@@ -152,6 +152,6 @@ func registerAgentConfig() {
		registerInt(flags, name("agent-reconnect-backoff"), defaults.AgentReconnectBackoff,


		- Singularity, Podman, and Apptainer Container runtimes for AgentRM: Launching a Singluarity/Podman/Apptainer container runtimes for Agent is no longer supported. Docker is the only option that is supported.

		- Determined Agent on Slurm/PBS: Slurmcluster with Determined Agents is not supported any more. For detailed instructions on existing ways to deploy, visit :ref:deploy-on-slurm-pbs. This change was announced in version 0.33.0.


		Deprecations

		- Singularity, Podman, and Apptainer Container runtimes for AgentRM: Launching a Singluarity/Podman/Apptainer container runtimes for Agent is no longer supported. Docker is the only option that is supported.

	- To specify which container runtime environment to use, pass in `FLAGS="-c {container_run_type}"` to `make slurmcluster`. Choose from either `singularity` (default), `podman`, or `enroot`. If utilizing Slurmcluster with Determined Agents, `docker` container runtime environment is the sole available option.
	- To specify which container runtime environment to use, pass ``FLAGS="-c {container_run_type}"`` to make slurmcluster. You can choose from ``singularity`` (default), ``podman``, or ``enroot``. If you are using Slurmcluster with Determined Agents, docker is the only available container runtime environment.

chore: deprecating container_runtime config, agentrm supporting singularity,podman, and apptainer #9516

chore: deprecating container_runtime config, agentrm supporting singularity,podman, and apptainer #9516

Conversation

ShreyaLnuHpe commented Jun 13, 2024 • edited Loading

Ticket

Description

Test Plan

Checklist

netlify bot commented Jun 13, 2024 • edited Loading

✅ Deploy Preview for determined-ui canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 14, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkunapuli Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyaLnuHpe Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyaLnuHpe Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyaLnuHpe Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

tara-det-ai Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tara-det-ai left a comment

Choose a reason for hiding this comment

dannysauer left a comment

Choose a reason for hiding this comment

NicholasBlaskey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyaLnuHpe commented Jun 13, 2024 •

edited

Loading

netlify bot commented Jun 13, 2024 •

edited

Loading

codecov bot commented Jun 14, 2024 •

edited

Loading

kkunapuli Jun 17, 2024 •

edited

Loading

ShreyaLnuHpe Jun 24, 2024 •

edited

Loading

ShreyaLnuHpe Jun 24, 2024 •

edited

Loading

ShreyaLnuHpe Jul 1, 2024 •

edited

Loading

tara-det-ai Jul 2, 2024 •

edited

Loading

hamidzr Jul 11, 2024 •

edited

Loading