Skip to content

Commit

Permalink
Add support for OCI containers
Browse files Browse the repository at this point in the history
Update the configure-eda.sh script to install rootless docker for compute nodes.

Add playbook and script to configure users to run rootless docker.

Resolves #29
  • Loading branch information
cartalla committed Dec 18, 2024
1 parent c8f5053 commit 49124b6
Show file tree
Hide file tree
Showing 18 changed files with 445 additions and 1 deletion.
109 changes: 109 additions & 0 deletions docs/containers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Containers

Slurm supports [running jobs in unprivileged OCI containers](https://slurm.schedmd.com/containers.html).
OCI is the [Open Container Initiative](https://opencontainers.org/), an open governance structure with the purpose of creating open industry standards around container formats and runtimes.

I'm going to document how to add OCI support to your EDA Slurm cluster.
Note that most EDA tools are not containerized and that some won't run in containers and that some may run in a container, but not correctly.
I recommend following the guidance of your EDA vendor and consult with them.

I've seen a couple of main motivations for using containers for EDA tools.
The first is because orchestration tools like Kubernetes and AWS Batch require jobs to run in containers.
The other is to have more flexibility managing the run time environment of the tools.
Since the EDA tools themselves aren't containerized, the container is usually used to manage file system mounts and packages that are used by the tools.
If new packages are required by a new tool, then it is easy to update and distribute a new version of the container.

## Compute node configuration

The compute node must be configured to use an unprivileged container runtime.
We'll show how to install and configure rootless Docker.

The following directions have been automated in the [creation of a custom EDA compute node AMI](custom-amis.md).

First, [install the latest Docker from the Docker yum repo](https://docs.docker.com/engine/install/rhel/).

Next, [configure Docker to run rootless](https://docs.docker.com/engine/security/rootless/).

Configure subuid and subgid.

Each user that will run Docker must have an entry in `/etc/subuid` and `/etc/subgid`.

## Per user configuration

You must configure docker to use a non-NFS storage location for storing images.

`~/.config/docker/daemon.json`:

```
{
"data-root": "/var/tmp/${USER}/containers/storage"
}
```

## Create OCI Bundle

Each container requires an [OCI bundle](https://slurm.schedmd.com/containers.html#bundle).

The bundle directories can be stored on NFS and shared between users.
For example, you could create an oci-bundles directory on your shared file system.

This shows how to create an ubuntu bundle.
You can do this as root with the docker service running, but it would be better to run
it using rootless Docker.

```
export OCI_BUNDLES_DIR=~/oci-bundles
export IMAGE_NAME=ubuntu
export BUNDLE_NAME=ubuntu
mkdir -p $OCI_BUNDLES_DIR
cd $OCI_BUNDLES_DIR
mkdir -p $BUNDLE_NAME
cd $BUNDLE_NAME
docker pull $IMAGE_NAME
docker export $(docker create $IMAGE_NAME) > $BUNDLE_NAME.tar
mkdir rootfs
tar -C rootfs -xf $IMAGE_NAME.tar
runc spec --rootless
runc run containerid
```

The same process works for Rocky Linux 8.

```
export OCI_BUNDLES_DIR=~/oci-bundles
export IMAGE_NAME=rockylinux:8
export BUNDLE_NAME=rockylinux8
mkdir -p $OCI_BUNDLES_DIR
cd $OCI_BUNDLES_DIR
mkdir -p $BUNDLE_NAME
cd $BUNDLE_NAME
docker pull $IMAGE_NAME
docker export $(docker create $IMAGE_NAME) > $BUNDLE_NAME.tar
mkdir rootfs
tar -C rootfs -xf $BUNDLE_NAME.tar
runc spec --rootless
runc run containerid2
```

## Test the bundle locally

```
export OCI_BUNDLES_DIR=~/oci-bundles
export BUNDLE_NAME=rockylinux8
cd $OCI_BUNDLES_DIR/$BUNDLE_NAME
runc spec --rootless
runc run containerid2
```

## Run a bundle on Slurm

```
export OCI_BUNDLES_DIR=~/oci-bundles
export BUNDLE_NAME=rockylinux8
srun -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --pty hostname
srun -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --pty bash
sbatch -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --wrap hostname
```
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ nav:
- 'job_preemption.md'
- 'rest_api.md'
- 'onprem.md'
- 'containers.md'
# - 'federation.md'
- 'delete-cluster.md'
# - 'implementation.md'
Expand Down
3 changes: 3 additions & 0 deletions source/cdk/cdk_slurm_stack.py
Original file line number Diff line number Diff line change
Expand Up @@ -1199,11 +1199,13 @@ def create_parallel_cluster_assets(self):
# Additions or deletions to the list should be reflected in config_scripts in on_head_node_start.sh.
files_to_upload = [
'config/bin/configure-eda.sh',
'config/bin/configure-rootless-docker.sh',
'config/bin/create_or_update_users_groups_json.sh',
'config/bin/create_users_groups_json.py',
'config/bin/create_users_groups_json_configure.sh',
'config/bin/create_users_groups_json_deconfigure.sh',
'config/bin/create_users_groups.py',
'config/bin/install-rootless-docker.sh',
'config/bin/on_head_node_start.sh',
'config/bin/on_head_node_configured.sh',
'config/bin/on_head_node_updated.sh',
Expand Down Expand Up @@ -1477,6 +1479,7 @@ def create_parallel_cluster_lambdas(self):
'ConfigureEdaScriptS3Url': self.custom_action_s3_urls['config/bin/configure-eda.sh'],
'ErrorSnsTopicArn': self.config.get('ErrorSnsTopicArn', ''),
'ImageBuilderSecurityGroupId': self.imagebuilder_sg.security_group_id,
'InstallDockerScriptS3Url': self.custom_action_s3_urls['config/bin/install-rootless-docker.sh'],
'ParallelClusterVersion': self.config['slurm']['ParallelClusterConfig']['Version'],
'Region': self.cluster_region,
'SubnetId': self.config['SubnetId'],
Expand Down
17 changes: 17 additions & 0 deletions source/resources/lambdas/CreateBuildFiles/CreateBuildFiles.py
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,23 @@ def lambda_handler(event, context):
Body = build_file_content
)

# Image with rootless Docker installed
template_vars['ImageName'] = f"parallelcluster-{parallelcluster_version_name}-docker-{distribution}-{version}-{architecture}".replace('_', '-')
template_vars['ComponentS3Url'] = environ['InstallDockerScriptS3Url']
build_file_s3_key = f"{assets_base_key}/config/build-files/{template_vars['ImageName']}.yml"
if requestType == 'Delete':
response = s3_client.delete_object(
Bucket = assets_bucket,
Key = build_file_s3_key
)
else:
build_file_content = build_file_template.render(**template_vars)
s3_client.put_object(
Bucket = assets_bucket,
Key = build_file_s3_key,
Body = build_file_content
)

# Image with EDA packages
template_vars['ImageName'] = f"parallelcluster-{parallelcluster_version_name}-eda-{distribution}-{version}-{architecture}".replace('_', '-')
template_vars['ComponentS3Url'] = environ['ConfigureEdaScriptS3Url']
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,4 +88,6 @@ ansible-playbook $PLAYBOOKS_PATH/eda_tools.yml \
-i inventories/local.yml \
-e @$ANSIBLE_PATH/ansible_head_node_vars.yml

popd
ansible-playbook $PLAYBOOKS_PATH/install-rootless-docker.yml \
-i inventories/local.yml \
-e @$ANSIBLE_PATH/ansible_head_node_vars.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/bin/bash -ex
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0

# Configure rootless docker for user.
# The slurm config directory must exist

script=$0
script_name=$(basename $script)

# Jinja2 template variables
assets_bucket={{assets_bucket}}
assets_base_key={{assets_base_key}}
export AWS_DEFAULT_REGION={{Region}}
ClusterName={{ClusterName}}
ErrorSnsTopicArn={{ErrorSnsTopicArn}}
playbooks_s3_url={{playbooks_s3_url}}

# Notify user of errors
function on_exit {
rc=$?
set +e
if [[ $rc -ne 0 ]] && [[ ":$ErrorSnsTopicArn" != ":" ]]; then
tmpfile=$(mktemp)
echo "See log files for more info:
/var/lib/amazon/toe/TOE_*
grep PCImageBuilderEDA /var/log/messages | less" > $tmpfile
aws --region $AWS_DEFAULT_REGION sns publish --topic-arn $ErrorSnsTopicArn --subject "${ClusterName} configure-rootless-docker.sh failed" --message file://$tmpfile
rm $tmpfile
fi
}
trap on_exit EXIT

# Redirect all IO to /var/log/messages and then echo to stderr
exec 1> >(logger -s -t configure-rootless-docker) 2>&1

# Install ansible
if ! yum list installed ansible &> /dev/null; then
yum install -y ansible || amazon-linux-extras install -y ansible2
fi

external_login_node_config_dir=/opt/slurm/${ClusterName}/config
if [ -e $external_login_node_config_dir ]; then
config_dir=$external_login_node_config_dir
else
config_dir=/opt/slurm/config
fi
config_bin_dir=$config_dir/bin
ANSIBLE_PATH=$config_dir/ansible
PLAYBOOKS_PATH=$ANSIBLE_PATH/playbooks
PLAYBOOKS_ZIP_PATH=$ANSIBLE_PATH/playbooks.zip

if ! [ -e $external_login_node_config_dir ]; then
mkdir -p $config_bin_dir

ansible_head_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_head_node_vars.yml"
ansible_compute_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_compute_node_vars.yml"
ansible_external_login_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_external_login_node_vars.yml"

# Download ansible playbooks
aws s3 cp $playbooks_s3_url ${PLAYBOOKS_ZIP_PATH}.new
if ! [ -e $PLAYBOOKS_ZIP_PATH ] || ! diff -q $PLAYBOOKS_ZIP_PATH ${PLAYBOOKS_ZIP_PATH}.new; then
mv $PLAYBOOKS_ZIP_PATH.new $PLAYBOOKS_ZIP_PATH
rm -rf $PLAYBOOKS_PATH
mkdir -p $PLAYBOOKS_PATH
pushd $PLAYBOOKS_PATH
yum -y install unzip
unzip $PLAYBOOKS_ZIP_PATH
chmod -R 0700 $ANSIBLE_PATH
popd
fi

aws s3 cp $ansible_head_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_head_node_vars.yml

aws s3 cp $ansible_compute_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_compute_node_vars.yml

aws s3 cp $ansible_external_login_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_external_login_node_vars.yml
fi

pushd $PLAYBOOKS_PATH

ansible-playbook $PLAYBOOKS_PATH/configure-rootless-docker.yml \
-i inventories/local.yml \
-e @$ANSIBLE_PATH/ansible_external_login_node_vars.yml

popd
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/bin/bash -ex
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0

# This script calls an ansible playbook that installs rootless docker on a compute node.
# It has 2 different use cases:
# * To build ParallelCluster AMIs
# * To install docker on VDIs or other login nodes using a ParallellCluster.
# The location of the config directory is different for those 2 use cases.
# For an AMI build, the config directory and scripts will not exist and must be downloaded from S3.
# For a login node, the playbooks and scripts will already exist.

script=$0
script_name=$(basename $script)

# Jinja2 template variables
assets_bucket={{assets_bucket}}
assets_base_key={{assets_base_key}}
export AWS_DEFAULT_REGION={{Region}}
ClusterName={{ClusterName}}
ErrorSnsTopicArn={{ErrorSnsTopicArn}}
playbooks_s3_url={{playbooks_s3_url}}

# Notify user of errors
function on_exit {
rc=$?
set +e
if [[ $rc -ne 0 ]] && [[ ":$ErrorSnsTopicArn" != ":" ]]; then
tmpfile=$(mktemp)
echo "See log files for more info:
/var/lib/amazon/toe/TOE_*
grep PCImageBuilderEDA /var/log/messages | less" > $tmpfile
aws --region $AWS_DEFAULT_REGION sns publish --topic-arn $ErrorSnsTopicArn --subject "${ClusterName} install-rootless-docker.sh failed" --message file://$tmpfile
rm $tmpfile
fi
}
trap on_exit EXIT

# Redirect all IO to /var/log/messages and then echo to stderr
exec 1> >(logger -s -t install-rootless-docker) 2>&1

# Install ansible
if ! yum list installed ansible &> /dev/null; then
yum install -y ansible || amazon-linux-extras install -y ansible2
fi

external_login_node_config_dir=/opt/slurm/${ClusterName}/config
if [ -e $external_login_node_config_dir ]; then
config_dir=$external_login_node_config_dir
else
config_dir=/opt/slurm/config
fi
config_bin_dir=$config_dir/bin
ANSIBLE_PATH=$config_dir/ansible
PLAYBOOKS_PATH=$ANSIBLE_PATH/playbooks
PLAYBOOKS_ZIP_PATH=$ANSIBLE_PATH/playbooks.zip

if ! [ -e $external_login_node_config_dir ]; then
mkdir -p $config_bin_dir

ansible_head_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_head_node_vars.yml"
ansible_compute_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_compute_node_vars.yml"
ansible_external_login_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_external_login_node_vars.yml"

# Download ansible playbooks
aws s3 cp $playbooks_s3_url ${PLAYBOOKS_ZIP_PATH}.new
if ! [ -e $PLAYBOOKS_ZIP_PATH ] || ! diff -q $PLAYBOOKS_ZIP_PATH ${PLAYBOOKS_ZIP_PATH}.new; then
mv $PLAYBOOKS_ZIP_PATH.new $PLAYBOOKS_ZIP_PATH
rm -rf $PLAYBOOKS_PATH
mkdir -p $PLAYBOOKS_PATH
pushd $PLAYBOOKS_PATH
yum -y install unzip
unzip $PLAYBOOKS_ZIP_PATH
chmod -R 0700 $ANSIBLE_PATH
popd
fi

aws s3 cp $ansible_head_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_head_node_vars.yml

aws s3 cp $ansible_compute_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_compute_node_vars.yml

aws s3 cp $ansible_external_login_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_external_login_node_vars.yml
fi

pushd $PLAYBOOKS_PATH

ansible-playbook $PLAYBOOKS_PATH/install-rootless-docker.yml \
-i inventories/local.yml \
-e @$ANSIBLE_PATH/ansible_compute_node_vars.yml

popd
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,13 @@ users_groups_json_s3_url="s3://$assets_bucket/$assets_base_key/config/users_grou
# Download all of the config scripts
config_scripts=(\
configure-eda.sh \
configure-rootless-docker.sh \
create_or_update_users_groups_json.sh \
create_users_groups_json.py \
create_users_groups_json_configure.sh \
create_users_groups_json_deconfigure.sh \
create_users_groups.py \
install-rootless-docker.sh \
on_head_node_start.sh \
on_head_node_configured.sh \
on_head_node_updated.sh \
Expand Down
1 change: 1 addition & 0 deletions source/resources/playbooks/ParallelClusterComputeNode.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@
- security_updates
- bug_fixes
- ParallelClusterComputeNode
- install-rootless-docker
8 changes: 8 additions & 0 deletions source/resources/playbooks/configure-rootless-docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
- name: Configure rootless docker for user
hosts:
- ExternalLoginNode
become_user: root
become: yes
roles:
- configure-rootless-docker
8 changes: 8 additions & 0 deletions source/resources/playbooks/install-rootless-docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
- name: Install rootless docker for OCI containers
hosts:
- ParallelClusterComputeNode
become_user: root
become: yes
roles:
- install-rootless-docker
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="runc --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeKill="runc --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="runc --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
RunTimeRun="runc --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b
Loading

0 comments on commit 49124b6

Please sign in to comment.