diff --git a/docs/containers.md b/docs/containers.md new file mode 100644 index 00000000..ddb27bc8 --- /dev/null +++ b/docs/containers.md @@ -0,0 +1,109 @@ +# Containers + +Slurm supports [running jobs in unprivileged OCI containers](https://slurm.schedmd.com/containers.html). +OCI is the [Open Container Initiative](https://opencontainers.org/), an open governance structure with the purpose of creating open industry standards around container formats and runtimes. + +I'm going to document how to add OCI support to your EDA Slurm cluster. +Note that most EDA tools are not containerized and that some won't run in containers and that some may run in a container, but not correctly. +I recommend following the guidance of your EDA vendor and consult with them. + +I've seen a couple of main motivations for using containers for EDA tools. +The first is because orchestration tools like Kubernetes and AWS Batch require jobs to run in containers. +The other is to have more flexibility managing the run time environment of the tools. +Since the EDA tools themselves aren't containerized, the container is usually used to manage file system mounts and packages that are used by the tools. +If new packages are required by a new tool, then it is easy to update and distribute a new version of the container. + +## Compute node configuration + +The compute node must be configured to use an unprivileged container runtime. +We'll show how to install and configure rootless Docker. + +The following directions have been automated in the [creation of a custom EDA compute node AMI](custom-amis.md). + +First, [install the latest Docker from the Docker yum repo](https://docs.docker.com/engine/install/rhel/). + +Next, [configure Docker to run rootless](https://docs.docker.com/engine/security/rootless/). + +Configure subuid and subgid. + +Each user that will run Docker must have an entry in `/etc/subuid` and `/etc/subgid`. + +## Per user configuration + +You must configure docker to use a non-NFS storage location for storing images. + +`~/.config/docker/daemon.json`: + +``` +{ + "data-root": "/var/tmp/${USER}/containers/storage" +} +``` + +## Create OCI Bundle + +Each container requires an [OCI bundle](https://slurm.schedmd.com/containers.html#bundle). + +The bundle directories can be stored on NFS and shared between users. +For example, you could create an oci-bundles directory on your shared file system. + +This shows how to create an ubuntu bundle. +You can do this as root with the docker service running, but it would be better to run +it using rootless Docker. + +``` +export OCI_BUNDLES_DIR=~/oci-bundles +export IMAGE_NAME=ubuntu +export BUNDLE_NAME=ubuntu +mkdir -p $OCI_BUNDLES_DIR +cd $OCI_BUNDLES_DIR +mkdir -p $BUNDLE_NAME +cd $BUNDLE_NAME +docker pull $IMAGE_NAME +docker export $(docker create $IMAGE_NAME) > $BUNDLE_NAME.tar +mkdir rootfs +tar -C rootfs -xf $IMAGE_NAME.tar +runc spec --rootless +runc run containerid +``` + +The same process works for Rocky Linux 8. + +``` +export OCI_BUNDLES_DIR=~/oci-bundles +export IMAGE_NAME=rockylinux:8 +export BUNDLE_NAME=rockylinux8 +mkdir -p $OCI_BUNDLES_DIR +cd $OCI_BUNDLES_DIR +mkdir -p $BUNDLE_NAME +cd $BUNDLE_NAME +docker pull $IMAGE_NAME +docker export $(docker create $IMAGE_NAME) > $BUNDLE_NAME.tar +mkdir rootfs +tar -C rootfs -xf $BUNDLE_NAME.tar +runc spec --rootless +runc run containerid2 +``` + +## Test the bundle locally + +``` +export OCI_BUNDLES_DIR=~/oci-bundles +export BUNDLE_NAME=rockylinux8 +cd $OCI_BUNDLES_DIR/$BUNDLE_NAME +runc spec --rootless +runc run containerid2 +``` + +## Run a bundle on Slurm + +``` +export OCI_BUNDLES_DIR=~/oci-bundles +export BUNDLE_NAME=rockylinux8 + +srun -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --pty hostname + +srun -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --pty bash + +sbatch -p interactive --container $OCI_BUNDLES_DIR/$BUNDLE_NAME --wrap hostname +``` diff --git a/mkdocs.yml b/mkdocs.yml index 9ea05a8f..6e58ead8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -15,6 +15,7 @@ nav: - 'job_preemption.md' - 'rest_api.md' - 'onprem.md' + - 'containers.md' # - 'federation.md' - 'delete-cluster.md' # - 'implementation.md' diff --git a/source/cdk/cdk_slurm_stack.py b/source/cdk/cdk_slurm_stack.py index 9037017e..2cdbf056 100644 --- a/source/cdk/cdk_slurm_stack.py +++ b/source/cdk/cdk_slurm_stack.py @@ -1199,11 +1199,13 @@ def create_parallel_cluster_assets(self): # Additions or deletions to the list should be reflected in config_scripts in on_head_node_start.sh. files_to_upload = [ 'config/bin/configure-eda.sh', + 'config/bin/configure-rootless-docker.sh', 'config/bin/create_or_update_users_groups_json.sh', 'config/bin/create_users_groups_json.py', 'config/bin/create_users_groups_json_configure.sh', 'config/bin/create_users_groups_json_deconfigure.sh', 'config/bin/create_users_groups.py', + 'config/bin/install-rootless-docker.sh', 'config/bin/on_head_node_start.sh', 'config/bin/on_head_node_configured.sh', 'config/bin/on_head_node_updated.sh', @@ -1477,6 +1479,7 @@ def create_parallel_cluster_lambdas(self): 'ConfigureEdaScriptS3Url': self.custom_action_s3_urls['config/bin/configure-eda.sh'], 'ErrorSnsTopicArn': self.config.get('ErrorSnsTopicArn', ''), 'ImageBuilderSecurityGroupId': self.imagebuilder_sg.security_group_id, + 'InstallDockerScriptS3Url': self.custom_action_s3_urls['config/bin/install-rootless-docker.sh'], 'ParallelClusterVersion': self.config['slurm']['ParallelClusterConfig']['Version'], 'Region': self.cluster_region, 'SubnetId': self.config['SubnetId'], diff --git a/source/resources/lambdas/CreateBuildFiles/CreateBuildFiles.py b/source/resources/lambdas/CreateBuildFiles/CreateBuildFiles.py index d0831d15..3c82d175 100644 --- a/source/resources/lambdas/CreateBuildFiles/CreateBuildFiles.py +++ b/source/resources/lambdas/CreateBuildFiles/CreateBuildFiles.py @@ -217,6 +217,23 @@ def lambda_handler(event, context): Body = build_file_content ) + # Image with rootless Docker installed + template_vars['ImageName'] = f"parallelcluster-{parallelcluster_version_name}-docker-{distribution}-{version}-{architecture}".replace('_', '-') + template_vars['ComponentS3Url'] = environ['InstallDockerScriptS3Url'] + build_file_s3_key = f"{assets_base_key}/config/build-files/{template_vars['ImageName']}.yml" + if requestType == 'Delete': + response = s3_client.delete_object( + Bucket = assets_bucket, + Key = build_file_s3_key + ) + else: + build_file_content = build_file_template.render(**template_vars) + s3_client.put_object( + Bucket = assets_bucket, + Key = build_file_s3_key, + Body = build_file_content + ) + # Image with EDA packages template_vars['ImageName'] = f"parallelcluster-{parallelcluster_version_name}-eda-{distribution}-{version}-{architecture}".replace('_', '-') template_vars['ComponentS3Url'] = environ['ConfigureEdaScriptS3Url'] diff --git a/source/resources/parallel-cluster/config/bin/configure-eda.sh b/source/resources/parallel-cluster/config/bin/configure-eda.sh index 1adb638b..c82b5e10 100755 --- a/source/resources/parallel-cluster/config/bin/configure-eda.sh +++ b/source/resources/parallel-cluster/config/bin/configure-eda.sh @@ -88,4 +88,6 @@ ansible-playbook $PLAYBOOKS_PATH/eda_tools.yml \ -i inventories/local.yml \ -e @$ANSIBLE_PATH/ansible_head_node_vars.yml -popd +ansible-playbook $PLAYBOOKS_PATH/install-rootless-docker.yml \ + -i inventories/local.yml \ + -e @$ANSIBLE_PATH/ansible_head_node_vars.yml diff --git a/source/resources/parallel-cluster/config/bin/configure-rootless-docker.sh b/source/resources/parallel-cluster/config/bin/configure-rootless-docker.sh new file mode 100755 index 00000000..4e563d1c --- /dev/null +++ b/source/resources/parallel-cluster/config/bin/configure-rootless-docker.sh @@ -0,0 +1,86 @@ +#!/bin/bash -ex +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +# Configure rootless docker for user. +# The slurm config directory must exist + +script=$0 +script_name=$(basename $script) + +# Jinja2 template variables +assets_bucket={{assets_bucket}} +assets_base_key={{assets_base_key}} +export AWS_DEFAULT_REGION={{Region}} +ClusterName={{ClusterName}} +ErrorSnsTopicArn={{ErrorSnsTopicArn}} +playbooks_s3_url={{playbooks_s3_url}} + +# Notify user of errors +function on_exit { + rc=$? + set +e + if [[ $rc -ne 0 ]] && [[ ":$ErrorSnsTopicArn" != ":" ]]; then + tmpfile=$(mktemp) + echo "See log files for more info: + /var/lib/amazon/toe/TOE_* + grep PCImageBuilderEDA /var/log/messages | less" > $tmpfile + aws --region $AWS_DEFAULT_REGION sns publish --topic-arn $ErrorSnsTopicArn --subject "${ClusterName} configure-rootless-docker.sh failed" --message file://$tmpfile + rm $tmpfile + fi +} +trap on_exit EXIT + +# Redirect all IO to /var/log/messages and then echo to stderr +exec 1> >(logger -s -t configure-rootless-docker) 2>&1 + +# Install ansible +if ! yum list installed ansible &> /dev/null; then + yum install -y ansible || amazon-linux-extras install -y ansible2 +fi + +external_login_node_config_dir=/opt/slurm/${ClusterName}/config +if [ -e $external_login_node_config_dir ]; then + config_dir=$external_login_node_config_dir +else + config_dir=/opt/slurm/config +fi +config_bin_dir=$config_dir/bin +ANSIBLE_PATH=$config_dir/ansible +PLAYBOOKS_PATH=$ANSIBLE_PATH/playbooks +PLAYBOOKS_ZIP_PATH=$ANSIBLE_PATH/playbooks.zip + +if ! [ -e $external_login_node_config_dir ]; then + mkdir -p $config_bin_dir + + ansible_head_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_head_node_vars.yml" + ansible_compute_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_compute_node_vars.yml" + ansible_external_login_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_external_login_node_vars.yml" + + # Download ansible playbooks + aws s3 cp $playbooks_s3_url ${PLAYBOOKS_ZIP_PATH}.new + if ! [ -e $PLAYBOOKS_ZIP_PATH ] || ! diff -q $PLAYBOOKS_ZIP_PATH ${PLAYBOOKS_ZIP_PATH}.new; then + mv $PLAYBOOKS_ZIP_PATH.new $PLAYBOOKS_ZIP_PATH + rm -rf $PLAYBOOKS_PATH + mkdir -p $PLAYBOOKS_PATH + pushd $PLAYBOOKS_PATH + yum -y install unzip + unzip $PLAYBOOKS_ZIP_PATH + chmod -R 0700 $ANSIBLE_PATH + popd + fi + + aws s3 cp $ansible_head_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_head_node_vars.yml + + aws s3 cp $ansible_compute_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_compute_node_vars.yml + + aws s3 cp $ansible_external_login_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_external_login_node_vars.yml +fi + +pushd $PLAYBOOKS_PATH + +ansible-playbook $PLAYBOOKS_PATH/configure-rootless-docker.yml \ + -i inventories/local.yml \ + -e @$ANSIBLE_PATH/ansible_external_login_node_vars.yml + +popd diff --git a/source/resources/parallel-cluster/config/bin/install-rootless-docker.sh b/source/resources/parallel-cluster/config/bin/install-rootless-docker.sh new file mode 100755 index 00000000..e97d607f --- /dev/null +++ b/source/resources/parallel-cluster/config/bin/install-rootless-docker.sh @@ -0,0 +1,91 @@ +#!/bin/bash -ex +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +# This script calls an ansible playbook that installs rootless docker on a compute node. +# It has 2 different use cases: +# * To build ParallelCluster AMIs +# * To install docker on VDIs or other login nodes using a ParallellCluster. +# The location of the config directory is different for those 2 use cases. +# For an AMI build, the config directory and scripts will not exist and must be downloaded from S3. +# For a login node, the playbooks and scripts will already exist. + +script=$0 +script_name=$(basename $script) + +# Jinja2 template variables +assets_bucket={{assets_bucket}} +assets_base_key={{assets_base_key}} +export AWS_DEFAULT_REGION={{Region}} +ClusterName={{ClusterName}} +ErrorSnsTopicArn={{ErrorSnsTopicArn}} +playbooks_s3_url={{playbooks_s3_url}} + +# Notify user of errors +function on_exit { + rc=$? + set +e + if [[ $rc -ne 0 ]] && [[ ":$ErrorSnsTopicArn" != ":" ]]; then + tmpfile=$(mktemp) + echo "See log files for more info: + /var/lib/amazon/toe/TOE_* + grep PCImageBuilderEDA /var/log/messages | less" > $tmpfile + aws --region $AWS_DEFAULT_REGION sns publish --topic-arn $ErrorSnsTopicArn --subject "${ClusterName} install-rootless-docker.sh failed" --message file://$tmpfile + rm $tmpfile + fi +} +trap on_exit EXIT + +# Redirect all IO to /var/log/messages and then echo to stderr +exec 1> >(logger -s -t install-rootless-docker) 2>&1 + +# Install ansible +if ! yum list installed ansible &> /dev/null; then + yum install -y ansible || amazon-linux-extras install -y ansible2 +fi + +external_login_node_config_dir=/opt/slurm/${ClusterName}/config +if [ -e $external_login_node_config_dir ]; then + config_dir=$external_login_node_config_dir +else + config_dir=/opt/slurm/config +fi +config_bin_dir=$config_dir/bin +ANSIBLE_PATH=$config_dir/ansible +PLAYBOOKS_PATH=$ANSIBLE_PATH/playbooks +PLAYBOOKS_ZIP_PATH=$ANSIBLE_PATH/playbooks.zip + +if ! [ -e $external_login_node_config_dir ]; then + mkdir -p $config_bin_dir + + ansible_head_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_head_node_vars.yml" + ansible_compute_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_compute_node_vars.yml" + ansible_external_login_node_vars_yml_s3_url="s3://$assets_bucket/$assets_base_key/config/ansible/ansible_external_login_node_vars.yml" + + # Download ansible playbooks + aws s3 cp $playbooks_s3_url ${PLAYBOOKS_ZIP_PATH}.new + if ! [ -e $PLAYBOOKS_ZIP_PATH ] || ! diff -q $PLAYBOOKS_ZIP_PATH ${PLAYBOOKS_ZIP_PATH}.new; then + mv $PLAYBOOKS_ZIP_PATH.new $PLAYBOOKS_ZIP_PATH + rm -rf $PLAYBOOKS_PATH + mkdir -p $PLAYBOOKS_PATH + pushd $PLAYBOOKS_PATH + yum -y install unzip + unzip $PLAYBOOKS_ZIP_PATH + chmod -R 0700 $ANSIBLE_PATH + popd + fi + + aws s3 cp $ansible_head_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_head_node_vars.yml + + aws s3 cp $ansible_compute_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_compute_node_vars.yml + + aws s3 cp $ansible_external_login_node_vars_yml_s3_url /opt/slurm/config/ansible/ansible_external_login_node_vars.yml +fi + +pushd $PLAYBOOKS_PATH + +ansible-playbook $PLAYBOOKS_PATH/install-rootless-docker.yml \ + -i inventories/local.yml \ + -e @$ANSIBLE_PATH/ansible_compute_node_vars.yml + +popd diff --git a/source/resources/parallel-cluster/config/bin/on_head_node_start.sh b/source/resources/parallel-cluster/config/bin/on_head_node_start.sh index 1e0e0300..e1bddb48 100755 --- a/source/resources/parallel-cluster/config/bin/on_head_node_start.sh +++ b/source/resources/parallel-cluster/config/bin/on_head_node_start.sh @@ -64,11 +64,13 @@ users_groups_json_s3_url="s3://$assets_bucket/$assets_base_key/config/users_grou # Download all of the config scripts config_scripts=(\ configure-eda.sh \ + configure-rootless-docker.sh \ create_or_update_users_groups_json.sh \ create_users_groups_json.py \ create_users_groups_json_configure.sh \ create_users_groups_json_deconfigure.sh \ create_users_groups.py \ + install-rootless-docker.sh \ on_head_node_start.sh \ on_head_node_configured.sh \ on_head_node_updated.sh \ diff --git a/source/resources/playbooks/ParallelClusterComputeNode.yml b/source/resources/playbooks/ParallelClusterComputeNode.yml index 3b78de91..6a35ee52 100644 --- a/source/resources/playbooks/ParallelClusterComputeNode.yml +++ b/source/resources/playbooks/ParallelClusterComputeNode.yml @@ -10,3 +10,4 @@ - security_updates - bug_fixes - ParallelClusterComputeNode + - install-rootless-docker diff --git a/source/resources/playbooks/configure-rootless-docker.yml b/source/resources/playbooks/configure-rootless-docker.yml new file mode 100644 index 00000000..e2853453 --- /dev/null +++ b/source/resources/playbooks/configure-rootless-docker.yml @@ -0,0 +1,8 @@ +--- +- name: Configure rootless docker for user + hosts: + - ExternalLoginNode + become_user: root + become: yes + roles: + - configure-rootless-docker diff --git a/source/resources/playbooks/install-rootless-docker.yml b/source/resources/playbooks/install-rootless-docker.yml new file mode 100644 index 00000000..a758d23d --- /dev/null +++ b/source/resources/playbooks/install-rootless-docker.yml @@ -0,0 +1,8 @@ +--- +- name: Install rootless docker for OCI containers + hosts: + - ParallelClusterComputeNode + become_user: root + become: yes + roles: + - install-rootless-docker diff --git a/source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/etc/oci.conf b/source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/etc/oci.conf new file mode 100644 index 00000000..f8b53b1b --- /dev/null +++ b/source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/etc/oci.conf @@ -0,0 +1,6 @@ +EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" +RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)=" +RunTimeQuery="runc --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t" +RunTimeKill="runc --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t" +RunTimeDelete="runc --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t" +RunTimeRun="runc --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b diff --git a/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/config-oci.yml b/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/config-oci.yml new file mode 100644 index 00000000..10952819 --- /dev/null +++ b/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/config-oci.yml @@ -0,0 +1,10 @@ +--- + +- name: Create oci.conf + when: primary_controller|bool + copy: + dest: "/opt/slurm/etc/oci.conf" + src: opt/slurm/etc/oci.conf + owner: root + group: root + mode: 0644 diff --git a/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/main.yml b/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/main.yml index 7541590c..a2971860 100644 --- a/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/main.yml +++ b/source/resources/playbooks/roles/ParallelClusterHeadNode/tasks/main.yml @@ -7,4 +7,5 @@ - { include_tasks: config-slurmrestd.yml, tags: slurmrestd } - { include_tasks: config-licenses.yml, tags: licenses } - { include_tasks: config-slurmdb-accounts.yml, tags: accounts } +- { include_tasks: config-oci.yml } - { include_tasks: config-pyxis.yml } diff --git a/source/resources/playbooks/roles/configure-rootless-docker/README.md b/source/resources/playbooks/roles/configure-rootless-docker/README.md new file mode 100644 index 00000000..970a87f4 --- /dev/null +++ b/source/resources/playbooks/roles/configure-rootless-docker/README.md @@ -0,0 +1,14 @@ +configure-rootless-docker +========= + +Configure user to run rootless docker. + +License +------- + +mit0 + +Author Information +------------------ + +Allan Carter (cartalla@amazon.com) diff --git a/source/resources/playbooks/roles/configure-rootless-docker/tasks/main.yml b/source/resources/playbooks/roles/configure-rootless-docker/tasks/main.yml new file mode 100644 index 00000000..eda93e73 --- /dev/null +++ b/source/resources/playbooks/roles/configure-rootless-docker/tasks/main.yml @@ -0,0 +1,6 @@ +--- + +- name: Configure rootless docker + shell: + cmd: | + dockerd-rootless-setuptool.sh install diff --git a/source/resources/playbooks/roles/install-rootless-docker/README.md b/source/resources/playbooks/roles/install-rootless-docker/README.md new file mode 100644 index 00000000..eb9098e7 --- /dev/null +++ b/source/resources/playbooks/roles/install-rootless-docker/README.md @@ -0,0 +1,14 @@ +install-rootless-docker +========= + +Install rootless docker for use by OCI containers. + +License +------- + +mit0 + +Author Information +------------------ + +Allan Carter (cartalla@amazon.com) diff --git a/source/resources/playbooks/roles/install-rootless-docker/tasks/main.yml b/source/resources/playbooks/roles/install-rootless-docker/tasks/main.yml new file mode 100644 index 00000000..48d6b9c3 --- /dev/null +++ b/source/resources/playbooks/roles/install-rootless-docker/tasks/main.yml @@ -0,0 +1,65 @@ +--- +# tasks file for install-rootless-docker + +- name: Remove Docker packages + yum: + state: removed + name: + - docker + - docker-client + - docker-client-latest + - docker-common + - docker-latest + - docker-latest-logrotate + - docker-logrotate + - docker-engine + - podman + - runc + +- name: Install dnf-plugins-core + when: (rhel8 or rhel8clone or rhel9 or rhel9clone) + yum: + state: present + name: + - dnf-plugins-core + +- name: Set up Docker repository + shell: + cmd: | + dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo + creates: /etc/yum.repos.d/docker-ce.repo + +- name: Install docker packages + yum: + state: present + name: + - docker-ce + - docker-ce-cli + - containerd.io + - docker-buildx-plugin + - docker-compose-plugin + - fuse-overlayfs + - iptables + +- name: Disable docker.service + systemd_service: + name: docker.service + enabled: false + state: stopped + +- name: Disable docker.socket + systemd_service: + name: docker.socket + enabled: false + state: stopped + +- name: Remove /var/run/docker.sock + file: + path: /var/run/docker.sock + state: absent + +- name: Load ip_tables kernel module + community.general.modprobe: + name: ip_tables + state: present + persistent: present