- FPGA Developer AMI
- - Deploy the Cluster
+
- Deploy or update the Cluster
@@ -130,10 +126,13 @@
Custom AMIs for ParallelCluster
ParallelCluster supports building custom AMIs for the compute nodes.
-The easiest way is to start an EC2 instances, update it with your changes, and create a new AMI from that instance.
+The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance.
You can then add the new AMI to your configuration file.
-ParallelCluster can also automate this process for you and when you build your cluster, example ParallelCluster build configuration files
-will be created for you in source/resources/parallel-cluster/config/build-files/parallelcluster-eda-*.yml.
+ParallelCluster can also automate this process for you using EC2 ImageBuilder.
+When you build your cluster, example ParallelCluster build configuration files
+will be created for you and stored on the head node at:
+/opt/slurm/ClusterName/config/build-files/parallelcluster-PCVersion-*.yml
+The build files with eda in the name build an image that installs the packages that are typically used by EDA tools.
The easiest way is to use the ParallelCluster UI to build the AMI using a build config file.
- Click on Images on the left
@@ -143,15 +142,23 @@ Custom AMIs for ParallelCluster
- Copy the image/name value into the Image Id field. It should begin with parallelcluster-
- Click Build Image
+The UI will create a cloudformation template that uses EC2 ImageBuilder.
+While it is being built it will show up as Pending in the UI.
+When the build is complete the AMI will show up either as Available or Failed.
+If it fails, the instance used to do the build will be left running.
+You can connect to it using SSM and lookin in /var/log/messages for error messages.
+When the build is successful, the stack will be deleted.
+There is currently a bug where the stack deletion will fail.
+This doesn't mean that the AMI build failed.
+Simply select the stack and delete it manually and it should successfully delete.
FPGA Developer AMI
-This tutorial shows how to create an AMI based on the AWS FPGA Developer AMI.
+
The build file with fpga in the name is based on the FPGS Developer AMI.
The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional
charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances.
-Subscribe To the AMI
First subscribe to the FPGA developer AMI in the AWS Marketplace.
There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2.
-Deploy the Cluster
-With the config updated, the AMIs for the compute nodes will be built using the specified base AMIs.
+Deploy or update the Cluster
+After the AMI is built, add it to the config and create or update your cluster to use the AMI.
diff --git a/index.html b/index.html
index fde9ae0c..1160ce93 100644
--- a/index.html
+++ b/index.html
@@ -306,5 +306,5 @@ Keyboard Shortcuts
diff --git a/search/search_index.json b/search/search_index.json
index fd2cec06..622cca43 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"AWS EDA Slurm Cluster This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters. Operating System and Processor Architecture Support This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture. Documentation View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs Security See CONTRIBUTING for more information. License This library is licensed under the MIT-0 License. See the LICENSE file.","title":"AWS EDA Slurm Cluster"},{"location":"#aws-eda-slurm-cluster","text":"This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters.","title":"AWS EDA Slurm Cluster"},{"location":"#operating-system-and-processor-architecture-support","text":"This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture.","title":"Operating System and Processor Architecture Support"},{"location":"#documentation","text":"View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs","title":"Documentation"},{"location":"#security","text":"See CONTRIBUTING for more information.","title":"Security"},{"location":"#license","text":"This library is licensed under the MIT-0 License. See the LICENSE file.","title":"License"},{"location":"CONTRIBUTING/","text":"Contributing Guidelines Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution. Reporting Bugs/Feature Requests We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment Contributing via Pull Requests Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request . Finding contributions to work on Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. Code of Conduct This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments. Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue. Licensing See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#contributing-guidelines","text":"Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#reporting-bugsfeature-requests","text":"We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment","title":"Reporting Bugs/Feature Requests"},{"location":"CONTRIBUTING/#contributing-via-pull-requests","text":"Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request .","title":"Contributing via Pull Requests"},{"location":"CONTRIBUTING/#finding-contributions-to-work-on","text":"Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.","title":"Finding contributions to work on"},{"location":"CONTRIBUTING/#code-of-conduct","text":"This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments.","title":"Code of Conduct"},{"location":"CONTRIBUTING/#security-issue-notifications","text":"If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue.","title":"Security issue notifications"},{"location":"CONTRIBUTING/#licensing","text":"See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Licensing"},{"location":"custom-amis/","text":"Custom AMIs for ParallelCluster ParallelCluster supports building custom AMIs for the compute nodes . The easiest way is to start an EC2 instances, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you and when you build your cluster, example ParallelCluster build configuration files will be created for you in source/resources/parallel-cluster/config/build-files/parallelcluster-eda-*.yml . The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image FPGA Developer AMI This tutorial shows how to create an AMI based on the AWS FPGA Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. Subscribe To the AMI First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 . Deploy the Cluster With the config updated, the AMIs for the compute nodes will be built using the specified base AMIs.","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#custom-amis-for-parallelcluster","text":"ParallelCluster supports building custom AMIs for the compute nodes . The easiest way is to start an EC2 instances, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you and when you build your cluster, example ParallelCluster build configuration files will be created for you in source/resources/parallel-cluster/config/build-files/parallelcluster-eda-*.yml . The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#fpga-developer-ami","text":"This tutorial shows how to create an AMI based on the AWS FPGA Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances.","title":"FPGA Developer AMI"},{"location":"custom-amis/#subscribe-to-the-ami","text":"First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 .","title":"Subscribe To the AMI"},{"location":"custom-amis/#deploy-the-cluster","text":"With the config updated, the AMIs for the compute nodes will be built using the specified base AMIs.","title":"Deploy the Cluster"},{"location":"debug/","text":"Debug For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation . Slurm Head Node If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv Compute Nodes If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd Log Files Logfile Description /var/log/slurmd.log slurmctld logfile Job Stuck in Pending State You can use scontrol to get detailed information about a job. scontrol show job *jobid* Job Stuck in Completing State When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Debug"},{"location":"debug/#debug","text":"For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation .","title":"Debug"},{"location":"debug/#slurm-head-node","text":"If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv","title":"Slurm Head Node"},{"location":"debug/#compute-nodes","text":"If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd","title":"Compute Nodes"},{"location":"debug/#log-files","text":"Logfile Description /var/log/slurmd.log slurmctld logfile","title":"Log Files"},{"location":"debug/#job-stuck-in-pending-state","text":"You can use scontrol to get detailed information about a job. scontrol show job *jobid*","title":"Job Stuck in Pending State"},{"location":"debug/#job-stuck-in-completing-state","text":"When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Job Stuck in Completing State"},{"location":"delete-cluster/","text":"Delete Cluster To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"delete-cluster/#delete-cluster","text":"To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"deploy-parallel-cluster/","text":"Deploy AWS ParallelCluster A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0. Prerequisites See Deployment Prerequisites page. Create ParallelCluster UI (optional but recommended) It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide . Create ParallelCluster Slurm Database The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting . Create the Cluster To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster. Create users_groups.json Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys. Configure submission hosts to use the cluster ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm. Customize the compute node AMI The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again. Run Your First Job Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash Slurm Documentation https://slurm.schedmd.com","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#deploy-aws-parallelcluster","text":"A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0.","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#prerequisites","text":"See Deployment Prerequisites page.","title":"Prerequisites"},{"location":"deploy-parallel-cluster/#create-parallelcluster-ui-optional-but-recommended","text":"It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide .","title":"Create ParallelCluster UI (optional but recommended)"},{"location":"deploy-parallel-cluster/#create-parallelcluster-slurm-database","text":"The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting .","title":"Create ParallelCluster Slurm Database"},{"location":"deploy-parallel-cluster/#create-the-cluster","text":"To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster.","title":"Create the Cluster"},{"location":"deploy-parallel-cluster/#create-users_groupsjson","text":"Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys.","title":"Create users_groups.json"},{"location":"deploy-parallel-cluster/#configure-submission-hosts-to-use-the-cluster","text":"ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm.","title":"Configure submission hosts to use the cluster"},{"location":"deploy-parallel-cluster/#customize-the-compute-node-ami","text":"The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again.","title":"Customize the compute node AMI"},{"location":"deploy-parallel-cluster/#run-your-first-job","text":"Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash","title":"Run Your First Job"},{"location":"deploy-parallel-cluster/#slurm-documentation","text":"https://slurm.schedmd.com","title":"Slurm Documentation"},{"location":"deployment-prerequisites/","text":"Deployment Prerequisites This page shows common prerequisites that need to be done before deployment. Deployment Server/Instance Requirements The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance. Configure AWS CLI Credentials You will needs AWS credentials that provide admin access to deploy the cluster. Clone or Download the Repository Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git Create SNS Topic for Error Notifications (Optional but recommended) The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket. Make sure using at least python version 3.7 This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5 Make sure required packages are installed cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment. Install Cloud Development Kit (CDK) (Optional) The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed. Create Configuration File Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None slurm/SubmitterSecurityGroupIds Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. sg-* None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml Configure the Compute Instances The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 Rocky 8 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1 Configure Fair Share Scheduling (Optional) Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities. Configure Licenses Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-prerequisites","text":"This page shows common prerequisites that need to be done before deployment.","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-serverinstance-requirements","text":"The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance.","title":"Deployment Server/Instance Requirements"},{"location":"deployment-prerequisites/#configure-aws-cli-credentials","text":"You will needs AWS credentials that provide admin access to deploy the cluster.","title":"Configure AWS CLI Credentials"},{"location":"deployment-prerequisites/#clone-or-download-the-repository","text":"Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git","title":"Clone or Download the Repository"},{"location":"deployment-prerequisites/#create-sns-topic-for-error-notifications-optional-but-recommended","text":"The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket.","title":"Create SNS Topic for Error Notifications (Optional but recommended)"},{"location":"deployment-prerequisites/#make-sure-using-at-least-python-version-37","text":"This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5","title":"Make sure using at least python version 3.7"},{"location":"deployment-prerequisites/#make-sure-required-packages-are-installed","text":"cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment.","title":"Make sure required packages are installed"},{"location":"deployment-prerequisites/#install-cloud-development-kit-cdk-optional","text":"The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed.","title":"Install Cloud Development Kit (CDK) (Optional)"},{"location":"deployment-prerequisites/#create-configuration-file","text":"Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None slurm/SubmitterSecurityGroupIds Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. sg-* None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml","title":"Create Configuration File"},{"location":"deployment-prerequisites/#configure-the-compute-instances","text":"The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 Rocky 8 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1","title":"Configure the Compute Instances"},{"location":"deployment-prerequisites/#configure-fair-share-scheduling-optional","text":"Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities.","title":"Configure Fair Share Scheduling (Optional)"},{"location":"deployment-prerequisites/#configure-licenses","text":"Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Configure Licenses"},{"location":"federation/","text":"Federation (legacy) To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"federation/#federation-legacy","text":"To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"implementation/","text":"Implementation Details (legacy) Slurm Infrastructure All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons. Directory Structure All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}} CloudWatch Metrics CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py Down Node Handling If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer Insufficient Capacity Exception (ICE) Handling When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Implementation Details (legacy)"},{"location":"implementation/#implementation-details-legacy","text":"","title":"Implementation Details (legacy)"},{"location":"implementation/#slurm-infrastructure","text":"All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons.","title":"Slurm Infrastructure"},{"location":"implementation/#directory-structure","text":"All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}}","title":"Directory Structure"},{"location":"implementation/#cloudwatch-metrics","text":"CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py","title":"CloudWatch Metrics"},{"location":"implementation/#down-node-handling","text":"If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer","title":"Down Node Handling"},{"location":"implementation/#insufficient-capacity-exception-ice-handling","text":"When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Insufficient Capacity Exception (ICE) Handling"},{"location":"job_preemption/","text":"Job Preemption The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license. Documentation https://slurm.schedmd.com/preempt.html","title":"Job Preemption"},{"location":"job_preemption/#job-preemption","text":"The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license.","title":"Job Preemption"},{"location":"job_preemption/#documentation","text":"https://slurm.schedmd.com/preempt.html","title":"Documentation"},{"location":"onprem/","text":"On-Premises Integration The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information. Network Requirements The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes. DNS Requirements Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC. File System Requirements All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm. Slurm Configuration of On-Premises Compute Nodes The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem Simulating an On-Premises Network Using AWS Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"On-Premises Integration"},{"location":"onprem/#on-premises-integration","text":"The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information.","title":"On-Premises Integration"},{"location":"onprem/#network-requirements","text":"The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes.","title":"Network Requirements"},{"location":"onprem/#dns-requirements","text":"Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC.","title":"DNS Requirements"},{"location":"onprem/#file-system-requirements","text":"All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm.","title":"File System Requirements"},{"location":"onprem/#slurm-configuration-of-on-premises-compute-nodes","text":"The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem","title":"Slurm Configuration of On-Premises Compute Nodes"},{"location":"onprem/#simulating-an-on-premises-network-using-aws","text":"Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"Simulating an On-Premises Network Using AWS"},{"location":"res_integration/","text":"RES Integration Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx SubmitterSecurityGroupIds The security group names and ids used by RES VDIs. The name will be something like EnvironmentName -vdc-dcv-host-security-group EnvironmentName - VDISG : sg-xxxxxxxx SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"res_integration/#res-integration","text":"Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx SubmitterSecurityGroupIds The security group names and ids used by RES VDIs. The name will be something like EnvironmentName -vdc-dcv-host-security-group EnvironmentName - VDISG : sg-xxxxxxxx SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"rest_api/","text":"Slurm REST API The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster. How to use the REST API The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"Slurm REST API"},{"location":"rest_api/#slurm-rest-api","text":"The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster.","title":"Slurm REST API"},{"location":"rest_api/#how-to-use-the-rest-api","text":"The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"How to use the REST API"},{"location":"run_jobs/","text":"Run Jobs This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages. Set Up Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format. Key Slurm Commands The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state sbatch The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script. Run a simulation build followed by a regression build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi srun The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs squeue The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue sprio Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11 sacct Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct sreport The sreport command can be used to generate report from the Slurm database. Other Slurm Commands Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Run Jobs"},{"location":"run_jobs/#run-jobs","text":"This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages.","title":"Run Jobs"},{"location":"run_jobs/#set-up","text":"Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format.","title":"Set Up"},{"location":"run_jobs/#key-slurm-commands","text":"The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state","title":"Key Slurm Commands"},{"location":"run_jobs/#sbatch","text":"The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script.","title":"sbatch"},{"location":"run_jobs/#run-a-simulation-build-followed-by-a-regression","text":"build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi","title":"Run a simulation build followed by a regression"},{"location":"run_jobs/#srun","text":"The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs","title":"srun"},{"location":"run_jobs/#squeue","text":"The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue","title":"squeue"},{"location":"run_jobs/#sprio","text":"Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11","title":"sprio"},{"location":"run_jobs/#sacct","text":"Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct","title":"sacct"},{"location":"run_jobs/#sreport","text":"The sreport command can be used to generate report from the Slurm database.","title":"sreport"},{"location":"run_jobs/#other-slurm-commands","text":"Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Other Slurm Commands"},{"location":"soca_integration/","text":"SOCA Integration Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx SubmitterSecurityGroupIds The ComputeNode security group name and id cluster-id - ComputeNodeSG : sg-xxxxxxxx ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"},{"location":"soca_integration/#soca-integration","text":"Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx SubmitterSecurityGroupIds The ComputeNode security group name and id cluster-id - ComputeNodeSG : sg-xxxxxxxx ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"AWS EDA Slurm Cluster This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters. Operating System and Processor Architecture Support This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture. Documentation View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs Security See CONTRIBUTING for more information. License This library is licensed under the MIT-0 License. See the LICENSE file.","title":"AWS EDA Slurm Cluster"},{"location":"#aws-eda-slurm-cluster","text":"This repository contains an AWS Cloud Development Kit (CDK) application that creates a Slurm cluster that is suitable for running production EDA workloads on AWS. The original (legacy) version of this repo that used a custom Python plugin to integrate Slurm with AWS has been deprecated and is no longer supported. It can be found on the v1 branch. The latest version of the repo uses AWS ParallelCluster for the core Slurm infrastructure and AWS integration. The big advantage of moving to AWS ParallelCluster is that it is a supported AWS service. Currently, some of the features of the legacy version are not supported in the ParallelCluster version, but work continues to add features to ParallelCluster so that those features may be supported in the future. Key features are: Automatic scaling of AWS EC2 instances based on demand Use any AWS EC2 instance type including Graviton2 Use of spot instances Handling of spot terminations Handling of insufficient capacity exceptions Batch and interactive partitions (queues) Manages tool licenses as a consumable resource User and group fair share scheduling Slurm accounting database CloudWatch dashboard Job preemption Manage on-premises compute nodes Configure partitions (queues) and nodes that are always on to support reserved instances (RIs) and savings plans (SPs). Features in the legacy version and not in the ParallelCluster version: Heterogenous clusters with mixed OSes and CPU architectures on compute nodes. Multi-AZ support. Supported by ParallelCluster, but not currently implemented. Multi-region support AWS Fault Injection Simulator (FIS) templates to test spot terminations Support for MungeKeySsmParameter Multi-cluster federation ParallelCluster Limitations Number of \"Compute Resources\" (CRs) is limited to 50 which limits the number of instance types allowed in a cluster. ParallelCluster can have multiple instance types in a CR, but with memory based scheduling enabled, they must all have the same number of cores and amount of memory. All Slurm instances must have the same OS and CPU architecture. Stand-alone Slurm database daemon instance. Prevents federation. Multi-region support. This is unlikely to change because multi-region services run against our archiectural philosophy. Federation may be an option but its current implementation limits scheduler performance and doesn't allow cluster prioritization so jobs land on random clusters. Slurm Limitations Job preemption based on licenses Federation doesn't support prioritizing federated clusters for job scheduling. Result is jobs scattered across the federated clusters.","title":"AWS EDA Slurm Cluster"},{"location":"#operating-system-and-processor-architecture-support","text":"This Slurm cluster supports the following OSes: ParallelCluster: Amazon Linux 2 CentOS 7 RedHat 7 and 8 This Slurm cluster supports both Intel/AMD (x86_64) based instances and ARM Graviton2 (arm64/aarch64) based instances. Graviton instances require Amazon Linux 2 or RedHat 8 operating systems. RedHat 7 and CentOS 7 do not support Graviton 2. This provides the following different combinations of OS and processor architecture. ParallelCluster: Amazon Linux 2 and arm64 Amazon Linux 2 and x86_64 CentOS 7 and x86_64 RedHat 7 and x86_64 RedHat 8 and arm64 RedHat 8 and x86_64 Note that in ParallelCluster, all compute nodes must have the same OS and architecture.","title":"Operating System and Processor Architecture Support"},{"location":"#documentation","text":"View on GitHub Pages You can also view the docs locally, The docs are in the docs directory. You can view them in an editor or using the mkdocs tool. I recommend installing mkdocs in a python virtual environment. python3 -m venv ~/.mkdocs_venv source ~/.mkdocs_venv/bin/activate pip install mkdocs Then run mkdocs. source ~/.mkdocs_venv/bin/activate mkdocs serve & firefox http://127.0.0.1:8000/ & Open a browser to: http://127.0.0.1:8000/ Or you can simply let make do this for you. make local-docs","title":"Documentation"},{"location":"#security","text":"See CONTRIBUTING for more information.","title":"Security"},{"location":"#license","text":"This library is licensed under the MIT-0 License. See the LICENSE file.","title":"License"},{"location":"CONTRIBUTING/","text":"Contributing Guidelines Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution. Reporting Bugs/Feature Requests We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment Contributing via Pull Requests Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request . Finding contributions to work on Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. Code of Conduct This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments. Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue. Licensing See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#contributing-guidelines","text":"Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution.","title":"Contributing Guidelines"},{"location":"CONTRIBUTING/#reporting-bugsfeature-requests","text":"We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: A reproducible test case or series of steps The version of our code being used Any modifications you've made relevant to the bug Anything unusual about your environment or deployment","title":"Reporting Bugs/Feature Requests"},{"location":"CONTRIBUTING/#contributing-via-pull-requests","text":"Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: You are working against the latest source on the main branch. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. You open an issue to discuss any significant work - we would hate for your time to be wasted. To send us a pull request, please: Fork the repository. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. Ensure local tests pass. Commit to your fork using clear commit messages. Send us a pull request, answering any default questions in the pull request interface. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on forking a repository and creating a pull request .","title":"Contributing via Pull Requests"},{"location":"CONTRIBUTING/#finding-contributions-to-work-on","text":"Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.","title":"Finding contributions to work on"},{"location":"CONTRIBUTING/#code-of-conduct","text":"This project has adopted the Amazon Open Source Code of Conduct . For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments.","title":"Code of Conduct"},{"location":"CONTRIBUTING/#security-issue-notifications","text":"If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page . Please do not create a public github issue.","title":"Security issue notifications"},{"location":"CONTRIBUTING/#licensing","text":"See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.","title":"Licensing"},{"location":"custom-amis/","text":"Custom AMIs for ParallelCluster ParallelCluster supports building custom AMIs for the compute nodes . The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you using EC2 ImageBuilder. When you build your cluster, example ParallelCluster build configuration files will be created for you and stored on the head node at: /opt/slurm/ ClusterName /config/build-files/parallelcluster- PCVersion -*.yml The build files with eda in the name build an image that installs the packages that are typically used by EDA tools. The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image The UI will create a cloudformation template that uses EC2 ImageBuilder. While it is being built it will show up as Pending in the UI. When the build is complete the AMI will show up either as Available or Failed . If it fails, the instance used to do the build will be left running. You can connect to it using SSM and lookin in /var/log/messages for error messages. When the build is successful, the stack will be deleted. There is currently a bug where the stack deletion will fail. This doesn't mean that the AMI build failed. Simply select the stack and delete it manually and it should successfully delete. FPGA Developer AMI The build file with fpga in the name is based on the FPGS Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 . Deploy or update the Cluster After the AMI is built, add it to the config and create or update your cluster to use the AMI.","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#custom-amis-for-parallelcluster","text":"ParallelCluster supports building custom AMIs for the compute nodes . The easiest way is to start an EC2 instance, update it with your changes, and create a new AMI from that instance. You can then add the new AMI to your configuration file. ParallelCluster can also automate this process for you using EC2 ImageBuilder. When you build your cluster, example ParallelCluster build configuration files will be created for you and stored on the head node at: /opt/slurm/ ClusterName /config/build-files/parallelcluster- PCVersion -*.yml The build files with eda in the name build an image that installs the packages that are typically used by EDA tools. The easiest way is to use the ParallelCluster UI to build the AMI using a build config file. Click on Images on the left Click on the Custom tab Click on Build Image Paste the contents of a config file into the window. Copy the image/name value into the Image Id field. It should begin with parallelcluster- Click Build Image The UI will create a cloudformation template that uses EC2 ImageBuilder. While it is being built it will show up as Pending in the UI. When the build is complete the AMI will show up either as Available or Failed . If it fails, the instance used to do the build will be left running. You can connect to it using SSM and lookin in /var/log/messages for error messages. When the build is successful, the stack will be deleted. There is currently a bug where the stack deletion will fail. This doesn't mean that the AMI build failed. Simply select the stack and delete it manually and it should successfully delete.","title":"Custom AMIs for ParallelCluster"},{"location":"custom-amis/#fpga-developer-ami","text":"The build file with fpga in the name is based on the FPGS Developer AMI. The FPGA Developer AMI has the Xilinx Vivado tools that can be used free of additional charges when run on AWS EC2 instances to develop FPGA images that can be run on AWS F1 instances. First subscribe to the FPGA developer AMI in the AWS Marketplace . There are 2 versions, one for CentOS 7 and the other for Amazon Linux 2 .","title":"FPGA Developer AMI"},{"location":"custom-amis/#deploy-or-update-the-cluster","text":"After the AMI is built, add it to the config and create or update your cluster to use the AMI.","title":"Deploy or update the Cluster"},{"location":"debug/","text":"Debug For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation . Slurm Head Node If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv Compute Nodes If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd Log Files Logfile Description /var/log/slurmd.log slurmctld logfile Job Stuck in Pending State You can use scontrol to get detailed information about a job. scontrol show job *jobid* Job Stuck in Completing State When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Debug"},{"location":"debug/#debug","text":"For ParallelCluster and Slurm issues, refer to the official AWS ParallelCluster Troubleshooting documentation .","title":"Debug"},{"location":"debug/#slurm-head-node","text":"If slurm commands hang, then it's likely a problem with the Slurm controller. Connect to the head node from the EC2 console using SSM Manager or ssh and switch to the root user. sudo su The first thing to do is to ensure that the Slurm controller daemon is running: systemctl status slurmctld If it isn't then first check for errors in the user data script. The following command will show the output: grep cloud-init /var/log/messages | less Then check the controller's logfile. /var/log/slurmctld.log The following command will rerun the user data. /var/lib/cloud/instance/scripts/part-001 Another way to debug the slurmctld daemon is to launch it interactively with debug set high. The first thing to do is get the path to the slurmctld binary. slurmctld=$(cat /etc/systemd/system/slurmctld.service | awk -F '=' '/ExecStart/ {print $2}') Then you can run slurmctld: $slurmctld -D -vvvvv","title":"Slurm Head Node"},{"location":"debug/#compute-nodes","text":"If there are problems with the compute nodes, connect to them using SSM Manager. Check for cloud-init errors the same way as for the slurmctl instance. The compute nodes do not run ansible; their AMIs are configured using ansible. Also check the slurmd.log . Check that the slurm daemon is running. systemctl status slurmd","title":"Compute Nodes"},{"location":"debug/#log-files","text":"Logfile Description /var/log/slurmd.log slurmctld logfile","title":"Log Files"},{"location":"debug/#job-stuck-in-pending-state","text":"You can use scontrol to get detailed information about a job. scontrol show job *jobid*","title":"Job Stuck in Pending State"},{"location":"debug/#job-stuck-in-completing-state","text":"When a node starts it reports it's number of cores and free memory to the controller. If the memory is less than in slurm_node.conf then the controller will mark the node as invalid. You can confirm this by searching for the node in /var/log/slurm/slurmctld.log on the controller. If this happens, fix the memory in slurm_nodes.conf and restart slurmctld. systemctl restart slurmctld Then reboot the node. Another cause of this is a hung process on the compute node. To clear this out, connect to the slurm controller and mark the node down, resume, and then idle. scontrol update node NODENAME state=DOWN reason=hung scontrol update node NODENAME state=RESUME scontrol update node NODENAME state=IDLE","title":"Job Stuck in Completing State"},{"location":"delete-cluster/","text":"Delete Cluster To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"delete-cluster/#delete-cluster","text":"To delete the cluster all you need to do is delete the configuration CloudFormation stack. This will delete the ParallelCluster cluster and all of the configuration resources. If you specified RESEnvironmentName then it will also deconfigure the creation of users_groups.json and also deconfigure the VDI instances so they are no longer using the cluster. If you deployed the Slurm database stack then you can keep that and use it for other clusters. If you don't need it anymore, then you can delete the stack. You will also need to manually delete the RDS database. If you deployed the ParallelCluster UI then you can keep it and use it with other clusters. If you don't need it anymore then you can delete the stack.","title":"Delete Cluster"},{"location":"deploy-parallel-cluster/","text":"Deploy AWS ParallelCluster A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0. Prerequisites See Deployment Prerequisites page. Create ParallelCluster UI (optional but recommended) It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide . Create ParallelCluster Slurm Database The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting . Create the Cluster To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster. Create users_groups.json Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys. Configure submission hosts to use the cluster ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm. Customize the compute node AMI The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again. Run Your First Job Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash Slurm Documentation https://slurm.schedmd.com","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#deploy-aws-parallelcluster","text":"A ParallelCluster configuration will be generated and used to create a ParallelCluster slurm cluster. The first supported ParallelCluster version is 3.6.0. Version 3.7.0 is the recommended minimum version because it supports compute node weighting that is proportional to instance type cost so that the least expensive instance types that meet job requirements are used. The current latest version is 3.8.0.","title":"Deploy AWS ParallelCluster"},{"location":"deploy-parallel-cluster/#prerequisites","text":"See Deployment Prerequisites page.","title":"Prerequisites"},{"location":"deploy-parallel-cluster/#create-parallelcluster-ui-optional-but-recommended","text":"It is highly recommended to create a ParallelCluster UI to manage your ParallelCluster clusters. A different UI is required for each version of ParallelCluster that you are using. The versions are list in the ParallelCluster Release Notes . The minimum required version is 3.6.0 which adds support for RHEL 8 and increases the number of allows queues and compute resources. The suggested version is at least 3.7.0 because it adds configurable compute node weights which we use to prioritize the selection of compute nodes by their cost. The instructions are in the ParallelCluster User Guide .","title":"Create ParallelCluster UI (optional but recommended)"},{"location":"deploy-parallel-cluster/#create-parallelcluster-slurm-database","text":"The Slurm Database is required for configuring Slurm accounts, users, groups, and fair share scheduling. It you need these and other features then you will need to create a ParallelCluster Slurm Database. You do not need to create a new database for each cluster; multiple clusters can share the same database. Follow the directions in this ParallelCluster tutorial to configure slurm accounting .","title":"Create ParallelCluster Slurm Database"},{"location":"deploy-parallel-cluster/#create-the-cluster","text":"To install the cluster run the install script. You can override some parameters in the config file with command line arguments, however it is better to specify all of the parameters in the config file. ./install.sh --config-file --cdk-cmd create This will create the ParallelCuster configuration file, store it in S3, and then use a lambda function to create the cluster. If you look in CloudFormation you will see 2 new stacks when deployment is finished. The first is the configuration stack and the second is the cluster.","title":"Create the Cluster"},{"location":"deploy-parallel-cluster/#create-users_groupsjson","text":"Before you can use the cluster you must configure the Linux users and groups for the head and compute nodes. One way to do that would be to join the cluster to your domain. But joining each compute node to a domain effectively creates a distributed denial of service (DDOS) attack on the demain controller when the cluster rapidly scales out or in and each node tries to join or leave the domain. This can lead to domain controller timeouts and widespread havoc in your environment. To solve this problem a script runs on a server that is joined to the domain which writes a JSON file with all of the non-privileged users and groups and their respective uids and gids. A script and cron job on the head and compute nodes reads this json file to create local users and groups that match the domain-joined servers. Select the server that you want to use to create and update the JSON file. The outputs of the configuration stack have the commands required. Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command02CreateUsersGroupsJsonConfigure Create /opt/slurm/{{ClusterName}}/config/users_groups.json and create a cron job to refresh it hourly. Before deleting the cluster you can undo the configuration by running the commands in the following outputs. Config Stack Output Description command10CreateUsersGroupsJsonDeconfigure Removes the crontab that refreshes users_groups.json. Now the cluster is ready to be used by sshing into the head node or a login node, if you configured one. If you configured extra file systems for the cluster that contain the users' home directories, then they should be able to ssh in with their own ssh keys.","title":"Create users_groups.json"},{"location":"deploy-parallel-cluster/#configure-submission-hosts-to-use-the-cluster","text":"ParallelCluster was built assuming that users would ssh into the head node or login nodes to execute Slurm commands. This can be undesirable for a number of reasons. First, users shouldn't be given ssh access to a critical infrastructure like the cluster head node. With ParallelCluster 3.7.0 you can configure login nodes, but if you have already provisioned desktop nodes then it's wasteful to have to provision login nodes. Second, it's just inconvenient to have to use ssh to access the cluster and use it. Fortunately, you can configure any server as a submission host so that users can run slurm commands. These commands must be run by an administrator that has root access to the submission host. The commands could also be run to create a custom AMI for user desktops so that they can access the clusters. The commands to configure submission hosts are in the outputs of the configuration CloudFormation stack. Run them in the following order: Config Stack Output Description Command01SubmitterMountHeadNode Mounts the Slurm cluster's shared file system, adds it to /etc/fstab. Command03SubmitterConfigure Configure the submission host so it can directly access the Slurm cluster. The first command simply mounts the head node's NFS file system so you have access to the Slurm commands and configuration. The second command runs an ansible playbook that configures the submission host so that it can run the Slurm commands for the cluster. It also configures the modulefile that sets up the environment to use the slurm cluster. The clusters have been configured so that a submission host can use more than one cluster by simply changing the modulefile that is loaded. On the submission host just open a new shell and load the modulefile for your cluster and you can access Slurm.","title":"Configure submission hosts to use the cluster"},{"location":"deploy-parallel-cluster/#customize-the-compute-node-ami","text":"The easiest way to create a custom AMI is to find the default ParallelCluster AMI in the UI. Create an instance using the AMI and make whatever customizations you require such as installing packages and configuring users and groups. Custom file system mounts can be configured in the aws-eda-slurm-cluster config file which will add it to the ParallelCluster config file so that ParallelCluster can manage them for you. When you are done create a new AMI and wait for the AMI to become available. After it is available you can add the custom ami to the aws-eda-slurm-cluster config file. slurm: ParallelClusterConfig: ComputeNodeAmi: ami-0fdb972bda05d2932 Then update your aws-eda-slurm-cluster stack by running the install script again.","title":"Customize the compute node AMI"},{"location":"deploy-parallel-cluster/#run-your-first-job","text":"Run the following command in a shell to configure your environment to use your slurm cluster. module load {{ClusterName}} To submit a job run the following command. sbatch /opt/slurm/$SLURM_CLUSTER_NAME/test/job_simple_array.sh To check the status run the following command. squeue To open an interactive shell on a slurm node. srun --pty /bin/bash","title":"Run Your First Job"},{"location":"deploy-parallel-cluster/#slurm-documentation","text":"https://slurm.schedmd.com","title":"Slurm Documentation"},{"location":"deployment-prerequisites/","text":"Deployment Prerequisites This page shows common prerequisites that need to be done before deployment. Deployment Server/Instance Requirements The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance. Configure AWS CLI Credentials You will needs AWS credentials that provide admin access to deploy the cluster. Clone or Download the Repository Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git Create SNS Topic for Error Notifications (Optional but recommended) The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket. Make sure using at least python version 3.7 This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5 Make sure required packages are installed cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment. Install Cloud Development Kit (CDK) (Optional) The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed. Create Configuration File Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None slurm/SubmitterSecurityGroupIds Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. sg-* None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml Configure the Compute Instances The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 Rocky 8 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1 Configure Fair Share Scheduling (Optional) Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities. Configure Licenses Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-prerequisites","text":"This page shows common prerequisites that need to be done before deployment.","title":"Deployment Prerequisites"},{"location":"deployment-prerequisites/#deployment-serverinstance-requirements","text":"The deployment process was developed and tested using Amazon Linux 2. It has also been tested on RHEL 8 and RHEL 9. An easy way to create a deployment instance is to use an AWS Cloud 9 desktop. This will give you a code editor IDE and shell environment that you can use to deploy the cluster. If the required packages aren't installed then you will need sudo or root access on the instance.","title":"Deployment Server/Instance Requirements"},{"location":"deployment-prerequisites/#configure-aws-cli-credentials","text":"You will needs AWS credentials that provide admin access to deploy the cluster.","title":"Configure AWS CLI Credentials"},{"location":"deployment-prerequisites/#clone-or-download-the-repository","text":"Clone or download the aws-eda-slurm-cluster repository to your system. git clone git@github.com:aws-samples/aws-eda-slurm-cluster.git","title":"Clone or Download the Repository"},{"location":"deployment-prerequisites/#create-sns-topic-for-error-notifications-optional-but-recommended","text":"The Slurm cluster allows you to specify an SNS notification that will be notified when an error is detected. You can provide the ARN for the topic in the config file or on the command line. You can use the SNS notification in various ways. The simplest is to subscribe your email address to the topic so that you get an email when there is an error. You could also use it to trigger a CloudWatch alarm that could be used to trigger a lambda to do automatic remediation or create a support ticket.","title":"Create SNS Topic for Error Notifications (Optional but recommended)"},{"location":"deployment-prerequisites/#make-sure-using-at-least-python-version-37","text":"This application requires at least python version 3.7. Many distributions use older versions of python by default such as python 3.6.8 in RHEL 8 and Rocky Linux 8. Newer versions are available, but can't be made the system default without breaking OS tools such as yum. The easiest way to get around this is to create a python virtual environment using a newer version of python. Simply install the newer version and then use it to create and activate a virtual environment. $ python3 --version Python 3.6.8 $ yum -y install python3.11 $ python3.11 -m venv ~/.venv-python3.11 $ source ~/.venv-python3.11/bin/activate $ python3 --version Python 3.11.5","title":"Make sure using at least python version 3.7"},{"location":"deployment-prerequisites/#make-sure-required-packages-are-installed","text":"cd aws-eda-slurm-cluster source setup.sh The setup script assumes that you have sudo access so that you can install or update packages. If you do not, then contact an administrator to help you do the updates. If necessary modify the setup script for your environment.","title":"Make sure required packages are installed"},{"location":"deployment-prerequisites/#install-cloud-development-kit-cdk-optional","text":"The setup script will attempt to install all of the prerequisites for you. If the install script fails on your system then you can refer to this section for instructions on how to install or update CDK. This cluster uses Cloud Development Kit (CDK) and Python 3 to deploy the cluster. Install the packages used by the installer. sudo yum -y install curl gcc-c++ make nfs-utils python3 tcl unzip wget The following link documents how to setup for CDK. Follow the instructions for Python. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites Note that CDK requires a pretty new version of nodejs which you may have to download from, for example, https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz sudo yum -y install wget wget https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz tar -xf node-v16.13.1-linux-x64.tar.xz ~ Add the nodjs bin directory to your path. https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install Note that the version of aws-cdk changes frequently. The version that has been tested is in the CDK_VERSION variable in the install script. The install script will try to install the prerequisites if they aren't already installed.","title":"Install Cloud Development Kit (CDK) (Optional)"},{"location":"deployment-prerequisites/#create-configuration-file","text":"Before you deploy a cluster you need to create a configuration file. A default configuration file is found in source/resources/config/default_config.yml . You should create a new config file and update the parameters for your cluster. Ideally you should version control this file so you can keep track of changes. The schema for the config file along with its default values can be found in source/cdk/config_schema.py . The schema is defined in python, but the actual config file should be in yaml format. The following are key parameters that you will need to update. If you do not have the required parameters in your config file then the installer script will fail unless you specify the --prompt option. You should save your selections in the config file. Parameter Description Valid Values Default StackName The cloudformation stack that will deploy the cluster. None slurm/ClusterName Name of the Slurm cluster For ParallelCluster shouldn't be the same as StackName Region Region where VPC is located $AWS_DEFAULT_REGION VpcId The vpc where the cluster will be deployed. vpc-* None SshKeyPair EC2 Keypair to use for instances None slurm/SubmitterSecurityGroupIds Existing security groups that can submit to the cluster. For SOCA this is the ComputeNodeSG* resource. sg-* None ErrorSnsTopicArn ARN of an SNS topic that will be notified of errors arn:aws:sns:{{region}}:{AccountId}:{TopicName} None slurm/InstanceConfig Configure instance types that the cluster can use and number of nodes. See default_config.yml","title":"Create Configuration File"},{"location":"deployment-prerequisites/#configure-the-compute-instances","text":"The slurm/InstanceConfig configuration parameter configures the base operating systems, CPU architectures, instance families, and instance types that the Slurm cluster should support. ParallelCluster currently doesn't support heterogeneous clusters; all nodes must have the same architecture and Base OS. Base OS CPU Architectures Amazon Linux 2 x86_64, arm64 CentOS 7 x86_64 RedHat 7 x86_64 RedHat 8 x86_64, arm64 Rocky 8 x86_64, arm64 You can exclude instances types by family or specific instance type. By default the InstanceConfig excludes older generation instance families. You can include instances by family or specific instance type. If no includes are specified then all non-excluded instance types will be used. You can also choose to only include the largest instance size within a family. The advantage of using the max instance size is that jobs running on the instance have the highest network bandwidth for that family and fewer instances are required to run the same number of jobs. This may help jobs run faster and allow jobs to wait less time for a new instance to start. The disadvantage is higher cost if the instance is lightly loaded. The default InstanceConfig includes all supported base OSes and architectures and burstable and general purpose instance types. default instance families default instance types default excluded instance families default excluded instance types Note that instance types and families are python regular expressions. slurm: InstanceConfig: Include: InstanceFamilies: - t3.* - m6a.* InstanceTypes: - r6a.large The following InstanceConfig configures instance types recommended for EDA workloads running on CentOS. slurm: InstanceConfig: Include: InstanceFamilies: - c5.* - c6g.* - f1 - m5.* - m6g.* - r5.* - r6g.* - x2gd - z1d If you have reserved instances (RIs) or savings plans then you can configure instances so that they are always on since you are paying for them whether they are running or not. To do this add a MinCount greater than 0 for the compute resources that contain the instance types. slurm: InstanceConfig: NodeCounts: DefaultMinCount: 1","title":"Configure the Compute Instances"},{"location":"deployment-prerequisites/#configure-fair-share-scheduling-optional","text":"Slurm supports fair share scheduling , but it requires the fair share policy to be configured. By default, all users will be put into a default group that has a low fair share. The configuration file is at source/resources/playbooks/roles/ParallelClusterHeadNode/files/opt/slurm/config/accounts.yml.example in the repository and is deployed to /opt/slurm/{{ClusterName}}/conf/accounts.yml . The file is a simple yaml file that allows you to configure groups, the users that belong to the group, and a fair share weight for the group. Refer to the Slurm documentation for details on how the fair share weight is calculated. The scheduler can be configured so that users who aren't getting their fair share of resources get higher priority. The following shows 3 top level groups. Note that the fairshare weights aren't a percentage. They are just a relative weight. In this example, the projects have 9 times higher weight than the jenkins group. jenkins: fairshare: 10 users: - jenkins project1: fairshare: 90 project2: fairshare: 90 The allocation of top level groups can be further subdivided to control the relative priority of jobs within that group. For example, a project may have design verification (dv), rtl design (rtl), physical design (dv), and formal verification (fv) teams. The following example shows how the project's allocation can be prioritized for the different teams. If a group is using more than it's fair share then its jobs will have lower priority than jobs whose users aren't getting their fair share. project1-dv: parent: project1 fairshare: 80 users: - dvuser1 project1-pd: parent: project1 fairshare: 10 users: - pduser1 project1-rtl: parent: project1 fairshare: 10 users: - rtluser1 project1-fv: parent: project1 fairshare: 10 users: - fvuser1 The scheduler uses the priority/multifactor plugin to calculate job priorities. Fair share is just one of the factors. Read the Multifactor Priority Plugin documentation for the details. This is the default configuration in slurm.conf. The partition weight is set the highest so that jobs in the interactive partition always have the highest priority. Fairshare and QOS are the next highest weighted factors. The next factor is the job age, which means all else being equal the jobs run in FIFO order with the jobs that have been waiting the longest getting higher priority. PriorityType=priority/multifactor PriorityWeightPartition=100000 PriorityWeightFairshare=10000 PriorityWeightQOS=10000 PriorityWeightAge=1000 PriorityWeightAssoc=0 PriorityWeightJobSize=0 These weights can be adjusted based on your needs to control job priorities.","title":"Configure Fair Share Scheduling (Optional)"},{"location":"deployment-prerequisites/#configure-licenses","text":"Slurm supports configuring licenses as a consumable resource . It will keep track of how many running jobs are using a license and when no more licenses are available then jobs will stay pending in the queue until a job completes and frees up a license. Combined with the fairshare algorithm, this can prevent users from monopolizing licenses and preventing others from being able to run their jobs. Licenses are configured using the slurm/Licenses configuration variable. If you are using the Slurm database then these will be configured in the database. Otherwises they will be configured in /opt/slurm/{{ClusterName}}/etc/slurm_licenses.conf . The example configuration shows how the number of licenses can be configured. In this example, the cluster will manage 800 vcs licenses and 1 ansys license. Users must request a license using the -L or --licenses options. slurm: Licenses: vcs: Count: 800 ansys: Count: 1","title":"Configure Licenses"},{"location":"federation/","text":"Federation (legacy) To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"federation/#federation-legacy","text":"To maximize performance, EDA workloads should run in a single AZ. If you need to run jobs in more than one AZ then you can use the federation feature of Slurm so that you can run jobs on multiple clusters. The config directory has example configuration files that demonstrate how deploy federated cluster into 3 AZs. source/config/slurm_eda_az1.yml source/config/slurm_eda_az2.yml source/config/slurm_eda_az3.yml These clusters should be deployed sequentially. The first cluster creates a cluster and a slurmdbd instance. The other 2 clusters are deployed into their own AZ by configuring the SubnetId of the cluster. They reuse the same slurmdbd instance so that they can reuse a common pool of licenses that is managed by the slurmdbd instance. The config files for the 2nd and 3rd clusters provide the stack names from the others so that the security groups can be updated to allow the required network traffic between the clusters. The following shows an example of the configuration. slurm_eda_az1: Federation: Name: slurmeda FederatedClusterStackNames: [] slurm_eda_az2: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 slurm_eda_az3: Federation: Name: slurmeda FederatedClusterStackNames: - slurmedaaz1 - slurmedaaz2","title":"Federation (legacy)"},{"location":"implementation/","text":"Implementation Details (legacy) Slurm Infrastructure All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons. Directory Structure All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}} CloudWatch Metrics CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py Down Node Handling If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer Insufficient Capacity Exception (ICE) Handling When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Implementation Details (legacy)"},{"location":"implementation/#implementation-details-legacy","text":"","title":"Implementation Details (legacy)"},{"location":"implementation/#slurm-infrastructure","text":"All hosts in the cluster must share a uniform user and group namespace. The munged service must be running before starting any slurm daemons.","title":"Slurm Infrastructure"},{"location":"implementation/#directory-structure","text":"All of the configuration files, scripts, and logs can be found under the following directory. /opt/slurm/{{ClusterName}}","title":"Directory Structure"},{"location":"implementation/#cloudwatch-metrics","text":"CloudWatch metrics are published by the following sources, but the code is all in SlurmPlugin.py . Slurm power saving scripts /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume_fail.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_stop.py /opt/slurm/{{ClusterName}}/bin/slurm_ec2_terminate.py Spot monitor running on compute nodes /opt/slurm/{{ClusterName}}/bin/spot_monitor.py Cron jobs running on the Slurm controller /opt/slurm/{{ClusterName}}/bin/slurm_ec2_publish_cw.py /opt/slurm/{{ClusterName}}/bin/terminate_old_instances.py","title":"CloudWatch Metrics"},{"location":"implementation/#down-node-handling","text":"If a node has a problem running jobs then Slurm can mark it DOWN. This includes if the resume script cannot start an instance for any reason include insufficient EC2 capacity. This can create 2 issues. First, if the compute node is running then it is wasting EC2 costs. Second, the node will be unavailable for scheduling which reduces the configured capacity of the cluster. The cluster is configured to periodically check for DOWN nodes so that they aren't left running and wasting compute costs. This is done by /opt/slurm/{{ClusterName}}/bin/slurm_down_nodes_clean.sh . The script is called every day by a systemd service: /etc/systemd/system/slurm_down_nodes_clean.service This service is run at boot and once a day as defined in /etc/systemd/system/slurm_down_nodes_clean.timer","title":"Down Node Handling"},{"location":"implementation/#insufficient-capacity-exception-ice-handling","text":"When Slurm schedules a powered down node it calls the ResumeScript defined in slurm.conf . This is in /opt/slurm/{{ClusterName}}/bin/slurm_ec2_resume.py . The script will attempt to start an EC2 instance and if it receives and InsufficientCapacityException (ICE) then the node will be marked down and Slurm will requeue the job. However, this is inadequate because if there are a large number of instances of that instance type configured then Slurm will schedule them and try to start them with the same result. Eventually all of the powered down nodes will be marked DOWN and depending on the job requirements the job will be allocated to a node with a different instance type or it will fail. This can take a substantial amount of time so SlurmPlugin.py does the following when it receives an ICE. Mark the node as DRAIN so no new jobs are scheduled on it. Find all other powered down nodes of the same type and mark them DOWN so that they won't be scheduled after this node is marked DOWN. Nodes that are running will be left alone. Requeue jobs on the node that failed to resume because of ICE. Mark the node DOWN. Power down the node. This is so that Slurm knows that the node is powered down so that when it is marked IDLE it will be powered up when a job is scheduled on it. The slurm_down_nodes_clean.service periodically finds all DOWN Slurm nodes, powers them down, and then marks them IDLE so that they can have jobs scheduled on them. This will allow Slurm to attempt to use more nodes of the instance type in the hopes that there is more capacity. If not, then the cycle repeats.","title":"Insufficient Capacity Exception (ICE) Handling"},{"location":"job_preemption/","text":"Job Preemption The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license. Documentation https://slurm.schedmd.com/preempt.html","title":"Job Preemption"},{"location":"job_preemption/#job-preemption","text":"The cluster is set up with an interactive partition that has a higher priority than all other partitions. All other partitions are configured to allow jobs to be preempted by the interactive queue. When an interactive job is pending because of compute resources then it can preempt another job and use the resources. The preempted job will be requeued so that it will rerun when resources become available. Jobs should rarely pend because of lack of compute resources if you've defined enough compute nodes in your configuration. The more likely reason for a job to pend is if it requires a license and all available licenses are already being used. However, it appears that Slurm doesn't support preemption based on licenses availability so if the reason a job is pending is because of licenses then it will not preempt jobs in a lower priority queue even if doing so would free up a license.","title":"Job Preemption"},{"location":"job_preemption/#documentation","text":"https://slurm.schedmd.com/preempt.html","title":"Documentation"},{"location":"onprem/","text":"On-Premises Integration The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information. Network Requirements The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes. DNS Requirements Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC. File System Requirements All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm. Slurm Configuration of On-Premises Compute Nodes The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem Simulating an On-Premises Network Using AWS Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"On-Premises Integration"},{"location":"onprem/#on-premises-integration","text":"The Slurm cluster can also be configured to manage on-premises compute nodes. The user must configure the on-premises compute nodes and then give the configuration information.","title":"On-Premises Integration"},{"location":"onprem/#network-requirements","text":"The on-prem network must have a CIDR range that doesn't overlap the Slurm cluster's VPC and the two networks need to be connected using VPN or AWS Direct Connect. The on-prem firewall must allow ingress and egress from the VPC. The ports are used to connect to the file systems, slurm controllers, and allow traffic between virtual desktops and compute nodes.","title":"Network Requirements"},{"location":"onprem/#dns-requirements","text":"Local network DNS must have an entry for the slurm controller or have a forwarding rule to the AWS provided DNS in the Slurm VPC.","title":"DNS Requirements"},{"location":"onprem/#file-system-requirements","text":"All of the compute nodes in the cluster, including the on-prem nodes, must have file system mounts that replicate the same directory structure. This can involve mounting filesystems across VPN or Direct Connect or synchronizing file systems using tools like rsync or NetApp FlexCache or SnapMirror. Performance will dictate the architecture of the file system. The onprem compute nodes must mount the Slurm controller's NFS export so that they have access to the Slurm binaries and configuration file. They must then be configured to run slurmd so that they can be managed by Slurm.","title":"File System Requirements"},{"location":"onprem/#slurm-configuration-of-on-premises-compute-nodes","text":"The slurm cluster's configuration file allows the configuration of on-premises compute nodes. The Slurm cluster will not provision any of the on-prem nodes, network, or firewall, but it will configure the cluster's resources to be used by the on-prem nodes. All that needs to be configured are the configuration file for the on-prem nodes and the CIDR block. InstanceConfig: OnPremComputeNodes: ConfigFile: 'slurm_nodes_on_prem.conf' CIDR: '10.1.0.0/16' slurm_nodes_on_prem.conf # # ON PREMISES COMPUTE NODES # # Config file with list of statically provisioned on-premises compute nodes that # are managed by this cluster. # # These nodes must be addressable on the network and firewalls must allow access on all ports # required by slurm. # # The compute nodes must have mounts that mirror the compute cluster including mounting the slurm file system # or a mirror of it. NodeName=Default State=DOWN NodeName=onprem-c7-x86-t3-2xl-0 NodeAddr=onprem-c7-x86-t3-2xl-0.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-1 NodeAddr=onprem-c7-x86-t3-2xl-1.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-2 NodeAddr=onprem-c7-x86-t3-2xl-2.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-3 NodeAddr=onprem-c7-x86-t3-2xl-3.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-4 NodeAddr=onprem-c7-x86-t3-2xl-4.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-5 NodeAddr=onprem-c7-x86-t3-2xl-5.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-6 NodeAddr=onprem-c7-x86-t3-2xl-6.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-7 NodeAddr=onprem-c7-x86-t3-2xl-7.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-8 NodeAddr=onprem-c7-x86-t3-2xl-8.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 NodeName=onprem-c7-x86-t3-2xl-9 NodeAddr=onprem-c7-x86-t3-2xl-9.onprem.com CPUs=4 RealMemory=30512 Feature=c7,CentOS_7_x86_64,x86_64,GHz:2.5 Weight=1 # # # OnPrem Partition # # The is the default partition and includes all nodes from the 1st OS. # PartitionName=onprem Default=YES PriorityTier=20000 Nodes=\\ onprem-c7-x86-t3-2xl-[0-9] # # Always on partitions # SuspendExcParts=onprem","title":"Slurm Configuration of On-Premises Compute Nodes"},{"location":"onprem/#simulating-an-on-premises-network-using-aws","text":"Create a new VPC with public and private subnets and NAT gateways. To simulate the latency between an AWS region and on-prem you can create the VPC in a different region in your account. The CIDR must not overlap with the Slurm VPC. Create a VPC peering connection to your Slurm VPC and accept the connection in the Slurm VPC. Create routes in the private subnets for the CIDR of the peered VPC and route it to the vpc peering connection. Add the on-prem VPC to the Slurm VPC's Route53 private local zone. Create a Route53 private hosted zone for the on-prem compute nodes and add it to the onprem VPC and the slurm VPC so that onprem compute nodes can be resolved. Copy the Slurm AMIs to the region of the on-prem VPC. Create an instance using the copied AMI. Connect to the instance and confirm that the mount points mounted correctly. You will probably have to change the DNS names for the file systems to IP addresses. I created A records in the Route53 zone for the file systems so that if the IP addresses ever change in the future I can easily update them in one place without having to create a new AMI or updated any instances. Create a new AMI from the instance. Create compute node instances from the new AMI and run the following commands on them get the slurmd daemon running so they can join the slurm cluster. # Instance specific variables hostname=onprem-c7-x86-t3-2xl-0 # Domain specific variables onprem_domain=onprem.com source /etc/profile.d/instance_vars.sh # munge needs to be running before calling scontrol /usr/bin/cp /opt/slurm/$ClusterName/config/munge.key /etc/munge/munge.key systemctl enable munged systemctl start munged ipaddress=$(hostname -I) $SLURM_ROOT/bin/scontrol update nodename=${hostname} nodeaddr=$ipaddress # Set hostname hostname_fqdn=${hostname}.${onprem_domain} if [ $(hostname) != $hostname_fqdn ]; then hostnamectl --static set-hostname $hostname_fqdn hostnamectl --pretty set-hostname $hostname fi if [ -e /opt/slurm/${ClusterName}/config/users_groups.json ] && [ -e /opt/slurm/${ClusterName}/bin/create_users_groups.py ]; then /opt/slurm/${ClusterName}/bin/create_users_groups.py -i /opt/slurm/${ClusterName}/config/users_groups.json fi # Create directory for slurmd.log logs_dir=/opt/slurm/${ClusterName}/logs/nodes/${hostname} if [[ ! -d $logs_dir ]]; then mkdir -p $logs_dir fi if [[ -e /var/log/slurm ]]; then rm -rf /var/log/slurm fi ln -s $logs_dir /var/log/slurm systemctl enable slurmd systemctl start slurmd # Restart so that log file goes to file system systemctl restart spot_monitor","title":"Simulating an On-Premises Network Using AWS"},{"location":"res_integration/","text":"RES Integration Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx SubmitterSecurityGroupIds The security group names and ids used by RES VDIs. The name will be something like EnvironmentName -vdc-dcv-host-security-group EnvironmentName - VDISG : sg-xxxxxxxx SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"res_integration/#res-integration","text":"Integration with Research and Engineering Studion (RES) is straightforward. You simply specify the --RESEnvironmentName option for the install.sh script or add the RESEnvironmentName configuration parameter to your configuration file. The install script will set the following configuration parameters based on your RES environment or check them if you have them set to make sure they are consistent with your RES environment. The intention is to completely automate the deployment of ParallelCluster and set up the RES environment so that it can easily be used. Parameter Description Value VpcId VPC id for the RES cluster vpc-xxxxxx SubnetId Subnet in the RES VPC. subnet-xxxxx SubmitterSecurityGroupIds The security group names and ids used by RES VDIs. The name will be something like EnvironmentName -vdc-dcv-host-security-group EnvironmentName - VDISG : sg-xxxxxxxx SubmitterInstanceTags The tag of VDI instances. 'res:EnvironmentName': EnvironmentName ' ExtraMounts The mount parameters for the /home directory. This is required for access to the home directory. ExtraMountSecurityGroups Security groups that give access to the ExtraMounts. These will be added to compute nodes so they can access the file systems. When you specify RESEnvironmentName , a lambda function will run SSM commands to create a cron job on a RES domain joined instance to update the users_groups.json file every hour. Another lambda function will also automatically configure all running VDI hosts to use the cluster. The following example shows the configuration parameters for a RES with the EnvironmentName=res-eda. --- #==================================================================== # EDA Slurm cluster for RES using ParallelCluster # # Defaults and valid configuration options are in source/config_schema.py. # Command line values override values in the config file. #==================================================================== StackName: res-eda-pc-3-8-0-rhel8-x86-config Region: SshKeyPair: RESEnvironmentName: res-eda ErrorSnsTopicArn: TimeZone: 'US/Central' slurm: ClusterName: res-eda-pc-3-8-0-rhel8-x86 ParallelClusterConfig: Version: '3.8.0' Image: Os: 'rhel8' Architecture: 'x86_64' Database: DatabaseStackName: pcluster-slurm-db-res SlurmCtl: {} # Configure typical EDA instance types # A partition will be created for each combination of Base OS, Architecture, and Spot InstanceConfig: UseSpot: true NodeCounts: DefaultMaxCount: 10 When the cluster deployment finishes you are ready to run jobs from your RES DCV desktop.","title":"RES Integration"},{"location":"rest_api/","text":"Slurm REST API The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster. How to use the REST API The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"Slurm REST API"},{"location":"rest_api/#slurm-rest-api","text":"The Slurm REST API give a programmatic way to access the features of Slurm. The REST API can be used, for example, to use a Lambda function to submit jobs to the Slurm cluster.","title":"Slurm REST API"},{"location":"rest_api/#how-to-use-the-rest-api","text":"The following shows how to run a simple REST call. source /opt/slurm/{{ClusterName}}/config/slurm_config.sh unset SLURM_JWT . <(scontrol token) wget --header \"X-SLURM-USER-TOKEN: $SLURM_JWT\" --header \"X-SLURM-USER-NAME: $USER\" -q $SLURMRESTD_URL/slurm/v0.0.38/diag/ -O - The REST API is documented at https://slurm.schedmd.com/rest_api.html . The token returned by scontrol token has a default lifetime of 3600 seconds (1 hour). For automation, a cron job on the Slurm controller creates a new token for the root and slurmrestd users every 30 minutes and stores them in SSM Parameter Store at /{{ClusterName}}/slurmrestd/jwt/{{user_name}} . These tokens can be used by automations such as a Lambda function to access the REST API. An example Lambda function called {{ClusterName}}-CallSlurmRestApiLambda shows how to call various API functions. You can use this as a template to write functions that use your Slurm cluster for automations.","title":"How to use the REST API"},{"location":"run_jobs/","text":"Run Jobs This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages. Set Up Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format. Key Slurm Commands The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state sbatch The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script. Run a simulation build followed by a regression build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi srun The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs squeue The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue sprio Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11 sacct Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct sreport The sreport command can be used to generate report from the Slurm database. Other Slurm Commands Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Run Jobs"},{"location":"run_jobs/#run-jobs","text":"This page is to give some basic instructions on how to run and monitor jobs on Slurm. Slurm provides excellent man pages for all of its commands, so if you have questions refer to the man pages.","title":"Run Jobs"},{"location":"run_jobs/#set-up","text":"Load the environment module for Slurm to configure your PATH and Slurm related environment variables. module load {{ClusterName}} The modulefile sets environment variables that control the defaults for Slurm commands. These are documented in the man pages for each command. If you don't like the defaults then you can set them in your environment (for example, your .bashrc) and the modulefile won't change any variables that are already set. The environment variables can always be overridden by the command line options. For example, the SQUEUE_FORMAT2 and SQUEUE_SORT environment variables are set so that the default output format is easier to read and contains useful information that isn't in the default format.","title":"Set Up"},{"location":"run_jobs/#key-slurm-commands","text":"The key Slurm commands are Command Description Example salloc Create a compute allocation. salloc -c 1 --mem 1G -C 'spot&GHz:3.1' srun Run a job within an allocation. srun --pty bin/bash sbatch Submit a batch script sbatch -c 1 --mem 1G -C 'spot&GHz:3.1' script squeue Get job status scancel Cancel a job scancel jobid sinfo Get info about Slurm node status sinfo -p all scontrol view or modify Slurm configuration and state scontrol show node nodename sstat Display various status information about a running job/step sshare Tool for listing fair share information sprio View the factors that comprise a job's scheduling priority sacct Display accounting data for jobs sreport Generate reports from the Slurm accounting data. sview Graphical tool for viewing cluster state","title":"Key Slurm Commands"},{"location":"run_jobs/#sbatch","text":"The most common options for sbatch are listed here. For more details run man sbatch . Options Description Default -p, --partition= partition-names Select the partition/partitions to run job on. Set by slurm.InstanceConfig.DefaultPartition in config file. -t, --time= time Set a limit on total run time of the job. SBATCH_TIMELIMIT=\"1:0:0\" (1 hour) -c, --cpus-per-task= ncpus Number of cores. Default is 1. --mem= size[units] Amount of memory. Default unit is M. Valid units are [K|M|G|T]. SBATCH_MEM_PER_NODE=100M -L, --licenses= license Licenses used by the job. -a, --array= indexes Submit job array -C, --constraint= list Features required by the job. Multiple constraints can be specified with AND(&) and OR( ). -d, --dependency= dependency-list Don't start the job until the dependencies have been completed. -D, --chdir= directory Set the working directory of the job --wait Do not exit until the job finishes, Exit code of sbatch will be the same as the exit code of the job. --wrap Wrap shell commands in a batch script.","title":"sbatch"},{"location":"run_jobs/#run-a-simulation-build-followed-by-a-regression","text":"build_jobid=$(sbatch -c 4 --mem 4G -L vcs_build -C 'GHz:4|GHz:4.5' -t 30:0 sim-build.sh) if sbatch -d \"afterok:$build_jobid\" -c 1 --mem 100M --wait submit-regression.sh; then echo \"Regression Passed\" else echo \"Regression Failed\" fi","title":"Run a simulation build followed by a regression"},{"location":"run_jobs/#srun","text":"The srun is usually used to open a pseudo terminal on a compute node for you to run interactive jobs. It accepts most of the same options as sbatch to request cpus, memory, and node features. To open up a pseudo terminal in your shell on a compute node with 4 cores and 16G of memory, execute the following command. srun -c 4 --mem 8G --pty /bin/bash This will queue a job and when it is allocated to a node and the node runs, the job control will be returned to your shell, but stdin and stdout will be on the compute node. If you set your DISPLAY environment variable and allow external X11 connections you can use this to run interactive GUI jobs on the compute node and have the windows on your instance. xhost + export DISPLAY=$(hostname):$(echo $DISPLAY | cut -d ':' -f 2) srun -c 4 --mem 8G --pty /bin/bash emacs . # Or whatever gui application you want to run. Should open a window. Another way to run interactive GUI jobs is to use srun 's --x11 flag to enable X11 forwarding. srun -c 1 --mem 8G --pty --x11 emacs","title":"srun"},{"location":"run_jobs/#squeue","text":"The squeue command shows the status of jobs. The output format can be customized using the --format or --Format options and you can configure the default output format using the corresponding SQUEUE_FORMAT or SQUEUE_FORMAT2 environment variables. squeue","title":"squeue"},{"location":"run_jobs/#sprio","text":"Use sprio to get information about a job's priority. This can be useful to figure out why a job is scheduled before or after another job. sprio -j10,11","title":"sprio"},{"location":"run_jobs/#sacct","text":"Display accounting information about jobs. For example, it can be used to get the requested CPU and memory and see the CPU time and memory actually used. sacct -o JobID,User,JobName,AllocCPUS,State,ExitCode,Elapsed,CPUTime,MaxRSS,MaxVMSize,ReqCPUS,ReqMem,SystemCPU,TotalCPU,UserCPU -j 44 This shows more details. sacct --allclusters --allusers --federation --starttime 1970-01-01 --format 'Submit,Start,End,jobid%15,State%15,user,account,cluster%15,AllocCPUS,AllocNodes,ExitCode,ReqMem,MaxRSS,MaxVMSize,MaxPages,Elapsed,CPUTime,UserCPU,SystemCPU,TotalCPU' | less For more information: man sacct","title":"sacct"},{"location":"run_jobs/#sreport","text":"The sreport command can be used to generate report from the Slurm database.","title":"sreport"},{"location":"run_jobs/#other-slurm-commands","text":"Use man command to get information about these less commonly used Slurm commands. Command Description sacctmgr View/modify Slurm account information sattach Attach to a job step sbcast Transmit a file to the nodes allocated to a Slurm job. scrontab Manage slurm crontab files sdiag Diagnostic tool for Slurm. Shows information related to slurmctld execution. seff sgather Transmit a file from the nodes allocated to a Slurm job. sh5util Tool for merging HDF5 files from the acct_gather_profile plugin that gathers detailed data for jobs. sjobexitmod Modify derived exit code of a job strigger Set, get, or clear Slurm trigger information","title":"Other Slurm Commands"},{"location":"soca_integration/","text":"SOCA Integration Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx SubmitterSecurityGroupIds The ComputeNode security group name and id cluster-id - ComputeNodeSG : sg-xxxxxxxx ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"},{"location":"soca_integration/#soca-integration","text":"Scale Out Computing on AWS (SOCA) is an AWS solution that was the basis for the Research and Engineering Studion (RES) service. Unless you are already a SOCA user, it is highly recommended that you use RES, which is a fully supported AWS service. Integration with SOCA is straightforward. Set the following parameters in your config file. Parameter Description Value VpcId VPC id for the SOCA cluster vpc-xxxxxx SubmitterSecurityGroupIds The ComputeNode security group name and id cluster-id - ComputeNodeSG : sg-xxxxxxxx ExtraMounts Add the mount parameters for the /apps and /data directories. This is required for access to the home directory. Deploy your slurm cluster. Connect to the SOCA Scheduler instance and follow the instructions to Create users_groups.json . Connect to a remote desktop instance and follow the instructions in Configure submission hosts to use the cluster . If all users need to use the cluster then it is probably best to create a custom AMI that is configured with the configuration commands. You are now ready to run jobs from your SOCA desktop.","title":"SOCA Integration"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index eb8c4068..7b5777e1 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ