Merge pull request #2333 from tilne/release-2.10.1

v2.10.1: Merge develop into release-2.10

.github/workflows/ci.yml

@@ -0,0 +1,74 @@
+name: ParallelCluster CI
+
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        name:
+          - Python 2.7 Tests
+          - Python 3.4 Tests
+          - Python 3.5 Tests
+          - Python 3.6 Tests
+          - Python 3.7 Tests
+          - Python 3.8 Tests
+          - Python 3.9 Tests
+          - Python 3.8 Tests Coverage
+          - Code Checks
+          - CloudFormation Templates Checks
+        include:
+          - name: Python 2.7 Tests
+            python: 2.7
+            toxdir: cli
+            toxenv: py27-nocov
+          - name: Python 3.4 Tests
+            python: 3.4
+            toxdir: cli
+            toxenv: py34-nocov
+          - name: Python 3.5 Tests
+            python: 3.5
+            toxdir: cli
+            toxenv: py35-nocov
+          - name: Python 3.6 Tests
+            python: 3.6
+            toxdir: cli
+            toxenv: py36-nocov
+          - name: Python 3.7 Tests
+            python: 3.7
+            toxdir: cli
+            toxenv: py37-nocov
+          - name: Python 3.8 Tests
+            python: 3.8
+            toxdir: cli
+            toxenv: py38-nocov
+          - name: Python 3.9 Tests
+            python: 3.9
+            toxdir: cli
+            toxenv: py39-nocov
+          - name: Python 3.8 Tests Coverage
+            python: 3.8
+            toxdir: cli
+            toxenv: py38-cov
+          - name: Code Checks
+            python: 3.6
+            toxdir: cli
+            toxenv: code-linters
+          - name: CloudFormation Templates Checks
+            python: 3.6
+            toxdir: cli
+            toxenv: cfn-format-check,cfn-lint,cfn-tests
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install Tox and any other packages
+        run: pip install tox
+      - name: Run Tox
+        run: cd ${{ matrix.toxdir }} && tox -e ${{ matrix.toxenv }}

CHANGELOG.md

@@ -1,14 +1,67 @@
 CHANGELOG
 =========
 
+2.10.1
+------
+
+**ENHANCEMENTS**
+
+- Add support for me-south-1 region (Bahrein), af-south-1 region (Cape Town) and eu-south-1 region (Milan)
+  - At the time of this version launch:
+    - Amazon FSx for Lustre and ARM instance types are not supported in me-south-1, af-south-1 and eu-south-1
+    - AWS Batch is not supported in af-south-1
+    - EBS io2 is not supported in af-south-1 and eu-south-1
+- Install Arm Performance Libraries (APL) 20.2.1 on ARM AMIs (CentOS8, Alinux2, Ubuntu1804).
+- Install EFA kernel module on ARM instances with `alinux2` and `ubuntu1804`. This enables support for `c6gn` instances.
+- Add support for io2 and gp3 EBS volume type.
+- Add `iam_lambda_role` parameter under `cluster` section to enable the possibility to specify an existing IAM role to 
+  be used by AWS Lambda functions in CloudFormation. 
+  When using `sge`, `torque`, or `slurm` as the scheduler, 
+  `pcluster` will not create any IAM role if both `ec2_iam_role` and `iam_lambda_role` are provided.
+- Improve robustness of a Slurm cluster when clustermgtd is down.
+- Configure NFS threads to be max(8, num_cores) for performance. This enhancement will not take effect on Ubuntu 16.04.
+- Optimize calls to DescribeInstanceTypes EC2 API when validating cluster configuration. 
+
+**CHANGES**
+
+- Upgrade EFA installer to version 1.11.1.
+  - EFA configuration: ``efa-config-1.7`` (from efa-config-1.5)
+  - EFA profile: ``efa-profile-1.3`` (from efa-profile-1.1)
+  - EFA kernel module: ``efa-1.10.2`` (no change)
+  - RDMA core: ``rdma-core-31.2amzn`` (from rdma-core-31.amzn0)
+  - Libfabric: ``libfabric-1.11.1amzn1.0`` (from libfabric-1.11.1amzn1.1)
+  - Open MPI: ``openmpi40-aws-4.1.0`` (from openmpi40-aws-4.0.5)
+- Upgrade Intel MPI to version U8.
+- Upgrade NICE DCV to version 2020.2-9662.
+- Set default systemd runlevel to multi-user.target on all OSes during ParallelCluster official AMI creation.
+  The runlevel is set to graphical.target on head node only when DCV is enabled. This prevents the execution of
+  graphical services, such as x/gdm, when they are not required.
+- Download Intel MPI and HPC packages from S3 rather than Intel yum repos.
+- Change the default of instance types from the hardcoded `t2.micro` to the free tier instance type 
+    (`t2.micro` or `t3.micro` dependent on region). In regions without free tier, the default is `t3.micro`.
+- Enable support for p4d as head node instance type (p4d was already supported as compute node in 2.10.0).
+- Pull Amazon Linux Docker images from public ECR when building docker image for `awsbatch` scheduler.
+- Increase max retry attempts when registering Slurm nodes in Route53.
+
+**BUG FIXES**
+
+- Fix pcluster createami for Ubuntu 1804 by downloading SGE sources from Debian repository and not from the EOL
+  Ubuntu 19.10.
+- Remove CloudFormation DescribeStacks API call from AWS Batch Docker entrypoint. This removes the risk of job
+  failures due to CloudFormation throttling.
+- Mandate the presence of `vpc_settings`, `vpc_id`, `master_subnet_id` in the config file to avoid unhandled exceptions.
+- Set the default EBS volume size to 500 GiB when volume type is `st1` or `sc1`.
+- Fix installation of Intel PSXE package on CentOS 7 by using yum4.
+- Fix routing issues with multiple Network Interfaces on Ubuntu 18.04.
+
 2.10.0
 ------
 
 **ENHANCEMENTS**
 
 - Add support for CentOS 8 in all Commercial regions.
 - Add support for P4d instance type as compute node.
-- Add the possibilty to enable NVIDIA GPUDirect RDMA support on EFA by using the new `enable_efa_gdr` configuration
+- Add the possibility to enable NVIDIA GPUDirect RDMA support on EFA by using the new `enable_efa_gdr` configuration
   parameter.
 - Enable support for NICE DCV in GovCloud regions.
 - Enable support for AWS Batch scheduler in GovCloud regions.

README.md

@@ -70,16 +70,16 @@ Automate VPC creation? (y/n) [n]:
 
 Enter ``n`` if you already have a VPC suitable for the cluster. Otherwise you can let ``pcluster configure``
 create a VPC for you. The same choice is given for the subnet: you can select a valid subnet ID for
-both the master and compute nodes, or you can let ``pcluster configure`` set up everything for you.
+both the head node and compute nodes, or you can let ``pcluster configure`` set up everything for you.
 The same choice is given for the subnet configuration: you can select a valid subnet ID for both
-the master and compute nodes, or you can let pcluster configure set up everything for you.
+the head node and compute nodes, or you can let pcluster configure set up everything for you.
 In the latter case, just select the configuration you prefer.
 
 ```
 Automate Subnet creation? (y/n) [y]: y
 Allowed values for Network Configuration:
-1. Master in a public subnet and compute fleet in a private subnet
-2. Master and compute fleet in the same public subnet
+1. Head node in a public subnet and compute fleet in a private subnet
+2. Head node and compute fleet in the same public subnet
 ```