Make GitLab instance CIS level 2 compliant

[nolunwa-ucsc]

We use AWS Config and AWS Security Hub for automated management and verification of configuration settings.
The Team should configure or verify Security Hub is configured to validate the AWS account against
CIS AWS Foundations Benchmark ( CIS level 2 Benchmark). This is required for NIST SP 800 53 rev 5 Standard.

[dsotirho-ucsc]

@hannes-ucsc: "During the 2023 assessment, we were asked to perform a scan of the GitLab instance against the CIS benchmark. There were a number of level 2 findings that we ignored because r4 of FedRamp only required CIS level 2. The scan was one-shot only and we used and still are using CIS level 1 hardened AMI, which we pay a subscription fee for. Spike to investigate if there is a similar AMI that is hardened against level 2."

[dsotirho-ucsc]

…we used and still are using CIS level 1 hardened AMI, which we pay a subscription fee for. Spike to investigate if there is a similar AMI that is hardened against level 2.

CIS Amazon Linux 2 Benchmark - Level 2
By: Center for Internet Security - Latest Version: 2.0.0.29
$0.045/hr

Other CIS hardened images can be found here: https://www.cisecurity.org/cis-hardened-images/amazon

[dsotirho-ucsc]

Spike to test this image in anvildev.gitlab and confirm that all containers start up and that timer units are functional.

[achave11-ucsc]

Assignee to summarize spike results.

[dsotirho-ucsc]

Assignee to summarize spike results.

Attempting to deploy the CIS Amazon Linux 2 Benchmark v2.0.0.29 AMI failed with an error attempting to connect the GitLab data volume to mount point /dev/sdf.
Upon creation of the new instance, two volumes are created instead of just one as with the previous AMI. The two volumes are a 20GB root volume and an 40GB volume of unknown purpose.

Subject: [PATCH] Make GitLab instance CIS level 2 compliant (#5990)
---
Index: terraform/gitlab/gitlab.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
--- a/terraform/gitlab/gitlab.tf.json.template.py	(revision 2639f5bd9f768879e5623113b19f694d86c99686)
+++ b/terraform/gitlab/gitlab.tf.json.template.py	(revision e83203fdc3055fea2b669d8ee709c079f43475ea)
@@ -239,10 +239,10 @@
 # For instructions on finding the latest CIS-hardened AMI, see
 # OPERATOR.rst#upgrading-linux-ami
 #
-# CIS Amazon Linux 2 Kernel 4.14 Benchmark v2.0.0.29 - Level 1-4c096026-c6b0-440c-bd2f-6d34904e4fc6
+# CIS Amazon Linux 2 Benchmark v2.0.0.29 - Level 2-c41d38c4-3f6a-4434-9a86-06dd331d3f9c
 #
 ami_id = {
-    'us-east-1': 'ami-02adfaf34663c8edb'
+    'us-east-1': 'ami-044f22d25234fc98b'
 }
 
 gitlab_mount = '/mnt/gitlab'

aws_volume_attachment.gitlab: Still destroying... [id=vai-3140407683, 1m40s elapsed]
aws_volume_attachment.gitlab: Destruction complete after 1m45s
aws_instance.gitlab: Destroying... [id=i-00ebb81df55939cd8]
aws_instance.gitlab: Still destroying... [id=i-00ebb81df55939cd8, 10s elapsed]
aws_instance.gitlab: Destruction complete after 11s
aws_instance.gitlab: Creating...
aws_instance.gitlab: Still creating... [10s elapsed]
aws_instance.gitlab: Creation complete after 14s [id=i-08cd638ab8733ae08]
aws_volume_attachment.gitlab: Creating...
aws_lb_target_group_attachment.gitlab_git: Creating...
aws_lb_target_group_attachment.gitlab_http: Creating...
aws_lb_target_group_attachment.gitlab_ssh: Creating...
aws_lb_target_group_attachment.gitlab_ssh: Creation complete after 0s [id=arn:aws:elasticloadbalancing:us-east-1:289950828509:targetgroup/azul-gitlab-ssh/d2912ab83eb390e7-20240318214800354000000002]
aws_lb_target_group_attachment.gitlab_git: Creation complete after 0s [id=arn:aws:elasticloadbalancing:us-east-1:289950828509:targetgroup/azul-gitlab-git/385495552953ed11-20240318214800426900000003]
aws_lb_target_group_attachment.gitlab_http: Creation complete after 0s [id=arn:aws:elasticloadbalancing:us-east-1:289950828509:targetgroup/azul-gitlab-http/a8a428c48fbd0f85-20240318214800450600000004]
aws_volume_attachment.gitlab: Still creating... [10s elapsed]
╷
│ Error: attaching EBS Volume (vol-0cd71b694fc4dc181) to EC2 Instance (i-08cd638ab8733ae08): InvalidParameterValue: Invalid value '/dev/sdf' for unixDevice. Attachment point /dev/sdf is already in use
│ 	status code: 400, request id: c765e70a-4817-41a3-b22c-c9826c1cfb79
│
│   with aws_volume_attachment.gitlab,
│   on gitlab.tf.json line 1663, in resource[24].aws_volume_attachment[0].gitlab:
│ 1663:                     }
│
╵
make: *** [apply] Error 1

(.venv) daniel@Crispin ~/repo/azul2 $ aws ec2 describe-volumes
{
    "Volumes": [
        {
            ## The 20GB GitLab root volume
            "Attachments": [
                {
                    "AttachTime": "2024-03-18T21:47:47.000Z",
                    "Device": "/dev/xvda",
                    "InstanceId": "i-08cd638ab8733ae08",
                    "State": "attached",
                    "VolumeId": "vol-0e6d5842b41f80dba",
                    "DeleteOnTermination": true
                }
            ],
            "AvailabilityZone": "us-east-1a",
            "CreateTime": "2024-03-18T21:47:47.185Z",
            "Encrypted": true,
            "KmsKeyId": "<REDACTED>",
            "Size": 20,
            "SnapshotId": "snap-019804939f4d0ff49",
            "State": "in-use",
            "VolumeId": "vol-0e6d5842b41f80dba",
            "Iops": 100,
            "VolumeType": "gp2",
            "MultiAttachEnabled": false
        },
        {
            ## A unexpected additional 40GB volume attached to /dev/sdf
            "Attachments": [
                {
                    "AttachTime": "2024-03-18T21:47:47.000Z",
                    "Device": "/dev/sdf",
                    "InstanceId": "i-08cd638ab8733ae08",
                    "State": "attached",
                    "VolumeId": "vol-0bc91eba36f4454a9",
                    "DeleteOnTermination": true
                }
            ],
            "AvailabilityZone": "us-east-1a",
            "CreateTime": "2024-03-18T21:47:47.326Z",
            "Encrypted": false,
            "Size": 40,
            "SnapshotId": "snap-04dac21bdd3172661",
            "State": "in-use",
            "VolumeId": "vol-0bc91eba36f4454a9",
            "Iops": 120,
            "VolumeType": "gp2",
            "MultiAttachEnabled": false
        },
        {
            ## The GitLab data volume
            "Attachments": [],
            "AvailabilityZone": "us-east-1a",
            "CreateTime": "2023-10-10T17:45:57.560Z",
            "Encrypted": true,
            "KmsKeyId": "<REDACTED>",
            "Size": 150,
            "SnapshotId": "snap-0d8b9b5cfede82dad",
            "State": "available",
            "VolumeId": "vol-0cd71b694fc4dc181",
            "Iops": 450,
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "azul-gitlab"
                },
                {
                    "Key": "owner",
                    "Value": "[email protected]"
                }
            ],
            "VolumeType": "gp2",
            "MultiAttachEnabled": false
        },
        {
            ## An older unused GitLab data volume (?)
            "Attachments": [],
            "AvailabilityZone": "us-east-1a",
            "CreateTime": "2023-08-01T21:29:38.920Z",
            "Encrypted": true,
            "KmsKeyId": "<REDACTED>",
            "Size": 101,
            "SnapshotId": "snap-06e8d6addfba7653f",
            "State": "available",
            "VolumeId": "vol-072b9ae2c42b3a206",
            "Iops": 303,
            "Tags": [
                {
                    "Key": "Name",
                    "Value": ""
                }
            ],
            "VolumeType": "gp2",
            "MultiAttachEnabled": false
        }
    ]
}

[dsotirho-ucsc]

@hannes-ucsc: "It turns out that CIS level 2 requires /var to reside on a separate partition and that the hardened AMI we plan to use satisfies that requirement by mounting a second volume, in addition to the root volume. The mount point conflicts with where we usually mount the GitLab data volume. Repeat experiment with the data volume moved to a different mount point to avoid the conflict."

[dsotirho-ucsc]

Index: terraform/gitlab/gitlab.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
--- a/terraform/gitlab/gitlab.tf.json.template.py	(revision d0713874b97e75a3413254ea566e74d346361610)
+++ b/terraform/gitlab/gitlab.tf.json.template.py	(date 1710875963591)
@@ -144,7 +144,7 @@
 # To then format the volume, you can then either attach it to some other Linux
 # instance and format it there or use `make terraform` to create the actual
 # Gitlab instance and attach the volume. For the latter you would need to ssh
-# into the Gitlab instance, format `/dev/xvdf` (`/dev/nvme1n1` on newer
+# into the Gitlab instance, format `/dev/xvdg` (`/dev/nvme2n1` on newer
 # instance types) and reboot the instance. For example:
 #
 # docker stop gitlab-runner
@@ -152,7 +152,7 @@
 # docker stop gitlab-dind
 # sudo mv /mnt/gitlab /mnt/gitlab.deleteme
 # sudo mkdir /mnt/gitlab
-# sudo mkfs.ext4 /dev/nvme1n1
+# sudo mkfs.ext4 /dev/nvme2n1
 # sudo reboot
 # sudo rm -rf /mnt/gitlab.deleteme
 #
@@ -1399,7 +1399,7 @@
         },
         'aws_volume_attachment': {
             'gitlab': {
-                'device_name': '/dev/sdf',
+                'device_name': '/dev/sdg',
                 'volume_id': '${data.aws_ebs_volume.gitlab.id}',
                 'instance_id': '${aws_instance.gitlab.id}',
                 'provisioner': {
@@ -1585,7 +1585,7 @@
                 'user_data_replace_on_change': True,
                 'user_data': '#cloud-config\n' + yaml.dump({
                     'mounts': [
-                        ['/dev/nvme1n1', gitlab_mount, 'ext4', '']
+                        ['/dev/nvme2n1', gitlab_mount, 'ext4', '']
                     ],
                     'packages': [
                         'docker',

[dsotirho-ucsc]

Wait with spike until spike on #6082 is done.