Skip to content

Commit

Permalink
Merge pull request #189 from mikegolubev/gitlab-spot-workshop
Browse files Browse the repository at this point in the history
Adding GitLab workshop
  • Loading branch information
jagpk authored Sep 9, 2022
2 parents 6996036 + a672cbd commit 29334c7
Show file tree
Hide file tree
Showing 61 changed files with 1,835 additions and 29 deletions.
27 changes: 3 additions & 24 deletions content/amazon-ec2-spot-cicd-workshop/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,6 @@ pre: "<b>8. </b>"
---

## Overview
During this workshop, you'll get hands-on with Amazon EC2 Spot and discover architectural best practices through the lens of DevOps and CI/CD. You'll deploy Jenkins build agents and test environments on Spot instances at a fraction of the cost of on-demand instances. You'll also implement mechanisms to ensure that your CI/CD tooling recovers from spot market events by decoupling application state from your compute resources. Finally, you'll migrate your CI/CD environment to a containered environment to eke out maximum performance and cost efficiency. In addition to covering the ins and outs of Spot, we'll share some of the Spot-based mechanisms used by customers to reduce the cost of their test and production workloads.

## Workshop Details
This workshop will be broken down into a series of labs that flow on from each other (that is, you must complete each lab in order before proceeding with the next). The lab exercises that will be covered are:

* Workshop preparation: Deploy pre-requisite resources through Amazon CloudFormation;
* Lab 1: Reduce the cost of builds using Amazon EC2 Spot Fleet;
* Lab 2: Deploy testing environments using Amazon EC2 Spot, Amazon CloudFormation & Amazon EC2 Launch Templates;
* Lab 3: Externalize state data to add resiliency and reduce cost for your CI/CD tooling;
* Lab 4: Using containers backed by Auto Scaling Groups comprised of both on-demand and Spot instances;
* Workshop clean up.

As a reminder, you should have a laptop device (Windows/OSX/Linux are supported - tablets are not appropriate) with the current version of Google Chrome or Mozilla Firefox installed. You should also have a clean AWS account, with **AdministratorAccess** policy-level access.

This workshop should take between two and three hours to complete, depending on your proficiency with the AWS services being featured.

#### Additional considerations when running this workshop in a corporate IT environment
If you are running this workshop from a corporate IT environment, contact your Systems Administrator to ensure that you will be able to establish outbound Secure Shell (SSH) connections to an Internet host:

* If you cannot establish SSH connections to Internet hosts (and do not have a suitable workaround), you will not be able to complete Labs 3 & 4;
* If you can establish SSH connections to Internet hosts, obtain from your Systems Administrator the source IP address CIDR block that connections will be established from.

If you access the Internet through a transparent proxy server running in your corporate IT environment and this proxy server uses a different source address than where SSH connections come from, additional configuration of AWS Security Groups will need to be carried out. The lab guide will indicate the configuration steps required when appropriate.

Amazon EC2 Spot Instances are a good fit in DevOps scenarios: to run your CI/CD pipelines (including build and test runners) and to deploy your testing environments. In this section you can select one of the two workshops for CI/CD depending on the tool that you use:
* [Jenkins](/amazon-ec2-spot-cicd-workshop/jenkins-spot.html)
* [GitLab](/amazon-ec2-spot-cicd-workshop/gitlab-spot.html)
63 changes: 63 additions & 0 deletions content/amazon-ec2-spot-cicd-workshop/gitlab-spot/010-prep.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
+++
title = "Workshop Preparation"
weight = 10
+++

### Log in to AWS Cloud9 environment

You can execute the steps of this workshop directly on your workstation, but then you will need to make sure that you have the command-line tools for Git, Terraform, kubectl and AWS CLI installed. Instead of that, to not change any local settings, we recommend to use [AWS Cloud9](https://aws.amazon.com/cloud9/): a cloud IDE where you can get access to the terminal and install all the required tools.

A Cloud9 environment has already been provisioned for you in the CloudFormation template (created in the [**Starting the workshop**](before.html) section). You will now log in to it and do the final configuration steps: disable the AWS managed temporary credentials and download the required workshop files.

1. In the AWS Console enter **Cloud9** in the search box at the top of the screen and open the service.
2. On the **Your environments** page find the environment called like `GitLabWorkshopC9Instance-...` and click the **Open IDE** button for it.

![AWS Console Screenshot: Cloud9 Environment](/images/gitlab-spot/AWSConsole-Cloud9Environment.png)

3. When the environment comes up, close all tabs inside it and open a new terminal by clicking **+** > **New Terminal**.

4. Open preferences tab by choosing the cogwheel icon at the top-right corner of the screen. Then choose **AWS Settings** in the navigation pane.
5. Disable **AWS managed temporary credentials** toggle:

![Cloud9 Screenshot: Preferences](/images/gitlab-spot/Cloud9-Preferences.png)

6. Close the **Preferences** tab and in the terminal tab, execute the following command to verify that you are using the correct role (in the output you should see `GitLabWorkshopC9Role` and green `OK`):
```
aws sts get-caller-identity --query Arn | grep GitLabWorkshopC9Role && echo -e "\033[0;32mOK\033[0m" || echo -e "\033[0;31mNOT OK\033[0m"
```

![Cloud9 Screenshot: Caller identity](/images/gitlab-spot/Cloud9-CallerIdentity.png)

7. During the workshop, you will need to refer to the identifiers of the resources created by the CloudFormation stack you deployed. To reduce copy and paste across the CloudFormation console and the Cloud9 environment, you will load the CloudFormation Stack Outputs to environment variables. Configure the stack_name environment variable with the name of your CloudFormation template (substitute the `mod-gitlab-spot-workshop` with the actual stack name that you created / identified in the [Starting the workshop](before.html) section):
```
export stack_name=mod-gitlab-spot-workshop
```

8. Now, load the CloudFormation stack outputs on environment variables running the following commands:
```
export AWS_REGION=$(curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
for output in $(aws cloudformation describe-stacks --stack-name $stack_name --query 'Stacks[].Outputs[].OutputKey' --output text)
do
export $output=$(aws cloudformation describe-stacks --stack-name $stack_name --query 'Stacks[].Outputs[?OutputKey==`'$output'`].OutputValue' --output text) >> ~/.bash_profile
eval "echo $output : \"\$$output\""
eval "echo export $output=\"\$$output\"" >> ~/.bash_profile
done
. ~/.bash_profile
```

If successful, the output should be similar to the following:
```
Subnet1 : subnet-0225d837e8401e3a7
VPC : vpc-0cef9a85fe90459e6
GitLabPassword : 5HfKdkUxc880EDnV
GitLabURL : https://d1in48s0v2ygmm.cloudfront.net
GitLabCacheBucket : mod-gitlab-spot-workshop-gitlabcachebucket-aoggbfq6go76
Subnet1Zone : us-east-1a
```

9. Finally, enable Cloud9 to show hidden files (you will need it to modify the scripts of GitLab CI/CD later). To do this choose the small cogwheel icon right above the file tree and choose **Show Hidden Files** if it has not been enabled yet (if there is a tick to the left of it, do not click again, as it will disable the display of hidden files):

![Cloud9 Screenshot: Show hidden files](/images/gitlab-spot/Cloud9-ShowHiddenFiles.png)

You are now ready to start the main sections of the workshop! Please proceed to [**Create a GitLab repository**](020-create-gitlab-repo.html).
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
+++
title = "Create a GitLab repository"
weight = 20
+++
In this lab, you will create a new repository in GitLab and configure it in AWS Cloud9. Next, you will create the source code of the demo app and commit it into the repository.

### Log in to GitLab

1. Run below command to get GitLab URL and password from saved environment variables:
```
echo GitLabURL = $GitLabURL
echo GitLabPassword = $GitLabPassword
```
2. Open a new browser tab and proceed to GitLab URL.
3. Log in to GitLab with username `root` and the password from **GitLabPassword** output value.

### Create a new repository
**Challenge:** If you are familiar with GitLab, then test your knowledge by creating a new empty private repository called `GitLab Spot Workshop`.

{{%expand "Click to reveal the instructions" %}}
1. Choose **New project** on the **Projects** page.
2. Choose **Create blank project**.
3. In the **Project name** field type `GitLab Spot Workshop`, in the dropdown next to **Project URL** select `root` user, and unmark the checkbox **Initialize repository with a README**. Leave the rest values as default, and choose **Create project**:

![GitLab Screenshot: Create blank project](/images/gitlab-spot/GitLab-CreateBlankProject.png)

4. Copy the repository URL (in format `https://xxx.cloudfront.net/root/gitlab-spot-workshop.git`) that is displayed, for example, in **Create a new repository** section on the screen.

{{% /expand%}}

### Add GitLab as origin to the demo app repository

You will now initialize Git repository in the demo app and add your GitLab environment as an origin to it.

1. Return to Cloud9 tab and execute the following command to switch to the directory with the demo application:

```
cd ~/environment/amazon-ec2-spot-cicd-workshop/gitlab-spot/demo-app/
```

2. Execute the following command to substitute the `ECR_ADDRESS` and other placeholders with their actual values (like the address of ECR repository created in the CloudFormation stack) and save it into `.gitlab-ci.yml` file as expected by GitLab CI/CD:

```
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 600")
REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region)
export ECR_INFO=$(aws ecr describe-repositories --region $REGION --query "repositories[?contains(repositoryName, 'gitlab-spot-demo')].repositoryUri" --output text)
export ECR_ADDRESS=$(echo $ECR_INFO | awk -F'/' '{print $1}')
export ECR_NAME=$(echo $ECR_INFO | awk -F'/' '{print $2}')
sed "s/\${ECR_ADDRESS}/${ECR_ADDRESS}/g" template-gitlab-ci.yml | sed "s/\${ECR_NAME}/${ECR_NAME}/g" | sed "s/\${AWS_REGION}/${REGION}/g" > .gitlab-ci.yml
```

3. In the file tree on the left open file `amazon-ec2-spot-cicd-workshop/gitlab-spot/demo-app/.gitlab-ci.yml` (if you don't see it, make sure you have enabled the hidden files in [**Workshop Preparation**](010-prep.html)). Look through it to understand what it does.
4. Return to the terminal tab and execute the following commands to define your name and email that will be used in Git (replace `Your Name` and `[email protected]` with the values you prefer):

```
git config --global user.name "Your Name"
git config --global user.email "[email protected]"
```

5. Optionally you can enable the storage of the GitLab login and password in Git configuration, otherwise you will need to enter them each time you work with the repository:

```
git config --global credential.helper store
```

6. Initialize Git inside the directory (use the actual URL of GitLab repositoy that you saved in the previous section) and do your first commit:

```
git init --initial-branch=main
git remote add origin https://xxx.cloudfront.net/root/gitlab-spot-workshop.git
git add .
git commit -m "Initial commit"
```

You are now ready to do the key step in configuring GitLab CI/CD on Spot instances: add the runners. Please proceed to [**Configure GitLab runners on Spot instances**](030-configure-gitlab-runners-on-spot.html).
Loading

0 comments on commit 29334c7

Please sign in to comment.