[WiP] add nextflow-draft

content/nextflow-on-aws-batch/10_prerequisites/30_ap-southeast-1.md

@@ -0,0 +1,8 @@
+---
+title: "Singapore"
+chapter: false
+disableToc: true
+hidden: true
+---
+
+Create a Cloud9 Environment: [https://ap-southeast-1.console.aws.amazon.com/cloud9/home?region=ap-southeast-1](https://ap-southeast-1.console.aws.amazon.com/cloud9/home?region=ap-southeast-1)

content/nextflow-on-aws-batch/10_prerequisites/30_eu-west-1.md

@@ -0,0 +1,8 @@
+---
+title: "Ireland"
+chapter: false
+disableToc: true
+hidden: true
+---
+
+Create a Cloud9 Environment: [https://eu-west-1.console.aws.amazon.com/cloud9/home?region=eu-west-1](https://eu-west-1.console.aws.amazon.com/cloud9/home?region=eu-west-1)

content/nextflow-on-aws-batch/10_prerequisites/30_us-east-1.md

@@ -0,0 +1,8 @@
+---
+title: "N. Virgina"
+chapter: false
+disableToc: true
+hidden: true
+---
+
+Create a Cloud9 Environment: [https://us-east-2.console.aws.amazon.com/cloud9/home?region=us-east-1](https://us-east-1.console.aws.amazon.com/cloud9/home?region=us-east-1)

content/nextflow-on-aws-batch/10_prerequisites/30_us-east-2.md

@@ -0,0 +1,8 @@
+---
+title: "Ohio"
+chapter: false
+disableToc: true
+hidden: true
+---
+
+Create a Cloud9 Environment: [https://us-east-2.console.aws.amazon.com/cloud9/home?region=us-east-2](https://us-east-2.console.aws.amazon.com/cloud9/home?region=us-east-2)

content/nextflow-on-aws-batch/10_prerequisites/30_us-west-2.md

@@ -0,0 +1,8 @@
+---
+title: "Oregon"
+chapter: false
+disableToc: true
+hidden: true
+---
+
+Create a Cloud9 Environment: [https://us-west-2.console.aws.amazon.com/cloud9/home?region=us-west-2](https://us-west-2.console.aws.amazon.com/cloud9/home?region=us-west-2)

content/nextflow-on-aws-batch/10_prerequisites/30_workspace.md

@@ -0,0 +1,43 @@
+---
+title: "Create a Workspace"
+chapter: false
+weight: 30
+---
+
+{{% notice warning %}}
+If you are running the workshop on your own, the Cloud9 workspace should be built by an IAM user with Administrator privileges, not the root account user. Please ensure you are logged in as an IAM user, not the root
+account user.
+{{% /notice %}}
+
+{{% notice info %}}
+If you are at an AWS hosted event (such as re:Invent, Kubecon, Immersion Day, or any other event hosted by 
+an AWS employee) follow the instructions on the region that should be used to launch resources
+{{% /notice %}}
+
+{{% notice tip %}}
+Ad blockers, javascript disablers, and tracking blockers should be disabled for
+the cloud9 domain, or connecting to the workspace might be impacted.
+Cloud9 requires third-party-cookies. You can whitelist the [specific domains]( https://docs.aws.amazon.com/cloud9/latest/user-guide/troubleshooting.html#troubleshooting-env-loading).
+{{% /notice %}}
+
+### Launch Cloud9 in your closest region:
+
+{{< tabs name="Region" >}}
+    {{< tab name="Oregon" include="30_us-west-2.md" />}}
+    {{< tab name="Ireland" include="30_eu-west-1.md" />}}
+    {{< tab name="N. Virginia" include="30_us-east-1.md" />}}
+    {{< tab name="Ohio" include="30_us-east-2.md" />}}
+    {{< tab name="Singapore" include="30_ap-southeast-1.md" />}}
+{{< /tabs >}}
+
+- Select **Create environment**
+- Name it **nextflowworkshop**, and take all other defaults
+- When it comes up, customize the environment by closing the **welcome tab**
+and **lower work area**, and opening a new **terminal** tab in the main work area:
+![c9before](/images/nextflow-on-aws-batch/prerequisites/c9before.png)
+
+- Your workspace should now look like this:
+![c9after](/images/nextflow-on-aws-batch/prerequisites/c9after.png)
+
+- If you like this theme, you can choose it yourself by selecting **View / Themes / Solarized / Solarized Dark**
+in the Cloud9 workspace menu.

content/nextflow-on-aws-batch/10_prerequisites/31_resize_ebs.md

@@ -0,0 +1,54 @@
+---
+title: "Resize Root Volume"
+chapter: false
+weight: 31
+---
+
+## Resize Cloud9 EBS
+
+The default 10GB is quite small when using a docker file for Genomics.
+Thus, let us resize the EBS volume used by the Cloud9 instance.
+
+To change the EBS volume, please do
+
+   1. Select the Cloud9 instance in the EC2 console [deep link to get there](https://console.aws.amazon.com/ec2/v2/home)
+   2. Click the root-device link
+   3. click on the EBS-ID in the box appearing
+
+![resize_ebs_0](/images/nextflow-on-aws-batch/prerequisites/resize_ebs_0.png)
+
+Afterward modify the EBS volume.
+
+![resize_ebs_1](/images/nextflow-on-aws-batch/prerequisites/resize_ebs_1.png)
+
+And chose a new volume size (e.g. 100GB).
+
+![resize_ebs_2](/images/nextflow-on-aws-batch/prerequisites/resize_ebs_2.png)
+
+{{% notice info %}}
+Please make sure that the changes went through and the EBS volume now reflects the new size of the volume.
+{{% /notice %}}
+
+## Resize FS
+
+Changing the block device does not increase the size of the file system.
+
+To do so head back to the Cloud9 instance and use the following commands.
+
+```bash
+sudo growpart /dev/xvda 1
+sudo resize2fs $(df -h |awk '/^\/dev/{print $1}')
+```
+
+The root file-system should now show 99GB.
+
+```bash
+df --human-readable
+```
+
+```bash
+Filesystem      Size  Used Avail Use% Mounted on
+devtmpfs        483M   60K  483M   1% /dev
+tmpfs           493M     0  493M   0% /dev/shm
+/dev/xvda1       99G  8.0G   91G   9% /
+```

content/nextflow-on-aws-batch/10_prerequisites/40_create_iam.md

@@ -0,0 +1,16 @@
+---
+title: "Create an IAM role for your Workspace"
+chapter: false
+weight: 39
+---
+
+## Create an IAM role for your Workspace
+
+1. Head over to the IAM console and find *create role* (**[2]**) under the *Roles* (**[1]**) section.
+![create_role_0](/images/nextflow-on-aws-batch/prerequisites/create_role-0.png)
+1. Pick the EC2 use-case (**[1]**) and hit **Next** button at the bottom.
+![create_role_1](/images/nextflow-on-aws-batch/prerequisites/create_role-1.png)
+1. Wait for the Permission tab to render and chose **AdministratorAccess** (**[1]**).
+![create_role_2](/images/nextflow-on-aws-batch/prerequisites/create_role-2.png)
+1. Add a tag **nextflow-workshop** without a value to identify the role for cleanup.
+![create_role_3](/images/nextflow-on-aws-batch/prerequisites/create_role-3.png)

content/nextflow-on-aws-batch/10_prerequisites/40_updateiam.md

@@ -0,0 +1,13 @@
+---
+title: "Attach the IAM role to your Workspace"
+chapter: false
+weight: 40
+---
+
+## Attach the IAM role to your Workspace
+
+1. Follow [this deep link to find your Cloud9 EC2 instance](https://console.aws.amazon.com/ec2/v2/home?#Instances:tag:Name=aws-cloud9-.*workshop.*;sort=desc:launchTime)
+1. Select the instance, then choose **Actions / Instance Settings / Attach/Replace IAM Role**
+![c9instancerole](/images/nextflow-on-aws-batch/prerequisites/c9instancerole.png)
+1. Choose **nextflow-workshop-admin** from the **IAM Role** drop down, and select **Apply**
+![c9attachrole](/images/nextflow-on-aws-batch/prerequisites/c9attachrole.png)

content/nextflow-on-aws-batch/10_prerequisites/45_disable_cred.md

@@ -0,0 +1,28 @@
+---
+title: "Disable Temporary Credentials"
+chapter: false
+weight: 45
+---
+
+## Managed Credential Handling from Cloud9
+
+To not have Cloud9 overwrite the IAM roles with temporary crendentials, we will disable temporary crendential management within Cloud9.
+
+![](/images/nextflow-on-aws-batch/prerequisites/disable_cred.png)
+
+Please verify that your IAM role is providing your identity by executing the following command:
+
+```bash
+aws sts get-caller-identity
+```
+
+The output should include the IAM role name (`nextflow-workshop-admin`), similar to:
+
+```bash
+$ aws sts get-caller-identity
+{
+    "UserId": "AROA4KFOLXRT3PYT5QRNW:i-0aca005990c36734f",
+    "Account": "846474230887",
+    "Arn": "arn:aws:sts::846474230887:assumed-role/nextflow-workshop-admin/i-0aca005990c36734f"
+}
+```

content/nextflow-on-aws-batch/10_prerequisites/50_install_tools.md

@@ -0,0 +1,46 @@
+---
+title: "Install Tools"
+chapter: false
+weight: 50
+---
+
+## Install Java and Nextflow
+
+The nextflow command-line tool uses the JVM. Thus, we will install AWS open-source variant [Amazon Corretto](https://docs.aws.amazon.com/corretto).
+
+{{% notice info %}}
+Amazon Corretto is a no-cost, multiplatform, production-ready distribution of the Open Java Development Kit (OpenJDK). Corretto comes with long-term support that will include performance enhancements and security fixes. Amazon runs Corretto internally on thousands of production services and Corretto is certified as compatible with the Java SE standard. With Corretto, you can develop and run Java applications on popular operating systems, including Linux, Windows, and macOS.
+{{% /notice %}}
+
+### Amazon Corretto
+
+To [install Corretto](https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/generic-linux-install.html), we are adding the repository first.
+
+```bash
+sudo rpm --import https://yum.corretto.aws/corretto.key 
+sudo curl -L -o /etc/yum.repos.d/corretto.repo https://yum.corretto.aws/corretto.repo
+```
+
+Afterwards install java-11 and check the installation.
+
+```bash
+sudo yum install -y java-11-amazon-corretto-devel
+java --version
+```
+
+### Graphviz
+
+Nextflow is able to render graphs for which it needs `graphviz` to be installed. `jq` will help us deal with JSON files.
+
+```bash
+sudo yum install -y graphviz jq
+```
+
+### AWS Region
+
+Even though we are depending on an IAM Role and not local permissions some tools  depend on having the `AWS_REGION` defined as environment variable - let's add it to our login shell configuration.
+
+```bash
+export AWS_REGION=$(curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
+echo "AWS_REGION=${AWS_REGION}" |tee -a ~/.bashrc
+```

content/nextflow-on-aws-batch/10_prerequisites/51_install_nextflow.md

@@ -0,0 +1,20 @@
+---
+title: "Install Nextflow"
+chapter: false
+weight: 51
+---
+
+## Nextflow
+
+This workshop will use Nextflow as a workflow engine. It provides a simple and fluent DSL and is portable by using containers for workflow execution.
+Furthermore it allows the use of a lot of traditional and modern execution engines; one of which is AWS Batch which we will use today.
+
+### Install Nextflow
+
+Installing Nextflow using the online installer.
+The snippet creates the nextflow launcher in the current directory. So we just move the command to `/usr/local/bin` to have it ready to be executed anywhere.
+
+```bash
+curl -s https://get.nextflow.io | bash
+sudo mv nextflow /usr/local/bin/
+```

content/nextflow-on-aws-batch/10_prerequisites/_index.md

@@ -0,0 +1,13 @@
+---
+title: "Start the workshop..."
+chapter: true
+weight: 10
+---
+
+# Getting Started
+To start the workshop, follow one of the following depending on whether you are...
+
+* ...[running the workshop on your own (in your own account)](/nextflow-on-aws-batch/10_prerequisites/nf_self_paced.html), or
+* ...[attending an AWS hosted event (using AWS provided hashes)](/nextflow-on-aws-batch/10_prerequisites/nf_aws_event.html)
+
+Once you have completed with either setup, continue with **[Create a Workspace]({{< relref "30_workspace.md" >}})**

content/nextflow-on-aws-batch/10_prerequisites/nf_aws_event.md

@@ -0,0 +1,25 @@
+---
+title: "...at an AWS event"
+chapter: false
+weight: 20
+---
+
+### Running the workshop at an AWS Event
+
+{{% notice warning %}}
+Only complete this section if you are at an AWS hosted event (such as re:Invent,
+Kubecon, Immersion Day, or any other event hosted by an AWS employee). If you are running the workshop on your own, go to: [Start the workshop on your own]({{< relref "self_paced.md" >}}).
+{{% /notice %}}
+
+### Login to the AWS Workshop Portal
+
+If you are at an AWS event, an AWS acccount was created for you to use throughout the workshop. You will need the **Participant Hash** provided to you by the event's organizers.
+
+1. Connect to the portal by browsing to [https://dashboard.eventengine.run/](https://dashboard.eventengine.run/).
+2. Enter the Hash in the text box, and click **Proceed**
+3. In the User Dashboard screen, click **AWS Console**
+4. In the popup page, click **Open Console**
+
+You are now logged in to the AWS console in an account that was created for you, and will be available only throughout the workshop run time.
+
+Once you have completed the step above, **you can head straight to [Create a Workspace]({{< relref "30_workspace.md" >}})**

content/nextflow-on-aws-batch/10_prerequisites/nf_self_paced.md

@@ -0,0 +1,42 @@
+---
+title: "...on your own"
+chapter: false
+weight: 10
+---
+
+{{% notice warning %}}
+Only complete this section if you are running the workshop on your own. If you are at an AWS hosted event (such as re:Invent, Kubecon, Immersion Day, etc), go to [Start the workshop at an AWS event]({{< relref "aws_event.md" >}}).
+{{% /notice %}}
+
+### Prepare for CleanUp
+
+{{% notice warning %}}
+In order to be able to clean up after the workshop is done, we will tag every resource with a tag **nextflow-workshop** and no value. This will allow us to filter for those resources and remove tehm savely.
+{{% /notice %}}
+
+### Running the workshop on your own
+
+{{% notice warning %}}
+Your account must have the ability to create new IAM roles and scope other IAM permissions.
+{{% /notice %}}
+
+1. If you don't already have an AWS account with Administrator access: [create
+one now by clicking here](https://aws.amazon.com/getting-started/)
+
+1. Once you have an AWS account, ensure you are following the remaining workshop steps
+as an IAM user with administrator access to the AWS account:
+[Create a new IAM user to use for the workshop](https://console.aws.amazon.com/iam/home?#/users$new)
+
+1. Enter the user details:
+![Create User](/images/using_ec2_spot_instances_with_eks/prerequisites/iam-1-create-user.png)
+
+1. Attach the AdministratorAccess IAM Policy:
+![Attach Policy](/images/using_ec2_spot_instances_with_eks/prerequisites/iam-2-attach-policy.png)
+
+1. Click to create the new user:
+![Confirm User](/images/using_ec2_spot_instances_with_eks/prerequisites/iam-3-create-user.png)
+
+1. Take note of the login URL and save:
+![Login URL](/images/using_ec2_spot_instances_with_eks/prerequisites/iam-4-save-url.png)
+
+Once you have completed the step above, **you can head straight to [Create a Workspace]({{<  relref "30_workspace.md"  >}})**