Skip to content

Latest commit

 

History

History
162 lines (121 loc) · 8.11 KB

README.md

File metadata and controls

162 lines (121 loc) · 8.11 KB
Raw

stack-external-worker

Deploy external workers pool for Cycloid.io

Create an Amazon Auto Scaling Group with Cycloid workers installed in EC2 servers.

The worker EC2 server will also contain Fluentd agent to send logs to Amazon CloudWatch logs. And also expose metrics in Prometheus format using Telegraf on port 9100.

note : The version of Cycloid workers is automatically generated by a curl on scheduler_api_address/api/v1/info URL.

Architecture

  • ASG: Amazon Auto Scaling Group

Requirements

In order to run this task, couple elements are required within the infrastructure:

  • Having a VPC with private & public subnets here
  • Having an S3 bucket (with versioning) to store Terraform remote states here

Details

Pipeline

Note The pipeline contains a manual approval between terraform plan and terraform apply. That means if you trigger a terraform plan, to apply it, you have to go on terraform apply job and click on the + button to trigger it.

Jobs description

  • build-worker-ami : Build Cycloid worker Amazon image AMI using Packer and Ansible.
  • terraform-plan: Terraform job that will simply make a plan of the external worker's stack.
  • terraform-apply: Terraform job similar to the plan one, but will actually create/update everything that needs to. Please see the plan diff for a better understanding.
  • deploy : Deploy the new AMI created into the ASG.
  • clean-old-worker-ami : Remove old AMI of Cycloid workers.
  • terraform-destroy: ⚠️ Terraform job meant to destroy the whole stack - NO CONFIRMATION ASKED. If triggered, the full project WILL be destroyed. Use with caution.

Params

Name Description Type Default Required
aws_access_key Amazon AWS access key for Terraform. See value format Here - ((aws.access_key)) True
aws_default_region Amazon AWS region to use for Terraform. - eu-west-1 True
aws_secret_key Amazon AWS secret key for Terraform. See value format Here - ((aws.secret_key)) True
config_ansible_path Path of Ansible files in the config git repository - ($ project $)/ansible True
config_git_branch Branch of the config git repository. - master True
config_git_private_key SSH key pair to fetch the config git repository. - ((ssh_config.ssh_key)) True
config_git_repository Git repository URL containing the config of the stack. - [email protected]:MyUser/config-external-workers.git True
config_terraform_path Path of Terraform files in the config git repository - ($ project $)/terraform/($ environment $) True
customer Name of the Cycloid Organization, used as customer variable name. - ($ organization_canonical $) True
debug_public_key Public ssh key injected by Packer during the build of the AMI for debug purpose - "ssh-rsa ..." False
env Name of the project's environment. - ($ environment $) True
external_worker_key Name of the Cycloid credential containing a public and a private SSH key to configure Cycloid workers. cycloid_creds_name cycloid-worker-keys True
packer_ansible_version Version of ansible used during Packer build to run Ansible playbook - "2.7" False
project Name of the project. - ($ project $) True
scheduler_api_address URL of the Pipeline scheduler API - https://myscheduler.com True
scheduler_host Host or IP of the Pipeline scheduler - "myscheduler.com" True
scheduler_port Port of the Pipeline scheduler - 2222 True
stack_git_branch Branch to use on the public stack git repository - master True
terraform_storage_bucket_name AWS S3 bucket name to store terraform remote state file. - ($ organization_canonical $)-terraform-remote-state True

Terraform

Inputs

Name Description Type Default Required
bastion_sg_allow Amazon source security group ID which will be allowed to connect on Magento front port 22 (ssh) - `` False
keypair_name SSH keypair name to use to deploy ec2 instances - cycloid-external-worker False
extra_tags Dict of extra tags to add on aws resources. format { "foo" = "bar" }. - {} False
metrics_sg_allow Additionnal security group ID to assign to Cycloid workers. Goal is to allow monitoring server to query metrics - "" False
public_subnets_ids Amazon public subnets IDs on which create each components. array `` True
vpc_id Amazon VPC id on which create each components. - `` True
worker_ami_id If you don't want to use the builded ami but a specific ami (mainly used for debug). string "" False
worker_asg_max_size Amazon Auto Scaling Group max size configuration. - 6 False
worker_asg_min_size Amazon Auto Scaling Group min size configuration. - 1 False
worker_count Number of Aws EC2 worker server to create. - `` True
worker_disk_size Root disk size in Go of Aws EC2 worker servers. - 20 False
worker_ebs_optimized Whether the Instance is EBS optimized or not, related to the instance type you choose. bool true False
worker_launch_template_profile Select launch template profile to use. Profile available "spot ondemand" - spot
worker_spot_price The maximum hourly price you're willing to pay for the Spot Instances. Linked to instance type. - 0.3 False
worker_type Type of AWS EC2 worker servers. This will be used for spot and ondemand launch config templates. - c5d.2xlarge False
worker_volume_disk_size Volume disk size in Go of Aws EC2 worker servers. - 130 False

Outputs

Name Description
asg_workers_sec_group_id Security group ID of workers

Ansible

Name Description Type Default Required
concourse_worker_tag A tag to set during registration to create dedicated workers for some specific jobs. Eg FastStorage, ... - `` False
concourse_worker_team The name of the team that this worker will be assigned to. Team ID is linked to your Cycloid organization. - `` False
fs_volume_type Filesystem used to store concourse datas. Possible values "btrfs ext4" - btrfs
use_local_device Use local storage drive from the instance. Default use EBS volume /dev/xvdg or /dev/local0. If not found keep the var_lib_device - false False
var_lib_device Volume device name to use for worker datas. If "nodevice" and use_local_device=false no dedicated volume will be used. - "nodevice" False

Deploy an external worker using Aws CloudFormation stack

Deploy using CloudFormation can be done with this url :

https://eu-central-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/create/review
?templateURL=https://s3-eu-west-1.amazonaws.com/cycloid-cloudformation/external-worker-aws-cf-template.yaml
&stackName=cycloid-ci-workers
&param_TeamId=...

Troubleshooting

Test ansible role with molecule

It is recommended to use virtualenv:

sudo apt install python3-virtualenv
or
python3 -m pip install --upgrade --user setuptools

Dependencies:

cd ansible

virtualenv .env --clear
source .env/bin/activate

python3 -m pip install ansible==2.9.* molecule molecule-docker yamllint ansible-lint flake8 pytest-testinfra

Run the tests:

molecule test

# or

molecule lint
molecule destroy
molecule converge
molecule verify

don't stop a failling worker

If the user-data script fail on boot, the worker is destroyed after 3 min. In case of debug, you can avoid termination of worker by creating a /var/tmp/keeprunning file.

ssh worker
touch /var/tmp/keeprunning