Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support multi-region deployments #310

Closed
gyuho opened this issue Apr 21, 2023 · 2 comments · Fixed by #312
Closed

Support multi-region deployments #310

gyuho opened this issue Apr 21, 2023 · 2 comments · Fixed by #312

Comments

@gyuho
Copy link
Collaborator

gyuho commented Apr 21, 2023 •
edited
Loading

Before

AWS_VOLUME_PROVISIONER_BIN_PATH=/home/ubuntu/volume-manager/target/release/aws-volume-provisioner
AWS_IP_PROVISIONER_BIN_PATH=/home/ubuntu/ip-manager/target/release/aws-ip-provisioner
AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH=/home/ubuntu/avalanche-telemetry/target/release/avalanche-telemetry-cloudwatch

AVALANCHED_AWS_BIN_PATH=/home/ubuntu/avalanche-ops/target/release/avalanched-aws
AVALANCHEGO_BIN_PATH=/home/ubuntu/avalanchego/build/avalanchego

cd /home/ubuntu/avalanche-ops
/home/ubuntu/avalanche-ops/target/release/avalancheup-aws default-spec \
--arch-type amd64 \
--rust-os-type ubuntu20.04 \
--anchor-nodes 4 \
--non-anchor-nodes 4 \
--region ap-northeast-2 \
--instance-mode=on-demand \
--instance-size=4xlarge \
--ip-mode=elastic \
--metrics-fetch-interval-seconds 60 \
--ingress-ipv4-cidr 0.0.0.0/0 \
--upload-artifacts-aws-volume-provisioner-local-bin ${AWS_VOLUME_PROVISIONER_BIN_PATH} \
--upload-artifacts-aws-ip-provisioner-local-bin ${AWS_IP_PROVISIONER_BIN_PATH} \
--upload-artifacts-avalanche-telemetry-cloudwatch-local-bin ${AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH} \
--upload-artifacts-avalanched-aws-local-bin ${AVALANCHED_AWS_BIN_PATH} \
--upload-artifacts-avalanchego-local-bin ${AVALANCHEGO_BIN_PATH} \
--network-name custom \
--keys-to-generate 50 \
--enable-nlb \
--primary-network-validate-period-in-days 32
version: 2
id: aops-custom-202304-2TKPeY
aad_tag: avalanche-ops-aad-tag
resources:
  identity:
    account_id: '931867039610'
    role_arn: arn:aws:sts::931867039610:assumed-role/jumpcloud-experimental-developer/[email protected]
    user_id: AROA5R542S55AJJWVOGVF:[email protected]
  region: ap-northeast-2
  ingress_ipv4_cidr: 0.0.0.0/0
  s3_bucket: avalanche-ops-202304-42ziec8wgg-ap-northeast-2
  kms_symmetric_default_encrypt_key:
    id: 95c87189-0e73-4462-ba31-3befb680888c
    arn: arn:aws:kms:ap-northeast-2:931867039610:key/95c87189-0e73-4462-ba31-3befb680888c
  ec2_key_name: aops-custom-202304-2TKPeY-ec2-key
  ec2_key_path: /home/ubuntu/aops-custom-202304-2TKPeY-ec2-access.key
  cloudformation_ec2_instance_role: aops-custom-202304-2TKPeY-ec2-instance-role
  cloudformation_ec2_instance_profile_arn: arn:aws:iam::931867039610:instance-profile/aops-custom-202304-2TKPeY-instance-profile
  cloudformation_vpc: aops-custom-202304-2TKPeY-vpc
  cloudformation_vpc_id: vpc-0439425269f4ebc93
  cloudformation_vpc_security_group_id: sg-04b6f70bcf8cfa4df
  cloudformation_vpc_public_subnet_ids:
  - subnet-035f4501874063a4f
  - subnet-0ec6e08f0adbc5301
  cloudformation_asg_anchor_nodes:
  - aops-custom-202304-2TKPeY-anchor-amd64-01
  - aops-custom-202304-2TKPeY-anchor-amd64-02
  - aops-custom-202304-2TKPeY-anchor-amd64-03
  - aops-custom-202304-2TKPeY-anchor-amd64-04
  cloudformation_asg_anchor_nodes_logical_ids:
  - aops-custom-202304-2TKPeY-anchor-amd64-01
  - aops-custom-202304-2TKPeY-anchor-amd64-02
  - aops-custom-202304-2TKPeY-anchor-amd64-03
  - aops-custom-202304-2TKPeY-anchor-amd64-04
  cloudformation_asg_non_anchor_nodes:
  - aops-custom-202304-2TKPeY-non-anchor-amd64-01
  - aops-custom-202304-2TKPeY-non-anchor-amd64-02
  - aops-custom-202304-2TKPeY-non-anchor-amd64-03
  - aops-custom-202304-2TKPeY-non-anchor-amd64-04
  cloudformation_asg_non_anchor_nodes_logical_ids:
  - aops-custom-202304-2TKPeY-non-anchor-amd64-01
  - aops-custom-202304-2TKPeY-non-anchor-amd64-02
  - aops-custom-202304-2TKPeY-non-anchor-amd64-03
  - aops-custom-202304-2TKPeY-non-anchor-amd64-04
  cloudformation_asg_nlb_arn: arn:aws:elasticloadbalancing:ap-northeast-2:931867039610:loadbalancer/net/aops-custom-202304-2TKPeY-nlb/7dc1e1fb90b53789
  cloudformation_asg_nlb_target_group_arn: arn:aws:elasticloadbalancing:ap-northeast-2:931867039610:targetgroup/aops-custom-202304-2TKPeY-tg/5d50684ef30aac69
  cloudformation_asg_nlb_dns_name: aops-custom-202304-2TKPeY-nlb-7dc1e1fb90b53789.elb.ap-northeast-2.amazonaws.com
  cloudformation_asg_launch_template_id: lt-083db80a88191d265
  cloudformation_asg_launch_template_version: '1'
  cloudformation_ssm_install_subnet_chain: aops-custom-202304-2TKPeY-ssm-install-subnet-chain
  cloudwatch_avalanche_metrics_namespace: aops-custom-202304-2TKPeY-avalanche
  created_nodes:
  - kind: anchor
    machineId: i-004218e7af59e6e3c
    nodeId: NodeID-9Cf6ZXc5xUjfS63ZZY6Qe5PtUdDWEE2Eg
    proofOfPossession:
      publicKey: 0xae84eb755fb323debc330252978275886c5558254dd61f76b3036c1eb67d72811fb7849a40e9878f8022c47aed1bf479
      proofOfPossession: 0xb9e4a894161b54f25c8fb48b0fc181019728f92a78b263faaa6d711461cd2689b53c84b69c83a1258c3d4cf0c5224a6208490fa8f5f828a38b61296c7118c05503b35e16e967d7c
be23308afa2d211c1b5ceb8df797fbda492b66738010af26c
    publicIp: 43.200.185.164
    httpEndpoint: http://43.200.185.164:9650
  - kind: anchor
    machineId: i-0004f1cda22ee6d84
    nodeId: NodeID-AGYHGNq3CZjuZ54fsJdrPiPvtChKTejZr
    proofOfPossession:
      publicKey: 0xb12778e3660bfc2f76741784f1c1e76ba58894178c88db1340879964bd175cb0d7ff7e3050bc22060164a89a9223ff97
      proofOfPossession: 0xb654102321216e99686c7a0645093f48fecf7531efbbf402fda9810ada92782b9c6fe564555e3d947fb89f290a46850a0424f8a6f82636eea91bb8ea3b86ddd2fba79c3bf7216f5
eacd972419766e3c663fda293837e6f907888b2ab90435e43
    publicIp: 43.201.208.149
    httpEndpoint: http://43.201.208.149:9650
  - kind: anchor
    machineId: i-0779d859b49002dd8
    nodeId: NodeID-5veLTLisivPaHokQYZpb7usoYw52u6hD8
    proofOfPossession:
      publicKey: 0x96b5c7c898c61f30c24bcc611d1cdf211091b2ba030452e29f02952d218cfa0b7c31306a141407da21e566dfa4faf914
      proofOfPossession: 0x8eb7ce0d1c60256d6888cb1b8d6099f6a7a2f3f67fcaa856f8ab1dd775ebf6a2f79d6539e2d92d9e3449624791482a0a0baebebbb7ee1f7bb3bf65bcc9a4b4d3e1d061cad1dfea0
171d7893885ee546793e2b99f94b1ebe8d4659458ea0a5c36
    publicIp: 52.78.19.151
    httpEndpoint: http://52.78.19.151:9650

machine:
  anchor_nodes: 4
  non_anchor_nodes: 4
  arch_type: amd64
  rust_os_type: ubuntu20.04
  instance_types:
  - m5.4xlarge
  - c5.4xlarge
  instance_mode: on-demand
  ip_mode: elastic
  volume_size_in_gb: 300
avalanched_config:
  log_level: info
  use_default_config: false
enable_nlb: true
disable_logs_auto_removal: false
metrics_fetch_interval_seconds: 60
primary_network_validate_period_in_days: 32
prefunded_keys:
- key_type: hot
  private_key_cb58: PrivateKey-ewoqjP7PxY4yr3iLTpLisriqt94hdyDFNgchSxGGztUrTXtNN
  private_key_hex: 0x56289e99c94b6912bfc12adc093c9b51124f0dc54ac7a766b2bc5ccf558d8027
  addresses:
    1000000:
      x: X-custom18jma8ppw3nhx5r4ap8clazz0dps7rv5u9xde7p
      p: P-custom18jma8ppw3nhx5r4ap8clazz0dps7rv5u9xde7p
  short_address: 6Y3kysjF9jnHnYkdS9yGAuoHyae2eNmeV
  eth_address: 0x8db97C7cEcE249c2b98bDC0226Cc4C2A57BF52FC
  h160_address: 0x8db97c7cece249c2b98bdc0226cc4c2a57bf52fc

After

AWS_VOLUME_PROVISIONER_BIN_PATH=/home/ubuntu/volume-manager/target/release/aws-volume-provisioner
AWS_IP_PROVISIONER_BIN_PATH=/home/ubuntu/ip-manager/target/release/aws-ip-provisioner
AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH=/home/ubuntu/avalanche-telemetry/target/release/avalanche-telemetry-cloudwatch

AVALANCHED_AWS_BIN_PATH=/home/ubuntu/avalanche-ops/target/release/avalanched-aws
AVALANCHEGO_BIN_PATH=/home/ubuntu/avalanchego/build/avalanchego

cd /home/ubuntu/avalanche-ops
/home/ubuntu/avalanche-ops/target/release/avalancheup-aws default-spec \
--arch-type amd64 \
--rust-os-type ubuntu20.04 \
--anchor-nodes 4 \
--non-anchor-nodes 4 \
--auto-regions 3 \
--instance-mode=on-demand \
--instance-size=4xlarge \
--ip-mode=elastic \
--metrics-fetch-interval-seconds 60 \
--ingress-ipv4-cidr 0.0.0.0/0 \
--upload-artifacts-aws-volume-provisioner-local-bin ${AWS_VOLUME_PROVISIONER_BIN_PATH} \
--upload-artifacts-aws-ip-provisioner-local-bin ${AWS_IP_PROVISIONER_BIN_PATH} \
--upload-artifacts-avalanche-telemetry-cloudwatch-local-bin ${AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH} \
--upload-artifacts-avalanched-aws-local-bin ${AVALANCHED_AWS_BIN_PATH} \
--upload-artifacts-avalanchego-local-bin ${AVALANCHEGO_BIN_PATH} \
--network-name custom \
--keys-to-generate 50 \
--enable-nlb \
--primary-network-validate-period-in-days 32
version: 3
id: aops-custom-202304-zeB7h6
aad_tag: avalanche-ops-aad-tag
resource:
  regions:
  - us-west-2
  - ap-northeast-2
  - eu-west-1
  s3_bucket: avalanche-ops-202304-42ziec8wgg-us-west-2
  ingress_ipv4_cidr: 0.0.0.0/0
  regional_resources:
    us-west-2:
      region: us-west-2
      ec2_key_name: aops-custom-202304-zeB7h6-ec2-key
      ec2_key_path: /home/ubuntu/aops-custom-202304-zeB7h6-ec2-access.us-west-2.key
    eu-west-1:
      region: eu-west-1
      ec2_key_name: aops-custom-202304-zeB7h6-ec2-key
      ec2_key_path: /home/ubuntu/aops-custom-202304-zeB7h6-ec2-access.eu-west-1.key
    ap-northeast-2:
      region: ap-northeast-2
      ec2_key_name: aops-custom-202304-zeB7h6-ec2-key
      ec2_key_path: /home/ubuntu/aops-custom-202304-zeB7h6-ec2-access.ap-northeast-2.key
machine:
  total_anchor_nodes: 4
  total_non_anchor_nodes: 4
  arch_type: amd64
  rust_os_type: ubuntu20.04
  instance_mode: on-demand
  ip_mode: elastic
  volume_size_in_gb: 300
  regional_machines:
    eu-west-1:
      anchor_nodes: 2
      non_anchor_nodes: 2
      instance_types:
      - c6a.4xlarge
      - m6a.4xlarge
      - m5.4xlarge
      - c5.4xlarge
    us-west-2:
      anchor_nodes: 1
      non_anchor_nodes: 1
      instance_types:
      - c6a.4xlarge
      - m6a.4xlarge
      - m5.4xlarge
      - c5.4xlarge
    ap-northeast-2:
      anchor_nodes: 1
      non_anchor_nodes: 1
      instance_types:
      - m5.4xlarge
      - c5.4xlarge
upload_artifacts:
  avalanched_local_bin: /home/ubuntu/avalanche-ops/target/release/avalanched-aws
  aws_volume_provisioner_local_bin: /home/ubuntu/volume-manager/target/release/aws-volume-provisioner
  aws_ip_provisioner_local_bin: /home/ubuntu/ip-manager/target/release/aws-ip-provisioner
  avalanche_telemetry_cloudwatch_local_bin: /home/ubuntu/avalanche-telemetry/target/release/avalanche-telemetry-cloudwatch
  avalanchego_local_bin: /home/ubuntu/avalanchego/build/avalanchego
  prometheus_metrics_rules_file_path: /home/ubuntu/aops-custom-202304-zeB7h6-prometheus-metrics-rules.yaml
avalanched_config:
  log_level: info
  use_default_config: false
enable_nlb: true
disable_logs_auto_removal: false
metrics_fetch_interval_seconds: 60
primary_network_validate_period_in_days: 32
prefunded_keys:
- key_type: hot
@gyuho
Copy link
Collaborator Author

gyuho commented Apr 25, 2023 •
edited
Loading

Region/zone-specific resources (need some special handling in control plane side for coordination)

  • KMS key
  • VPC
  • ASG
  • EIP
  • EBS
  • Cloudformation templates
  • Cloudwatch logs/metrics
  • NLB

Possibly shared across regions

  • IAM instance role
  • S3

This was referenced Apr 25, 2023
Merged
Merged
Merged
Merged
@gyuho
Copy link
Collaborator Author

gyuho commented Apr 25, 2023

Decided to create region-specific IAM roles (as it grants KMS keys per region)

This was referenced Apr 26, 2023
Closed
Closed
@meaghanfitzgerald meaghanfitzgerald mentioned this issue Jun 29, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support multi-region in control plane ava-labs/avalanche-ops
1 participant
@gyuho