[Buildkite] Intrastructure

[brianseeders]

Intrastructure

• New GCP project for Kibana CI
• Developer IAM + Roles?
• Migrate current Buildkite infra to new GCP project
• Check quotas and do a bit of capacity planning

Notes:

• Separate GCP project for CI
• Things that will live here:
  • GCS Buckets
  • Artifact Registry
  • Cloud Build
  • Cloud Run
    • PR bot, Slack bot?
  • GKE Auto-Pilot
    • Agent Manager
  • Agent Instances / Images
  • Cloud NAT + Cloud Router (for agents)
• Cloud NAT
  • To scale beyond about 1,800 agents, we will likely need to implement Cloud NAT and remove external IP addresses from our agent instances
  • Needs Cloud Router set up as well
  • Automatic IP allocation works like this:
    • Each VM is reserved a specified number of ports per IP address in Cloud NAT
    • When the ports are maxed, a new IP address is added to the pool, which counts against static IP quota (confirm static IP quota?)
    • Each IP can have about 64K ports
    • Each VM can reserve up to 1024 ports
    • So, with auto scaling of IPs, at least 64 agents will share an IP
• Allow SSH via IAP
  • IAP allows authorized users (auth via GCP/Google IAM) to SSH into agent instances that don't have public IPs
  • This is already done, except for assigning user/group permissions

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)