Skip to content

Commit

Permalink
Added terraform support for NIFCLOUD (kubernetes-sigs#10227)
Browse files Browse the repository at this point in the history
* Add NIFCLOUD

* Add tf-validate-nifcloud in gitlab-ci
  • Loading branch information
ystkfujii authored and maciej-markowski committed Jun 23, 2023
1 parent 9661dde commit c56316b
Show file tree
Hide file tree
Showing 15 changed files with 844 additions and 0 deletions.
6 changes: 6 additions & 0 deletions .gitlab-ci/terraform.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,12 @@ tf-validate-upcloud:
PROVIDER: upcloud
CLUSTER: $CI_COMMIT_REF_NAME

tf-validate-nifcloud:
extends: .terraform_validate
variables:
TF_VERSION: $TERRAFORM_VERSION
PROVIDER: nifcloud

# tf-packet-ubuntu20-default:
# extends: .terraform_apply
# variables:
Expand Down
5 changes: 5 additions & 0 deletions contrib/terraform/nifcloud/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
*.tfstate*
.terraform.lock.hcl
.terraform

sample-inventory/inventory.ini
137 changes: 137 additions & 0 deletions contrib/terraform/nifcloud/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Kubernetes on NIFCLOUD with Terraform

Provision a Kubernetes cluster on [NIFCLOUD](https://pfs.nifcloud.com/) using Terraform and Kubespray

## Overview

The setup looks like following

```text
Kubernetes cluster
+----------------------------+
+---------------+ | +--------------------+ |
| | | | +--------------------+ |
| API server LB +---------> | | | |
| | | | | Control Plane/etcd | |
+---------------+ | | | node(s) | |
| +-+ | |
| +--------------------+ |
| ^ |
| | |
| v |
| +--------------------+ |
| | +--------------------+ |
| | | | |
| | | Worker | |
| | | node(s) | |
| +-+ | |
| +--------------------+ |
+----------------------------+
```

## Requirements

* Terraform 1.3.7

## Quickstart

### Export Variables

* Your NIFCLOUD credentials:

```bash
export NIFCLOUD_ACCESS_KEY_ID=<YOUR ACCESS KEY>
export NIFCLOUD_SECRET_ACCESS_KEY=<YOUR SECRET ACCESS KEY>
```

* The SSH KEY used to connect to the instance:
* FYI: [Cloud Help(SSH Key)](https://pfs.nifcloud.com/help/ssh.htm)

```bash
export TF_VAR_SSHKEY_NAME=<YOUR SSHKEY NAME>
```

* The IP address to connect to bastion server:

```bash
export TF_VAR_working_instance_ip=$(curl ifconfig.me)
```

### Create The Infrastructure

* Run terraform:

```bash
terraform init
terraform apply -var-file ./sample-inventory/cluster.tfvars
```

### Setup The Kubernetes

* Generate cluster configuration file:

```bash
./generate-inventory.sh > sample-inventory/inventory.ini

* Export Variables:

```bash
BASTION_IP=$(terraform output -json | jq -r '.kubernetes_cluster.value.bastion_info | to_entries[].value.public_ip')
API_LB_IP=$(terraform output -json | jq -r '.kubernetes_cluster.value.control_plane_lb')
CP01_IP=$(terraform output -json | jq -r '.kubernetes_cluster.value.control_plane_info | to_entries[0].value.private_ip')
export ANSIBLE_SSH_ARGS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand=\"ssh root@${BASTION_IP} -W %h:%p\""
```

* Set ssh-agent"
```bash
eval `ssh-agent`
ssh-add <THE PATH TO YOUR SSH KEY>
```
* Run cluster.yml playbook:
```bash
cd ./../../../
ansible-playbook -i contrib/terraform/nifcloud/inventory/inventory.ini cluster.yml
```
### Connecting to Kubernetes
* [Install kubectl](https://kubernetes.io/docs/tasks/tools/) on the localhost
* Fetching kubeconfig file:
```bash
mkdir -p ~/.kube
scp -o ProxyCommand="ssh root@${BASTION_IP} -W %h:%p" root@${CP01_IP}:/etc/kubernetes/admin.conf ~/.kube/config
```
* Rewrite /etc/hosts
```bash
sudo echo "${API_LB_IP} lb-apiserver.kubernetes.local" >> /etc/hosts
```
* Run kubectl
```bash
kubectl get node
```
## Variables
* `region`: Region where to run the cluster
* `az`: Availability zone where to run the cluster
* `private_ip_bn`: Private ip address of bastion server
* `private_network_cidr`: Subnet of private network
* `instances_cp`: Machine to provision as Control Plane. Key of this object will be used as part of the machine' name
* `private_ip`: private ip address of machine
* `instances_wk`: Machine to provision as Worker Node. Key of this object will be used as part of the machine' name
* `private_ip`: private ip address of machine
* `instance_key_name`: The key name of the Key Pair to use for the instance
* `instance_type_bn`: The instance type of bastion server
* `instance_type_wk`: The instance type of worker node
* `instance_type_cp`: The instance type of control plane
* `image_name`: OS image used for the instance
* `working_instance_ip`: The IP address to connect to bastion server
* `accounting_type`: Accounting type. (1: monthly, 2: pay per use)
64 changes: 64 additions & 0 deletions contrib/terraform/nifcloud/generate-inventory.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/bin/bash

#
# Generates a inventory file based on the terraform output.
# After provisioning a cluster, simply run this command and supply the terraform state file
# Default state file is terraform.tfstate
#

set -e

TF_OUT=$(terraform output -json)

CONTROL_PLANES=$(jq -r '.kubernetes_cluster.value.control_plane_info | to_entries[]' <(echo "${TF_OUT}"))
WORKERS=$(jq -r '.kubernetes_cluster.value.worker_info | to_entries[]' <(echo "${TF_OUT}"))
mapfile -t CONTROL_PLANE_NAMES < <(jq -r '.key' <(echo "${CONTROL_PLANES}"))
mapfile -t WORKER_NAMES < <(jq -r '.key' <(echo "${WORKERS}"))

API_LB=$(jq -r '.kubernetes_cluster.value.control_plane_lb' <(echo "${TF_OUT}"))

echo "[all]"
# Generate control plane hosts
i=1
for name in "${CONTROL_PLANE_NAMES[@]}"; do
private_ip=$(jq -r '. | select( .key=='"\"${name}\""' ) | .value.private_ip' <(echo "${CONTROL_PLANES}"))
echo "${name} ansible_user=root ansible_host=${private_ip} access_ip=${private_ip} ip=${private_ip} etcd_member_name=etcd${i}"
i=$(( i + 1 ))
done

# Generate worker hosts
for name in "${WORKER_NAMES[@]}"; do
private_ip=$(jq -r '. | select( .key=='"\"${name}\""' ) | .value.private_ip' <(echo "${WORKERS}"))
echo "${name} ansible_user=root ansible_host=${private_ip} access_ip=${private_ip} ip=${private_ip}"
done

API_LB=$(jq -r '.kubernetes_cluster.value.control_plane_lb' <(echo "${TF_OUT}"))

echo ""
echo "[all:vars]"
echo "upstream_dns_servers=['8.8.8.8','8.8.4.4']"
echo "loadbalancer_apiserver={'address':'${API_LB}','port':'6443'}"


echo ""
echo "[kube_control_plane]"
for name in "${CONTROL_PLANE_NAMES[@]}"; do
echo "${name}"
done

echo ""
echo "[etcd]"
for name in "${CONTROL_PLANE_NAMES[@]}"; do
echo "${name}"
done

echo ""
echo "[kube_node]"
for name in "${WORKER_NAMES[@]}"; do
echo "${name}"
done

echo ""
echo "[k8s_cluster:children]"
echo "kube_control_plane"
echo "kube_node"
36 changes: 36 additions & 0 deletions contrib/terraform/nifcloud/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
provider "nifcloud" {
region = var.region
}

module "kubernetes_cluster" {
source = "./modules/kubernetes-cluster"

availability_zone = var.az
prefix = "dev"

private_network_cidr = var.private_network_cidr

instance_key_name = var.instance_key_name
instances_cp = var.instances_cp
instances_wk = var.instances_wk
image_name = var.image_name

instance_type_bn = var.instance_type_bn
instance_type_cp = var.instance_type_cp
instance_type_wk = var.instance_type_wk

private_ip_bn = var.private_ip_bn

additional_lb_filter = [var.working_instance_ip]
}

resource "nifcloud_security_group_rule" "ssh_from_bastion" {
security_group_names = [
module.kubernetes_cluster.security_group_name.bastion
]
type = "IN"
from_port = 22
to_port = 22
protocol = "TCP"
cidr_ip = var.working_instance_ip
}
Loading

0 comments on commit c56316b

Please sign in to comment.