Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

Persistence with Eirini in SCF

Jump to bottom
Christian edited this page Jul 12, 2019 · 3 revisions

Persistence volume support with Eirini in SCF

Note: Applies to builds only from develop branch

The eirini-persi-broker is shipped in SCF along with Eirini, this component allows to map Kubernetes StorageClasses as broker plans in Cloud Foundry.

How it works

To make a (native) persistent volume claim accessible to a Cloud Foundry application, we create a new service from a plan available in the Eirini persistence broker, and we associate it to the application.

Persistence Setup

To enable persistence with Eirini, it is only needed to define Plans that the broker will expose to the Cloud Foundry instance.

By default no Plans are mapped to the Service broker, they need to be specified manually before deployment.

Note This means that unless one Plan is defined (at least) in the deployment configuration the Eirini broker won't be accessible inside SCF ( See Test Persistence below ).

In the SCF config values, add (one, or more) Plan block(s) in the environment variable EIRINI_PERSI_PLANS for each Storage Class in Kubernetes you wish to expose in Cloud Foundry:

env:
  EIRINI_PERSI_PLANS: |
      - id: "first-storageclass"
        name: "default"
        description: "Eirini persistence broker"
        free: true
        kube_storage_class: "persistent_storageclass"
        default_size: "1Gi"
      - id: "second-storageclass"
        name: "hostpath"
        description: "Eirini persistence broker"
        free: true
        kube_storage_class: "hostpath_storageclass"
        default_size: "1Gi"

EIRINI_PERSI_PLANS expects a list of Plans. A Plan lets you define the access to one Kubernetes Storage Class.

All the available parameters for a Plan are:

  # Unique plan ID
  id: "first-storageclass"

  # Display name of the plan
  name: "default"

  # Plan description
  description: "Eirini persistence broker"

  # Free or Paid tier
  free: true

  # Kubernetes storage class
  kube_storage_class: "persistent_storageclass"

  # Default quota size
  default_size: "1Gi"

Fields:

  • id: Unique identifier for the broker plan
  • name: Plan name which will be displayed when listing in the marketplace (cf marketplace -s eirini-persi )
  • description: Plan description for the marketplace
  • free: Specify if the plan is a Free or a Paid tier (boolean)
  • kube_storage_class: Kubernetes Storage Class associated with the plan. eirini-persi-broker will create PersistentVolumeClaims from this Storage Class
  • default_size: Default quota size for services creation. Any service created from this plan will get a default quota (if not specified) from this value, it can be overridden when creating the service. e.g. to create a service which provides a 20MB of persistent volume cf create-service eirini-persi default eirini-persi-1-20M-quota -c '{"size": "20M"}'

Test Persistence works

1) Check if persistence is enabled

After deployment, if everything went well, you should see eirini-persi-broker among the service brokers in the Cloud Foundry instance.

List all the brokers:

$> cf service-brokers

Getting service brokers as admin...

name                  url
eirini-persi-broker   http://eirini-eirini-persi-broker.scf.svc.cluster.local:8999

If eirini-persi-broker is showed among the list of the available brokers, it means that it is configured and plans supplied during deployment are available to be consumed.

2) Check if plans are available

To list the plans available from the broker, we need to enable it.

Create a space if you haven't already:

$> cf create-space tmp
$> cf target -s tmp

And enable the service for the space (here, doing it globally):

$> cf enable-service-access eirini-persi

Enabling access to all plans of service eirini-persi for all orgs as admin...
OK

Now the broker should be available in the marketplace, list all the broker services:

$> cf marketplace
Getting services from marketplace in org system / space tmp as admin...
OK

service        plans     description
eirini-persi   default   Eirini persistence broker

List the plans available from the broker (output might differ):

$> cf marketplace -s eirini-persi
Getting service plan information for service eirini-persi as admin...
OK

service plan   description                 free or paid
default        Eirini persistence broker   free

3) Create a Service

To verify that everything works correctly it is possible to check after creating a service that the persistent volume claims are present in the Kubernetes cluster.

Create a service:

$> cf create-service eirini-persi default eirini-persi-1
Creating service instance eirini-persi-1 in org system / space tmp as admin...
OK

List all the PersistentVolumeClaim and verify that a new one was created:

$> kubectl get pvc -n eirini
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
f9d4b977-0b9e-4fcb-a92e-652bc77dd7e7   Bound    pvc-9960cc11-7895-11e9-afac-024267c84a6b   1Gi        RWO            persistent     6s

To create a service with a custom capacity, you can define the quota when creating the service.

E.g. create a service providing a volume with 20M of quota:

$> cf create-service eirini-persi default eirini-persi-1-20M-quota -c '{"size": "20M"}'

4) Bind the service to a Cloud Foundry application

List the available services:

$> cf services
Getting services in org system / space tmp as admin...

name                          service        plan      bound apps    last operation
eirini-persi-1                eirini-persi   default   dizzylizard   create succeeded

Let's associate the service to an application (in this case dizzylizard, See Testing Workflow below) using bind-service:

$> cf bind-service dizzylizard eirini-persi-1

Binding service eirini-persi-1 to app dizzylizard in org system / space tmp as admin...
OK

Restage our app so our change takes effect:

$> cf restage dizzylizard

Restaging app dizzylizard in org system / space tmp as admin...

5) Access the Volume

Once the service is associated, the application can access to the data of the volume mount by reading the mounted path inside the VCAP_SERVICES environment variable.

An example of VCAP_SERVICES generated by eirini-persi-broker:

{"eirini-persi": [	  {
		"credentials": { "volume_id": "the-volume-id" },
		"label": "eirini-persi",
		"name": "my-instance",
		"plan": "hostpath",
		"tags": [
			"erini",
			"kubernetes",
			"storage"
		],
		"volume_mounts": [
		  {
			"container_dir": "/var/vcap/data/de847d34-bdcc-4c5d-92b1-cf2158a15b47",
			"device_type": "shared",
			"mode": "rw"
		  }
		]
	  }
	]
}

You can refer to the Cloud Foundry documentation regarding of how to access to the Volume Service, with the difference that the eirini broker will create services with the id eirini-persi.

Example

Before deployment, list the Storage Classes available in your Kubernetes cluster:

$> kubectl get storageclass

NAME                   PROVISIONER   AGE
persistent (default)   hostpath      36m

In this case, on the Kubernetes cluster it is present only one storageclass, called persistent which is the default.

To the scf-config-values, add the following EIRINI_PERSI_PLANS variable in the env block to map the storage class above:

env:
  EIRINI_PERSI_PLANS: |
      - id: "default"
        name: "default"
        description: "Eirini persistence broker"
        free: true
        kube_storage_class: "persistent"
        default_size: "2Gi"

Note You don't need to specify any provisioner in the plan section. The relevant part which provides the link to kubernetes is the kube_storage_class parameter that needs to point to the name of the storage class associated with it.

Full scf-config-values.yaml example:

env:
  # Enter the domain you created for your CAP cluster
  DOMAIN: ip.nip.io
  EIRINI_PERSI_PLANS: |
      - id: "default"
        name: "default"
        description: "Eirini persistence broker"
        free: true
        kube_storage_class: "persistent"
        default_size: "2Gi"

  # UAA host and port
  UAA_HOST: uaa.ip.nip.io
  UAA_PORT: 2793

enable:
  eirini: true

kube:
  # Run kubectl get storageclasses
  # to view your available storage classes
  storage_class:
    persistent: "persistent"
    shared: "shared"

  # The registry the images will be fetched from.
  # The values below should work for
  # a default installation from the SUSE registry.
  registry:
    hostname: ""
    username: ""
    password: ""
  organization: "cap"
  auth: rbac

secrets:
  # Create a password for your CAP cluster
  CLUSTER_ADMIN_PASSWORD: password

  # Create a password for your UAA client secret
  UAA_ADMIN_CLIENT_SECRET: password

Test workflow

$> git clone https://github.com/scf-samples/dizzylizard.git
$> cd dizzylizard
$> cf push --no-start
$> cf service-brokers
# You should see eirini-persi

$> cf enable-service-access eirini-persi
$> cf marketplace -s eirini-persi
# Default plan should be available, based on the storageclass used for install

# Each service corresponds to a volume mount for the app, in this case mounting 2 volumes:
$> cf create-service eirini-persi default eirini-persi-1
$> cf create-service eirini-persi default eirini-persi-2
$> cf bind-service dizzylizard eirini-persi-1
$> cf bind-service dizzylizard eirini-persi-2


$> cf start dizzylizard
# If you ssh to the app you would see in the env VCAP_SERVICES
# the paths in the container which gives you access to the volumes

$> kubectl get pvc -n eirini
# Verify the Persistent volume claims are bound (similarly for service deletion)

Development notes

Vagrantbox

In the vagrant box, a default plan which uses the storage class used by the k8s deployment is already registered. It is sufficient to only run make run-eirini, and no need to tweak the config value.

Debugging

eirini-persi-broker and eirini-extensions are the two components that together provides Persistence support to Eirini in SCF.

eirini-persi-broker is responsible of creating PersistentVolumeClaims, and eirini-extensions mutates Eirini application pods by binding volume mounts.

Those components are in the eirini instance group.

It is possible to inspect the status of the components internally with monit inside the eirini pod:

$> kubectl exec -ti eirini-0 -n scf /bin/bash
eirini/0:/# monit summary
The Monit daemon 5.2.5 uptime: 3h 16m

Process 'crond'                     running
File 'cron_bin'                     accessible
Directory 'cron_spool'              accessible
Process 'rsyslogd'                  running
File 'rsyslogd_bin'                 accessible
File 'rsyslog_file'                 accessible
Process 'opi'                       running
Process 'eirini-persi-broker'       running
Process 'eirini-extensions'         running
File 'post-start'                   accessible
System 'system_eirini-0.eirini-set.scf.svc.cluster.local' running
eirini/0:/#

Logs are available in /var/vcap/sys/log/eirini-extensions and /var/vcap/sys/log/eirini-persi-broker

eirini/0:/# tail -f /var/vcap/sys/log/eirini-*/*.log

Known Issues

Kubernetes fails to contact the eirini-extensions mutating webhook if they are set in mandatory mode. This will make any pod fail that is meant to be patched by eirini. An indication that this is happening is that any app being publishesd using cf push is creating timeouts. When running kubectl get events -n eirini lines of log containing

Job Warning FailedCreate job-controller Error creating: Internal error occured

are shown.

Fix for a running cluster

In order to trigger re-generation of the mutating webhook certificate, we have to delete the secrets and the associated mutating webhook:

  • run kubectl delete secret eirini-extensions-webhook-server-cert -n eirini
  • run kubectl delete mutatingwebhookconfiguration eirini-x-mutating-hook-default
  • connect to the eirini-0 pod (kubectl exec -it eirini-0 -n eirini /bin/bash)
  • run monit restart eirini-extensions

Fix on redeploy on an existing k8s (which had a scf deployed before):

  • In case of multiple re-deployments on the same cluster it can happen that old secrets are still present on the cluster. The eirinix library then tries to reuse those, resulting in a failed connection since the service will have a different IP-address.

  • Before redeploying run kubectl get secrets -n eirini, if there is an eirini-x-setupcertificate (the name may vary depending on the operator fingerprint set on the extension, see https://godoc.org/github.com/SUSE/eirinix#ManagerOptions for details) present, delete it using kubectl delete secret eirini-x-setupcertificate -n eirini

    We need also to remove the older mutatingwebhook:

    $> kubectl get mutatingwebhookconfiguration
NAME                                     CREATED AT
eirini-x-mutating-hook-default   2019-06-10T08:55:30Z

$> kubectl delete mutatingwebhookconfiguration eirini-x-mutating-hook-default
Clone this wiki locally