Ceph SkyhookDM cluster setup

This document describes how to setup a Ceph (luminous) cluster with Skyhook Data Management extensions. There are 3 options:

Use one of our public Cloudlab cluster profiles, then do a full install of Ceph (lumionus) with SkyhookDM extensions. This will install a completely vanilla version of Ceph Luminous, then compile and inject our SkyhookDM extensions into the running cluster as a shared library on each node.
Use your own blank cluster that meets the below requirements, then do a full install of Ceph (lumionus) with SkyhookDM extensions. This will install a completely vanilla version of Ceph Luminous, then compile and inject our SkyhookDM extensions into the running cluster as a shared library on each node.
Use an existing Ceph cluster (luminous) and just add the SkyhookDM extensions to your running cluster. This will only compile and inject our SkyhookDM extensions into the running cluster as a shared library on each node.

Use one of our Cloudlab profiles

Boot any one of these.
You can now do a full install of Ceph (lumionus) with SkyhookDM extensions.

Use your own blank cluster

Requirements below for running our setup script using your own cluster, Ceph not yet installed.

Supported OS (Jan. 2019):

Ubuntu18 or Centos7

1 head node and 1 or more storage nodes.

The head node stores no data but acts as a client, a Ceph MGR and a Ceph MON. Ceph is installed on all nodes.

Disk space to compile Ceph extensions locally.

Requires about 4GB in the user's home directory on the head node only (referred to as client0).

An available full storage device on each of the storage nodes.

A full, non-partitioned storage-device (example: /dev/sdb) that will be clobbered then used for Ceph data storage. Device partitions such as dev/sdb1 are not recommended. To set this device in the install script, use the storage-device parameter.

Passwordless sudo on each machine

for the user running the installation script, as well as ssh access from the head node (client0) to all other nodes (osd0--osdn).

Machine hostnames resolvable as:

client0,osd0,osd1,osd2,...,osdn
- NOTE: This can be easily achieved by adding/augmenting lines in the /etc/hosts file of each machine such as:

10.X.X.x  current-name client0
10.X.X.y  current-name osd0
...

You can now do a full install of Ceph (lumionus) with SkyhookDM extensions.

Use an existing Ceph cluster

Assumes Ceph luminous version. Just build and deploy our extensions library as below, there is no need to stop or restart the cluster.

Clone and build SkyhookDM as per our Build instructions, requires ~10 GB of disk space.
Copy the skyhookdm library file into the libcls directory on each of the OSDs, requires sudo
- Centos7: LIB_CLS_DIR=/usr/lib64/rados-classes/
- Ubuntu18: LIB_CLS_DIR=/usr/lib/x86_64-linux-gnu/rados-classes/

# from the BUILD dir:
for ((i = 0 ; i < $nosds ; i++)); do
    echo "copying shared lib to osd$i;"
    scp ./lib/libcls_tabular.so.1.0.0  osd$i:/tmp/;
    ssh  osd$i "sudo cp /tmp/libcls_tabular.so.1.0.0 ${LIB_CLS_DIR};";
    ssh  osd$i "cd ${LIB_CLS_DIR}; if test -f libcls_tabular.so.1; then sudo  unlink libcls_tabular.so.1; fi";
    ssh  osd$i "cd ${LIB_CLS_DIR}; if test -f libcls_tabular.so; then sudo unlink libcls_tabular.so; fi";
    ssh  osd$i "cd ${LIB_CLS_DIR}; sudo ln -s libcls_tabular.so.1.0.0 libcls_tabular.so.1;";
    ssh  osd$i "cd ${LIB_CLS_DIR}; sudo ln -s libcls_tabular.so.1 libcls_tabular.so;";
done;

Run some test queries

You can now go to the Run test queries page.

Full install of Ceph (lumionus) with SkyhookDM extensions

Assumes an available cluster of 2+ nodes, either Cloudlab profile or one of your own that meets the above requirements.

Copy your ssh key over to your home dir on client0, this should be a key that can access each of the machines, then login to that machine.

scp -i ~/.ssh/mykey.id_rsa  ~/.ssh/mykey.id_rsa client0:~/;
ssh -i ~/.ssh/mykey.id_rsa client0;

From ${HOME} on client0 get the install script.

wget https://raw.githubusercontent.com/uccross/skyhookdm-ceph/skyhook-luminous/src/progly/cloudlab_cluster_setup/skyhookdm-ceph-install-wrapper.sh;

Run the install script specifying ubuntu or centos; typically takes < 15 minutes for a 1--8 node cluster.

bash skyhookdm-ceph-install-wrapper.sh <nosds> <clusterkey> <ubuntu|centos> <storage-device>;
Example:
bash skyhookdm-ceph-install-wrapper.sh 8 myclusterkey.id_rsa  ubuntu sdb;