Bears-R-Us · Ethan-DeBandi99 · Aug 10, 2022 · Jul 13, 2022 · Jul 14, 2022 · Jul 14, 2022
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
@@ -22,7 +22,7 @@ jobs:
     - uses: actions/checkout@v2
     - name: Install dependencies
       run: |
-        apt-get update && apt-get install -y python3-pip libhdf5-dev hdf5-tools libzmq3-dev
+        apt-get update && apt-get install -y python3-pip libhdf5-dev hdf5-tools libzmq3-dev libcurl4-openssl-dev
         python3 -m pip install types-pkg_resources versioneer
         python3 -m pip install -e .[dev]
     - name: Arkouda mypy
@@ -37,7 +37,7 @@ jobs:
     - uses: actions/checkout@v2
     - name: Install dependencies
       run: |
-        apt-get update && apt-get install -y python3-pip libhdf5-dev hdf5-tools libzmq3-dev
+        apt-get update && apt-get install -y python3-pip libhdf5-dev hdf5-tools libzmq3-dev libcurl4-openssl-dev
         python3 -m pip install versioneer
         python3 -m pip install -e .[dev]
     - name: Arkouda make doc
@@ -52,7 +52,7 @@ jobs:
       - uses: actions/checkout@v2
       - name: Install Dependencies
         run: |
-          apt-get update && apt-get install -y python3-pip libhdf5-dev hdf5-tools libzmq3-dev
+          apt-get update && apt-get install -y python3-pip libhdf5-dev hdf5-tools libzmq3-dev libcurl4-openssl-dev
           python3 -m pip install types-pkg_resources versioneer
           python3 -m pip install -e .[dev]
       - name: Arkouda flake8
@@ -77,7 +77,7 @@ jobs:
         apt-get update && apt-get install -y -V ca-certificates lsb-release wget
         wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
         apt-get install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
-        apt-get update && apt-get install -y libhdf5-dev hdf5-tools libzmq3-dev python3-pip libarrow-dev libparquet-dev
+        apt-get update && apt-get install -y libhdf5-dev hdf5-tools libzmq3-dev python3-pip libarrow-dev libparquet-dev libcurl4-openssl-dev
         echo "\$(eval \$(call add-path,/usr/lib/x86_64-linux-gnu/hdf5/serial/))" > Makefile.paths
     - name: Check python version
       run: |
@@ -104,7 +104,7 @@ jobs:
         apt-get update && apt-get install -y -V ca-certificates lsb-release wget
         wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
         apt-get install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
-        apt-get update && apt-get install -y libhdf5-dev hdf5-tools libzmq3-dev python3-pip libarrow-dev libparquet-dev
+        apt-get update && apt-get install -y libhdf5-dev hdf5-tools libzmq3-dev python3-pip libarrow-dev libparquet-dev libcurl4-openssl-dev
         echo "\$(eval \$(call add-path,/usr/lib/x86_64-linux-gnu/hdf5/serial/))" > Makefile.paths
     - name: Check chpl version
       run: |
@@ -137,7 +137,7 @@ jobs:
         apt-get update && apt-get install -y -V ca-certificates lsb-release wget
         wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
         apt-get install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
-        apt-get update && apt-get install -y libhdf5-dev hdf5-tools libzmq3-dev python3-pip libarrow-dev libparquet-dev
+        apt-get update && apt-get install -y libhdf5-dev hdf5-tools libzmq3-dev python3-pip libarrow-dev libparquet-dev libcurl4-openssl-dev
         echo "\$(eval \$(call add-path,/usr/lib/x86_64-linux-gnu/hdf5/serial/))" > Makefile.paths
     - name: Build/Install Arkouda
       run: |

diff --git a/EXTERNAL_INTEGRATION.md b/EXTERNAL_INTEGRATION.md
@@ -0,0 +1,188 @@
+# External Integration
+
+## Overview
+
+Given the crucial Exploratory Data Analysis (EDA) role Arkouda fulfills in a variety of data science workflows, coupled with the popular trend of deploying machine learning (ML) and deep learning (DL) workloads to cloud environments, enabling Arkouda to be seamlessly integrated into cloud environments such as Kubernetes is an increasingly important use case.
+
+## Design
+
+Delivering integration with external systems such as Kubernetes requires four elements, all of which are encapsulated within the [ExternalIntegration](src/ExternalIntegration.chpl) module with the exception of one enum: 
+
+1. Channel--implements logic for writing to external systems via export channels such as file systems and HTTP/S servers.
+2. ChannelParams--encapsulates configuration parameters needed by Channel objects to connect to external systems such as file systems and HTTP/S servers.
+3. register/deregister--various register and deregister functions that register/deregister Arkouda with external systems via the corresponding Channel.
+4. Enums--there are several enum classes that provide controlled vocabulary for external system and channel parameters.
+
+### Channel
+
+Channel derived classes override the Channel.write function to write the string payload parameter to an external system. For example, the HttpChannel class leverages the Chapel Curl module to write the JSON-formatted payloads used to register and deregister Arkouda with Kubernetes.
+
+### ChannelParams
+
+The ChannelParams derived classes encapsulate the metadata required to connect and write to external systems via a Channel.
+
+### register and deregister Functions
+
+The ExternalIntegration register and deregister functions encapsulate logic to (1) generate the payload required to register and deregister Arkouda with/from external systems and (2) utilize a Channel object to deliver the register/deregister payload.
+
+### Enums
+
+The following enums provide controlled vocabulary to configure external integration:
+
+1. SystemType--indicates the external system type, examples of which are KUBERNETES, CONSUL, and REDIS.
+2. ChannelType--defines the type of channel used to integrate with an external systems, examples of which are FILE and HTTP.
+3. ServiceEndpoint--indicates if the socket is for Arkouda client requests (for Arkouda server commands) or for metrics requests. 
+4. HttpRequestType, HttpRequestFormat--enums used internally within the ExternalIntegration module to configure the HttpChannel in terms of request type (e.g., POST or PUT) and request format (e.g., TEXT or JSON).
+5. Deployment--defined in the [ServerConfig](ServerConfig.chpl) module, the Deployment enum indicates whether Arkouda is deployed in a STANDARD environment (Slurm, bare metal) or KUBERNETES.
+
+## Building Arkouda with External Integration Support
+
+Since the ExternalIntegration module delegates HttpChannel registration logic to the Chaple Curl module, building Arkouda with ExternalIntegration requires the libcurl4-openssl-dev lib to be installed. For Debian and Ubuntu Linux distros, the install command is as follows:
+
+```
+sudo apt-get install libcurl4-openssl-dev
+```
+
+## Preparing External Systems for Integration
+
+### Kubernetes 
+
+The initial use case for Arkouda external integration is Kubernetes as described below. 
+
+#### Required Files for Registering with Kubernetes
+
+The Chapel Curl logic must use HTTPS to register/deregister with Kubernetes via the Kubernetes Rest API. Accordingly, SSL .crt and .key files signed with the certificate authority (CA) file configured for the target Kubernetes cluster must be deployed to all bare-metal/Slurm nodes or as a secret for Arkouda-on-Kubernetes deployments.
+
+An example of generating the required files is as follows:
+
+```
+# Generate base key file
+openssl genrsa -out arkouda.key 2048
+
+# Generate the certificate signing request (CSR)
+openssl req -new -key arkouda.key -out arkouda.csr
+
+# Sign with Kubernetes-configured CA
+sudo openssl x509 -req -in arkouda.csr -CA /etc/kubernetes/ssl/kube-ca.pem -CAkey /etc/kubernetes/ssl/kube-ca-key.pem -CAcreateserial -out arkouda.crt -days 730
+```
+
+#### Creating the Kubernetes User
+
+With the private key and signed cert file, create the arkouda user as follows:
+
+```
+kubectl config set-credentials arkouda --client-certificate=arkouda.crt --client-key=arkouda.key
+```
+
+#### Authorize read/write Access to Kubernetes Client API
+
+With the Kubernetes arkouda user and corresponding credentials composed of the arkouda.key and arkouda.crt in place, create the ClusterRoleBinding needed to authorize the arkouda user read/write access to the Kubernetes Client API.
+
+```
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: arkouda-rbac
+subjects:
+- kind: User
+  name: arkouda
+  apiGroup: rbac.authorization.k8s.io
+roleRef:
+  kind: ClusterRole #this must be Role or ClusterRole
+  name: cluster-admin # must match the name of the Role
+  apiGroup: rbac.authorization.k8s.io
+```
+
+```
+kubectl apply -f arkouda-rbac.yaml
+```
+
+Important note: while this cluster role binding is valid, there may be some environments where it is desirable to narrow the arkouda user privileges.
+
+## Use Cases and Examples
+
+### Kubernetes
+
+A stated above, integrating Arkouda with ML and DL workflows on Kubernetes is an increasingly important use case given the popularity of deploying ML/DL workflows to cloud environments generally, and Kubernetes specifically.
+
+The registerWithKubernetes function generates the JSON blob containing either the standard (if Arkouda is deployed on Kubernetes) or external (Arkouda is deployed outside of Kubernetes on Slurm or bare-metal) service definition.
+
+#### Registering Arkouda with Kubernetes
+
+There are two inner functions that registerWithKubernetes delegates to:
+
+1. registerAsInternalService--registers Arkouda as a standard Kubernetes service for Arkouda-on-Kubernetes deployments
+2. registerAsExternalService--registers Arkouda-on-Slurm or bare metal--in other words, Arkouda deployed outside of Kubernetes--as an external Kubernetes service
+
+The result of both registration inner functions is making Arkouda accessible to applications such as ML and DL workflows deployed in Kubernetes. 
+
+#### Deregistering Arkouda from Kubernetes
+
+The deregisterFromKubernetes function deletes the Kubernetes service and is triggered by the ak.shutdown() Arkouda client request.
+
+#### Kubernetes Integration Configuration Parameters
+
+The following environmental variables are required to configure Arkouda to register/deregister with Kubernetes:
+
+1. K8S_HOST--the Kubernetes API connect string
+2. NAMESPACE--Kubernetes namespace the service is deployed to
+3. KEY_FILE--TLS key file corresponding to a Kubernetes user that has service create/read/delete privileges
+4. CERT_FILE--TLS cert file corresponding to a Kubernetes user that has service create/read/delete privileges
+5. EXTERNAL_SERVICE_PORT--port Arkouda will be accessible from
+6. EXTERNAL_SERVICE_NAME--service name to access Arkouda
+
+#### Kubernetes Internal Service Registration (Kubernetes)
+
+Deployment of Arkouda-on-Kubernetes involves two Helm charts: one for the driver (locale0) pod, and one for 1..n locale (locale1...#locales-1) pods. The Helm installation process is detailed [here](https://github.com/hokiegeek2/arkouda/wiki/Arkouda-on-Docker-and-Kubernetes#deploying-multi-locale-arkouda-on-kubernetes). 
+
+Note that the ExternalIntegration.externalSystem param is SystemType.KUBERNETES and the ServerConfig deployment param is Deployment.KUBERNETES (Arkouda is deployed on Kubernetes)
+
+#### Kubernetes External Service Registration (Slurm)
+
+An example Slurm BATCH file for an Arkouda instance that registers/deregisters with Kubernetes is shown below. Note that the ExternalIntegration.externalSystem param is SystemType.KUBERNETES and the deployment param is not specified because Slurm is considered a DEFAULT deployment type.
+
+```
+#!/bin/bash
+#
+#SBATCH --job-name=arkouda-3-node
+#SBATCH --output=/tmp/arkouda.out
+#SBATCH --mem=1024
+#SBATCH --ntasks=3
+#SBATCH --nodes=3
+
+export CHPL_COMM_SUBSTRATE=udp
+export GASNET_MASTERIP='server1'
+export SSH_SERVERS='server1 server2 server3'
+export GASNET_SPAWNFN=S
+
+export NAMESPACE=arkouda
+export K8S_HOST=https://ace:6443 #result from kubectl cluster-info command
+export EXTERNAL_SERVICE_NAME=arkouda-external
+export EXTERNAL_SERVICE_PORT=5555
+export KEY_FILE=/opt/arkouda/tls.key #on all slurm hosts
+export CERT_FILE=/opt/arkouda/tls.crt #on all slurm hosts
+export CACERT_FILE=/etc/kubernetes/ssl/kube-ca.pem #on slurm hosts
+
+./arkouda_server -nl 3 --ExternalIntegration.systemType=SystemType.KUBERNETES \
+                       --ServerDaemon.daemonTypes=ServerDaemonType.INTEGRATION
+```
+
+#### Kubernetes External Service Registration (Bare Metal)
+
+An example bare metal deployment script for an Arkouda instance that registers/deregisters with Kubernetes is shown below. As is the case with the Arkouda-on-Slurm deployment, the ExternalIntegration.externalSystem param is SystemType.KUBERNETES and the deployment param is not specified because bare metal is considered a DEFAULT deployment type.
+
+```
+#!/bin/bash
+
+export GASNET_MASTERIP='server1'
+export SSH_SERVERS='server1 server2 server3'
+export NAMESPACE=arkouda
+export EXTERNAL_SERVICE_NAME=arkouda-external
+export EXTERNAL_SERVICE_PORT=5555 
+export K8S_HOST=https://ace:6443 #result from kubectl cluster-info command
+export KEY_FILE=/opt/arkouda.key #on all bare metal hosts
+export CERT_FILE=/opt/arkouda.crt #on all bare metal hosts
+
+./arkouda_server -nl 3 --ExternalIntegration.systemType=SystemType.KUBERNETES \
+                       --ServerDaemon.daemonTypes=ServerDaemonType.INTEGRATION
+```
diff --git a/README.md b/README.md
@@ -101,8 +101,8 @@ This yielded a >20TB dataframe in Arkouda.
 6. [Type Checking in Arkouda](#typecheck-ak)
 7. [Environment Variables](#env-vars-ak)
 8. [Versioning](#versioning-ak)
-9. [Contributing](#contrib-ak)
-
+9. [External Systems Integration](#external-integration)
+10. [Contributing](#contrib-ak)
 
 <a id="prereqs"></a>
 ## Prerequisites <sup><sup><sub><a href="#toc">toc</a></sub></sup></sup>
@@ -363,6 +363,10 @@ python -m arkouda |tail -n 2
 git push --tags
 ```
 
+<a id="external-integration"></a>
+## External Systems Integration <sup><sup><sub><a href="#toc">toc</a></sub></sup></sup>
+
+Integrating Arkouda with cloud environments enables users to access Arkouda from machine learning (ML) and deep learning (DL) workflows deployed to Kubernetes as an example. Detailed discussions regarding Arkouda systems integration and specific instructions for registering/deregistering Arkouda with Kubernetes are located in [EXTERNAL INTEGRATION](EXTERNAL_INTEGRATION.md)
 <a id="contrib-ak"></a>
 ## Contributing to Arkouda <sup><sup><sub><a href="#toc">toc</a></sub></sup></sup>