Add telemetry to quickstart

README.md

@@ -14,6 +14,7 @@ using Docker Compose.
     + [Run the demo app](#run-the-demo-app)
   * [Next steps](#next-steps)
   * [Explore the Conjur database](#explore-the-conjur-database)
+  * [Use Conjur with telemetry](#use-conjur-with-telemetry)
   * [Configuring Conjur with predefined admin password](#configuring-conjur-with-predefined-admin-password)
   * [Using persistent Conjur configuration](#using-persistent-conjur-configuration)
     + [Set up a Conjur Open Source environment with persistence](#set-up-a-conjur-open-source-environment-with-persistence)
@@ -346,6 +347,57 @@ To explore the database
 
 ![image](https://user-images.githubusercontent.com/8653164/115864622-03da7a00-a42f-11eb-974f-dc2cb034ca09.png)
 
+### Use Conjur with telemetry
+
+Conjur supports telemetry as an opt-in feature. The telemetry feature has a general purpose mechanism for collection, but currently only supports a single method for export, a Prometheus scrape target endpoint. Below are instructions for enabling and exploring the telemetry feature.
+
+In order to enable telemetry in Conjur you must opt-in via configuration.
+You have a choice between setting an environment variable:
+
+```sh
+CONJUR_TELEMETRY_ENABLED=true
+```
+
+or updating a value in the `conjur.conf` configuration file:
+
+```yaml
+telemetry_enabled: true
+```
+
+Note that the environment variables takes precedence.
+
+1. If you are already running the Conjur Open Source quickstart environment without
+   telemetry, bring down the Conjur container:
+
+   ```
+   docker-compose down conjur
+   ```
+
+1. Modify `docker-compose.yml` in this repository to enable telemetry
+   by setting the `CONJUR_TELEMETRY_ENABLED` environment variable to the value `'true'` (It needs to be a string otherwise the docker-compose YAML parser will not be happy). Below is an illustration of the required change:
+
+   ```yaml
+   services:
+      # ...
+      conjur:
+         environment:
+            # ...
+            CONJUR_TELEMETRY_ENABLED: 'true'
+   ```
+
+1. Start the Conjur Open Source environment using telemetry:
+
+   - If you had previously been running the Conjur Open Source environment,
+     follow the steps 2 and 3 of the
+     [Set up a Conjur Open Source environment](#set-up-a-conjur-open-source-environment)
+     section above in order to recreate the Conjur container.
+   - Otherwise, follow the steps starting from Step 1 of the
+     [Set up a Conjur Open Source environment](#set-up-a-conjur-open-source-environment)
+     section above.
+
+1. Head over to the [telemetry README](./telemetry/README.md) and follow the instructions on setting up the telemetry related services.
+   The telemetry README provides a comprehensive quickstart for setting up services such as Prometheus and Grafana, creating relevant connections between those services and the Conjur Prometheus scrape target endpoint, and providing an example dashboard with the metrics collected by Conjur.
+ 
 ### Configuring Conjur with predefined admin password
 
 The following command will allow you to specify the admin user's password:

docker-compose.yml

@@ -57,6 +57,7 @@ services:
       DATABASE_URL: postgres://postgres@database/postgres
       CONJUR_DATA_KEY:
       CONJUR_AUTHENTICATORS:
+      CONJUR_TELEMETRY_ENABLED: 'true'
     depends_on:
     - database
     restart: on-failure

telemetry/README.md

@@ -0,0 +1,70 @@
+# Conjur telemetry quickstart
+
+In order to consume the telemetry data collected by Conjur, you need to setup a Prometheus node to scrape the
+`/metrics` endpoint in Conjur. You can then use the myriad of tools that work on top of Prometheus to visualise the data. 
+
+Here, we provide a quickstart that sets up Prometheus and Graphana to allow you to quickly explore visualisation of the telemetry data collected by Conjur. To be comprehensive in our demostration, this quickstart additionally provides examples of gathering of metrics from services adjacent to Conjur such as Postgres and metrics about the node where Conjur is running. 
+
+This quickstart auto-magically carries out the following important tasks:
+1. Configure Prometheus to consume metrics from the 3 exporters (node, postgres and conjur).
+1. Configure Graphana to use Prometheus as a data source. 
+1. Create an example dashboard in Graphana that brings together all these metrics.
+
+Once the services are set-up, all the metrics will available within Graphana for visual consumption.
+
+The "important" tasks listed above are really just carried out by providing configuration files, which can be found under [./files](./files/).
+
+## Getting Started 
+
+1. Pull the Docker images
+
+   Open a terminal session and change directory to the folder containg this README. Pull the Docker
+   images defined in `docker-compose.yml`:
+   ```
+   docker-compose pull
+   ```
+
+   **Verification**
+   When the required images are successfully pulled, the terminal returns the
+   following:
+   ```
+   [+] Running 4/4
+   ⠿ node-exporter Pulled                                                                                           1.4s
+   ⠿ prometheus Pulled                                                                                              1.4s
+   ⠿ pg-prometheus-exporter Pulled                                                                                  1.4s
+   ⠿ grafana Pulled                                                                                                 1.4s
+   ```
+
+1. Start all the services. The services are the Prometheus exporters for postgres and the node (pg-prometheus-exporter and node-exporter), prometheus and graphana. 
+
+   Start the services:
+   ```
+   docker-compose up -d
+   ```
+
+   When the services start the terminal returns the following:
+   ```
+   [+] Running 4/0
+   ⠿ Container telemetry-grafana-1                 Running                                                          0.0s
+   ⠿ Container telemetry-node-exporter-1           Running                                                          0.0s
+   ⠿ Container telemetry-pg-prometheus-exporter-1  Running                                                          0.0s
+   ⠿ Container telemetry-prometheus-1              Running                                                          0.0s
+   ```
+
+   **Verification**
+   Run the following command to see a list of running containers:
+   ```
+   docker-compose ps
+   ```
+
+1. Use the Graphana UI
+
+   1. On a local browser, navigate to the Graphana UI available at [http://localhost:2345](http://localhost:2345).
+   1. Log in as `admin`/`admin`. You'll be prompted to change password, but for evaluation purposes you have the option to skip that step.
+   1. Once logged in, on the left column, select Dashboards.
+   ![](./select-dashboard.png)
+   1. Under `General` in the `Browse` tab , select `Conjur Dashboard` from the list of available dashboards. The page should then
+   be populated with the Conjur Dashboard:
+   ![](./dashboard.png)
+   1. Use Conjur as usual and observe how your activity is reflected in the updates to the Conjur dashboard.
+

telemetry/docker-compose.yml

@@ -0,0 +1,37 @@
+version: '3'
+services:
+  pg-prometheus-exporter:
+    image: quay.io/prometheuscommunity/postgres-exporter
+    environment: 
+      DATA_SOURCE_NAME: postgres://postgres@database/postgres?sslmode=disable
+      PG_EXPORTER_CONSTANT_LABELS: component=pg
+
+  prometheus:
+    image: prom/prometheus
+    volumes:
+      - ./files/prometheus:/etc/prometheus
+    ports:
+      - 9090:9090
+    command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
+
+  grafana:
+    image: grafana/grafana
+    environment:
+      - GF_SERVER_HTTP_PORT=2345
+    ports:
+      - 2345:2345
+    volumes:
+      - ./files/grafana/storage:/var/lib/grafana
+      - ./files/grafana/grafana.ini:/etc/grafana/grafana.ini
+      - ./files/grafana/provisioning:/etc/grafana/provisioning
+      - ./files/grafana/dashboards:/etc/grafana/dashboards
+
+  # Node exporter provides CPU and Memory metrics to Prometheus for the Docker
+  # host machine.
+  node-exporter:
+    image: quay.io/prometheus/node-exporter:latest
+
+networks:
+  default:
+    name: conjur-quickstart_default
+    external: true