Chore: Development sandbox Docker Compose configuration; Tests: Fix PROMETHEUS_URL; deactivate failing test; Misc.

DEVELOP.rst

@@ -10,16 +10,16 @@ Building from source
 ====================
 
 To build the CrateDB Prometheus Adapter from source, you need to have a working
-Go environment with **Golang version 1.16** installed.
+Go environment with a recent version of Golang installed.
 
 Use the ``go`` tool to download and install the ``cratedb-prometheus-adapter``
 executable into your ``GOPATH``::
 
    go get github.com/crate/cratedb-prometheus-adapter
 
 
-Setup sandbox
-=============
+Set up Sandbox
+==============
 
 For working on the source code, it is advised to setup a development sandbox.
 To start things off, clone the repository and change into the newly checked out
@@ -74,17 +74,58 @@ Run integration tests::
 
     pytest
 
-Ad hoc deployment
-=================
+Sandbox Deployment
+==================
+
+You can also use a Docker Compose service definition to run both CrateDB
+and Prometheus on your workstation, for evaluation and testing purposes.
+
+The setup will sample metrics each five seconds, for both ``prometheus`` and
+``cratedb`` collection jobs::
+
+    docker compose --file sandbox/docker-compose.yaml up
+
+.. note::
+
+    Please note that the Docker Compose configuration for the integration
+    testing harness is not suitable for regular operations, because it uses
+    a very high sampling rate. As such, it will significantly consume disk
+    space when it is run for a longer time.
+
+Provision database schema::
+
+    crash < sql/ddl.sql
+
+This currently needs to be accompanied by running the Prometheus Adapter
+service on your workstation, like::
+
+    go run .
+
+Navigate to Prometheus UI, and verify that both endpoints are connected::
+
+    open http://localhost:9090/targets
+
+Set up Python sandbox::
+
+    python3 -m venv .venv
+    source .venv/bin/activate
+    pip install -r requirements-test.txt
+
+Submit Prometheus query expressions and display results::
+
+    python sandbox/promquery.py
+
+Background Services
+-------------------
 
-You can also use the Docker Compose service definition to run both CrateDB
-and Prometheus in the background, by using DC's ``--detach`` option::
+In order to start the services in the background, use DC's ``--detach``
+option::
 
-    docker compose --file tests/docker-compose.yaml up --detach
+    docker compose --file sandbox/docker-compose.yaml up --detach
 
-Then, for inspecting the log output of the background services, use this command::
+Then, for inspecting the log output of the background services, run::
 
-    docker compose --file tests/docker-compose.yaml logs --follow
+    docker compose --file sandbox/docker-compose.yaml logs --follow
 
 Maintaining dependencies
 ========================

backlog.md

@@ -8,9 +8,14 @@
   - https://prometheus.io/docs/prometheus/latest/querying/examples/
 - Expose metrics about both database connection pools
   https://github.com/crate/cratedb-prometheus-adapter/pull/105
- 
+
 ## Iteration +2
 - Document how to connect to CrateDB Cloud
+- Log flooding:
+  ```log
+  ts=2024-01-14T00:27:24.941Z caller=server.go:349 level=error msg="Failed to write data to CrateDB" err="error closing write batch: error preparing write statement: ERROR: Relation 'metrics' unknown (SQLSTATE 42P01)"
+  ```
+- Implement "Subquery": https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery
 - Improve documentation
   https://community.crate.io/t/storing-long-term-metrics-with-prometheus-in-cratedb/1012
 

pyproject.toml

@@ -12,7 +12,7 @@ addopts = """
 env = [
     "CRATEDB_CONNECTION_STRING=crate://crate@localhost/?ssl=false",
     "CRATEDB_PROMETHEUS_ADAPTER_METRICS_URL=http://localhost:9268/metrics",
-    "PROMETHEUS_URL=http://localhost:9090/",
+    "PROMETHEUS_URL=http://localhost:9090",
 ]
 
 log_level = "DEBUG"

sandbox/docker-compose.yaml

@@ -0,0 +1,34 @@
+version: '3.0'
+
+services:
+
+  cratedb:
+    image: crate/crate:latest
+    ports:
+      - "4200:4200/tcp"
+      - "5432:5432/tcp"
+
+    # Make `host.docker.internal` work on Linux.
+    # https://stackoverflow.com/a/67158212
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  prometheus:
+    image: prom/prometheus:main
+    ports:
+      - "9090:9090/tcp"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+
+    # Make `host.docker.internal` work on Linux.
+    # https://stackoverflow.com/a/67158212
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  cratedb-version:
+    image: crate/crate:nightly
+    command: -version
+
+  prometheus-version:
+    image: prom/prometheus:main
+    command: --version

sandbox/prometheus.yml

@@ -0,0 +1,75 @@
+global:
+  # By default, scrape targets every 15 seconds.
+  scrape_interval: 15s
+
+  # Attach these labels to any time series or alerts when communicating with
+  # external systems (federation, remote storage, Alertmanager).
+  external_labels:
+    subsystem: 'database-sandbox'
+
+# Scrape configurations for two endpoints: It scrapes metrics from both
+# Prometheus, and from CrateDB Prometheus Adapter.
+
+# The job name will be added as a label `job=<job_name>` to any
+# time-series scraped from the corresponding configuration item.
+
+# `scrape_interval` overrides the global default scrape interval for
+# individual targets. For testing purposes, this value is tuned down
+# to permit fast metrics convergence.
+scrape_configs:
+
+  - job_name: 'prometheus'
+    scrape_interval: 5s
+    static_configs:
+      - targets: ['localhost:9090']
+
+  - job_name: 'cratedb'
+    scrape_interval: 5s
+    static_configs:
+      - targets: ['host.docker.internal:9268']
+
+remote_write:
+
+  # The URL of the endpoint to send samples to.
+
+  # When Prometheus is running on Docker, and the Prometheus adapter on localhost.
+  - url: http://host.docker.internal:9268/write
+
+  # When the Prometheus adapter is also running on Docker.
+  #- url: http://cratedb-prometheus-adapter:9268/write
+
+  # When the Prometheus adapter is running on localhost.
+  #- url: http://localhost:9268/write
+
+    # Timeout for requests to the remote write endpoint.
+    remote_timeout: 5s
+
+    # Configure the sending of series metadata to remote storage.
+    metadata_config:
+      # Configure how frequently metric metadata is sent to remote storage.
+      send_interval: 1s
+
+    # Configure the queue used to write to remote storage.
+    # For testing purposes, the deadline value is tuned down to avoid buffering.
+    queue_config:
+      # Maximum time a sample will wait in buffer.
+      batch_send_deadline: 1s
+      # Minimum and maximum number of shards, i.e. amount of concurrency.
+      min_shards: 1
+      max_shards: 1
+      # Initial and maximum retry delay. `min` gets doubled for every retry.
+      # min_backoff: 1ms
+      # max_backoff: 1ms
+
+remote_read:
+  # When Prometheus is running on Docker, and the Prometheus adapter on localhost.
+  - url: http://host.docker.internal:9268/read
+
+  # When the Prometheus adapter is also running on Docker.
+  #- url: http://cratedb-prometheus-adapter:9268/read
+
+  # When the Prometheus adapter is running on localhost.
+  #- url: http://localhost:9268/read
+
+    # Timeout for requests to the remote read endpoint.
+    remote_timeout: 5s

sandbox/promquery.py

@@ -0,0 +1,60 @@
+"""
+## About
+Submit Prometheus query expressions and display results.
+Similar to `promtool query`.
+
+## Setup
+```
+python3 -m venv .venv
+source .venv/bin/activate
+pip install --upgrade prometheus-api-client
+```
+
+## Synopsis
+```
+python sandbox/promquery.py 'foobar'
+python sandbox/promquery.py 'prometheus_http_requests_total'
+python sandbox/promquery.py 'prometheus_http_requests_total{code!="200"}'
+python sandbox/promquery.py 'prometheus_http_requests_total{code!~"2.."}'
+python sandbox/promquery.py 'rate(prometheus_http_requests_total[5m])[30m:1m]'
+```
+
+## Resources
+- https://pypi.org/project/prometheus-api-client/
+- https://prometheus.io/docs/prometheus/latest/querying/examples/
+- https://github.com/crate/cratedb-prometheus-adapter/blob/0.5.0/server.go#L124-L160
+"""
+import sys
+from pprint import pprint
+
+from prometheus_api_client import (
+    PrometheusConnect,
+)
+
+
+class PrometheusAdapter:
+
+    def __init__(self, url: str, disable_ssl: bool = False):
+        self.url = url
+        self.disable_ssl = disable_ssl
+        self.prometheus: PrometheusConnect
+
+    def connect(self):
+        self.prometheus = PrometheusConnect(url=self.url, disable_ssl=self.disable_ssl)
+        return self
+
+    def query(self, expression: str):
+        # Default timeout is 5 seconds. Perfect.
+        return self.prometheus.custom_query(query=expression)
+
+
+def run_query(url: str, expression: str):
+    prom = PrometheusAdapter(url=url, disable_ssl=True).connect()
+    metrics = prom.query(expression=expression)
+    pprint(metrics)
+
+
+if __name__ == "__main__":
+    url = "http://localhost:9090/"
+    expression = sys.argv[1]
+    run_query(url, expression)

tests/conftest.py

@@ -27,6 +27,8 @@ def adapter_metrics():
         "cratedb_prometheus_adapter_read_timeseries_samples_count",
         "cratedb_prometheus_adapter_write_crate_failed_total",
         "cratedb_prometheus_adapter_write_crate_latency_seconds_bucket",
+        "cratedb_prometheus_adapter_write_crate_latency_seconds_sum",
+        "cratedb_prometheus_adapter_write_crate_latency_seconds_count",
         "cratedb_prometheus_adapter_write_failed_total",
         "cratedb_prometheus_adapter_write_latency_seconds_bucket",
         "cratedb_prometheus_adapter_write_latency_seconds_sum",

tests/prometheus.yml

@@ -5,7 +5,7 @@ global:
   # Attach these labels to any time series or alerts when communicating with
   # external systems (federation, remote storage, Alertmanager).
   external_labels:
-    subsystem: 'database'
+    subsystem: 'database-testdrive'
 
 # Scrape configurations for two endpoints: It scrapes metrics from both
 # Prometheus, and from CrateDB Prometheus Adapter.

tests/test_adapter.py

@@ -1,4 +1,5 @@
 import os
+import socket
 from unittest import mock
 
 import pytest
@@ -101,6 +102,7 @@ def test_verify_write_activity(prometheus_client):
     assert value > 1_000
 
 
+@pytest.mark.skipif("sink" in socket.gethostname(), reason="Does not work on developer workstation")
 def test_verify_read_activity(prometheus_client, flush_database):
     """
     Verify the read-path works well.