Prometheus is an excellent systems monitoring and alerting toolkit, which uses a pull model for collecting metrics. The pull model is problematic when a firewall separates a Prometheus server and its metrics endpoints. Prometheus Proxy enables Prometheus to reach metrics endpoints running behind a firewall and preserves the pull model.
The prometheus-proxy
runtime comprises 2 services:
proxy
: runs in the same network domain as Prometheus server (outside the firewall) and proxies calls from Prometheus to theagent
behind the firewall.agent
: runs in the same network domain as all the monitored hosts/services/apps (inside the firewall). It maps the scraping queries coming from theproxy
to the actual/metrics
scraping endpoints of the hosts/services/apps.
Here's a simplified network diagram of how the deployed proxy
and agent
work:
Endpoints running behind a firewall require a prometheus-agent
(the agent) to be run inside the firewall. An agent can
run as a stand-alone server, embedded in another java server, or as a java agent. Agents connect to
a prometheus-proxy
(the proxy) and register the paths for which they will provide data. One proxy can work one or many
agents.
Requires Java 11 or newer.
Download the proxy and agent uber-jars from here.
Start a proxy
with:
java -jar prometheus-proxy.jar
Start an agent
with:
java -jar prometheus-agent.jar -Dagent.proxy.hostname=mymachine.local --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf
If the prometheus-proxy were running on a machine named mymachine.local and the
agent.pathConfigs
value in
the myapps.conf
config file had the contents:
agent {
pathConfigs: [
{
name: "App1 metrics"
path: app1_metrics
labels: "{\"key1\": \"value1\", \"key2\": 2}"
url: "http://app1.local:9100/metrics"
},
{
name: "App2 metrics"
path: app2_metrics
labels: "{\"key3\": \"value3\", \"key4\": \"value4\"}"
url: "http://app2.local:9100/metrics"
},
{
name: "App3 metrics"
path: app3_metrics
labels: "{\"key5\": \"value5\", \"key6\": \"value6\"}"
url: "http://app3.local:9100/metrics"
}
]
}
then the prometheus.yml scrape_config would target the three apps with:
- http://mymachine.local:8080/app1_metrics
- http://mymachine.local:8080/app2_metrics
- http://mymachine.local:8080/app3_metrics
If the endpoints were restricted with basic auth/bearer authentication, you could either include the basic-auth credentials in the URL with: http://user:pass@hostname/metrics
or they could be configured with basic_auth
/bearer_token
in the scrape-config.
The prometheus.yml
file would include:
scrape_configs:
- job_name: 'app1 metrics'
metrics_path: '/app1_metrics'
bearer_token: 'eyJ....hH9rloA'
static_configs:
- targets: [ 'mymachine.local:8080' ]
- job_name: 'app2 metrics'
metrics_path: '/app2_metrics'
basic_auth:
username: 'user'
password: 's3cr3t'
static_configs:
- targets: [ 'mymachine.local:8080' ]
- job_name: 'app3 metrics'
metrics_path: '/app3_metrics'
static_configs:
- targets: [ 'mymachine.local:8080' ]
The docker images are available via:
docker pull pambrose/prometheus-proxy:1.23.0
docker pull pambrose/prometheus-agent:1.23.0
Start a proxy container with:
docker run --rm -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080 \
--env ADMIN_ENABLED=true \
--env METRICS_ENABLED=true \
pambrose/prometheus-proxy:1.23.0
Start an agent container with:
docker run --rm -p 8083:8083 -p 8093:8093 \
--env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
pambrose/prometheus-agent:1.23.0
Using the config file simple.conf, the proxy and the agent metrics would be available from the proxy on localhost at:
If you want to use a local config file with a docker container (instead of the above HTTP-served config file), use the
docker mount option. Assuming the config file prom-agent.conf
is in your current directory, run an agent container with:
docker run --rm -p 8083:8083 -p 8093:8093 \
--mount type=bind,source="$(pwd)"/prom-agent.conf,target=/app/prom-agent.conf \
--env AGENT_CONFIG=prom-agent.conf \
pambrose/prometheus-agent:1.23.0
Note: The WORKDIR
of the proxy and agent images is /app
, so make sure to use /app
as the base directory in the
target for --mount
options.
If you are running a JVM-based program, you can run with the agent embedded directly in your app and not use an external agent:
EmbeddedAgentInfo agentInfo = startAsyncAgent("configFile.conf", true);
The proxy and agent use the Typesafe Config library for configuration. Highlights include:
- support for files in three formats: Java properties, JSON, and a human-friendly JSON superset (HOCON)
- config files can be files or urls
- config values can come from CLI options, environment vars, Java system properties, and/or config files.
- config files can reference environment variables
All the proxy and agent properties are
described here. The only required
argument is an agent config value, which should have an agent.pathConfigs
value.
Options | ENV VAR Property |
Default | Description |
---|---|---|---|
--config, -c | PROXY_CONFIG | Agent config file or url | |
--port, -p | PROXY_PORT proxy.http.port |
8080 | Proxy listen port |
--agent_port, -a | AGENT_PORT proxy.agent.port |
50051 | gRPC listen port for agents |
--admin, -r | ADMIN_ENABLED proxy.admin.enabled |
false | Enable admin servlets |
--admin_port, -i | ADMIN_PORT proxy.admin.port |
8092 | Admin servlets port |
--debug, -b | DEBUG_ENABLED proxy.admin.debugEnabled |
false | Enable proxy debug servlet on admin port |
--metrics, -e | METRICS_ENABLED proxy.metrics.enabled |
false | Enable proxy metrics |
--metrics_port, -m | METRICS_PORT proxy.metrics.port |
8082 | Proxy metrics listen port |
--sd_enabled | SD_ENABLED proxy.service.discovery.enabled |
false | Service discovery endpoint enabled |
--sd_path | SD_PATH proxy.service.discovery.path |
"discovery" | Service discovery endpoint path |
--sd_target_prefix | SD_TARGET_PREFIX proxy.service.discovery.targetPrefix |
"http://localhost:8080/" | Service discovery target prefix |
--tf-disabled | TRANSPORT_FILTER_DISABLED proxy.transportFilterDisabled |
false | Transport filter disabled |
--ref-disabled | REFLECTION_DISABLED proxy.reflectionDisabled |
false | gRPC Reflection disabled |
--cert, -t | CERT_CHAIN_FILE_PATH proxy.tls.certChainFilePath |
Certificate chain file path | |
--key, -k | PRIVATE_KEY_FILE_PATH proxy.tls.privateKeyFilePath |
Private key file path | |
--trust, -s | TRUST_CERT_COLLECTION_FILE_PATH proxy.tls.trustCertCollectionFilePath |
Trust certificate collection file path | |
--version, -v | Print version info and exit | ||
--usage, -u | Print usage message and exit | ||
-D | Dynamic property assignment |
Options | ENV VAR Property |
Default | Description |
---|---|---|---|
--config, -c | AGENT_CONFIG | Agent config file or url (required) | |
--proxy, -p | PROXY_HOSTNAME agent.proxy.hostname |
Proxy hostname (can include :port) | |
--name, -n | AGENT_NAME agent.name |
Agent name | |
--admin, -r | ADMIN_ENABLED agent.admin.enabled |
false | Enable admin servlets |
--admin_port, -i | ADMIN_PORT agent.admin.port |
8093 | Admin servlets port |
--debug, -b | DEBUG_ENABLED agent.admin.debugEnabled |
false | Enable agent debug servlet on admin port |
--metrics, -e | METRICS_ENABLED agent.metrics.enabled |
false | Enable agent metrics |
--metrics_port, -m | METRICS_PORT agent.metrics.port |
8083 | Agent metrics listen port |
--consolidated, -o | CONSOLIDATED agent.consolidated |
false | Enable multiple agents per registered path |
--timeout | SCRAPE_TIMEOUT_SECS agent.scrapeTimeoutSecs |
15 | Scrape timeout time (seconds) |
--max_retries | SCRAPE_MAX_RETRIES agent.scrapeMaxRetries |
0 | Scrape maximum retries (0 disables scrape retries) |
--chunk | CHUNK_CONTENT_SIZE_KBS agent.chunkContentSizeKbs |
32 | Threshold for chunking data to Proxy and buffer size (KBs) |
--gzip | MIN_GZIP_SIZE_BYTES agent.minGzipSizeBytes |
1024 | Minimum size for content to be gzipped (bytes) |
--tf-disabled | TRANSPORT_FILTER_DISABLED agent.transportFilterDisabled |
false | Transport filter disabled |
--trust_all_x509 | TRUST_ALL_X509_CERTIFICATES agent.http.enableTrustAllX509Certificates |
false | Disable SSL verification for agent https endpoints |
--cert, -t | CERT_CHAIN_FILE_PATH agent.tls.certChainFilePath |
Certificate chain file path | |
--key, -k | PRIVATE_KEY_FILE_PATH agent.tls.privateKeyFilePath |
Private key file path | |
--trust, -s | TRUST_CERT_COLLECTION_FILE_PATH agent.tls.trustCertCollectionFilePath |
Trust certificate collection file path | |
--override | OVERRIDE_AUTHORITY agent.tls.overrideAuthority |
Override authority (for testing) | |
--version, -v | Print version info and exit | ||
--usage, -u | Print usage message and exit | ||
-D | Dynamic property assignment |
Misc notes:
- If you want to customize the logging, include the java arg
-Dlogback.configurationFile=/path/to/logback.xml
- JSON config files must have a .json suffix
- Java Properties config files must have a .properties or .prop suffix
- HOCON config files must have a .conf suffix
- Option values are evaluated in the order: CLI, environment vars, and finally config file vals
- Property values can be set as a java -D arg to or as a proxy or agent jar -D arg
- For more information about the proxy service discovery options, see the Prometheus documentation
- A pathConfig
labels
value is a quote-escaped JSON string with key/value pairs. It is used to add additional service discovery context to a target.
These admin servlets are available when the admin servlet is enabled:
- /ping
- /threaddump
- /healthcheck
- /version
The admin servlets can be enabled with the ADMIN_ENABLED
environment var, the --admin
CLI option, or with the
proxy.admin.enabled
and agent.admin.enabled
properties.
The debug servlet can be enabled with the DEBUG_ENABLED
environment var, the --debug
CLI option , or with the
proxy.admin.debugEnabled
and agent.admin.debugEnabled
properties. The debug servlet requires that the admin servlets
are enabled. The debug servlet is at: /debug
on the admin port.
Descriptions of the servlets are here. The path names can be changed in the configuration file. To disable an admin servlet, assign its property path to "".
Agents connect to a proxy using gRPC. gRPC supports TLS with or without mutual authentication. The necessary certificate and key file paths can be specified via CLI args, environment variables and configuration file settings.
The gRPC docs describe how to set up TLS. The repo includes the certificates and keys necessary to test TLS support.
Running TLS without mutual authentication requires these settings:
certChainFilePath
andprivateKeyFilePath
on the proxytrustCertCollectionFilePath
on the agent
Running TLS with mutual authentication requires these settings:
certChainFilePath
,privateKeyFilePath
andtrustCertCollectionFilePath
on the proxycertChainFilePath
,privateKeyFilePath
andtrustCertCollectionFilePath
on the agent
Run a proxy and an agent with TLS (no mutual auth) using the included testing certs and keys with:
java -jar prometheus-proxy.jar --config examples/tls-no-mutual-auth.conf
java -jar prometheus-agent.jar --config examples/tls-no-mutual-auth.conf
Run a proxy and an agent docker container with TLS (no mutual auth) using the included testing certs and keys with:
docker run --rm -p 8082:8082 -p 8092:8092 -p 50440:50440 -p 8080:8080 \
--mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
--mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
--env PROXY_CONFIG=tls-no-mutual-auth.conf \
--env ADMIN_ENABLED=true \
--env METRICS_ENABLED=true \
pambrose/prometheus-proxy:1.23.0
docker run --rm -p 8083:8083 -p 8093:8093 \
--mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
--mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
--env AGENT_CONFIG=tls-no-mutual-auth.conf \
--env PROXY_HOSTNAME=mymachine.lan:50440 \
--name docker-agent \
pambrose/prometheus-agent:1.23.0
Note: The WORKDIR
of the proxy and agent images is /app
, so make sure to use /app
as the base directory in the
target for --mount
options.
Disable SSL verification for agent https endpoints with the TRUST_ALL_X509_CERTIFICATES
environment var,
the --trust_all_x509
CLI option, or the agent.http.enableTrustAllX509Certificates
property.
It's possible to scrape an existing prometheus server using the /federate
endpoint.
This enables using the existing service discovery features already built into Prometheus.
An example config can be found in federate.conf.
To use the prometheus_proxy with nginx as a reverse proxy, disable the transport filter with the
TRANSPORT_FILTER_DISABLED
environment var, the --tf-disabled
CLI option, or the agent.transportFilterDisabled
/
proxy.transportFilterDisabled
properties. Agents and the Proxy must run with the same transporFilterDisabled
value.
When using transporFilterDisabled
, you will not see agent contexts immediately removed
from the proxy when agents are terminated. Instead, agent contexts will be removed from the proxy
after they age out from inactivity. The maximum age is controlled by the proxy.internal.maxAgentInactivitySecs
value.
The default value is 1 minute.
An example nginx conf file is here and an example agent/proxy conf file is here
The gRPC Reflection service is enabled by default.
To disable gRPC Reflection support, set the REFLECTION_DISABLED
environment var,
the --reflection_disabled
CLI option, or the proxy.reflectionDisabled
property to true.
To use grpcurl to test the reflection service, run:
grpcurl -plaintext localhost:50051 list
If you use the grpcurl -plaintext
option, make sure that you run the proxy in plaintext
mode, i.e., do not define any TLS properties.