Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing metrics on Grafana #20

Closed
omata opened this issue Jan 3, 2018 · 14 comments
Closed

Missing metrics on Grafana #20

omata opened this issue Jan 3, 2018 · 14 comments
Labels
question

Comments

@omata
Copy link

omata commented Jan 3, 2018

Hello,

I followed your instructions to get prometheus-nginxlog-exporter installed on my nginx node, so it is working and providing metrics, but there isn't any thing similar to the queries that you use in Grafana.

I'm getting the following:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.00010609
go_gc_duration_seconds{quantile="0.25"} 0.00010609
go_gc_duration_seconds{quantile="0.5"} 0.000119762
go_gc_duration_seconds{quantile="0.75"} 0.000119762
go_gc_duration_seconds{quantile="1"} 0.000119762
go_gc_duration_seconds_sum 0.000225852
go_gc_duration_seconds_count 2
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 14
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.61784e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 7.487352e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445075e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 11234
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 405504
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.61784e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.94912e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.424832e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 5961
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 5.373952e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5150157159249094e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 93
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 17195
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 27776
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 31160
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.023013e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 917504
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 917504
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 9.246968e+06
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1880.586
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 2703.691
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 2703.691
http_request_duration_microseconds_sum{handler="prometheus"} 14334.25
http_request_duration_microseconds_count{handler="prometheus"} 7
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 432
http_request_size_bytes{handler="prometheus",quantile="0.9"} 432
http_request_size_bytes{handler="prometheus",quantile="0.99"} 432
http_request_size_bytes_sum{handler="prometheus"} 2773
http_request_size_bytes_count{handler="prometheus"} 7
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 7
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 1318
http_response_size_bytes{handler="prometheus",quantile="0.9"} 1359
http_response_size_bytes{handler="prometheus",quantile="0.99"} 1359
http_response_size_bytes_sum{handler="prometheus"} 9293
http_response_size_bytes_count{handler="prometheus"} 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.02
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 7.757824e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.51501569491e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.517568e+07

I'm missing something?

Best Regards.
@martin-helmich
Copy link
Owner

Your response contains metrics like http_response_size_bytes{handler="prometheus",quantile="0.5"}, which seem to belong to Prometheus itself. Are you sure you queried the /metrics endpoint of the NGINX exporter, and not the /metrics endpoint of Prometheus itself?

@omata
Copy link
Author

omata commented Jan 4, 2018

Hello,

I'm sure, my Prometheus server is located at Google Cloud, my nginx server is on a dedicated server with other provider. Only node exporter is running at the nginx server but metrics are different from exposed by prometheus-nginxlog-exporter at port 4040.

So i don't know what is happening.

@omata
Copy link
Author

omata commented Jan 4, 2018

BTW I'm running the 64 bits version downloaded from: https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.0.1/prometheus-nginxlog-exporter-linux-amd64-static

@omata
Copy link
Author

omata commented Jan 4, 2018 •
edited
Loading

I did tests with the other binaries and I got the same result 😕
Tested versions: 1.0.0, 1.0.1 1.0.1-static

P.D. Do I need to install go language?

@martin-helmich
Copy link
Owner

My mistake, the http_response_size_bytes{handler="prometheus",quantile="0.5"} metrics and others are actually printed by the NGINX exporter, after all. Sorry for the confusion.

The exporter will only start producing NGINX metrics as soon as data has been read from an access log file. Please make sure that the access log file that you have supplied (via command-line parameter or configuration file) is present.

If you have configured your exporter to use an existing access log file, please provide the following information to help me further debug this issue:

  • Command-line flags and (if present) contents of your configuration file
  • STDOUT contents of the exporter (if running under systemd, use the command journalctl -u prometheus-nginxlog-exporter)
  • (If possible) a short excerpt from your NGINX access log file (with all sensitive information redacted, of course)

@omata
Copy link
Author

omata commented Jan 5, 2018 •
edited
Loading

The exporter will only start producing NGINX metrics as soon as data has been read from an access log file. Please make sure that the access log file that you have supplied (via command-line parameter or configuration file) is present

Yes it's present, but I don't know if the owner is related withe the issue, the file has www-data:adm ownership. It is present and increasing in size every second.

  • Command-line flags and (if present) contents of your configuration file

/etc/systemd/system/prometheus-nginxlog-exporter.service:

[Unit]
Description=NGINX metrics exporter for Prometheus
After=network-online.target

[Service]
ExecStart=/usr/local/bin/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.hcl
Restart=always
ProtectSystem=full
CapabilityBoundingSet=

[Install]
WantedBy=multi-user.target

/etc/prometheus-nginxlog-exporter.hcl

listen {
  port = 4040
  address = "127.0.0.1"
}

namespace "app" {
  source_files = [
    "/home/user/app/logs/nginx/app_access.log"
  ]
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""

  labels {
    app = "myapp"
    environment = "production"
  }
}

Content of /home/user/app/logs/nginx/app_access.log:

000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/politica/madre-sindicalista-preso-marcelo-balcedo-defendio-gritos-tv-inventaron-causa-infame_0_By2KwepQz.html" "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 6.0.1; SM-G570M Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/ciudades/tragedia-villa-crespo-mujer-murio-atropellada-auto-control_0_SyOQjQpmf.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 6.0.1; SM-G570M Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "GET /status_clip/?hash_id=y9i6 HTTP/1.1" 200 72 "https://app.com/media_map/view/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "GET /status_clip/?hash_id=y9IE HTTP/1.1" 200 67 "https://app.com/media_map/view/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "GET /player/y9Od/ HTTP/1.1" 200 5461 "https://www.filo.news/actualidad/La-mama-de-la-joven-que-murio-haciendo-parapente-A-mi-hija-la-mataron-20180105-0036.html" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G531M Build/LMY48B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.98 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 6.0.1; SM-G570M Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 7.0; Moto G (5) Plus Build/NPN25.137-83) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "GET /player/y9Oc/ HTTP/1.1" 200 5464 "https://www.filo.news/actualidad/La-mama-de-la-joven-que-murio-haciendo-parapente-A-mi-hija-la-mataron-20180105-0036.html" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G531M Build/LMY48B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.98 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "GET /player/y9Od/ HTTP/1.1" 200 5461 "https://www.filo.news/actualidad/La-mama-de-la-joven-que-murio-haciendo-parapente-A-mi-hija-la-mataron-20180105-0036.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_1 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0 Mobile/15C153 Safari/604.1"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/ciudades/tragedia-villa-crespo-mujer-murio-atropellada-auto-control_0_SyOQjQpmf.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G531M Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/148.0.0.51.62;]"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G531M Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/148.0.0.51.62;]"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/autos/flota-autos-balcedo-valuada-millones-dolares_0_ByjGkbamG.html" "Mozilla/5.0 (Linux; Android 7.0; HUAWEI VNS-L31 Build/HUAWEIVNS-L31) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:01 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G531M Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/148.0.0.51.62;]"
000.000.000.000 - - [05/Jan/2018:17:11:02 -0300] "GET /static/js/xxxxxxxx/logo.png HTTP/1.1" 200 1959 "https://vodgc.com/html5/html5lib/v2.57.1/mwEmbedFrame.php/p/102/uiconf_id/23449569/entry_id/0_zwoo4r5x?wid=_102&iframeembed=true&playerId=mediaplayer_5a4f8c515e67fe0e66487ae6_1515166687292_179&entry_id=0_zwoo4r5x&flashvars[streamerType]=auto" "Mozilla/5.0 (Linux; Android 6.0.1; MotoG3 Build/MPIS24.107-55-2-17; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/136.0.0.22.91;]"
000.000.000.000 - - [05/Jan/2018:17:11:02 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Linux; Android 5.0.1; ALE-L23 Build/HuaweiALE-L23) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36"
000.000.000.000 - - [05/Jan/2018:17:11:02 -0300] "POST /users/activity/ HTTP/1.1" 200 33 "https://www.xxxxxxxx.com/nnnnn/sss/titulo_0_xyxyxyxyxyxy.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"

SORRY I FORGOT SOMETHING

  • STDOUT contents of the exporter (if running under systemd, use the command journalctl -u prometheus-nginxlog-exporter)
Jan 04 16:21:24 server prometheus-nginxlog-exporter[15611]: loading configuration file /etc/prometheus-nginxlog-exporter.hcl
Jan 04 16:21:24 server prometheus-nginxlog-exporter[15611]: &{{4040 127.0.0.1} {false     {  []}} [{myapp [/home/username/app-server/logs/nginx/app_access.log] $remote_addr - $re
Jan 04 16:21:24 server prometheus-nginxlog-exporter[15611]: using configuration {{%!s(int=4040) 127.0.0.1} {%!s(bool=false)     {  []}} [{myapp [/home/username/app-server/logs/nginx/app
Jan 04 16:21:24 server prometheus-nginxlog-exporter[15611]: starting listener for namespace myapp
Jan 04 16:21:24 server prometheus-nginxlog-exporter[15611]: running HTTP server on address 127.0.0.1:4040
Jan 05 12:28:04 server systemd[1]: Stopping NGINX metrics exporter for Prometheus...
Jan 05 12:28:04 server systemd[1]: Stopped NGINX metrics exporter for Prometheus.
Jan 05 12:28:34 server systemd[1]: Started NGINX metrics exporter for Prometheus.
Jan 05 12:28:34 server prometheus-nginxlog-exporter[22448]: loading configuration file /etc/prometheus-nginxlog-exporter.hcl
Jan 05 12:28:34 server prometheus-nginxlog-exporter[22448]: using configuration {{%!s(int=4040) 127.0.0.1} {%!s(bool=false)     {  []}} [{myapp [/home/username/app-server/logs/nginx/app
Jan 05 12:28:34 server prometheus-nginxlog-exporter[22448]: starting listener for namespace myapp
Jan 05 12:28:34 server prometheus-nginxlog-exporter[22448]: running HTTP server on address 127.0.0.1:4040
Jan 05 12:38:31 server systemd[1]: Stopping NGINX metrics exporter for Prometheus...
Jan 05 12:38:31 server systemd[1]: Stopped NGINX metrics exporter for Prometheus.
Jan 05 12:38:31 server systemd[1]: Started NGINX metrics exporter for Prometheus.
Jan 05 12:38:31 server prometheus-nginxlog-exporter[21726]: loading configuration file /etc/prometheus-nginxlog-exporter.hcl
Jan 05 12:38:31 server prometheus-nginxlog-exporter[21726]: using configuration {{%!s(int=4040) 127.0.0.1} {%!s(bool=false)     {  []}} [{my [/home/username/app-server/logs/nginx/app_ac
Jan 05 12:38:31 server prometheus-nginxlog-exporter[21726]: starting listener for namespace my
Jan 05 12:38:31 server prometheus-nginxlog-exporter[21726]: running HTTP server on address 127.0.0.1:4040

@martin-helmich
Copy link
Owner

Thanks for the response. I'm starting to suspect this might be a permission-related issue. In your case, the exporter runs as root, but is started with an empty capability bounding set (CapabilityBoundingSet=), which takes away most of the privileges that root usually has.

Could you try removing the CapabilityBoundingSet= line from the systemd unit file (followed by systemctl daemon-reload and systemctl restart prometheus-nginxlog-exporter)? This will result in the exporter running with full root privileges -- which is obviously not a suitable solution for a production setup, but it's a quick&easy way to pinpoint a permissions issue.

@omata
Copy link
Author

omata commented Jan 5, 2018 •
edited
Loading

It worked!

The exporter did start to parse the log file, but in some cases I'm getting the message: error while parsing line

BTW: I deleted from format: the end of the expression \"$http_x_forwarded_for\"

@omata
Copy link
Author

omata commented Jan 5, 2018 •
edited
Loading

I'm getting only two metrics right now. Maybe because a wrong format?

app_http_response_count_total
app_http_response_size_bytes

@martin-helmich martin-helmich mentioned this issue Jan 5, 2018
Closed
@martin-helmich
Copy link
Owner

martin-helmich commented Jan 5, 2018 •
edited
Loading

Thanks for the feedback. I've opened a new issue to track the error reporting when opening files -- this is obviously a bug; if a file cannot be opened due to permissions, the exporter should terminate with a comprehensive error message, and not just keep on running.

Regarding the metrics: All other metrics (like app_http_upstream_time_seconds or app_http_response_time_seconds or app_http_response_size_bytes) will only become available when your access log contains the $upstream_response_time,
$request_time and/or $body_bytes_sent variables. These need to be enabled in the NGINX configuration (more precisely, the log_format setting) and then added to the format specified for the exporter.

@omata
Copy link
Author

omata commented Jan 5, 2018

Do you have an example on how to configure nginx to get those metrics?

And a lot of thanks for your help and great work!!!

Regards,

@martin-helmich
Copy link
Owner

Have a look at NGINX's Logging and Monitoring guide. It contains some good examples that contain the $request_time and $upstream_response_time:

    log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent"'
                             'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

@omata
Copy link
Author

omata commented Jan 5, 2018

Thank you!!!

@martin-helmich
Copy link
Owner

Happy to help 👍. Closing this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@omata @martin-helmich