probe_ssl_earliest_cert_expiry randomly flips between giving right and wrong result #1047
Comments
|
I installed ssl_prober and got the same result. Due to ssl_prober's extra labels, I could see that two certificates with two different serial number are in play. In other words, Apache is somehow sometimes presenting one certificate and sometimes another. This no longer seems like a bug in blackbox_exporter. |
|
Hi @pieter-lautus, we get different results for a wildcard certificate in Apache, could you please add a link to the ssl-prober tool you used? |
|
@mfriedenhagen I closed this issue on 23 March, but I see now I was not particularly clear why. This was an apache issue, not a ssl-prober or blackbox-exporter issue. For some reason, when the certificate was changed, apache reload left one process with the old configuration running, and kept routing requests to it. So some of our requests were responded to with an old certificate, and some with a new one. blackbox_exporter was accurately reporting on an apache weirdness. This is not a blackbox_exporter bug. |
|
Hi @pieter-lautus, I was obviously not clear enough: I would just like to get a link to the ssl-prober tool you used to see whether my problem is the same you reported here. I am pretty sure that blackbox-exporter is innocent :-). I would have contacted you directly for this information if this would be possible via GitHub. |
|
Maybe this one: https://github.com/ribbybibby/ssl_exporter? |
|
@mfriedenhagen Yes, I was using https://github.com/ribbybibby/ssl_exporter Sorry, I though you were a maintainer that was about to waste time investigating an issue I had already closed. |
Host operating system: output of
uname -aLinux monitoring 5.13.0-1021-aws #23~20.04.2-Ubuntu SMP Thu Mar 31 11:36:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
blackbox_exporter version: output of
blackbox_exporter --versionWhat is the blackbox.yml module config.
What is the prometheus.yml scrape config.
What logging output did you get from adding
&debug=trueto the probe URL?Output at a time when probe_ssl_earliest_cert_expiry gave correct value:
Output at a time when probe_ssl_earliest_cert_expiry gave wrong value:
What did you do that produced an error?
I drew a graph in Prometheus for
probe_ssl_earliest_cert_expiry{hostname="tau-psg-clone-02.lautus.net", job="apache_probe"}. I did this to debug intermittent false positives on our prometheus monitoring for expiring SSL certificates.What did you expect to see?
A flatline graph with a value corresponding to the certificate's actual expiry.
What did you see instead?
Further background
The host
tau-psg-clone-02.lautus.netis the only host in our fleet for which blackbox_exporter currently exhibits this behaviour.The correct value of the metric, 1.685816036e+09, corresponds to Saturday, 3 June 2023 18:13:56. The incorrect value, 1.680631454e+09, corresponds to Tuesday, 4 April 2023 18:04:14. The two are 60 days, 0 hours, 9 minutes and 42 seconds apart.
I verified that it is not the host itself that is intermittently presenting a different certificate by reloading a Chrome page for that host at one of the points in time when the prometheus graph dipped to the incorrect value. However, Chrome still saw the same certificate with the same expiry at that time.
I thought this issue might be related to #340 , and that the host was presenting multiple certificate chains from the root. So I inspected the certificate the host presents:
I inspected the chain of three certificates the host presented by copying the certificate output above to individual files. None of them expire earlier than Jun 3 18:13:56 2023 GMT.
The text was updated successfully, but these errors were encountered: