Kapacitor subscriptions fail with InfluxDB when SSL is enabled for Kapacitor

[mglazer]

I'm a little unsure if I should file this in Influx or in Kapacitor, but I'll file it in Kapacitor anyways.

Steps to reproduce

1. Enable SSL for Kapacitor
2. Enable SSL for InfluxDB
3. Create an alert which subscribes to any retention policy from InfluxDB

What you see

In kapacitor error logs:

log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:50:00 http: TLS handshake error from 127.0.0.1:53820: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:50:10 http: TLS handshake error from 127.0.0.1:53821: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:50:20 http: TLS handshake error from 127.0.0.1:53822: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:50:30 http: TLS handshake error from 127.0.0.1:53823: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:50:40 http: TLS handshake error from 127.0.0.1:53824: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:50:50 http: TLS handshake error from 127.0.0.1:53825: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:51:00 http: TLS handshake error from 127.0.0.1:53834: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:51:10 http: TLS handshake error from 127.0.0.1:53835: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:51:20 http: TLS handshake error from 127.0.0.1:53836: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 15:51:30 http: TLS handshake error from 127.0.0.1:53837: tls: oversized record received with length 21536

In Influx Error logs:

[subscriber] 2016/09/26 16:02:50 Post http://~subscriber:ZGVmYXVsdDtXjfyv1pddFdxjjIgYd6qMdUoWsjg01YztLwfKx7VVH-kn2d26mCZWLpCLBxc4Nrdym8htOVijA4RB5-U4p3ZA@localhost:9092/write?consistency=&db=_internal&precision=ns&rp=monitor: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

It seems the issue is in subscriber/http.go:

https://github.com/influxdata/influxdb/blob/1.0/services/subscriber/http.go

where subscriber doesn't connect to the subscribee (in this case, I believe it's Kapacitor) using SSL.

Product Versions

Kapacitor: 1.0.0
InfluxDB: 1.0.0

[nathanielc]

@mglazer You need to set the subscription-protocol to https instead of http in the Kapacitor configuration file.

[mglazer]

Hey Nathaniel:

Thanks for the quick response. I did that, but now I'm seeing the following:

In influx:

[subscriber] 2016/09/26 20:41:50 Post https://~subscriber:ZGVmYXVsdDsOCS9d8Ucm4C_OOPr1t4KrCzYxnLKcj4SiX0cDjPsUEORHcJO69yPq9-HTHcnzpCBzU1Gmz5gsI63RXpQcjCS5@localhost:9092/write?consistency=&db=_internal&precision=ns&rp=monitor: x509: certificate signed by unknown authority
[subscriber] 2016/09/26 20:41:50 Post http://~subscriber:ZGVmYXVsdDtXjfyv1pddFdxjjIgYd6qMdUoWsjg01YztLwfKx7VVH-kn2d26mCZWLpCLBxc4Nrdym8htOVijA4RB5-U4p3ZA@localhost:9092/write?consistency=&db=_internal&precision=ns&rp=monitor: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

In kapacitor:

log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 20:42:50 http: TLS handshake error from 127.0.0.1:55566: remote error: bad certificate
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 20:43:00 http: TLS handshake error from 127.0.0.1:55568: tls: oversized record received with length 21536
log messages must have 'L!' prefix where L is one of 'D', 'I', 'W', 'E'[log] 2016/09/26 20:43:00 http: TLS handshake error from 127.0.0.1:55569: remote error: bad certificate

I'm using the exact same certificates/keys between influx, grafana, kapacitor. In kapacitor.conf I have:

  enabled = true
  urls = ["https://localhost:8086"]
  username = "admin"
  password = "PASSWORD"
  #ssl-ca = "var/conf/ca.crt"
  #ssl-cert = "var/conf/ca.crt"
  #ssl-key = "var/conf/ssl.pem"
  insecure-skip-verify = true
  subscription-protocol = "https"
  timeout = 0
  udp-buffer = 1000
  udp-read-buffer = 0
  [influxdb.subscriptions]
  [influxdb.excluded-subscriptions]

(note: I've tried uncommenting the ssl- lines and commenting them, doesn't seem to do much). I'm not totally certain how the relationship between kapacitor/influx is supposed to work, so I know where the missing trust relationship setting is happening.

[mglazer]

Dove into this a bit. This ended up being an InfluxDB issue.

See PR: influxdata/influxdb#7392

Feel free to close this.

[nathanielc]

@mglazer Thanks for the InfluxDB PR! Closing this...