Metricbeat compatibility with HAProxy 1.5 and 1.7

[brandonmensing]

1.6 is supported today and a one-off test showed that 1.5 and 1.7 might work as well. They need to be tested automatically to make them supported.

(Related: #5210)

[joelstewart]

I am unable to get the HAProxy to work with 1.7 (via marathon-lb docker).

for marathon-lb, I provide a head template snippet to make sure tcp stats are enabled:

listen tcpstats
  bind 0.0.0.0:9090
  stats enable

listen httpstats
  bind 0.0.0.0:9092
  balance
  mode http
  stats enable
  monitor-uri /_haproxy_health_check

And I configure the modules.d/haproxy

- module: haproxy
  metricsets: ["info", "stat"]
  period: 10s
  hosts: ["tcp://xxxxxxxxxxx:9090"]

The result is haproxy adds errors to the metricbeat index, with errors stating:

 "error" : "strconv.ParseFloat: parsing \"\": invalid syntax",

and

"error" : "failed fetching haproxy stat: error parsing CSV: empty csv file given",

Debug mode gives little more information in metricbeat console output.

I started to compare csv formats between 1.6 and 1.7, and it looks like it only adds additional fields. But the docs for tcp socket communicate have changed as well. Not a go programmer, but it looks like some work was done to anticipate extra ",". But by the logs it looks like it does not get any csv back.

I tried metricbeat 5.6.4 and 5.6.3.

The haproxy tcp connection might not be compatible version to version. Why not use csv with http api at /haproxy?stats;csv for stats and /_haproxy_getconfig for info?

[ruflin]

@joelstewart Thanks for the detailed investigation. I'm not an expert on HAProxy and din't know we could also get the metrics though http. Perhaps @hartfordfive can share some more thoughts here?

Could you share the links to docs you found about the socket changes and the http endpoints? I struggle a bit to find the right pages, probably I just use the wrong Google queries :-(

[joelstewart]

links to 1.6 and 1.7 docs. shows new section for "tcp commands (section 9.2 is new)"

https://cbonte.github.io/haproxy-dconv/1.6/management.html#9
https://cbonte.github.io/haproxy-dconv/1.7/management.html#9

Note also section 9.1 has the list of CSV, and I believe that it should be backward compatible as it only appends new fields to the format.

When enabling http mode on "listen stats" port, there are configs to mirror in HAProxy.yml module...

https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-stats%20enable

Upon further investigation, It appears that the _haproxy_getconfig method is NOT HAProxy, this was added by marathon-lb (the wrapper I am using) https://github.com/mesosphere/marathon-lb/blob/master/getconfig.lua . So HAProxy "info" may not be available this route.

However, many of the fields that are separated into "info" module (https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-haproxy-info.html) appear in the stats as well:

pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,

ty

[ruflin]

@joelstewart Thanks a lot for all the links and notes. This is super helpful. I need to have a deeper look at all this when I find the time to figure out what our future approach here should be.

[ruflin]

@jsoriano Did your PR in #5819 solve the above issue, meaning we can close this one? Or did it "only" add the http endpoint.

[jsoriano]

@ruflin the issue about connecting using http should be solved after #5819 yes, but we are still not testing multiple haproxy versions, we may want to keep this issue open to track this.

[ruflin]

@jsoriano Let's close this issue as the PR solve on what the issue was opened for but lets open a new one to track the testing of both versions (or directly a PR ;-) ).