Use HAProxy to route Postgresql and ElasticSearch connections #808
Conversation
This provides two new configuration options: use_chef_backend chef_backend_members When use_chef_backend is true, HAProxy will be enabled with a configuration that will route requests to Postgresql and Elasticsearch to the current Chef Backend leader.
stevendanna
changed the title
| # When Chef Backend is configured, this is too early to verify | ||
| # postgresql accessibility since we need to configure HAProxy | ||
| # first | ||
| if ! PrivateChef['use_chef_backend'] |
There was a problem hiding this comment.
With this flag, we can at some point get rid of the hacky "derive whether we really care about bootstrap" logic that's used to allow a front-end to do bootstrapping.
- The BootstrapValidator checks can't run because they require postgresql to be up. - HAProxy needs some time to mark the non-leader backends as down. Thus we wait for that to happen or error out if it doesn't happen in a reasonable amount of time.
stevendanna
changed the title
- Move code for parsing HAProxy status socket responses into a HAProxyStatus class. - Move code for determining backend cluster members into a ChefBackend module.
| mode tcp | ||
| option httpchk GET /leader HTTP/1.1\r\nHost:localhost:<%= @leaderl_healthcheck_port %>\r\n\r\n | ||
| <% @chef_backend_members.each do |name, ip| -%> | ||
| server <%= name %> <%= ip %>:<%= @remote_postgresql_port %> check port <%= @leaderl_healthcheck_port %> rise 1 fall 1 |
There was a problem hiding this comment.
I think we can specify defaults here up above for all the servers. Like:
default-server inter 3s fall 3 rise 2
server name ip:port check port
server name ip:port check port
...
https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html
Reading the docs a little more there's actually built in support for postgres as well:
backend be_pgsql
[...]
option pgsql-check
server srv1 10.0.0.1:5432 check
server srv2 10.0.0.2:5432 check
There was a problem hiding this comment.
We are using an HTTP check here because the /leader endpoint knows about the leader/follower state. (postgresql could be erroneously up as a leader while leaderl is in the process of taking it down. The /leader endpoint will return the correct thing in that case while a postgresql-based check may not).
|
@dmccown Updated the PR to use the default-server config directive. |
| name "haproxy" | ||
| default_version "1.6.4" | ||
|
|
||
| dependency "zlib" |
There was a problem hiding this comment.
Are all of the dependencies run time dependencies? Is it worth it to prune the non-runtime deps at this time? I'm OK if there's another task to clean this up later.
There was a problem hiding this comment.
All of these are runtime deps since we don't do static linking, so zlib, pcre, openssl all have to be in the package.
|
Other than my comment on dependencies and the comment you cleaned up on your last PR nothing is really jumping out at me. So overall I'm 👍 |
This provides two new configuration options:
use_chef_backend
chef_backend_members
When use_chef_backend is true, HAProxy will be enabled with a
configuration that will route requests to Postgresql and Elasticsearch
to the current Chef Backend leader.