HAProxy: TLS passthrough with HTTPS checks

HAProxy can easily be configured to load balance SSL/TLS traffic. This post is going to look at adding HTTPS health checks to ensure a service is up, while keeping HAProxy in tcp mode. The diagram below gives an outline of the setup:

Note: two TCP connections are made during a request, one between the client and HAProxy and one from HAProxy to a back end. Although two TCP connections are made, the SSL/TLS connection passes straight though HAProxy (SSL/TLS passthrough).

Initial setup

On CentOS, HAProxy can be installed using the package manager:

yum install -y haproxy

Basic HAProxy configuration to load balance traffic in TCP mode will look something like:

frontend fe_service
    bind service.example.com:443
    mode tcp
    default_backend be_service

backend be_service
    balance roundrobin
    mode tcp
    server backend1.example.com backend1.example.com:8443
    server backend2.example.com backend1.example.com:8443

This configuration should be added to /etc/haproxy/haproxy.cfg. The service can then be enabled and started:

systemctl enable haproxy.service
systemctl start haproxy.service

TCP health checks

Very simple TCP health checking can be enabled by adding check to the end of each of the servers in the back end:

backend be_service
    balance roundrobin
    mode tcp
    server backend1.example.com backend1.example.com:8443 check
    server backend2.example.com backend1.example.com:8443 check

Note: HAProxy will need to be restarted to pick up the config change. This can be done by running systemctl restart haproxy.service.

On CentOS the default HAProxy configuration makes statistics available via a socket:

global
    ...
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

A command similar to the following can be used to query the socket and pull back the status of the two back ends:

$ echo 'show stat' | nc -U /var/lib/haproxy/stats | \
   awk -F, '/backend[1-2]/ {print $2,$18,$37}'
backend1.example.com UP L4OK
backend2.example.com UP L4OK

Note: you may need to install the nmap-netcat package if nc is not available.

Alternatively you can enable the HAProxy status page by adding the following additional configuration:

listen stats :9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /

Note: to add authentication, use the stats auth option.

HTTPS health checks

TCP health checks are better than nothing, however back end applications often provide a health check API which gives a better indication of the status of the service.

The configuration below will periodically make an HTTP request to /v1/status and check the return code of the HTTP request:

backend be_service
    balance roundrobin
    mode tcp
    option httpchk GET /v1/status
    http-check expect status 200
    server backend1.example.com backend1.example.com:8443 check check-ssl verify required ca-file /etc/haproxy/servicebox1.example.com.pem maxconn 5
    server backend2.example.com backend1.example.com:8443 check check-ssl verify required ca-file /etc/haproxy/servicebox2.example.com.pem maxconn 5

Note: to disable verifying certificates, verify none can be used in place of verify required ca-file SOMEFILE.

Once HAProxy has been restarted to pick up the new configuration, HAProxy will periodically poll the status API to check if the service is healthy. Checking the socket file should show layer 7 (L7) checks are being made:

$ echo 'show stat' | nc -U /var/lib/haproxy/stats | \
   awk -F, '/backend[1-2]/ {print $2,$18,$37}'
backend1.example.com DOWN L7STS
backend2.example.com UP L7OK

Note: in the example above backend1.example.com is responding to requests, however the status API is not returning 200, so it's marked as down. This would not be picked up by a standard TCP check.