Dev stuff from Juanga Covas

Building HAProxy from sources for performance, latest 2.4 under RHEL / CentOS 7

Building HAProxy from sources for performance, latest 2.4 under RHEL / CentOS 7

Some performance tips and how to upgrade or install haproxy in 2021

Juan Gabriel Covas's photo
Juan Gabriel Covas
·

8 min read

This article summarizes my adventures on installing HAProxy 2.4 which is the latest LTS version as of 2021. I wanted to upgrade an old 2.0.x LTS, compiled from sources, under CentOS 7.

So, we're going to build HAProxy ourselves, as it should be done to know a lot more about this proxy / load balancer and squeeze some extra performance out of it.

The goal of exploiting the maximum possible speed from HAProxy node(s) make sense since nowadays using HAProxy is useful not only for load balancing, but for SSL offloading: it provides high-performance SSL termination (so certificates management can be centralized there and backend webservers just receive unencrypted traffic).

I'm writing this because:

  1. There is a myriad of "how to install haproxy" guides on the internet, but you have to do your own research for specific version gotchas (related to haproxy/OS/distro). Unless I'm missing something, HAProxy itself seems to enforce the "Read the F. Manual" motto about how to install / customize, which is not a bad thing, I'd also want my users read my carefully updated documentation overtime, but you know.

  2. I was upgrading my own scripted installation of HAProxy 2.0.x to 2.4.x so I said: OK, let's try to do it and learn something new while doing it, since wanted to upgrade HAProxy in some production servers.

TLDR; I'm in a hurry

For you, lazy people 😜

How to compile haproxy from sources:

  • for systemd, in CentOS 7
  • with CPU optimizations (if suitable on bare-metal hardware)
  • using a modern gcc and letting haproxy decide proper CFLAGS.
# Build HAProxy 2.4.x from sources under CentOS 7 -------------> scroll

$ sudo yum -y install virt-what curl wget gcc pcre-static pcre-devel make \
perl pcre-devel zlib-devel openssl openssl-devel readline-devel systemd-devel
# grab haproxy sources and cd into dir.
$ _haproxy_v="2.4.3"; mkdir -p ~/src; cd ~/src; wget \
http://www.haproxy.org/download/${_haproxy_v:0:3}/src/haproxy-${_haproxy_v}.tar.gz \
&& tar xzf haproxy-${_haproxy_v}.tar.gz -C ~/src && cd ~/src/haproxy-$_haproxy_v
# install newer gcc without messing the system stock gcc
$ sudo yum -y install centos-release-scl; sudo yum -y install devtoolset-10
# the following scl command will open a new bash to use gcc10
# If you want to script it, use the this "source" command instead: source scl_source enable devtoolset-10
$ scl enable devtoolset-10 bash
# compile haproxy using gcc 10 now
$ _MY_HAPROXY_CPU_NATIVE=; _VMTEST=$(sudo virt-what); \
if [[ -z $_VMTEST ]] ;then _MY_HAPROXY_CPU_NATIVE="CPU=native"; \
echo "This seems a Bare-metal server, using CPU=native"; else \
echo "This seems virtualized: $_VMTEST"; fi; \
make -j $(nproc) TARGET=linux-glibc $_MY_HAPROXY_CPU_NATIVE \
USE_PCRE=1 USE_PCRE_JIT=1 USE_OPENSSL=1 USE_TFO=1 USE_LIBCRYPT=1 USE_THREAD=1 USE_SYSTEMD=1
$ sudo make install
# here is where you could just restart haproxy and exit if this is an "update only" task
# the following is only needed if it is a first install
$ sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy
$ cd admin/systemd && make
$ sudo cp haproxy.service /usr/lib/systemd/system
$ sudo systemctl daemon-reload
$ sudo mkdir -p /etc/haproxy; sudo mkdir -p /run/haproxy
$ sudo mkdir -p /var/lib/haproxy; sudo touch /var/lib/haproxy/stats
$ sudo useradd -r haproxy

# manually create config, new file: /etc/haproxy/haproxy.cfg

# setup systemd service
$ sudo systemctl start haproxy
$ sudo systemctl status haproxy
# enable haproxy on boot
$ sudo systemctl enable haproxy

Or here you have the long, explained stuff:

Pre-requisites

This guide assumes that we're a non-root user with sudo privileges.

Let's install the following pre-requisites. I've tested myself that will work from a fresh, minimal CentOS 7 installation (after applying the latest yum update).

$ sudo yum -y install virt-what curl wget gcc pcre-static pcre-devel make perl \
pcre-devel zlib-devel openssl openssl-devel readline-devel systemd-devel

Get the sources

Using this one-liner you can easily change the desired version and get the sources into a src directory in your home:

$ _haproxy_v="2.4.3"; \
mkdir -p ~/src; cd ~/src; \
wget http://www.haproxy.org/download/${_haproxy_v:0:3}/src/haproxy-${_haproxy_v}.tar.gz && \
tar xzf haproxy-${_haproxy_v}.tar.gz -C ~/src && \
cd ~/src/haproxy-$_haproxy_v

Using a modern gcc

The stock gcc from CentOS 7 is pretty old... from 2015:

$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)

From this site we learn that:

Software Collections, also known as SCL is a community project that allows you to build, install, and use multiple versions of software on the same system, without affecting system default packages. By enabling Software Collections, you gain access to the newer versions of programming languages and services which are not available in the core repositories.

The SCL repositories provide a package named Developer Toolset, which includes newer versions of the GNU Compiler Collection, and other development and debugging tools.

So let's go:

$ sudo yum -y install centos-release-scl

Currently, the Developer Toolset collections consists of #6 to 10. We install the #10:

$ sudo yum -y install devtoolset-10

To access GCC version 10 you need to launch a new shell instance using the Software Collection scl tool:

scl enable devtoolset-10 bash

Now check the GCC version, you’ll notice that GCC 10 is the default version in your current shell. 2021 sounds much better and in fact you can tell the difference by looking at the CFLAGS that will get auto-injected on make by doing haproxy -vv later:

$ gcc --version
gcc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)

⚠️ If you want to script this stuff, opening a new bash is a call for trouble (fork-bomb) so you could do the following to "enable devtoolset-x" just for your shell script: source scl_source enable devtoolset-10

Make options

We should be at ~/src/haproxy-2.4.3

A pretty safe make command for haproxy is the following:

make -j $(nproc) TARGET=linux-glibc \
USE_PCRE=1 USE_PCRE_JIT=1 USE_OPENSSL=1 USE_TFO=1 \
USE_LIBCRYPT=1 USE_THREAD=1 USE_SYSTEMD=1

Let's disect some of the make compilation options (I like when an article explain this stuff). If you're extra curious, see the INSTALL and/or Makefile files from haproxy sources.

  • Note there is no CFLAGS. See below for more about it.

  • We should use CPU=native to get the build machine's specific processor optimizations, but should be used only on non-virtualized environments (see INSTALL file: "known to break" in section "5) How to build HAProxy"). The default CPU setting for haproxy's make is generic so no CPU-specific optimization.

If you want a one-liner to detect whenever to use CPU=native or not in the make command, here is one:

_MY_HAPROXY_CPU_NATIVE=; _VMTEST=$(sudo virt-what); \
if [[ -z $_VMTEST ]] ;then _MY_HAPROXY_CPU_NATIVE="CPU=native"; \
echo "This seems a Bare-metal server, using CPU=native"; \
else echo "This seems virtualized: $_VMTEST"; fi; \
make -j $(nproc) TARGET=linux-glibc $_MY_HAPROXY_CPU_NATIVE \
USE_PCRE=1 USE_PCRE_JIT=1 USE_OPENSSL=1 USE_TFO=1 \
USE_LIBCRYPT=1 USE_THREAD=1 USE_SYSTEMD=1

You can later check the options used to build haproxy by issuing haproxy -vv

  • -j $(nproc) will try to use all the available CPUs to speed up compilation times.

  • TARGET=linux-glibc is the default for linux (kernel 2.6.28 and above). Any other is just other platforms or custom stuff.

  • USE_SYSTEMD=1 if you want to integrate with systemd, since is not enabled by default. This is "new", lot of haproxy how-tos just use the init.d way. To properly install, after compiling haproxy you need to go to admin/systemd directory under sources and make to produce an usable service file for systemd. This one was "hard" to guess, had to google a bit.

  • USE_ZLIB=1 if you really want to change the default, faster compression of HTTP responses that HAProxy can do (see INSTALL file: "4.6) Compression"). Lot of guides out there just put USE_ZLIB=1, but please read the INSTALL file about it. Don't use it and let the libslz library that is embedded in haproxy do the job.

  • USE_LUA=1 if you really need the Lua scripting capabilities of HAProxy. You'll need to provide Lua 5.3+ and tell LUA_INC=/path/to/lua/include and LUA_LIB=/path/to/lua/lib as seen here

About CFLAGS

You'll see many guides on the Internet that pass CFLAGS to make to compile HAProxy. That overwrites the default CFLAGS which is bad.

In fact latest HAProxy versions (in several branches) complain about how you messed up things doing that. So just leave CFLAGS out of make for HAProxy.

This is the full CFLAGS complain message from a bad built haproxy, which BTW is very kind advice from the devs:

$ haproxy -v
FATAL ERROR: invalid code detected -- cannot go further, please recompile!
The source code was miscompiled by the compiler, which usually indicates that
some of the CFLAGS needed to work around overzealous compiler optimizations
were overwritten at build time. Please do not force CFLAGS, and read Makefile
and INSTALL files to decide on the best way to pass your local build options.

Make install ... and a bit more

Now, the typical make install will put the haproxy binary haproxy in /usr/local/sbin among other stuff:

$ sudo make install

Now test if haproxy program is available:

$ haproxy -v
HAProxy version 2.4.3-4dd5a5a 2021/08/17 - https://haproxy.org/
[...]

Typing haproxy -vv (two 'v') should show how haproxy was built.

Create a symbolic link for the binary to allow you to run HAProxy commands as a normal user.

$ sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy

Let's generate a valid haproxy.service file to use with systemd:

$ cd admin/systemd && make
sed -e 's:@SBINDIR@:'/usr/local/sbin':' haproxy.service.in > haproxy.service

Let's copy the generated systemd service file to /usr/lib/systemd/system

$ sudo cp haproxy.service /usr/lib/systemd/system

Now let's create some more needed directories and files:

$ sudo mkdir -p /etc/haproxy; sudo mkdir -p /run/haproxy
$ sudo mkdir -p /var/lib/haproxy; sudo touch /var/lib/haproxy/stats

Create a system user haproxy (this also creates the group):

$ sudo useradd -r haproxy

Making a simple config

The default config file is /etc/haproxy/haproxy.cfg.

We manually create one with something like the following (or look for the examples directory at haproxy sources).

This is just an example, but will let you start haproxy without complaining.

global
    daemon
    user haproxy
    group haproxy
    chroot /var/lib/haproxy
    maxconn 1024

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    bind *:80
    default_backend servers

backend servers
    balance roundrobin  
    server ws01 1.1.1.1:80 
    server ws02 1.1.1.2:80

Systemctl

Let's setup the service:

$ sudo systemctl daemon-reload
$ sudo systemctl start haproxy
$ sudo systemctl status haproxy

To enable on boot:

$ sudo systemctl enable haproxy

And you can start to play around. Another article could be done about different haproxy configurations, but you have general good articles on that on the internet.

Conclusion

I hope this helps some people. It's not easy to find a proper "build haproxy from sources" article for all cases / distros / haproxy versions overtime.

Please correct me if anything is [terribly] wrong in this guide.

serverservices