Infrastructure. In production, few choices matter more than how you balance database connections and how you keep them highly available. For years, one organization ran a Global Load Balancer (GLB) on every web server to distribute traffic across four Galera MySQL nodes. It worked—until the operational tax piled up: install + configure on many hosts, touch multiple files for every tweak, and reconfigure again whenever the cluster scaled. The pivot was clear: centralize load balancing with HAProxy, and handle failover with Keepalived.
The outcome isn’t a promise, it’s a measurable change: less maintenance, more reliability, and easier scaling. Here is the full technical story.
Why move away from a GLB on every web server
Experience made the cost of the decentralized approach obvious:
- Maintenance overhead. Installing, configuring, and maintaining GLB on each web host multiplies effort. Any update—timeouts, a new health-check policy, an added IP—turns into a relay race across machines.
- Scaling pain. Adding a Galera node or growing the backend pool meant touching GLB on every single host that connects to MySQL.
- Fragmented visibility. Troubleshooting spread across hosts, logs, and local states.
The alternative is to centralize the entry point: two dedicated servers with HAProxy in front of the Galera cluster, and Keepalived to expose a floating Virtual IP (VIP) across them. If one fails, the other takes over immediately.
The architecture: a VIP, centralized balancing, and rapid failover
Current layout (at a glance):
- Web entry. Users reach the web tier via two VIPs managed by Keepalived (at the web layer).
- Database connections. Web servers no longer point at individual Galera nodes. They connect to one VIP for MySQL (e.g.,
192.168.1.180), also managed by Keepalived. - Load balancing via HAProxy. That VIP lands on HAProxy, which load balances plain TCP connections across the four Galera nodes:
192.168.1.81to192.168.1.84. - High availability. Keepalived uses VRRP to keep the VIP alive. If the primary HAProxy goes down, the backup promotes to MASTER and assumes the VIP—applications don’t notice.
This design centralizes control, simplifies the web servers (a single DB endpoint), and introduces a fast, predictable failover mechanism.
HAProxy for Galera: raw TCP, health checks, and consistent distribution
For MySQL/Galera, you want TCP mode (not L7), a round-robin of connections, and active health checks so traffic avoids sick nodes.
Configuration example:
frontend mysql_frontend
bind 192.168.1.180:3306
mode tcp
default_backend galera_backend
backend galera_backend
mode tcp
balance roundrobin
option mysql-check user haproxy
server db1 192.168.1.81:3306 check
server db2 192.168.1.82:3306 check
server db3 192.168.1.83:3306 check
server db4 192.168.1.84:3306 check
Code language: CSS (css)Key backend points:
mode tcp: treat sessions as raw MySQL connections.balance roundrobin: even distribution across Galera nodes.option mysql-check user haproxy: a native MySQL probe with a dedicated health user. Unhealthy nodes are pulled from the pool until they recover.
This prevents traffic from hitting degraded or split-brain nodes and gives you a single pane for logs, metrics, and stats.
Keepalived: one VIP, two HAProxy nodes, and VRRP to cover your back
Keepalived implements VRRP, the protocol that lets multiple hosts share a VIP and define priorities and roles (MASTER/BACKUP).
Primary example:
vrrp_instance VI_MYSQL {
state MASTER
interface eth0
virtual_router_id 53
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass S3cr3t
}
virtual_ipaddress {
192.168.1.180
}
track_script {
chk_haproxy
}
}
Code language: PHP (php)Health script:
#!/bin/bash
systemctl is-active --quiet haproxy
exit $?
Code language: PHP (php)Make it executable:
chmod +x /usr/local/bin/chk_haproxy
How it behaves: the MASTER announces itself and holds the VIP. If the script sees HAProxy is down, the node loses priority and the BACKUP (e.g., priority 90) promotes and takes over the VIP. From an app’s perspective, the MySQL IP never changes; the data plane continues seamlessly.
Immediate gains: fewer moving parts, more HA, and cleaner scaling
1) Less to maintain. Instead of shepherding GLB on many web servers, the team manages two HAProxy and two Keepalived nodes. Full centralization: one place for balancing, one for failover.
2) Real high availability. If the primary HAProxy dies, Keepalived moves the VIP in seconds. Web/app servers do not need to retry a list of hosts; they keep pointing to one IP.
3) Scaling is straightforward. Adding Galera nodes or extra HAProxy instances means changing one config, not dozens. That cuts change time and error probability.
4) Concentrated observability. HAProxy owns the balancing telemetry: stats, counters, per-backend errors, open sessions. Troubleshooting becomes simpler.
Operational notes: users, privileges, and drills
- Health-check user. The
mysql-check user haproxyrequires a minimally privileged MySQL account (only what’s needed for the probe). Rotate credentials and avoid unnecessary privileges. - Controlled failover drills. Run VRRP drills in a maintenance window (stop HAProxy on MASTER, watch failover + failback). This builds confidence and prevents surprises.
- Sessions and stickiness. Many apps keep long-lived pooled connections. In those cases, round-robin is fine. If your load relies on very long tasks per node, tune health-checks (intervals and thresholds) to avoid ejecting a server on a transient spike.
- IPv4/IPv6 and routing. In dual-stack environments, document which family the VIP uses and which interfaces HAProxy listens on; it reduces ambiguity.
- Keepalived security. Simple auth (
auth_type PASS) adds basic protection; in shared networks, add segmentation and controls on switches/routers.
From “a GLB per web server” to “one IP to rule them all”
Shifting from per-host GLBs to HAProxy + Keepalived lets you operate the database like a service with its own IP, independent of node count or layout. Apps no longer need to know about individual MySQLs or maintain host rotation lists; the balancer becomes the lever.
The team that made the transition put it this way: “less maintenance, more availability, easier to scale.” Galera remains the same—multi-master with synchronous replication—but the access point is now coherent and fault-tolerant.
Practical steps to replicate the pattern
- Inventory and tidy-up. List all web servers that talk to MySQL and map the Galera cluster. Consolidate DB access into one IP for apps.
- Deploy two HAProxy nodes. Prepare dedicated hosts (or VMs) for HAProxy and Keepalived.
- Configure HAProxy. Add a frontend on VIP:3306, define the backend with all Galera nodes and native MySQL health-checks.
- Configure Keepalived (VRRP). Define the VIP and the health script for HAProxy. Tune priorities and advert intervals.
- Test failover. Stop HAProxy on MASTER, observe how the VIP moves to BACKUP. Bring it back and confirm failback.
- Point apps to the VIP. Update DB connection settings on each web server to the VIP and retire local GLBs.
- Monitoring and tuning. Enable HAProxy stats, adjust health-check intervals/timeouts to match real load.
Reference configs (complete)
HAProxy (excerpt):
global
log /dev/log local0
maxconn 20000
defaults
log global
mode tcp
timeout connect 5s
timeout client 50s
timeout server 50s
option tcplog
frontend mysql_frontend
bind 192.168.1.180:3306
default_backend galera_backend
backend galera_backend
balance roundrobin
option mysql-check user haproxy
server db1 192.168.1.81:3306 check
server db2 192.168.1.82:3306 check
server db3 192.168.1.83:3306 check
server db4 192.168.1.84:3306 check
Code language: PHP (php)Keepalived (MASTER):
vrrp_script chk_haproxy {
script "/usr/local/bin/chk_haproxy"
interval 2
fall 2
rise 2
}
vrrp_instance VI_MYSQL {
state MASTER
interface eth0
virtual_router_id 53
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass S3cr3t
}
virtual_ipaddress {
192.168.1.180/24
}
track_script {
chk_haproxy
}
}
Code language: PHP (php)Keepalived (BACKUP):
vrrp_instance VI_MYSQL {
state BACKUP
interface eth0
virtual_router_id 53
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass S3cr3t
}
virtual_ipaddress {
192.168.1.180/24
}
track_script {
chk_haproxy
}
}
Code language: PHP (php)Lessons learned
- Centralization trims tech debt. A balancer per web server is hard to scale for humans. Two well-run HAProxy nodes are not.
- VRRP binds your IP to health. If HAProxy stops, priority drops and the VIP moves to whoever is healthy.
- Galera + HAProxy play nicely with no tricks: TCP, round-robin, mysql-check, done.
- Docs and runbooks. Write down failover/failback procedures and test them regularly—saves pain.
FAQ
How do I configure HAProxy to balance a Galera MySQL cluster?
Use TCP mode. Bind a frontend to the VIP:3306 and define a backend with all Galera nodes (server dbX IP:3306 check). Turn on option mysql-check user <user> for native MySQL health probes, and roundrobin to distribute connections.
How does Keepalived (VRRP) expose a floating MySQL VIP?
Keepalived defines a VRRP instance with a VIP and priorities. The host with the highest priority and a healthy HAProxy holds the VIP as MASTER. If HAProxy fails, the BACKUP promotes and assumes the VIP, preserving connection continuity.
What advantages does HAProxy + Keepalived offer over per-host GLB?
Less maintenance (two servers vs. many), real HA with a floating VIP, cleaner scaling (add nodes by changing one config), and centralized observability for diagnosis.
Best practices for MySQL health checks in HAProxy?
Use option mysql-check with a least-privilege user, tune intervals/timeouts to your real workload, and monitor metrics (connect times, rise/fall counts, per-server errors). Rotate credentials and avoid extra privileges on the check account.
Bottom line: replacing per-host GLB with HAProxy + Keepalived tangibly simplifies operating a Galera cluster: fewer moving parts, higher availability, and a clearer path to scale. For production MySQL, this has become a safe bet: a floating VIP with VRRP and a central balancer that does the heavy lifting.
source: systemdeveloper
