HAProxy Complete Configuration & Management Guide¶

📋 Overview¶

This comprehensive guide covers the complete HAProxy Aloha infrastructure, configuration, management, and operational procedures for the WithinEarth platform.

🏗️ HAProxy Infrastructure¶

HAProxy Aloha HA Pair¶

graph TB
    INTERNET["🌍 Internet<br/>Public Traffic"]

    subgraph HAPROXY_CLUSTER["HAProxy Aloha HA Cluster"]
        VIP["Floating VIP<br/>89.149.192.33<br/><b>Public Entry Point</b>"]

        MASTER["HAProxy Master (aloha-1)<br/>10.32.8.36<br/>Status: <b>ACTIVE</b><br/>Role: Primary LB"]

        STANDBY["HAProxy Standby (aloha-2)<br/>10.32.8.38<br/>Status: <b>STANDBY</b><br/>Role: Failover LB"]

        SYNC["Peer Synchronization<br/>Stick tables sync<br/>Config sync"]
    end

    subgraph API_BACKEND["Backend: 7 API Servers × 43 Pools = 301 Pools"]
        API1["API-1: 10.32.8.134<br/>Ports: 19169-19211"]
        API2["API-2: 10.32.8.135<br/>Ports: 19212-19254"]
        API3["API-3: 10.32.8.139<br/>Ports: 19255-19297"]
        API4["API-4: 10.32.8.137<br/>Ports: 19298-19340"]
        API5["API-5: 10.32.8.35<br/>Ports: 19341-19383"]
        API6["API-6: 10.32.8.166<br/>Ports: 19384-19426"]
        API7["API-7: 10.32.8.167<br/>Ports: 19427-19635"]
    end

    INTERNET --> VIP
    VIP -->|"Active"| MASTER
    VIP -.->|"Failover"| STANDBY

    MASTER <-->|"Sync"| SYNC
    STANDBY <-->|"Sync"| SYNC

    MASTER --> API1 & API2 & API3 & API4 & API5 & API6 & API7
    STANDBY -.->|"On Failover"| API1 & API2 & API3 & API4 & API5 & API6 & API7

    style VIP fill:#90EE90
    style MASTER fill:#87CEEB
    style STANDBY fill:#FFE4B5

Infrastructure Details¶

Component	Value	Details
HAProxy Master	10.32.8.36	aloha-1, active load balancer
HAProxy Standby	10.32.8.38	aloha-2, failover load balancer
Public Floating IP	89.149.192.33	Main entry point for all traffic
Additional IPs	212.7.202.151, 212.7.202.152	Reserved/secondary
Host Type	VM on Proxmox	Cluster: 10.32.8.13, 14
Version	HAProxy Aloha (Enterprise)	Hardware appliance virtualized
Total Backends	2 (consolidated)	shared_backend + recheck22_backend
Total Backend Servers	301 unique pools	7 API servers × 43 pools each
Admin Panel	http://10.32.8.36:4444	Master control panel
Stats Page	http://10.32.8.36:45001/stats	Real-time statistics
Management UI	https://haproxy-ui.withinearth.com/home	Client management interface

⚙️ HAProxy Configuration Architecture¶

Configuration File Structure¶

/etc/haproxy/haproxy.cfg (695 lines - Consolidated)
├── Global Section (Lines 1-50)
│   ├── Process/thread settings
│   ├── SSL configuration
│   └── Logging setup
│
├── Defaults Section (Lines 51-100)
│   ├── Timeouts
│   ├── Connection limits
│   └── Error handling
│
├── Peer Section (Lines 93-95)
│   └── aloha-1 ↔ aloha-2 synchronization
│
├── Frontend Section (Lines 101-215)
│   ├── SSL termination (Port 443)
│   ├── HTTP listener (Port 80)
│   ├── Host header capture
│   ├── Rate limiting logic
│   ├── IP whitelisting
│   ├── ACLs and routing rules
│   └── Backend selection
│
├── Backend Section (Lines 216-695)
│   ├── shared_backend (301 servers)
│   │   ├── API-1 pools (19169-19211)
│   │   ├── API-2 pools (19212-19254)
│   │   ├── API-3 pools (19255-19297)
│   │   ├── API-4 pools (19298-19340)
│   │   ├── API-5 pools (19341-19383)
│   │   ├── API-6 pools (19384-19426)
│   │   └── API-7 pools (19427-19635)
│   │
│   └── recheck22_backend (7 servers)
│       └── Special routing for recheck endpoint
│
└── Map Files (External)
    ├── rate_limit.txt (926 rate limit rules)
    ├── ip_whitelist.txt (6,199 whitelisted IPs)
    └── frontend_backend_mapping.txt (legacy, optional)

Key Configuration Parameters¶

Global Settings¶

global
    maxconn 100000                    # Maximum connections
    tune.ssl.default-dh-param 2048   # SSL DH param size
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:...
    nbthread 4                        # CPU threads

Frontend Configuration¶

frontend web-fe
    bind *:443 ssl crt /etc/ssl/certificate.pem
    bind *:80
    maxconn 33000                     # Session limit

    # Capture headers for logging
    capture request header Host len 50
    capture request header User-Agent len 100

    # Rate limiting
    http-request set-var(txn.host) req.hdr(host),lower
    http-request set-var(txn.rate_limit) var(txn.host),map(/etc/haproxy/files-net/rate_limit.txt)
    http-request track-sc0 src table rate_limit_table
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt var(txn.rate_limit) }

    # IP whitelisting
    tcp-request connection reject unless { src -f /etc/haproxy/files-net/ip_whitelist.txt }

    # Backend selection
    use_backend shared_backend

Backend Configuration¶

backend shared_backend
    balance uri whole              # Consistent hashing by URL
    hash-type consistent           # Stable distribution

    # Timeouts
    timeout server 120s
    timeout connect 10s

    # Health checks
    default-server inter 30s rise 2 fall 3 check

    # API-1 pools (43 pools)
    server api1_pool1  10.32.8.134:19169 check
    server api1_pool2  10.32.8.134:19170 check
    # ... (41 more pools)
    server api1_pool43 10.32.8.134:19211 check

    # API-2 pools (43 pools)
    server api2_pool1  10.32.8.135:19212 check
    # ... (continues for all 7 API servers)

🔒 Security Features¶

Rate Limiting¶

graph LR
    REQUEST["Incoming Request"]

    subgraph RATE_LIMIT["Rate Limiting Logic"]
        EXTRACT["Extract Host Header<br/>(e.g., 65nuconnect.withinearth.com)"]
        LOOKUP["Map Lookup<br/>rate_limit.txt"]
        LIMIT["Get Limit<br/>(e.g., 50 req/s)"]
        TRACK["Track in Stick Table<br/>Count requests per IP"]
        CHECK["Check: Current > Limit?"]
    end

    ALLOW["✅ Allow Request<br/>Forward to Backend"]
    DENY["❌ HTTP 429<br/>Rate Limit Exceeded"]

    REQUEST --> EXTRACT
    EXTRACT --> LOOKUP
    LOOKUP --> LIMIT
    LIMIT --> TRACK
    TRACK --> CHECK

    CHECK -->|"Within Limit"| ALLOW
    CHECK -->|"Exceeded"| DENY

    style DENY fill:#FFB6C1
    style ALLOW fill:#90EE90

Rate Limit Configuration¶

Map File: /etc/haproxy/files-net/rate_limit.txt (926 rules)

Example entries:

65nuconnect.withinearth.com 50
didaconnect.withinearth.com 420
masterconnect.withinearth.com 100

Per-URL Rate Limits (also supported):

/api/xconnect/availability 100
/api/xconnect/book 50
/api/xconnect/cancel 30

IP Whitelisting¶

Map File: /etc/haproxy/files-net/ip_whitelist.txt (6,199 IPs)

Configuration:

# Reject connections from non-whitelisted IPs
tcp-request connection reject unless { src -f /etc/haproxy/files-net/ip_whitelist.txt }

# Exception: Channel Manager bypass
acl is_channel_manager hdr(host) -i channelmanager.withinearth.com
tcp-request connection accept if is_channel_manager

SSL/TLS Configuration¶

# SSL termination
bind *:443 ssl crt /etc/ssl/certificate.pem

# SSL cipher suite (modern, secure)
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:...
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11

# Force HTTPS redirect
redirect scheme https code 301 if !{ ssl_fc }

📊 Monitoring & Statistics¶

HAProxy Stats Page¶

URL: http://10.32.8.36:45001/stats

Shows real-time metrics: - Current sessions - Session rate - Backend server status (UP/DOWN) - Queue length - Error rates - Response times

Admin Control Panel¶

URL: http://10.32.8.36:4444

Administrative functions: - Server management (enable/disable) - Configuration changes - View logs - System status

HAProxy Management UI¶

URL: https://haproxy-ui.withinearth.com/home

Client management interface: - Add/remove clients - Update rate limits - Manage IP whitelist - View client statistics

Key Metrics to Monitor¶

Metric	Normal Range	Alert Threshold	Action
Session Count	5,000-15,000	>30,000	Scale backend
Session Rate	100-500/s	>1,000/s	Check for attack
Backend Errors	<0.01%	>1%	Check API servers
Queue Length	0-10	>100	Add backend servers
Response Time	<100ms	>500ms	Investigate slowness
Failed Health Checks	0	>3 consecutive	Check server

🔄 High Availability & Failover¶

Failover Architecture¶

sequenceDiagram
    participant Client
    participant VIP as Floating VIP<br/>89.149.192.33
    participant Master as HAProxy Master<br/>10.32.8.36
    participant Standby as HAProxy Standby<br/>10.32.8.38
    participant API as API Servers

    Note over Master,Standby: Normal Operation
    Client->>VIP: Request
    VIP->>Master: Route to Master
    Master->>API: Forward to backend
    API-->>Master: Response
    Master-->>VIP: Response
    VIP-->>Client: Response

    Note over Master: ❌ Master Fails

    VIP->>Standby: Automatic VIP failover<br/>(VRRP/Keepalived)

    Note over Standby: ✅ Standby becomes Active

    Client->>VIP: Request
    VIP->>Standby: Route to Standby
    Standby->>API: Forward to backend
    API-->>Standby: Response
    Standby-->>VIP: Response
    VIP-->>Client: Response

Failover Behavior¶

Aspect	Details
Detection Time	<3 seconds (VRRP heartbeat)
Failover Time	<1 second (VIP takeover)
Session Handling	Active connections maintained via stick tables
Config Sync	Automatic sync via peer configuration
Stick Table Sync	Real-time sync between master/standby
Zero Downtime	Yes (seamless VIP transfer)

Peer Synchronization¶

# Peer configuration
peers WITHINEARTH
    peer aloha-1 10.32.8.36:1024
    peer aloha-2 10.32.8.38:1024

# Stick table with peer sync
stick-table type ip size 1m expire 60s store http_req_rate(10s) peers WITHINEARTH

What is Synced: - Rate limiting counters - Session persistence data - IP tracking tables - Connection statistics

🛠️ Operational Procedures¶

Daily Operations¶

Check HAProxy Status¶

# On HAProxy server (10.32.8.36 or 10.32.8.38)
systemctl status haproxy

# Check if process is running
ps aux | grep haproxy

# Check listening ports
netstat -tlnp | grep haproxy

View Real-Time Stats¶

# Via stats page (recommended)
curl -s http://10.32.8.36:45001/stats

# Via command line
echo "show stat" | socat stdio /var/run/haproxy.sock

Monitor Logs¶

# Follow HAProxy logs
tail -f /var/log/haproxy.log

# Check for errors
grep " 5[0-9][0-9] " /var/log/haproxy.log | tail -20

# Check rate limit denials
grep "429" /var/log/haproxy.log | tail -20

Configuration Management¶

Validate Configuration¶

# Syntax check
haproxy -c -f /etc/haproxy/haproxy.cfg

# Should output: "Configuration file is valid"

Reload Configuration (Zero Downtime)¶

# Graceful reload
systemctl reload haproxy

# Or manual reload
haproxy -f /etc/haproxy/haproxy.cfg -sf $(pidof haproxy)

Backup Configuration¶

# Backup current config
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.backup.$(date +%Y%m%d_%H%M%S)

# Backup map files
tar -czf /backup/haproxy-maps-$(date +%Y%m%d).tar.gz /etc/haproxy/files-net/

Backend Server Management¶

Check Backend Server Status¶

# Show all backend servers
echo "show servers state" | socat stdio /var/run/haproxy.sock

# Show specific backend
echo "show backend" | socat stdio /var/run/haproxy.sock | grep shared_backend

Disable Backend Server¶

# Disable server (drain connections, stop new connections)
echo "disable server shared_backend/api1_pool1" | socat stdio /var/run/haproxy.sock

# Verify
echo "show stat" | socat stdio /var/run/haproxy.sock | grep api1_pool1

Enable Backend Server¶

# Re-enable server
echo "enable server shared_backend/api1_pool1" | socat stdio /var/run/haproxy.sock

Rate Limit Management¶

Update Rate Limits¶

# Edit rate limit map
nano /etc/haproxy/files-net/rate_limit.txt

# Add new client
echo "newclient.withinearth.com 100" >> /etc/haproxy/files-net/rate_limit.txt

# Reload map (no HAProxy restart needed)
echo "clear map /etc/haproxy/files-net/rate_limit.txt" | socat stdio /var/run/haproxy.sock
echo "add map /etc/haproxy/files-net/rate_limit.txt newclient.withinearth.com 100" | socat stdio /var/run/haproxy.sock

Update IP Whitelist¶

# Add IP to whitelist
echo "203.0.113.50" >> /etc/haproxy/files-net/ip_whitelist.txt

# Remove IP from whitelist
sed -i '/203.0.113.50/d' /etc/haproxy/files-net/ip_whitelist.txt

# Reload HAProxy (required for IP whitelist changes)
systemctl reload haproxy

🚨 Troubleshooting¶

Common Issues¶

Issue: Backend Server Shows DOWN¶

Symptoms: - Stats page shows server status: DOWN - Clients getting 503 Service Unavailable

Diagnosis:

# Check health check status
echo "show stat" | socat stdio /var/run/haproxy.sock | grep api1_pool1

# Manual health check
curl -I http://10.32.8.134:19169/health

Solution:

# If server is actually UP, force enable
echo "enable server shared_backend/api1_pool1" | socat stdio /var/run/haproxy.sock

# If server is DOWN, check API server
# SSH to API server and check IIS pool

Issue: Rate Limiting Not Working¶

Symptoms: - Clients not getting HTTP 429 - Rate limits not enforced

Diagnosis:

# Check rate limit map is loaded
echo "show map /etc/haproxy/files-net/rate_limit.txt" | socat stdio /var/run/haproxy.sock

# Check stick table
echo "show table rate_limit_table" | socat stdio /var/run/haproxy.sock

Solution:

# Reload map file
echo "clear map /etc/haproxy/files-net/rate_limit.txt" | socat stdio /var/run/haproxy.sock

# Reload HAProxy
systemctl reload haproxy

Issue: High CPU Usage¶

Symptoms: - HAProxy CPU >80% - Slow response times

Diagnosis:

# Check CPU usage
top -p $(pidof haproxy)

# Check connection count
echo "show info" | socat stdio /var/run/haproxy.sock | grep CurrConns

# Check session rate
echo "show info" | socat stdio /var/run/haproxy.sock | grep SessRate

Solution: - Check for DDoS attack (high session rate) - Verify rate limits are working - Consider scaling backend servers - Check SSL termination load

Issue: VIP Failover Not Working¶

Symptoms: - Floating IP not moving to standby - Traffic stops when master fails

Diagnosis:

# Check VRRP/Keepalived status
systemctl status keepalived

# Check IP addresses
ip addr show | grep 89.149.192.33

# Check peer sync
echo "show peers" | socat stdio /var/run/haproxy.sock

Solution:

# Restart keepalived
systemctl restart keepalived

# Check firewall allows VRRP (protocol 112)
iptables -L | grep vrrp

📝 Configuration Examples¶

Adding a New Client¶

Step 1: Add to rate limit map

echo "newclient.withinearth.com 75" >> /etc/haproxy/files-net/rate_limit.txt

Step 2: Add IP to whitelist (if needed)

echo "203.0.113.100" >> /etc/haproxy/files-net/ip_whitelist.txt

Step 3: Reload HAProxy

systemctl reload haproxy

Step 4: Verify

# Check rate limit is loaded
echo "show map /etc/haproxy/files-net/rate_limit.txt" | socat stdio /var/run/haproxy.sock | grep newclient

# Test from client IP
curl -H "Host: newclient.withinearth.com" https://89.149.192.33/api/xconnect/health

Adding a New API Server¶

Step 1: Edit HAProxy config

nano /etc/haproxy/haproxy.cfg

Step 2: Add 43 new server entries

backend shared_backend
    # ... existing servers ...

    # API-8 pools (NEW)
    server api8_pool1  10.32.8.200:19636 check inter 30s rise 2 fall 3
    server api8_pool2  10.32.8.200:19637 check inter 30s rise 2 fall 3
    # ... (41 more pools)
    server api8_pool43 10.32.8.200:19678 check inter 30s rise 2 fall 3

Step 3: Validate and reload

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl reload haproxy

Step 4: Verify

echo "show stat" | socat stdio /var/run/haproxy.sock | grep api8

📚 Quick Reference¶

Important Files¶

File	Path	Purpose
Main Config	`/etc/haproxy/haproxy.cfg`	HAProxy configuration
Rate Limits	`/etc/haproxy/files-net/rate_limit.txt`	926 rate limit rules
IP Whitelist	`/etc/haproxy/files-net/ip_whitelist.txt`	6,199 whitelisted IPs
SSL Certificate	`/etc/ssl/certificate.pem`	SSL/TLS certificate
Logs	`/var/log/haproxy.log`	HAProxy access/error logs
Socket	`/var/run/haproxy.sock`	Runtime API socket

Important Commands¶

Task	Command
Check Status	`systemctl status haproxy`
Reload Config	`systemctl reload haproxy`
Restart	`systemctl restart haproxy`
Validate Config	`haproxy -c -f /etc/haproxy/haproxy.cfg`
View Stats	`curl http://10.32.8.36:45001/stats`
Show Backends	`echo "show backend" \\| socat stdio /var/run/haproxy.sock`
Show Sessions	`echo "show sess" \\| socat stdio /var/run/haproxy.sock`
Disable Server	`echo "disable server backend/server" \\| socat stdio /var/run/haproxy.sock`

Important URLs¶

Purpose	URL	Access
Stats Page	http://10.32.8.36:45001/stats	Internal
Admin Panel	http://10.32.8.36:4444	Admin only
Management UI	https://haproxy-ui.withinearth.com/home	Admin only
Standby Stats	http://10.32.8.38:45001/stats	Internal

🔍 Performance Tuning¶

Recommended Settings¶

global
    maxconn 100000               # Maximum connections
    nbthread 4                   # Number of threads (= CPU cores)
    tune.ssl.default-dh-param 2048
    tune.bufsize 32768          # Buffer size

defaults
    timeout client 90s          # Client timeout
    timeout server 120s         # Backend timeout
    timeout connect 10s         # Connection timeout
    timeout http-request 10s    # HTTP request timeout
    maxconn 33000               # Max concurrent connections

Health Check Optimization¶

# Balanced health checks (not too aggressive)
default-server inter 30s rise 2 fall 3 check

# Explanation:
# inter 30s    = Check every 30 seconds
# rise 2       = 2 successful checks to mark UP
# fall 3       = 3 failed checks to mark DOWN

Last Updated: 2025-11-16 HAProxy Version: Aloha (Enterprise Edition) Configuration: Consolidated (695 lines, 2 backends, 301 servers)