There are various situations where clients can overload a website server, but you don’t want to return a 4xx or 5xx simply because the system is getting bogged down, instead it would be better to have a mechanism which tries to take the pressure off the backend server to give all clients a fair chance. Thankfully HAProxy has a couple of ways that requests can be slowed down on their way to the backend, the cleanest mechanism is to use a small lua script ( HAProxy 1.6.0+ )

The fun comes when looking at how to trigger this delay and the example below uses a few different triggers to catch different scenarios.

/etc/haproxy/delay.lua

function delay_request(txn)
    core.msleep(1500 + math.random(1000))
end
core.register_action("delay_request", { "http-req" }, delay_request);

/etc/haproxy/haproxy.cfg

global
    lua-load /etc/haproxy/delay.lua

backend Abuse
    stick-table type ip size 100k expire 1m store gpc0,conn_cur,conn_rate(3s),http_req_rate(10s),http_err_rate(20s)

backend Abuse_Req
   stick-table type ip size 1m expire 12h store gpc0,gpc0_rate(5s)

frontend VIP1
    bind 1.2.3.4:80

    acl acl_ua_blacklist hdr_sub(User-Agent) -i slurp
    acl acl_ua_blacklist hdr_sub(User-Agent) -i spider
    acl acl_ua_blacklist hdr_sub(User-Agent) -i bot

    acl acl_ignore_this  path_end -i .png
    acl acl_ignore_this  path_end -i .jpg
    acl acl_ignore_this  path_end -i .jpeg
    acl acl_ignore_this  path_end -i .gif
    acl acl_ignore_this  path_end -i .woff
    acl acl_ignore_this  path_end -i .otf
    acl acl_ignore_this  path_end -i .ttf
    acl acl_ignore_this  path_end -i .svg
    acl acl_ignore_this  path_end -i .eot
    acl acl_ignore_this  path_end -i .css
    acl acl_ignore_this  path_end -i .js

    tcp-request connection track-sc0 src table Abuse

    acl acl_abuse sc0_conn_rate(Abuse) ge 10
    acl acl_abuse sc0_http_req_rate(Abuse) ge 100
    acl acl_abuse sc0_http_err_rate(Abuse) ge 20
    ## sc1_gpc0_rate = request rate specific to requests which get through the acl_ignore_this filter below
    acl acl_abuse sc1_gpc0_rate(Abuse_Req) ge 10

    acl acl_flag_abuser sc0_inc_gpc0(Abuse) ge 0

    ## delay requests when abuse is detected
    http-request lua.delay_request if !acl_ignore_this { sc1_inc_gpc0(Abuse_Req) gt 0 } acl_abuse acl_flag_abuser

    ## continue to delay all requests for the duration of the table expire time
    http-request lua.delay_request if { sc0_get_gpc0(Abuse) gt 0 }

    ## delay requests when there are more than 10 concurrent connections
    http-request lua.delay_request if { sc0_conn_cur(Abuse) ge 10 }

    ## delay requests from Spider Bots
    http-request lua.delay_request if acl_ua_blacklist

The above User-Agent blacklist has a very relaxed matching rule “bot” but because this setup does not deny requests and just adds the small delay to each request it would not matter too much if this ACL matched to something it should not have.

The general purpose counter gpc0 is used to record that abuse has been detected for the specific source IP

http-request lua.delay_request  if !acl_ignore_this { sc1_inc_gpc0(Abuse_Req) gt 0 } acl_abuse acl_flag_abuser

if the request is not in the acl_ignore_this ACL then increment the gpc0 in the sc1 table. Then acl_abuse contains the actually trigger levels for detecting abuse and it’s only when acl_abuse returns true that haproxy will call acl_flag_abuser which then increments the gpc0 counter and also returns true resulting in the request being delayed.

http-request lua.delay_request if { sc0_get_gpc0(Abuse) gt 0 }

This continues to delay requests while the gpc0 counter is greater than 0, which will be true for any IPs which have previously been flagged by one of the acl_abuse rules. This state will continue until the table entry has expired.

sc1_gpc0(Abuse_Req) is also useful for reporting the clients who make the most dynamic html and json calls in the last 12h, which is why this table has such a high expire time

View the Abuse table rows

root@lb1:~# echo "show table Abuse" | socat unix-connect:/run/haproxy/admin.sock stdio
# table: Abuse, type: ip, size:102400, used:380
0x20e1b1c: key=81.153.52.70 use=0 exp=33074 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
0x210b62c: key=82.132.236.240 use=0 exp=23402 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
0x1fa0c3c: key=82.132.242.131 use=0 exp=52647 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=3 http_err_rate(20000)=0
0x209ac9c: key=85.255.234.39 use=0 exp=41840 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
0x214d27c: key=86.113.162.57 use=0 exp=23190 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
...

View the Abuse_Req table – Top Requests

root@lb1:~# echo "show table Abuse_Req" | socat unix-connect:/run/haproxy/admin.sock stdio  | sed -e "s/gpc0=//" | sort -t" " -nk5 | tail -10 
0x111c430: key=5.80.128.235 use=0 exp=42921961 21 gpc0_rate(5000)=0
0x1015340: key=109.153.251.211 use=0 exp=42562885 22 gpc0_rate(5000)=0
0xf5c5a0: key=80.229.228.105 use=0 exp=42974321 22 gpc0_rate(5000)=0
0xf5bed0: key=90.198.127.228 use=0 exp=43145733 23 gpc0_rate(5000)=0
0xfb81a0: key=92.40.249.168 use=0 exp=43030832 25 gpc0_rate(5000)=0
0x1085760: key=188.29.165.159 use=0 exp=43153401 28 gpc0_rate(5000)=0
0x1121ce0: key=94.197.120.126 use=0 exp=43064693 28 gpc0_rate(5000)=0
0xfebec0: key=79.69.141.143 use=0 exp=42881105 31 gpc0_rate(5000)=0
0x1129cb0: key=147.148.167.129 use=0 exp=42926477 46 gpc0_rate(5000)=0
0xf98e00: key=86.145.59.214 use=0 exp=42937952 48 gpc0_rate(5000)=0

References:
godevops.net/2015/06/24/adding-random-delay-specific-http-requests-haproxy-lua/
gist.github.com/jeremyj/e964a951634f1997daea
www.loadbalancer.org/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2