HAProxy – Slowing down abuse with user friendly rate controls

There are various situations where clients can overload a website server, but you don’t want to return a 4xx or 5xx simply because the system is getting bogged down, instead it would be better to have a mechanism which tries to take the pressure off the backend server to give all clients a fair chance. Thankfully HAProxy has a couple of ways that requests can be slowed down on their way to the backend, the cleanest mechanism is to use a small lua script ( HAProxy 1.6.0+ )

The fun comes when looking at how to trigger this delay and the example below uses a few different triggers to catch different scenarios.

/etc/haproxy/delay.lua

function delay_request(txn)
    core.msleep(1500 + math.random(1000))
end
core.register_action("delay_request", { "http-req" }, delay_request);

/etc/haproxy/haproxy.cfg

global
    lua-load /etc/haproxy/delay.lua

backend Abuse
    stick-table type ip size 100k expire 1m store gpc0,conn_cur,conn_rate(3s),http_req_rate(10s),http_err_rate(20s)

backend Abuse_Req
   stick-table type ip size 1m expire 12h store gpc0,gpc0_rate(5s)

frontend VIP1
    bind 1.2.3.4:80

    acl acl_ua_blacklist hdr_sub(User-Agent) -i slurp
    acl acl_ua_blacklist hdr_sub(User-Agent) -i spider
    acl acl_ua_blacklist hdr_sub(User-Agent) -i bot

    acl acl_ignore_this  path_end -i .png
    acl acl_ignore_this  path_end -i .jpg
    acl acl_ignore_this  path_end -i .jpeg
    acl acl_ignore_this  path_end -i .gif
    acl acl_ignore_this  path_end -i .woff
    acl acl_ignore_this  path_end -i .otf
    acl acl_ignore_this  path_end -i .ttf
    acl acl_ignore_this  path_end -i .svg
    acl acl_ignore_this  path_end -i .eot
    acl acl_ignore_this  path_end -i .css
    acl acl_ignore_this  path_end -i .js

    tcp-request connection track-sc0 src table Abuse

    acl acl_abuse sc0_conn_rate(Abuse) ge 10
    acl acl_abuse sc0_http_req_rate(Abuse) ge 100
    acl acl_abuse sc0_http_err_rate(Abuse) ge 20
    ## sc1_gpc0_rate = request rate specific to requests which get through the acl_ignore_this filter below
    acl acl_abuse sc1_gpc0_rate(Abuse_Req) ge 10

    acl acl_flag_abuser sc0_inc_gpc0(Abuse) ge 0

    ## delay requests when abuse is detected
    http-request lua.delay_request if !acl_ignore_this { sc1_inc_gpc0(Abuse_Req) gt 0 } acl_abuse acl_flag_abuser

    ## continue to delay all requests for the duration of the table expire time
    http-request lua.delay_request if { sc0_get_gpc0(Abuse) gt 0 }

    ## delay requests when there are more than 10 concurrent connections
    http-request lua.delay_request if { sc0_conn_cur(Abuse) ge 10 }

    ## delay requests from Spider Bots
    http-request lua.delay_request if acl_ua_blacklist

The above User-Agent blacklist has a very relaxed matching rule “bot” but because this setup does not deny requests and just adds the small delay to each request it would not matter too much if this ACL matched to something it should not have.

The general purpose counter gpc0 is used to record that abuse has been detected for the specific source IP

http-request lua.delay_request  if !acl_ignore_this { sc1_inc_gpc0(Abuse_Req) gt 0 } acl_abuse acl_flag_abuser

if the request is not in the acl_ignore_this ACL then increment the gpc0 in the sc1 table. Then acl_abuse contains the actually trigger levels for detecting abuse and it’s only when acl_abuse returns true that haproxy will call acl_flag_abuser which then increments the gpc0 counter and also returns true resulting in the request being delayed.

http-request lua.delay_request if { sc0_get_gpc0(Abuse) gt 0 }

This continues to delay requests while the gpc0 counter is greater than 0, which will be true for any IPs which have previously been flagged by one of the acl_abuse rules. This state will continue until the table entry has expired.

sc1_gpc0(Abuse_Req) is also useful for reporting the clients who make the most dynamic html and json calls in the last 12h, which is why this table has such a high expire time

View the Abuse table rows

root@lb1:~# echo "show table Abuse" | socat unix-connect:/run/haproxy/admin.sock stdio
# table: Abuse, type: ip, size:102400, used:380
0x20e1b1c: key=81.153.52.70 use=0 exp=33074 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
0x210b62c: key=82.132.236.240 use=0 exp=23402 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
0x1fa0c3c: key=82.132.242.131 use=0 exp=52647 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=3 http_err_rate(20000)=0
0x209ac9c: key=85.255.234.39 use=0 exp=41840 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
0x214d27c: key=86.113.162.57 use=0 exp=23190 gpc0=0 conn_rate(3000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(20000)=0
...

View the Abuse_Req table – Top Requests

root@lb1:~# echo "show table Abuse_Req" | socat unix-connect:/run/haproxy/admin.sock stdio  | sed -e "s/gpc0=//" | sort -t" " -nk5 | tail -10 
0x111c430: key=5.80.128.235 use=0 exp=42921961 21 gpc0_rate(5000)=0
0x1015340: key=109.153.251.211 use=0 exp=42562885 22 gpc0_rate(5000)=0
0xf5c5a0: key=80.229.228.105 use=0 exp=42974321 22 gpc0_rate(5000)=0
0xf5bed0: key=90.198.127.228 use=0 exp=43145733 23 gpc0_rate(5000)=0
0xfb81a0: key=92.40.249.168 use=0 exp=43030832 25 gpc0_rate(5000)=0
0x1085760: key=188.29.165.159 use=0 exp=43153401 28 gpc0_rate(5000)=0
0x1121ce0: key=94.197.120.126 use=0 exp=43064693 28 gpc0_rate(5000)=0
0xfebec0: key=79.69.141.143 use=0 exp=42881105 31 gpc0_rate(5000)=0
0x1129cb0: key=147.148.167.129 use=0 exp=42926477 46 gpc0_rate(5000)=0
0xf98e00: key=86.145.59.214 use=0 exp=42937952 48 gpc0_rate(5000)=0

References:
http://godevops.net/2015/06/24/adding-random-delay-specific-http-requests-haproxy-lua/
https://gist.github.com/jeremyj/e964a951634f1997daea
http://www.loadbalancer.org/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2

Frsky Taranis setup for Ardupilot

The Frsky Taranis is a very cool RC transmitter with some serious advanced functionality. Here are a couple of URLs covering how to configure the unit for Adrupilot mode switching

and also how to simplify auto tuning

UAV parachute recovery system

In the event that something goes horribly wrong while at high altitude it makes sense that things come back down under a bit of control!

http://droneyard.com/2013/07/01/uav-parachute-recovery-system/

What i’m struggling to solve is how to design the deployment system so that it does not tangle around the props when the UAV is falling in a tumble. I’m currently imagining a tub with either spring loaded deployment mechanism or a ballistic launch system.

Circular Wireless 5.8GHz Helical Antenna

A great antenna for long range FPV with the same circular polarisation as the clover leaf and skew antennas

http://www.circular-wireless.com/en/products-page/helical-modo-axial/heliaxial58/

Available in the UK from

http://quadcopters.co.uk/circular-wireless-58ghz-heliaxial-fpv-antenna-532-p.asp

UAV Gimbals

A brushless motor gimbal is the only way to really get a smooth filming platform

 

http://www.goodluckbuy.com/aluminum-brushless-camera-mount-gimbal-kit-w-motors-and-gimbal-controller-for-compact-camera-aerial-photography.html

http://www.unmannedtechshop.co.uk/2-axis-brushless-gimbal-controller.html

http://desirerc.co.uk/index.php?option=com_jcommerce&pgn=product_info&products_id=105

http://www.rctimer.com/product_950.html

Programmable LED strips

These are perfect for so many cool applications. I really want to upgrade the static red and white LED strips on my Quadcopter with these so that the whole strips and function as a white strobe as well as navaigation lights and also to show warning events, like low battery etc.

32 LEDs per meter – http://www.adafruit.com/products/306

 

60 LEDs per meter – http://www.adafruit.com/products/1461

Arduino code library –  https://github.com/adafruit/Adafruit_NeoPixel

It is a shame that the newer 60 LEDs per meter strips require a more accurate timing signal but the link above suggests that this should not be a problem for the modern Arduino boards.

Adafruit NeoPixel Digital RGB LED Weatherproof Strip 60 LED -1m

Giff Gaff – call waiting

Call waiting activate call waiting *43#
Call waiting cancel call waiting #43#

Arducopter – the reason electric motors were invented

http://copter.ardupilot.com

Copyright © All Rights Reserved · Green Hope Theme by Sivan & schiy · Proudly powered by WordPress