Portal Home > Knowledgebase > Articles Database > syn flood - help
syn flood - help
| Posted by rehana, 05-11-2009, 05:38 PM |
| I've been getting a syn flood for the last week or so.
I've pretty much tried everything I could online but have been unsuccessful in stopping them.
I talked to the data center techs and they basically can't put a stop to it either.
Here's a very small portion of my netstat
tcp 0 0 xxx.xxx.xxx.xxx:80 86.50.121.144:8540 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 41.100.16.152:7824 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 52.53.22.7:3146 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 77.217.49.124:1659 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 75.162.93.151:3230 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 113.85.63.249:1656 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 15.253.35.29:8849 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 24.56.59.180:6911 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 33.185.99.83:1917 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 103.5.8.249:4782 SYN_RECV -
root@xxx [/]# netstat -nap |grep SYN |wc -l
2008
The IP's change often and it's not possible to narrow it down.
So far the things I have done;
syn cookies enabled
reduced time out
echo "1" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv
increased
echo "150000" > /proc/sys/net/ipv4/ip_conntrack_max
installed apf but it slowed down the server to a crawl which made my clients really unhappy so had to remove it.
The bandwidth is constantly staying at 30Mbps with slight bumps here and there but every day around 7pm it drops completely to normal levels and the flood stops. It starts back up around 7 in the morning.
I could really use some help, because I'm seeing the bandwidth going higher right now and this is driving me insane.
It's already started on another server now, so this is getting insane.
thanks
machine has centos 5.3 with cpanel
|
| Posted by CiscoMike, 05-11-2009, 06:21 PM |
| it has to be dealt with upstream. Syn cookies, in this case, are making things worse as well. For every 1/2 open connection (and don't kid yourself, they are 1/2 open as the other side will never complete the handshake) you're losing memory. Granted they'll time out in a more graceful fashion with syn cookies but you're checking state in a place where state checking is going against you.
The simple answer is to d/rtbh your server (aka null route to you) but that obviously will take you off-line. s/rtbh won't work with a number of different IPs. At 30Mbps you could look at some sort of hardware device (just saying it makes me cringe) but depending on the flow setup rate, that device will also take a shot in the pants.
Moral of the story: this isn't something you can really address on your server. You need the upstream network to assist by filtering out the non-completed connections. a dedicated DDoS appliance will help but again, depending on your flow setup rate (if it's in the magnitude of high ten-thousands, i.e. 90k cps or higher) then even a dedicated appliance will fall over on itself.
edit: assuming it's all traffic destined for your web daemon, you could leverage mod_evasive but that still doesn't alleviate the issue with your server getting hammered, it just protects the web daemon at the expense of more memory being chewed up. There are other negative implications from an untuned mod_evasive as well such as accidentally blocking your own connections (or other legitimate connections) if you tune it too tightly.
Last edited by CiscoMike; 05-11-2009 at 06:25 PM.
|
| Posted by The Universes, 05-11-2009, 06:23 PM |
| syncookies enabled by the way? Do you have DDOS Deflate installed?
APF should not slow down the server, in terms of CPU usage and what not. I'd be willing to bet the slowdown as due to conntrack filling up (which APF sets on based on its config file).
I would put APF back on the machine, adjust and tighten some of the settings. More than likely, you will need a hardware solution to truly mitigate this.
|
| Posted by atariko, 05-11-2009, 10:14 PM |
| You can try to determine the TTL of the attackers and then drop SYNs in IPTables that have that TTL. It looks like the source IP and port are being randomized?
|
| Posted by rehana, 05-11-2009, 10:46 PM |
| Some of the people at the datacenter as well as cpanel recommended a dedicated system but at the moment its' quite expensive and not in the budget. I've been looking at mod_evasive but a little hesitant because just as you said, it could be a problem.
I can't null route the traffic either since it's got clients being hosted. I'll probably turn off syn cookies and give that a try.
The machine is sturdy, its' got 32GB of RAM and dual quad cores. Load hasn't gone above 2 but apache is getting bogged down from time to time.
|
| Posted by rehana, 05-11-2009, 10:49 PM |
| I do have DDOS Deflate installed and running already. From what I see, it's looking for a particular number of connections from an IP, rather than a large number of different IP's. I still get "an IP banned" email every 10 minutes or so.
Conntrack was set to 150,000 (which I changed it to after setting up APF)
I wish I had the money to get a hardware solution. This has just been a huge headache for the last week or so. Not much sleep
|
| Posted by rehana, 05-11-2009, 10:52 PM |
| I'm intrigued. Can you explain how to do this more? I'm pretty noobish when it comes to this stuff.
Yes, the IPs and ports are randomized and they change very often.
Thanks for the info again, everyone. Would appreciate more comments still.
|
Add to Favourites 
Print this Article
Also Read
