High disk utilization when 5k connections on nginx

Portal Home > Knowledgebase > Articles Database > High disk utilization when 5k connections on nginx

Posted by Mr_Parham, 04-19-2014, 11:17 AM
Hi We have a download server which before it used to serve 800-900Mbps no problems but since a while ago the number of the connections has increased (It use to be under 3000 and now during the day it's 5000 and in peak time it would reach 20000 connections) it seems to only be able to max 450-500Mbps (disk ultilization would be %100 and there is no way to push more thn that number out of server), I played around with the server but there was nothing that I could find, the only thing that I could think of was that nginx need more ram but it seems like we still have plenty of ram free root@expressleech-new:~# free -m total used free shared buffers cached Mem: 32218 32017 200 0 0 17565 -/+ buffers/cache: 14451 17767 Swap: 11572 3131 8441 here is server detail 2*Intel(R) Xeon(R) CPU E5-2620 @ 2.00GHz 32GB Ram 16*4TB Enterprise 7200rpm disk (Hardware Raid 50) 2*500GB Enterprise 7200rpm disk (Hardware Raid 1) 2*1Gbps dedicated unmetered uplink Anyone have any idea what the issue might be? PS - I did a test from my other servers, I can reach 800-900Mbps again if I close all the nginx connections (well I blocked them all and only allow the ones which were related to my test)
Posted by net, 04-19-2014, 11:27 AM
Moved > Hosting Security and Technology .
Posted by fabin, 04-19-2014, 12:06 PM
You cannot say that there is plenty of ram in server. At some point if time 3GB of swap was needed.
Posted by fabin, 04-19-2014, 12:08 PM
During peek time if CPU is utilized more for IO, then try SSD hard disk.
Posted by Steven, 04-19-2014, 12:12 PM
What does your IO wait look like? You may be running out of IOPS.
Posted by Mr_Parham, 04-19-2014, 08:06 PM
I would try upgrading the ram then to see how it goes =/ Here is some peak time stats Active connections: 18052 ===================================== Linux 3.2.0-4-amd64 (sh4) 04/20/2014 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.14 0.00 2.61 4.65 0.00 91.59 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sdb 326.63 58584.39 13107.94 18652269932 4173344088 sda 8.03 55.82 67.63 17771427 21530648 ===================================== top - 04:35:06 up 3 days, 16:27, 1 user, load average: 62.02, 43.81, 57.13 Tasks: 236 total, 1 running, 235 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.8 us, 0.9 sy, 0.0 ni, 85.8 id, 9.3 wa, 0.0 hi, 2.2 si, 0.0 st KiB Mem: 32991836 total, 29777408 used, 3214428 free, 3816 buffers KiB Swap: 11850748 total, 11964 used, 11838784 free, 2379308 cached ===================================== total used free shared buffers cached Mem: 32218 28857 3361 0 3 2325 -/+ buffers/cache: 26528 5690 Swap: 11572 11 11561
Posted by funkywizard, 04-19-2014, 09:50 PM
Try turning off proxy buffering in Nginx. What's your disk configuration? Raid 1 / raid 10? Sometimes tweaking readahead values can help tremendously.
Posted by Mr_Parham, 04-20-2014, 05:54 AM
Raid 50, we already tried playing around with readahead but no luck, and I would try proxy buffering thing as well
Posted by funkywizard, 04-20-2014, 06:02 AM
Proxy buffering turned on may cause nginx to write to disk in some situations needlessly. I would suggest turning it off just to be safe. The correct readahead value will depend upon your raid stripe size. For loading large files, your ideal configuration would be a 2MB stripe size on the raid array with a 512kb readahead value. This should provide approximately 4x the performance of the default values for loading large files. If you'd like I could go into detail as to why these are the optimal values, but suffice it to say that a 512kb readahead is the optimal value generally, but in a raid you want the readahead to ideally be 1/4 of the stripe size to avoid unnecessarily reading from multiple disks for a single read request. For the optimal 512kb readahead value to be advisable, this would therefore require a 2mb stripe size, which is much larger than the default and therefore you probably have a much smaller stripe size.
Posted by Infinitnet, 04-20-2014, 06:13 AM
Like @funkywizard said, you could try increasing your readahead size. Furthermore you could try mounting with noatime, disable the NGINX access logs and disable proxy buffering and you can also try caching static files, so it doesn't have to fetch them all the time (if the files aren't on the same machine).
Posted by Mr_Parham, 04-20-2014, 02:05 PM
About the readahead I checked with my server admin and he said this is not possible on our server (I think because of the raid controller model, I wasn't sure how to do it so I asked him), about the access log it's set to be off now, the proxy buffering we didn't configure it on server as nginx is just serving static files, the disks has already been mounted with noatime and about caching I don't see how that would work on this server PS - We have upgraded the ram today from 32GB to 128GB, I would wait and see how it goes in the peak time but I don't think it made that much of a difference Parham
Posted by funkywizard, 04-20-2014, 04:40 PM
It is always possible to change the readahead. It may not be possible to change the stripe size. This is why hardware raid controllers are not a good match for large file hosting such as tube sites / upload sites. Software raid will allow whatever stripe size you want, hardware raid may not. To get adequate disk performance you *need* a 2MB stripe size, and most raid cards do not support that. Even with software raid, you will need to set the stripe size when you first create the array, you cannot change it afterwards without formatting data. Perhaps this is what your technician meant by saying it cannot be done?
Posted by Mr_Parham, 04-20-2014, 04:44 PM
Well I checked with him and he said yes that's what he meant, is there any raid controller which you recommend for 16 disks so I can use that for me new server which I can change the stripe size? Regards Parham
Posted by funkywizard, 04-20-2014, 04:46 PM
You could actually pass through the drives individually to the OS from the raid controller (jbod mode), and then form a software raid inside the OS even though you're using a hardware raid card. That's how I'd do it in this scenario.
Posted by Mr_Parham, 05-04-2014, 05:42 AM
We tried this on a smaller server and we have which was easier to move the data of the hard disks so we can apply the changed and bring them back on the server but it didn't make such a massive difference, I checked with the datacenter and they recommended adding SAN to my server which would hopefully solve my issue, now there is few question PS - I never ever worked with SAN before so I'm sorry if the questions seems a bit strange 1- Does the SAN have it's own raid controller because they said the model they are going to get me have 96 disks 2- If I add SSD caching to my server is that going to help with the SAN storage performance as well or does it need it's own SSD to be able to boost up the performance 3- if SAN don't have it's own raid controller then what is the best raid controller which I would be able to use to be able to use my entire bandwidth, I can connect up to 10*1Gbps uplinks to my server and cpu and ram should not be a problem so if my disks can handle this server should be enough for 6 month of my entire website
Posted by funkywizard, 05-04-2014, 05:55 AM
A san generally adds large amounts of cost and reduces performance. They have their place but I would seriously doubt it is an appropriate solution to your problem.
Posted by Mr_Parham, 05-04-2014, 06:03 AM
Our website is storage hungry so that's why they recommended it, we have 24 cores on the server and usually only use 2 of them and the rest are not used at all and the ram we can upgrade whenever Also adding a SAN would be cheaper for us then adding new server, do you still not recommend it?
Posted by singaporevps, 05-04-2014, 06:07 AM
I suggest don't use SAN. I uses SANless technology where data replicas across many nodes. This helps me save a lot of IO
Posted by MassiveGRID, 05-04-2014, 06:11 AM
Hello, It's easy to see high disk utilization when using 7200 spinning drives in high traffic servers. The easiest and fastest answer to your question is: use SSDs If you can't use SSDs due to cost, then you'll have to dig into your problem to find a solution. Questions you should ask and explore solutions based on the answers are: a) Is the most traffic comes from some files or it is distributed to all our files hosted on my RAID50? b) if (a) yes, are these files small enough to use a RAM caching mechanism (and increase the RAM maybe) c) is the bottleneck the RAID controller? d) can I use an ssd for the highest traffic files and the rest will be served from the RAID50? Depending on the answers you can find the solution.
Posted by Mr_Parham, 05-04-2014, 01:59 PM
Buying SSD disks is not a problem but we only can get hold of the 500GB SSD disks, I would need quite a lot of servers to be able to host the 100TB data that we have at the moment and each one of my servers can only support up to 16 disks so that would be the problem a) It's distributed on all the files b) we tried increasing the ram from 32GB to 128GB no difference, and we get files from 0.5MB to 50GB and most of them are about 400-500MB so c) I don't know because I don't know any good raid controller, I did ask for recommendation on this threat before d) nope, not possible
Posted by Mr_Parham, 05-15-2014, 04:06 PM
We ordered a new server hoping we would be able to cache the hot files on the SSD disks and the rest of the traffic from SAS disks but the problem I'm having is that for each download our script generate a symlink to the file, now after doing some configuration on nginx it seems like nginx is caching a file with high download multiple times because the link that user open to download the file is the symlink, anyone have any idea how can go around this? Parham

Add to Favourites Print this Article