Pile up of crond processes at certain times?

Portal Home > Knowledgebase > Articles Database > Pile up of crond processes at certain times?

Posted by iUnknown, 05-10-2009, 06:04 PM
To me, this sounds really strange and really unrelated, hopefully it will ring a bell in somebody-who-reads-this' mind... After enabled suexec in my webserver (litespeed), every day at two regular times, I notice a ton of crond processes on many different users. This slows down my entire server until they go away and it must be resolved. I have only recently noticed that it is infact crond processes causing this and I hope I have finally found the correct issue. It causes a chain reaction of events and so I have slowly had to crawl up the chain to find the root of the issue... (high iowait, high swapping, high memory, normal processes taking up more cpu%, etc. etc. ........ crond processes) I have checked every single file in /var/spool/cron and none of the cron jobs run remotely near these times (except some that run every minute/15 minutes for example). Also, I don't understand how so many crond processes could be made at once. I will make a better rough estimate when it next happens. What could it be causing this to happen? Thanks very much.
Posted by sahsanu, 05-10-2009, 07:04 PM
Hello, Did you check /etc/crontab file?. After check this file, take a look into these directories: /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly I hope this helps. Regards, sahsanu
Posted by iUnknown, 05-10-2009, 07:07 PM
Crontab just runs: /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly And those run hourly/daily/weekly/monthly, not at say 5.17PM and 7.02PM.
Posted by sahsanu, 05-10-2009, 07:17 PM
Yes, crond uses /etc/crontab to run scripts hourly, daily.... but times are not as every hour o'clock, exactly midnight... This is my crontab, just as example: 6 * * * * root cd / && run-parts --report /etc/cron.hourly 37 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 39 0 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 46 2 27 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) If you checked your /etc/crontab file and /etc/cron.d/ directory and you can't see anything weird, then I have no idea where your problem is .
Posted by sahsanu, 05-10-2009, 07:21 PM
I forget it, check your syslog to see if any cron task was running at those hours. grep -i cron /var/log/syslog
Posted by iUnknown, 05-11-2009, 02:31 AM
grep: /var/log/syslog: No such file or directory Looking at /var/log/cron .... What does /usr/lib/sa/sa1 1 1 do? What does /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1 do? (I understand it monitors CPU in some way, but is that all it does - monitor? Thanks.
Posted by jphilipson, 05-11-2009, 02:44 AM
dcpumon is a CPU/Memory accounting program run by cpanel that stores percentage values of CPU/Memory/mysql usage under /var/log/dcpumon. Then you can view those stats with /usr/local/cpanel/bin/dcpumonview, or click the CPU/Memory/MySQL usage link in WHM. sa or sar is a utility to monitor system performance and activity, normal on linux systems. When you have these moments of high load, what processes are you seeing using the top and su commands, and tailing /var/log/cron?
Posted by iUnknown, 05-11-2009, 12:25 PM
The top processes are normal processes in the normal order. I have been tailing the cron log file for a while now and the same standard crons are coming up. The load just spiked and the only odd cron in the list was: /usr/lib/sa/sa1 1 1 Is this a required cron? Do I really need this cron or the dcpumon one? Edit: server load recovering now: Cpu(s): 12.4%us, 6.0%sy, 0.0%ni, 0.8%id, 78.8%wa, 0.2%hi, 1.8%si, 0.0%st Clearly the iowait is the problem, but there is no easy way of seeing what is causing the iowait? As a little swap is being used, I assume that is what is causing the iowait, perhaps /usr/lib/sa/sa1 1 1 causes an extremely high spike in memory usage? Last edited by iUnknown; 05-11-2009 at 12:28 PM.
Posted by sahsanu, 05-11-2009, 06:32 PM
sa1 command collect and store binary data in the system activity daily data file. It's part of sar command, if you aren't using a tool to show system activity graphs you may disable it but, I don't think it is causing high memory usage. This command runs once and in one second, so discard it as your memory/load usage issue. dcpumon is a CPU/Memory accounting script. You can see it using dcpumonview command (/usr/local/cpanel/bin/dcpumonview). Same as above, if you don't care about these kind of statistics you can disable it. As you said, once your system starts using swap, your load will increase a lot due io wait usage. Try to free memory (optimizing services or disabling those not needed). Then monitor it for a while and pay attention to your io wait percentages because it could not be your fault, maybe some neighbour is running backups or using too much disk resources. Good luck. sahsanu
Posted by iUnknown, 05-11-2009, 06:46 PM
Thanks for the advice. My systems run fine normally, with no lag at all. It is at these two times of the day where for some reason memory usage spikes causing a chain reaction of events. I will take a careful look next time at the ps (aux) output and try and work out what exactly is making the memory usage more than usual.

Add to Favourites Print this Article