Portal Home > Knowledgebase > Articles Database > Server comprised steps for recovery.
Posted by RelativeDesign-Jerret, 04-12-2008, 03:31 AM Well, one of the worst things (in hosting) has happened. I received a notice this morning from lfd (configserver) that someone had logged into my server as root and it wasn’t me. Unfortunately I didn’t notice it until eight hours later so I have no idea (yet) what happened during that period. Thankfully I don’t have any really critical data on the system that could have been stolen. I’m in the process of restoring from a full system backup right now. After that’s done I’m going to look to see what the differences are between the files from the backup and that on the comprised drive. I’m not sure if I’ll get anything useful from the diff but hopefully I’ll find a clue as to how they got root access. Then of course I need to get my server back up. However, I don’t want to do this until I’ve taken some steps to identify how the individual got in and take some additional preventative steps. Here’s what I am planning on doing: 1) Check to make sure all exposed services are patched and look at some security sites to see if there are any known vulnerabilities for these services. Anyone know which sites are good to look at? 2) Change firewall to only allow ssh access from a couple specific IP addresses. 3) Disable root ssh access so I have to login via a different account and perform sudos, etc. 4) ???? I’ll also look for a good server-hardening guide to see if there are some obvious things I forgot to secure. Do any of you find folks have any other suggestions or resources that I should check out? Thanks much, Jerret
Posted by andrewtayloruk, 04-12-2008, 04:44 AM There's lots of things to do to harden your server, and, some of it won't be appropriate depending on the services you need to offer, however, we do the following: 1) Disable all services which aren't being used (SSH / HTTP / HTTPS are it for us, we run email on a seperate cluster, we don't allow remote MySQL, we don't allow FTP) 2) Remove SSH password logins, generate private/public keys for authentication 3) Remove SSH root login 4) Use of SUDO 5) Strict IP tables in and out 6) Use of Logwatcher to keep a daily eye on your systems 7) Disable execution of scripts from Apache owned directories, or, directories with world rwx privileges. 8) Start logging to a remote syslog host, then you should always have good logs to track back the issue with. I've also just started to look at SELINUX and how that can be used to stop problems caused by PHP scripts. What else do people suggest? Andrew
Posted by tix3, 04-12-2008, 05:22 AM 1)Move ssh port to something high and not 22 2)Do not permit root login. 3)Install bfd so you are not vulnerable to brute force. 4)Try to have only updated to the last web scripts.Usually they are the root of all evil. 5)Regurarly update allo of your server software (ftp server,mysql server etc) 6)Use string passwords. 7)Disable all unneeded services. 8)Configure apache with suphp and/or suhosin so you would in a safer enviroment. 9)Install mod_security
Posted by whmcsguru, 04-12-2008, 07:40 AM A few things here: Changing the SSH port is what is known as 'security by obscurity', it's not going to actually STOP anything, it can still easily be found what port you're running on, and most script kiddies don't care about ssh in the first place. Disallowing root login, again, not a good idea at all. If services are down , or the server needs emergency work, you're effectively screwed. Oh sure, you can login as normal user, then su to root, but that takes time, and on a loaded server, time is one commodity that is precious and you don't hve. Now, the simple solution to both of those? Use access control lists. Of course, this, again, is relying on 'security by obscurity', but at least it's not complicating the matter any further than it has to be. To do this, simply enter into /etc/hosts.deny and What does this do? this prevents individuals outside of that range from ever getting ssh access to your server. Using SuPHP isn't recommended, as this can and has interefered with web scripts. Granted, many of these things are 'permissions' related, but many aren't. SuPHP/phpsuexec has known issues with say php values in .htaccess , symlinks, and a few other problems, and if the site is detected as using them, then it will just throw an error suhosin, on the other hand is a great recommendation. It's geared toward fixing the buggs without causing major interference. Rather than using BFD/APF, I would recommend using CSF / LFD. This software does just so much more out of the box to protect your server. As long as you're listening to what the server is saying, you're good to go. Now, all of that matters not if you don't pay attention to what your server is telling you. Your server talks to you, believe it or not. It sends you logs that you need to interpret. Not doing so is bad. Not reading through them means you don't know what your server is trying to tell you. Yet another reason I love CSF/LFD over APF is the amount of LOGGING that actually goes on. You know every single thing about a server that could be used against you, and, believe me, that's good. Most importantly: If you don't have the administrative skills to keep an eye on a server, don't claim you do. Find an administrator, even if it's only one of the < $30/month packages, just make SURE they are proactive, and not reactive to situations.
Posted by andrewtayloruk, 04-12-2008, 08:06 AM hosts.deny / hosts.allow is a good solution to some problems but I don't think it's better than public/private key for controlling SSH access. What happens if you're out on site, or on vacation, or your ISP goes down and you're at starbucks. No access to the server. As long as you can protect your own private key, you don't have to worry about this. Andrew
Posted by whmcsguru, 04-12-2008, 08:16 AM I fully agree with private keys, just not disabling direct root login. In my own scenario, I've always got a backdoor built in to my server, it's my cell. Sure, it's not the easiest to maneuver, but it works, and it always connects me on one specfic network . This way, I can at least modify the allow/deny files quickly if need be.
Posted by 3RH-Stan, 04-12-2008, 08:29 AM Try securing your system with ELS, its mostly automated and it does the job.
Posted by derek.bodner, 04-12-2008, 11:17 AM I'm not sure I could disagree more with this. If a server is so overloaded that you can't perform a sudo, then you're not going to get any pertinent work done with a server in that state anyway. If you're calling it a 'waste of time' that you don't have, it just took me 3 seconds to sudo on one of my servers. That's an absolutely inconsequential tradeoff for security. Logging in directly as root to save time is one of the worst pieces of security advice I've seen in quite some time. Besides cutting off one attack vector (the vast majority of brute force attacks are done as root), it's an absolute necessity if there are more than one person logging in to your server. Rather than setting up hosts.deny with a deny all (which IS complicating the matter. Imagine if you're on the road, with your laptop, which has changed IP's. Now imagine that even if you have a server setup at home that you ssh into, then ssh into your server from there, so you can keep your IP that's in hosts.allow, there WILL be a time when that server you're proxying through goes down, and now you've locked yourself out of the box), setup something like denyhosts that blocks people trying to brute force. Having your access lists with a global deny on the actual server, rather than on a dedicated hardware device like an external firewall, is setting yourself up for trouble down the line. If you have a dedicated appliance like a firewall that you can log into and add an IP should you lock yourself out of a box, then setup a deny all with a specific access list. Otherwise, I'd stay away from doing so, and use denyhosts instead. Password strength, password strength, password strength. Setting up private keys (with passphrases!) is also a very good idea. Finally, keep track of security vulnerabilities. Things like privilege escalations, buffer overflows, etc. There are various RSS feeds/alerts by security focus, redhat errata (or your distro of choice), linuxsecurity.com, etc that can help you stay up to date on found security vulnerabilities and how to go about fixing them. I would also not consider that box safe to continue using. Once a box has been rooted, I would never trust its integrity again. I would migrate off of the box asap, or at the minimum reformat and re-install. Last edited by derek.bodner; 04-12-2008 at 11:28 AM.
Posted by whmcsguru, 04-12-2008, 11:43 AM I've done PLENTY of work in 'loaded down' servers. Typically it's only one or two services that need to be stopped, or killed, and that output isn't as hard to get to. Authentication is always going to take more time to deal with and process than simply killing a process. As far as the 'on the road' scenaario, re-read the post. If you put in global access masks (ie: .yourprovider.net) , then you don't have to worry about changing things, or getting locked out. The alternative is to allow ssh key only access for root, which stops brute forcers right there, at least from the ssh side. Again, ssh attacks and the like are not the main priority. Disabling root access entirely (requiring a sudo user) is a backwards approach to a problem that is minimalistic at best, IF handled properly. Rather than complicate matters with requiring users to lookup passwords, usernames, etc, for each host, using ssh keys to authenticate is the perfect solution to this, on top of the ssh access limits. Whatever is done, the server most definitely needs to be investigated by a professional. There are a number of people here who know what they're doing, and, there are a number who don't. Start looking for someone to scan through things now.
Posted by derek.bodner, 04-12-2008, 12:39 PM That's nice. I'm sure you're the only one. You are VASTLY overstating the burden authentication places on a server. I logged into a server the other day with a 400 load average (4 core box) and sub 5% cpu idle. It took me virtually no time to sudo. Logging in directly as root is poor practice in so many ways (besides ssh attacks), this isn't even an argument worth having. To the OP, it's a trade off worth having. Using sudo allows for improved security, and better practices. You shouldn't run any commands as root that don't need to be run as root, and sudo helps prevent mistakes as well as being a great security tool. Two problems with that: 1) You're assuming he'd only use one provider 2) Let's say he uses both comcast and verizon frequently. Opening up *.comcast and *.verizon, and it's practically pointless. With a hardware firewall, you could set it up based on MAC address, so you can always have a connection from your laptop regardless of your IP or service provider. If you don't have the resources for a hardware firewall, and absolutely must have a global deny all policy, iptables is much more configurable and you can do something similar (allow based on MAC address). There is no one step to security. Disallowing root logins and setting up sudo, while not complete protection, is absolutely a step worth taking, and will not "effectively screw you".
Posted by RelativeDesign-Jerret, 04-13-2008, 01:53 AM Thank you all for your responses! I haven't had a chance yet to go through everything posted but I'll respond to all of you as soon as I get a chance.
Posted by RelativeDesign-Jerret, 04-14-2008, 02:53 AM First and foremost I would like to let you all know how much I appreciate your help. I’ve considered everything put forward here and tried to incorporate it into my server hardening process. I’ve set aside the drive that was in the compromised server and my provider performed a fresh centos 5 install. After completing installation I’ve performed the following steps: 1)Used yum to make sure all components were up to date. In addition I updated the yum update daemon to send e-mail notifications. 2)Removed all non-critical services (quite a few). In addition I’ve decided to move all DNS services to DNSMadeEasy and found no need to run an ftp server anymore as I always use SFTP anyways. 3)Updated sshd to listen on a "non-published" IP and different port 4)Removed sshd root login capability and forced protocol 2 5)Installed configserver (cfs and lfd) in place of APF + BFD 6)Installed ossec - OSSEC is an Open Source Host-based intrusion detection system 7)Installed RKHunter 8)Installed CHROOTKIT 9)Installed Clam AntiVirus 10) Secured /tmp, /var/tmp and /dev/shm 11) Installed mod_security 12) Installed mod_evasive 13) Installed suphp 14) Modified directadmin to accept ssl logins only 15) Used nmap to check for open ports 16) Updated all websites running 3rd party software. @andrewtayloruk - I've completed 1,3,4,5,6 and the rest will be implemented shortly. @tix3 - I've done everything you suggested with the exception of choosing CSF vs APF+BFD @linux-tech: a) Should sshd be restricted via hosts.deny or some other mechanism like csf? b) I've installed SuPHP for now since I operate a handful of sites on this server and can work through the inconveniences. I'll also look into suhosin though. c) LFD is what alerted me to the root login. Unfortunately I didn’t notice the e-mail until ~8 hours later. @3RH-Stan - I've looked at what ELS does and tried to cover all security related modifications that are performed. I've heard of people having issues with running this with my setup so I choose to do it all manually. @derek.bodner: a) I'll setup several jump boxes to "proxy" into my server with. That way I should be able to block everyone via deny.hosts and still have a fairly high assurance of being able to get into the box. As a last resort I’ll have my webhost do it, they’re awesome. b) Every password is randomly generated (I use RoboForm) and consists of at least eight uppercase and lowercase characters plus numbers. I also only copy / paste these. c) SSH private keys w/ passphrases is on my todo list. d) Subscribing to security vulnerability providers - great idea, I'll check out the ones you have listed and look for some others. e)Yup, the drive has been reformatted and we've put a fresh install on it. My remaining ToDo list (hopefully in the next day or two): 1) Remove SSH password logins, generate private/public keys for authentication 2) Disable execution of scripts from Apache owned directories, or, directories with world rwx privileges – I need to look into this more, I’m not sure who exactly to do this. 3) Start logging to a remote syslog host, then you should always have good logs to track back the issue with. 4) Restrict SSH login to specific IP addresses (or ranges). Cheers, Jerret Last edited by RelativeDesign-Jerret; 04-14-2008 at 02:55 AM. Reason: extra spaces
Add to Favourites 
Print this Article
