Unfortunately sometimes our servers might get hacked. Following are the few steps for our initial investigation.
1. Check the csf/lfd logs under /var/log, you may see messages like
Time: Wed Feb 20 12:05:22 2009 -0700
Uptime: 46951 seconds
Command Line (often faked in exploits):
Network connections by the process (if any):
tcp: 0.0.0.0:80 -> 0.0.0.0:0
tcp: 0.0.0.0:443 -> 0.0.0.0:0
tcp: Server-IP:45500 -> Some public-IP:7000
2. Check the nobody process alerts of server.
3. If rkhunter is installed in the server, check the rkhunter scan alert to know whether any binary of recent changes happened in server.
4. Login into server and run the below commands to identify it.
netstat -plan| grep :6667
lsof -p PID | grep cwd
find the root cause and directory in which it is running.
5. Check the sendmail and check whether it is binary or perlscript. If it is perl, most probably it will be setup to append the path info to /var/log/formmail.log or /var/log/spamd.log.
6. Run the “clamscan -ir DIRECTORYPATH” to verify whether any suspicious information in it.
7. Kill the associated running processes.
8. Restart the service which causing the issue, move the folder/file contents to some other location. Add attribute, permission, ownership modification on it. Set immutable bit on it.
9. Check the “netstat -plan| grep :6667” and block the IP from which connection is coming.
csf -d IP “REASON”
(eg. csf -d IP “Blocked the IP due to IRC connection from it”)