It has been two weeks since I have not made a new blog post. There are some reasons behind this. I am busy with the work.
However, I don’t ignore my blog and actually was writing 2 new blog posts; one for the e-mail security with GPG and another one for my third Nessus blog post. Those are still in progress. I just saved them and will complete as soon as I have more free time.
I am currently visiting Rackspace Cloud at San Antonio. I started to write this blog post in the plane and now I will complete it in my hotel room…
I am currently writing an article for the Slicehost customers to show them how to investigate their slices (Linux VPS) during a possible compromise.
I am doing some research and implementing my knowledge on the Slicehost environment which takes quite time to complete the article.
I thought it would be good to have a blog post about a more general environment. This is the my first forensic related post. Yes, I have huge interest on Computer Forensics.
First of all, this article covers only the basic of Linux forensics. By saying that I won’t cover any highly sophisticated forensic techniques here ( at least in first two articles)
The aim of this blog post is simply showing you the way you can investigate your compromised Linux machine and learn from your mistakes. ( I will have articles about some advanced forensics tools such as autopsy, vinetto and MboxGrep later)
IMPORTANT WARNING: Before you do anything, you need to make an important decision—do you plan to involve law enforcement and prosecute the attacker? If the answer is yes, you should leave the compromised system alone and make no changes to it.
Any changes you make post-attack could complicate and taint the evidence, and because of that, many people have a policy of unplugging a system once they detect an attack and leaving it off until law enforcement arrives.
Investigators likely will want the complete system, or at least the drives, so they can store it safely; thus, your forensics analysis might end here until your system is returned. 
Nobody is perfect. Everybody can make mistakes. However, I avoid as much as possible to make same mistake twice. I believe only stupid people do that. At least, I feel stupid if I do same mistakes.
Ok, back to our lesson. We have a compromised Linux machine. First be calm. It is ok to get hacked. We are not only ones whose boxes got cracked. Of course, good system administrators will do everything to avoid this type of situation.
However, even if you believe you are so knowledgeable system admin, your machines can be hacked by an attacker who exploited a new discovered vulnerability…
Checking Network Connection
Check the network connections and open ports with netstat command.
By running this command you can see the any backdoors that are listening
tcp 0 0 0.0.0.0:6697 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN
In this case we see port 6697 is open. It is not a good sign because that port is used by IRC. We can sniff the connection by tcpdump. For more info on tcpdump, check this blog post.
tcpdump src port 6697
You can check here for more info on IRC bots.
Checking Last Logged in IPs
Brute force attack is a very popular type of attack. You may be able to find who was attacked you by checking last logged in IPs with the last command.
Using last you can determine the time a user logged in and out. It also provide you the hostname / IP address from where the user logged in from.
This will give us last 25 users’ IP who logged in the system.
/var/log/auth_log file can also have valuable information regarding to successful or failed login attempts.
Checking Last Commands
You may have heard “No crime is perfect” a lot if you have ever watched the Forensic Files TV show. It is true. Only a few good hackers cannot leave their finger prints on their digital crime.
For example, most of the time intruders leave their their .bash_history files. .bash_history file contains the last commands used with the bash shell.
This can give us a lot information about what they did, what they installed and where they got their files from. Typical entries may include,
tar xf malware.tar
This tells us the url they got the malware from, how they ran it, and where it was
installed. A good starting point for looking for their directory! 
Be aware of the way .bash_history store the information! It only show the all commands which has been run by a spesific user after he logged out.
In case attacker is logged in and you are trying to check his .bash_history, you may see an empty file.
Use who command to see active users on the machine.
user1 pts/0 Nov 18 23:33 (220.127.116.11)
user2 pts/1 Nov 16 10:22 (18.104.22.168)
We see two active users on the system. If user2 is compromised account, we should tcpdump and monitor his activity:
tcpdump host 22.214.171.124 -w demo.dump
You can also use thehistory command to list the history of the last executed commands.
To get more useful information from history command and .bash_history file, let’s modify /etc/profile directory.
Add following line at the end of the file:
export HISTTIMEFORMAT=”%h/%d – %H:%M:%S “
You can now see the time when the commands run. (You will be able to see all commands with time stamps on the history ‘s output.
However, for .bash_history, you will be only able to see time stamps for new commands which is not useful for us.
We learn some basic information for investigating compromised Linux machines such as checking network connection, active users on the system, getting bash history, last logged in users IP etc… All of these are so critical information to track intruders and find security holes on the system.
The next post will discuss integrity checks and some helpful tools such as rootkit scanners.