Tonight I just saw that my Freebsd server has been compromised since 3 days ago.
I did a fresh install of Freebsd 7.2 at the end of November and then setup an account to take care of this blog backup.
Today I tried to take a look on the backup data and am unable to log on the account. Maybe I typed a wrong password… mmmh, in fact I used an obvious one to not forget it (same as login…). And the account was accessible by SSH.
After a few tries the fact strikes me: if I can not login on the account, it is because it has been changed by somebody else…
First step: I login as root and change all my account passwords. Now I can go on to look for what happened.
Connection attempts are logged in /var/log/auth.log* files.
In auth.log.1.bz2, bingo: I find the first successful foreign connection. Obviously it also contains a LOT of other attempts from various attackers armed with dictionaries of common login/password.
Dec 7 01:53:03 dragon sshd: Accepted keyboard-interactive/pam for backup from XX.YY.28.27 port 44670 ssh2 Dec 7 01:53:03 dragon sshd: Received disconnect from XX.YY.28.27: 11: Bye Bye
The account has been detected by a China IP. It is accessed for real a few hours later by a Romanian IP:
Dec 7 14:04:52 dragon sshd: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT! Dec 7 14:04:52 dragon sshd: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 1215 ssh2 Dec 7 14:15:25 dragon sshd: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT! Dec 7 14:15:27 dragon sshd: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 1232 ssh2 Dec 7 22:41:42 dragon sshd: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT! Dec 7 22:41:43 dragon sshd: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 4241 ssh2 Dec 7 22:43:03 dragon sshd: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT! Dec 7 22:43:03 dragon sshd: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 4243 ssh2 Dec 7 22:43:03 dragon sshd: subsystem request for sftp
10 minutes then 8 hours then another 2 minutes, it looks like somebody has taken the time to install something…
Footprints of shell activity can be found in the .bash_history file of the account:
w ps x uname -a exit
First the pirate looks if there is anybody there with w (display who is logged in and what they are doing), then he checks the running processes, then at last he checks the OS version. For me:
dragon# uname -a FreeBSD dragon.local 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May 1 08:49:13 UTC 2009 email@example.com:/usr/obj/usr/src/sys/GENERIC i386
Second attempt, same recipe but this time he request a password change:
w ps x uname -a passwd exit
Third connection, things are added:
w ps x clear cd /var/tmp mkdir " " cd " " uname -a cd all chmod +x * ./bash exit
The attacker created an “invisible” directory named with 3 spaces, cd’ed in a “all” directory and then executed a “bash” program. Where this program comes from I don’t know. How has it been copied here ?
I can check that the program is in fact still running:
dragon# ps -x ... backup 6886 0.0 0.1 544 384 ?? Ss Mon10PM 0:01.67 ./bash ...
What is this program doing ? I can check if any socket connection is opened with netstat and sockstat:
dragon# netstat ... tcp4 0 0 192.168.1.2.60487 irc2.xxxxxxxxx..6666 ESTABLISHED ... dragon# sockstat USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS ... backup bash 6886 0 tcp4 192.168.1.2:60487 XX.YY.175.21:6666 backup bash 6886 3 udp4 *:54977 *:* ...
Ouch! A connection to an IRC server is established. Smells like I am now part of a botnet…
More information from /var/log
Nothing interesting in cron*. Only my backup scripts are visible.
Nothing interesting in maillog*.
lastlog contains records of the last connections to the system. They can be consulted with the “last” command:
dragon# last ... backup ttyp0 XX.YY.180.196 Mon Dec 7 22:41 - 22:43 (00:02) backup ttyp0 XX.YY.180.196 Mon Dec 7 14:15 - 14:16 (00:00) backup ttyp0 XX.YY.180.196 Mon Dec 7 14:04 - 14:04 (00:00) ...
It only confirms what I saw in the authlog* files.
What has been installed
Here is what I can find in “/var/tmp/ /all”:
dragon# ls -l total 344 -rwxr-xr-x 1 backup wheel 85 Dec 11 00:00 1 -rwxr-xr-x 1 backup wheel 133 Nov 27 12:09 2 -rwxr-xr-x 1 backup wheel 133 Nov 27 12:09 3 -rwxr-xr-x 1 backup wheel 297784 Dec 31 2002 bash -rwxr-xr-x 1 backup wheel 942 Dec 31 2002 checkmech -rwxr-xr-x 1 backup wheel 4147 Dec 31 2002 genuser -rwxr-xr-x 1 backup wheel 1015 Dec 11 00:00 mech.levels -rwxr-xr-x 1 backup wheel 5 Dec 7 22:43 mech.pid -rw-r--r-- 1 backup wheel 1113 Dec 11 00:00 mech.session -rwxr-xr-x 1 backup wheel 1304 Dec 7 22:42 mech.set drwxr-xr-x 2 backup wheel 512 Dec 7 22:43 randfiles
bash is the program currently running in the compromised account.
Looking in the strings inside the executable, I can see that it is an EnergyMech IRC bot program.
The checkmech script can be installed as a cron job to check if the bot is running and restarting it if not. It does not seem to be installed in my cron jobs.
dragon# ./bash -v EnergyMech 2.8.5, December 30th, 2002 Compiled on Dec 30 2002 18:29:09 Features: LNK, TEL, PIP, DYN, NEW, ALS, WIN, SEF
Is this a standard or patched build or EnergyMech ? I checked with the emech-2.8.5-freebsd-static.tar.gz version distributed on energymech.net. It appears to contains the same binary as my bash executable.
So at least it’s not an “evil” version of the IRC bot. It can only do what the stock source code can do.
I can see some login/nickname/server information in the mech config files. But nothing I can do something with.
So I kill the process and delete the directory.
Next time I install Freebsd and create accounts, I will avoid using obvious passwords and I will limit the SSH incoming connection to the main account I need to access externally.