Elastic Sheep

Tonight I just saw that my Freebsd server has been compromised since 3 days ago.

I did a fresh install of Freebsd 7.2 at the end of November and then setup an account to take care of this blog backup.

Today I tried to take a look on the backup data and am unable to log on the account. Maybe I typed a wrong password… mmmh, in fact I used an obvious one to not forget it (same as login…). And the account was accessible by SSH.

After a few tries the fact strikes me: if I can not login on the account, it is because it has been changed by somebody else…

First step: I login as root and change all my account passwords. Now I can go on to look for what happened.

Connection history

Connection attempts are logged in /var/log/auth.log* files.

In auth.log.1.bz2, bingo: I find the first successful foreign connection. Obviously it also contains a LOT of other attempts from various attackers armed with dictionaries of common login/password.

Dec  7 01:53:03 dragon sshd[3904]: Accepted keyboard-interactive/pam for backup from XX.YY.28.27 port 44670 ssh2
Dec  7 01:53:03 dragon sshd[3907]: Received disconnect from XX.YY.28.27: 11: Bye Bye

The account has been detected by a China IP. It is accessed for real a few hours later by a Romanian IP:

Dec  7 14:04:52 dragon sshd[5749]: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT!
Dec  7 14:04:52 dragon sshd[5749]: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 1215 ssh2
Dec  7 14:15:25 dragon sshd[5782]: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT!
Dec  7 14:15:27 dragon sshd[5782]: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 1232 ssh2
Dec  7 22:41:42 dragon sshd[6866]: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT!
Dec  7 22:41:43 dragon sshd[6866]: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 4241 ssh2
Dec  7 22:43:03 dragon sshd[6877]: reverse mapping checking getaddrinfo for XX-YY-180-196.rdsnet.ro [XX.YY.180.196] failed - POSSIBLE BREAK-IN ATTEMPT!
Dec  7 22:43:03 dragon sshd[6877]: Accepted keyboard-interactive/pam for backup from XX.YY.180.196 port 4243 ssh2
Dec  7 22:43:03 dragon sshd[6880]: subsystem request for sftp

10 minutes then 8 hours then another 2 minutes, it looks like somebody has taken the time to install something…

Bash history

Footprints of shell activity can be found in the .bash_history file of the account:

First attempt:

w
ps x
uname -a
exit

First the pirate looks if there is anybody there with w (display who is logged in and what they are doing), then he checks the running processes, then at last he checks the OS version. For me:

dragon# uname -a
FreeBSD dragon.local 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May  1 08:49:13 UTC 2009
root@walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

Second attempt, same recipe but this time he request a password change:

w
ps x
uname -a
passwd
exit

Third connection, things are added:

w
ps x
clear
cd /var/tmp
mkdir "   "
cd "   "
uname -a
cd all
chmod +x *
./bash
exit

The attacker created an “invisible” directory named with 3 spaces, cd’ed in a “all” directory and then executed a “bash” program. Where this program comes from I don’t know. How has it been copied here ?

I can check that the program is in fact still running:

dragon# ps -x
...
backup   6886  0.0  0.1   544   384  ??  Ss   Mon10PM   0:01.67 ./bash
...

What is this program doing ? I can check if any socket connection is opened with netstat and sockstat:

dragon# netstat
...
tcp4       0      0 192.168.1.2.60487      irc2.xxxxxxxxx..6666  ESTABLISHED
...
dragon# sockstat
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
...
backup   bash       6886  0  tcp4   192.168.1.2:60487     XX.YY.175.21:6666
backup   bash       6886  3  udp4   *:54977               *:*
...

Ouch! A connection to an IRC server is established. Smells like I am now part of a botnet…

More information from /var/log

Nothing interesting in cron*. Only my backup scripts are visible.
Nothing interesting in maillog*.

lastlog contains records of the last connections to the system. They can be consulted with the “last” command:

dragon# last
...
backup           ttyp0    XX.YY.180.196   Mon Dec  7 22:41 - 22:43  (00:02)
backup           ttyp0    XX.YY.180.196   Mon Dec  7 14:15 - 14:16  (00:00)
backup           ttyp0    XX.YY.180.196   Mon Dec  7 14:04 - 14:04  (00:00)
...

It only confirms what I saw in the authlog* files.

What has been installed

Here is what I can find in “/var/tmp/ /all”:

dragon# ls -l
total 344
-rwxr-xr-x  1 backup  wheel      85 Dec 11 00:00 1
-rwxr-xr-x  1 backup  wheel     133 Nov 27 12:09 2
-rwxr-xr-x  1 backup  wheel     133 Nov 27 12:09 3
-rwxr-xr-x  1 backup  wheel  297784 Dec 31  2002 bash
-rwxr-xr-x  1 backup  wheel     942 Dec 31  2002 checkmech
-rwxr-xr-x  1 backup  wheel    4147 Dec 31  2002 genuser
-rwxr-xr-x  1 backup  wheel    1015 Dec 11 00:00 mech.levels
-rwxr-xr-x  1 backup  wheel       5 Dec  7 22:43 mech.pid
-rw-r--r--  1 backup  wheel    1113 Dec 11 00:00 mech.session
-rwxr-xr-x  1 backup  wheel    1304 Dec  7 22:42 mech.set
drwxr-xr-x  2 backup  wheel     512 Dec  7 22:43 randfiles

bash is the program currently running in the compromised account.

Looking in the strings inside the executable, I can see that it is an EnergyMech IRC bot program.

The checkmech script can be installed as a cron job to check if the bot is running and restarting it if not. It does not seem to be installed in my cron jobs.

dragon# ./bash -v
EnergyMech 2.8.5, December 30th, 2002
Compiled on Dec 30 2002 18:29:09
Features: LNK, TEL, PIP, DYN, NEW, ALS, WIN, SEF

Is this a standard or patched build or EnergyMech ? I checked with the emech-2.8.5-freebsd-static.tar.gz version distributed on energymech.net. It appears to contains the same binary as my bash executable.

So at least it’s not an “evil” version of the IRC bot. It can only do what the stock source code can do.

I can see some login/nickname/server information in the mech config files. But nothing I can do something with.

So I kill the process and delete the directory.

Lesson learned

Next time I install Freebsd and create accounts, I will avoid using obvious passwords and I will limit the SSH incoming connection to the main account I need to access externally.