Catching Outbound spam from compromised scripts

Outgoing spam is likely to come from two sources:

1. Indirectly from a compromised web script in a clients account
2. Directly from a client

The starting point for both will be the exim mainlog:

/var/log/exim_mainlog (Linux)
/var/log/exim/mainlog (FreeBSD)

For the purpose of this document I am going to assume a Linux OS.

The most laborious way to track messages down is to trawl the exim mainlog and to look for anomalous behaviour. This is actually very difficult to do and you really need to narrow down exactly what you are looking for.

Tracking down spammers is a difficult affair, but can be made easier with some preparation of your servers environment. I would strongly advise that you add the following to the exim configuration to enable some extended logging that greatly improves the ease in tracking down on-server spammers:

In WHM > Exim Configuration Editor > Switch to Advanced Mode > in the first textbox add the following line and then Save:

log_selector = +arguments +subject

This tells exim to log the path on disk from where the email was executed and the subject of the email. You can then interrogate the exim mainlog more easily.

The best way to do this is to obtain the original email header from the spam originating from your server. This you should receive either from the person reporting the spam, or from remnants of a spam attack in the exim mail queue.

The part required in the email is the exim message id in the Received: header line within the email header of the spam.

As an example, take the following email header:

Return-path: <bob@barfoo.com>
Received: from [11.22.33.44] (helo=barfoo.com)
by foobar.com with esmtps (TLSv1:AES256-SHA:256)
(Exim 4.52)
id 1FZ8z3-0006M4-Do
for fred@foobar.com; Thu, 27 Apr 2006 17:04:49 +0100
Received: from forums by barfoo.com with local (Exim 4.43)
id 1FZ8zt-0005lz-E7
for fred@foobar.com; Thu, 27 Apr 2006 12:05:41 -0400
To: fred@foobar.com
Subject: Buy Me!
From: bob@barfoo.com

The Received: header lines are added to the email header, so the original Received: line that we're interested in is:

Received: from forums by barfoo.com with local (Exim 4.43)
id 1FZ8zt-0005lz-E7
for fred@foobar.com; Thu, 27 Apr 2006 12:05:41 -0400

And the id we want is 1FZ8zt-0005lz-E7

This is the unique identifier for this email that has originated from the server. With this, we can follow the exim transaction on the server to see how it was processed using:

grep 1FZ8zt-0005lz-E7 /var/log/exim_mainlog

(be aware that the exim_mainlog files may have been rotated so you may have to expand compressed archives and search them instead)

This transaction may look something like this:

2006-04-27 17:43:41 1FZ8zt-0005lz-E7 <= bob@barfoo.com U=nobody P=local S=4001 T="Buy Me!"
2006-04-27 17:43:50 cwd=/home/ClientX/public_html/phpBB/ 5 args: /usr/sbin/exim -Mc 1FZ8zt-0005lz-E7
2006-04-27 17:43:53 1FZ8zt-0005lz-E7 => fred@foobar.com R=lookuphost T=remote_smtp H=foobar.com [44.33.22.11] X=TLSv1:AES256-SHA:256
2006-04-27 17:43:53 1FZ8zt-0005lz-E7 Completed

In this example, we can see that the email originated from the nobody user locally on the server. This means that the likely spam was sent from a script on the server. The nobody user is used to run the Apache web server and is the default username and group that Apache will execute web scripts as. Two things can affect this:

1. suexec, if enabled, will run CGI scripts as the owner of the script file, typically the cPanel account name
2. phpsuexec, if enabled, will run PHP scripts in the same manner as CGI scripts

suexec is typically always enabled on web servers and phpsuexec may or may not be. If phpsuexec is not enabled, then in all likelihood, the script run under the nobody account will be a PHP script.

From the example above we can see that a script was run from with the /home/ClientX/public_html/phpBB/ directory on the server, which would suggest a compromised PHP script within that directory.

Here's another example of a spam originating from a client instead of a script. This can happen either with malicious intent, or if the clients PC has been compromised by a virus or worm:

2006-04-27 17:54:51 1FZ9lT-000707-O2 <= bob@barfoo.com H=someisp.com ([192.168.254.2]) [11.22.33.44] P=esmtpa A=fixed_plain:bob@barfoo.com S=715 id=ABCDEFG T="Buy Me!"
2006-04-27 17:54:51 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1FZ9lT-000707-O2
2006-04-27 17:54:51 1FZ9lT-000707-O2 => fred@foobar.com R=boxtraper_autowhitelist T=boxtrapper_autowhitelist
2006-04-27 17:54:52 1FZ9lT-000707-O2 => fred@foobar.com R=lookuphost T=remote_smtp H=foobar.com [44.33.22.11] X=TLSv1:AES256-SHA:256
2006-04-27 17:54:52 1FZ9lT-000707-O2 Completed

In this example, the key part is:

A=fixed_plain:bob@barfoo.com

This shows that the email was authenticated for relaying using SMTP AUTH (i.e. fixed_plain) and the username bob@barfoo.com from that clients PC.

As you can see, there is a great depth to the amount of work needed to track down spammers on a server, plus there's the additional work of closing holes in insecure scripts if they are the cause. Some instances can be much more complex and require trawling through the Apache logs for domains in /usr/local/apache/domlogs/* which is not a trivial matter.

The best security from such exploitation is to keep your server secure and to be aware of who and what you allow on your server.