I wanted to share my methodology for analyzing access logs coming from a shared hosting environment.
The steps break down like this:
- Gather the log files
- Analyze the log files
Once you get into it, things can get a bit annoying. When things are annoying, you won’t do them. You need to pay attention to security!
Most hosting companies offer you some sort of control panel where you an download your raw webserver logs. In my case it is the ever popular cPanel software.
Navigate to the logs section and click the ‘Raw Access Log’ icon. This will display a list of all your add-on domains. I have about 20 of them – Yikes! Who wants to click each of these links and download them? Not us. Enter Download Them All, the amazing Firefox plugin. Install this plugin and set the download directory to where you would like to store these logs. I used ‘my home’/Documents/hacked/logs/’. For the type of file to download, select archives and start the downloads. They’ll zip along depending on their size and will end up in your log folder. Select them all in your file browser and unzip them. Sort the directory by file type and delete all of the archive files.
The next step is to combine these files somehow so we can look at one file. My first attempt at this I used the linux command cat – which means concatenate.
cat * > bigole.txt
There is a problem with this approach though – all entries are lumped together and it can be hard to tell what domain the logs come from. We’ll fix this with another approach later.
This put all of the files in one file that I could search for some certain files that had been placed on my site. After I found the files I went back in time a bit and discovered how the files had been placed.
Bingo! The file was named setting.php and was a uploaded through a CMS software. This .php file was a hacker ‘shell’ and let some little shit browse around and hide phishing site urls in my domains. So I needed to track these files down and get rid of them.
Find file ‘setting.php’ in public_html
find public_html -name 'setting.php'
Find files created in the last 11 days
find . -mtime -11 -ls
So all of this worked out O.K. , but a few minutes later I remember something we’d been using at my day job – splunk. Splunk will index your log files and make them searchable through a web interface. Works on Mac, Windows and Linux.
So I followed the same procedure but stopped at the cat command. I installed splunk and configured the inputs to look at my log directory. Splunk sucked all the files into it’s internal database and showed a timeline with a simple search box above it.
I typed in ‘setting.php’ and searched. I see a list of entries that matched. I then clicked on the IP address that had been accessing setting.php. It then added that IP to my search terms. Deleting setting.php from the search bar let me see all activity associated with that IP across all domains.
At the bottom of each log entry you can see what file it was pulled from. The file names identify the domain, so I could tie the entry back to the domain – solving the one problem I ran into.
Using these rudimentary functions of splunk I was able to get a much better idea of the activity happening on my sites – ALL MY SITES! You can create reports using splunk, letting you dig deeper into your sites usage than traditional web trending software. You can also save searches, schedule searches and have actions take place if results are returned. This thing is great!
My next step is to figure out how to get these logs shipped to me automatically or install splunk on my server.Comment?