I wanted to share my methodology for analyzing access logs coming from a shared hosting environment.
The steps break down like this:
- Gather the log files
- Analyze the log files
Once you get into it, things can get a bit annoying. When things are annoying, you won’t do them. You need to pay attention to security!
Most hosting companies offer you some sort of control panel where you an download your raw webserver logs. In my case it is the ever popular cPanel software.
Navigate to the logs section and click the ‘Raw Access Log’ icon. This will display a list of all your add-on domains. I have about 20 of them – Yikes! Who wants to click each of these links and download them? Not us. Enter Download Them All, the amazing Firefox plugin. Install this plugin and set the download directory to where you would like to store these logs. I used ‘my home’/Documents/hacked/logs/’. For the type of file to download, select archives and start the downloads. They’ll zip along depending on their size and will end up in your log folder. Select them all in your file browser and unzip them. Sort the directory by file type and delete all of the archive files.
The next step is to combine these files somehow so we can look at one file. My first attempt at this I used the linux command cat – which means concatenate.
cat * > bigole.txt
There is a problem with this approach though – all entries are lumped together and it can be hard to tell what domain the logs come from. We’ll fix this with another approach later.
This put all of the files in one file that I could search for some certain files that had been placed on my site. After I found the files I went back in time a bit and discovered how the files had been placed.
Bingo! The file was named setting.php and was a uploaded through a CMS software. This .php file was a hacker ‘shell’ and let some little shit browse around and hide phishing site urls in my domains. So I needed to track these files down and get rid of them.
Find file ‘setting.php’ in public_html
find public_html -name 'setting.php'
Find files created in the last 11 days
find . -mtime -11 -ls
So all of this worked out O.K. , but a few minutes later I remember something we’d been using at my day job – splunk. Splunk will index your log files and make them searchable through a web interface. Works on Mac, Windows and Linux.
So I followed the same procedure but stopped at the cat command. I installed splunk and configured the inputs to look at my log directory. Splunk sucked all the files into it’s internal database and showed a timeline with a simple search box above it.
I typed in ‘setting.php’ and searched. I see a list of entries that matched. I then clicked on the IP address that had been accessing setting.php. It then added that IP to my search terms. Deleting setting.php from the search bar let me see all activity associated with that IP across all domains.
At the bottom of each log entry you can see what file it was pulled from. The file names identify the domain, so I could tie the entry back to the domain – solving the one problem I ran into.
Using these rudimentary functions of splunk I was able to get a much better idea of the activity happening on my sites – ALL MY SITES! You can create reports using splunk, letting you dig deeper into your sites usage than traditional web trending software. You can also save searches, schedule searches and have actions take place if results are returned. This thing is great!
My next step is to figure out how to get these logs shipped to me automatically or install splunk on my server.
Ubercart is a popular module for Drupal eCommerce. I setup a system recently. It works well out of the box. The majority of the work you will put in will be payments and shipping quotes. I’ll dig into how to get going here.
- Preparation: Sign up for (or have your customer sign up for) google analytics, UPS merchant and/or USPS merchant accounts so you can calculate shipping, otherwise you can just do a flat rate(base rate + per item rate).
Here are links to info on signing up:
Get access to SSH into your hosting provider if you don’t already. Optionally, ask your hosting provider to enable cron and wget for you. Cron will let you schedule a PHP page to be run at your desired times. wget will let you download files directly to your server and then unzip them. Sometimes you can schedule cron from your hosting control panel.
Go to your hosting control panel and create a mySQL database and a user for that database. You can usually do this in your ‘control panel’ in your hosting software.
The user must have these permissions:
- CREATE TEMPORARY TABLES
- LOCK TABLES
Write down your user name, password and database name. Be wary that some hosting services will prepend your user and database with different characters – perhaps your username. Also your host name for your database may not be localhost, which is the default for drupal in the ‘advanced’ part of the setup page. I know that GoDaddy will give you a numbered server to connect to. This info is presented to you at the end of the web control panels I have used that have a mySQL database creation wizard.
- Sign up for a paypal developer account – https://developer.paypal.com/ – if you are going to use paypal (which I will be explaining here). This will let you create test transactions. Create a test seller account and a test buyer account. Don’t forget to put money in the buyer account. Take note of the email addresses and passwords, you’ll need them later. An important note: copying the email address from the paypal page may result in a space in the address be sure to take that out. That caused some grief for me.
- Download the ubercart deluxe package (http://install.ubercart.org/files/install.ubercart.org/uberinstaller/ube… ) which includes all the core Drupal modules and Ubercart add-on modules.
If you have wget installed in your ssh shell you can run these commands(otherwise use ftp):
// these are comments
// change directory to your home directory
// make a new directory called drupal
// change to that directory
// download ubercart to that directory
wget <a href="http://install.ubercart.org/files/install.ubercart.org/uberinstaller/ubercart_deluxe.tar.gz">http:
// unzip the file
tar xzf ubercart_deluxe.tar.gz
// view your unzipped file
// copy deluxe contents to your public html directory
cp -r ./ubercart_deluxe/* ~/public_html/
// If you're new to the command line...
// . (dot) stands for your current directory. ~ represents your home directory.
- Navigate to your home domain and install drupal. Get your database name and logon credentials and follow the instructions below. Getting started with Drupal 5 from: http://drupal.org/handbooks. The Drupal community is strong, use it!
- Go to Admin › Site Building › Modules. Expand ‘Ubercart – fulfillment’ and ‘Ubercart – payment’. You should enable one or more of: flat-rate, UPS or USPS shipping. In payment, enable Paypal if you want to use that. Be sure to save those options.
- Go to Home › Administer › Store administration › Configuration › Shipping Quote Settings › Quote Methods › General Settings and enable your preferred shipping methods. There will be links to configure the fulfillment modules you activated in the previous step. As mentioned before, both UPS and USPS require registration for UPS ‘WebTools’ and USPS XML services. You can also enable error display for admins in the ‘edit’ tab.
Both can be configured in ubercart to direct to a testing server. USPS requires that you contact them to gain access to their production site. Their test site did not work for me – I have yet to get the production working, but haven’t tried, UPS has been the priority.
- Go to Home › Administer › Store administration › Configuration › Payment Settings › Payment Methods › PayPal Website Payments Standard settings. Check the ‘enable’ box for PayPal payments standard. Enter the email address for your PayPal test business account. PayPal server should be set to ‘sandbox’ for testing. Change this back before going live and be sure to change the email address to your actual PayPal email.
- Create products and test out your shipping!
- Ask for help here or at http://www.ubercart.org if you need it.
Putting audio on your site seems like it should be easy. It is, once you know what to do.
- Download the audio module from drupal.org. At the moment, if you want to have multiple files in one player you need to get the 5.x-2.x-dev download – http://ftp.drupal.org/files/projects/audio-5.x-2.x-dev.tar.gz. If not, you can get the 5.x release of your choice.
- Unzip the file and copy the new folder to your sites/all/modules folder on your webserver
- Download the getid3 library from sourceforge and put in sites/all/modules/audio/getid3
- Enable the module in your Drupal Admin
- Configure the audio import directory
- Configure the id3 directory
- Upload some properly tagged mp3s to the audio import directory – do a few that have album name in common.
- Import the audio files
- Create a new view with type of audio:xspf and a filter on audio:album if desired. Select the Page or Block and Menu options based on your preference.
- Create a new view with type of full nodes. Select and complete the Page or Block and Menu sections. Filter on an audio related criteria and Sort as you like.
- You will now have two pages: one with a player containing the songs belonging to the album filter you created, and one with a list of single entries with their own players. You could change the latter to only return one node, change the sorting to descending by node:created date and place in a block – then you’d have the most recent audio entry for your site, placed in a block perhaps.