Alan Ivey

Systems Administrator and person of the internets.

Posts

January 25, 10:13 AM

I had previously written a post on one method of upgrading PHP from 5.1 to 5.2 on CentOS and Red Hat servers by creating new RPMs. Since then, I have found a much better way to create PHP 5.2.17 (or newer) RPMs to easily upgrade (and later remove, if you want) the older version available by default. I'll presume you have no prior experience building PHP RPMs.

We start by adding the Extra Packages for Enterprise Linux (EPEL) repository to our server. EPEL is safe to add as an additional repository and leave enabled because it provides complementary packages only and will not conflict with the CentOS or Red Hat repositories. Since their installation method may change, I won't reproduce it here; instead head over to the installation portion of the FAQ and it's just one line to get started with EPEL.

With EPEL enabled, we will need just one new package to build RPMs: Mock. Mock will create a chroot, download what's necessary to build your RPM, build, and cleanup afterwards. Mock only requires a couple of python packages as dependencies; it will not require a compiler or anything like that since mock will grab them as needed during operation. I will presume you are not running as root so I will use sudo as necessary. To install mock:

$ sudo yum install mock

That's it. You can run all mock operations with the 'sudo' command to run as root, but your packages will be built inside the chroot as a non-root user. Or, if you prefer to not use sudo when running mock, add your user to the mock group (where username is your user):

$ sudo /usr/sbin/usermod -G mock username

Using the source code from php.net and creating a RPM spec file from scratch would be a lot of work, so we'll grab a source RPM for PHP 5.2. I've found that Remi has the most up-to-date RPM spec with the latest patches and robust install scripts. Browse Remi's SRPMS and download the latest PHP 5.2 file. As of this writing, it's php-5.2.17-1.remi.src.rpm. On your system, download it with wget in your home directory:

$ cd ~
$ wget http://rpms.famillecollet.com/SRPMS/php-5.2.17-1.remi.src.rpm

Now, we could just run mock and rebuild this source RPM and hopefully install the results, but it would fail due to not having sqlite2-devel installed. Remi provides sqlite2 and sqlite2-devel, so you could add Remi's yum repository to your mock settings, but then it would grab Remi's other updated packages like MySQL 5.5 and others during the PHP build. My goal here is to only build PHP against Base CentOS and EPEL packages. So, we have to edit the spec file and rebuild the source RPM before passing it off to mock. Don't worry; while there are a few steps, it's not very difficult if you follow closely.

To unpack the source RPM we just downloaded into our home directory, we'll need to set up the .rpmmacros file in our home directory. Why? Without it, it will attempt to extract to /usr/src/redhat, and only root can use that directory, and you should never build any packages as a privileged user. We'll create the macros file and the directories it needs from your home directory:

$ cd ~
$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros
$ mkdir -p ~/rpmbuild/{SOURCES,SPECS,SRPMS}

Unpack the source RPM. This will extract the PHP 5.2.17 source and any patches and configuration files into ~/rpmbuild/SOURCES, and the spec file will be in ~/rpmbuild/SPECS:

$ rpm -ivh php-5.2.17-1.remi.src.rpm

Now we need to make a couple of changes to the spec file. We'll change to the SPECS directory and make a copy of Remi's spec file so we can keep it for reference:

$ cd ~/rpmbuild/SPECS
$ cp -a php52.spec php.spec

Open up php.spec in your favorite text editor and let's have the PHP configure script use the bundled SQLite2 libraries instead of searching for ones on your system:

  1. Find the line "BuildRequires: sqlite2-devel >= 2.8.0" and delete it
  2. Find "--with-sqlite=shared,%{_prefix} \" and replace it with "--with-sqlite=shared \"
  3. Optionally: Add your own lines to the %changelog describing that you removed the sqlite2-devel build requirement
  4. Optionally: Remove the first 4 lines after "%pre common" to remove Remi's note about their forums

It took 2 small tweaks to remove the sqlite2-devel RPM requirement not found in our standard repositories. See php.spec.diff to see only the changes we made; or php.spec for the complete file if you do not want to edit it yourself.

The hard part is now over! It's easy from here on out. First step is to build the newly-edited spec into your own source RPM. But first, if you don't have "rpmbuild" available, it's a quick install via yum:

$ sudo yum install rpm-build

It's possible your system may have already had the yum package rpm-build installed. We'll build our source RPM with "-bs" to build the source, and "--nodeps" to prevent our system from checking all of the "Requires" in the spec since we're just building the source:

$ rpmbuild --nodeps -bs php.spec
Wrote: /home/username/rpmbuild/SRPMS/php-5.2.17-1.src.rpm

rpmbuild will show you where it wrote your source RPM. All we have to do is provide it to mock; "-v" will give us verbose output and "-r epel-5-x86_64" will use the mock profile at "/etc/mock/epel-5-x86_64.cfg". If you're on a 32-bit system, change "epel-5-x86_64" to "epel-5-i386":

$ cd ~/rpmbuild/SRPMS
$ mock -v -r epel-5-x86_64 php-5.2.17-1.src.rpm

The output shows you that mock is creating a chroot, downloading the bare essentials needed for a chroot and compiling tools, ccache to speed up compilation, and the packages in all "Requires" lines in the spec file. When mock is finished with the build after several minutes, you'll see output similar to:

INFO: Done(php-5.2.17-1.src.rpm) Config(epel-5-x86_64) 10 minutes 20 seconds
INFO: Results and/or logs in: /var/lib/mock/epel-5-x86_64/result

The /var/lib/mock/epel-5-x86_64/results/ folder will contain all of your PHP RPMs, including another source RPM. You can now either install these PHP packages by hand, or you can dump them into your own Yum repository and install with yum. 

TL;DR

$ sudo yum install mock
$ sudo /usr/sbin/usermod -G mock username
$ cd ~
$ wget http://rpms.famillecollet.com/SRPMS/php-5.2.17-1.remi.src.rpm
$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros
$ mkdir -p ~/rpmbuild/{SOURCES,SPECS,SRPMS}
$ rpm -ivh php-5.2.17-1.remi.src.rpm
$ cd ~/rpmbuild/SPECS
$ cp -a php52.spec php.spec
$ sed -i "/BuildRequires\: sqlite2-devel/d" php.spec
$ sed -i "s/--with-sqlite=shared,%{_prefix}/--with-sqlite=shared/" php.spec
$ sudo yum install rpm-build
$ rpmbuild --nodeps -bs php.spec
$ cd ~/rpmbuild/SRPMS
$ mock -v -r epel-5-x86_64 php-5.2.17-1.src.rpm

Resulting RPMs will be in /var/lib/epel-5-x86_64/results. If on a 32-bit system, replace epel-5-x86_64 with epel-5-i386.

Permalink | Leave a comment  »

January 05, 08:31 PM

In addition to hosting Drupal sites, we also host a number of WordPress sites. Similar to checking all of our Drupal core versions, we needed an easy way to quickly see what versions we are running on all of our WordPress sites. Kudos to Ethan for getting this one kicked off; here is a bash script to check your definable $WEBHOME (where you deposit all of your WordPress webroots) and scan for WordPress versions.

Note: this requires bash, grep, wc, sed, awk, and tree. If your system doesn't have tree, it's usually pretty easy to get from EPEL/Homebrew/apt-get/[mac]ports/etc.

#!/bin/bash
 
# Where to start scanning
WEBHOME=/var/www/vhosts
 
# Workaround to fix awk counting below
WEBHOMECOUNT=$(($(echo "${WEBHOME}"|grep -o "/"|wc -l| sed s/\ //g)+2))
 
# Other projects could use 'version.php' so we include 
# 'wp-includes/' in our search to limit it to WordPress
for i in $(tree -L 5 -if ${WEBHOME} | grep 'wp-includes/version.php'); do
  SITE=$(echo $i|awk -v count="$WEBHOMECOUNT" -F/ '{for(j=count;j<=NF-2;j++) \
        printf $j"/"}' | sed 's/.$//g')
  VERSION=$(grep "wp_version = " $i|awk -F\' '{print $2}')
  echo $SITE - $VERSION
done

As a result, you'll see output like this:

$ ./wordpress-version-check.sh
speedyupdates.com - 3.0.4
littlebehind.org - 3.0.3
scaredtoupgrade.com/blog - 2.8.2
kickinitoldschool.org/web/wordpress - 2.0.2

If you think there's a better way, or have any questions, let me know in the comments.

Permalink | Leave a comment  »

January 05, 07:25 PM

At EchoDitto, we host a lot of Drupal sites. Just about all of our web servers contain more than one completely separate Drupal cores since we develop most of our websites independently. We also keep an eye out on Drupal Security and when to apply core updates. Since we're not using Aegir, nor do we run every site off of a single multi-site core install, we need a quick and easy way to see what versions we're running.

Here's a shell script we recently developed to report the Drupal Core versions for all of our sites under a single parent folder. It works on Drupal 4 through 7. It's not been tested outside of our platform, or non-RHEL systems, but it should be universal as long as you have bash, find, awk, and grep. I decided not to use Drush because it would involve bootstrapping every site and invoking PHP; this is just more lightweight. Drush also would not work for Drupal 4 sites, which we surprisingly still have a couple kicking around still.

#!/bin/bash
 
# Where to start looking
WEBHOME=/var/www/vhosts
 
# Workaround to fix awk counting below, but it works
WEBHOMECOUNT=$(($(echo "${WEBHOME}"|grep -o "/"|wc -l| sed s/\ //g)+2))
 
# Change maxdepth from 5 to some other value if your webroot is not being detected
for i in $(find $WEBHOME -maxdepth 5 -type f -name system.module); do 
  # Grabs only the version number from the file as described 
  # at http://drupal.org/handbook/version-info
  VERSION=$(grep "VERSION" ${i}|awk -F\' '{print $4}')
 
  # Drupal 4 has the system.module file in a lower directory than 5+
  if [[ $VERSION == 4* ]]; then
    SITE=$(echo $i|awk -v count="$WEBHOMECOUNT" -F/ '{for(j=count;j<=NF-2;j++) \
           printf $j"/"}' | sed 's/.$//g')
  else
    SITE=$(echo $i|awk -v count="$WEBHOMECOUNT" -F/ '{for(j=count;j<=NF-3;j++) \
           printf $j"/"}' | sed 's/.$//g')
    if [ -z $VERSION ]; then
      # Drupal 7 does not keep the version number on system.module, despite the 
      # drupal.org page. Grab from changelog (disabled b/c it includes extra info)
      #VERSION="$(sed '3q;d' ${WEBHOME}/${SITE}/CHANGELOG.txt)"
      # All versions of Drupal have the version of the module in system.info, 
      # so we grab it here for D7. The only reason I'm not using this for all 
      # versions is because of the aforementioned drupal.org page
      VERSION=$(grep "version = \"" ${WEBHOME}/${SITE}/modules/system/system.info \
                | awk -F\" '{print $2}')
    fi
  fi
  echo $SITE - $VERSION
done

As a result, you'll see the directory tree after $WEBHOME where the root of the site is and the version number. Below is an example:

$ ./drupal-version-check.sh
mysite.org/www - 5.22
thissite.net - 4.7.11
subfolder/drupalftw.com - 6.12
blah.org/trunk - 7.0

If you have suggestions, please send them over. This script is still young and I'd love to see if anyone has any better ideas.

Permalink | Leave a comment  »

December 07, 02:34 PM

When dealing with a load spike or unexplained poor performance, once of the first things to do is to check the Apache web server logs. However, when dealing with multiple web servers behind a load balancer, it can be cumbersome to check through multiple logs on different machines.

Luckily, if you have shared storage, it’s pretty simple to write to a single log file. Apache allows for log file locations to go to a pipe, which means you can redirect the output anywhere else. This was initially done to allow Apache to release open file descriptors since it can begin to create problems with more than 300 domains. But you don’t need to have 300 websites to see the benefits of using a pipe for logging.

In your VirtualHost declaration, change your logging options from:

CustomLog logs/domain.com-access_log combined

to:

CustomLog “|/bin/cat » /var/www/vhosts/_logs/domain.com-access_log” combined

In this example, /var/www/vhosts is the shared SAN storage device, and by using “cat »” we append Apache output in real-time to the log file. As multiple machines do this simultaneously, you begin to have a single log file from multiple machines. You can do the same thing with your ErrorLog:

ErrorLog  ”|/bin/cat » /var/www/vhosts/_logs/domain.com-error_log”

There is a slight problem with this approach: it doesn’t give us enough information. If one of the 4 servers behind the load balancer was the source of a problem, it would be helpful to determine which server each log entry is coming from. 

For CustomLog, we can create a new LogType that adds the server’s IP address at the end of the line, and the LogType name will be “combinedIP”:

LogFormat “%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %A” combinedIP

This format is identical to “combined” and just adds %A at the end. Then, simply change the end of the CustomLog line:

CustomLog “|/bin/cat » /var/www/vhosts/_logs/domain.com-access_log” combinedIP

Here’s an example of what your log file will look like, showing entries from two servers:

1.2.3.4 - - [07/Dec/2010:11:22:08 -0500] “GET /blog/feed HTTP/1.1” 304 - “-” “-” 192.168.1.6 5.6.7.8 - - [07/Dec/2010:11:22:08 -0500] “GET /blog/feed HTTP/1.1” 304 - “-” “-” 192.168.1.8

These log entries are the same as the typical “combined” LogType but with the addition of the IP address at the end.

But what about the ErrorLog? There is no way to define a LogType for the ErrorLog, and it would be helpful to know which server originated error, say, if MaxClients was reached, or there is a misconfiguration in a configuration file. Since we’re piping the ErrorLog output to /bin/cat, we can instead pipe it to a script that simply adds the server hostname on the end (I chose not to use an IP address because 1) some servers have multiple IP addresses, and 2) I cannot insert the VirtualHost’s IP address automatically. Better to know the server and then determine the IP address is necessary). We’ll start by creating a simple script at /usr/local/bin/httpd_errors (I’m not using bash since I was not able to get it to write to the log file in real time, it would only write once Apache was reloaded. I did not spend much time to determine why, and I knew it would work in PHP):

#!/usr/bin/php -q<?php$stdin = fopen (‘php://stdin’, ‘r’);ob_implicit_flush (true); // Use unbuffered outputwhile ($line = fgets ($stdin)){    print chop($line).” - “.system(‘hostname -f’).”\n”;} ?>

Make the script executable with “chmod +x /usr/local/bin/httpd_errors”, and then change your ErrorLog in your Apache configuration:

ErrorLog  ”|/usr/local/bin/httpd_errors » /var/www/vhosts/_logs/domain.com-error_log”

And your error logs will now reveal which server the error originated from:

[Tue Dec 07 11:20:51 2010] [error] [client 1.2.3.4] File does not exist: /var/www/vhosts/domain.com/favicon.ico - wsrv1

[Tue Dec 07 11:24:27 2010] [error] [client 5.6.7.8] File does not exist: /var/www/vhosts/domain.com/favicon.ico - wsrv2

There you have it, combined Apache logs for multiple web servers. If you have any suggestions, questions, tips, etc, leave a comment!

Permalink | Leave a comment  »

Profile

Systems Administrator at EchoDitto
Internet | Lancaster, Pennsylvania Area, US

Summary

Unix Systems Administrator with dynamic experience with federal contracts, non-profits, and small businesses, focusing on security and standards. Open source software used as much as possible, though always the right tool for the job.
Specialties: Unix systems administration with security focus

Experience

  • May 2008 - Present
    Systems Administrator / EchoDitto
    Linux (LAMP) and Leopard server administration, Leopard desktop support
  • May 2007 - Present
    Creative Arts Resident / Momentum Christian Church
    Web site team leader and developer for volunteer-based team for momentumchurch.com In-office Network and Computer Administrator for systems based on OS X 10.4 and 10.5 and Windows XP and Vista Video production with Final Cut Studio 2 DVD production Graphics design
  • Feb 2005 - Present
    Systems Administrator / NASA
    Systems Analyst & Unix Systems Administrator (Solaris, Red Hat-based Linux, Ubuntu, OS X)

Education

  • 2006 - 2010
    University of Mary Washington
    Bachelor's of Professional Studies in Network Security

Additional Information

Websites:
abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz