alanthing

Linux and other things

Certbot on CentOS 6

CentOS 6 is getting updates through November 30, 2020, but it’s getting more and more difficult to find newer packages for the operating system. If you want to use Certbot for obtaining and renewing Let’s Encrypt TLS certificates, you can use certbot-auto and let it handle the work for you, but I wanted to try only the PyPi package via pip.

Note: I’m using Docker to test these commands via: docker run --rm -it centos:6 bash

1
2
3
4
5
6
7
# No Python wheels available, so we need gcc and some devel packages
$ yum install gcc libffi-devel openssl-devel python-devel python-pip

# Let's try and install certbot
$ pip install --upgrade certbot
...snip...
Successfully installed ConfigArgParse-0.12.0 PyOpenSSL-17.3.0 acme-0.18.2 certbot-0.18.2 cffi-1.11.0 configobj-5.0.6 cryptography-2.0.3 funcsigs-1.0.2 mock-2.0.0 pbr-3.1.1 pyrfc3339-1.0 requests-2.18.4 zope.component-4.4.0 zope.event-4.3.0

Seemed to work, right? Nope 😠:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ certbot --help
Traceback (most recent call last):
  File "/usr/bin/certbot", line 7, in <module>
    from certbot.main import main
  File "/usr/lib/python2.6/site-packages/certbot/main.py", line 7, in <module>
    import zope.component
  File "/usr/lib/python2.6/site-packages/zope/component/__init__.py", line 28, in <module>
    from zope.component.globalregistry import getGlobalSiteManager
  File "/usr/lib/python2.6/site-packages/zope/component/globalregistry.py", line 18, in <module>
    from zope.interface.registry import Components
  File "/usr/lib64/python2.6/site-packages/zope/interface/registry.py", line 167
    filtered_state = {k: v for k, v in reduction[2].items()
                             ^
SyntaxError: invalid syntax

Of course this doesn’t work, it’s Python 2.6!

Software Collections, to the rescue! Let’s use the Python 2.7 SCL and try again:

1
2
3
4
5
6
7
8
9
10
# Install the SCL Yum repo files
$ yum install centos-release-scl

# Grab Python 2.7, pip, and related depedencies
$ yum install python27-python-pip

# Install into the SCL root with pip. Wheels are available, so no compiler or devel packages necessary!
$ scl enable python27 'pip install certbot'
...snip...
Successfully installed ConfigArgParse-0.12.0 PyOpenSSL-17.3.0 acme-0.18.2 asn1crypto-0.22.0 certbot-0.18.2 certifi-2017.7.27.1 cffi-1.11.0 chardet-3.0.4 configobj-5.0.6 cryptography-2.0.3 enum34-1.1.6 funcsigs-1.0.2 future-0.16.0 idna-2.6 ipaddress-1.0.18 mock-2.0.0 parsedatetime-2.4 pbr-3.1.1 pycparser-2.18 pyrfc3339-1.0 pytz-2017.2 requests-2.18.4 setuptools-36.5.0 six-1.11.0 urllib3-1.22 zope.component-4.4.0 zope.event-4.3.0 zope.interface-4.4.2

Success! Right?

1
2
$ certbot --help
bash: certbot: command not found

Hrm, what happened?

1
2
3
$ find /opt/rh/python27/ -name certbot -executable
/opt/rh/python27/root/usr/bin/certbot
/opt/rh/python27/root/usr/lib/python2.7/site-packages/certbot

Right, it’s in the SCL. You have some options. You could enable the SCL in userspace through commands like source scl_source enable python27 or scl enable python27 bash. Or, because I want to use certbot interactively and in crons without thinking about the SCL, we can use a “shim”:

1
2
3
4
5
6
$ cat > /usr/local/bin/certbot <<'EOF'
#!/bin/bash
source scl_source enable python27
certbot "$@"
EOF
$ chmod -c 755 /usr/local/bin/certbot

Assuming /usr/local/bin/ is in your $PATH, we can test this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ certbot register --agree-tos -m 'le@example.com' -n
Saving debug log to /var/log/letsencrypt/letsencrypt.log

IMPORTANT NOTES:
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

$ find /etc/letsencrypt \! -type d
/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8/meta.json
/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/a1bx2c3d4e5f6a7b8a1b2c3d4e5f6a7b8/regr.json
/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8/private_key.json

Now you have certbot on CentOS 6 from PyPi and without certbot-auto (Nothing wrong with certbot-auto, I just prefer to have more control over the installation process). Let me know how it goes for you!

Python 3 and uWSGI on CentOS 7

Wherever I can, I avoid compiling applications from source or packaging up my own versions of things that exist in popular repositories. I recently needed to run a Python 3 application via uWSGI on CentOS 7 and was frustrated at how many search results were recommending building uWSGI from source when it’s available in EPEL. I’ve only deployed a handful of Python applications, and was not interested in keeping up with source-compiled dependencies to support this small project over time. We’ll discuss a couple of different ways to run these programs together with minimal customization.

Python 3.4

In CentOS 7, you can run Python 3.4 alongside Python 2.7 as it is available in EPEL.

1
2
3
rpm -q epel-release || yum -q -y install epel-release

yum -y install python34

pip

If you want to install pip, simply yum install python34-pip and it will install at /usr/bin/pip3 and /usr/bin/pip3.4, leaving /usr/bin/pip alone as that is part of the python2-pip package. This was only available recently (late 2016); previously you had to manually install and deal with /usr/bin/pip being overwritten.

Virtual Environments

Since 3.3, Python supports creating virtual environments without an external module like virtualenv. pyvenv was the “recommended tool” for this initially, though it is deprecated in Python 3.6. As such, to lean into the preferred Python nomenclature, you can use the venv module directly to create Python 3.4 virtual environments:

1
2
3
python3.4 -m venv /path/to/venv34dirname

source /path/to/venv34dirname/bin/activate

If you like using virtualenv, you’ll be disappointed to see that there is no python34-virtualenv package available in EPEL. While you could pip3 install virtualenv, that would install at /usr/bin/virtualenv, potentially overwriting the same file provided by the python-virtualenv package. But, if you do, consider the following:

1
2
3
4
5
6
7
8
9
10
11
# Install virtualenv with the pip provided the 'python34' package:
pip3 install virtualenv

# Then, a virtualenv based on /usr/bin/python3.4:
virtualenv /path/to/venv34dirname

# Now, to create based on /usr/bin/python2.7, using a newer version of virtualenv than is provided by the 'python-virtualenv' package:
virtualenv --python=/usr/bin/python2.7 /path/to/venv27dirname

# Or, recommended, use the alternate path from the 'python-virtualenv' package:
virtualenv-2.7 /path/to/venv27dirname

Unless you have a compelling reason for virtualenv on your servers, use the built-in venv module. I don’t like to overwrite files provided by packages, and we should use recommended upstream tools where possible.

uWSGI

Since uWSGI is provided by EPEL, there is no need to install it with pip.

1
yum -y install uwsgi uwsgi-plugin-python3

This will install the Python 3.4 uWSGI plugin to /usr/lib64/uwsgi/python3_plugin.so. Be sure to include the directive plugin = python3 in your uWSGI definition and proceed as normal (beyond the scope of this post).

Python 3.5 Software Collection

I love Software Collections. With Python 2.7 being the default version of Python with CentOS 7, I find it much better to keep another version in a completely separate directory structure than integrating with /usr and such. We’ll use a rebuild of the Red Hat SCL for Python 3.5, and everything will be installed to /opt/rh/rh-python35.

1
2
3
yum -y install centos-release-scl

yum -y install rh-python35-python

Using

You’ll quickly notice that the newly-installed Python 3.5 binary is not in your $PATH; it’s in /opt/rh/rh-python35/root/bin. You have a few options:

Launch a new instance of bash:

1
scl enable rh-python35 bash

Alternatively, you can modify your existing session by running one of the following manually, or add to ~/.bashrc:

1
2
3
4
5
# Preferred method:
source scl_source enable rh-python35

# Source the file directly:
#source /opt/rh/rh-python35/enable

pip

1
yum -y install rh-python35-python-pip

After installing, you can use pip, pip3, or pip3.5

Virtual Environments

You can use the built-in method:

1
python3.5 -m venv /path/to/venv35dirname

Or, you can use virtualenv (though, see the 3.4 section for the rationale I recommend against this):

1
2
3
yum -y install rh-python35-python-virtualenv

virtualenv-3.5 /path/to/venv35dirname

uWSGI

While there is a uwsgi-plugin-python3 RPM, it’s for the aforementioned python34 package. This is where you will have to compile something manually; so carefully consider the pros and cons of using a Software Collection. (Later, I may add to this post where you can use a yum post-install or post-upgrade action to ensure the compiled plugin is kept up to date.)

This is a helpful exercise to learn how you can integrate packages that install to default paths and a Software Collection which is kept away from default paths.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
yum -y install uwsgi uwsgi-plugin-common

UWSGI_VERSION="$( rpm -q uwsgi --queryformat '%{VERSION}' )"
UWSGI_PLUGINS_DIR="$( dirname "$( rpm -q uwsgi-plugin-common -l | grep '_plugin\.so' | tail -1 )" )"

source scl_source enable rh-python35

if [[ $X_SCLS == *rh-python35* ]]; then
  UWSGI_PLUGINS_DIR="/opt/rh/rh-python35/root${UWSGI_PLUGINS_DIR:?}"
fi

if [[ ! -f "${UWSGI_PLUGINS_DIR:?}/python35_plugin.so" ]]; then
  yum -y install gcc libcap-devel libuuid-devel make openssl-devel rh-python35-python-devel

  mkdir -pv /opt/rh/rh-python35/root/src
  cd /opt/rh/rh-python35/root/src

  curl -LO "https://projects.unbit.it/downloads/uwsgi-${UWSGI_VERSION:?}.tar.gz"
  tar zxvf "uwsgi-${UWSGI_VERSION:?}.tar.gz"
  cd "uwsgi-${UWSGI_VERSION:?}/"

  make PROFILE=nolang
  PYTHON=python3.5 /usr/sbin/uwsgi --build-plugin "plugins/python python35"

  [[ ! -d "${UWSGI_PLUGINS_DIR:?}/" ]] && mkdir -pv "${UWSGI_PLUGINS_DIR:?}/"
  mv -v python35_plugin.so "${UWSGI_PLUGINS_DIR:?}/"
fi

(Reference: http://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html)

After completing, your resulting uWSGI file will need to contain:

1
2
plugins-dir = /opt/rh/rh-python35/root/usr/lib64/uwsgi
plugin = python35

Which should you use?

If you need Python 3 but want to reduce the amount of customization that you need to do, I recommend installing python34 from EPEL. You can use it with uWSGI with no compiling from source.

If Python 3.4 is too old and/or you need 3.5+ functionality, the Software Collection can be an alternative to compiling Python from source and knowing that you’re getting quality packages designed by Red Hat and rebuilt by CentOS.

If you need Python 3.6, or a newer release of 3.4 or 3.5 not provided quickly enough by the package maintainers, and you don’t want to deal with all of the pain of ./configure && make && sudo make install, I would suggest looking at pyenv to meet your needs.

Please let me know if I missed something or should add anything. I’m cumulatively green with running Python in production but wanted to share my learnings as I was frustrated by the lack of good information for my needs. Feedback is welcome.

Robots.txt Disallow All With Nginx

If you’re managing an environment similar to a production and want to keep bots from indexing traffic, it’s customary to add a robots.txt file at the root of your website to disallow all. Instead of creating a two-line plain text file, you can do this with only nginx:

1
2
3
4
location = /robots.txt {
  add_header  Content-Type  text/plain;
  return 200 "User-agent: *\nDisallow: /\n";
}

Add this into your configuration management as determined by environment, or add it by hand, and no longer worry if Google might start broadcasting your dev site to the world.

Auto-restart Tomcat With Systemd

It’s commonplace that if you’ve run services on Linux and wanted something to auto-restart if it crashed, then you’ve looked at monit. You may not know, however, that systemd can provide the ability to restart a failed process without adding another daemon. In this example, we’ll keep Tomcat running if it stops for any reason, unless we systemctl stop tomcat, by using a systemd override:

1
2
3
4
5
6
7
8
9
10
11
mkdir -pv /etc/systemd/system/tomcat.service.d/

cat > /etc/systemd/system/tomcat.service.d/restart.conf <<'EOF'
[Service]
Restart=always
RestartSec=30
TimeoutStartSec=240
TimeoutStopSec=240
EOF

systemctl daemon-reload

You don’t need to restart Tomcat; systemd will begin to honor the Restart setting immediately. You can view all of these values before and after to verify the changes were honored:

1
2
3
4
5
$ systemctl show tomcat | grep -E '^(Restart(|USec)|Timeout(Start|Stop)USec)='
Restart=always
RestartUSec=30s
TimeoutStartUSec=4min
TimeoutStopUSec=4min

I recommend reading the documentation about each of the options, but a quick explanation:

  • Restart=always: Unless running systemctl stop, systemd will always attempt to start the process if it dies for any reason. That means if you were to pkill -f tomcat or any other kill command, regardless of the SIG code, it will be restarted. This would mean you should always be using systemctl to stop. If you’d like the ability to stop the process another way and only want systemd to restart if the application crashed or exited with a non-zero exit code, you can use Restart=on-failure or another value. Check the docs for your use-case.
  • RestartSec=30: First, the documentation above lists RestartSec despite systemctl show returning RestartUSec; I try and stick with the docs where I can so you can follow along with the official guidance rather than something I have stumbled upon that may not work in the future. This value is how long to sleep before restarting the service. The default is only 100ms, so I like to set it higher to allow for a busy operation to continue before systemd gets too aggressive with restarting.
  • TimeoutStartSec and TimeoutStopSec instruct systemd how long to wait for the stop and start commands before considering the operation failed and trying again. Change as needed, but I set it to 4 minutes to allow for our Tomcat applications to completely initialize.

It’s worth noting that you can run the interactive command systemctl edit tomcat and it will open /etc/systemd/system/tomcat.service.d/override.conf in your $EDITOR for adding in the contents above, eliminating the need to remember the directory structure and where to put override directives. I prefer adding files by hand as shown above for two reasons: 1) adding a file directly is obviously easier in a non-interactive server initialization script, and 2) you can have multiple files with the pattern /etc/systemd/system/name.service.d/*.conf. For organizing purposes, you could have a restart.conf where you specify your custom Restarting parameters, and another environment.conf file where you might add more environmental variables.

OS X 10.10 Yosemite Local Development Environment: Apache, PHP, and MySQL With Homebrew

This post originally featured on the Echo & Co. blog.

OS X 10.10 Yosemite comes with Apache and PHP pre-installed, but it’s not in a great configuration, requires root to make lots of changes, and can introduce issues with file ownership and permissions. We prefer to use Homebrew to avoid these problems and because it’s easier to keep up to date with newer versions of each component and extend customization. We can also set things up to be fully automatic so you can create new websites locally without needing to edit any configuration files.

With the arrival of Yosemite, some of the changes previously used in 10.9 for setting up Apache, PHP, and MySQL with Homebrew don’t work quite the same. This guide will walk you through using Homebrew to install Apache, PHP, and MySQL for a “MAMP” development environment. We’ll also use DNSMasq and Apache’s VirtualDocumentRoot to set up “auto-VirtualHosts” so you don’t need to edit configuration files when starting new projects. Finally, we’ll add a firewall rule to allow the default http port 80 to be used without running Apache as root.

The following steps are intended for use on a Yosemite system without any previous attempts to use Homebrew for Apache, PHP, or MySQL. If you have attempted to install a similar stack and run into conflicts, or you’ve upgraded your operating system from 10.9 and things broke, the final section has some troubleshooting pointers. If that fails, leave a comment and I’ll try my best to help you out.

At the conclusion of this guide, you’ll be able to create a directory like ~/Sites/project and access it immediately at http://project.dev without editing your /etc/hosts file or editing any Apache configuration. We’ll configure PHP and MySQL to allow for enough flexibility for development.

Because some of the commands span several lines, each command will be in a separate code block. This means you should copy and paste each code block in its entirety as a single command.

Before diving in, yes, this is a lot of steps. You can do it faster and pay money for something like MAMP Pro, but this is more fun, and you may learn something along the way! And, while you can simplify things with Vagrant or other virtual machine format, some people prefer to run things on “bare metal” and not have the overhead of a virtual machine.

Homebrew Setup

If you’ve not already installed Homebrew, you can follow the instructions at http://brew.sh. I used to include the command in previous walkthrough blogs, but it could change after posting, so definitely check their website to install it properly.

If you do not have git available on your system, either from Homebrew, Xcode, or another source, you can install it with Homebrew now (if you already have it installed, feel free to skip this step to keep the version of git you already have):

1
brew install -v git

PATH Variable

In previous guides on 10.9 and earlier, I added a change to $PATH in ~/.bash_profile to ensure that Homebrew-installed applications would run by default over similar ones that were already installed on OS X. Thankfully, Yosemite’s $PATH order is different than earlier OS versions and now includes the default Homebrew location of /usr/local/bin in front. If you installed Homebrew to a custom location, or are not seeing /usr/local/bin at the beginning of your shell’s $PATH, check out the file /etc/paths or the directory /etc/paths.d/.

MySQL

Install MySQL with Homebrew:

1
brew install -v mysql

Copy the default my-default.cnf file to the MySQL Homebrew Cellar directory where it will be loaded on application start:

1
cp -v $(brew --prefix mysql)/support-files/my-default.cnf $(brew --prefix)/etc/my.cnf

This will configure MySQL to allow for the maximum packet size, only appropriate for a local or development server. Also, we’ll keep each InnoDB table in separate files to keep ibdataN-type file sizes low and make file-based backups, like Time Machine, easier to manage multiple small files instead of a few large InnoDB data files. This is the first of many multi-line single commands. The following is a single, multi-line command; copy and paste the entire block at once:

1
2
3
4
5
6
cat >> $(brew --prefix)/etc/my.cnf <<'EOF'

# Echo & Co. changes
max_allowed_packet = 1073741824
innodb_file_per_table = 1
EOF

Uncomment the sample option for innodb_buffer_pool_size to improve performance:

1
sed -i '' 's/^#[[:space:]]*\(innodb_buffer_pool_size\)/\1/' $(brew --prefix)/etc/my.cnf

Now we need to start MySQL using OS X’s launchd. This used to be an involved process with launchctl commands, but now we can leverage the excellent brew services command:

1
brew tap homebrew/services
1
brew services start mysql

By default, MySQL’s root user has an empty password from any connection. You are advised to run mysql_secure_installation and at least set a password for the root user:

1
$(brew --prefix mysql)/bin/mysql_secure_installation

Apache

Start by stopping the built-in Apache, if it’s running, and prevent it from starting on boot. This is one of very few times you’ll need to use sudo:

1
sudo launchctl unload /System/Library/LaunchDaemons/org.apache.httpd.plist 2>/dev/null

The formula for building Apache is not in the default Homebrew repository that you get by installing Homebrew. While we can use the format of brew install external-repo/formula, if an external formula relies on another external formula, you have to use the brew tap command first. I know, it’s weird. So, we need to tap homebrew-dupes because “homebrew-apache/httpd22” relies on “homebrew-dupes/zlib”. Whew:

1
brew tap homebrew/dupes

A slight deviation from my prior walkthroughs: we’ll install Apache 2.2 with the event MPM and set up PHP-FPM instead of mod_php. If those terms mean anything to you and you’re curious as to why I decided to go this route; it’s because: 1) switching PHP versions is far easier with PHP-FPM and the default 9000 port instead of also editing the Apache configuration to switch the mod_php module location, and 2) if we’re therefore not using mod_php, we don’t have to use the prefork MPM and can get better performance with event or worker. As to why I’m using 2.2 instead of 2.4, popular FOSS projects like Drupal and WordPress still ship with 2.2-style .htaccess files. Using 2.4 sometimes means you have to set up “compat” modules, and that’s above the requirement for a local environment, in my opinion.

Onward! Let’s install Apache 2.2 with the event MPM, and we’ll use Homebrew’s OpenSSL library since it’s more up-to-date than OS X’s:

1
brew install -v homebrew/apache/httpd22 --with-brewed-openssl --with-mpm-event

In order to get Apache and PHP to communicate via PHP-FPM, we’ll install the mod_fastcgi module:

1
brew install -v homebrew/apache/mod_fastcgi --with-brewed-httpd22

To prevent any potential problems with previous mod_fastcgi setups, let’s remove all references to the mod_fastcgi module (we’ll re-add the new version later):

1
sed -i '' '/fastcgi_module/d' $(brew --prefix)/etc/apache2/2.2/httpd.conf

Add the logic for Apache to send PHP to PHP-FPM with mod_fastcgi, and reference that we’ll want to use the file ~/Sites/httpd-vhosts.conf to configure our VirtualHosts. The parenthesis are used to run the command in a subprocess, so that the exported variables don’t persist in your terminal session afterwards. Also, you’ll see export USERHOME a few times in this guide; I look up the full path for your user home directory from the operating system wherever a full path is needed in a configuration file and “~” or a literal “$HOME” would not work.

This is all one command, so copy and paste the entire code block at once:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
(export USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F"\: " '{print $2}') ; export MODFASTCGIPREFIX=$(brew --prefix mod_fastcgi) ; cat >> $(brew --prefix)/etc/apache2/2.2/httpd.conf <<EOF

# Echo & Co. changes

# Load PHP-FPM via mod_fastcgi
LoadModule fastcgi_module    ${MODFASTCGIPREFIX}/libexec/mod_fastcgi.so

<IfModule fastcgi_module>
  FastCgiConfig -maxClassProcesses 1 -idle-timeout 1500

  # Prevent accessing FastCGI alias paths directly
  <LocationMatch "^/fastcgi">
    <IfModule mod_authz_core.c>
      Require env REDIRECT_STATUS
    </IfModule>
    <IfModule !mod_authz_core.c>
      Order Deny,Allow
      Deny from All
      Allow from env=REDIRECT_STATUS
    </IfModule>
  </LocationMatch>

  FastCgiExternalServer /php-fpm -host 127.0.0.1:9000 -pass-header Authorization -idle-timeout 1500
  ScriptAlias /fastcgiphp /php-fpm
  Action php-fastcgi /fastcgiphp
  
  # Send PHP extensions to PHP-FPM
  AddHandler php-fastcgi .php
  
  # PHP options
  AddType text/html .php
  AddType application/x-httpd-php .php
  DirectoryIndex index.php index.html
</IfModule>

# Include our VirtualHosts
Include ${USERHOME}/Sites/httpd-vhosts.conf
EOF
)

We’ll be using the file ~/Sites/httpd-vhosts.conf to configure our VirtualHosts, but the ~/Sites folder doesn’t exist by default in newer versions of OS X. We’ll also create folders for logs and SSL files:

1
mkdir -pv ~/Sites/{logs,ssl}

Let’s populate the ~/Sites/httpd-vhosts.conf file. The biggest difference from my previous guides are that you’ll see the port numbers are 8080/8443 instead of 80/443. OS X 10.9 and earlier had the ipfw firewall which allowed for port redirecting, so we would send port 80 traffic “directly” to our Apache. But ipfw is now removed and replaced by pf which “forwards” traffic to another port. We’ll get to that later, but know that “8080” and “8443” are not typos but are acceptable because of later port forwarding. Also, I’ve now added a basic SSL configuration (though you’ll need to acknowledge warnings in your browser about self-signed certificates):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
touch ~/Sites/httpd-vhosts.conf

(export USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F"\: " '{print $2}') ; cat > ~/Sites/httpd-vhosts.conf <<EOF
#
# Listening ports.
#
#Listen 8080  # defined in main httpd.conf
Listen 8443

#
# Use name-based virtual hosting.
#
NameVirtualHost *:8080
NameVirtualHost *:8443

#
# Set up permissions for VirtualHosts in ~/Sites
#
<Directory "${USERHOME}/Sites">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    <IfModule mod_authz_core.c>
        Require all granted
    </IfModule>
    <IfModule !mod_authz_core.c>
        Order allow,deny
        Allow from all
    </IfModule>
</Directory>

# For http://localhost in the users' Sites folder
<VirtualHost _default_:8080>
    ServerName localhost
    DocumentRoot "${USERHOME}/Sites"
</VirtualHost>
<VirtualHost _default_:8443>
    ServerName localhost
    Include "${USERHOME}/Sites/ssl/ssl-shared-cert.inc"
    DocumentRoot "${USERHOME}/Sites"
</VirtualHost>

#
# VirtualHosts
#

## Manual VirtualHost template for HTTP and HTTPS
#<VirtualHost *:8080>
#  ServerName project.dev
#  CustomLog "${USERHOME}/Sites/logs/project.dev-access_log" combined
#  ErrorLog "${USERHOME}/Sites/logs/project.dev-error_log"
#  DocumentRoot "${USERHOME}/Sites/project.dev"
#</VirtualHost>
#<VirtualHost *:8443>
#  ServerName project.dev
#  Include "${USERHOME}/Sites/ssl/ssl-shared-cert.inc"
#  CustomLog "${USERHOME}/Sites/logs/project.dev-access_log" combined
#  ErrorLog "${USERHOME}/Sites/logs/project.dev-error_log"
#  DocumentRoot "${USERHOME}/Sites/project.dev"
#</VirtualHost>

#
# Automatic VirtualHosts
#
# A directory at ${USERHOME}/Sites/webroot can be accessed at http://webroot.dev
# In Drupal, uncomment the line with: RewriteBase /
#

# This log format will display the per-virtual-host as the first field followed by a typical log line
LogFormat "%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedmassvhost

# Auto-VirtualHosts with .dev
<VirtualHost *:8080>
  ServerName dev
  ServerAlias *.dev

  CustomLog "${USERHOME}/Sites/logs/dev-access_log" combinedmassvhost
  ErrorLog "${USERHOME}/Sites/logs/dev-error_log"

  VirtualDocumentRoot ${USERHOME}/Sites/%-2+
</VirtualHost>
<VirtualHost *:8443>
  ServerName dev
  ServerAlias *.dev
  Include "${USERHOME}/Sites/ssl/ssl-shared-cert.inc"

  CustomLog "${USERHOME}/Sites/logs/dev-access_log" combinedmassvhost
  ErrorLog "${USERHOME}/Sites/logs/dev-error_log"

  VirtualDocumentRoot ${USERHOME}/Sites/%-2+
</VirtualHost>
EOF
)

You may have noticed that ~/Sites/ssl/ssl-shared-cert.inc is included multiple times; create that file and the SSL files it needs:

1
2
3
4
5
6
7
8
(export USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F"\: " '{print $2}') ; cat > ~/Sites/ssl/ssl-shared-cert.inc <<EOF
SSLEngine On
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile "${USERHOME}/Sites/ssl/selfsigned.crt"
SSLCertificateKeyFile "${USERHOME}/Sites/ssl/private.key"
EOF
)
1
2
3
4
5
6
7
8
9
openssl req \
  -new \
  -newkey rsa:2048 \
  -days 3650 \
  -nodes \
  -x509 \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=$(whoami)/CN=*.dev" \
  -keyout ~/Sites/ssl/private.key \
  -out ~/Sites/ssl/selfsigned.crt

Start Apache

Start Homebrew’s Apache and set to start on login:

1
brew services start httpd22

Run with port 80

You may notice that httpd.conf is running Apache on ports 8080 and 8443. Manually adding “:8080” each time you’re referencing your dev sites is no fun, but running Apache on port 80 requires root. The next two commands will create and load a firewall rule to forward port 80 requests to 8080, and port 443 requests to 8443. The end result is that we don’t need to add the port number when visiting a project dev site, like “http://projectname.dev/” instead of “http://projectname.dev:8080/”.

The following command will create the file /Library/LaunchDaemons/co.echo.httpdfwd.plist as root, and owned by root, since it needs elevated privileges:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
sudo bash -c 'export TAB=$'"'"'\t'"'"'
cat > /Library/LaunchDaemons/co.echo.httpdfwd.plist <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
${TAB}<key>Label</key>
${TAB}<string>co.echo.httpdfwd</string>
${TAB}<key>ProgramArguments</key>
${TAB}<array>
${TAB}${TAB}<string>sh</string>
${TAB}${TAB}<string>-c</string>
${TAB}${TAB}<string>echo "rdr pass proto tcp from any to any port {80,8080} -> 127.0.0.1 port 8080" | pfctl -a "com.apple/260.HttpFwdFirewall" -Ef - &amp;&amp; echo "rdr pass proto tcp from any to any port {443,8443} -> 127.0.0.1 port 8443" | pfctl -a "com.apple/261.HttpFwdFirewall" -Ef - &amp;&amp; sysctl -w net.inet.ip.forwarding=1</string>
${TAB}</array>
${TAB}<key>RunAtLoad</key>
${TAB}<true/>
${TAB}<key>UserName</key>
${TAB}<string>root</string>
</dict>
</plist>
EOF'

This file will be loaded on login and set up the 80->8080 and 443->8443 port forwards, but we can load it manually now so we don’t need to log out and back in:

1
sudo launchctl load -Fw /Library/LaunchDaemons/co.echo.httpdfwd.plist

PHP

The following is for the latest release of PHP, version 5.6. If you’d like to use 5.3, 5.4 or 5.5, simply change the “5.6” and “php56” values below appropriately.

1
brew install -v homebrew/php/php56

Set timezone and change other PHP settings (sudo is needed here to get the current timezone on OS X) to be more developer-friendly, and add a PHP error log (without this, you may get Internal Server Errors if PHP has errors to write and no logs to write to):

1
(export USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F"\: " '{print $2}') ; sed -i '-default' -e 's|^;\(date\.timezone[[:space:]]*=\).*|\1 \"'$(sudo systemsetup -gettimezone|awk -F"\: " '{print $2}')'\"|; s|^\(memory_limit[[:space:]]*=\).*|\1 512M|; s|^\(post_max_size[[:space:]]*=\).*|\1 200M|; s|^\(upload_max_filesize[[:space:]]*=\).*|\1 100M|; s|^\(default_socket_timeout[[:space:]]*=\).*|\1 600|; s|^\(max_execution_time[[:space:]]*=\).*|\1 300|; s|^\(max_input_time[[:space:]]*=\).*|\1 600|; $a\'$'\n''\'$'\n''; PHP Error log\'$'\n''error_log = '$USERHOME'/Sites/logs/php-error_log'$'\n' $(brew --prefix)/etc/php/5.6/php.ini)

Fix a pear and pecl permissions problem:

1
chmod -R ug+w $(brew --prefix php56)/lib/php

The optional Opcache extension will speed up your PHP environment dramatically, so let’s install it. Then, we’ll bump up the opcache memory limit:

1
brew install -v php56-opcache
1
/usr/bin/sed -i '' "s|^\(\;\)\{0,1\}[[:space:]]*\(opcache\.enable[[:space:]]*=[[:space:]]*\)0|\21|; s|^;\(opcache\.memory_consumption[[:space:]]*=[[:space:]]*\)[0-9]*|\1256|;" $(brew --prefix)/etc/php/5.6/php.ini

Finally, let’s start PHP-FPM:

1
brew services start php56

Optional: At this point, if you want to switch between PHP versions, you’d want to: brew services stop php56 && brew unlink php56 && brew link php54 && brew services start php54. No need to touch the Apache configuration at all!

DNSMasq

A difference now between what I’ve shown before, is that we don’t have to run on port 53 or run dnsmasq as root. The end result here is that any DNS request ending in .dev reply with the IP address 127.0.0.1:

1
brew install -v dnsmasq
1
echo 'address=/.dev/127.0.0.1' > $(brew --prefix)/etc/dnsmasq.conf
1
echo 'listen-address=127.0.0.1' >> $(brew --prefix)/etc/dnsmasq.conf
1
echo 'port=35353' >> $(brew --prefix)/etc/dnsmasq.conf

Similar to how we run Apache and PHP-FPM, we’ll start DNSMasq:

1
brew services start dnsmasq

With DNSMasq running, configure OS X to use your local host for DNS queries ending in .dev:

1
sudo mkdir -v /etc/resolver
1
sudo bash -c 'echo "nameserver 127.0.0.1" > /etc/resolver/dev'
1
sudo bash -c 'echo "port 35353" >> /etc/resolver/dev'

To test, the command ping -c 3 fakedomainthatisntreal.dev should return results from 127.0.0.1. If it doesn’t work right away, try turning WiFi off and on (or unplug/plug your ethernet cable), or reboot your system.

Great! So, what did I do?

We set up Apache to run on boot on ports 8080 and 8443 with auto-VirtualHosts for directories in the ~/Sites folder and PHP-FPM via mod_fastcgi. The OS X firewall will forward all port 80 traffic to port 8080 and port 443 to port 8443, so we don’t have specify the port number when visiting web pages in local web browsers or run Apache as root. MySQL is installed and set to run on boot as well. DNSMasq and some OS X configuration is used to direct any hostname ending in .dev to the local system to work in conjunction with Apache’s auto-VirtualHosts.

What do I do now?

You shouldn’t need to edit the Apache configuration or edit /etc/hosts for new local development sites. Simply create a directory in ~/Sites and then reference “http://” + that foldername + “.dev/” in your browser to access it.

For example, download Drupal 7 to the directory ~/Sites/firstproject, and it can then be accessed at http://firstproject.dev/* without any additional configuration. A caveat – you will need to uncomment the line in Drupal’s .htaccess* containing “RewriteBase /” to work with the auto-VirtualHosts configuration.

What if this “auto-VirtualHost” doesn’t work for [other project]?

If you need to create a manual VirtualHost in Apache because the auto-VirtualHost does not work for your configuration, you will need to declare it before the auto-VirtualHosts if you’re also going to use .dev as the TLD. Otherwise the auto-VirtualHost block will be accessed first. I have a commented-out sample for a manual VirtualHost entry in “~/Sites/httpd-vhosts.conf” you may use.

Troubleshooting

The commands above can be run on a fresh Yosemite system without issue as I’ve tested with fresh installs in VMware. Nearly every problem I’ve heard about has been related to upgrading from 10.9 or earlier, or switching to this style of setup from another or similar setup. The easiest thing to do would be to brew uninstall each component referenced above, delete all related config files in $(brew —prefix)/etc and any Homebrew files in ~/Library/LaunchAgents, and start over. If that’s a bit heavy-handed, check each of the configuration files edited in this guide and look for duplicate entries or typos.

You can also check log files for error output:

  • Apache: $(brew —prefix)/var/log/apache2/error_log, or run httpd -DFOREGROUND and look for output
  • PHP-FPM: $(brew —prefix)/var/log/php-fpm.log
  • MySQL: $(brew —prefix)/var/mysql/$(hostname).err
  • DNSMasq: no log file, run dnsmasq --keep-in-foreground and look for output

Leave a comment if you have any questions or problems!

OS X 10.9 Local Development Environment: Apache, PHP, and MySQL With Homebrew

This post originally featured on the Echo & Co. blog.

There’s nothing quite like setting up everything on your Mac for Drupal (or other PHP) development in a way that things just work and don’t need constant fiddling. This guide will walk you through using Homebrew to install Apache, PHP, and MySQL for a “MAMP” development environment. We’ll also use DNSMasq and Apache’s VirtualDocumentRoot to set up “auto-VirtualHosts,” and add a firewall rule to allow the default http port 80 to be used without running Apache as root.

At the conclusion of this guide, you’ll be able to create a directory like ~/Sites/project and access it at http://project.dev without editing your /etc/hosts file or editing any Apache configuration. You’ll also be able to use xip.io with auto-VirtualHosts for accessing your sites on other devices in your local network.

We also configure PHP and MySQL to allow for enough flexibility for complex operations generally only reserved for development and not production.

The OS X operating system comes with Apache and PHP pre-installed, and I’ve previously recommended utilizing them to some degree for getting a local PHP development environment on your Mac. Since then, the community around Homebrew has improved dramatically and I now recommend that our developers at Echo & Co. use Homebrew exclusively for all components.

Homebrew Setup

If you’ve not already install Homebrew, you can follow the instructions at brew.sh, or simply run the following command:

1
2
3
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
brew doctor
brew update

Also of note, if you do not have git available on your system, either from Homebrew, Xcode, or another source, you can install it with Homebrew now (if you already have it installed, feel free to skip this step to keep the version of git you already have):

1
brew install -v git

PATH Variable

Since OS X already comes with PHP and Apache, we’ll want to make sure that our brew-installed versions appear in the shell path before the built-in ones. The following command adds logic to your ~/.bash_profile to ensure the Homebrew directory is in the beginning of $PATH.

1
echo "export PATH=\$(echo \$PATH | sed 's|/usr/local/bin||; s|/usr/local/sbin||; s|::|:|; s|^:||; s|\(.*\)|/usr/local/bin:/usr/local/sbin:\1|')" >> ~/.bash_profile && source ~/.bash_profile

MySQL

The following commands will download and install the latest version of MySQL and do some basic configuration to allow for large imports and a couple other miscellaneous configuration changes.

1
2
3
4
5
6
7
8
9
10
11
brew install -v mysql

cp -v $(brew --prefix mysql)/support-files/my-default.cnf $(brew --prefix mysql)/my.cnf

cat >> $(brew --prefix mysql)/my.cnf <<'EOF'
# Echo & Co. changes
max_allowed_packet = 2G
innodb_file_per_table = 1
EOF

sed -i '' 's/^# \(innodb_buffer_pool_size\)/\1/' $(brew --prefix mysql)/my.cnf

Now we need to start MySQL using OS X’s launchd, and we’ll set it to start when you login.

1
2
3
4
5
[ ! -d ~/Library/LaunchAgents ] && mkdir -v ~/Library/LaunchAgents

[ -f $(brew --prefix mysql)/homebrew.mxcl.mysql.plist ] && ln -sfv $(brew --prefix mysql)/homebrew.mxcl.mysql.plist ~/Library/LaunchAgents/

[ -e ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist ] && launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist

By default, MySQL’s root user has an empty password from any connection. You are advised to run mysql_secure_installation and at least set a password for the root user.

1
$(brew --prefix mysql)/bin/mysql_secure_installation

Apache

Start by stopping the built-in Apache, if it’s running, and prevent it from starting on boot. This is one of very few times you’ll need to use sudo.

1
sudo launchctl unload -w /System/Library/LaunchDaemons/org.apache.httpd.plist 2>/dev/null

Apache is not in the default repository of Homebrew formulas, nor are some dependencies, so we use the brew tap command to add other formula repositories.

1
2
brew tap homebrew/dupes
brew tap homebrew/apache

We’ll install Apache 2.2 with Apr and OpenSSL from Homebrew as well instead of utilizing the built-in versions of those tools.

1
brew install -v httpd22 --with-brewed-openssl

We’ll be using the file ~/Sites/httpd-vhosts.conf to configure our VirtualHosts, so we create necessary directories and then include the file in httpd.conf.

1
2
3
4
5
6
7
8
[ ! -d ~/Sites ] && mkdir -pv ~/Sites

touch ~/Sites/httpd-vhosts.conf

USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F": " '{print $2}') cat >> $(brew --prefix)/etc/apache2/2.2/httpd.conf <<EOF
# Include our VirtualHosts
Include ${USERHOME}/Sites/httpd-vhosts.conf
EOF

We’ll create a folder for our logs in ~/Sites as well.

1
[ ! -d ~/Sites/logs ] && mkdir -pv ~/Sites/logs

Now to fill in the contents of ~/Sites/httpd-vhosts.conf that we included in httpd.conf earlier. Note that this is one command to copy and paste! Start with USERHOME through the second EOF has a single copy and paste block for the terminal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F": " '{print $2}') cat > ~/Sites/httpd-vhosts.conf <<EOF
#
# Use name-based virtual hosting.
#
NameVirtualHost *:80

#
# Set up permissions for VirtualHosts in ~/Sites
#
<Directory "${USERHOME}/Sites">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

# For http://localhost in the users' Sites folder
<VirtualHost _default_:80>
    ServerName localhost
    DocumentRoot "${USERHOME}/Sites"
</VirtualHost>

#
# VirtualHosts
#

## Manual VirtualHost template
#<VirtualHost *:80>
#  ServerName project.dev
#  CustomLog "${USERHOME}/Sites/logs/project.dev-access_log" combined
#  ErrorLog "${USERHOME}/Sites/logs/project.dev-error_log"
#  DocumentRoot "${USERHOME}/Sites/project.dev"
#</VirtualHost>

#
# Automatic VirtualHosts
# A directory at ${USERHOME}/Sites/webroot can be accessed at http://webroot.dev
# In Drupal, uncomment the line with: RewriteBase /

# This log format will display the per-virtual-host as the first field followed by a typical log line
LogFormat "%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedmassvhost

# Auto-VirtualHosts with .dev
<VirtualHost *:80>
  ServerName dev
  ServerAlias *.dev

  CustomLog "${USERHOME}/Sites/logs/dev-access_log" combinedmassvhost
  ErrorLog "${USERHOME}/Sites/logs/dev-error_log"

  VirtualDocumentRoot ${USERHOME}/Sites/%-2+
</VirtualHost>

# Auto-VirtualHosts with xip.io
<VirtualHost *:80>
  ServerName xip
  ServerAlias *.xip.io

  CustomLog "${USERHOME}/Sites/logs/dev-access_log" combinedmassvhost
  ErrorLog "${USERHOME}/Sites/logs/dev-error_log"

  VirtualDocumentRoot ${USERHOME}/Sites/%-7+
</VirtualHost>
EOF

Run with port 80

You may notice that httpd.conf is running Apache on port 8080, but the above are using port 80. The next two commands will create and load a firewall rule to forward 8080 requests to 80. The end result is that we can use port 80 in VirtualHosts without needing to run Apache as root.

The following single command will create the file /Library/LaunchDaemons/co.echo.httpdfwd.plist as root, and owned by root, since it needs elevated privileges.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
sudo bash -c 'export TAB=$'"'"'\t'"'"'
cat > /Library/LaunchDaemons/co.echo.httpdfwd.plist <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
${TAB}<key>Label</key>
${TAB}<string>co.echo.httpdfwd</string>
${TAB}<key>ProgramArguments</key>
${TAB}<array>
${TAB}${TAB}<string>sh</string>
${TAB}${TAB}<string>-c</string>
${TAB}${TAB}<string>ipfw add fwd 127.0.0.1,8080 tcp from any to me dst-port 80 in &amp;&amp; sysctl -w net.inet.ip.forwarding=1</string>
${TAB}</array>
${TAB}<key>RunAtLoad</key>
${TAB}<true/>
${TAB}<key>UserName</key>
${TAB}<string>root</string>
</dict>
</plist>
EOF'

This file will be loaded on login and set up the 80->8080 port forward, but we can load it manually now so we don’t need to log out and back in.

1
sudo launchctl load -w /Library/LaunchDaemons/co.echo.httpdfwd.plist

PHP

The following is for the latest release of PHP, version 5.5. If you’d like to use 5.3, 5.4 or 5.6, simply change the “5.5” and “php55” values below appropriately. (Note: if you use 5.3, the OpCache extension instructions are different. They will be posted below after the instructions for newer versions.)

Start by adding the PHP tap for Homebrew. PHP 5.3 needs an additional tap, so skip the second command if you are using 5.4 or higher.

1
2
3
4
brew tap homebrew/php

# Skip this if using PHP 5.4 or higher
brew tap homebrew/versions

Install PHP and mod_php. This command will also load the PHP module in the httpd.conf file for you.

1
brew install -v php55 --homebrew-apxs --with-apache

Add PHP configuration to Apache’s httpd.conf file.

1
2
3
4
5
6
cat >> $(brew --prefix)/etc/apache2/2.2/httpd.conf <<EOF
# Send PHP extensions to mod_php
AddHandler php5-script .php
AddType text/html .php
DirectoryIndex index.php index.html
EOF

Set timezone and change other PHP settings (this is a single command). sudo is needed here to get the current timezone on OS X (in previous versions of OS X it wasn’t needed, I’m not sure why it is now).

1
sed -i '-default' "s|^;\(date\.timezone[[:space:]]*=\).*|\1 \"$(sudo systemsetup -gettimezone|awk -F": " '{print $2}')\"|; s|^\(memory_limit[[:space:]]*=\).*|\1 256M|; s|^\(post_max_size[[:space:]]*=\).*|\1 200M|; s|^\(upload_max_filesize[[:space:]]*=\).*|\1 100M|; s|^\(default_socket_timeout[[:space:]]*=\).*|\1 600|; s|^\(max_execution_time[[:space:]]*=\).*|\1 300|; s|^\(max_input_time[[:space:]]*=\).*|\1 600|;" $(brew --prefix)/etc/php/5.5/php.ini

Add a PHP error log; without this, you may get Internal Server Errors if PHP has errors to write and no logs to write to (this is a single command; be sure to copy and paste the lines containing USERHOME through the last EOF as a single command).

1
2
3
4
USERHOME=$(dscl . -read /Users/`whoami` NFSHomeDirectory | awk -F": " '{print $2}') cat >> $(brew --prefix)/etc/php/5.5/php.ini <<EOF
; PHP Error log
error_log = ${USERHOME}/Sites/logs/php-error_log
EOF

This weird little “hack” is needed to fix a permissions problem with using pear or pecl.

1
touch $(brew --prefix php55)/lib/php/.lock && chmod 0644 $(brew --prefix php55)/lib/php/.lock

The OpCache extension will speed up your PHP environment dramatically, and it’s easy to install for 5.4 and higher. If you are looking to install PHP 5.3, skip this block.

1
brew install -v php55-opcache

Skip this block unless you are using PHP 5.3. Because there is no php53-opcache Homebrew formula, we can install it with pecl and replicate the same configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
pecl install zendopcache-beta

sed -i '' "s|^\(zend_extension=\"\)\(opcache\.so\"\)|\1$(php -r 'print(ini_get("extension_dir")."/");')\2|" $(brew --prefix)/etc/php/5.3/php.ini

[ ! -d $(brew --prefix)/etc/php/5.3/conf.d ] && mkdir -pv $(brew --prefix)/etc/php/5.3/conf.d

echo "[opcache]" > $(brew --prefix)/etc/php/5.3/conf.d/ext-opcache.ini

grep -E '^zend_extension.*opcache\.so' $(brew --prefix)/etc/php/5.3/php.ini >> $(brew --prefix)/etc/php/5.3/conf.d/ext-opcache.ini

sed -i '' '/^zend_extension.*opcache\.so/d' $(brew --prefix)/etc/php/5.3/php.ini

# "php54" is not a typo here- I'm using a sample config file from
# another recipe for my config file in php53
grep -E '^[[:space:]]*opcache\.' \
  $(brew --prefix)/Library/Taps/homebrew/homebrew-php/Formula/php54-opcache.rb \
  | sed 's/^[[:space:]]*//g' >> $(brew --prefix)/etc/php/5.3/conf.d/ext-opcache.ini

And continue with steps for all PHP versions: give OpCache some more memory to keep more opcode caches.

1
sed -i '' "s|^\(opcache\.memory_consumption=\)[0-9]*|\1256|;" $(brew --prefix)/etc/php/5.5/conf.d/ext-opcache.ini

Start Apache

Start Homebrew’s Apache and set to start on boot.

1
2
3
ln -sfv $(brew --prefix httpd22)/homebrew.mxcl.httpd22.plist ~/Library/LaunchAgents

launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.httpd22.plist

DNSMasq

I’ve covered this before, but we’ll list the commands here again for completeness. This example will have any DNS request ending in .dev reply with the IP address 127.0.0.1.

1
2
3
4
5
brew install -v dnsmasq

echo 'address=/.dev/127.0.0.1' > $(brew --prefix)/etc/dnsmasq.conf

echo 'listen-address=127.0.0.1' >> $(brew --prefix)/etc/dnsmasq.conf

Because DNS services run on a lower port, we need to have this run out of /Library/LaunchDaemons, so we do need to use sudo for the initial setup.

1
2
3
sudo cp -v $(brew --prefix dnsmasq)/homebrew.mxcl.dnsmasq.plist /Library/LaunchDaemons

sudo launchctl load -w /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

With DNSMasq running, configure OS X to use your local host for DNS queries ending in .dev

1
2
3
sudo mkdir -v /etc/resolver

sudo bash -c 'echo "nameserver 127.0.0.1" > /etc/resolver/dev'

Great! So, what did I do?

We set up Apache to run on boot on port 8080 with mod_php with auto-VirtualHosts for directories in the ~/Sites folder. The OS X firewall will forward all port 80 traffic to port 8080, so we don’t have specify the port number when visiting web pages in web browsers. MySQL is installed and set to run on boot as well. DNSMasq and some OS X configuration is used to direct any hostname ending in .dev to the local system to work in conjunction with Apache’s auto-VirtualHosts.

What do I do now?

You shouldn’t need to edit the Apache configuration or edit /etc/hosts for new local development sites. Simply create a directory in ~/Sites and then reference that foldername + .dev in your browser to access it.

For example, use drush to download Drupal 7 to the directory ~/Sites/firstproject, and it can then be accessed at http://firstproject.dev/ without any additional configuration. A caveat – you will need to uncomment the line in Drupal’s .htaccess containing “RewriteBase /” to work with the auto-VirtualHosts configuration.

What about this xip.io thing?

If your Mac’s LAN IP address is 192.168.0.10, you can access sites from any other device on your local network using http://firstproject.192.168.0.10.xip.io/. You can test a websites on your Mac with your phone/tablet/etc. Pretty great, right?

What if this “auto-VirtualHost” doesn’t work for me?

If you need to create a manual VirtualHost in Apache because the auto-VirtualHost does not work for your configuration, you may need to declare it before the auto-VirtualHosts if you’re also going to use .dev as the TLD. Otherwise the auto-VirtualHost block will be accessed first.

What’s the Deal With Pressflow 7?

This post originally featured on the Echo & Co. blog.

Anyone who used Drupal 6 at scale knew that Pressflow 6 was pretty great. It forked Drupal and added several performance enhancements, including the ability to use external caches. Since Drupal 7 incorporated lots of the Pressflow 6 features, what’s the deal with Pressflow 7?

After Drupal 7 launched, it seemed like Pressflow 7 would continue the momentum from Pressflow 6 and greatly enhance Drupal 7. But, here we are in 2013, and you don’t hear about Pressflow 7 very often, and searching for differences does not yield much more than some diffs between baseline Drupal core 7 and the Pressflow 7 repository on GitHub. Since there are clearly some differences, let’s dive in and see what they are exactly.

A full diff as of October 2013 can be found at https://gist.github.com/alanthing/6064500 for further reading. Original post at drupal.stackexchange.com.

Creating PHP Packages as a Software Collection

Maintaining multiple versions of software can be tricky. Most of the explanations I have come across suggest using the version provided by your Linux distribution and installing others from source using a custom path, like ./configure --prefix=/opt/php-5.3.26; make; make install, but it’s not pleasant to update, manage small sub-packages, or make init.d scripts. The ideal option is to use packages that install to another location alongside the defaults, but hacking up spec files to override the default RPM macros like %configure (for CentOS and RHEL anyway) is not fun. This is where Red Hat Software Collections come in to play. While Red Hat recently announced the Red Hat Software Collections 1.0 Beta for alternate versions of PHP, Perl, MySQL, and more, let’s step through how to create your own Software Collection, or SCL. I’ll show how to create PHP 5.3.26 to be installed in /opt/rh/ by converting existing SPEC files, and I’ll also include diff files for how I built PHP 5.2.17 as an SCL.

If you’re just here to get RPMs and don’t much care for how they were made, or if you’d like to get the source RPMs and learn for yourself, you can visit our repository and get whatever you’d like, including Yum repos:

Before I get too deep into the instructions, I’d like to share the resources I used:

An important point to be learned is: Software Collections are essentially easy-to-use RPM macros that make installing packages into non-standard directories a piece of cake. As I explain various steps, I’ll attempt to explain how the Software Collections function and how I’ve seen others set them up so we’re following best practices in ours.

Explaining the Software Collection base

Every Software Collection needs a base package that sets up the directory structure for the packages. When you use Red Hat’s method of building SCLs, specifically via the scl-utils-build package, the base package will create /opt/rh/%{scl_prefix}/root with subdirectories like bin/, usr/, etc/, as so on.

For the SCL named php53, here is the base directory structure:

/opt/rh/php53/root/bin
/opt/rh/php53/root/boot
/opt/rh/php53/root/dev
/opt/rh/php53/root/etc
/opt/rh/php53/root/home
/opt/rh/php53/root/lib
/opt/rh/php53/root/lib64
/opt/rh/php53/root/media
/opt/rh/php53/root/mnt
/opt/rh/php53/root/opt
/opt/rh/php53/root/proc
/opt/rh/php53/root/root
/opt/rh/php53/root/sbin
/opt/rh/php53/root/selinux
/opt/rh/php53/root/srv
/opt/rh/php53/root/sys
/opt/rh/php53/root/tmp
/opt/rh/php53/root/usr
/opt/rh/php53/root/var

Most of the SCLs I have seen contain an enable script at /opt/rh/%{scl_prefix}/enable that will add the binaries, man pages, and library paths to the session. In this case, running source /opt/rh/php53/enable will run the following:

1
2
3
export PATH=/opt/rh/php52/root/usr/bin:/opt/rh/php52/root/usr/sbin:$PATH
export LD_LIBRARY_PATH=/opt/rh/php52/root/usr/lib64:$LD_LIBRARY_PATH
export MANPATH=/opt/rh/php52/root/usr/share/man:$MANPATH

The base package will also create a %{scl_prefix}-build subpackage that creates RPM macros that will come in handy as the SPEC file is edited.

Building the Software Collection base

  • Start with http://people.redhat.com/rcollet/php54/rhel/5/SRPMS/php54-1-2.el5.src.rpm because it’s ready for CentOS 5 and 6

  • Rename or copy php54.spec to php53.spec

  • Set the global variable scl to be php53:

    %global scl php53
    
  • If you haven’t already, install rpm-build and mock, and you’ll also need scl-utils-build for later:

    yum -y install rpm-build mock scl-utils-build
    
  • CentOS 5 needs additional rpmbuild options when using a CentOS 6 build host because CentOS 6 uses stronger hashes by default

    • CentOS 5:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" -bs --nodeps php53.spec
      
    • CentOS 6:

      rpmbuild -bs --nodeps php53.spec
      
  • Build with mock (repeat el6 instructions for el5):

    mock -v -r epel-6-x86_64 php53-1-2.el5.src.rpm
    
  • Copy resulting files to local folder and create a local Yum repo to be used later (repeat el6 instructions for el5):

    mkdir -vp ~/rpmbuild/RPMS/repo/el6/{x86_64,SRPMS}
    cp -va /var/lib/mock/epel-6-x86_64/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    createrepo -v -d ~/rpmbuild/RPMS/repo/el6/x86_64
    

Mock

We used mock to build the SCL base package, and to make our lives significantly easier, we’ll continue to use it with some modifications specific to our SCL.

For all remaining steps, unless there are specific notes for CentOS 5 and 6, repeat the 6/el6 instructions for 5/el5.

  • Change to the mock configuration directory:

    cd /etc/mock
    
  • Copy the default EPEL files to new ones specific to our SCL:

    cp -a epel-6-x86_64.cfg epel-6-x86_64-scl-php53.cfg
    
  • Edit epel-6-x86_64-scl-php53.cfg and:

    • Add -scl-php53 to the existing value of config_opts[‘root’]:

      config_opts['root'] = 'epel-6-x86_64-scl-php53'
      
    • CentOS 5 and 6 have different options for config_opts[‘chroot_setup_cmd’]:

      • CentOS 5: Add scl-utils-build php53-build the following to the existing value:

        config_opts['chroot_setup_cmd'] = 'install buildsys-build scl-utils-build php53-build'
        
      • CentOS 6: Change from ‘groupinstall buildsys-build’ to install @buildsys-build scl-utils-build php53-build (the @ will grab the group and then still allow for non-group packages to be added):

        config_opts['chroot_setup_cmd'] = 'install @buildsys-build scl-utils-build php53-build'
        
  • Add the local repo where you left the files from the section above as a new repo; I added mine above [base]. Change /home/alan to the appropriate path for your system!

    [localrepo-6-x86_64]
    name=localrepo-6-x86_64
    enabled=1
    baseurl=file:///home/alan/rpmbuild/RPMS/repo/el6/x86_64
    

PHP

  • Use latest php-5.3.x SRPM from http://www2.atomicorp.com/channels/source/php/

  • Install the SRPM

    rpm -ivh php-5.3.26-19.art.src.rpm
    
  • Go to the SPECS folder, typically at ~/rpmbuild/SPECS, and copy php-art.spec to php.spec

    cp -v php-art.spec php.spec
    
  • Several changes are needed to add the SCL macros to the SPEC file; see the Red Hat documentation. Download my patch file for a full list of changes needed to convert AtomiCorp’s PHP SPEC to one that can be used with and without SCL options: php_scl.diff

    curl --remote-name --location http://alanthing.com/files/php_scl.diff
    

    Let’s step through the changes:

    • The first section allows for variables beginning with _root_ to be available if it’s not being built with a SCL. As I said earlier, I heavily borrowed from Remi’s PHP 5.4 SCL SRPM. The _root_ macros are provided by the scl-utils-build package, but this block allows them to work if an SCL is not being used.

      +%if 0%{?scl:1}
      +%scl_package php
      +%else
      +%global pkg_name          %{name}
      +%global _root_sysconfdir  %{_sysconfdir}
      +%global _root_bindir      %{_bindir}
      +%global _root_sbindir     %{_sbindir}
      +%global _root_includedir  %{_includedir}
      +%global _root_libdir      %{_libdir}
      +%global _root_prefix      %{_prefix}
      +%if 0%{?rhel} < 6
      +%global _root_initddir    /etc
      +%else
      +%global _root_initddir    %{_initddir}
      +%endif
      +%endif
      
    • These values allow you to use the SCL mod_php with the built-in version of Apache

      +%global _httpd_confdir     %{_root_sysconfdir}/httpd/conf.d
      +%global _httpd_moddir      %{_libdir}/httpd/modules
      +%global _root_httpd_moddir %{_root_libdir}/httpd/modules
      
    • There are other changes scattered throughout the file to force using locations outside of the SCL, like:

      -%global httpd_mmn %(cat %{_includedir}/httpd/.mmn || echo missing-httpd-devel)
      +%global httpd_mmn %(cat %{_root_includedir}/httpd/.mmn || echo missing-httpd-devel)
      
    • In all of the Name: lines, add the macro %{?scl_prefix}, which will only be used when building an SCL. For example, if you use the SCL rpmbuild options, the common package here would be called php53-php-common, but without it would still be called php-common. This applies for the main Name and sub-packages. It’s also used for Obsoletes, Conflicts, Provides, and Requires.

      -Name: %{phpname}
      +Name: %{?scl_prefix}%{phpname}
      
      -Obsoletes: php-mhash
      -Conflicts: php-zend-optimizer
      +Obsoletes: %{?scl_prefix}php-mhash
      +Conflicts: %{?scl_prefix}php-zend-optimizer
      
    • The main package would provide mod_php for the built-in Apache, so this addition prevents a collision of having both php and php53-php installed:

      +%if 0%{?scl:1}
      +Conflicts: php53, php
      +%else
       # php53
       Obsoletes: php53
      +%endif
      
    • As per the Red Hat documentation, a main package (not php, but php-common, since php-common is needed by every php package) must require the base package

      %{?scl:Requires:%scl_runtime}
      
    • The AtomiCorp SPEC is carefully designed to prevent conflicts with the CentOS 5 packages called php53 that are not part of an SCL. I added a change to only keep the Obsoletes if it’s not an SCL

      +%if 0%{?!scl:1}
       Obsoletes: php53-imap
      +%endif
      
    • I made a few improvements that aren’t related to SCL. I won’t list them all here, but one that I noticed was that libevent is no longer needed after PHP 5.3.4, as my comment suggests explaining why I comment it out:

      +# libevent needed *until* 5.3.4, see http://bugzilla.redhat.com/show_bug.cgi?id=835671
      +#BuildRequires: libevent-devel >= 1.4.11
      
    • A lot of the compile options will use the built-in libraries, like:

      - --with-freetype-dir=%{_prefix} \
      - --with-png-dir=%{_prefix} \
      - --with-xpm-dir=%{_prefix} \
      + --with-freetype-dir=%{_root_prefix} \
      + --with-png-dir=%{_root_prefix} \
      + --with-xpm-dir=%{_root_prefix} \
      
    • In a few places I use sed to fix config files, here’s one where I ensure the SCL root is used for the session.save_path by changing /var/lib to %{_localstatedir}/lib

      -%{__sed} -e '/session.save_path/s/php/%{phpname}/' %{SOURCE2} >$RPM_BUILD_ROOT%{_sysconfdir}/php.ini
      +%{__sed} -e 's|/var/lib/php/session|%{_localstatedir}/lib/%{phpname}/session|' %{SOURCE2} >$RPM_BUILD_ROOT%{_sysconfdir}/php.ini
      
    • Instead of installing the Apache module files in the system default directories, I decided to keep them in the SCL root and create symbolic links, though I do put the mod_php Apache configuration file in /etc/httpd/conf.d

      -  >$RPM_BUILD_ROOT/%{_origsysconfdir}/httpd/conf.d/%{phpname}.conf
      +  >$RPM_BUILD_ROOT/%{_httpd_confdir}/%{?scl_prefix}%{phpname}.conf
      +%endif
      +
      +# Symlink SCL DSOs into real httpd moddir, mod_php config into real httpd config directory
      +%if %{?scl:1}0
      +install -m 755 -d $RPM_BUILD_ROOT%{_root_httpd_moddir}
      +install -m 755 -d $RPM_BUILD_ROOT%{_httpd_confdir}
      +%if %{phpname} == php
      +ln -s %{_httpd_moddir}/libphp5.so $RPM_BUILD_ROOT%{_root_httpd_moddir}/libphp5.so
      +ln -s %{_httpd_moddir}/libphp5-zts.so $RPM_BUILD_ROOT%{_root_httpd_moddir}/libphp5-zts.so
      +%else
      +ln -s %{_httpd_moddir}/libphp5.so $RPM_BUILD_ROOT%{_root_httpd_moddir}/lib%{phpname}.so
      +ln -s %{_httpd_moddir}/libphp5-zts.so $RPM_BUILD_ROOT%{_root_httpd_moddir}/lib%{phpname}-zts.so
      +%endif
       %endif
      
    • A common practice I observed in other SCL files was putting the init.d script in /etc/init.d instead of the SCL root with prepending the SCL name. For example, if you build with SCL options, the PHP-FPM script would be at /etc/init.d/php53-php-fpm, or without it would be /etc/init.d/php-fpm, either one using the correct paths:

      -install -m 755 -d $RPM_BUILD_ROOT%{_originitdir}
      -install -m 755 %{SOURCE6} $RPM_BUILD_ROOT%{_originitdir}/php-fpm
      +install -m 755 -d $RPM_BUILD_ROOT%{_root_initddir}
      +install -m 755 %{SOURCE6} $RPM_BUILD_ROOT%{_root_initddir}/%{?scl_prefix}%{phpname}-fpm
      +sed -e '/daemon/s:php-fpm:/usr/sbin/php-fpm:' \
      +    -i $RPM_BUILD_ROOT%{_root_initddir}/%{?scl_prefix}%{phpname}-fpm
      +sed -e '/php-fpm.pid/s:/var:%{_localstatedir}:' \
      +    -e '/subsys/s/php-fpm/%{?scl_prefix}%{phpname}-fpm/' \
      +    -e 's:/etc/sysconfig/php-fpm:%{_sysconfdir}/sysconfig/%{phpname}-fpm:' \
      +    -e 's:/etc/php-fpm.conf:%{_sysconfdir}/%{phpname}-fpm.conf:' \
      +    -e 's:/usr/sbin:%{_sbindir}:' \
      +    -i $RPM_BUILD_ROOT%{_root_initddir}/%{?scl_prefix}%{phpname}-fpm
      
    • Same concept for logrotate configs:

       # LogRotate
      -install -m 755 -d $RPM_BUILD_ROOT%{_origsysconfdir}/logrotate.d
      -install -m 644 %{SOURCE7} $RPM_BUILD_ROOT%{_origsysconfdir}/logrotate.d/php-fpm
      +install -m 755 -d $RPM_BUILD_ROOT%{_root_sysconfdir}/logrotate.d
      +install -m 644 %{SOURCE7} $RPM_BUILD_ROOT%{_root_sysconfdir}/logrotate.d/%{?scl_prefix}%{phpname}-fpm
      +sed -e 's:/var:%{_localstatedir}:' \
      +    -i $RPM_BUILD_ROOT%{_root_sysconfdir}/logrotate.d/%{?scl_prefix}%{phpname}-fpm
      
    • While you can source the enable script to get the SCL binaries in your $PATH, we can also create symlinks with the SCL prefix in system folders. This is ideal if you would only need occasional access to the SCL binary, then you could use php53-php instead of /opt/rh/php53/root/usr/bin/php:

      +# make the cli commands available in standard root for SCL build
      +%if 0%{?scl:1}
      +install -m 755 -d $RPM_BUILD_ROOT%{_root_bindir}
      +ln -s %{_bindir}/php       $RPM_BUILD_ROOT%{_root_bindir}/%{?scl_prefix}php
      +ln -s %{_bindir}/phar.phar $RPM_BUILD_ROOT%{_root_bindir}/%{?scl_prefix}phar
      +%endif
      
    • You have previously seen if 0%{?scl:1} as a way to define an if block if an SCL is being built. For single lines, you can use %{?scl: --item--}, like conditionally adding directories:

      -%{_libdir}/httpd/modules/libphp5.so
      -%{_libdir}/httpd/modules/libphp5-zts.so
      +%{?scl: %dir %{_httpd_moddir}}
      +%{_httpd_moddir}/libphp5.so
      +%{?scl: %{_root_httpd_moddir}/libphp5.so}
      +%{_httpd_moddir}/libphp5-zts.so
      +%{?scl: %{_root_httpd_moddir}/libphp5-zts.so}
      
    • TL;DR: Read through the diff file, but you’re making the SPEC file to be compatible with and without SCL build options. This is great because you don’t have to maintain a second SPEC file for building SCL packages. There are several conditions that if you are using an SCL, you can have files/scripts outside of the SCL root, like init.d scripts, logrotate configs, symlinks to binaries, etc.

  • Apply the diff file as a patch to the AtomiCorp SPEC file. I’m ignoring whitespace because the text editor I used (Sublime Text) removed some spaces for me.

    patch -p0 --ignore-whitespace < php_scl.diff
    
  • Use rpmbuild to build the SRPM. The —define “scl php53” option enables the Software Collection (without it, you’d be building PHP packages that install into system default paths), -bs will only build a source package, and —nodeps will not check for Requires or BuildRequires on the build system (we’ll use mock to build the binary RPMs and it will install required packages into the chroot):

    • CentOS 5: Use MD5 for checksums:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" --define "scl php53" -bs --nodeps php.spec
      
    • CentOS 6:

      rpmbuild --define "scl php53" -bs --nodeps php.spec
      
  • Use mock with the new chroot configurations we created in /etc/mock earlier:

    mock -v -r epel-6-x86_64-scl-php53 php53-php-5.3.26-19.el6.src.rpm
    
  • After several minutes, you should have ready-to-install binary packages in /var/lib/mock/epel-6-x86_64-scl-php53/result/. We’ll need some of them to build other packages, like PEAR, APC, Memcache, etc, so we’ll add them to our local Yum repo we set up earlier (and this location will serve as where we collect all of our completed RPMs):

    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    createrepo -v -d ~/rpmbuild/RPMS/repo/el6/x86_64
    

    At this point, you should have several packages in ~/rpmbuild/RPMS/repo/el6/x86_64. You’ll generally want to copy completed .rpm files out of /var/lib/mock as soon as they’re finished being built so they aren’t destroyed the next time to run mock with the same chroot (option -r).

PEAR

  • Get latest php-pear from Red Hat at http://ftp.redhat.com/redhat/linux/enterprise/6Server/en/os/SRPMS/ and install:

    rpm -ivh php-pear-1.9.4-4.el6.src.rpm
    
  • Grab php-pear_scl.diff to use with patch to convert the SPEC file for use with SCL:

    curl --remote-name --location http://alanthing.com/files/php-pear_scl.diff
    
    • I won’t rehash with the same amount of detail as we covered for php.spec, but as a quick summary, this patch file will:

    • Define the SCL package name:

      +%{?scl:%scl_package php-pear}
      +%{!?scl:%global pkg_name %{name}}
      
    • Add %{scl_prefix} to Name, Requires, BuildRequires, Obsoletes, Conflicts, and Provides. Here’s one example:

      -Name: php-pear
      +Name: %{?scl_prefix}php-pear
      
    • Add %{?scl:Requires: %scl_runtime} so the SCL Base Package is a requirement:

      +%{?scl:Requires: %scl_runtime}
      
    • Some macros need to use the system default paths. Remember, %{_root_*}-style macros are provided by the scl-utils-build package:

      -export PHP_PEAR_SIG_BIN=%{_bindir}/gpg
      +export PHP_PEAR_SIG_BIN=%{_root_bindir}/gpg
      
    • The executable scripts for PEAR need to use the actual rpm macro instead of static paths like “/usr”:

      +for exe in pear pecl peardev; do
      +    sed -e 's:/usr:%{_prefix}:' \
      +        -i $RPM_BUILD_ROOT%{_bindir}/$exe
      +done
      
    • Make symbolic links in the system default directories for the SCL executables:

      +# make the cli commands available in standard root for SCL build
      +%if 0%{?scl:1}
      +install -m 755 -d $RPM_BUILD_ROOT%{_root_bindir}
      +ln -s %{_bindir}/pear      $RPM_BUILD_ROOT%{_root_bindir}/%{scl_prefix}pear
      +ln -s %{_bindir}/pecl      $RPM_BUILD_ROOT%{_root_bindir}/%{scl_prefix}pecl
      +%endif
      
  • Apply the diff file as a patch to the Red Hat SPEC file:

    patch -p0 --ignore-whitespace < php-pear_scl.diff
    
  • Use rpmbuild to build the SRPM:

    • CentOS 5:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" --define "scl php53" -bs --nodeps php-pear.spec
      
    • CentOS 6:

      rpmbuild --define "scl php53" -bs --nodeps php-pear.spec
      
  • Use mock build the packages in our SCL chroot:

    mock -v -r epel-6-x86_64-scl-php53 php53-php-pear-1.9.4-4.el6.src.rpm
    
  • Copy the files to the local Yum repo folder where we’re keeping all of our completed RPMs:

    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    
  • Since APC and Memcache will need php53-php-pear as a build dependency, we’ll need to update the Yum repo files:

    createrepo -v -d ~/rpmbuild/RPMS/repo/el6/x86_64
    

APC

For the remaining packages, I will skip explaining the differences in the SPEC files and simply list the commands. If you look at the diff patch for php-pecl-apc.spec, you’ll notice that it’s very similar to the changes made for PEAR.

  • Get latest php-pecl-apc from Red Hat at http://ftp.redhat.com/redhat/linux/enterprise/6Server/en/os/SRPMS/ and install:

    rpm -ivh php-pecl-apc-3.1.9-2.el6.src.rpm
    
  • Grab php-pecl-apc_scl.diff to use with patch to convert the SPEC file for use with SCL:

    curl --remote-name --location http://alanthing.com/files/php-pecl-apc_scl.diff
    
    • Actually, I will point out an important change in this spec file: adding the directory path in front of php and php-config to ensure we’re using the SCL binary and not the system default php:

      -%global php_zendabiver %((echo 0; php -i 2>/dev/null | sed -n 's/^PHP Extension => //p') | tail -1)
      -%global php_version %((echo 0; php-config --version 2>/dev/null) | tail -1)
      +%global php_zendabiver %((echo 0; %{_bindir}/php -i 2>/dev/null | sed -n 's/^PHP Extension => //p') | tail -1)
      +%global php_version %((echo 0; %{_bindir}/php-config --version 2>/dev/null) | tail -1)
      
  • Apply the diff file as a patch to the Red Hat SPEC file:

    patch -p0 --ignore-whitespace < php-pecl-apc_scl.diff
    
  • Use rpmbuild to build the SRPM:

    • CentOS 5:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" --define "scl php53" -bs --nodeps php-pecl-apc.spec
      
    • CentOS 6:

      rpmbuild --define "scl php53" -bs --nodeps php-pecl-apc.spec
      
  • Use mock build the packages in our SCL chroot:

    mock -v -r epel-6-x86_64-scl-php53 php53-php-pecl-apc-3.1.9-2.el6.src.rpm
    
  • Copy the files to the local Yum repo folder where we’re keeping all of our completed RPMs:

    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    
  • No other packages require the completed RPMs to build, so we skip the createrepo command here and move onto the next page.

Memcache

We’re going to upgrade php-pecl-memcache from 3.0.5 to 3.0.8. Both are considered “beta” on pecl.php.net so it’s not like we’re losing stability, but we are gaining features.

  • Get latest php-pecl-memcache from Red Hat at http://ftp.redhat.com/redhat/linux/enterprise/6Server/en/os/SRPMS/ and install:

    rpm -ivh php-pecl-memcache-3.0.5-4.el6.src.rpm
    
  • Grab php-pecl-memcache_scl.diff to use with patch to convert the SPEC file for use with SCL:

    curl --remote-name --location http://alanthing.com/files/php-pecl-memcache_scl.diff
    
  • Download the memcache 3.0.8 source from the PECL website into the ~/rpmbuild/SOURCES directory

    curl --output ~/rpmbuild/SOURCES/memcache-3.0.8.tgz --location http://pecl.php.net/get/memcache-3.0.8.tgz
    
  • Apply the diff file as a patch to the Red Hat SPEC file, which will upgrade 3.0.5 to 3.0.8 and remove the need for 3 patches:

    patch -p0 --ignore-whitespace < php-pecl-memcache_scl.diff
    
  • Use rpmbuild to build the SRPM:

    • CentOS 5:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" --define "scl php53" -bs --nodeps php-pecl-memcache.spec
      
    • CentOS 6:

      rpmbuild --define "scl php53" -bs --nodeps php-pecl-memcache.spec
      
  • Use mock build the packages in our SCL chroot:

    mock -v -r epel-6-x86_64-scl-php53 php53-php-pecl-memcache-3.0.8-1.el6.src.rpm
    
  • Copy the files to the local Yum repo folder where we’re keeping all of our completed RPMs:

    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64-scl-php53/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    

PHP 5.3 SCL Completed

You should now have several binary RPMs in ~/rpmbuild/RPMS/repo/el6/x86_64 ready to be installed as a Software Collection. If you want to distribute your SRPMs, you’ll probably want to use the ones generated by mock that are in ~/rpmbuild/RPMS/repo/el6/SRPMS.

PHP 5.2

PHP 5.2 is quite old, but if 5.3 or higher broke compatibility with old websites, you can install PHP 5.2 as a Software Collection and use PHP-FPM to use only the older version for a specific website. As previously mentioned, I’m using a SRPM that has backported patches (presumably from https://code.google.com/p/php52-backports/), so it’s certainly more up-to-date than the last 5.2 release from php.net (Released: 06 January 2011).

I won’t go into detail of the diff patches since they are largely similar to the changes made on the PHP 5.3 SPECs.

Software Collection base

  • As with php53, start with http://people.redhat.com/rcollet/php54/rhel/5/SRPMS/php54-1-2.el5.src.rpm

  • Rename or copy php54.spec to php52.spec

  • Set the global variable scl to be php52:

    %global scl php52
    
  • Use rpmbuild to build the SRPM:

    • CentOS 5:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" -bs --nodeps php52.spec
      
    • CentOS 6:

      rpmbuild -bs --nodeps php52.spec
      
  • Build with mock (repeat el6 instructions for el5):

    mock -v -r epel-6-x86_64 php52-1-2.el5.src.rpm
    
  • Copy resulting files to local folder and create a local Yum repo to be used later (repeat el6 instructions for el5):

    mkdir -vp ~/rpmbuild/RPMS/repo/el6/{x86_64,SRPMS}
    cp -va /var/lib/mock/epel-6-x86_64/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    createrepo -v -d ~/rpmbuild/RPMS/repo/el6/x86_64
    

Mock

These steps are nearly identical are for php53, but using php52 instead. As mentioned previously, if you need to repeat something for CentOS 5, use the CentOS 6 and change 6 to 5, el5 to el6, etc.

  • Change to the mock configuration directory:

    cd /etc/mock
    
  • Copy the default EPEL files to new ones specific to our SCL:

    cp -a epel-6-x86_64.cfg epel-6-x86_64-scl-php52.cfg
    
  • Edit epel-6-x86_64-scl-php52.cfg and:

    • Add -scl-php53 to the existing value of config_opts[‘root’]:

      config_opts['root'] = 'epel-6-x86_64-scl-php52'
      
    • CentOS 5 and 6 have different options for config_opts[‘chroot_setup_cmd’]:

      • CentOS 5: Add scl-utils-build php52-build the following to the existing value:

        config_opts['chroot_setup_cmd'] = 'install buildsys-build scl-utils-build php52-build'
        
      • CentOS 6: Change from ‘groupinstall buildsys-build’ to install @buildsys-build scl-utils-build php52-build:

        config_opts['chroot_setup_cmd'] = 'install @buildsys-build scl-utils-build php52-build'
        
  • Add the local repo where you left the files from the section above as a new repo; I add mine above [base]. Change /home/alan to the appropriate path for your system!

    [localrepo-6-x86_64]
    name=localrepo-6-x86_64
    enabled=1
    baseurl=file:///home/alan/rpmbuild/RPMS/repo/el6/x86_64
    

PHP

  • Use latest php-5.2.x SRPM from http://centos.alt.ru/pub/repository/centos/5/SRPMS/

  • Install the SRPM

    rpm -ivh php-5.2.17-29.el5.src.rpm
    
  • Several changes are needed to add the SCL macros to the SPEC file; see the Red Hat documentation. Download my patch file for a full list of changes needed to convert the alt.ru PHP SPEC to one that can be used with and without SCL options: php52_scl.diff

    curl --remote-name --location http://alanthing.com/files/php52_scl.diff
    
  • Apply the diff file as a patch to the alt.ru SPEC file:

    patch -p0 --ignore-whitespace < php52_scl.diff
    
  • Use rpmbuild to build the SRPM:

    • CentOS 5:

      rpmbuild --define "_source_filedigest_algorithm md5" --define "_binary_filedigest_algorithm md5" --define "scl php52" -bs --nodeps php.spec
      
    • CentOS 6:

      rpmbuild --define "scl php52" -bs --nodeps php.spec
      
  • Use mock with the new chroot configurations we created in /etc/mock earlier:

    mock -v -r epel-6-x86_64-scl-php52 php52-php-5.2.17-29.el6.src.rpm
    
  • After mock is completed, copy the files to the local repo folder. Like before, we’ll need some of these packages to build other packages:

    mkdir -vp ~/rpmbuild/RPMS/repo/el6/{SRPMS,x86_64}
    cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
    cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
    createrepo -v -d ~/rpmbuild/RPMS/repo/el6/x86_64
    

PEAR, APC, and Memcache

You can repeat the same steps for php-pear, php-pecl-apc, and php-pecl-memcache as above for PHP 5.3. The only difference would be to change the SCL name during the rpmbuild and mock steps. Here are all of the steps without explanation:

PEAR

1
2
3
4
5
6
7
8
9
10
11
cd ~/rpmbuild/SRPMS
curl --remote-name --location http://ftp.redhat.com/redhat/linux/enterprise/6Server/en/os/SRPMS/php-pear-1.9.4-4.el6.src.rpm
rpm -ivh php-pear-1.9.4-4.el6.src.rpm
cd ~/rpmbuild/SPECS
curl --remote-name --location http://alanthing.com/files/php-pear_scl.diff
patch -p0 --ignore-whitespace < php-pear_scl.diff
rpmbuild --define "scl php52" -bs --nodeps php-pear.spec
mock -v -r epel-6-x86_64-scl-php52 php52-php-pear-1.9.4-4.el6.src.rpm
cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS
createrepo -v -d ~/rpmbuild/RPMS/repo/el6/x86_64

APC

1
2
3
4
5
6
7
8
9
10
cd ~/rpmbuild/SRPMS
curl --remote-name --location http://ftp.redhat.com/redhat/linux/enterprise/6Server/en/os/SRPMS/php-pecl-apc-3.1.9-2.el6.src.rpm
rpm -ivh php-pecl-apc-3.1.9-2.el6.src.rpm
cd ~/rpmbuild/SPECS
curl --remote-name --location http://alanthing.com/files/php-pecl-apc_scl.diff
patch -p0 --ignore-whitespace < php-pecl-apc_scl.diff
rpmbuild --define "scl php52" -bs --nodeps php-pecl-apc.spec
mock -v -r epel-6-x86_64-scl-php52 php52-php-pecl-apc-3.1.9-2.el6.src.rpm
cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS

Memcache

1
2
3
4
5
6
7
8
9
10
11
cd ~/rpmbuild/SRPMS
curl --remote-name --location http://ftp.redhat.com/redhat/linux/enterprise/6Server/en/os/SRPMS/php-pecl-memcache-3.0.5-4.el6.src.rpm
rpm -ivh php-pecl-memcache-3.0.5-4.el6.src.rpm
cd ~/rpmbuild/SPECS
curl --remote-name --location http://alanthing.com/files/php-pecl-memcache_scl.diff
curl --output ~/rpmbuild/SOURCES/memcache-3.0.8.tgz --location http://pecl.php.net/get/memcache-3.0.8.tgz
patch -p0 --ignore-whitespace < php-pecl-memcache_scl.diff
rpmbuild --define "scl php52" -bs --nodeps php-pecl-memcache.spec
mock -v -r epel-6-x86_64-scl-php52 php52-php-pecl-memcache-3.0.8-1.el6.src.rpm
cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.{noarch,x86_64}.rpm ~/rpmbuild/RPMS/repo/el6/x86_64
cp -va /var/lib/mock/epel-6-x86_64-scl-php52/result/*.src.rpm ~/rpmbuild/RPMS/repo/el6/SRPMS

Conclusion

It does seem like a little bit of work to convert a SPEC file for SCL compatibility, but once the work is done, incremental updates are easy, and you can use the same SPEC to build a non-SCL package as well. I wouldn’t be surprised if Red Hat begins to create SCL-compatible SPEC files for major applications in RHEL 7 and beyond because of the flexibility it offers, just see their Red Hat Software Collections 1.0 Beta. Unfortunately, it’s difficult to fully grasp how to build one, so hopefully this guide has been helpful.

Personally, after having some trouble quickly switching from MySQL to MariaDB or Percona, I’ll probably look to rebuild those as Software Collections instead of replacing the default packages, but that’s another blog post!

Speed Up PHP on NFS With Turbo_realpath

This post originally featured on the Echo & Co. blog.

If you run a website based on PHP, and have your source files on a network file system like NFS, OCFS2, or GlusterFS, and combine it with PHP’s open_basedir protection, you’ll quickly notice that the performance will degrade substantially. Normally, PHP can cache various path locations it learns after processing include_once and require_once calls via the realpath_cache. There’s a bug in PHP that effectively disables the realpath_cache entirely when combined with open_basedir. Popular PHP applications with Drupal and WordPress make heavy use of these functions to include other files, so you would very quickly notice the drop in performance in this scenario. If you want to isolate your websites from each other (or from the rest of the operating system), how can you retain any shred of performance?

This is where Artur Graniszewski’s turbo_realpath extension really comes in handy. I won’t retype his installation instructions, so follow the previous link to get it installed manually.

If you’re running CentOS 5 or CentOS 6, check out yum.echoditto.com and you’ll find source and compiled RPMs that will install alongside the RedHat/CentOS-supplied PHP packages. The RPM will create a basic configuration file at /etc/php.d/turbo_realpath.ini. Essentially, it enables the PHP module but defaults all settings off, so you will need to read the comments (taken from Artur’s most recent post on turbo_realpath) to determine how you want to use it.

Configuration

We frequently use turbo_realpath on a per-VirtualHost basis with Apache 2.2 and mod_php. If you use PHP-FPM, you can apply similar settings in your FPM pool configuration files. If you install our RPM and don’t edit /etc/php.d/turbo_realpath.ini, add something similar to the following to each VirtualHost:

1
2
3
<IfModule php5_module>
  php_admin_value realpath_cache_basedir "/var/www/vhosts/domain.com:/usr/share/pear:/usr/share/php:/usr/lib64/php:/usr/lib/php:/tmp:/var/tmp"
</IfModule>

This is effectively the same using open_basedir; any directories referenced in realpath_cache_basedir will be the only ones the website is allowed to access, and they will be cached as determined by the realpath_cache_size and realpath_cache_ttl. If you look in php.ini, you may notice the default values for these are:

1
2
3
4
5
6
7
; Determines the size of the realpath cache to be used by PHP. This value should
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-size
realpath_cache_size = 16k

; Duration of time, in seconds for which to cache realpath information for a given
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-ttl
realpath_cache_ttl = 120

You may want to increase these if you’re finding your website is still not loading quickly. On our systems, we have bumped the realpath_cache_size and realpath_cache_ttl settings up to 1m and 300, respectively.

Speed and Security!

With turbo_realpath enabled, realpath_cache_basedir set to appropriate open_basedir-like values, and realpath_cache_size and realpath_cache_ttl increased from defaults, we’re able to have isolated PHP sites and have better performance by caching the locations of included/required files effectively. Hopefully, our RPMs will help you on your system for a quick installation of the excellent turbo_realpath module!

References

VirtualBox Extension Pack With One Line

Just installed VirtualBox on OS X and need the Extension Pack? This (really long) one-liner will download and install it for you. Run in terminal and enter your password once and you’re off to the races!

1
export version=$(/usr/bin/vboxmanage -v) && export var1=$(echo ${version} | cut -d 'r' -f 1) && export var2=$(echo ${version} | cut -d 'r' -f 2) && export file="Oracle_VM_VirtualBox_Extension_Pack-${var1}-${var2}.vbox-extpack" && curl --silent --location http://download.virtualbox.org/virtualbox/${var1}/${file} -o ~/Downloads/${file} && VBoxManage extpack install ~/Downloads/${file} --replace && rm ~/Downloads/${file} && unset version var1 var2 file

Note that VirtualBox must be installed first :)