alanthing

Linux and other things

Python 3 and uWSGI on CentOS 7

Wherever I can, I avoid compiling applications from source or packaging up my own versions of things that exist in popular repositories. I recently needed to run a Python 3 application via uWSGI on CentOS 7 and was frustrated at how many search results were recommending building uWSGI from source when it’s available in EPEL. I’ve only deployed a handful of Python applications, and was not interested in keeping up with source-compiled dependencies to support this small project over time. We’ll discuss a couple of different ways to run these programs together with minimal customization.

Python 3.4

In CentOS 7, you can run Python 3.4 alongside Python 2.7 as it is available in EPEL.

1
2
3
rpm -q epel-release || yum -q -y install epel-release

yum -y install python34

pip

If you want to install pip, simply yum install python34-pip and it will install at /usr/bin/pip3 and /usr/bin/pip3.4, leaving /usr/bin/pip alone as that is part of the python2-pip package. This was only available recently (late 2016); previously you had to manually install and deal with /usr/bin/pip being overwritten.

Virtual Environments

Since 3.3, Python supports creating virtual environments without an external module like virtualenv. pyvenv was the “recommended tool” for this initially, though it is deprecated in Python 3.6. As such, to lean into the preferred Python nomenclature, you can use the venv module directly to create Python 3.4 virtual environments:

1
2
3
python3.4 -m venv /path/to/venv34dirname

source /path/to/venv34dirname/bin/activate

If you like using virtualenv, you’ll be disappointed to see that there is no python34-virtualenv package available in EPEL. While you could pip3 install virtualenv, that would install at /usr/bin/virtualenv, potentially overwriting the same file provided by the python-virtualenv package. But, if you do, consider the following:

1
2
3
4
5
6
7
8
9
10
11
# Install virtualenv with the pip provided the 'python34' package:
pip3 install virtualenv

# Then, a virtualenv based on /usr/bin/python3.4:
virtualenv /path/to/venv34dirname

# Now, to create based on /usr/bin/python2.7, using a newer version of virtualenv than is provided by the 'python-virtualenv' package:
virtualenv --python=/usr/bin/python2.7 /path/to/venv27dirname

# Or, recommended, use the alternate path from the 'python-virtualenv' package:
virtualenv-2.7 /path/to/venv27dirname

Unless you have a compelling reason for virtualenv on your servers, use the built-in venv module. I don’t like to overwrite files provided by packages, and we should use recommended upstream tools where possible.

uWSGI

Since uWSGI is provided by EPEL, there is no need to install it with pip.

1
yum -y install uwsgi uwsgi-plugin-python3

This will install the Python 3.4 uWSGI plugin to /usr/lib64/uwsgi/python3_plugin.so. Be sure to include the directive plugin = python3 in your uWSGI definition and proceed as normal (beyond the scope of this post).

Python 3.5 Software Collection

I love Software Collections. With Python 2.7 being the default version of Python with CentOS 7, I find it much better to keep another version in a completely separate directory structure than integrating with /usr and such. We’ll use a rebuild of the Red Hat SCL for Python 3.5, and everything will be installed to /opt/rh/rh-python35.

1
2
3
yum -y install centos-release-scl

yum -y install rh-python35-python

Using

You’ll quickly notice that the newly-installed Python 3.5 binary is not in your $PATH; it’s in /opt/rh/rh-python35/root/bin. You have a few options:

Launch a new instance of bash:

1
scl enable rh-python35 bash

Alternatively, you can modify your existing session by running one of the following manually, or add to ~/.bashrc:

1
2
3
4
5
# Preferred method:
source scl_source enable rh-python35

# Source the file directly:
#source /opt/rh/rh-python35/enable

pip

1
yum -y install rh-python35-python-pip

After installing, you can use pip, pip3, or pip3.5

Virtual Environments

You can use the built-in method:

1
python3.5 -m venv /path/to/venv35dirname

Or, you can use virtualenv (though, see the 3.4 section for the rationale I recommend against this):

1
2
3
yum -y install rh-python35-python-virtualenv

virtualenv-3.5 /path/to/venv35dirname

uWSGI

While there is a uwsgi-plugin-python3 RPM, it’s for the aforementioned python34 package. This is where you will have to compile something manually; so carefully consider the pros and cons of using a Software Collection. (Later, I may add to this post where you can use a yum post-install or post-upgrade action to ensure the compiled plugin is kept up to date.)

This is a helpful exercise to learn how you can integrate packages that install to default paths and a Software Collection which is kept away from default paths.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
yum -y install uwsgi uwsgi-plugin-common

UWSGI_VERSION="$( rpm -q uwsgi --queryformat '%{VERSION}' )"
UWSGI_PLUGINS_DIR="$( dirname "$( rpm -q uwsgi-plugin-common -l | grep '_plugin\.so' | tail -1 )" )"

source scl_source enable rh-python35

if [[ $X_SCLS == *rh-python35* ]]; then
  UWSGI_PLUGINS_DIR="/opt/rh/rh-python35/root${UWSGI_PLUGINS_DIR:?}"
fi

if [[ ! -f "${UWSGI_PLUGINS_DIR:?}/python35_plugin.so" ]]; then
  yum -y install gcc libcap-devel libuuid-devel make openssl-devel rh-python35-python-devel

  mkdir -pv /opt/rh/rh-python35/root/src
  cd /opt/rh/rh-python35/root/src

  curl -LO "https://projects.unbit.it/downloads/uwsgi-${UWSGI_VERSION:?}.tar.gz"
  tar zxvf "uwsgi-${UWSGI_VERSION:?}.tar.gz"
  cd "uwsgi-${UWSGI_VERSION:?}/"

  make PROFILE=nolang
  PYTHON=python3.5 /usr/sbin/uwsgi --build-plugin "plugins/python python35"

  [[ ! -d "${UWSGI_PLUGINS_DIR:?}/" ]] && mkdir -pv "${UWSGI_PLUGINS_DIR:?}/"
  mv -v python35_plugin.so "${UWSGI_PLUGINS_DIR:?}/"
fi

(Reference: http://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html)

After completing, your resulting uWSGI file will need to contain:

1
2
plugins-dir = /opt/rh/rh-python35/root/usr/lib64/uwsgi
plugin = python35

Which should you use?

If you need Python 3 but want to reduce the amount of customization that you need to do, I recommend installing python34 from EPEL. You can use it with uWSGI with no compiling from source.

If Python 3.4 is too old and/or you need 3.5+ functionality, the Software Collection can be an alternative to compiling Python from source and knowing that you’re getting quality packages designed by Red Hat and rebuilt by CentOS.

If you need Python 3.6, or a newer release of 3.4 or 3.5 not provided quickly enough by the package maintainers, and you don’t want to deal with all of the pain of ./configure && make && sudo make install, I would suggest looking at pyenv to meet your needs.

Please let me know if I missed something or should add anything. I’m cumulatively green with running Python in production but wanted to share my learnings as I was frustrated by the lack of good information for my needs. Feedback is welcome.

Comments