All the housekeeping: migrating to Django 2, Python 3 and Ubuntu 18

Python 2.7’s end-of-life countdown clock has been ticking for 18 months and I’ve spent most of that time ignoring the warnings. Likewise Django 2 has been out for quite a while; Django 1.11 will be EOL in April. I’ve got a lot of python/Django projects in production and the idea of finding time to upgrade them all didn’t seem like a particularly realistic one, but this week I found myself at home and between gigs, so I thought I’d better make a start.

In my case there was a bit of a dependency cascade. I wanted to update to Django 2.2, which requires python 3.6. But my development box (and the production server for this project) were running the now-ancient Ubuntu 14.04, which doesn’t bundle an up-to-date version of python 3. So I would need to upgrade those as well. And to further complicate the Django side of things, while I trusted that most of the libraries I use would work with Django 2.x, I knew I was relying on a couple of Django extensions I’d written myself, which would probably also need to be updated for this brave modern world.

The first thing was to upgrade Ubuntu on my local dev VM. I’d originally set this up with Vagrant and salt states to provide a reproducible provisioning system, but you know how it goes. I got too fond of the machine and all the data accumulating within it, didn’t keep it ephemeral and didn’t keep it up-to-date. Fortunately Ubuntu’s do-release-upgrade works well and I was able to use this (twice!) to upgrade first from 14.04 to 16.04 and then to 18.04.

That accomplished, it was time to upgrade to python 3.6. Upgrading Ubuntu had broken my virtualenvs (due to the different python versions available) but I needed to recreate them in any case, this time specifying python 3. This was a case of backing up my postactivate scripts before running (these commands are provided by virtualenvwrapper):

$ rmvirtualenv sitename
$ mkvirtualenv -p python3 sitename

I know python 3 comes with its own virtualenv implementation, venv, but for now I’m happy to stick with virtualenv, as I already know its ins and outs.

Having reinstated the postactivate script for the virtualenv, it was time to reinstall my requirements (as frozen in the Django 1.11 world), so I could deal with the python migration before updating Django:

$ python setup.py develop
$ pip install -r requirements.txt

I was then able to run the project and check for python errors. Django’s docs have some great info on the likely issues that surface when moving to python 3. There were no problems in the project’s first-party code but I found I needed to update a couple of Django extensions that I’d written myself – django-simpleinliner and django-admin-thumbnails – which were relying on python2 APIs like dict.iteritems() and str(). I found the six compatibility layer really helpful in working around these to provide code that would run in both python 2 and python 3 environments.

With that done it was time to update Django and any other outdated requirements in the project – pip-review is handy for this (it used to be part of pip-tools but has since been forked into its own package):

$ pip-review --interactive

That pulled down the latest version of Django (2.2.6) along with other library updates that had been held back due to not being compatible with python 2.7. Of note was that the legacy MySQL adapter I’d been using, MySQL-python, is now obsolete and has been replaced with mysqlclient. I then needed to try running the project, going through the code correcting references to deprecated or updated APIs. I found this blog post very useful, and found many of the same common issues:

• Rewrite urlpatterns to use path() and new regex syntax
• Add app_name directive to all included urls.py files
• Change all class __unicode__ magic methods to __str__
• Update all references to reverse: change from django.core.urlresolvers import reverse to from django.urls import reverse

I had to change a couple of database settings to reflect changes between MySQL 5.5 and 5.7. I also needed to update the project’s CircleCI config to use their python3.6 docker images, as well as the latest MySQL version. I had to update the virtualenv creation command to use python 3 (-p python3). And of course there were a few schema changes which Django took care of when I ran:

$ django-admin migrate

With the project successfully running locally it was time to look at deployment. Rather than upgrade the production server I decided to spin up a new one with the latest Ubuntu 18.04 image (all very easy on Digital Ocean [affiliate link]) and set things up from scratch. Because I don’t maintain that many servers, I usually use a setup script of my own design rather than getting involved with a state management or reproducible deployment system. The script is open source and I updated it for Ubuntu 18.04 as part of this exercise.

The main change on the deployment side of things was Ubuntu’s move to systemd replacing init.d, which meant I needed to overhaul the way the project’s gunicorn service is managed. Digital Ocean have a great tutorial on setting up Django with nginx and gunicorn on Ubuntu 18.04, and the gunicorn section was exactly what I needed. After I’d updating my gunicorn and nginx configs and copied the project database from the old server, I was able to get the site running on the new one.

The only other thing was tweaking my process for using Let’s Encrypt’s certbot, which is now a fairly sophisticated Ubuntu package. The official certbot instructions are really handy and with a bit of tweaking I was able to get certificate generation working on the new server.

Having followed this tangled thread all the way to its conclusion, I’m glad I took the time to do it now at relative leisure rather than in a panic after the python 2.7 EOL in January. I have several other projects to migrate but now I’ve established the process it should be pretty straightforward. Getting with the times!