Using ManifestStaticFilesStorage with Django Storages and Amazon S3

Many of the Django websites I’ve been working on lately are being deployed to Amazon Elastic Compute Cloud (EC2) servers and utilize Amazon Simple Storage Service (S3) for hosting static files.

The simplest method I’ve found for integrating Amazon S3 for hosting static media inside a Django project is to use the django-storages project to handle creating the necessary storage backend. I setup the static file storage backend to use the S3Botostorage backend via settings:

And the next time I run Collectstatic all of my static assets are pushed to the configured S3 account and static files served from there when I use the static templatetag.

For the most part this system runs smoothly but I have run into a couple of common issues with each project:

Static files are loaded into the root of the S3 Bucket.

Most of my clients will use the same bucket for static assets as well as user uploads. So, how do we get the static files into a key prefix (the S3 version of a directory) on Amazon S3? For that I make use of the AWS LOCATION  settings tag made available by django-storages S3 backend.

With that settings variable in place set all of my static files will go into the key prefix I indicated when I run Collectstatic. So, for example, if I set AWS_LOCATION = 'static'  in my Django settings then all of my static files will now be available in https://mybucket.s3.amazonaws.com/static/

Using Django’s ManifestStaticFilesStorage along with storages.backends.s3boto.S3BotoStorage.

I always configure storages with ‘Expires’ and ‘Cache-Control’ headers set well into the future so that browsers know to cache the media which brings improved performance to the site as static files don’t have to be fetched on every page load.

However, I do want to be able to take advantage of Django’s ManifestStaticFilesStorage backend so that I can have Collectstatic “bust the cache” for my static files by renaming them when uploading new or updated files to S3. The problem I first ran into was that I’m already calling the S3BotoStorage backend with django-storages. I can’t define two separate storage backends!

Thankfully, the community of developers working on Django were wise enough to create a ManifestFilesMixin that we can utilize with S3BotoStorage and accomplish what we need. To get this working we’ll need to create a custom storage backend that brings these two classes together.

I create file named CustomS3BotoStorage.py (you can call it whatever you’d like). Inside that file I create a simple class that combines the ManifestFilesMixin and the S3BotoStorage class:

Then, I set the STATICFILES_STORAGE setting to point to my custom storage class:

With this custom class I can now have the benefits of django-storages with the cache busting of ManifestStaticFilesStorage.

Custom authentication scheme for Django Rest Framework

I’m developing a native iOS app that needs to interface with an existing Django web project via a RESTful API. For this particular project I’ve chosen to use the well-maintained Django Rest Framework to handle creating the API.

I’ve decided to use OAuth2 authentication between the iOS app and the Django project and I’ve set everything up following the Django Rest Framework documentation for using OAuth2 authentication. Followed the simple CURL test and I’m able to communicate with all of my endpoints successfully.

So, what’s the problem?

Let me give a little background to help you understand how I arrived at the need for a custom authentication scheme.

Restricting the API to Authenticated Users

As the title indicates, this project requires user authentication to access the API regardless of the method (GET, POST, PUT, etc). Django Rest Framework makes this easy enough by providing a permissions class called IsAuthenticated which will deny permission to any unauthenticated user, and allow permission otherwise.

I implemented this in the settings files per the documentation to override the default of allowing anyone to view the API:

With the default authentication class of  'rest_framework.authentication.SessionAuthentication' and the permission class above I have now restricted my API to authenticated users.

Adding OAuth2 API Access

To get OAuth2 authentication up and running I configured the DEFAULT_AUTHENTICATION_CLASSES  in my settings file as indicated in the documentation. Here’s the recommended settings configuration for OAuth2 with my permission class from above:

 Configure Super User Access to the Web Browsable API

Now we’ve arrived at the problem. Our developers can’t access the web browsable API because we’re requiring authentication and the only authentication scheme available is OAuth2.

As a developer, the web browsable API is one of my favorite parts of the Django Rest Framework project. The web browsable API makes it much easier to work with, visualize, and debug our API endpoints.

Adding  'rest_framework.authentication.SessionAuthentication' back to the  'DEFAULT_AUTHENTICATION_CLASSES' tuple would certainly work. However, it would also make the web browsable API available to any authenticated user which we don’t want.

My solution was to write a custom authentication scheme for the Django Rest Framework that restricted session authentication to only allow superusers to view the web browsable API. In order to do so I decided to extend the  'rest_framework.authentication.SessionAuthentication' class and overwrite the authentication method.

I created a file called authentication.py  in my projects  Accounts app which we use to give other custom functionality to users. Of course, you could name and place this file wherever it best suits your project.

Here is my entire custom authentication.py file:

I implemented it into my settings file as I did the other authentication classes:

There you have it. All authenticated users have access to the API via OAuth2 Authentication and only super users are able to view the web browsable API.

Migrate URL Tags to Django 1.5

Prior to Django 1.3, the syntax for the url tag was {% url app-view %}. Django considered the first variable in the URL tag to be a special case unquoted constant instead of a context variable named app-view.

In Django 1.5 this functionality was reversed so that an unquoted first argument to the URL tag is considered a context variable. So, when I recently upgraded Brite Revolution, Brite Me, and Brite for Brands to Django 1.5 I received a Deprecation Warning about my legacy use of the URL template tag. I needed to convert all of my existing URL tags (over 200 instances) to use quoted literals as the first argument.

Thankfully, PyCharm 3’s “Replace in Path” with a regular expression made quick work of this for me:

PyCharm 3 "Replace in Path" example
PyCharm 3 “Replace in Path” example

More info on the URL template tag can be found in the Django docs.

Get the latest tweet from your Twitter account using Twython and Django

Twython is the premier Python library providing an easy (and up-to-date) way to access Twitter data. Actively maintained and featuring support for Python 2.6+ and Python 3. It’s been battle tested by companies, educational institutions and individuals alike.

At Brite Revolution we were using version 1.0 of the Twitter REST API to pull in the latest tweet from the public timeline of our company twitter account (@briterevolution) to post in the footer of our site.

We had everything working great using python-twitter until Twitter deprecated v1.0 of their REST API. This forced all users to migrate to v1.1 which in turn required authentication on every API endpoint. This includes the public timeline API endpoints which had not required authentication in previous versions. What to do?

My initial thought was to upgrade our python-twitter library to start working with version 1.1 of the new API. To my dismay, as of the time of this post python-twitter still hadn’t released an official update that supports version 1.1 and I’m not keen on deploying beta software into a live production environment. Instead, I turned to the official Twitter API docs and began looking through the recommended Python libraries for a solution. Enter Twython.

Twython

A few new lines in Django settings (we’ll get to that) and Twython made quick work out of grabbing the latest tweet from our public timeline. I’d encourage you to check out Twython when considering a Twitter library for your next project.

Getting the latest tweets with Twython

This tutorial assumes you have a basic Django project up and running. Let’s get started.

1. Install Twython and any dependencies

2. Create your Twitter application

Head over to https://dev.twitter.com/apps and register your Twitter application!

In our particular use case we’re only using this Twitter application for our Twitter account. We still have to use Oauth to authenticate our own Twitter account. The good news is that Twitter makes this really easy. Once you have your Twitter app created scroll down to the bottom of the details section. There you’ll see a “Create my access token” button…click it. This will generate the access token and access token secret necessary for you to properly authenticate with the Twitter REST API.

3. Configure Django settings

We need to tell Twython what our Oauth settings are. There are several ways to do this with some methods being more secure than others. For the sake of simplicity we’re going to put our Oauth credentials into our projects settings.py file.

4. Custom context processor

We’re going to write a custom context processor for our Django project so that our latest tweet is added to the template context.

Some helpful hints for the code above

  1. As of this writing you can make 180 requests per 15 minute window to the REST API while other API calls have different rate limiting. Since our recent tweet loads on every page we don’t want to slow down each page load while waiting for the Twitter API to respond. We chose to set our cache timeout at 10 minutes (600 seconds). You can modify this via the setting for TWITTER_CACHE_TIMEOUT.
  2. I prefer the Tweet parser included with twitter-text-python. I installed it into my projects utils folder as twitter_parser.py.

Put this code into a context_processors.py file in the root of your project. Make sure to add the new context_processors.py file into your settings files so Django knows about it..

5. Template

We’re ready to load our latest tweet into our template. By using a context processor we don’t need to do anything additional with our views to get access to the latest tweet. We just need to use the variable name we created in the context processor (“latest_tweet”).