django-compressor

django-compressor docs

Warning: much of the documentation is casual about saying things that are only true in some scenarios, without making clear that that’s the case.

ACTUALLY USING

Here are some practical scenarios for using django-compressor.

For what to put in your templates, you can go by the django-compressor documentation, and be sure to use {% static %} and not STATIC_URL.

For what to put in your settings… it’s a lot more complicated. Set the compressor filters and precompilers however you want. For the rest, keep reading.

Scenario: Development using runserver, DEBUG, not offline

If DEBUG is True, then compressor won’t even do anything and so everything should just work.

Scenario: Running using local files, not offline

This is the typical small server situation. You unpack your project on the server, run collectstatic, point nginx or some other server at STATIC_ROOT and go.

Example settings:

# Django settings
DEBUG = False
STATIC_ROOT = '/var/www/static/'
STATIC_URL = '/static/'
# set compressor filters and precompilers as desired.
# leave other compressor settings at defaults.

# nginx settings
location /static {
   alias /var/www/static;
}

Scenario: running using local files, with offline

Like the previous scenario, but you want compressor to do all its work at deploy time so the results are cached and ready to go immediately when you start your server.

# Django settings like before, plus:
COMPRESS_OFFLINE = True

Now at deploy time you have more steps:

$ python manage.py collectstatic
$ python manage.py compress

Run compress after collectstatic so that compressor can find its input files. It’ll write its output files under {STATIC_ROOT}/CACHE, and get them from there at runtime.

Scenario: running with storage on the network, with offline

In this scenario, you’re putting your static files somewhere off of the server where you’re running Django. For example, S3. Or just your own static file server somewhere. Whatever.

Let’s start with how this would be setup without django-compressor, then we can modify it to add django-compressor.

# settings/no_compressor.py
STATIC_ROOT = None  # Unused
STATIC_URL = None  # Unused
STATIC_FILE_STORAGE = 'package.of.FileStorageClass'

At deploy time you can just run collectstatic, and all your static files will be pushed to the network:

$ python manage.py collectstatic

And at runtime, {% static %} will ask your file storage class to come up with a URL for each file, which will turn out to be on your other server, or S3, or whatever.

Now, suppose we want to add compressor with offline processing (not using offline makes no sense with network storage). Here are the settings you can use at runtime for that, assuming things have been prepared correctly:

# settings/deployed.py
# Django settings we'll use in production
STATIC_ROOT = None  # Unused
STATIC_URL = None  # Unused
STATIC_FILE_STORAGE = 'path.to.network.filestorage'
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True

The preparation is the tricky part. It turns out that for the compress command to work, a copy of the static files must be gathered in a local directory first. Most of the tools we might use to compile, compress, etc. are going to read local files and write local output.

To gather the static files into a local directory, we might, for example, use a different settings file that uses the default file storage class, and run collectstatic. E.g.:

# settings/gather.py
# Django settings when first running collectstatic
from .deployed import *

# Override a few settings to make storage local
STATIC_ROOT = '/path/to/tmp/dir'
STATIC_URL = None  # Unused
STATIC_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'

$ python manage.py collectstatic --settings=settings.gather

After running collectstatic with these settings, all your source static files will be gathered under ‘/path/to/tmp/dir’.

Now you could run compress, and the resulting files would be added under /path/to/tmp/dir. There’s an important gotcha that will cause problems, though - for compressor to match up the output it makes now with what it’ll be looking for later, the contents of each {% compress %} tag must be identical now to what it’ll be then, which means the URLs must point at the production file server. We can accomplish this by setting STATIC_URL before running the compress:

# settings/compress.py
# Django settings when running compress command
from .deployed import *

# Override a few settings to make storage local, but URLs look remote
STATIC_ROOT = '/path/to/tmp/dir'
STATIC_URL = 'https://something.s3.somewhere/static/'  # URL prefix for runtime
STATIC_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'

$ python manage.py compress --settings=settings.compress

The problem now is to get all these files onto the remote server. You could just use rsync or s3cmd or something, which will work fine. But for maximum flexibility, let’s figure out a way to do it using Django. Our approach will be to tell Django that our SOURCE static files are in ‘/path/to/tmp/dir’, and we want them collected using our production file storage class, which will put them where we want them.

# Django settings when running collectstatic again after compress,
# to copy the resulting files to the network
# settings/copy_upstream.py
from .deployed import *  # Set up for network file storage
# Tell collectstatic to use the files we collected and compressed
STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder']
STATICFILES_DIRS = ['/path/to/tmp/dir']

$ python manage.py collectstatic --settings=settings.copy_upstream

That should copy things to the network. Then if you run using the ‘deployed’ settings, things should work!

TODO: TEST THAT!!!!!!!!!!!!!!!!!!!!

Other approaches

The compressor docs suggest a different approach – hack the storage class you’re using so when you run collectstatic, it saves a copy of each file into a local directory in addition to pushing it upstream. Then you can use the same storage class for collectstatic, compress, and runtime.

More detailed notes

Cache

For some things, compressor uses the cache named by COMPRESS_CACHE_BACKEND, which defaults to None, which gives us the default Django cache.

Principles of compression

Whether compressor is processing templates offline ahead of time or at runtime, there are some common principles.

First, if COMPRESS_ENABLED is False, the {% compress %} tag will simply render as its contents; compressor won’t change anything.

Otherwise, compressor will

  1. parse the contents of the tag and figure out which css and javascript files would be included

  2. fetch those files (See “accessing the files to be compressed”)

  3. run those files through any configured preprocessors

  4. concatenate the result and save it using COMPRESS_STORAGE

  5. at rendering, the tag and contents will be replaced with one or two HTML elements that will load the compressed file instead of the original ones.

Offline

If COMPRESS_OFFLINE is True, compressor expects all uses of {% compress ... %} in templates to have been pre-processed by running manage.py compress ahead of time, which puts the results in compressor’s offline cache. If anything it needs at run-time is not found there, things break/throw errors/render wrong etc.

Note

If COMPRESS_OFFLINE is True and files have not been pre-compressed, compressor will not compress them at runtime. Things will break.

The offline cache manifest is a json file, stored using COMPRESS_STORAGE, in the subdirectory COMPRESS_OUTPUT_DIR (default: CACHE), using the filename COMPRESS_OFFLINE_MANIFEST (default: manifest.json).

The keys in the offline cache manifest are generated from the template content inside each compress tag, not the contents of the compressed files. So, you must arrange to re-run the offline compression anytime your content files might have changed, or it’ll be serving up compressed files generated from the old file contents.

Note

It sounds like you must also be sure the contents of the compress tags don’t change between precompressing and runtime, for example by changing the URL prefix!

The values in the offline cache manifest are paths of the compressed files in COMPRESS_STORAGE.

Note

RECOMMENDATION FROM DOCS: make COMPRESS_OFFLINE_MANIFEST change depending on the current code revision, so that during deploys, servers running different versions of the code will each use the manifest appropriate for the version of the code they’re running. Otherwise, servers might use the wrong manifest and strange things could happen.

Not offline

If COMPRESS_OFFLINE is False, compressor will look in COMPRESS_STORAGE for previously processed results, but if not found, will create them on the fly and save them to use again.

Storage

Compressor uses a Django storage class for some of its operations, controlled by the setting COMPRESS_STORAGE.

The default storage class is compressor.storage.CompressorFileStorage, which is a subclass of the standard filesystem storage class. It uses COMPRESS_ROOT as the base directory in the local filesystem to store files in, and builds URLs by prefixing file paths within the storage with COMPRESS_URL.

If you change COMPRESS_STORAGE, then ignore anything in the docs about COMPRESS_ROOT and COMPRESS_URL as they won’t apply anymore (except in a few cases… see exceptions noted as they come up, below).

Accessing the files to be compressed

For each file to be compressed, compressor starts with the URL from the rendered original content inside the compress tag. For example, if part of the content is <script src="http://example.com/foo.js"></script>, then it extracts "http://example.com/foo.js" as the URL.

It checks that the URL starts with COMPRESS_STORAGE’s base_url, or if accessing that fails (quite possible since base_url is not a standard part of the file storage class API), uses COMPRESS_URL.

Note

This is a place where compressor can use COMPRESS_URL even if it’s not using its default storage.

If the URL doesn’t start with that string, compressor throws a possibly misleading error, “’%s’ isn’t accessible via COMPRESS_URL (‘%s’) and can’t be compressed”.

Otherwise, compressor tries to come up with a local filepath to access the file, as follows:

  • Try to get a local filepath from COMPRESS_STORAGE using .path().

  • If that’s not implemented (for example, for remote storages), it tries again using compressor.storage.CompressorFileStorage (regardless of what COMPRESS_STORAGE is set to), so basically it’s going to look for it under COMPRESS_ROOT.

  • If it still can’t get a local filepath, throws an error: “’%s’ could not be found in the COMPRESS_ROOT ‘%s’%s” which is very misleading if you’re not using a storage class that looks at COMPRESS_ROOT.