Once you have a working website or reusable app, you will want to make it public. Deploying websites is one of the most difficult activities of development with Django, because there are lots of moving parts that you have to tackle:
Managing the web server
Configuring the database Serving static and media files
Processing the Django project
Setting up email sending
Arranging background tasks and cron jobs
Setting up continuous integration
Other tasks, depending on your project's scale and complexity
You have developed and tested a fully functional web application in Django. Deploying this application can involve a diverse set of activities from choosing your hosting provider to performing installations.
A production environment is simply one where end-users use your application. It should be available, resilient, secure, responsive, and must have abundant capacity for current (and future) needs.
Professional websites usually have development, staging, and production environments. Each of them has a specific purpose. Development environments are used for creating the project. The production environment is the server (or servers) on which your public website is hosted. The staging environment is a system technically analogous to production but is used to check the new features and optimizations before publishing them.
Choosing a web stack
By a web stack, we refer to the set of technologies that are used to build a web application. It is usually displayed as a series of components, such as OS, database, and web server, all piled on top of one another. Hence, it is referred to as a stack.
Components of a stack
While constructing your web stack, some of the choices you might need to make are as follows:
Which OS and distribution? For example, Debian, Red Hat, or OpenBSD.
Which WSGI server? For example, Gunicorn or uWSGI.
Which webserver? For example, Apache or Nginx.
Which database? For example, PostgreSQL, MySQL, or Redis.
Which caching system? For example, Memcached or Redis.
Which process control and monitoring system? For example, Upstart,
Systemd, or Supervisord.
How to store static media? For example, Amazon S3 or CloudFront
How you choose to host your Django application is one of the key factors in determining your production setup.
Virtual machines or Docker
Virtual machines isolate your application (guest machine) from the underlying infrastructure (host machine). Container technologies such as Docker are increasingly being used for cloud deployments, either complementing or replacing virtual machines.
Containers are a means to create multiple user-space instances over the same kernel. Unlike virtual machines, containers avoid the need to start and run separate guest operating systems. Typically, each container packages an application and its dependencies in a user-space instance separate from other containers. Unlike virtual machines, they do not have a separate instance of the operating system, making them lighter, and faster to start or stop.
Docker has become the containerization technology of choice with a large ecosystem and wide support among cloud vendors. Docker images are created from a
binary image called base image or automatically built from a script called a Dockerfile. This helps you recreate the same environment in production for
development or testing purposes, thus ending the infamous excuse but it worked in my machine.
The most common design pattern using Docker is breaking down applications and services into microservices. The advantage is that individual microservices can be developed and deployed independently while being more elastic and resilient in demanding situations. Hence, containerization technologies such as Docker is a natural fit due to its minimal overhead and application-level isolation.
The following is a simplistic example of a Django web application implemented as a microservice using containers:
This single microservice is composed of three containers with separate logical components: Nginx container (web server), Gunicorn/Django container (web application), and PostgreSQL container (database). Each container is instantiated from a Docker image, which may be built using a Dockerfile.
Docker containers have an ephemeral file system, so persistent data is managed by explicitly creating a volume. Volumes can be used to share data between containers. In this case, the static files of the Django project can be shared to the Nginx container to serve them directly.
As you can imagine, most real-world applications will be composed of multiple Microservices and each of them would require multiple containers. Kubernetes is the most widely recommended solution for managing such container clusters.
Platform as a service
A Platform as a Service (PaaS) is defined as a cloud service where the solution stack is already provided and managed for you. Popular platforms for Django hosting include Heroku, PythonAnywhere, and Google App Engine.
Virtual private servers
A virtual private server (VPS) is a virtual machine hosted in a shared environment. From the developer's perspective, it would seem like a dedicated machine (hence, the word private) preloaded with an operating system.
A more appropriate term would be Function as a Service (FaaS), as these platforms support execution of an application logic like a small Python function but does not store any state.
Infrastructure as a service (IaaS) is a type of cloud computing service that offers essential compute, storage, and networking resources on demand, on a pay-as-you-go basis. IaaS is one of the four types of cloud services, along with software as a service (SaaS), platform as a service (PaaS), and serverless.
Migrating your organization's infrastructure to an IaaS solution helps you reduce maintenance of on-premises data centers, save money on hardware costs, and gain real-time business insights. IaaS solutions give you the flexibility to scale your IT resources up and down with demand. They also help you quickly provision new applications and increase the reliability of your underlying infrastructure.
An IaaS by contrast provides total flexibility is typically cheaper, but it requires a high degree of knowledge and effort to properly set up. Prominent IaaS options include DigitalOcean, Linode, Amazon EC2, and Google Compute Engine among many others.
If you are interested in maximizing performance, you can opt for a bare metal server with a collocation from providers, such as Rackspace.
Fabric is favored among Python web developers for its simplicity and ease of use. It expects a file named fabfile.py that defines all the actions (for deployment or otherwise) in your project. Each of these actions can be a local or remote shell command. The remote host is connected via SSH.
The key strength of Fabric is its ability to run commands on a set of remote hosts. For instance, you can define a web group that contains the hostnames of all web servers in production.
When you run your Fabric deployment command, say, fab deploy, the following scripted sequence of actions take place:
1. Runs all tests locally
2. Commits all local changes to Git
3. Pushes to a remote central Git repository
4. Resolves merge conflicts, if any
5. Collects the static files (CSS, images)
6. Copies the static files to the static file server
7. At the remote host, pulls changes from a central Git repository
8. At the remote host, runs (database) migrations
9. At the remote host, touches app. wsgi to restart WSGI server
The entire process is automatic and should be completed in a few seconds. By default, if any step fails, then the deployment gets aborted. Though not explicitly mentioned, there would be checks to ensure that the process is idempotent.
Configuration management tools such as Chef, Puppet, or Ansible try to bring a server to a certain desired state. Unlike Fabric, which requires the deployment process to be specified in an imperative manner, these configuration-management tools are declarative. You just need to define the final state you want the server to be in, and it will figure out how to get there.
One of the most important advantages of configuration-management tools is that they are idempotent by default. Your servers can go from an unknown state to a known state, resulting in easier server configuration management and reliable deployment.
Among configuration-management tools, Chef, and Puppet enjoy wide popularity since they were one of the earliest tools in this category. For such folks, we have Salt and Ansible as excellent alternatives.
Monitoring also helps with the early detection of problems. Unusual patterns, such as spikes or a gradually increasing load, can be signs of bigger underlying problems, such as memory leak. A good monitoring system can alert site owners of problems before they happen.
Monitoring tools usually need a backend service (sometimes called agents) to collect the statistics and a frontend service to display dashboards or generate reports. Popular data collection backends include StatsD and Monit. This data can be passed to front-end tools, such as Graphite.
There are several hosted monitoring tools, such as New Relic and Status.io, which are easier to set up and use.
The key to improving performance is finding where the bottlenecks are. As Lord Kelvin would say:
"If you can't measure it, you can't improve it."
A good starting point for frontend optimization would be to check your site with Google page speed or Yahoo! YSlow (commonly used as browser plugins). These tools will rate your site and recommend various best practices, such as minimizing the number of HTTP requests or gzipping the content.
Django can help you improve frontend performance in a number of ways:
Cache infinitely with CachedStaticFilesStorage: The fastest way to load static assets is to leverage the browser cache. By setting a
long caching time, you can avoid re-downloading the same asset again and again. However, the challenge is to know when not to use the cache when
the content changes.
CachedStaticFilesStorage class solves this elegantly by appending the asset's MD5 hash to its filename. This way, you can extend the TTL of the cache for these files infinitely.
To use this, set the CACHES setting named staticfiles to CachedStaticFilesStorage or, if you have a custom storage, inherit from CachedFilesMixin. Also, it is best to configure your caches to use the local memory cache backend to perform the static filename to its hashed name lookup.
Use a static asset manager: An asset manager can pre-process your static assets to minify, compress, or concatenate them, thereby reducing their size and minimizing requests. It can also preprocess them, enabling you to write them in other languages, such as CoffeeScript and Syntactically awesome stylesheets (Sass). There are several Django packages that offer static asset management such as django-pipeline or webassets.
Backend performance improvements covers your entire server-side web stack, including database queries, template rendering, caching, and background jobs.
For quick and easy profiling needs, django-debug-toolbar is quite handy. We can also use Python profiling tools, such as the hotshot module for detailed analysis. In Django, you can use one of the several profiling middleware snippets to display the output of hotshot in the browser. A recent live-profiling solution is django-silk. It stores all the requests and responses in the configured database, allowing aggregated analysis over an entire
user session, say, to find the worst-performing views.
As the documentation suggests, you should enable the cached template loader in production.The cached template is compiled the first time it is needed and then stored in memory. Subsequent requests for the same template are served from memory.
Sometimes, the Django ORM can generate inefficient SQL code. There are several optimization patterns to improve this, as follows:
Reduce database hits with select_related: If you are using a OneToOneField or a Foreign key relationship, in forwarding
direction, for a large number of objects, then select_related() can perform a SQL join and reduce the number of database hits.
Reduce database hits with prefetch_related: For accessing a ManyToManyField method or, a Foreign key relation, in reverse
direction, or a Foreign key relation in a large number of objects, consider using prefetch_related to reduce the number of database hits.
Fetch only needed fields with values or values_list: You can save time and memory usage by limiting queries to return only the needed fields and
skipping model instantiation using values() or values_list().
Denormalize models: Selective denormalization improves performance by reducing joins at the cost of data consistency. It can also be used for
precomputing values, such as the sum of fields or the active status report into an extra column. Compared to using annotated values in queries,
denormalized fields are often simpler and faster.
Add an index: If a non-primary key gets searched a lot in your queries, consider setting that field's db_index to True in your model definition.
Create, update, and delete multiple rows at once: Multiple objects can be operated upon in a single database query with the bulk_create(), update(), and delete() methods. However, they come with several important caveats such as skipping the save() method on that model. So, read the documentation carefully before using them.
Django has a flexible cache system that allows you to cache anything from a template fragment to an entire site. It allows a variety of pluggable backends such as file-based or data-based backed storage. Most production systems use a memory-based caching system, such as Redis or Memcached. This is purely because volatile memory is many orders of magnitude faster than disk-based storage.
By default, Django stores its user session in the database. This usually gets retrieved for every request. To improve performance, the session data can be stored in memory by changing the SESSION_ENGINE setting. For instance, add the following in settings.py to store the session data in your cache:
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
For basic caching strategies, it might be easier to use a caching framework. Among the popular ones are django-cache-machine and django-cachalot. They can handle common scenarios, such as automatically caching results of queries to avoid database hits every time you perform a read.
Varnish can make pages load extremely fast. However, if used improperly, it might serve static pages to your users. Varnish can be easily configured to recognize dynamic pages or dynamic parts of a page such as a shopping cart.
Russian doll caching, popular in the rails community, is an interesting template cache-invalidation pattern. Imagine a user's timeline page with a series of posts, each containing a nested list of comments. In fact, the entire page can be considered as several nested lists of content. At each level, the rendered template fragment gets cached.
Another common caching pattern is to cache forever. Even after the content changes, the user might get served stale data from the cache. However, an asynchronous job, such as a Celery job, also gets triggered to update the cache. You can also periodically warm the cache at a certain interval to refresh the content.
Essentially, a successful caching strategy identifies the static and dynamic parts of a site. For many sites, the dynamic parts are the user-specific data when you are logged in. If this is separated from the generally available public content, then implementing caching becomes easier.