2017-04-17

Recap of DjangoConEurope 2017

"DjangoCon, why is everybody wearing this t-shirt?" wondered the security guys in the airport of Florence, Italy, in the beginning of April. The reason for that was DjangoCon Europe 2017 happening there for a week, full of interesting talks in an exciting location.

What I liked, was that the conference was not only about technical novelties in Django world, but also about human issues that programmers deal with in everyday life.

Interesting Non-tech Topics

According to a manifest, the conference had a goal to strengthen the Django community and to shape responsible attitude towards the works done with Django.

Healthy and Successful Community

We have to build stronger communities including everyone who wants to participate without discrimination. Although, at first, it might be difficult as people have biases, i.e. prejudices for or against one person or group; by being emphatic we can accept and include everyone no matter what is their gender identity or expression, sexual orientation, ethnicity, race, neurodiversity, age, religion, disabilities, geographical location, food diversities, body size, or family status.

Valuing diversity and individual differences is the key for a healthy, positive and successful community, that empowers its members and helps them grow stronger and happier.

Responsibility for How We Use Technology

Information technology companies (Apple, Alphabet, Microsoft, Amazon, and Facebook) are among the most traded companies in the world. IT connects people and their things, automates processes, stores and treats historical data. Usually you don't need many physical resources to start an IT business. Software developers have a power to shape the future, but should use this power responsibly:

With this, great responsibility is upon us: to make the future a better place, to make the future more evenly distributed, across gender gaps and discriminations, breaking economical, political and geographical barriers.

Business

  • When creating an online business, it is important to think about the business value that your product will give to people and the way you will make money with it. Don't make assumptions without talking to your customers.
  • When choosing employees for your company, give them freedom how to prove their knowledge: by a quiz, or whiteboard interview, or a take-home programming task. Different people have different ways how to best represent their skills.
  • Launch as early as possible. Track the usage statistics with Google Analytics or other analytics service. Collect emails for the mailing list. Write about your product in a blog and personalized emails.
  • Django is an open-source project based on the hard work of many professionals, and if you gain any commercial value of it and appreciate the framework, you should donate to the Django Software Foundation.

Interesting Tech Topics

From the technical point of view, I liked several ideas mentioned in the conference:

Automate your processes

  • For starting new projects, you can have boilerplates with the mostly used functionalities already prepared. Django management command startproject has a parameter --template for that where you can pass a URL to a zip file.
  • Developers should have troubleshooting checklists for debugging, just like airplane pilots.
  • There are several types of tests. Unit tests check the functionality of individual functions or methods. Integration tests check how different units work together. The functional tests check how the processes of business requirements work from start to end. Finally, there is manual testing requiring people to click through the website and fill in the forms. Some tests like the ones involving third-party authentication or mobile phones, are hardly possible to automate. Anyway, manual testing is the most expensive in time and resources (besides it being boring for the tester), functional tests go after them, then integration tests, and lastly unit tests. Although automatic testing adds up to the development time, in the long run it makes the systems more stable and error proof.

What about Django

  • You can extend the Django ORM with custom lookups, transactions, and filtered prefetchings, to make your QuerySets more readable and more capable.
  • Once again, PostgreSQL has more capabilities than MySQL and is more stable. Use EXPLAIN ANALYZE ... SQL command to find the bottlenecks of your database queries. You can usually fix them by adding indexes.
  • You can have custom indexes for your database tables, to optimize the performance on PostgreSQL (or some other vendor) database.
  • Django 1.11 is out and it's a long-term support version.

What about Third Party Django Packages

  • After analyzing the 6 most popular model translation packages (parler, hvad, klingon, modeltranslation, nece, and i18nfield) from different angles (database support, integration in django admin and forms, performance, etc.), django-hvad seemed to be the winning approach.
  • You can visually build static websites with very little coded configuration using django-cms and djangocms-cascade. The djangocms-cascade provides an alternative nested-plugins system for Django CMS.

What about Django Projects

  • If you build web apps for developing countries, you have to keep these things in mind: people might be using cell phones instead of computers (you need responsive design with small or no images), Internet connectivity is slow and unstable (websites have to be fast and should preferably have offline versions), the users do not always understand English (the websites should be translated and adapted), and locations where people live do not always have street addresses.
  • Some interesting use cases: tracking the health of the heart with Arduino and Django, providing weather data to the whole Europe using Django, managing a radio station in Birmingham using Django.

Thanks

Finally, thanks to the organizers for making this conference as great as it was. The city was beautiful, the food and coffee was delicious, the location for the talks was impressive. Looking forward to the next DjangoConEurope!

2017-03-01

Tracking the Results of Cron Jobs

Every Django website needs some automatic background tasks to execute regularly. The outdated sessions need to be cleaned up, search index needs to be updated, some data needs to be imported from RSS feeds or APIs, backups need to be created, you name it. Usually, if not all the time, those regular tasks are being set as cron jobs. However, when some task is run in the background, by default, you don't get any feedback whether it was successfully completed, or whether it crashed on the way. In this post I will show you how I handle the results of cron jobs.

In a Django project, all those tasks are usually implemented as management commands. For each such command I write a short bash script, that will call the management command with specific parameters and will print the verbose output to a log file.

Let's say my project structure is like this on a remote server:

/home/myproject
├── bin
├── include
├── lib
├── public_html
├── backups
├── project
│   └── myproject
├── scripts
└── logs

A virtual environment is created in the home directory of myproject linux user. The Django project itself is kept under project directory. The scripts directory is for my bash scripts. And the logs directory is for the verbose output of the bash scripts.

For example, for the default clearsessions command that removes outdated sessions, I would create scripts/cleanup.sh bash script as follows:

#!/usr/bin/env bash
SECONDS=0
PROJECT_PATH=/home/myproject
CRON_LOG_FILE=${PROJECT_PATH}/logs/cleanup.log

echo "Cleaning up the database" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}

cd ${PROJECT_PATH}
source bin/activate
cd project/myproject    
python manage.py clearsessions --verbosity=2 --traceback >> ${CRON_LOG_FILE}  2>&1

echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}

To run this command every night at 1 AM, you could create a file myproject_crontab with the following content:

MAILTO=""
00 01 * * * /home/myproject/scripts/cleanup.sh

Then register the cron jobs with:

$ crontab myproject_crontab

By such a bash script, I can track:

  • At what time the script was last executed.
  • What is the verbose output of the management command.
  • If the management command broke, what was in the traceback.
  • Whether the command finished executing or hung up.
  • How long it took to run the command.

In addition, this gives me information whether the crontab was registered and whether the cron service was running at all. As I get the total time of execution in minutes and seconds, I can decide how often I can call the cron job regularly so that it doesn't clash with another cron job.

When you have multiple Django management commands, you can group them thematically into single bash script, or you can wrap them into individual bash scripts. After putting them into the crontab, the only thing left is manually checking the logs from time to time.

If you have any suggestions how I could even improve this setup, I would be glad to hear your opinion in the comments.

Here is the Gist of the scripts in this post. To see some examples of custom Django management commands, you can check Chapter 9, Data Import and Export in my book Web Development with Django Cookbook - Second Edition.

2016-12-16

Django Administration: Inlines for Inlines

The default Django model administration comes with a concept of inlines. If you have a one-to-many relationship, you can edit the parent and its children in the same form. However, you are limited in a way that you cannot have inlines under inlines at nested one-to-many relations. For example, you can't show models Painter, Picture, and Review in the same form if one painter may have drawn multiple pictures and each picture may have several reviews.

In this article I would like to share a workaround allowing you to quickly access the inlines of an inline model. The idea is that for every inline you can provide a HTML link leading to the separate form where you can edit the related model and its own relations. It's as simple as that.

For example, in the form of Painter model, you have the instances of Picture listed with specific links "Edit this Picture separately":

When such a link is clicked, the administrator goes to the form of the Picture model which shows the instances of Review model listed underneath:

Let's have a look, how to implement this.

First of all, I will create a gallery app and define the three models there. Nothing fancy here. The important part there is just that the Picture model has a foreign key to the Painter model and the Review model has a foreign key to the Picture model.

# gallery/models.py
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals

import os

from django.db import models
from django.utils.encoding import python_2_unicode_compatible
from django.utils.translation import ugettext_lazy as _
from django.utils.text import slugify


@python_2_unicode_compatible
class Painter(models.Model):
    name = models.CharField(_("Name"), max_length=255)

    class Meta:
        verbose_name = _("Painter")
        verbose_name_plural = _("Painters")

    def __str__(self):
        return self.name


def upload_to(instance, filename):
    filename_base, filename_ext = os.path.splitext(filename)
    return "painters/{painter}/{filename}{extension}".format(
        painter=slugify(instance.painter.name),
        filename=slugify(filename_base),
        extension=filename_ext.lower(),
    )

@python_2_unicode_compatible
class Picture(models.Model):
    painter = models.ForeignKey(Painter, verbose_name=_("Painter"), on_delete=models.CASCADE)
    title = models.CharField(_("Title"), max_length=255)
    picture = models.ImageField(_("Picture"), upload_to=upload_to)

    class Meta:
        verbose_name = _("Picture")
        verbose_name_plural = _("Pictures")

    def __str__(self):
        return self.title


@python_2_unicode_compatible
class Review(models.Model):
    picture = models.ForeignKey(Picture, verbose_name=_("Picture"), on_delete=models.CASCADE)
    reviewer = models.CharField(_("Reviewer name"), max_length=255)
    comment = models.TextField(_("Comment"))

    class Meta:
        verbose_name = _("Review")
        verbose_name_plural = _("Reviews")

    def __str__(self):
        return self.reviewer

Then I will create the administration definition for the models of the gallery app. Here I will set two types of administration for the Picture model:

  • By extending admin.StackedInline I will create administration stacked as inline.
  • By extending admin.ModelAdmin I will create administration in a separate form.

In Django model administration besides usual form fields, you can also include some computed values. This can be done by your fields (or fieldsets) and readonly_fields attributes referring to a callable or a method name.

You can set a translatable label for those computed values by defining short_description attribute for the callable or method. If you want to render some HTML, you can also set the allow_tags attribute to True (otherwise your HTML string will be escaped).

# gallery/admin.py
# -*- coding: UTF-8 -*-
from django.contrib import admin
from django.core.urlresolvers import reverse
from django.utils.translation import ugettext_lazy as _
from django.utils.text import force_text

from .models import Painter, Picture, Review

def get_picture_preview(obj):
    if obj.pk:  # if object has already been saved and has a primary key, show picture preview
        return """<a href="{src}" target="_blank"><img src="{src}" alt="{title}" style="max-width: 200px; max-height: 200px;" /></a>""".format(
            src=obj.picture.url,
            title=obj.title,
        )
    return _("(choose a picture and save and continue editing to see the preview)")
get_picture_preview.allow_tags = True
get_picture_preview.short_description = _("Picture Preview")


class PictureInline(admin.StackedInline):
    model = Picture
    extra = 0
    fields = ["get_edit_link", "title", "picture", get_picture_preview]
    readonly_fields = ["get_edit_link", get_picture_preview]

    def get_edit_link(self, obj=None):
        if obj.pk:  # if object has already been saved and has a primary key, show link to it
            url = reverse('admin:%s_%s_change' % (obj._meta.app_label, obj._meta.model_name), args=[force_text(obj.pk)])
            return """<a href="{url}">{text}</a>""".format(
                url=url,
                text=_("Edit this %s separately") % obj._meta.verbose_name,
            )
        return _("(save and continue editing to create a link)")
    get_edit_link.short_description = _("Edit link")
    get_edit_link.allow_tags = True


@admin.register(Painter)
class PainterAdmin(admin.ModelAdmin):
    save_on_top = True
    fields = ["name"]
    inlines = [PictureInline]


class ReviewInline(admin.StackedInline):
    model = Review
    extra = 0
    fields = ["reviewer", "comment"]


@admin.register(Picture)
class PictureAdmin(admin.ModelAdmin):
    save_on_top = True
    fields = ["painter", "title", "picture", get_picture_preview]
    readonly_fields = [get_picture_preview]
    inlines = [ReviewInline]

In this administration setup, the get_edit_link() method creates a HTML link between the inline and the separate administration form for the Picture model. As you can see, I also added the get_picture_preview() function as a bonus. It is included in both administration definitions for the Picture model and its purpose is to show a preview of the uploaded picture after saving it.

To recap, nested inlines are not supported by Django out of the box. However, you can have your inlines edited in a separate page with the forms linked to each other. For the linking you would use some magic of the readonly_fields attribute.

What if you really need to have inlines under inlines in your project? In that case you might check django-nested-admin and don't hesitate to share your experience with it in the comments.

2016-05-26

Debugging Django Management Commands in PyCharm

My favorite editor for Python projects is PyCharm. Besides editing code, it allows you to inspect the database, work with Git repositories, run management commands, execute bash commands and Python scripts, and debug code just in the same window. In this article, I will show you how to set breakpoints and debug Django management commands visually in PyCharm.

Django management commands are scripts that can be executed on your Django project to do something with the project database, media files, or code. Django itself comes with a bunch of commands like: migrate, runserver, collectstatic, makemessages, and clearsessions. Management commands can be executed like this:

(myproject_env)$ python manage.py clearsessions

If you want to create a custom management command in your project, you can find how to do that in the official Django documentation. Also you can find some practical examples in the Chapter 9, Data Import and Export of the Web Development with Django Cookbook - Second Edition.

In this example, I won't create any new management command, but will debug the clearsessions command that is coming from Django and is located at django/contrib/sessions/management/commands/clearsessions.py.

First of all, let's click on "Edit Configurations..." in the top toolbar just before the Run button (with the Play icon). In the opened dialog box "Run/Debug Configurations" click on the "Add New Configuration" button (with the Plus icon) and choose "Python".

Let's fill in the configuration with these values:

Name: Clear Outdated Sessions
Script: /Users/me/DjangoProjects/myproject_env/project/myproject/manage.py
Script paramethers: clearsessions --traceback --verbosity=2
Python interpreter: Python 2.7.6 virtualenv at ~/DjangoProjects/myproject_env
Working directory: /Users/me/DjangoProjects/myproject_env/project/myproject/

Then open the file with the definition of the management command django/contrib/sessions/management/commands/clearsessions.py. Click on the left padding of the editor to add a breakpoint (marked with a red circle) where the script should stop executing for inspection.

Normally to run this script, you could click on the Run button (with the Play icon). But as we want to debug the script, we will click on the Debug button (with the Bug icon) in the toolbar.

The script will start executing and will stop temporarily at the breakpoint you made. You will be able to inspect all local variables in the debug panel that is opened at the bottom of your window by default.

You can navigate through code execution with the arrow buttons "Step Over", "Step Into", etc. To evaluate local or global variables or values, click on the "Evaluate Expression" button (with the Calculator icon) and enter some Python code.

Click on the "Resume Program" button (with the Fast Play icon) to continue execution of the script, when you are ready.

Analogously, you can debug your models, views, forms, or middlewares by running a development server ("Django server") in debug mode.

2016-05-21

Deploying a Django Website on Heroku

Once you have a working project, you have to host it somewhere. One of the most popular deployment platforms nowadays is Heroku. Heroku belongs to a Platform as a Service (PaaS) category of cloud computing services. Every Django project you host on Heroku is running inside a smart container in a fully managed runtime environment. Your project can scale horizontally (adding more computing machines) and you pay for what you use starting with a free tier. Moreover, you won't need much of system administrator's skills to do the deployment - once you do the initial setup, the further deployment is as simple as pushing Git repository to a special heroku remote.

However, there are some gotchas to know before choosing Heroku for your Django project:

  • One uses PostgreSQL database with your project. MySQL is not an option.
  • You cannot store your static and media files on Heroku. One should use Amazon S3 or some other storage for that.
  • There is no mailing server associated with Heroku. One can use third-party SendGrid plugin with additional costs, GMail SMTP server with sent email amount limitations, or some other SMTP server.
  • The Django project must be version-controlled under Git.
  • Heroku works with Python 2.7. Python 3 is not yet supported.

Recently I deployed a small Django project on Heroku. To have a quick reference for the future, I summed up the process here providing instructions how to do that for future reference.

1. Install Heroku Toolbelt

Sign up for a Heroku account. Then install Heroku tools for doing all the deployment work in the shell.

To connect your shell with Heroku, type:

$ heroku login

When asked, enter your Heroku account's email and password.

2. Prepare Pip Requirements

Activate your project's virtual environment and install Python packages required for Heroku:

(myproject_env)$ pip install django-toolbelt

This will install django, psycopg2, gunicorn, dj-database-url, static3, and dj-static to your virtual environment.

Install boto and Django Storages to be able to store static and media files on an S3 bucket:

(myproject_env)$ pip install boto
(myproject_env)$ pip install django-storages

Go to your project's directory and create the pip requirements that Heroku will use in the cloud for your project:

(myproject_env)$ pip freeze -l > requirements.txt

3. Create Heroku-specific Files

You will need two files to tell Heroku what Python version to use and how to start a webserver.

In your project's root directory create a file named runtime.txt with the following content:

python-2.7.11

Then at the same location create a file named Procfile with the following content:

web: gunicorn myproject.wsgi --log-file -

4. Configure the Settings

As mentioned in the "Web Development with Django Cookbook - Second Edition", we keep the developmnent and production settings in separate files both importing the common settings from a base file.

Basically we have myproject/conf/base.py with the settings common for all environments.

Then myproject/conf/dev.py contains the local database and dummy email configuration as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *

DATABASES = {
    "default": {
        "CONN_MAX_AGE": 0,
        "ENGINE": "django.db.backends.postgresql",
        "HOST": "localhost",
        "NAME": "myproject",
        "PASSWORD": "",
        "PORT": "",
        "USER": "postgres"
    }
}

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"

Lastly for the production settings we need myproject/conf/prod.py with special database configuration, non-debug mode, and unrestrictive allowed hosts as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *
import dj_database_url

DATABASES = {
    "default": dj_database_url.config()
}

ALLOWED_HOSTS = ["*"]

DEBUG = False

Now let's open myproject/settings.py and add the following content:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .conf.dev import *

Finally, open the myproject/wsgi.py and change the location of the DJANGO_SETTINGS_MODULE there:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.conf.prod")

5. Set Up Amazon S3 for Static and Media Files

Create an Amazon S3 bucket myproject.media at the AWS Console (web interface for Amazon Web Services). Go to the properties of the bucket, expand "Permissions" section, click on the "add bucket policy" button and enter the following:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::myproject.media/*"
        }
    ]
}

This ensures that files on the S3 bucket will be accessible publicly without any API keys.

Go back to your Django project and add storages to the INSTALLED_APPS in myproject/conf/base.py:

INSTALLED_APPS = [
    # ...
    "storages",
]

Media files and static files will be stored on different paths under S3 bucket. To implement that, we need to create two Python classes under a new file myproject/s3utils.py as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from storages.backends.s3boto import S3BotoStorage

class StaticS3BotoStorage(S3BotoStorage):
    """
    Storage for static files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'static'
        super(StaticS3BotoStorage, self).__init__(*args, **kwargs)


class MediaS3BotoStorage(S3BotoStorage):
    """
    Storage for uploaded media files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'media'
        super(MediaS3BotoStorage, self).__init__(*args, **kwargs)

Finally, let's edit the myproject/conf/base.py and add AWS settings:

AWS_S3_SECURE_URLS = False       # use http instead of https
AWS_QUERYSTRING_AUTH = False                # don't add complex authentication-related query parameters for requests
AWS_S3_ACCESS_KEY_ID = "..."                # Your S3 Access Key
AWS_S3_SECRET_ACCESS_KEY = "..."            # Your S3 Secret
AWS_STORAGE_BUCKET_NAME = "myproject.media"
AWS_S3_HOST = "s3-eu-west-1.amazonaws.com"  # Change to the media center you chose when creating the bucket

STATICFILES_STORAGE = "myproject.s3utils.StaticS3BotoStorage"
DEFAULT_FILE_STORAGE = "myproject.s3utils.MediaS3BotoStorage"

# the next monkey patch is necessary to allow dots in the bucket names
import ssl
if hasattr(ssl, '_create_unverified_context'):
   ssl._create_default_https_context = ssl._create_unverified_context

Collect static files to the S3 bucket:

(myproject_env)$ python manage.py collectstatic --noinput

6. Set Up Gmail to Send Emails

Open myproject/conf/prod.py and add the following settings:

EMAIL_USE_TLS = True
EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = "myproject@gmail.com"
EMAIL_HOST_PASSWORD = "mygmailpassword"
EMAIL_PORT = 587

7. Push to Heroku

Commit and push all the changes to your Git origin remote. Personally I prefer using SourceTree to do that, but you can also do that in the command line, PyCharm, or another software.

In your project directory type the following:

(myproject_env)$ heroku create my-unique-project

This will create a Git remote called "heroku", and a new Heroku project "my-unique-project" which can be later accessed at http://my-unique-project.herokuapp.com.

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

8. Transfer Your Local Postgres Database To Heroku

Create local database dump:

(myproject_env)$ PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump

Upload the database dump temporarily to some server, for example, S3 bucket: http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump. Then import that dump into the Heroku database:

(myproject_env)$ heroku pg:backups restore 'http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump' DATABASE_URL

Remove the database dump from S3 server.

9. Set Environment Variables

If your Git repository is not private, put your secret values in environment variables rather than in the Git repository directly.

(myproject_env)$ heroku config:set AWS_S3_ACCESS_KEY_ID=ABCDEFG123
$ heroku config:set AWS_S3_SECRET_ACCESS_KEY=aBcDeFg123

To read out the environment variables you can type:

(myproject_env)$ heroku config

To read out the environment variables in the Python code open myproject/conf/base.py and type:

import os
AWS_S3_ACCESS_KEY_ID = os.environ.get("AWS_S3_ACCESS_KEY_ID", "")
AWS_S3_SECRET_ACCESS_KEY = os.environ.get("AWS_S3_SECRET_ACCESS_KEY", "")

10. Set DNS Settings

Open your domain settings and set CNAME to "my-unique-project.herokuapp.com".

At last, you are done! Drop in the comments if I missed some part. For the new updates, see the next section.

*. Update Production

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

If you have changed something in the static files, collect them again:

(myproject_env)$ python manage.py collectstatic --noinput

Collecting static files to S3 bucket takes quite a long time, so I do not recommend to do that automatically every time when you want to deploy to Heroku.

Further Reading

You can read more about Django on Heroku in the following resources:

2016-02-10

Special Offer for the Readers of DjangoTricks Blog

Packt Publishing, the company that published my Django book, has a special offer for enthusiast and professional developers reading this blog. For two weeks you can get the eBook "Web Development with Django Cookbook - Second Edition" for half price. The eBook is available in PDF, ePub, Mobi, and Kindle formats. Also you will get access to download the related code files.

Use the discount code DJGTRK50 at the Packt Publishing bookstore.
The discount is valid until the 24th of February, 2016.

2016-02-06

Fresh Book for Django Developers

This week the post office delivered a package that made me very satisfied. It was a box with three paper versions of my "Web Development with Django Cookbook - Second Edition". The book was published at the end of January after months of hard, but fulfilling work in the late evenings and at weekends.

The first Django Cookbook was dealing with Django 1.6. Unfortunately, the support for that version is over. So it made sense to write an update for a newer Django version. The second edition was adapted for Django 1.8 which has a long-term support until April 2018 or later. This edition introduces new features added to Django 1.7 and Django 1.8, such as database migrations, QuerySet expressions, or System Check Framework. Most concepts in this new book should also be working with Django 1.9.

My top 5 favourite new recipes are these:

  • Configuring settings for development, testing, staging, and production environments
  • Using database query expressions
  • Implementing a multilingual search with Haystack
  • Testing pages with Selenium
  • Releasing a reusable Django app

The book is worth reading for any Django developer, but will be best understood by those who already know the basics of web development with Django. You can learn more about the book and buy it at the Packt website or Amazon.

I thank the Packt Publishing very much for long cooperation in the development of this book. I am especially thankful to acquisition editor Nadeem N. Bagban, content development editors Arwa Manasawala and Sumeet Sawant, and technical editor Bharat Patil. Also I am grateful for insightful feedback from the reviewer Jake Kronika.

What 5 recipes do you find the most useful?

2015-01-30

How to Find the Performance Bottlenecks in Your Django Views?

Once you have your Django projects running, you come to situations, when you need to optimize for performance. The rule of thumb is to find the bottlenecks and then to take action to eliminate them by more idiomatic Python code, database denormalization, caching, or other techniques.

What is a bottleneck? Literally it refers to the top narrow part of a bottle. In engineering, bottleneck is a case where the performance or capacity of an entire system is limited by a single or small number of components or resources.

How to find these parts of your code? The most trivial way is to check the current time before specific code execution and after that code execution, and then count the time difference:

from datetime import datetime
start = datetime.now()
# heavy execution ...
end = datetime.now()
d = end - start  # datetime.timedelta object
print d.total_seconds()  # prints something like 7.861985

However, measuring code performance for Django projects like this is inefficient, because you need a lot of such wrappers for your code until you find which part is the most critical. Also you need a lot of manual computation to find the critical parts.

Recently I found line_profiler module that can inspect the performance of the code line by line. By default, to use line_profiler for your functions, you should decorate them with @profile decorator and then to execute the script:

$ kernprof -l some_script_to_profile.py

This script will execute your script, analize the decorated function, and will save results to a binary file that can later be inspected with:

$ python -m line_profiler some_script_to_profile.py.lprof

That's quite complicated, but to use line_profiler for Django views, you can install django-devserver which replaces the original development server of Django and will output the performance calculations immediately in the shell like this:

[30/Jan/2015 02:26:40] "GET /quotes/json/ HTTP/1.1" 200 137
    [sql] 1 queries with 0 duplicates
    [profile] Total time to render was 0.01s
    [profile] Timer unit: 1e-06 s

          Total time: 0.001965 s
          File: /Users/archatas/Projects/quotes_env/project/inspirational/quotes/views.py
          Function: quote_list_json at line 27

          Line #      Hits         Time  Per Hit   % Time  Line Contents
          ==============================================================
              27                                           def quote_list_json(request):
              28         1            2      2.0      0.1      quote_dict_list = []
              29         2         1184    592.0     60.3      for quote in InspirationQuote.objects.all():
              30         1            1      1.0      0.1          quote_dict = {
              31         1            1      1.0      0.1              'author': quote.author,
              32         1            1      1.0      0.1              'quote': quote.quote,
              33         1          363    363.0     18.5              'picture': quote.get_medium_picture_url(),
              34                                                   }
              35         1            1      1.0      0.1          quote_dict_list.append(quote_dict)
              36
              37         1           42     42.0      2.1      json_data = json.dumps(quote_dict_list)
              38         1          370    370.0     18.8      return HttpResponse(json_data, content_type="application/json")

The most interesting data in this table is the "% Time" column, giving an overview in percentage which lines of the Django view function are the most time-consuming. For example, here it says that I should pay the most attention to the QuerySet, the method get_medium_picture_url() and the HttpResponse object.

To setup line profiling, install line_profiler and django-devserver to you virtual environment:

(myproject_env)$ pip install line_profiler
(myproject_env)$ pip install django-devserver

Then make sure that you have the following settings in your settings.py or local_settings.py:

# settings.py
INSTALLED_APPS = (
    # ...
    'devserver',
)

MIDDLEWARE_CLASSES = (
    # ...
    'devserver.middleware.DevServerMiddleware',
)

DEVSERVER_MODULES = (
    'devserver.modules.sql.SQLRealTimeModule',
    'devserver.modules.sql.SQLSummaryModule',
    'devserver.modules.profile.ProfileSummaryModule',

    # Modules not enabled by default
    'devserver.modules.profile.LineProfilerModule',
)

DEVSERVER_AUTO_PROFILE = True  # profiles all views without the need of function decorator

When you execute

(myproject_env)$ python manage.py runserver

it will run the development server from django-devserver and for each visited view, it will show the analysis of code performance. I have tested this setup with Django 1.7, but it should work since Django 1.3.

Do you know any more useful tools to check for performance bottlenecks?

2014-11-04

Win Free Copies of “Web Development with Django Cookbook”

Readers would be pleased to know that I have teamed up with Packt Publishing to organize a Giveaway of my book Web Development with Django Cookbook.

Three lucky winners stand a chance to win an e-copy of the book. Keep reading to find out how you can be one of the lucky ones.

Book Overview

  • Improve your skills by developing models, forms, views, and templates
  • Create a rich user experience using Ajax and other JavaScript techniques
  • A practical guide to writing and using APIs to import or export data

How to Enter?

All you need to do is head on over to the book page and look through the product description of the book and drop a line via the comments below this post to let us know what interests you the most about this book. It’s that simple.

Prize

Winners will get an e-copy of the Book.

Deadline

The contest will close on the 1st of December, 2014 at 00:00 GMT. Winners will be contacted by email, so be sure to use your real email address when you comment!


...And the winners are...

Out of the 21 participants I finally chose these winners: Mark Phillips, Faria Chowdhury, and Mario Gudelj. The guys from Packt Publishing will contact you by email and will give you the free copies of "Web Development with Django Cookbook". Have a nice reading!

2014-10-27

Contributing Back to the Community - Django Cookbook

In the early beginning of year 2014, the IT book publishing company "Packt Publishing" contacted me with an interesting offer: to share my Django experience in a form of a book. I thought it might be a good challenge for me and also value for the Django community, as I had been working with Django for 7 years or so, and during that time there was quite a lot of knowledge gathered and used practically. So for the next 9 months on late evenings and weekends I was adapting some of the most useful code snippets and describing them in the book. The great staff from the Packt Publishing helped me to structure the content, make everything clear and understandable, and get the grammar correct. Finally, the book was released and it's called "Web Development with Django Cookbook".

Word cloud of the Web Development with Django Cookbook

This book is written for intermediate or advanced Django web developers. When writing the book, my own purpose was not to fully cover every possible web development task, but rather to have enough useful bits of knowledge for those who seek information about web development with Django. The book was written in the format of recipes. There are over 70 recipes giving you instructions how to deal with different challenges of web development. The code mentioned in the book is optimized for Django 1.6, but most of it should also work with older Django versions as well as with Django 1.7.

The cookbook consists of 10 chapters:

  1. Getting started with Django 1.6. This chapter will guide you through the basic configuration which is necessary to start any Django project. It will cover topics like virtual environment, version control, and project settings.
  2. Database Structure. When you create a new app, the first thing to do is to define your models. In this chapter you will learn how to write reusable pieces of code to use in your models. You will learn how to deal with multilingual data in your database. Also you will be told how to manage database schema changes using South migrations.
  3. Forms and Views. To show your data or let one create it, you need views and forms. This chapter will show you some patterns for creating them.
  4. Templates and JavaScript. Information is presented to the user by rendered templates. In modern websites, JavaScript is a must for richer user experience. This chapter shows practical examples of using templates and JavaScript together.
  5. Custom Template Filters and Tags. The default Django template system is quite extensive, but there are more things to add for different cases. This chapter shows you how to create and use own template tags and filters.
  6. Model Administration. Django framework comes with a handy pre-build administration. This chapter shows how to extend the default administration with your own functionality.
  7. Django CMS. Django CMS is the most popular open source content management system made in Django. This chapter deals with best practices using Django CMS and extending it for your needs.
  8. Hierarchical Structures. When you need to create a tree-like structure in Django, django-mptt module comes to hand. This chapter shows you how to use it and how to set administration for hierarchical structures.
  9. Data Import and Export. Very often there are cases when you need to transfer data from and to different formats, retrieve it from and provide it to different sources. This chapter deals with management commands for data import and also APIs for data export.
  10. Bells and Whistles. There is more to Django. This chapter shows additional snippets and tricks useful in web development with Django. Also you will learn about deployment of Django projects.

Get a copy of this book at Packt Publishing and tell me what you think about it in the comments below. Which recipes did you find the most useful? What would you like to read more in this blog or another edition of the cookbook?

Web Development with Django Cookbook