OWASP Top 10 # 1 – Injection Attacks

Injection

What is it?

An injection attack is where a program, database or interpreter is tricked into executing code or performing functionality which was not intended to be run.

Though SQL injection is probably the most commonly known type of injection attack, it is a vulnerability that can affect many technologies;

  • SQL
  • XPath
  • LDAP
  • Applications / Operating Systems

SQL injection is when a query or command parameter which should contain a simple variable value actually contains additional syntax which is inadvertently executed by the database.

How does it happen?

For many web applications a URL such as http://www.TheFooSite.com/MyDatabaseTable?Id=1 would translated into a query such as:

Select * From MyDatabaseTable Where Id = 1

The integrity of the value of the id has come from the user and as such can contain potentially any malicious payload.

For example, we can change the sql query to return all data within the table:

http://www.TheFooSite.com/MyDatabaseTable?Id=1 or 1=1

Alternatively we could make the database throw an error which could potentially propagate to the user and show our technology stack, this in turn would invite any potential hacker to start a more in depth probe:

http://www.TheFooSite.com/MyDatabaseTable?Id=x

Additionally we could append on queries which query the schema tables such sysobjects and could start to expose our database structure to the user. This could then be used to append queries which exposes the potentially sensitive data contained within these tables such as credit card numbers and identity numbers:

http://www.TheFooSite.com/MyDatabaseTable?Id=SELECT name FROM master..sysobjects WHERE xtype = 'U';

Untrusted data sources

Any information which comes from the user, third party or external entity should be considered as untrusted and therefore potentially dangerous.

The sources of untrusted data are numerous:

  • From the user
    • In the URL via a query string or route parameter
    • Posted via a form
  • From the browser
    • In cookies
    • In the request headers
  • Web services calls
  • Database
  • Files such as XML, XSD, CSV

In fact anything which does not come directly from our running source code should be treated as potentially malicious. That includes data and services from within the bounds of our company as well as data entered directly from our users, even if they are our colleagues.

How can I protect myself from this?

Security is all about addressing the security concern at multiple levels; this way if one level is breached there are more levels which can catch the breach.

The principle of least privilege

The principle of least privilege is about giving any entity that requires permissions to perform a job, the minimum level of permissions it requires to perform that job.

Why grant access for the user account which is used by a web application to query an entire database, delete data, modify schema or even grant administrator level access? By granting the minimum level of permissions you dramatically reduced the level of damage that a breach can perform.

Don’t give the account db_owner access; this can read, update, delete, truncate and drop any table as well as execute any stored procedures and modify the schema.

A user account should be given access to:

  • Read from only the individual table(s) required
  • Write to only the individual table(s) required
  • Execute only the required stored procedures required

Any account accessed by a web application should probably never be given access to modify any schema, read or write to any schema or system tables and should probably not even be given access to physically delete data from any tables.

If you need to delete data, consider marking records as deleted by a status field and archiving off the records by a system task running in the background.

Where different levels of access is required between types of user account groups or roles, consider having multiple accounts. For example a public account used by public user groups and an admin account which is used by back office staff.

Please note: just because the user role is ‘admin’ it does not mean they should be given anything but the bare minimum permissions within the database.

In line SQL parametrisation

Most issues from SQL injection are from parameters being attached to the query or command via string concatenation. Take the following example which concatenates a parameter taken from the user via a URL query string parameter.

var id = Request.QueryString["Id"];
var sqlCommand = "SELECT * FROM MyTable WHERE id = " + id;

The data within the parameter is run as is within the database allowing the data within the parameter to break out of its parameter scope and into a query scope.

For example “1 or 1=1” will be searched upon as if it has two parts of the predicate.

By parametrising the query, the data contained within the parameter will stay within the scope of the parameter; here we will try and search for an id which is equal to “1 or 1=1” which at some level will either retrieve no data or throw an error when trying to convert between data types.

The following shows how we can parametrise an line query:

var id = Request.QueryString["id"];

var sqlCommand = "SELECT * FROM Foo WHERE Id = @id";

using (var connection = new SqlConnection(connString))
{
using (var command = new SqlCommand(sqlString, connection))
{
command.Parameters.Add("@d", SqlDbType.VarChar).Value = id;
command.Connection.Open();
var data = command.ExecuteReader();
}
}

We now have an error thrown within by the database as it cannot convert the string “1 or 1=1” into an integer; the data type of the id field.

Stored procedure Parametrisation

The problem with in line parametrised queries is that you have to remember to do them. By using stored procedures you are enforced to perform this step.

var id = Request.QueryString["id"];

var sqlString = "GetFoo";
using (var conn = new SqlConnection(connString))
{
using (var command = new SqlCommand(sqlString, conn))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("@id", SqlDbType.VarChar).Value = id;
command.Connection.Open();
var data = command.ExecuteReader();
}
}

This will cause a similar issue that our bad string cannot be converted to an integer.

White listing untrusted data

Where possible, all untrusted data should always be validated against a white list of known safe values.

A white list states a set of rules which defines the data. Any incoming data must be validated against these rules to be considered safe. Failure of the validation should interpret the data as an expected attack.

There are multiple ways of performing white list:

  • Type conversion
    • Integer, date, GUID
  • Regular expression
    • Email address, phone number, name
  • Enumerated list of known good values
    • Titles, countries and colours for example

The following shows how we can use the try parse function to white list our id parameter by trying to convert it to an integer.

int idString;
if(!int.TryParse(idString, out id))
{
throw new Exception("The id is not a valid integer.");
}

Note: any errors should be handled, where required audited and then the user should be displayed a human readable message which gives nothing away about our implemented technology stack or in-depth details of the error.

Now that we have the parameter as integer we can add our parameter into the command as an integer:

command.Parameters.Add("@id", SqlDbType.Int).Value = id;

Any suspect attacks now should be caught way before the database which is a good place to be.

ORM’s

ORM’s should automatically parametrise any SQL.

The only thing to note is that when using ORM’s it is often required to drop to native SQL for performance reasons. When doing this care should be made to ensure all untrusted data is parametrised using the steps above.

Injection through stored procedures

Care should be taken when using stored procedures that they are not themselves concatenating parameters before calling exec.

declare @query varchar(max)
set @query = 'select * from dbo.MyTable where MyField like ''%'+ @parameter + '%''
EXEC(@query)

The stored procedure sp_executesql allows parametrising sql statements which are held as string.

declare @query varchar(max)
set @query = 'select * from dbo.Foo where MyField like ''%'+ @myFieldParameter + '%''

EXEC sp_executesql @query, N'@search varchar(50)', @myFieldParameter=@parameter

Automated SQL injection tools

There are a number of tools which can automate the hacking process, allowing people with next to no technical experience to perform automated attacks or to test the strength of web applications.

Best SQL Injection Tools

Advertisements

Self Closing MVC Html Helpers

The following is an example of how to write a self closing MVC Html helper similar to BeginForm(). It takes the form of a self closing DIV tag; in fact we will write a Bootstrap panel.

We will be profiting from the IDisposable and the Dispose method which will write our closing div tag

First we create a class which will be passed an Action which will write our end tag; this class implements IDisposable and calls out required action.

using System;

namespace WebApps.HtmlHelpers
{
  internal class DisposableHtmlHelper : IDisposable
  {
    private readonly Action _end;

    public DisposableHtmlHelper(Action end)
    {
      _end = end;
    }

    public void Dispose()
    {
      _end();
    }
  }
}

Now we write our help methods; a BeingPanel method which writes a div tag to the ViewContect response stream. It returns an instance of our newly created DisposableHtmlHelper as defined above. We register with it our method which will write out closing tag.

using System;
using System.Web.Mvc;

namespace WebApps.HtmlHelpers
{
  public static class HtmlHelpers
  {
    public static IDisposable BeginPanel(this HtmlHelper htmlHelper)
    {
      htmlHelper.ViewContext.Writer.Write(@"<div class="">");

      return new DisposableHtmlHelper(htmlHelper.EndPanel);
    }

    public static void EndDiv(this HtmlHelper htmlHelper)
    {
      htmlHelper.ViewContext.Writer.Write("</div>");
    }

    public static void EndPanel(this HtmlHelper htmlHelper)
    {
      htmlHelper.EndDiv();
    }
  }
}

We can now call this within a using statement in our view.

@using (Html.BeginPanel()) {
 // Monkey business would be here
}

Django – AJAX

This post is part of my Django series. You can see an overview of the series along with instruction on how to get all the source code here.

This article assumes you are comfortable creating a Django project, can create apps and register them into the INSTALLED_APPS list of the settings file. If not please read my Django HelloWorld article.

This article assumes you have a project called DjangoSandBox with an application called ajaxintroduction.

Overview

In these examples are we are going to create a page which uses AJAX to GET and POST data between the server.

The page is based around a ‘Quote’ object which contains the quote and the author. We will provide a single page which has the following functionality:

  • Get all quotes for an author
  • Get a random quote
  • Create a quote.

The following examples are split out but in the repository this is a single page. I strongly urge you to work from the source code within the repository.

Initial Page

So before we start on the AJAX calls we need an initial page up and running.

First lets create our model.

#models.py
from django.db import models
from django.db.models import Model

class Quote(Model):
    quote = models.TextField()
    author = models.CharField(max_length=50)

    def __str__(self):
        return "{0}: {1}".format(self.author, self.quote)

Make and run the migrations:

python manage.py makemigrations ajaxintroduction
python manage.py migrate

Next is our view, currently we simply want to return our template:

#views.py

from django.shortcuts import render

def get_quotes(request):
    return render(request, "ajaxintroduction/random_quotes.html")

Next our template; nothing more than a heading currently.

<!-- templates/ajaxintroduction/random_quotes.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Daily Quote!</title>

    <style>
        span {
            display: block;
        }
        span span {
            display: inline;
        }
    </style>
</head>
<body>

<h1>AJAX Examples</h1>
</body>

Then we hook in our view into our URL routing configuration.

# urls.py
from django.conf.urls import patterns, url

from . import views

urlpatterns = 
    patterns('',
    url(r'^$', views.get_quotes, name="index"),
)

Now hook our application into our project URL routing configuration:

#DjangoSandBox/urls.py

from django.conf.urls import patterns, include, url

from ajaxintroduction import urls as ajax_urls

urlpatterns = 
    patterns('',
             url(r'^ajax/', include(ajax_urls, namespace="ajax")),
             )

To test run the development server and navigate to:

http://127.0.0.1:8000/ajax/

You should see the template page loaded with our heading.

Static Files

Next we are going to configure our static files:

  • JQuery which we use for performing AJAX calls in JavaScript
  • JQuery-Cookies which will use to grab the CSRF token from our cookie
  • A new js file which we will place all our required JavaScript code in

First create an empty text file in ajaxintroduction/static/ajaxintroduction/js/ called ajax-example.js

// ajaxintroduction/static/ajaxintroduction/js/ajax-example.js

// Currently empty

Next download JQuery and JQueryCookies. You can download them from the references below if you have not downloaded the repository code of my examples. Alternatively you can download the files directly from GitHub

Note: if you are not downloading the files from my repository ensure you file names match with regards to the versions throughout the examples.

Copy in jquery and jquery.cookie into and DjangoSandBox/static/js

We now need to register these files as being static files which can be served directly to the user.

Within the settings file register the paths of the static directory of the project as well as the application directory of our ajaxintroduction application.

# settings.py
STATIC_URL = '/static/'

STATICFILES_DIRS =  (
    os.path.join(PROJECT_ROOT, 'static'),
    os.path.join(PROJECT_ROOT, 'ajaxintroduction', 'static'),
)

Now are are going to reference all these static files in our html head.

  • Call “load staticfiles” to enable the use of static files in our template
  • Reference the file with the static template tag. Files are referenced relative to the composition of STATICFILES_DIRS/STATIC_URL:
    • For our project files we reference js/file.js
    • For our application files we reference ajaxintroduction/js/file.js

Note: similar to templates we duplicate the application name within the directory path of our application static files as a way of name spacing our files.

<!-- templates/ajaxintroduction/random_quotes.html -->
<head>
    {% load staticfiles %}

    <script src="{% static "js/jquery-2.1.4.js" %}"></script>
    <script src="{% static "js/jquery.cookie.js" %}"></script>
    <script src="{% static "ajaxintroduction/js/ajax-example.js" %}"></script>    
</head>

To test run the development server and navigate to:

http://127.0.0.1:8000/ajax/

You should be able to use FireBug or Developer Tools to view the contents of the JavaScript files which have been downloaded along side our generated HTML.

If you look at the HTML code you will see the file path of the static file has been completed as a relative URL from our website route prefixed with static; static/ajaxintroduction/js/ajax-example.js

Note: if the files have not been downloaded use FireBug or Developer Tools to check the network requests of these resource files. Check the URL of the request is as expected. Check the response. Try entering in the full URL of the resource file into the browser:

http://127.0.0.1:8000/static/ajaxintroduction/js/ajax-example.js

Get – A Random Quote

We are now going to create the functionality to get a random quote. First we create a view which gets a random Quote record. We return the record as JSON data using the inbuilt Python json library:

import json
context_data = {random_quote.author: random_quote.quote}
json.dumps(context_data)

We then return the JSON data via a HTTPResponse setting the content_type to “application/json”. Add the following view function:

#views.py
import json
from random import randint

from django.http import HttpResponse

from .models import Quote

def get_random_quote(request):
    random_number = randint(0, Quote.objects.all().count() - 1)

    random_quote = Quote.objects.all()[random_number]
    context_data = {random_quote.author: random_quote.quote}

    return HttpResponse(json.dumps(context_data), content_type="application/json")

We then create a simple button and a span element to display our random quote. Both require an id as this is how we will be finding the element via JQuery. Add the following into our template file.

<!-- templates/ajaxintroduction/random_quotes.html -->

<h2>Get Random Quote:</h2>
<span>
    <button type="button" id="getquote">Get:</button>
    <span id="quote"></span><span id="author"></span>
</span>

Note: the above should be added into our existing html template file and not replacing its contents.

We now hook up our button to our view which returns our random quote as JSON data within our JavaScript file:

// ajaxintroduction/js/ajax-example.js
$(document).ready(function () {
    $("#getquote").click(function () {
        getQuote();
    });
});

function getQuote() {
    $.getJSON("/ajax/random", function (data) {
        console.log("getQuote.response:");
        console.log(data);
        $.each(data, function (author, quote) {
            $('#quote').text(quote);
            $('#author').text(author);
        });
    });
}
  • The $(document).ready is a special JQuery function which only runs when the html page has finished loading within the browser. We use it to bind our getQuote function onto the click event of our button.
  • The getJSON function is used to make a call to our view via AJAX. The first parameter is a relative URL which we will configure up below in our URL routing configuration. The second parameter is a call back function which is passed our returned random quote as JSON data.
  • We use the JQuery each function to iterate and unpack our JSON data which we display within our span elements,

We now hook in our view to our URL routing configuration.

# urls.py
from django.conf.urls import patterns, url

from . import views

urlpatterns = 
    patterns('',
    url(r'^random/$', views.get_random_quote, name="random"),
)

To test run the development server and navigate to:

http://127.0.0.1:8000/ajax/

Pressing the button will return and display a random quote.

Note: if the AJAX request does not appear to work use FireBug or Developer Tools to check the request. Check the URL, the parameters and it’s status. To see any errors which might be on the actual AJAX view navigate to the URL of the AJAX request.

You can also navigate directly to the new view function via:

http://127.0.0.1:8000/ajax/random

Get Multiple

In our next example we are going to allow the retrieval of all quotes for an author.

First we change our initial page view to return a list of all distinct authors.

Note: SQLite3 does not support the distinct SQL command so we use the Python set collection to create a distinct list of authors.

Secondly we create a new view which looks up all quotes for an author which we will pass in as a GET parameter. We return the quotes as a QuerySet.

The Django QuerySet is not serializable via the inbuilt Python json library. Django provides the serializers class which can.

#views.py
import json
from random import randint

from django.core import serializers
from django.http import HttpResponse, JsonResponse
from django.shortcuts import render

from .models import Quote

def get_quotes(request):
    authors = set(r.author for r in Quote.objects.all())

    return render(request, "ajaxintroduction/random_quotes.html", context={"authors": authors})


def get_author_quotes(request):
    if "author" not in request.GET or request.GET["author"] == "all":
        quotes = Quote.objects.all()
    else:
        author_name = request.GET["author"]
        quotes = Quote.objects.filter(author=author_name)

    return HttpResponse(serializers.serialize('json', quotes), content_type="application/json", )

We next now create our UI. This will consist of a choice list of the authors, a button and a span element for displaying the results.

<!-- templates/ajaxintroduction/random_quotes.html -->

<h2>Get Quotes By Author:</h2>
<span>
    <select id="authors" title="Fpop dfic">
        <option value="all">All</option>
        {% for author in authors %}
            <option value="{{ author }}">
                {{ author }}
            </option>
        {% endfor %}
    </select>
    <button type="button" id="getAuthorQuotes">Get</button>
</span>
<span id="quotesbyauthor"></span>

Now our JavaScript; first hook our function onto our button within the JQuery document ready function.

The only real difference between our call to getJSON in the previous example is that we pass in the selected author from the select box as a GET parameter. GET parameters are constructed from key value pairs represented by a JSON string.

Again we use the JQuery function ‘each’ to loop through all quotes unpacking them into a function as key value pair. We construct a series of list elements and add them into our span element.

// ajaxintroduction/js/ajax-example.js
$(document).ready(function () {
    $("#getAuthorQuotes").click(function () {
        getAuthorQuotes();
    });
});

function getAuthorQuotes() {
    $.getJSON("/ajax/author/", {author: $('#authors').val()}, function (data) {
            console.log("getAuthorQuotes.response:");
            console.log(data);

            var $quotesbyauthor = $('#quotesbyauthor');
            $quotesbyauthor.text("");

            var items = [];

            $.each(data, function (key, value) {
                items.push("<ul>" + value.fields["quote"] + ". " + value.fields["author"] + "</ul>");
            });

            $("<ul/>", {
                "class": "quotes",
                html: items.join("")
            }).appendTo($quotesbyauthor);
        }
    );
}

Now we simply need to hook in our view into our URL routing configuration.

# urls.py
from django.conf.urls import patterns, url

from . import views

urlpatterns = 
    patterns('',
    url(r'^author/$', views.get_author_quotes, )

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/ajax/

You can now ask for all quotes for an author. If you cannot get this to work check the requests being made along with the results. Try navigating to the URL directly.

Post -Create A Quote

There is no real fundamental difference between a HTTP POST or a GET. The method is used as a status; both can contain URL parameters and both can contain a body.

It is advisable that GET requests should never perform CRUD operations. We will use a POST to create our Quote record.

We are going to allow the user to create a quote asynchronously from our page. We will create a new view function which takes our quote and author as strings, ensures they are set and then attempts to create the quote. We will use get_or_create to ensure we don’t create duplicate quotes.

We simply use request.POST[“XXX”] to get our form element value on the server side. For more complicated examples you should think about writing a dedicated Django model form.

We then return an indication of whether the quote was created, a validation message and also the new number of quotes in the database. We return the data as JSON but using the JsonResponse; this will automatically serialise the data to JSON as well as setting the content type of the response to JSON.

# views.py
import json
from random import randint

from django.core import serializers
from django.http import HttpResponse, JsonResponse
from django.shortcuts import render

from .models import Quote

# @csrf_exempt # Use for debugging only.
def new_quote(request):
    message = ""

    is_valid = True
    created = False

    if "quote" not in request.POST or "author" not in request.POST:
        message = "Invalid post data"
        is_valid = False

    if request.POST["quote"] == "" or request.POST["author"] == "":
        message = "Please provide a quote and an author"
        is_valid = False

    if is_valid:
        quote, created = Quote.objects.get_or_create(quote=request.POST["quote"], author=request.POST["author"])

        if created:
            message = "Quote Created"
        else:
            message = "Quote already exists"

    data = {"count": Quote.objects.all().count(), "message": message, "created": created}

    return JsonResponse(data)

We now add a form onto our template which will be used to capture the data required to create our Quote record. We also add a button. We include a csrf_token in our form to prevent from cross site forgery attacks.

<!-- templates/ajaxintroduction/random_quotes.html -->

<h2>Create Quote:</h2>
<form method="post" action="">
    {% csrf_token %}
    <table>
        <tr>
            <th>Quote</th>
            <th>Author</th>
        </tr>
        <tr>
            <td><input type="text" id="txtquote"/></td>
            <td><input type="text" id="txtauthor"/></td>
        </tr>
    </table>
    <button type="button" id="newquote">New Quote</button>
</form>

We now hook in our JavaScript; this is where it starts to get a bit tricky as we need to post back our csrf_token as part of the POST data.

Django writes the token value to a cookie called csrftoken which we can access in JavaScipt via the JQuery-Cookies library. We hooked this file in above as a static file. We should only include this if our HTTP method is not one of GET, HEAD, OPTIONS or TRACE. We set the token in the X-CSRFToken HTTP request header parameter. We use the JQuery ajaxSetup function to do this.

If you want to test posting back without Django validating the presence and validity of the token you can decorate your view with the @csrf_exempt decorator. This should only be used for testing/development purposes.

We then call our new view URL passing in the data of the new quote we would like to create.

We then display our validation message and the new count of elements.

// ajaxintroduction/js/ajax-example.js
function csrfSafeMethod(method) {
    // these HTTP methods do not require CSRF protection
    return (/^(GET|HEAD|OPTIONS|TRACE)$/.test(method));
}

$.ajaxSetup({
    beforeSend: function (xhr, settings) {
        if (!csrfSafeMethod(settings.type) && !this.crossDomain) {
            xhr.setRequestHeader("X-CSRFToken", $.cookie('csrftoken'));
        }
    }
});

$(document).ready(function () {
    $("#newquote").click(function () {
        createNewQuote();
    });
});

function createNewQuote() {
    $.ajax({
        type: "POST",
        url: "/ajax/new/",
        data: {
            'quote': $('#txtquote').val(),
            'author': $('#txtauthor').val()
        },
        success: function (data) {
            console.log("createNewQuote.response:");
            console.log(data);

            alert(data.message);

            if (data.created) {
                alert("There are now " + data.count + " quotes");
            }
        }
    });
}

The last thing we need to do is to hook our new view into our URL routing configuration.

# urls.py
from django.conf.urls import patterns, url

from . import views

urlpatterns = 
    patterns('',
    url(r'^new/$', views.new_quote, name="new"),
)

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/ajax

##References## {#References}

Django: Testing

This post is part of my Django series. You can see an overview of the series along with instruction on how to get all the source code here.

This article assumes you are comfortable creating a Django project, can create apps and register them into the INSTALLED_APPS list of the settings file. If not please read my Django HelloWorld article.

Django uses the Python package unittest. This article assumes that you have a basic knowledge of creating unittests with this package. If this is not the case please see my blog post on unittest.

This article assumes you have a project called DjangoSandBox in an application called testintroduction.

This article tests the various parts we have built up over the series; please see here for any explanations of the code we are testing.

Testing Models

Models are pretty easy to test; our first example is using the HelloWorld class we have created previously. Nothing special, a name and date field, a __str()__ function and a Django get_absolute_url function.

#helloworld/models.py
from django.db import models
from django.core.urlresolvers import reverse

class HelloWorld(models.Model):
    first_name = models.CharField(max_length=20)
    now = models.DateTimeField()

    def __str__(self):
        return "{0} {1}".format(self.first_name, self.now)

    @staticmethod
    def get_absolute_url():
        return reverse('helloworld:model')

Testing these functions is nothing more than creating an instance of the class, calling the require function or property and asserting that our result is as required.

#test_models.py

from datetime import date
from django.test import TestCase
from helloworld.models import HelloWorld

class TestHelloWorld(TestCase):
    def setUp(self):
        self.first_name = "Luke"
        self.now = date.today()

        self.a_helloworld = HelloWorld(first_name=self.first_name, now=self.now)

    def test_str(self):
        self.assertEqual("{0} {1}".format(self.first_name, self.now), str(self.a_helloworld))

    def test_get_absolute_url(self):
        self.assertEqual("/helloworld/model/", self.a_helloworld.get_absolute_url())

Our model could contain validation. We can test validation by creating an instance of the model in an invalid state and calling full_clean. If the model is invalid an exception is raised with all the errors contained within a dictionary upon the message_dict property.

# test_models.py
from datetime import date
from django.core.exceptions import ValidationError
from django.test import TestCase
from modelsadvanced.models import ContactDetails, ProxyChild

class TestModelValidation(TestCase):

    def test_model_validation(self):

        try:
            a_contact = ContactDetails(name="Luke", age=101, contactDate=date(year=2015, month=7, day=2))
            a_contact.full_clean()
        except ValidationError as e:

            self.assertEqual({'name': ['Ensure this value has at least 10 characters (it has 4).', 'Luke is barred'],
                              'age': ['Ensure this value is less than or equal to 100.'],
                              'contactDate': ['2015-07-02 is not a future date.']}, e.message_dict)

If we want to test class meta data we can access this via the _meta property:

#test_models.py
class TestModelMeta(TestCase):
    def test_model_meta(self):
        self.assertEqual(True, ProxyChild()._meta.proxy)

We can use list and map to get a list of all field names contained within the fields property of the meta class.:

#test_models.py
class TestModelFields(TestCase):
    def test_model_meta(self):
        field_names = list(map((lambda x: x.name), ContactDetails._meta.fields))

        self.assertEqual(4, len(ContactDetails._meta.fields))
        self.assertEqual(['id', 'age', 'name', 'contactDate'], field_names)

Testing Views

When we test views we can use the test client; it acts as a dummy Web browser. We access it via the client property of our unit test class.

We call get upon the test client along with the URL.

response = self.client.get("/index")

We can use reverse to access the URL via our routing config. Don’t worry we will explicitly test the routing config later/

response = self.client.get(reverse("home:home"))

The response.status_code can be used to assure our response code is as expected.

The response.resolver_match can be use to ensure that the view function used was as expected.

The function self.assertTemplateUsed can be used to assure that the correct template was used.

We can assert that the html is as expected by calling self.assertContains along with sections of html or strings which should be contained within the rendered HTML.

#test_views.py

from django.core.urlresolvers import reverse
from django.test import TestCase
from home import views as home_views

class TestFunctionView(TestCase):
    def test_function_view(self):
        response = self.client.get(reverse("home:home"))

        self.assertEqual(200, response.status_code)
        self.assertEqual(home_views.home, response.resolver_match.func)
        self.assertTemplateUsed(response, template_name='home/home.html')

        self.assertContains(response, '<a href="/helloworld/index/">Hello World</a>')
        self.assertContains(response, '<a href="/templatesintroduction/index">Templates</a>')
        self.assertContains(response, '<a href="/formsintroduction/basic/">Forms</a>')
        self.assertContains(response, '<a href="/viewsintroduction/contact/list">Views</a>')

We can access the context data which was passed to the view via the context property of the response.

#test_views.py

from django.core.urlresolvers import reverse
from django.test import TestCase
from helloworld import views as helloworld_views

class TestContextData(TestCase):
    def test_context_data(self):
        response = self.client.get(reverse("helloworld:template"))

        self.assertEqual('Obelix', response.context["who"])
        self.assertEqual(False, response.context["islong"])

Testing Class Based Views

In the example two up we used the following code to ensure the correct view function was used:

self.assertEqual(home_views.home, response.resolver_match.func)

If we want to assure that the correct class view was used we need to compare the name of the class against that which was used:

self.assertEqual(PhoneAddressListView.as_view().__name__, response.resolver_match.func.__name__)

Where a base template is used we can simply call the function assertTemplateUsed with the base template name.

In the example below we test the list view with and without records to ensure our “no records here” message is displayed to the user.

When records are returned to our template we have a few ways we can test the results along with the assertContains function we have seen in the previous examples.

  • Check the record count of the object list in the contect view
  • Compare elements
  • Compare the query set. To do this we need to convert the expected query set to a list of strings from the repr function as this is what is contained in the response.
#test_views.py

class TestClassView(TestCase):
    def test_with_no_records(self):
        response = self.client.get(reverse("viewsintroduction:address_list"))

        self.assertEqual(200, response.status_code)
        self.assertEqual(PhoneAddressListView.as_view().__name__, response.resolver_match.func.__name__)

        self.assertTemplateUsed(response, template_name='viewsintroduction/base.html')
        self.assertTemplateUsed(response, template_name='viewsintroduction/phoneaddresslist.html')

        self.assertEqual(0, len(response.context["phoneaddress_list"]))
        self.assertContains(response, '<tr><td colspan="6">There are no addresses</td></tr>')

    def test_with_one_record(self):
        an_address = PhoneAddress(number=1, street_name="A", city="A City")
        an_address.save()

        response = self.client.get(reverse("viewsintroduction:address_list"))

        self.assertEqual(200, response.status_code)
        self.assertEqual(PhoneAddressListView.as_view().__name__, response.resolver_match.func.__name__)

        self.assertTemplateUsed(response, template_name='viewsintroduction/base.html')
        self.assertTemplateUsed(response, template_name='viewsintroduction/phoneaddresslist.html')

        context_addresses = response.context['phoneaddress_list']
        expected_addresses = [repr(r) for r in PhoneAddress.objects.all()]

        self.assertEqual(1, len(context_addresses))
        self.assertEqual(an_address, context_addresses.first())

        self.assertQuerysetEqual(context_addresses, expected_addresses, ordered=False)

        self.assertContains(response, an_address.city)
        self.assertContains(response, an_address.number)
        self.assertContains(response, an_address.street_name)

        self.assertNotContains(response, '<tr><td colspan="6">There are no addresses</td></tr>')

Testing Errors

We now move on to test some of our error views; by comparing the status code.

#test_views.py

class Test404Error(TestCase):
    def test_404_error_is_raised(self):
        response = self.client.get(reverse("helloworld:error_as_404"))
        self.assertEqual(404, response.status_code)

class TestNotAllowed(TestCase):
    def test_not_allowed(self):
        response = self.client.get(reverse("helloworld:not_allowed"))

        self.assertEqual(405, response.status_code)


class TestOnlyPost(TestCase):
    def test_raises_error_on_get(self):
        response = self.client.get(reverse("helloworld:error_if_not_post"))

        self.assertEqual(405, response.status_code)

    def test_all_ok_on_post(self):
        response = self.client.post(reverse("helloworld:error_if_not_post"), {'name': 'Jim'})

        self.assertEqual(200, response.status_code)
        self.assertContains(response, "{0}, this can only be called with a post".format("Jim"))

When we expect a redirect is made we can use the assertRedirects function to validate this. The function takes the status codes of our page load and also the page we direct to. 301 indicates that this is a page which has moved permanently.

#test_views.py

class TestRedirect(TestCase):
    def test_redirect(self):
        response = self.client.post(reverse("viewsintroduction:redirect_view"))

        self.assertTrue(301, response.status_code)
        self.assertRedirects(response, "http://djangoproject.com", 301, 200)  # Status codes Us/Them

Test URLS

When we test URLs we should test two things;

  • The reverse function returns the correct URL string
  • The resolve function returns the correct view function. This does not only include the function and URL name but the app name of the view, the URL namespace and any arguments which will be passed into the view function.
#test_urls.py

from django.core.urlresolvers import reverse, resolve
from django.test import TestCase

from home import views

class TestHomeUrls(TestCase):
    urls = 'home.urls'

    def test_reverse(self):
        self.assertEqual('/', reverse('home'))

    def test_resolve(self):
        resolver = resolve(reverse('home'))

        self.assertEqual(views.home, resolver.func)
        self.assertEqual('home', resolver.url_name)

        self.assertEqual((), resolver.args)
        self.assertEqual({}, resolver.kwargs)
        self.assertEqual(None, resolver.app_name)
        self.assertEqual('', resolver.namespace)

Note: Setting urls = ‘home.urls’ upon the test class above defines the URL routing config we are using. Here we test only an apps config which explains why we reference the URL by their name only without the namespace which we set in the project’s urls.py

When our URL config contains get query parameters we can pass them into the reverse function with the kwargs parameter which is a dictionary.

#test_urls.py

class TestViewsIntroductionUrls(TestCase):
    urls = 'viewsintroduction.urls'

    def test_reverse(self):
        self.assertEqual('/address/1/update/', reverse('address_update', kwargs={'pk': 1}))

Testing Forms

When testing forms we pass the input data as a dictionary into the constructor.

We can access the form fields via the fields property; below we ensure that we have the correct fields and that they contain the right configuration.

We can assert that the form validates as expected with form.is_valid(). Any errors raised will be found within the property errors on the form which is a dictionary of errors. Field errors will be found under a key named after the field name; it returns a list of errors.

#test_forms.py

from django.core.urlresolvers import reverse
from django.forms import DecimalField
from django.test import TestCase

from formsintroduction.forms.bastic_form_example import BasicFormExample
from formsintroduction.forms.class_form_example import ClassBasedForm
from viewsintroduction.models import PhoneAddress


class TestBasicForm(TestCase):
    def test_fields(self):
        form = BasicFormExample({})

        self.assertEqual(2, len(form.fields))
        self.assertTrue("height" in form.fields.keys())
        self.assertTrue("name" in form.fields.keys())
        self.assertIsInstance(form.fields['height'], DecimalField)
        self.assertEqual(2, form.fields['height'].max_value)

    def test_valid_data(self):
        form = BasicFormExample({"name": "Luke", "height": 1.11})

        self.assertTrue(form.is_valid())

    def test_invalid_data(self):
        form = BasicFormExample({"name": "Luke"})

        self.assertFalse(form.is_valid())
        self.assertTrue("name" not in form.errors.keys())
        self.assertTrue("height" in form.errors.keys())

        self.assertEqual(form.errors["height"], ['This field is required.'])

Testing Class Derived Forms

Class based views are tested in pretty much the same way. However when we call save we get an instance of the record created which we can test against.

In the example below we test the form meta data to ensure that the correct fields are set along with the right model class.

#test_forms.py

class TestClassBasedForm(TestCase):
    def test_valid_data(self):
        form = ClassBasedForm({"number": 123, "street_name": "The high street", "city": "Bristol"})

        self.assertTrue(form.is_valid())

        a_record = form.save()

        self.assertEqual(123, a_record.number)
        self.assertEqual("The high street", a_record.street_name)
        self.assertEqual("Bristol", a_record.city)

    def test_invalid_data(self):
        form = ClassBasedForm({})

        self.assertEqual(form.errors["number"],
                         ['We need a number of the house!', 'None of the fields have been set!!!'])

    def test_meta(self):
        self.assertEqual(PhoneAddress, ClassBasedForm.Meta.model)
        self.assertEqual(("city", "street_name", "number"), ClassBasedForm.Meta.fields)

Posting Forms

We use the test client to test posting a form to the server. We pass the form data as a dictionary along with the URL.

#test_forms.py

    def test_post_invalid(self):
        response = self.client.post(reverse("formsintroduction:form_class_create"),
                                    {"city": "Word_Word", "street_name": "street", "number": 1},
                                    follow=True)

        self.assertEqual(200, response.status_code)
        self.assertFormError(response, 'form', 'city', ["Name must be a capitalised single word"])

    def test_post_valid(self):
        response = self.client.post(reverse("formsintroduction:form_class_create"),
                                    {"city": "Foo", "street_name": "Foo", "number": 1},
                                    follow=True)

        self.assertEqual(200, response.status_code)
        self.assertRedirects(response, "http://testserver/viewsintroduction/address/1/", 302)

Testing An Authorised View

Unsuccessful Authentication

The Django authorisation system will automatically redirect a user to the registered login page when they try to access a view which requires authentication. Our first test case assures that we are redirected successfully.

from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.test import TestCase


class TestAuthedView(TestCase):
    def test_redirect_to_loging(self):
        response = self.client.post(reverse("auth:users"))

        self.assertTrue(302, response.status_code)
        self.assertRedirects(response, "auth/login/?next=/auth/users/", 302, 200)

In the above example the 302 and 200 parameters passed into assertRedirects are status codes of the HTTP for the initial page and then the redirect page. 302 is a redirect while 200 is a successful request.

Successful Authentication

The Django test web client comes with the ability to log a user in:

self.client.login(username='username', password='password')

Note: don’t forget that when we are running tests we start with a blank database so you will need to create a login first. The example below creates a new user and activates it in line; this should be moved to a test fixture set-up function.

from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.test import TestCase

class TestAuthedView(TestCase):

    def test_can_login(self):
        user = User.objects.create_user(username='FOOTESTFOO', password='pwd')
        user.is_active = True
        user.save()

        login = self.client.login(username='FOOTESTFOO', password='pwd')

        self.assertTrue(login)

        response = self.client.post(reverse("auth:users"))

        self.assertTrue(200, response.status_code)
        self.assertContains(response, "Registered Users")
        self.assertContains(response, "First Name")
        self.assertContains(response, "Last Name")
        self.assertContains(response, "FOOTESTFOO")

Testing Cookies

We can access cookies from the cookies property of the Django test web client once we have called post or get. It is a dictionary keyed upon the cookie name:

self.client.cookies
self.client.cookies["hits"]

The cookie is returned as a http.cookies.Morsel. We can access the cookie value with the value property. The class responds as a dictionary for other data, in the following example we ensure the expiry has been set correctly with the max-age key.

from django.core.urlresolvers import reverse
from django.test import TestCase

class TestSessions(TestCase):
    def test_cookie(self):
        response = self.client.post(reverse("sessionsandcookies:cookies"))

        self.assertTrue(200, response.status_code)

        self.assertTrue("hits" in self.client.cookies)
        self.assertEqual('1', self.client.cookies["hits"].value)
        self.assertEqual(30, self.client.cookies["hits"]["max-age"])

Testing Session Data

We can access session data upon the Django test web client after we have called post or get. It returns a SessionBase which responds as a dictionary to retrieve the session data from their key names. All functionality of the SessionBase class is available; below we call get_expiry_age to ensure our session expiry policy has worked as expected:

from django.core.urlresolvers import reverse
from django.test import TestCase


class TestSessions(TestCase): 

    def test_sessions(self):
        response = self.client.post(reverse("sessionsandcookies:sessions"))

        self.assertTrue(200, response.status_code)

        self.assertTrue("has_visited" in self.client.session)
        self.assertEqual('True', self.client.session["has_visited"])
        self.assertEqual(10, self.client.session.get_expiry_age())

Additional Assertions

As mentioned previously, Django provides some additional assert functions to the standard Python unittest. class We have covered many of them above though there are two others which I think will be quite useful.

The assertFieldOutput assertion allows us to validate against good and bad field input data. I tried to get this to work against the defined fields or a model or form however it seems to only work with the class type.

The assertNumQueries assertion will ensure that the database has been touched the correct number of times.

#test_additionl_assertions.py

from django.forms import EmailField, CharField
from django.test import TestCase

from viewsintroduction.models import PhoneAddress


class TestAdditionalAssertions(TestCase):
    def test_assertFieldOutput(self):
        self.assertFieldOutput(EmailField, {'a@a.com': 'a@a.com'}, {'aaa': ['Enter a valid email address.']})

        self.assertFieldOutput(CharField, {'aa': 'aa'},
                               {'word': ['Ensure this value has at most 3 characters (it has 4).'],
                                'a': ['Ensure this value has at least 2 characters (it has 1).']}, [],
                               {"max_length": 3, "min_length": 2})

    def test_assert_num_queries(self):
        with self.assertNumQueries(1):
            PhoneAddress.objects.create(city="Foo", street_name="Foo", number=1)

        self.assertNumQueries(1, lambda: PhoneAddress.objects.create(city="Foo", street_name="Foo", number=1))

References

Sessions & Cookies

This post is part of my Django series. You can see an overview of the series along with instruction on how to get all the source code here.

This article assumes you are comfortable creating a Django project, can create apps and register them into the INSTALLED_APPS list of the settings file. If not please read my Django HelloWorld article.

This article assumes you have a project called DjangoSandBox in an application called sessionsandcookies.

Sessions

By default Django comes with the sessions middleware and applications installed:

#settings.py

# Application definition
INSTALLED_APPS = (
    'django.contrib.sessions',
)

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
)

Note: If the settings were not present or you are starting a new project, ensure you have run migrate to create all the required schema and run all the required configuration scripts:

python manage.py migrate

Setting

Session data is stored as key value pairs within request.session which behaves as a dictionary. We set a value with [“key”] = value and get a value with get(“key”).

Session data is only valid for a certain amount of time. This is set to SESSION_COOKIE_AGE which defaults to two weeks.

We can use the get_expiry_date function to see the expiry date of the session data.

We can set the expiry with the set_expiry function which takes a time in seconds.

The following uses session data to set mark that a user has visited a page.

#views.py

from datetime import datetime

def sessions(request):
    context = {
        "has_visited": request.session.get("has_visited", False),
        "visited": request.session.get("visited", None),
        "expiry": request.session.get_expiry_date().strftime("%x %X")
    }

    if "has_visited" not in request.session:
        request.session['has_visited'] = str(True)
        request.session['visited'] = datetime.now().strftime("%x %X")
        request.session.set_expiry(10)  # Time in seconds. 0 expires when the browser is closed

    return render(request, 'sessionsandcookies/sessions.html', context)

A simple template to display the data we store and retrieve from the session data.

<!-- templates/sessionsandcookies/sessions.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Sessions</title>
</head>
<body>
<h1>Sessions</h1>

{% if has_visited %}
    <span>Visited: {{ visited }}</span>
    <span>Expires: {{ expiry }}</span>
{% else %}
    First Page Land
{% endif %}

<span>
    <a href="{% url 'sessionsandcookies:sessions_clear' %}">Clear</a>
</span>
</body>
</html>

Hooking in our view function into the URL routing config:

# urls.py
url(r'^sessions/$', views.sessions, name="sessions"),

Note: don’t forget to hook our app into the project URL routing config.

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/sessions/

Clearing

We can use the clear function upon request.session to remove all session data. In this example we clear the session data and then redirect to the page in the previous example to prove we revert to the default value.

#view.py
def sessions_clear(request):
    if 'has_visited' in request.session:
        del request.session['has_visited']

    request.session.clear()

    return redirect(reverse("sessionsandcookies:sessions"))

Note: Above we call delete upon the session data element as well as clear; we can do both depending upon what we require.

Hooking in our view function into the URLrouting config.

#urls.py
url(r'^sessions/clear/$', views.sessions_clear, name="sessions_clear"),

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/sessions/clear/

Cookies

Not all browsers accept cookies so our first example is a way to test if the browser accepts cookies. We call set_test_cookie and then test_cookie_worked. The latter returns true if we can create cookies.

Testing

#views.py
def cookies_test(request):
    can_create_cookie = False
    request.session.set_test_cookie()

    if request.session.test_cookie_worked():
        request.session.delete_test_cookie()
        can_create_cookie = True

    return render(request, "sessionsandcookies/cookies_test.html", {"can_create_cookie": can_create_cookie})

A simple view which displays our result.

<!-- templates/sessionsandcookies/cookies_test.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Test Cookie</title>
</head>
<body>
<h1>Test Cookie</h1>

{% if can_create_cookie %}
    <span>Test cookie worked</span>
{% else %}
    <span>Test cookie did not work</span>
{% endif %}

<span>
    <a href="{% url 'sessionsandcookies:cookies' %}">Back</a>
</span>
</body>
</html>

Hooking in our function view into the URL routing config.

#urls.py
url(r'^cookies/test$', views.cookies_test, name="cookies_test"),

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/cookies/test

Setting

Setting cookie information is a little clunky in Django; we need to call set_cookie on the response object though we have to render our template to get hold of the response. We call render_to_response before calling set_cookies upon the returned response object.

#views.py
def cookies(request):
    hits = int(request.COOKIES.get('hits', '0'))
    first_hit = 'hits' not in request.COOKIES
    response = render_to_response('sessionsandcookies/cookies.html',
                                  {"first_hit": first_hit, "hits": hits}, RequestContext(request))

    response.set_cookie('hits', hits + 1, max_age=30)
    return response

A simple template view displays the number of hits which is passed into our template.

<!-- templates/sessionsandcookies/cookies.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Cookies</title>
</head>
<body>

<h1>Cookies</h1>

<span>
{% if first_hit %}
    Your first hit!
{% else %}
    Hits: {{ hits }}
{% endif %}
</span>

</body>
</html>

Hooking in our view function into our URL routing config.

#urls.py
url(r'^cookies/$', views.cookies, name="cookies"),

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/cookies/

Expiring

We can expire our cookie by calling delete_cookie along with our cookie name.

#views.py
def cookies_expire(request):
    response =         render_to_response('sessionsandcookies/cookies_clear.html',
            {"first_hit": True, "hits": 0}, RequestContext(request))

    response.delete_cookie('hits')
    return response

A simple template which tells the user the cookies have been cleared along with a URL back to our page which shows the cookies data. We can then see the cookie hit count is reset to ‘first hit!’.

<! templates/sessionsandcookies/cookies_clear.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Cookie Clear</title>
</head>
<body>
<h1>Cookie Clear</h1>

<span>The cookie was cleared</span>

<span>
    <a href="{% url 'sessionsandcookies:cookies' %}">Back</a>
</span>
</body>
</html>

Hooking in our view function into our URLrouting config.

#urls.py
url(r'^cookies/expire', views.cookies_expire, name="cookies_expire"),

To test run the development server and navigate to:

http://127.0.0.1:8000/sessionsandcookies/cookies/expire/

References

Django: Errors and Logging

This post is part of my Django series. You can see an overview of the series along with instruction on how to get all the source code here.

This article assumes you are comfortable creating a Django project, can create apps and register them into the INSTALLED_APPS list of the settings file. If not please read my Django HelloWorld article.

This article assumes you have a project called DjangoSandBox in an application called errorsandlogging.

Errors

We can enforce HTTP GET/POST methods upon a view function with the @require_http_methods decorator. Access denied error methods are raised for other HTTP methods.

#views.py
from django.views.decorators.http import require_http_methods

@require_http_methods(["GET", "POST"])
def error_if_not_get_or_post(request):
    return HttpResponse("This can only be called with a get or a post")

If we only want HTTP POST calls to our view function we can use the @require_POST decorator; access denied errors are raised for other HTTP methods.

#views.py

from django.views.decorators.http import require_POST

@require_POST
def error_if_not_post(request):
    return HttpResponse("{0}, this can only be called with a post".format(request.POST.get("name", "")))

If we only want HTTP GET method calls to our view function we can use the @require_GET decorator; access denied errors are raised for other http methods.

#views.py

from django.views.decorators.http import require_GET

@require_GET
def error_if_not_get(request):
    return HttpResponse("This can only be called with a get")

We can return a HTTP 404 error (response not found) by raising a Http404 error. This will use the 404 html file placed within the site templates directory.

#views.py
from django.http import Http404

def error_as_404(request):
    raise Http404("There is nothing to see here")

We can also return a custom HTTP 404 error by returning an instance of HttpResponseNotFound. This is similar to raising a Http404 but allows us to use custom html rather than the stock site 404 page.

#views.py

from django.http import HttpResponseNotFound

def error_as_custom_404(request):
    return HttpResponseNotFound('<h1>There is nothing to see here</h1>')

We can return a HTTP 405 error (response not allowed) by returning an instance of HttpResponseNotAllowed.

#views.py
from django.http import HttpResponseNotAllowed

def not_allowed(request):
    return HttpResponseNotAllowed("<h1>You are not allowed to see this</h1>")

Logging

This assumes that you have a basic working knowledge of Python logging. If this is not the case then you can come up to speed with my post here.

The logging config is defined within the settings config of the Django project; any instances of the logger will then automatically pick these settings up.

Below we create two loggers as defined within the ‘handlers’ section.

  • A terminal logger which will log anything with a level of DEBUG or higher; it is called ‘console’.
  • A file logger which will log anything with a level of ERROR or higher; it is called ‘file’.

We create two formatters and assign them to a logger via the formatter variable.

  • A simple formatter which will prefix our log message with the time and the log level. It is called ‘simple’ and is assigned to the console logger.
  • A verbose formatter which will prefix our log message with the level, time, module, process id and thread id. It is called verbose and is assigned to the file logger.

The loggers or handlers are then assigned in the loggers section to our helloworld app.

#settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(asctime)s %(levelname)s %(message)s'
        }
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
        'file': {
            'level': 'ERROR',
            'class': 'logging.FileHandler',
            'filename': 'debug.log',
            'formatter': 'verbose'
        },
    },
    'loggers': {
        'helloworld': {
            'handlers': ['console', 'file'],
            'level': 'WARNING',
            'propagate': True,
        },
    },
}

The following is a test view to check our custom Django log config is working. We throw a log message of each logging level at the logger as well as a trace stack and an exception.

#views.py

import logging
from django.http import HttpResponse

logger = logging.getLogger(__name__)

def with_logger(request):
    logger.debug('This is some debug')
    logger.info('This is information')
    logger.warning('This is a warning')
    logger.error('This is a error')
    logger.critical('This is a critical')

    logger.critical("Output with trace", exc_info=True)

    try:
        boom = 1 / 0
        logger.debug("Looks like we have broke the compiler as 1/0 = {0}".format(boom))
    except ZeroDivisionError as e:
        logger.exception(e)

    logger.critical('Final output to show we are logging still')

    return HttpResponse("Logged output")

The logging.getLogger function call passes in the name of the file; this ensures that we will only ever use one logger instance for any view function within this file; as long as they use the logger returned into the logger variable.

We can hook this view into our URL routing config.

#urls.py

from django.conf.urls import patterns, url

from . import views

urlpatterns = 
    patterns('',
             url(r'^logger/$', views.with_logger, name="logger")
             )

We hook into our errorsandlogging application into our project URL routing config:

# DjangoSandBox/urls.py
from django.conf.urls import patterns, include, url

from django.contrib import admin

from errorsandloggingimport urls as errorsandlogging_urls

urlpatterns = 
    patterns('',
url(r'^errorsandlogging/', include(errorsandlogging_urls, namespace="errorsandlogging")),
             )

To test run the development server and navigate to:

http://127.0.0.1:8000/errorsandlogging/logger/

References

Python: Parameter Scope

This is part of my Python & Django Series which can be found here including information on how to download all the source code.

Variables can be local or global.

Global Variables exist at the top level; i.e they are not inside a class or a method. They are implicitly global.

Local Variables are any variables inside a method. They will shadow a variable with the same name in an outer scope, i.e the calling method or class class etc. Changes to a local variable does not affect any variables which they shadow.

Global keyword is used to import a global variable. It allows a global variable to be used and set inside a method. It can only import global variables and not outer scope variables such as those within an outer method or class.

Nonlocal keyword is similar to global but will import any variables which are in an outer scope regardless if they are global or not global.

The following code and output can be used to demonstrate the usage of the above concepts.

Code:

def print_scope_msg(a_msg):
    print("Inside scope_test:", a_msg)

msg = "Initial"

def scope_test():

    def local_test():
        # msg is local. Changes are only existing inside the method
        msg = "local"
        print("Inside local_test():", msg)

    def nonlocal_test():
        # msg is non local. This allows the msg inside scope_test() to be imported and edited
        nonlocal msg
        msg = "nonlocal"
        print("Inside nonlocal_test:", msg)

    def global_test():
        # msg is global. This allows the msg inside global but not scope_test() to be imported and edited
        # Changes to msg affect the global msg but not the scope_test (it skips it)
        global msg
        msg = "global"
        print("Inside global_test: ", msg)

    msg = "scope"

    local_test()
    print_scope_msg(msg)

    nonlocal_test()
    print_scope_msg(msg)

    global_test()
    print_scope_msg(msg)

scope_test()
print("Inside global:", msg)

Output:

1: Inside local_test(): local
2: Inside scope_test: scope
3: Inside nonlocal_test: nonlocal
4: Inside scope_test: nonlocal
5: Inside global_test: global
6: Inside scope_test: nonlocal
7: Inside global: global

So what happened? Each numbered output above has an explanation as described below.

  1. Reports “local” as we have changed msg to local.
  2. Reports “scope” as the change inside local_test() was made to a shadowed version of msg which does not affect the value of msg inside scope_test().
  3. Reports “nonlocal” as we have changed msg to nonlocal.
  4. Reports “nonlocal” as inside nonlocal_test() we imported msg with the nonlocal keyword before changing the value.
  5. Reports “globaltest” as we have changed msg to globaltest.
  6. Reports “nonlocal” as the change to msg in global_test() used the global keyword which imported the global reference of msg and not that defined within scope_test() where we are now.
  7. Reports “globaltest” as the change to msg in global_test() used the global keyword.