Introduction and Audience

It's been over two years since I wrote the first version of this tutorial. I decided to give it another run with some of the tools that have come about since then (particularly WebOb).

Sometimes Python is accused of having too many web frameworks. And it's true, there are a lot. That said, I think writing a framework is a useful exercise. It doesn't let you skip over too much without understanding it. It removes the magic. So even if you go on to use another existing framework (which I'd probably advise you do), you'll be able to understand it better if you've written something like it on your own.

This tutorial shows you how to create a web framework of your own, using WSGI and WebOb. No other libraries will be used.

What Is WSGI?

At its simplest WSGI is an interface between web servers and web applications. We'll explain the mechanics of WSGI below, but a higher level view is to say that WSGI lets code pass around web requests in a fairly formal way. That's the simplest summary, but there is more -- WSGI lets you add annotation to the request, and adds some more metadata to the request.

WSGI more specifically is made up of an application and a server. The application is a function that receives the request and produces the response. The server is the thing that calls the application function.

A very simple application looks like this:

>>> def application(environ, start_response):
...     start_response('200 OK', [('Content-Type', 'text/html')])
...     return ['Hello World!']

The environ argument is a dictionary with values like the environment in a CGI request. The header Host:, for instance, goes in environ['HTTP_HOST']. The path is in environ['SCRIPT_NAME'] (which is the path leading up to the application), and environ['PATH_INFO'] (the remaining path that the application should interpret).

We won't focus much on the server, but we will use WebOb to handle the application. WebOb in a way has a simple server interface. To use it you create a new request with req = webob.Request('http://localhost/test'), and then call the application with resp = req.get_response(app). For example:

>>> from webob import Request
>>> req = Request.blank('http://localhost/test')
>>> resp = req.get_response(application)
>>> print resp
200 OK
Content-Type: text/html
<BLANKLINE>
Hello World!

This is an easy way to test applications, and we'll use it to test the framework we're creating.

About WebOb

WebOb is a library to create a request and response object. It's centered around the WSGI model. Requests are wrappers around the environment. For example:

>>> req = Request.blank('http://localhost/test')
>>> req.environ['HTTP_HOST']
'localhost:80'
>>> req.host
'localhost:80'
>>> req.path_info
'/test'

Responses are objects that represent the... well, response. The status, headers, and body:

>>> from webob import Response
>>> resp = Response(body='Hello World!')
>>> resp.content_type
'text/html'
>>> resp.content_type = 'text/plain'
>>> print resp
200 OK
Content-Length: 12
content-type: text/plain; charset=UTF-8
<BLANKLINE>
Hello World!

Responses also happen to be WSGI applications. That means you can call resp(environ, start_response). Of course it's much less dynamic than a normal WSGI application.

These two pieces solve a lot of the more tedious parts of making a framework. They deal with parsing most HTTP headers, generating valid responses, and a number of unicode issues.

Serving Your Application

While we can test the application using WebOb, you might want to serve the application. Here's the basic recipe, using the Paste HTTP server:

1
2
3

if __name__ == '__main__':
    from paste import httpserver
    httpserver.serve(app, host='127.0.0.1', port=8080)

you could also use wsgiref from the standard library, but this is mostly appropriate for testing as it is single-threaded:

1
2
3
4

if __name__ == '__main__':
    from wsgiref import simple_server
    server = make_server('127.0.0.1', 8080, app)
    server.serve_forever()

Making A Framework

Well, now we need to start work on our framework.

Here's the basic model we'll be creating:

• We'll define routes that point to controllers
• We'll create a simple framework for creating controllers

Routing

We'll use explicit routes using URI templates (minus the domains) to match paths. We'll add a little extension that you can use {name:regular expression}, where the named segment must then match that regular expression. The matches will include a "controller" variable, which will be a string like "module_name:function_name". For our examples we'll use a simple blog.

So here's what a route would look like:

1
2
3
4
5
6
7
8
9

app = Router()
app.add_route('/', controller='controllers:index')
app.add_route('/{year:\d\d\d\d}/',
              controller='controllers:archive')
app.add_route('/{year:\d\d\d\d}/{month:\d\d}/',
              controller='controllers:archive')
app.add_route('/{year:\d\d\d\d}/{month:\d\d}/{slug}',
              controller='controllers:view')
app.add_route('/post', controller='controllers:post')

To do this we'll need a couple pieces:

• Something to match those URI template things.
• Something to load the controller
• The object to patch them together (Router)

>>> import re
>>> var_regex = re.compile(r'''
...     \{          # The exact character "{"
...     (\w+)       # The variable name (restricted to a-z, 0-9, _)
...     (?::([^}]+))? # The optional :regex part
...     \}          # The exact character "}"
...     ''', re.VERBOSE)
>>> def template_to_regex(template):
...     regex = ''
...     last_match = None
...     for match in var_regex.finditer(template):
...         if last_match is None:
...             regex += re.escape(template[:match.start()])
...         else:
...             regex += re.escape(template[last_match.end():match.start()])
...         expr = match.group(2) or '[^/]+'
...         expr = '(?P<%s>%s)' % (match.group(1), expr)
...         regex += expr
...         last_match = match
...     if not last_match:
...         # There were no {} groups
...         regex = re.escape(template)
...     else:
...         regex += re.escape(template[match.end():])
...     regex = '^%s$' % regex
...     return regex

>>> print template_to_regex('/a/static/path')
^\/a\/static\/path$
>>> print template_to_regex('/{year:\d\d\d\d}/{month:\d\d}/{slug}')
^\/(?P<year>\d\d\d\d)\/(?P<month>\d\d)\/(?P<slug>[^/]+)$

>>> import sys
>>> def load_controller(string):
...     module_name, func_name = string.split(':', 1)
...     __import__(module_name)
...     module = sys.modules[module_name]
...     func = getattr(module, func_name)
...     return func

>>> from webob import Request
>>> from webob import exc
>>> class Router(object):
...     def __init__(self):
...         self.routes = []
...
...     def add_route(self, template, controller, **vars):
...         if isinstance(controller, basestring):
...             controller = load_controller(controller)
...         self.routes.append((re.compile(template_to_regex(template)),
...                             controller,
...                             vars))
...
...     def __call__(self, environ, start_response):
...         req = Request(environ)
...         for regex, controller, vars in self.routes:
...             match = regex.match(req.path_info)
...             if match:
...                 req.urlvars = match.groupdict()
...                 req.urlvars.update(vars)
...                 return controller(environ, start_response)
...         return exc.HTTPNotFound()(environ, start_response)

Controllers

The router just passes the request on to the controller, so the controllers are themselves just WSGI applications. But we'll want to set up something to make those applications friendlier to write.

To do that we'll write a decorator. A decorator is a function that wraps another function. After decoration the function will be a WSGI application, but it will be decorating a function with a signature like controller_func(req, **urlvars). The controller function will return a response object (which, remember, is a WSGI application on its own).

If you remember webob.exc.HTTPNotFound, those objects can also be used as exceptions. Unless you are willing to restrict yourself to Python 2.5+, you'll have to do raise exc.HTTPNotFound().exception because the actual objects like HTTPNotFound don't work as exceptions before 2.5. We'll allow the function to raise these exceptions, and turn them into normal responses.

We'll also let the function return just a string, and in that case create a standard response object (200 OK, with a Content-Type of text/html).

>>> from webob import Request, Response
>>> from webob import exc
>>> def controller(func):
...     def replacement(environ, start_response):
...         req = Request(environ)
...         try:
...             resp = func(req, **req.urlvars)
...         except exc.HTTPException, e:
...             resp = e
...         if isinstance(resp, basestring):
...             resp = Response(body=resp)
...         return resp(environ, start_response)
...     return replacement

You use this controller like:

>>> @controller
... def index(req):
...     return 'This is the index'

Putting It Together

Now we'll show a basic application. Just a hello world application for now. Note that this document is the module __main__.

>>> @controller
... def hello(req):
...     if req.method == 'POST':
...         return 'Hello %s!' % req.params['name']
...     elif req.method == 'GET':
...         return '''<form method="POST">
...             You're name: <input type="text" name="name">
...             <input type="submit">
...             </form>'''
>>> hello_world = Router()
>>> hello_world.add_route('/', controller=hello)

Now let's test that application:

>>> req = Request.blank('/')
>>> resp = req.get_response(hello_world)
>>> print resp
200 OK
content-type: text/html; charset=UTF-8
Content-Length: 131
<BLANKLINE>
<form method="POST">
            You're name: <input type="text" name="name">
            <input type="submit">
            </form>
>>> req.method = 'POST'
>>> req.body = 'name=Ian'
>>> resp = req.get_response(hello_world)
>>> print resp
200 OK
content-type: text/html; charset=UTF-8
Content-Length: 10
<BLANKLINE>
Hello Ian!

Another Controller

There's another pattern that might be interesting to try for a controller. Instead of a function, we can make a class with methods like get, post, etc. The urlvars will be used to instantiate the class.

We could do this as a superclass, but the implementation will be more elegant as a wrapper, like the decorator is a wrapper. Python 3.0 will add class decorators which will work like this.

We'll allow an extra action variable, which will define the method (actually action_method, where _method is the request method). If no action is given, we'll use just the method (i.e., get, post, etc).

>>> def rest_controller(cls):
...     def replacement(environ, start_response):
...         req = Request(environ)
...         try:
...             instance = cls(req, **req.urlvars)
...             action = req.urlvars.get('action')
...             if action:
...                 action += '_' + req.method.lower()
...             else:
...                 action = req.method.lower()
...             try:
...                 method = getattr(instance, action)
...             except AttributeError:
...                 raise exc.HTTPNotFound("No action %s" % action)
...             resp = method()
...             if isinstance(resp, basestring):
...                 resp = Response(body=resp)
...         except exc.HTTPException, e:
...             resp = e
...         return resp(environ, start_response)
...     return replacement

Here's the hello world:

>>> class Hello(object):
...     def __init__(self, req):
...         self.request = req
...     def get(self):
...         return '''<form method="POST">
...             You're name: <input type="text" name="name">
...             <input type="submit">
...             </form>'''
...     def post(self):
...         return 'Hello %s!' % self.request.params['name']
>>> hello = rest_controller(Hello)

We'll run the same test as before:

>>> hello_world = Router()
>>> hello_world.add_route('/', controller=hello)
>>> req = Request.blank('/')
>>> resp = req.get_response(hello_world)
>>> print resp
200 OK
content-type: text/html; charset=UTF-8
Content-Length: 131
<BLANKLINE>
<form method="POST">
            You're name: <input type="text" name="name">
            <input type="submit">
            </form>
>>> req.method = 'POST'
>>> req.body = 'name=Ian'
>>> resp = req.get_response(hello_world)
>>> print resp
200 OK
content-type: text/html; charset=UTF-8
Content-Length: 10
<BLANKLINE>
Hello Ian!

URL Generation and Request Access

You can use hard-coded links in your HTML, but this can have problems. Relative links are hard to manage, and absolute links presume that your application lives at a particular location. WSGI gives a variable SCRIPT_NAME, which is the portion of the path that led up to this application. If you are writing a blog application, for instance, someone might want to install it at /blog/, and then SCRIPT_NAME would be "/blog". We should generate links with that in mind.

The base URL using SCRIPT_NAME is req.application_url. So, if we have access to the request we can make a URL. But what if we don't have access?

We can use thread-local variables to make it easy for any function to get access to the currect request. A "thread-local" variable is a variable whose value is tracked separately for each thread, so if there are multiple requests in different threads, their requests won't clobber each other.

The basic means of using a thread-local variable is threading.local(). This creates a blank object that can have thread-local attributes assigned to it. I find the best way to get at a thread-local value is with a function, as this makes it clear that you are fetching the object, as opposed to getting at some global object.

Here's the basic structure for the local:

>>> import threading
>>> class Localized(object):
...     def __init__(self):
...         self.local = threading.local()
...     def register(self, object):
...         self.local.object = object
...     def unregister(self):
...         del self.local.object
...     def __call__(self):
...         try:
...             return self.local.object
...         except AttributeError:
...             raise TypeError("No object has been registered for this thread")
>>> get_request = Localized()

Now we need some middleware to register the request object. Middleware is something that wraps an application, possibly modifying the request on the way in or the way out. In a sense the Router object was middleware, though not exactly because it didn't wrap a single application.

This registration middleware looks like:

>>> class RegisterRequest(object):
...     def __init__(self, app):
...         self.app = app
...     def __call__(self, environ, start_response):
...         req = Request(environ)
...         get_request.register(req)
...         try:
...             return self.app(environ, start_response)
...         finally:
...             get_request.unregister()

Now if we do:

>>> hello_world = RegisterRequest(hello_world)

then the request will be registered each time. Now, lets create a URL generation function:

>>> import urllib
>>> def url(*segments, **vars):
...     base_url = get_request().application_url
...     path = '/'.join(str(s) for s in segments)
...     if not path.startswith('/'):
...         path = '/' + path
...     if vars:
...         path += '?' + urllib.urlencode(vars)
...     return base_url + path

Now, to test:

>>> get_request.register(Request.blank('http://localhost/'))
>>> url('article', 1)
'http://localhost/article/1'
>>> url('search', q='some query')
'http://localhost/search?q=some+query'

Templating

Well, we don't really need to factor templating into our framework. After all, you return a string from your controller, and you can figure out on your own how to get a rendered string from a template.

But we'll add a little helper, because I think it shows a clever trick.

We'll use Tempita for templating, mostly because it's very simplistic about how it does loading. The basic form is:

1
2

import tempita
template = tempita.HTMLTemplate.from_filename('some-file.html')

But we'll be implementing a function render(template_name, **vars) that will render the named template, treating it as a path relative to the location of the render() call. That's the trick.

To do that we use sys._getframe, which is a way to look at information in the calling scope. Generally this is frowned upon, but I think this case is justifiable.

We'll also let you pass an instantiated template in instead of a template name, which will be useful in places like a doctest where there aren't other files easily accessible.

>>> import os
>>> import tempita
>>> def render(template, **vars):
...     if isinstance(template, basestring):
...         caller_location = sys._getframe(1).f_globals['__file__']
...         filename = os.path.join(os.path.dirname(caller_location), template)
...         template = tempita.HTMLTemplate.from_filename(filename)
...     vars.setdefault('request', get_request())
...     return template.substitute(vars)

Conclusion

Well, that's a framework. Ta-da!

Of course, this doesn't deal with some other stuff. In particular:

• Configuration
• Making your routes debuggable
• Exception catching and other basic infrastructure
• Database connections
• Form handling
• Authentication

But, for now, that's outside the scope of this document.