Add a web api to your app with django-piston

More and more often I found myself in a situation where I would like to add a web api to my django applications. I have recently tried to used django-piston http://bitbucket.org/jespern/django-piston for this and I found the learning curve a bit steep. It is not particularly hard but it requires you to understand few things before being able to enjoy it.

This post should help you to understand how to create the handlers to read, create, update, delete an object and see how you can call this web api from the command line using curl. I have forked django-piston to extend the example "blogserver". I would recommend you to also read the README inside the example

You can grab the code like this :

hg clone http://bitbucket.org/yml/django-piston/


curl is a command line tool to transfer data from or to a server, using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE). The command is designed to work without user interaction.

Here it is the 4 things that I found a bit hard to understand.

1. @require_extended

This decorator is handy if you want to restrict the access to your handler to only the request that have have one of the header listed below :
• application/json
• application/x-yaml
• text/xml
• application/python-pickle

The direct effect on the curl command line is that you will need to add a -H 'Content-Type:', this will give you something like :

$ curl -u testuser:foobar -H 'Content-Type:application/json' http://127.0.0.1:8000/api/posts/?format=json


If the header is omitted django-piston will return a 'Bad Request'.

2. Request Method

The handler for your resource can be composed of the following method : read, create, update, delete that are respectively mapped to the following request method : GET, POST, PUT, DELETE. "-X " is used to specify the method you want to use. Here it is an example that execute the update method of the blogserver :

$ curl -u testuser:foobar -H 'Content-Type:application/json' -X PUT -d '{"content": "Update test", "title": "Update test"}' http://127.0.0.1:8000/api/post/1/



3. Passing data to your handler

In this example I am going to demonstrate how to pass JSON string with curl. In order to do this you should use the -d followed by the JSON string :

$ curl -u testuser:foobar -H 'Content-Type:application/json' -X PUT -d '{"content": "Update test", "title": "Update test"}' http://127.0.0.1:8000/api/post/1/


django-piston will automatically turn this string into a python dictionary that is ready to be used by your handler. You will find this data in 'request.data' the raw string is available in "request.raw_post_data"

4. How to pass parameter to your handler

It took me a while to understand that the same handler can be mounted to several urls this will allow you to pass additional parameters like the primary key ID or the slug :

....
blogposts = Resource(handler=BlogpostHandler, authentication=auth)
urlpatterns = patterns('',
url(r'^posts/$', blogposts),
url(r'^post/(?P.+)/$', blogposts),
.....


In the example above "blockposts" is mapped to 2 different urls, the second one will be particularly handy to read, update, delete a particular post. Additional parameters from the URL will be passed to the method : read, create, update, delete. I encourage you to check out the code from bitbucket to see how you could take advantage of this technique.

This post barely scratch the surface of how to use django-piston, to put it in a nutshell this reusable application makes creating a web api for your own project simple. It avoids you to write a lot of boilerplate code because it abstracts all the machinery and let you focus on important things.

I would be glad to hear from you how you use it and what is your favorite trick.

Fabric factory

This is a project I have been working on recently after I spent a day to look at the existing solution to run periodically a test suite. Most of the project I look at were either difficult to setup or require to learn yet another specific domain specific language or had dependency on a larger software stack.

As a reaction to this situation I have decided to see if I could write something simple that achieves gracefully this task. I also try to make it as easy to setup as possible.

I have decided to use cpython as platform, django as web framework for the server and Fabric as library to automate the task execution.

The result of this mix can be found on bitbucket in a project called Fabric Factory. This will eventually become a complete Job Server that could be used to distribute any kind of task scripted in Fabric.


Installation

This assumes that python is installed on your computer and that you have an internet conection.

You can download the code using mercurial:
* hg clone http://bitbucket.org/yml/fabric_factory/
A fabfile will help you to quickly setup your environment.
* fab quickstart

Note : In order to run the command above you will need the latest version of Fabric the following command will take care of this:

pip install -e git://github.com/bitprophet/fabric.git#egg=Fabric

Usage

"quickstart" has created a virtualenv which must be actived before you continue.

. ve/bin/activate


Once the virtualenv is activated you can go inside "src/project". This is a django project so from there you can do several things :

* create an sqlite db : python manage.py syncdb
* run the server : python manage.py runserver
* run the test suite : python manage.py test

The main app of this django project called fabric factory is called "factory".

Once the server is started and that you have created some "Build" in django's admin interface you can open a new terminal and run the client side of the project:

cd src/worker
python run_worker.py --daemon=start
python run_worker.py --daemon=stop


Use case

Now that you have understood the layout of the project. Let us see how we can achieve something useful with it.

We are going to create a Build that will :
* download the Fabric Factory
* setup the environement
* run the test suite
* Report the result

1> Direct your browser to that url http://127.0.0.1:8000/admin/ and key in the username/password you have chosen for your administrator.
2> Add the fabfile recipe store in docs http://127.0.0.1:8000/admin/factory/fabfilerecipe/add/ and call it "fabric factory use case"
3> Replace example.com by 127.0.0.1:8000 in sites : http://127.0.0.1:8000/admin/sites/site/1/
4> Create a Build that will download setup and run the test here : http://127.0.0.1:8000/admin/factory/build/add/

The fabfile recipe that we have downloaded earlier contains a task called : 'download_setup_and_test' This task as been writen to do what we want.

We are now going to configure the client to run this task. However before doing this let us see how the server publish the tasks that need to be executed. Point your browser to this url : http://127.0.0.1:8000/factory/build/oldest_not_executed/

5> Open a new terminal and move into the the worker directory then you can start the worker in daemon mode :

. ve/bin/activate
cd src/worker
python run_worker.py --daemon=start


If you want to look at what is happening in the background you can watch at the log file in realtime

tail -f worker.log

6> You can see the status of the build here : http://127.0.0.1:8000/admin/factory/build/ Keep in mind this task is pretty long to run because we are downloading all the dependancies (django, fabric). It took me almost 5 minutes to execute this task and to see the result in the admin.

Conclusion

This project which is still very immature seems to prove that this stack is well suited to build this kind of tool. I would be glad to hear your experience about this kind of tool. Please do not hesitate to copy, fork, contribute to this project to make sure that soon we have a simple easy to setup yet flexible tool to distribute tasks.

sphinx autodoc and django app

Today I have been getting my foot wet with autodoc extension from sphinx. Here it is what the documentation of sphinx say about it :

"""
Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.
"""

I have to say that before today my experience with sphinx as been excellent. It is relatively straightforward to get started once you pass the first little annoyances that come with the fact that you are learning a new tool.

The module I have been using to conduct this experiment is django-geotagging this reusable app enables you to geotag any object in the database. If you want to know more about this you can read the sphinx based documentation. This is the goal of the documentation, isn't it ? :-)
If adding a manually written documentation is very easy and well documented. The modification required to move from a very primitive documentation written using some rst file to sphinx is shown here. Most of it is automatically generated by : sphinx-quickstart.

Taking it a step further have been more difficult than anticipated and this for several reasons : lack of example, and a bug
Today I wanted to add to the existing documentation in django-geotagging and API section. In order to do so I have spoted in the sphinx documentation an extension called autodoc. It seems to be exactly what I need, here it is an extract from its doc :

"""

This extension can import the modules you are documenting, and pull in documentation from docstrings in a semi-automatic way.

"""

The first modification I add to do in sphinx's conf.py was to setup the settings in my environment. This can be done by adding the following 3 lines to conf.py

"""
from geotagging_demo_project import settings
from django.core.management import setup_environ
setup_environ(settings)
"""

Once this is done you should just be able to create a file that will be used as placeholder to describe the documentation you want to extract. Let us take an example now, since I want to describe the API of "models.py" I am going to create a file called "model.rst". In this file I need to add the following lines :

"""
:mod:`models` -- geotag models
==========================================

.. automodule:: geotags.models
:members:
:show-inheritance:

.. autoclass:: Point
"""

I will let you read the documentation for each of this directive to understand what they are doing. This is were the bug come into play because once you have done this you should be able to enjoy the automatically extrated documentation the next time you build it. Instead of this I have observed this bug. To put it in a nutshell sphinx is complaining about the line 27 of
"/usr/lib/python2.6/django/contrib/gis/db/models/proxy.py"

I am not really sure where the bug is however it seems that doing a small modification there enable me to build the documentation.

I hope that this feedback about my experience will help you to get started with using sphinx and its autodoc extension. If you have an opinion about the django ticket #11353 I am very interested to hear it.

Serving Django via CherryPy behind cherokee

Almost one year ago "Peter Baumgartner" wrote about Serving Django via cherrypy I am going to explain in this post how to take this approach one step further and to add load and balancing of the requests to several instances of cherrypy. This sounds like a lot of "*.conf" editing, isn't it ? In fact the nice thing about this approach is that the only file you will have to edit is your settings.py to add one line. To do so I am going to use cherokee mainly because it has a user friendly interface called cherokee-admin which provides a very easy way to configure your server. For this article I have used cherokee Version 0.98.1.

I will assume in this article that you have a working django project in a virtualenv. First you will need to install cherrypy and django-cpserver. You have several way to do this the easier is probably to use pip

pip install cherrypy
pip install -e git://github.com/lincolnloop/django-cpserver.git#egg=django-cpserver

Then you need to edit you settings.py to add "django_cpserver" in the list of your INSTALLED_APPS. This will give you a convenient django management command to start cherrypy server.
./manage.py runcpserver port=8089

Believe it or not this was the hardest part of the recipe from now to the end we will use a nice web interface. In order to launch cherokee-admin on ubuntu I use the following command :

sudo cherokee-admin

We need to define 2 remote sources in the admin interface :

127.0.0.1:8088 and 127.0.0.1:8088 are the addresses on which cherokee can contact the cherrypy instances. Several interested things to note here, the adresses can be spread on several computer and several ports.

Then we need to define a new target "/django" (alias) that will load and balance the requests to cherrypy instances.



Then for this target we need to set the handler to "HTTP reverse proxy".


It is time to use the remote sources we have defined earlier.



The last bit is to rewrite the url before passing it to the cherrypy instances



This is the end of the recipe you can now save the modification and restart the cherokee. I would be glad to read from you the enhancements that could be added to this recipe.

django full text search with solango

2 months ago I wrote a post titled "django fulltext search part-1" this post was explaining how to take advantage of djangosearch to interface between django and solr.

The big advantage of djangosearch is the fact that it comes with a plugable backend architecture. This can be a strength since in theory it enables you to abstract the details of the the full text engine you are using however in practice I ended up writing a patch to bypath the abstraction layer because it was preventing me of doing the query I wanted. So to make a long story short djangosearch was not working out of the box for my needs.

However 2 week ago Sean Creeley released solango and this significantly changed landscape of full text search in the django eco-system. I will not go in the details about solango in this post. It comes with some management commands that are so convenient that I still wonder why I haven't thought at implementing them on top of djangosearch. I will copy below a short extract form the documentation which is excellent :

./manage.py solr --help
#solango schema options
--fields              Prints out the fields the schema.xml will create
--flush               Will remove the data directory from Solr.
--reindex             Will reindex Solr from the registry.
--schema              Will create the schema.xml in SOLR_SCHEMA_PATH or in the --path.
--start               Start solr running java -jar start.jar
--path=SCHEMA_PATH    Tells Solango where to create config file

In the rest of the post I am going to assume that you have installed and configured solango. When I have done this I have not seen any major obstacle. Once again Sean has done an excellent job at documenting this project. one thing that annoys me while implementing solango in one of the project I am working on is the fact that you cannot restrict your search in what is often call an "advanced search". The good news is that solango has been recently improved to be easily extended. The only thing you will have to do is to defined a django form to represent your advanced search and to add an url that use it. This can literally be done in less than 30 lines of code including the comments and the imports.

Let us start by the writing the forms.py :

from django import forms
from solango.solr import get_model_from_key
import solango

def model_choices():
    """
    Return a list of tuple with all the models that have been indexed.
    This tuple is used in the AdvancedSearchForm to selects the models
    you want to search your term in.
    """
    models = [(model_key, get_model_from_key(model_key)._meta.verbose_name_plural)
             for model_key in solango.registry.keys()]
    return models

class AdvancedSearchForm(forms.Form):
    """
    Form that represents an advanced search
    """
    q = forms.CharField(required=False)
    model = forms.MultipleChoiceField(choices=model_choices(), required=False,
        widget=forms.CheckboxSelectMultiple)
    def clean_q(self):
        q = self.cleaned_data.get("q")
        if q == '':
            raise forms.ValidationError("You cannot query for an empty string")
        return q

Nothing really complex there we have defined a form with a charfield named "q" that will be used to enter your search terms and a set of check boxes displaying all the models indexed. The user will be able to select one or several models to restrict its query.
Now that the form is defined we are going to use it. In order to do so you need to add the following url somewhere in your project.

from django.conf.urls.defaults import *

from solango.views import select
from populous.search.forms import AdvancedSearchForm

urlpatterns = patterns('',
            url(r'^advanced/$', select,
                {
                    "form_class":AdvancedSearchForm,
                    "template_name":None,
                },name="search-advanced"),
)

You will recognize there the same pattern used in the generic views. You can customize the solango's view, called "select", by passing a "form_class" and a "template_name". In this example I have not over loaded the template_name thus django will used the solango default template.

I would be glad to read from you the customized form you have built on top of solango. Please do not hesitate to post them as comment.

Updated the 22th Feb 2009 : Correct the name of Sean Creeley, sorry for that.

Enable distutils for a django reusable app

As you might already be aware Pinax move to distutils, this change has been described by James Tauber in this post. Today I have decided to make my feet wet with this approach for a django reusable app. I have been guided by jezdez on #pinax.
Before starting I would recommend you to read this page, yes I know, it is a bit long but very interesting. after this reading you will be well prepared to start to work on you django reusable app.

I have used one of my project django-geotagging to experiment with this approach. The project can be found on launchpad there. The good news about this approach is that all the major rcs are supported : SVN, BZR, HG, ... If your favorite versionning system is missing there is a good chance that I just forget to mention it. This list is not exhaustive.

The core of this approach is a file called setup.py that need to be paced at the root of your repository, most of its argument are self explanatory.

This file enable you to setup a complete env in 4 steps :

• Create a virtual env : virtualenv geotagging_env
• Move into that env : cd geotagging_env
• Activate the env : . bin/activate
• Install pip : easy_install pip
• Use pip to grab django-geotagging : pip install -e bzr+http://bazaar.launchpad.net/~yml/django-geotagging/geotags/#egg=django-geotags

The coolness of this increase with the number of reusable apps you have to install in order to build your web project.

I would be glad to learn from you what kind of cool things can be done once this infrastructure is in place.

How to use the same widget as GeoDjango

At the end of this post you will be able to use the same widget than the automatically generated admin interface in geodjango. I spent quite sometimes today to rediscover how to do this. Yes, rediscover because I have already written about this a couple of month ago. My first post on that topic can be read there. Happily "jbronn" on #geodjango gave the solution to me.

# Getting an instance so we can generate the map widget; also
# getting the geometry field for the model.
admin_instance = PointAdmin(Point, admin.site)
point_field = Point._meta.get_field('point')

# Generating the widget.
PointWidget = admin_instance.get_map_widget(point_field)

In fact all the complication at the moment there is no static widget that widget you could use in your own form. You have to build them dynamically.

I am now going to break down the 3 lines of code.
PointAdmin is the ModelAdmin class which is a representation of a model in the admin interface. Here it is an example :

from django.contrib.gis import admin
from geotagging.models import Point

class PointAdmin(admin.GeoModelAdmin):
list_filter = ('content_type','point' )
list_display = ('object', 'point', 'content_type', 'object_id')

Point in the model we are working on so Point._meta.get_field('point') is accessing the field called point of the Point mode. The code below should help you to understand :

class Point(models.Model):
  """
  """
  point = models.PointField(verbose_name=_("point"),srid=4326)
  content_type   = models.ForeignKey(ContentType,
          related_name="content_type_set_for_%(class)s")
  object_id      = models.CharField(_('object ID'),max_length=50)
  object = generic.GenericForeignKey(ct_field="content_type", fk_field="object_id")
  objects = models.GeoManager()

  def __unicode__(self):
      return 'Point for %s' % self.object

The last line is actually where the geodjango specificity is :
* PointWidget = admin_instance.get_map_widget(point_field)
get_map_widget is defined here

Now that we have a PointWidget we can use it in our form. Here is is a small example :

class PointForm(forms.ModelForm):
    point = forms.CharField(widget=PointWidget())
    class Meta:
        model = Point
        exclude = ("content_type","object_id")
    class Media:
        js = ("http://openlayers.org/api/2.6/OpenLayers.js",)

Now you can use geodjango super widgets in your forms.