uWSGI reaches the 0.9.3 Milestone

uWSGI project [1] reaches the 0.9.3 milestone. You can review the complete announcement [2] on the mailing list  and download the code here [3]

This new release brings some new exiting features :

- Nginx 0.7.x module
- configuration via python module
- support (non-standard) for Python 3.x
- Twisted client resource adapter
- graceful restart of worker processes and hot-plug substitution/upgrade
of the uWSGI server
- shared memory area to share data between workers/processes
- Tomcat handler
- support for virtualenv

In this post [4] I have covered the usage of both the "configuration via a python module" and the "virtualenv support".

uWSGI allows you to run your favorite wsgi application on top of prefered web server (apache, cherokee or nginx).

[1] http://projects.unbit.it/uwsgi/
[2] http://lists.unbit.it/pipermail/uwsgi/2009-November/000020.html
[3] http://projects.unbit.it/downloads/uwsgi-0.9.3.tar.gz
[4] http://yml-blog.blogspot.com/2009/11/setting-up-django-project-with-cherokee.html

Continous sphinx build

I have always found the cycle : Edit the documentation source => build the sphinx based documentation [1] => open a browser pointing the updated documentation and re iterate until your are satisfied a bit painful.

Today I have finally found an automated version of this workflow that
gave me one of this nice WAHOUUUU feeling that happens after you remove this little stone from your shoe.

The idea is to continuously build the documentation using watch [2]

watch -n 2 make htlm

This will update the generated documentation every 2 seconds. This is nice in itself and it leads me to discover that Epiphany automatically reload the page when it changes.

[1] http://sphinx.pocoo.org/
[2] http://linux.die.net/man/1/watch
[3] http://projects.gnome.org/epiphany/

Setting up a django project with Cherokee, uWSGI and vitualenv (continued)

In my previous post [1] I have started to talk about this stack but I have avoided to mention that the django project was indeed encapsulated into a virtualenv. This brought a couple small quirks and questions to the surface. So I spent some times investigating and exchanging with the community behind uWSGI and cherokee in order to solve them. The enhancements I discuss here require that you use the tip of uWSGI [2] because some of them are new capability added in the past days or small improvements to my recipe of using these products.

Before I start to dive into the enhancements let me sum up what we had to do in order to serve our WSGI project, in our case a django project.

1> I had to add 2 configurations files : uwsgi.conf, django_wsgi.py
The first one is an XML file where you define the PYTHONPATH and the mount point for your application, the second one is a python module where you define your WSGI configuration.

2> I had to use the cherokee's wizard to create the appropriate configuration to serve my django app using uWSGI. This step add an "Information Sources" with the following settings in the interpreter section :

/usr/local/bin/uwsgi -s /tmp/cherokee-source1.sock -C -x /opt/webapps/example-ve/src/project/uwsgi.conf

The first thing that can be improved in this recipe is that you can reduce the configuration to only one python module. This allows you to avoid the uwsgi.conf file. uWSGI comes with an option that allows you to directly specify a python module.

-w     name of wsgi config module (no ROCK_SOLID)

This would give us the following command :

uwsgi -s /tmp/cherokee-source1.sock -C -w project.django_wsgi

The django_wsgi file is very simple and looks like this :

import os
import django.core.handlers.wsgi

# Set the django settings and define the wsgi app
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'
application = django.core.handlers.wsgi.WSGIHandler()

# Mount the application to the url
applications = {'/':'application', }

This is all good and simple but it only works if "project" is in the PYHTONPATH. I handle all my projects in a dedicated virtualenv. uWSGI handle this gracefully now, there is a new option that allows you to define your Python Home.

-H     set python home/virtualenv

Here it the improved settings for the interpreter section of the information sources :

uwsgi -s /tmp/cherokee-source1.sock -C -w project.django_wsgi -H /opt/webapps/example-ve

I would be glad to hear from you and very happy if you give this setup a spin on your django project and let me know how it performs compare to your actual setup.

[1] http://yml-blog.blogspot.com/2009/11/how-to-set-up-cherokee-and-uwsgi.html
[2] http://projects.unbit.it/uwsgi/changeset/87%3Ae78ef65c833e

How to set up cherokee and uWSGI

A couple of post ago I have "blogged" about cherokee which is a light weight web server, fast and extremely simple to configure thanks to its great admin interface.

The most common configuration to use cherokee with django is described here and to put is in a nutshell it was based on flup using SCGI or FASTCGI protocol.

However a couple of weeks ago a new contender pop up on my radar. It is called uWSGI and it might become in the near future very popular to power django application. You can find more information about it on on the pages below :
* example [2]
* Documentation [3]

From an architecture stand point it might provide us python web developer an interesting piece of software that could operate with virtually any webserver. There is already an apache module and cherokee handler. The documentation mentions that the core developers have started to develop an nginx adaptor.


Software prerequisites

cherokee 0.99.26
uWSGI (changeset: 85:61dcfd718023)

I have installed cherokee using the PPA on launchpad [1] and uWSGI directly from the mercurial repository.

hg clone http://projects.unbit.it/hg/uwsgi

Installation

Installation of both product is relatively straight forward and well documented. I will not detail it more here but rather point you to cherokee [4] and uWSGI website [5].
Here it is how you would compile uWSGI on Ubuntu 9.04

make -f Makefile.Linux.Py26

Note: The compilation will create a binairy called uwsgi26. It is important to add it into your PATH under the name of uwsgi. If you don't do this cherokee's wizard  will complain about not being able to find uWSGI server. I have created a symbolic link but you could also copy it into /usr/local/bin.

Configuration

The amazing admin interface of cherokee make it dead simple once you know what need to be done. In the latest version of cherokee a wizzard has been added to make the configuration even simpler.

First thing first you need to create 2 configurations file into your django project directory called respectively : django_wsgi.py and uwsgi.conf

------ django_wsgi.py ------------------

  import os import django.core.handlers.wsgi  
  os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings' 
  application = django.core.handlers.wsgi.WSGIHandler() 
 

------ uwsgi.conf --------------------

<uwsgi>

    <pythonpath>/home/yml/workdir/webdev/fabric_factory/20091023/fabric_factory/ve/lib/python2.6/site-packages/</pythonpath>

    <pythonpath>/home/yml/workdir/webdev/fabric_factory/20091023/fabric_factory/src/</pythonpath>

    <app mountpoint="/">

        <script>project.django_wsgi</script>

    </app>

</uwsgi>

Once this is done you can launch cherokee admin : 'cherokee-admin -u' and point your browser to this URL :
* http://127.0.0.1:9090/

Once in cherokee you can use the uWSGI wizard to create automatically your information sources and add a new rule. In order to do so you need to go into the virtual server section on the behavior tab [6]. Below the table there is an hyperlink called wizards. It will take you to a list of wizards, click on uWSGI.

Cherokee asks for the path to the uwsgi.conf file that we have created earlier. The last bit you might want to do is to add a rule to serve the admin media.

Troubleshooting

Here it is some basic technics to understand what is going on and solve your issues. Look at cherokee's log files :

sudo tail -f /var/log/cherokee/cherokee.error
and
sudo tail -f /var/log/cherokee/cherokee.access 

Look into the section called "information sources" in cherokee-admin. Explore the information sources created by the uWSGI wizard. Copy the text in the interpreter field, in my case it is :

/usr/local/bin/uwsgi -s /tmp/cherokee-source1.sock -C -x /[Path to my django project]/uwsgi.conf

Then you can paste it in a terminal this will give you a way to see what it is going on.

This new combination is still very young but it looks promising to me. I am sure that if we get enough people to hammer on it can quickly becomes popular.

[1] https://launchpad.net/~cherokee-webserver/+archive/ppa
[2] http://projects.unbit.it/uwsgi/wiki/Example
[3] http://projects.unbit.it/uwsgi/wiki/Doc
[4] http://www.cherokee-project.com/doc/basics_installation.html
[5] http://projects.unbit.it/uwsgi/wiki/Install
[6] http://127.0.0.1:9090/vserver/10

django-piston authentication against django auth -- part 4

The previous post ends with one known "Open Issue". The authentication for the ajax call was sitll hard coded [1]. In this post I am going to show you how you can extend django-piston to authenticate your users against django.contrib.auth. This might be important if your web app already take advantage of this module to manage and authenticate users.


A bit of reading of the source code and the documentation of django-piston [2] leads me to understand that django-piston has been designed from the ground to enable you to easily write your own authentication handler. Out of the box it comes with "HttpBasicAuthentication" and OAuthAuthentication. "HttpBasicAuthentication" is a very good example how to implement a DjangoAuthentication.

An authentication handler is a class, which must have 2 methods: is_authenticated, challenge

class DjangoAuthentication(object):
    """
    Django authentication. 
    """
    def __init__(self, login_url=None, redirect_field_name=REDIRECT_FIELD_NAME):
        if not login_url:
            login_url = settings.LOGIN_URL
        self.login_url = login_url
        self.redirect_field_name = redirect_field_name
        self.request = None
    
    def is_authenticated(self, request):
        """
        This method call the `is_authenticated` method of django
        User in django.contrib.auth.models.
        
        `is_authenticated`: Will be called when checking for
        authentication. It returns True if the user is authenticated
        False otherwise.
        """
        self.request = request
        return request.user.is_authenticated()
        
    def challenge(self):
        """
        `challenge`: In cases where `is_authenticated` returns
        False, the result of this method will be returned.
        This will usually be a `HttpResponse` object with
        some kind of challenge headers and 401 code on it.
        """
        path = urlquote(self.request.get_full_path())
        tup = self.login_url, self.redirect_field_name, path 
        return HttpResponseRedirect('%s?%s=%s' %tup)

The implementation is relatively simple, is_authenticated uses the method with the same name available from a django User and the challenge redirect the user to the login page.

This post will end the series on django-piston it was for me a pretext to get familliar with this great app that make creating a RESTFUL api for django easy. You can find all the modifications I did into my fork on bitbucket [3]


[1] http://bitbucket.org/yml/django-piston/src/a8bcb7f9756e/examples/blogserver/templates/edit_ajaxy_post.html#cl-6
[2] http://bitbucket.org/jespern/django-piston/wiki/Documentation#authentication
[3] http://bitbucket.org/yml/django-piston/

django-piston form validation -- part 3

In my previous post titled "Exploration of django-piston -- part 2" [1] I end the post by 2 open issues. This post will propose a solution to the first one:

The validation errors are returned as a pseudo xml string.

Bad Request <ul class="errorlist"><li>content<ul class="errorlist"><li>This field is required.</li></ul></li><li>title<ul class="errorlist"><li>This field is required.</li></ul></li></ul>

To put it in other words I would prefer if django-piston return the validation errors of the form in the format specified in the request.

JSON

$ curl -u testuser:foobar -X POST -d content="This post is created using the api" http://127.0.0.1:8000/api/posts/?format=json
Bad Request {"title": ["This field is required."]}

yaml

$ curl -u testuser:foobar -X POST -d content="This post is created using the api" http://127.0.0.1:8000/api/posts/?format=yaml
Bad Request title: [!!python/unicode 'This field is required.']

xml

$ curl -u testuser:foobar -X POST -d content="This post is created using the api" http://127.0.0.1:8000/api/posts/?format=xml
Bad Request <ul class="errorlist"><li>title<ul class="errorlist"><li>This field is required.</li></ul></li></ul>

After a while reading the source of django-piston and poking around I have implemented this feature in my branch.

The biggest hurdle was that for some reasons "form.errors" cannot be serialized directly.

simplejson.dumps(form.errors)
*** TypeError:  is not JSON serializable

In order to work around this issue you need to force python to evaluate this proxy object.

dict((key, [unicode(v) for v in values]) for key,values in form.errors.items())

The last bit is to change the JS in order to adapt it to the fact that now the validation errors are returned as a json string.

    response = response.substring(12, response.length);
    errors = eval("("+response+")");

    $.each($(':input'), function(index, value) {
        field =$(value).attr('id');
        field = field.substring(3, field.length);
        field_errors = errors[field];
        if (field_errors) {
            ul = "<ul class=\"errorlist\">\n";
            $.each(field_errors, function(index, value){
              ul += "<li>"+value+"</li>\n";
            });
            ul += "</ul>\n";
            $(value).parent().prepend(ul);
        };
    });

The complete code related to the implementation of this feature is available in my branch [3].

Open Issue

Authentication is still hardcoded into the html pages. What is the best way to keep HttpBasicAuthentication to the external api that could easily be used by curl and add single sign on with django admin interface ?

The example is still very incomplete. It is missing among other things the capability to update/delete a blog post

I would be glad to read from you on how to improve this example or to get pointer of reusable application that have been built on top of django-piston.


[1] http://yml-blog.blogspot.com/2009/10/exploration-of-django-piston-part-2.html
[2] http://bitbucket.org/yml/django-piston/changeset/a8bcb7f9756e/
[3] http://bitbucket.org/yml/django-piston/

exploration of django-piston -- part 2

Djangocon 09 was really food for thought and this post follow my previous post about django-piston. I found the talk from Avi Bryant about his experience building Trendly [1].

My take away of this talk was that they have a single HTML page with a lot of JS. The JS is the used to get the JSON data the JSON from the server. Unfortunately I am unable to find a link to a better description of its talk.

My last post hopefully convince you that it is easy relatively easy to create a web api to your django application [2]. This Post aims to show you how you can create a page to list and create a blogpost.

List and create a blogpost

The idea here is to reuse the web api that we have created in the previous article to build a web page. One of the interesting feature of django-piston is the decorator @validate(BlogpostForm, 'PUT'). It enables you to validate the data send to your server and it takes as input a django form. A longuer description of this feature is available here [4].

The view cannot be simpler [5] :

def create_ajaxy_post(request):
    form = BlogpostForm()
    return direct_to_template(request,
                              template='edit_ajaxy_post.html',
                              extra_context={'form':form} )

The template is using jQuery [6] which ease the ajax call and the DOM manipulation. The template [7] has a very simple structure :

{% block content %}

<h1>Ajaxy</h1>

<div id="content"></div>

<h2>Ajaxy form</h1>

<form method="POST" action="{% url posts %}" id="post_form">

    <div id="post_form_error"></div>

    {{ form.as_p }}

    <input type='submit' name='create' onclick="return send_form();" value='Create'></input>

</form>

{% endblock %}

In order to give life to this page you need some JS which will get from the JSON string of the blogposts and POST the DATA to create a new blogpost and add it to the list.

If you try to POST an empty blogpost the data will be validated on the server and the validation errors will be displayed to the user :

* content
  o This field is required.
* title
  o This field is required.

Open Issue

The validation error are returned as a pseudo xml string :

Bad Request <ul class="errorlist"><li>content<ul class="errorlist"><li>This field is required.</li></ul></li><li>title<ul class="errorlist"><li>This field is required.</li></ul></li></ul>

It will be much more convenient to get a json dict with the error. Any one has an idea on how to get this ?

Authentication is still hardcoded into the html pages. What is the best way to keep HttpBasicAuthentication to the external api that could easily be used by curl and add single sign on wiht the admin ?

The example is still very incomplete. It is missing among other things the capability to update/delete a blog post

I would be glad to read from you on how to improve this example or to get pointer of reusable application that have been built on top of django-piston.


[1] http://trendly.com/
[2] http://bitbucket.org/yml/django-piston/src/tip/examples/blogserver/README.txt
[3] http://bitbucket.org/yml/django-piston/src/72c72e0b4b7e/examples/blogserver/templates/edit_ajaxy_post.html
[4] http://bitbucket.org/jespern/django-piston/wiki/Documentation#form-validation
[5] http://bitbucket.org/yml/django-piston/src/tip/examples/blogserver/blog/views.py
[6] http://jquery.com/
[7] http://bitbucket.org/yml/django-piston/src/72c72e0b4b7e/examples/blogserver/templates/edit_ajaxy_post.html