Continous sphinx build

I have always found the cycle : Edit the documentation source => build the sphinx based documentation [1] => open a browser pointing the updated documentation and re iterate until your are satisfied a bit painful.

Today I have finally found an automated version of this workflow that
gave me one of this nice WAHOUUUU feeling that happens after you remove this little stone from your shoe.

The idea is to continuously build the documentation using watch [2]

watch -n 2 make htlm

This will update the generated documentation every 2 seconds. This is nice in itself and it leads me to discover that Epiphany automatically reload the page when it changes.

[1] http://sphinx.pocoo.org/
[2] http://linux.die.net/man/1/watch
[3] http://projects.gnome.org/epiphany/

Serving Django via CherryPy behind cherokee

Almost one year ago "Peter Baumgartner" wrote about Serving Django via cherrypy I am going to explain in this post how to take this approach one step further and to add load and balancing of the requests to several instances of cherrypy. This sounds like a lot of "*.conf" editing, isn't it ? In fact the nice thing about this approach is that the only file you will have to edit is your settings.py to add one line. To do so I am going to use cherokee mainly because it has a user friendly interface called cherokee-admin which provides a very easy way to configure your server. For this article I have used cherokee Version 0.98.1.

I will assume in this article that you have a working django project in a virtualenv. First you will need to install cherrypy and django-cpserver. You have several way to do this the easier is probably to use pip

pip install cherrypy
pip install -e git://github.com/lincolnloop/django-cpserver.git#egg=django-cpserver

Then you need to edit you settings.py to add "django_cpserver" in the list of your INSTALLED_APPS. This will give you a convenient django management command to start cherrypy server.
./manage.py runcpserver port=8089

Believe it or not this was the hardest part of the recipe from now to the end we will use a nice web interface. In order to launch cherokee-admin on ubuntu I use the following command :

sudo cherokee-admin

We need to define 2 remote sources in the admin interface :

127.0.0.1:8088 and 127.0.0.1:8088 are the addresses on which cherokee can contact the cherrypy instances. Several interested things to note here, the adresses can be spread on several computer and several ports.

Then we need to define a new target "/django" (alias) that will load and balance the requests to cherrypy instances.



Then for this target we need to set the handler to "HTTP reverse proxy".


It is time to use the remote sources we have defined earlier.



The last bit is to rewrite the url before passing it to the cherrypy instances



This is the end of the recipe you can now save the modification and restart the cherokee. I would be glad to read from you the enhancements that could be added to this recipe.

How to use the same widget as GeoDjango

At the end of this post you will be able to use the same widget than the automatically generated admin interface in geodjango. I spent quite sometimes today to rediscover how to do this. Yes, rediscover because I have already written about this a couple of month ago. My first post on that topic can be read there. Happily "jbronn" on #geodjango gave the solution to me.

# Getting an instance so we can generate the map widget; also
# getting the geometry field for the model.
admin_instance = PointAdmin(Point, admin.site)
point_field = Point._meta.get_field('point')

# Generating the widget.
PointWidget = admin_instance.get_map_widget(point_field)

In fact all the complication at the moment there is no static widget that widget you could use in your own form. You have to build them dynamically.

I am now going to break down the 3 lines of code.
PointAdmin is the ModelAdmin class which is a representation of a model in the admin interface. Here it is an example :

from django.contrib.gis import admin
from geotagging.models import Point

class PointAdmin(admin.GeoModelAdmin):
list_filter = ('content_type','point' )
list_display = ('object', 'point', 'content_type', 'object_id')

Point in the model we are working on so Point._meta.get_field('point') is accessing the field called point of the Point mode. The code below should help you to understand :

class Point(models.Model):
  """
  """
  point = models.PointField(verbose_name=_("point"),srid=4326)
  content_type   = models.ForeignKey(ContentType,
          related_name="content_type_set_for_%(class)s")
  object_id      = models.CharField(_('object ID'),max_length=50)
  object = generic.GenericForeignKey(ct_field="content_type", fk_field="object_id")
  objects = models.GeoManager()

  def __unicode__(self):
      return 'Point for %s' % self.object

The last line is actually where the geodjango specificity is :
* PointWidget = admin_instance.get_map_widget(point_field)
get_map_widget is defined here

Now that we have a PointWidget we can use it in our form. Here is is a small example :

class PointForm(forms.ModelForm):
    point = forms.CharField(widget=PointWidget())
    class Meta:
        model = Point
        exclude = ("content_type","object_id")
    class Media:
        js = ("http://openlayers.org/api/2.6/OpenLayers.js",)

Now you can use geodjango super widgets in your forms.

Django fulltext search -- part 1

I have been recently working on adding full text search to a django based web application since this is not yet particularly well documented. Here it is my little stone to the amazing pyramid already in place waiting for you to use it. In order to add this "must have" feature to your great web application you will need 3 components :

• djangosearch
• pysolr
• solr
  

Starting by the latest the first challenge is to get it installed, I haven't had luck with the usual "apt-get install solr". It must be me but after an installation I was only getting a blank page instead of solr admin interface. On #solr channel someone advise me to just grab the latest release tar ball and to run with it. Then I have performed the tutorial this help me a lot to get started with solr. Believe it or not when you are there you have done the hardest part of it.

Now you need to go inside the django project you want to work on and follow the following recipe :

1. download and put into your PYHTONPATH djangosearch and pysolr
   
2. Add djangosearch inside the INSTALLED_APPS tupple
3. add the following settings :

• SEARCH_ENGINE = "solr" # solr, lucene, xapian, estraier
• SEARCH_RESULTS_PER_PAGE = 10
• SOLR_URL= "http://localhost:8983/solr"
  

At this point you are almost done the last operations you will need to do is to define inside models.py the fields you want to index with solr. In order to do this djangosearch come with a class called : ModelIndex
For example if you want to index : tease, and story fields of your Story model you will do something like this :

from djangosearch import ModelIndex
class Story(models.Model):
  ...
 tease = models.TextField(_('tease'), blank=True)
 story = models.TextField(_('story'), blank=True)
  ...
  index = ModelIndex(fields=["tease","story"])

Now you need to tell solr about this 2 new fields this is in fact easier that it seems to be, the theory is explained there. In practice what you need to do open this file (apache-solr-1.3.0/example/solr/conf/schema.xml) with your editor of choice and add this 2 lines inside the fields section:




You will now need to restart solr in order to take this modification into account.

The meal is now ready you can go and taste it.

• Open a browser go to the admin interface and add a new story or update an existing one.
• Open a django shell and enjoy your new full text search

from dajngosearch import search
search_results = search("foo")

I hope this will be useful to someone and will avoid you the pain I had to get all these pieces dancing together.
I would be glad to read from you what can be improve in this raw recipe and also what you are doing with your search and search_results.

How to explore dynamically your django code

I have been using the methods presented by Simon Willison on this post.
For quite sometimes now and I always felt that it would be nice to have the tab completion when using the debugger inside my views or elsewhere in my django code.

Here it is the magical line:
import ipdb; ipdb.set_trace()

Note : That "pdb" as been replace by "ipdb" which is the IPython debugger.

You then get all the nice stuff from IPython (tab completion, nice tracebacks) right in pdb. In order to install it you can for example do : easy_python ipdb

How to evaluate the coverage of a django test suite

I am using this recipe to estimate the coverage of the test suites arounds my django's projects.

Software Prerequisites

In order to follow this recipe you will need to have the following software installed :

• Django
• python (obviously)
• coverage.py

I will not explain how to get the first 2 items of this list installed since if you are reading this I am assuming that you are familiar with them. "Coverage" is a bit different, python package index make installing this component a piece of cake.

# easy_install coverage

Usage of coverage with django test suite

This presentation give a recipe to evaluate the coverage of your test suite. I found this page useful to understand how to use it. If you are looking for a project to test this recipe on you can checkout "django-survey". The 3 lines below is all you need to get a report on the coverage of your test suite.

# coverage.py -e                            (1)
# coverage.py -x manage.py test survey      (2)
# coverage.py -r -m >report.txt             (3)

The first line erases collected coverage data, the second executes the module and collect the coverage data, the third line reports on the statement coverage for the given files and show line numbers of the statements that weren't executed.

Then you need to analyze the file called "report.txt" and extract the information usefull to your project. In our case all the files located in django-survey :

Name 
------------------------------------------------------------------------------------------------------------------------------------------------
__init__                                                                                                               0      0   100%
[...]
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\__init__                                              0      0   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\forms                                               172    117    68%   32, 47-49, 56, 74-86, 96, 107, 117-120, 126-136, 139-145, 147-159, 212, 226-227, 230
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\models                                              157    142    90%   28-29, 70, 82-83, 88-89, 102-104, 109, 126, 159, 211, 236
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\templatetags\__init__                                 0      0   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\templatetags\survey                                  10      7    70%   8, 16-17
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\tests\__init__                                        2      2   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\tests\test_images                                     1      1   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\tests\test_models                                     1      1   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\tests\test_urls                                       1      1   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\urls                                                  6      6   100%
c:\yml\_myscript_\dj_things\web_development\django-survey\survey\views                                               183    126    68%   47, 54-65, 79-83, 90, 94, 96, 124-125, 150-151, 157-158, 183-186, 230-231, 246, 261-263, 280-281, 311-313, 328, 341-342, 359-360, 379, 419-425, 444-456, 462-474
[...]
manage                                                                                                                13      9    69%   10-13
management\__init__                                                                                                    0      0   100%
settings                                                                                                              31     31   100%
urls                                                                                                                   4      4   100%