Feb 21, 2009

django full text search with solango

2 months ago I wrote a post titled "django fulltext search part-1" this post was explaining how to take advantage of djangosearch to interface between django and solr.

The big advantage of djangosearch is the fact that it comes with a plugable backend architecture. This can be a strength since in theory it enables you to abstract the details of the the full text engine you are using however in practice I ended up writing a patch to bypath the abstraction layer because it was preventing me of doing the query I wanted. So to make a long story short djangosearch was not working out of the box for my needs.

However 2 week ago Sean Creeley released solango and this significantly changed landscape of full text search in the django eco-system. I will not go in the details about solango in this post. It comes with some management commands that are so convenient that I still wonder why I haven't thought at implementing them on top of djangosearch. I will copy below a short extract form the documentation which is excellent :
./manage.py solr --help
#solango schema options
--fields Prints out the fields the schema.xml will create
--flush Will remove the data directory from Solr.
--reindex Will reindex Solr from the registry.
--schema Will create the schema.xml in SOLR_SCHEMA_PATH or in the --path.
--start Start solr running java -jar start.jar
--path=SCHEMA_PATH Tells Solango where to create config file

In the rest of the post I am going to assume that you have installed and configured solango. When I have done this I have not seen any major obstacle. Once again Sean has done an excellent job at documenting this project. one thing that annoys me while implementing solango in one of the project I am working on is the fact that you cannot restrict your search in what is often call an "advanced search". The good news is that solango has been recently improved to be easily extended. The only thing you will have to do is to defined a django form to represent your advanced search and to add an url that use it. This can literally be done in less than 30 lines of code including the comments and the imports.

Let us start by the writing the forms.py :

from django import forms
from solango.solr import get_model_from_key
import solango

def model_choices():
"""
Return a list of tuple with all the models that have been indexed.
This tuple is used in the AdvancedSearchForm to selects the models
you want to search your term in.
"""
models = [(model_key, get_model_from_key(model_key)._meta.verbose_name_plural)
for model_key in solango.registry.keys()]
return models

class AdvancedSearchForm(forms.Form):
"""
Form that represents an advanced search
"""
q = forms.CharField(required=False)
model = forms.MultipleChoiceField(choices=model_choices(), required=False,
widget=forms.CheckboxSelectMultiple)
def clean_q(self):
q = self.cleaned_data.get("q")
if q == '':
raise forms.ValidationError("You cannot query for an empty string")
return q

Nothing really complex there we have defined a form with a charfield named "q" that will be used to enter your search terms and a set of check boxes displaying all the models indexed. The user will be able to select one or several models to restrict its query.
Now that the form is defined we are going to use it. In order to do so you need to add the following url somewhere in your project.

from django.conf.urls.defaults import *

from solango.views import select
from populous.search.forms import AdvancedSearchForm

urlpatterns = patterns('',
url(r'^advanced/$', select,
{
"form_class":AdvancedSearchForm,
"template_name":None,
},name="search-advanced"),
)

You will recognize there the same pattern used in the generic views. You can customize the solango's view, called "select", by passing a "form_class" and a "template_name". In this example I have not over loaded the template_name thus django will used the solango default template.

I would be glad to read from you the customized form you have built on top of solango. Please do not hesitate to post them as comment.

Updated the 22th Feb 2009 : Correct the name of Sean Creeley, sorry for that.

Jan 31, 2009

Enable distutils for a django reusable app

As you might already be aware Pinax move to distutils, this change has been described by James Tauber in this post. Today I have decided to make my feet wet with this approach for a django reusable app. I have been guided by jezdez on #pinax.
Before starting I would recommend you to read this page, yes I know, it is a bit long but very interesting. after this reading you will be well prepared to start to work on you django reusable app.

I have used one of my project django-geotagging to experiment with this approach. The project can be found on launchpad there. The good news about this approach is that all the major rcs are supported : SVN, BZR, HG, ... If your favorite versionning system is missing there is a good chance that I just forget to mention it. This list is not exhaustive.

The core of this approach is a file called setup.py that need to be paced at the root of your repository, most of its argument are self explanatory.

This file enable you to setup a complete env in 4 steps :
The coolness of this increase with the number of reusable apps you have to install in order to build your web project.

I would be glad to learn from you what kind of cool things can be done once this infrastructure is in place.

Jan 19, 2009

How to use the same widget as GeoDjango

At the end of this post you will be able to use the same widget than the automatically generated admin interface in geodjango. I spent quite sometimes today to rediscover how to do this. Yes, rediscover because I have already written about this a couple of month ago. My first post on that topic can be read there. Happily "jbronn" on #geodjango gave the solution to me.


# Getting an instance so we can generate the map widget; also
# getting the geometry field for the model.
admin_instance = PointAdmin(Point, admin.site)
point_field = Point._meta.get_field('point')

# Generating the widget.
PointWidget = admin_instance.get_map_widget(point_field)

In fact all the complication at the moment there is no static widget that widget you could use in your own form. You have to build them dynamically.

I am now going to break down the 3 lines of code.
PointAdmin is the ModelAdmin class which is a representation of a model in the admin interface. Here it is an example :

from django.contrib.gis import admin
from geotagging.models import Point

class PointAdmin(admin.GeoModelAdmin):
list_filter = ('content_type','point' )
list_display = ('object', 'point', 'content_type', 'object_id')


Point in the model we are working on so Point._meta.get_field('point') is accessing the field called point of the Point mode. The code below should help you to understand :


class Point(models.Model):
"""
"""
point = models.PointField(verbose_name=_("point"),srid=4326)
content_type = models.ForeignKey(ContentType,
related_name="content_type_set_for_%(class)s")
object_id = models.CharField(_('object ID'),max_length=50)
object = generic.GenericForeignKey(ct_field="content_type", fk_field="object_id")
objects = models.GeoManager()

def __unicode__(self):
return 'Point for %s' % self.object



The last line is actually where the geodjango specificity is :
* PointWidget = admin_instance.get_map_widget(point_field)
get_map_widget is defined here

Now that we have a PointWidget we can use it in our form. Here is is a small example :


class PointForm(forms.ModelForm):
point = forms.CharField(widget=PointWidget())
class Meta:
model = Point
exclude = ("content_type","object_id")
class Media:
js = ("http://openlayers.org/api/2.6/OpenLayers.js",)


Now you can use geodjango super widgets in your forms.

Dec 13, 2008

Social Product Development Network

I am a bit overwhelmed by the number of excelent PLM related blog posts. Today I am going to try to answer the question asked by Oleg : How to implement Social Networking for PLM ?

PLM has always been the concept of managing the information produce by a variety of Computer Aided Authoring tools. In a PLM system everything is organized around the BOM, so all this information is linked to Part and the Parts are organized in a tree.
PLM vendor love to claim that they are also capturing the communication thanks to their workflow system.
But the reality is that these workflows capture at best 2 or 3 very well formalized process : RFQ, ECN, ...

But everybody in the companies knows that the most important discussion about the product developement are done else where : by emails, over phone, meetings... This is even more true with the concept of global engineering team located in several countries. Some companies have a very aggressive objective in term offshore engineering and design, I know a couple of companies that are targeting 50 % of these activities in low cost countries.

I think that time has come to invent yet another acronym Social Product Development Network -- SPDN. This new marketing beast will offer :
  • an internal blog to every employee with the possibility to publish article on internet.
  • a wiki to capture the user generated documentation around the Product
  • An issue tracker to follow the issue related to the product development
  • A twitter clone to exchange short messages
  • gallery to share pictures of the product prototypes, tests , ...
  • BOM management
  • PDM
  • Workflow to control the 2 or 3 key processes
You are going to tell me that this is impossible because PLM vendor does not have the resources/skills to implement all this software stack. You are right but the good news is that you don't have to do it. All this and much more is already available out there in many flavors. For example I have been recently contributing and using such platform recently. It is called Pinax a video presenting the solution is available there. The only pieces that you will have to worry about are the core of the PLM (BOM, PDM, Workflow) and for each of them you will have only to focuse on implementing the vision on an leveraging a modern infrastructure.
No you don't have to develop yet another ORM it is there. No you don't have to invent yet another template language, or an url dispatcher, a cache stragetgy. You use all your ressources to focus on the implementation of the PLM vision rather than the technical details. This technology power web site that receive million hits a day. So I assume it will scale the PLM needs.

But all this requires a complete shift from selling software licences to sell a vision.

Conclusion :
If you really have a vision for PLM sell it with the service to implement it rather than the licences.

Dec 12, 2008

Django fulltext search -- part 1

I have been recently working on adding full text search to a django based web application since this is not yet particularly well documented. Here it is my little stone to the amazing pyramid already in place waiting for you to use it. In order to add this "must have" feature to your great web application you will need 3 components :

Starting by the latest the first challenge is to get it installed, I haven't had luck with the usual "apt-get install solr". It must be me but after an installation I was only getting a blank page instead of solr admin interface. On #solr channel someone advise me to just grab the latest release tar ball and to run with it. Then I have performed the tutorial this help me a lot to get started with solr. Believe it or not when you are there you have done the hardest part of it.

Now you need to go inside the django project you want to work on and follow the following recipe :
  1. download and put into your PYHTONPATH djangosearch and pysolr
  2. Add djangosearch inside the INSTALLED_APPS tupple
  3. add the following settings :
  • SEARCH_ENGINE = "solr" # solr, lucene, xapian, estraier
  • SEARCH_RESULTS_PER_PAGE = 10
  • SOLR_URL= "http://localhost:8983/solr"

At this point you are almost done the last operations you will need to do is to define inside models.py the fields you want to index with solr. In order to do this djangosearch come with a class called : ModelIndex
For example if you want to index : tease, and story fields of your Story model you will do something like this :

from djangosearch import ModelIndex
class Story(models.Model):
...
tease = models.TextField(_('tease'), blank=True)
story = models.TextField(_('story'), blank=True)
...
index = ModelIndex(fields=["tease","story"])


Now you need to tell solr about this 2 new fields this is in fact easier that it seems to be, the theory is explained there. In practice what you need to do open this file (apache-solr-1.3.0/example/solr/conf/schema.xml) with your editor of choice and add this 2 lines inside the fields section:




You will now need to restart solr in order to take this modification into account.

The meal is now ready you can go and taste it.
  • Open a browser go to the admin interface and add a new story or update an existing one.
  • Open a django shell and enjoy your new full text search

from dajngosearch import search
search_results = search("foo")

I hope this will be useful to someone and will avoid you the pain I had to get all these pieces dancing together.
I would be glad to read from you what can be improve in this raw recipe and also what you are doing with your search and search_results.

Dec 8, 2008

Blindness of closed source software company

Since yesterday I am using this blog in a slightly different way than usual. For some reasons it seems that something is happening in PLM realm.

This post is again a reaction to the post from Oleg Shilovitsky's. The post was titled :

Who will play role of Google Maps in PLM?

I think this is an excelent question. I am taking this opportunity to add some information on how this could happen. I have been working closely with both concept, recently I have been working hard to developed a Geographic Information System (GIS) platform to manage Geo localized content. I will make a post related to this platform as soon as it is ready for prime time.
Let us get back to the question, just a bit of history before google map GIS was complex, expensive, not web based software, closed source, understood and used by a very small subset of the population. Do you see any resemblance with PLM ? :-)
Google maps brings it to the mass.


I agree with you that none of the application or technology below deliver something close to what google maps bring.
  • Enterprise Mashups (application area emerged from ability to mix data coming from different sources and present it to user.
  • Microsoft Excel Services (high acceptance of Excel user interface make it usable everywhere)
  • Dassault Systemes 3DLive (very promising lifelike user experience comes last year in new DS products).
My explanation to this is that google maps was just the tip of the iceberg the cherry on the cake. The nice UI that you present to your user. But the truth is that in fact there is now a complete open source stacked available to build geo enabled mashup:
This is just to named few but in reality there is plenty of component available with a vibrant community around them.
So if PLM want to get close to that ubiquity PLM vendor will need to open the doors and the windows in order to get some fresh air, idea in. This put them in front of 2 challenges :
- Ability to deliver code able to compete with other open source software. In my eyes that means improve their quality and documentation
- Build a business model to keep up this effort and make benefits from it.

Afresco manage to do it in a different realm I am sure someone will do it the only questions left are who and when.

Dec 7, 2008

What if SMB as a vision for PLM and PLM vendors don't undertand it.

This post has been started as a comment of a post from the virtualducthman but since it start to be too long I have decided to write my own post. This is not a rant but rather the impression I had after I moved from a famous PLM Vendor to tier one automotive supplier.

As always extremely well written post without any flaw in the reasoning if you buy in the initial hypothesis. However I disagree with the statement that you use as foundation : Software vendor are proposing excellent product with a good ROI and SMB customer don't understand it because they do not have a vision.

In fact as PLM manager for a tier one supplier in automotive I have seen quite the opposite. I have seen all the big player coming and trying to approach me with solutions that have all the exact same problem :
  • No interoperability between authoring tools in the Engineering department and Plant
  • No backward or forward compatibility between version with the tools that they are supposed to integrate with.
  • All these tools share an interesting feature, no open standard to exchange data (xml, json, rss, atom, ...)
  • Impossible to mash up the information you put in their repository with other source of information. Ok, I agree impossible is a bit to strong you can always develop this feature and add it to your application. At least by designed these applications are making such thing hard.

What if rather than saying that SMB do not have a vision we try to find a different explanation to your observation. I tend to believe that they have an extremely well defined vision. SMB know that if they want to be in the business in five years from know they need to be agile in order to transform the way they do things to adapt themselves to the new constraints of the OEM. It is interesting to note that nobody is blaming Ford, GM, ... of not being able to see that they have good chance to go bankrupt in some month from now. It is interesting that many people blame now these companies of not being able re-invent/adapt their products to their market. when all of them where using PLM systems and had huge PLM projects on going. May be this can be an interesting post to develop this idea. Why PLM system only help you to develop faster/cheaper product that you are used to produce ? or Why PLM system can hurt your creativity in introducing innovative to your customer ?

In order to develop this agility SMB need to put very high in the list of capabilities for their information system the following features : Agility, re-configuration, continuous evolution / transformation, openness, ease of integration with unknown system, overall strategy of the PLM vendor.

That put us in front of a slightly different picture, isn't it? I am not going to develop in that post why and how each of this term could help to contribute to convince the SMB to change their mind on the value proposition your are doing but I am convince that the impact will be significant. Also I hope that you understand that now known on the solution on the market give them such freedom.

As an example, Alfresco has proven that this approach can work, I understand that this is not in the PLM realm but in content management and web content management but this area shared a couple of similarity with PLM. The market was dominated by closed source vendors providing the same kind of solution for the last 15 years, with no major evolution nor integration of the new tools introduced by the development of internet : wiki, Blog, chat, tag, fulltext search, low cost, commodity software, ... Of course each PLM product have changed several times their names but unfortunately the code, concepts, people behind them are the same it was just a marketing transformation.

Where is the fresh air, idea, innovation ... May be this industry need to show some code rather than powerpoint. It would be nice also if they change their approach in trying to force their solution rather than listening to their customer SMB is not OEM. They do not have will/money to spend in top/down project they need to be able to scale their solution with a bottom up approach. They need to be able to scale from 1 project to a department then a division and at the end to the whole company.

Any way Mr virtualduchman thanks for writing your excellent posts. This post shouldn't be read as PLM do not bring value to SMB but rather like PLM bring value but this can be highly increased by listening to their needs and feedbacks.