ESI : Edge Side Includes (ESI) is an XML-based markup language that provides a means to assemble resources in HTTP clients. Unlike other in-markup languages, ESI is designed to leverage client tools like caches to improve end-user perceived performance, reduce processing overhead on the origin server, and enhanced availability. ESI allows for dynamic content assembly at the edge of the network.
On of the most common difficulties that leads you to not cache a page for a logged in user is that you want to display some custom information for that user. For example you want to be able to display:
* " welcome joes, [ link to his profile ] "
However the rest of the page will be common for all the users. The diagram below explains the composition of the page :
This is a very common pattern, you can also have a header, footer and a navigation block that don't change very often and the rest of the page which is more dynamic like: recent activity, last articles, ... So the idea here is to use varnish to assemble information coming from different urls and having a different lifetime in cache for each item.
So in our example we will cache the "dyn_page" (yellow in the diagram) for five minutes and we will never cache the user info. I am going to start by dumping the code for this toy app and then explain it as we progress.
Here it is the code of the views.py:
import time
from django.views.generic.simple import direct_to_template
from django.views.decorators.cache import cache_page, never_cache
@never_cache
def user_info(request):
return direct_to_template(request, "esi_app/user_info.html")
@cache_page(60*5)
def view_dyn_page(request):
#Simulate the fact that this page take a lot of time to be built
time.sleep(2)
return direct_to_template(request, "esi_app/dyn_page.html")
@never_cache
def view_page(request):
USE_ESI = True
if not USE_ESI:
#Simulate the fact that the dynamic part of the page
# take a lot of time to be built
time.sleep(2)
return direct_to_template(request,
template="esi_app/page.html",
extra_context={'USE_ESI':USE_ESI,})
The views.py contains 3 views one that can display the information for each individual block (yellow and green in the diagram above) and one that can displays the complete page. Note the "USE_ESI" variable that we will utilize in our template. I have added a sleep of 2 seconds in the code to simulate an operation which is taking a lot of time thus the caching strategies make more sense and my ab test later or will be more meaningful.
Here it is the code of of the urls.py:
from django.conf.urls.defaults import *
urlpatterns = patterns('esi_app.views',
url(r'^page/', 'view_page', name='view_page'),
url(r'^dyn_page/', 'view_dyn_page', name='viewd_dyn_page'),
url(r'^user_info/', 'user_info', name='user_info'),
)
Here it is the code for the base.html:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>{% block title %} {%endblock%}</title>
<style type="text/css">
#user_info {
background:lightgreen;
}
#content {
background:lightyellow;
}
</style>
</head>
<body>
<div id="user_info">
{% block user_info %} {%endblock%}
</div>
<div id="content">
{% block content %} {%endblock%}
</div>
</body>
</html>
Here it is the code for the page.html:
{% extends "esi_app/base.html" %}
{% block user_info %}
{% if USE_ESI %}
<esi:include src="http://192.168.1.18:6081/esi/user_info/"/>
{% else %}
{% include "esi_app/user_info.html" %}
{% endif %}
{%endblock%}
{% block content %}
{%if USE_ESI %}
<esi:include src="http://192.168.1.18:6081/esi/dyn_page/"/>
{% else %}
{% include "esi_app/dyn_page.html" %}
{% endif %}
{%endblock%}
This template use the variables "USE_ESI" to decide whether the page will be built using an ESI server or not. This allows a graceful degradation and will help you to debug your page. In a real life situation this variable might come from a django's context processor. The idea here is that "/esi/page/" is built using "/esi/dyn_page/" and "/esi/user_info/"
{% load webdesign %}
<h1>{% lorem 2 w random %}</h1>
<p>{% lorem 2 p random %}</p>
{% if user.is_authenticated %}
Welcome, {{ user }} -- <a href="/admin/logout">Logout</a>
{% else %}
<a href="/admin/login">Login</a>
{% endif %}
Then you will need to configure varnish to make it do the ESI transformation on the page with the url equal to "/esi/page/".
Here it is the varnish configuration for the /etc/varnish/default.vcl:
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
unset req.http.Accept-Encoding;
#unset req.http.Vary;
}
sub vcl_fetch {
if (req.url == "/esi/page/") {
esi;
}
}
The code below is rather self explanatory it tells Varnish to do ESI substitution on the page located at "/esi/page/". has in my previous post Cherokee is located on the port 8080 and Varnish on the port 6081. The trickiest part there is the vcl_recv, in this section varnish explicitly prevents the backend from gzipping the content.
All the machinery is in place now so you can use your favorite browser to visualize the result :
* go to http://192.168.1.18:6081/esi/page/ to view the page generated by varnish
* go to http://192.168.1.18:8080/esi/page/ to view the page returned by Cherokee
Curl is another tools that is useful when playing cached page :
* curl [url] wil display in the console the html source code
* curl -I [url] will show the document info in a console.
* curl [url] -H "Accept-g: gzip,deflate"
Here it is the page rendered by cherokee :
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title> </title>
<style type="text/css">
#user_info {
background:lightgreen;
}
#content {
background:lightyellow;
}
</style>
</head>
<body>
<div id="user_info">
<esi:include src="http://192.168.1.18:6081/esi/user_info/"/>
</div>
<div id="content">
<esi:include src="http://192.168.1.18:6081/esi/dyn_page/"/>
</div>
</body>
</html>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title> </title>
<style type="text/css">
#user_info {
background:lightgreen;
}
#content {
background:lightyellow;
}
</style>
</head>
<body>
<div id="user_info">
<a href="/admin/login">Login</a>
</div>
<div id="content">
<h1>ipsa sed</h1>
<p><p>Nemo perferendis delectus pariatur aliquid repellendus repellat explicabo facilis, molestiae veritatis odit, accusantium repellat culpa ab laboriosam, iste laborum amet et harum, iusto illum ipsa a quos necessitatibus voluptatem consectetur cumque? Doloremque atque delectus ipsa ad veniam incidunt cum exercitationem voluptates labore, sapiente ducimus deserunt expedita aperiam temporibus omnis magnam qui architecto, pariatur voluptates nesciunt nam ab dolore omnis, quo voluptatem nihil accusamus aperiam excepturi exercitationem? Consectetur mollitia neque quod quas.</p>
<p>Tempore amet voluptate ipsum suscipit placeat exercitationem labore nam voluptas, debitis esse dignissimos, fugiat illum asperiores suscipit deleniti maiores consequuntur, doloribus architecto repellendus dicta nemo corporis explicabo? A fuga ex, voluptates quam dignissimos aspernatur, reprehenderit accusantium id magni ut debitis adipisci esse voluptas tempora, quas doloribus blanditiis voluptatum veniam nam magni et adipisci fuga pariatur provident? Aut ipsum quam quia earum quod cum sapiente officia inventore delectus, expedita ex quia ipsam consectetur exercitationem ut ad sunt illum minus voluptatum, accusantium maxime facere eos numquam explicabo, rerum ab dolorem repellendus, praesentium debitis tempora aut facere sapiente odit veniam quae?</p></p>
</div>
</body>
</html>
I would be glad to hear from you what other varnish tricks can be used on top of a django web application.
[1] http://varnish.projects.linpro.no/
[2]http://yml-blog.blogspot.com/2010/01/response-time-optimisation-with-varnish.html
[3] http://varnish.projects.linpro.no/wiki/ESIfeatures
[4] http://www.w3.org/TR/esi-lang
[5] http://www.holovaty.com/writing/django-two-phased-rendering/
