Dec 29, 2009

flup vs uWSGI with cherokee

Cherokee is one of this many web servers that supports both deployment strategies. Since I have recently blogs about uwsgi
  Several people asked me how do they compare performance wise. Up to now my answer was I don't know and I don't really care because this is not the most critical aspect in my opinion.

However since this question keep comming I have decided to give it a more accurate answer. This blog post takes you on how to setup cherokee with both alternatives and compare their performance. The guinea pig application I have chosen to do this comparison is django-cms because I think it represents fairly well the overhead introduced by the dynamic generation of the page.

Note : in this example django-cms will not be mounted under "/" this will cause all sort of issues if you try to do this but none of them impact the particular page that we try to access.

One of the nice Cherokee's feature is its documentation, here it is the 2 pages related to this blog post :
flup [1]
uswgi [2]

In this blog post I will assume that django-cms is installed, its example project properly configured and that you have added a page to test test against it. I will aslo assume that cherokee, flup and uwsgi are setup.
During this test I have used uwsgi changeset: 145:db356717823c.

Cherokee provides 2 wizard that enables you to get running very fast. First thing first you will need to start cherokee admin :


sudo cherokee-admin -u

media admin
Add a rule to serve the "/media/admin" in my case these files are in "/opt/www/django-cms_tutorial/ve/lib/python2.6/site-packages/django/contrib/admin/media"



media cms
Add a rule to serve the "/media/cms" in my case these files are in "/opt/www/django-cms_tutorial/ve/src/django-cms/cms/media/cms"



Flup


In this case you will need to use the wizard called "Plateforms > django". This wizard takes the web directory and the project directory. In our case :

web directory : /flup
project directory : /opt/www/django-cms_tutorial/ve/src/django-cms/example






Then we need to modify the interpreter command from the "django 1" source because we used virtualenv instead of the global python.
So we need to change it from :


python /opt/www/django-cms_tutorial/ve/src/django-cms/example/manage.py runfcgi protocol=scgi host=127.0.0.1 port=37134
to:


/opt/www/django-cms_tutorial/ve/src/django-cms/example/start_fcgi.sh


Then you need to create this file called start_fcgi.sh

example$ cat > start_fcgi.sh << EOF
> source /opt/www/django-cms_tutorial/ve/bin/activate;
> python /opt/www/django-cms_tutorial/ve/src/django-cms/example/manage.py runfcgi protocol=scgi host=127.0.0.1 port=37134
> EOF


chmod +x start_fcgi.sh

Restarts cherokee and direct your browser to "/flup/".

uWSGI


In order to setup the uwsgi server we are going to use the cherokee's wizard and then modify the result to adapt it to our particular use case.

uwsgi wizard takes only one argument the path to the uwsgi configuration file :

/opt/www/django-cms_tutorial/ve/src/django-cms/example/example_uwsgi.py





We are going to edit the interpreter command to adapt it a bit to our use case :


/usr/local/bin/uwsgi -s 127.0.0.1:42597 -t 10 -M -p 1 -C -w example.example_uwsgi -H /opt/www/django-cms_tutorial/ve/src/django-cms

to :

/usr/local/bin/uwsgi -s 127.0.0.1:46075 -t 10 -M -p 10 -C -w example.example_uwsgi -H /opt/www/django-cms_tutorial/ve/

The file called example_uwsgi looks like this :


import os
import django.core.handlers.wsgi
# Set the django settings and define the wsgi app
os.environ['DJANGO_SETTINGS_MODULE'] = 'example.settings'
application = django.core.handlers.wsgi.WSGIHandler()
# Mount the application to the url
applications = {'/uwsgi':application, }

Comparison: uwsgi vs flup

flup


Resource usage for concurency equal to 50 :






(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 50 http://192.168.1.18:8080/flup/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /flup/
Document Length: 3485 bytes

Concurrency Level: 50
Time taken for tests: 82.385 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3688000 bytes
HTML transferred: 3485000 bytes
Requests per second: 12.14 [#/sec] (mean)
Time per request: 4119.256 [ms] (mean)
Time per request: 82.385 [ms] (mean, across all concurrent requests)
Transfer rate: 43.72 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 3
Processing: 750 4042 1424.0 3791 11665
Waiting: 750 4041 1424.0 3791 11665
Total: 753 4042 1424.2 3791 11666

Percentage of the requests served within a certain time (ms)
50% 3791
66% 4098
75% 4350
80% 4509
90% 4972
95% 5566
98% 10388
99% 10974
100% 11666 (longest request)

(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 100 http://192.168.1.18:8080/flup/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /flup/
Document Length: 3485 bytes

Concurrency Level: 100
Time taken for tests: 82.614 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3688000 bytes
HTML transferred: 3485000 bytes
Requests per second: 12.10 [#/sec] (mean)
Time per request: 8261.424 [ms] (mean)
Time per request: 82.614 [ms] (mean, across all concurrent requests)
Transfer rate: 43.59 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 3
Processing: 1209 7920 1519.3 7694 14118
Waiting: 1209 7920 1518.1 7694 14118
Total: 1213 7920 1519.5 7694 14120

Percentage of the requests served within a certain time (ms)
50% 7694
66% 8161
75% 8405
80% 8574
90% 9215
95% 10456
98% 13048
99% 13983
100% 14120 (longest request)


Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /flup/
Document Length: 3485 bytes

Concurrency Level: 100
Time taken for tests: 81.737 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3688000 bytes
HTML transferred: 3485000 bytes
Requests per second: 12.23 [#/sec] (mean)
Time per request: 8173.722 [ms] (mean)
Time per request: 81.737 [ms] (mean, across all concurrent requests)
Transfer rate: 44.06 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.2 0 5
Processing: 830 7843 1455.0 7574 14385
Waiting: 830 7843 1455.0 7574 14385
Total: 835 7843 1455.3 7574 14388

Percentage of the requests served within a certain time (ms)
50% 7574
66% 7912
75% 8137
80% 8335
90% 8893
95% 10624
98% 13435
99% 13670
100% 14388 (longest request)

Resource usage for concurency equal to 100 :




uwsgi


uWSGI resource usage for concurrency equal to 50 :






(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 50 http://192.168.1.18:8080/uwsgi/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /uwsgi/
Document Length: 3494 bytes

Concurrency Level: 50
Time taken for tests: 84.013 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3697000 bytes
HTML transferred: 3494000 bytes
Requests per second: 11.90 [#/sec] (mean)
Time per request: 4200.629 [ms] (mean)
Time per request: 84.013 [ms] (mean, across all concurrent requests)
Transfer rate: 42.97 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 2
Processing: 682 4115 472.7 4189 4707
Waiting: 682 4115 472.7 4189 4707
Total: 684 4115 472.5 4189 4707

Percentage of the requests served within a certain time (ms)
50% 4189
66% 4246
75% 4286
80% 4316
90% 4378
95% 4440
98% 4515
99% 4544
100% 4707 (longest request)
(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 100 http://192.168.1.18:8080/uwsgi/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
apr_socket_recv: Connection reset by peer (104)
Total of 839 requests completed

This reveals the conservative nature of uwsgi. I had a discussion about this with Roberto De Ioris. He mades a detailled answer detailled answer explaining the situation. So the bottom line is that we will need to modify our configuration and increase the socket timeout "-z" and the socket listen queue "-l"


/usr/local/bin/uwsgi -i -l 120 -z 60 -p 10 -M -s 127.0.0.1:46075 -w example.example_uwsgi -H /opt/www/django-cms_tutorial/ve/

(ve)yml@yml-laptop:django-cms_tutorial$ ab -n 1000 -c 100 http://192.168.1.18:8080/uwsgi/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.18 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: Cherokee/0.99.37
Server Hostname: 192.168.1.18
Server Port: 8080

Document Path: /uwsgi/
Document Length: 3494 bytes

Concurrency Level: 100
Time taken for tests: 86.246 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 3697000 bytes
HTML transferred: 3494000 bytes
Requests per second: 11.59 [#/sec] (mean)
Time per request: 8624.594 [ms] (mean)
Time per request: 86.246 [ms] (mean, across all concurrent requests)
Transfer rate: 41.86 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.7 0 7
Processing: 762 8248 1460.7 8509 9706
Waiting: 762 8248 1460.7 8509 9706
Total: 766 8248 1459.5 8509 9706

Percentage of the requests served within a certain time (ms)
50% 8509
66% 8669
75% 8852
80% 8998
90% 9182
95% 9360
98% 9474
99% 9566
100% 9706 (longest request)

uWSGI resource usage for concurrency equal to 100 :





Conclusion

flup is slightly faster than uWSGI at this point but this has to be put into perspective and you need to take into consideration the features that come with uWSGI, an exhaustive list can be found hereuWSGI is unable to complete the ab test with the following argument -n 1000 -c 100 with its default settings you will need to adjust the timeout socket and socket listen queue. However it is interesting to note that the memory footprint for uwsgi is lower by an order of magnitude and that my laptop remains responsive during the ab test using uWSGI where it was almost taken down by same test with flup.

[1] http://www.cherokee-project.com/doc/cookbook_django.html
[2] http://www.cherokee-project.com/doc/cookbook_uwsgi.html
[3] http://projects.unbit.it/uwsgi/
blog comments powered by Disqus