Joomla 1.5 & Drupal 6.1 Performance Comparison

Introduction

Disclaimer: The absolute numbers here are not representative for losooperimes of either application on your server.
The thing we're looking for here is how results compare to eachother.

Hardware:
-Acer Aspire 7720G l
-Intel Core 2 du0 T5250 with Santa Rosa chipset (1.5Ghz 667MHz FSB 2MB L2 cache)
-2GB DDR2

Software
-XAMPP package with Zend Optimzer disabled
-No opcode caching enabled
-MySQL query caching enabled with default settings
-Windows Vista running classic theme (yes vista is lame, I know)

-SVN checkout of joomla 1.5
-CVS checkout Drupal 6.1
-Jmeter
-Apache Benchmark

Part 1: Testing for load times with Jmeter

Test Setup:

• concurrency: 1 user
• Requests: 1000
• Jmeter will run these requests as fast as it can

What we are trying to test here is the user experience on a drupal/joomla site that is backed by an adequate hosting provider. Needless to say, the faster the pageload/latency time, the better is the user experience.
I'm not stress-testing this with multiple concurrent users because as for pageload times this would yield spurious comparative results.

I tested Drupal and Joomla firstly as a fresh install+1 content item, and then with a setup that is a more realistic approximation of a real-life website, the latter option holds 11 nodes/articles, of which 10 are displayed on the frontpage (this invokes the pager or both systems), and 2 sidebar blocks (login/search).

From my log, my steps of making the fresh Joomla and Drupal "equal":
-Add 1 5paragraph lipsum page and display it on frontpage of either site
-removed login form from drupal site (joomla does not display login form by default
-In joomla I added a "readmore" in the content where Drupal had made a teaser break automatically.

Caching:

The benchmarks with caching enabled will use the caching features that a novice user would be able to set up (so no op-code caching/custom mysql cache settings).
On each site I hit the frontpage a few times as anonymous user in order to write its cachable contents to the cache table

In Drupal I went to admin/settings/performance and turned on the following options:

• Caching mode: normal
• Page Compression: enabled
• Block cache: enabled (though there are nog blocks)
• Optimize CSS files: enabled
• Optimize JS: enabled
  

For Joomla:

• Gzip compression: Yes
• Cache:yes

Results:

Results in milliseconds; median pageload time/median latency. Pageload time is the time is takes before all objects of the page are loaded. Latency is the time until the first response from the server, this is when a page starts building in front of the user. You can see that the difference between latency and pageload time is larger in the test setups with 11 nodes because there is more html content to download.

I chose the median instead of the average because the average is more sensitive to outliers.

Conclusions

1. Drupal is significantly faster than Joomla in all 4 setups
2. Drupal cuts down pageload time by ~74% when caching is enabled on the fresh install and ~86% with the more populated setup
3. Joomla cuts down pageload time by ~23% on the fresh install and ~20% on the more populated setup

Limitations

This test has several important limitations that should be taken into account when making inferences from the data presented:

1. This benchmark was all done on a laptop, so there was no internet conenction involved in the downloading of objects. This means that when you take a (slow) internet connection into account the pageload/latency difference is going to be significantly larger.
2. Because this was done on a single machine, Jmeter will have put a load on the computer in itself, thereby increasing response and load times, but comparative results will still be allright.
3. This setup was done on a Windows+Xampp setupt which probably differs in many ways from the more common Linux (LAMP) setup. I don't think it matters a lot for comparative test results but I'm not entirely sure, maybe someone can comment on this
4. This comparative test lacks a benchmark for a "fully optmized" setup of a drupal and a Joomla site. I would like to do such a test some time but I'm not aware of all the hacks and modules that are available to make Joomla faster. If someone has a nice tutorial/article about Joomla optimization for me that would be great.

Appendix

Here is a graph that displays the pageload times for all the setups:

For the avid reader who noted that I have a header labeled "Part 1"... Part 2 (regarding scalability) is for another day. This testing stuff takes more time than you'd think :).

p.s. Thanks Khalid for 2bits for giving me some pointers and feedback before publishing this.