Test of Liferay performance on quad core CPU running Linux

User Rating: 4 / 5

Star ActiveStar ActiveStar ActiveStar ActiveStar Inactive
 

Performance testing Liferay 5.1 on a Intel Core2 quad core CPU running Linux

 

Introduction

I have performed a performance analysis of Liferay 5.1 running on Linux 2.6 operating system using an Intel Core2 Quad core CPU, Q9450.

The results will allow us to draw some conclusions about the performance, but they also open up new areas where further testing should be performed, to find the factors limiting the performance and settings which can be tuned to increase the performance even more.

My objective has been to investigate the performance of Liferay serving a public facing, no-login, web site.

 

Short summary

Liferay 5.1.0 running under Linux 2.6.27 on an Intel Core2 Quad 2.67GHz CPU, is able to handle about 2.200 requests per second from at least 1.000 simultaneous users, with average response time below 1 second. This performance was achieved using the JRE setting  "-X:incgc", which increases the performance up to 30%. Only a few URLs were sampled in the tests used to gather these data.

When all the 7500 different URLs for the sample site used were tested, the performance dropped significantly, down to only being able to handle 26 requests per second from at least 1.000 simultaneous users. An article that looks into the details of this performance drop will be published in the coming weeks.

Update 7. September : It seems like the significant drop in performance was caused by missing "cacheable" settings in the database, this was a bug in Liferay 5.0.0 that has now been fixed. See this forum post for more details. I will update the database and rerun the tests, and come back with more info.

Update 13. September : Using VisualVM to look at cache statistics, and then editing Ehcache file for Liferay, I was able to improve the performance from 26 requests per second to 300 request per second. See this article for all the details.

 

Test setup

The following table lists the hardware and software used for testing, you will find the full specification here.

Server running the actual Liferay instance
 Hardware 
 Intel Q9450 Quad core CPU, 2.67 Ghz, 8GB RAM
 Software 
 Liferay 5.1.0 bundled with Tomcat 6.0.16
 OpenJDK 1.6 and Sun Java 1.6,64 bit
 MySQL 5.0.51
 Ubuntu 8.04, 64 bit
 Client running JMeter to generate load on the server
 Hardware 
 Intel E8400 Dual core CPU, 3.0GHz, 2GB RAM
 SoftwareJMeter 2.3.2
 OpenJDK 1.6
 Ubuntu 8.04, 64 bit

 

The machines were connected using a Linksys 5 port Gigabit switch.

 

Test procedure

Liferay 5.1 bundled with Tomcat 6 .0.16 was installed. The portal database was residing on MySQL 5 on the same machine.

The liferay database script for version 5.0 was loaded into the portal database, since this contains the testdata for the liferay web site, and provides a useable real life database. The database was upgraded to version 5.1 by running database upgrade scripts. This sample web site, representing a copy of the Liferay.com website, does not seem to be part of the Liferay 5.1.0 release, and therefore the version from 5.0 was used.

The Liferay server was started with different Java JRE settings, which is documented below. The maximum heap memory size and the garbage collection strategy was varied, to see what effects it had on the performance.

The " -XX:MaxPermSize=128m" JRE setting was used for all tests.

 

The JMeter running on the client PC was started with JRE settings "-Xmx1500m -Xms1500m".

Tomcat was running configured with maxThreads set to 300 on the HttpConnector.

For some of the tests, the Tomcat native libraries was also installed. Note that in order to get the native libraries to work, I had to turn of the IPv6 support in Ubuntu.

 

JMeter test scripts

The script homepage_without_cookies.jmx, just loads the start page of the portal, the /web/guest/home. Most of the tests were run using this script.

This script will only request the homepage of Liferay, and the Liferay server will respond with just the HTML for the homepage, which is 49kb in size.

In some test this script was modified to download all the images, javascript and css references by the homepage, and this requires 36 requests for each request for /web/guest/home, and totals 410kb in size.

There is no cookie manager in this script, which means that each and every request creates a new session on the Liferay server. So each request represents a brand new "user" hitting the page.

Each test run therefore created between 100.000 and 500.000 sessions on the Liferay server.

 

The script homepage_with_cookies.jmx is similar homepage_without_cookies.jmx mentioned above, but this script has a cookie manager, and only each thread running in Jmeter will create a

new session on the Liferay server. This means that only between 100 and 500 sessions were created in each test run.


The script main_menuitems_without_cookies.jmx is similar homepage_without_cookies.jmx mentioned above, except that this script requests the homepage plus four of the top menu item pages.

 

The script whole_site_without_cookies.jmx is using av text file containing about 7500 different URLs, which represents all the different URLs for the Liferay sample site used. These URL were gathered using the "wget" command to crawl the site, and then using some "sed" magic to get all the URLs in one file. The file with the URLs can be downloaded here.

 

Test results

Running the test "homepage_without_cookies"

OpenJDK 1.6.0b11, without Tomcat native libraries, and JRE setting "-Xmx4000m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 100 100.000 No 55 2 3.428 1.578
 200 200.000 No 100 1 6.270 1.731
 300 300.000 No 162 1 7.247 1.714
 400 400.000 No 300 1193.891 956
 500 250.000 No 263 2 93.205 1.577
 100 50.000 Yes 867 291 7.954 113

What is worth observing here, is that the maximum response time is getting awfully high. When monitoring the CPU load on the Liferay server, I noticed that one of the cores went to 100% while the other three droppend to 0%. At the same time the network traffic went from between 80-120MiB/s to 0MiB/s. This is a clear indication that the JRE was doing garbage collection, and in that time period no requests were served. These garbage collection consuming 100% CPU lasted about 5 seconds, and were occurring about every 30 seconds in the beginning of the test. At the end of the tests, at least with 500 threads (and 250.000 samples -> 250.000 sessions), the one core were running at 100% while the rest were running at 0% almost constantly.

Normally the  four CPU cores were runinng at between 20-60 %, and the network card was sending between 80 and 120 MiB/s.

But the results show that it is possible to serve several hundred threads about 1700 pages per second with average response around a few hundred milliseconds.

 

OpenJDK 1.6.0b11, without Tomcat native libraries, and JRE setting "-Xmx6500m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 100 100.000 No 45 2 1.621 1.773
 300 300.000 No 167 1 9.566 1.773
 500 250.000 No 248 1 45.325 1.707
 500 250.000 No 230 1 45.314 1.867
 100 50.000 Yes 865 291 11.561 109

With the larger heap size, we see a performance increase.

 

OpenJDK 1.6.0b11, with Tomcat native libraries, and JRE setting "-Xmx6500m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 100
 100.000 No 44 1 2.532 1.865
 300 300.000 No 154 2 4.710 1.805
 500 250.000 No 270 1 9.166 1.711
 100 50.000 Yes 2.009 300 40.381 45

With the Tomcat native libraries in place, we see a small performance increase for most of the tests, but a big drop for the case where all references images are downloaded. More investigation should be done on the effects of the Tomcat native library.

 

OpenJDK 1.6.0b11, without Tomcat native libraries, and JRE setting "-Xmx6500m"  and "-X:incgc":

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 100
 100.000 No 36 2 240 2.191
 300 300.000 No 118 2 3.570 2.286
 500 250.000 No 204 2 9.740 2.082
 100 50.000 Yes 755 288 1.829 128

The JRE setting "X:incgc", which means that the JRE is during incremental garbage collection, improves the performance significantly. The average response time is reduced by 20-30%, and the throughput increases 20-30%. When looking at the CPU utilization at the server, the CPU usage never reached 100% for any cores, and all the cores were working at more or less the same utilization. This means the there were no situations where the garbage collection was forcing the other Java threads to wait for several seconds.

 

Sun JDK 1.6.0_06, with Tomcat native libraries, and JRE setting "-Xmx6500m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 300 300.000 No 145 2 4.616 1.904
 500 250.000 No 247 2 9.173 1.813

Sun's JDK seems to perform better than the OpenJDK. When I was monitor the CPU utilization on the server, I could see that there were no situations where one core would go to 100% while the rest dropped to 0%. This seems to suggest that the garbage collector is performing better. More investigation needs to be done here, but it seems like the Sun JDK is performing more similair to the OpenJDK running with "X:incgc".

 

Sun JDK 1.5.0_15, with Tomcat native libraries, and JRE setting "-Xmx6500m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 500 250.000 No 243 2 3.891 1.853

The Sun JDK 1.5 seems to perform similar to the Sun JDK 1.6.


Running the test "homepage_with_cookies"

OpenJDK 1.6.0b11, with Tomcat native libraries, and JRE setting "-Xmx6500m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 100 100.000 No 38 2 1.123 2.028
 500 250.000 No 219 2 8.614 2.150
 100 50.000 Yes 819 320 1.385 120

Since this test is using a CookieManager to handle the cookies sent from the server, it means that a new session will not be started on the server for each request. There will only be a new request per thread. The average response time decreases slightly, and the throughput increases about 10-20%. But it is the "Max Response Time" that show a significant decrease, especially in the case where the images are downloaded as well. This descrease is caused by the fact that the garbage collector has far less session related objects to reap, and the garbage collector never uses 100% of a CPU core.

 

Running the test "main_menuitems_without_cookies"

OpenJDK 1.6.0b11, with Tomcat native libraries, and JRE setting "-Xmx6500m" :

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 100 100.000 No 46 1 4.752 1.715
 300 300.000 No 136 1 4.394 1.998

 In this test, five different pages are requested. The numbers are similar to the "homepage_without_cookies". This seems to suggest that Liferay / Tomcat is able to cache all these requests, and the performance is therefore equal to the situation were only one URL is tested.

 

Running the test "whole_site_without_cookies"

OpenJDK 1.6.0b11, without Tomcat native libraries, and JRE setting "-Xmx6500m"  and "-X:incgc":

Number of Threads
Total Samples
Download images
Average Respone Time (ms)
Min Respone Time (ms)
Max Respone Time (ms)
Throughput (requests per second)
 10 30.000 No 241 1 2.723 41
 50 30.000 No 1.802 1 28.997 27
 100 30.000 No 3.899 1 86.483 26

Here we see a significant dropin perfomance. The CPU utilization on the server is constantly above 90%. This major drop in performance can be caused by at least two issues. Either the caching mechanism is not able to cache the 7500 different pages / images that the site contains, and therefore the JSP engine must recreate the pages all the times. Or there could be some portlets on some pages that are not performing well, and these are causing the performance to drop. More investigation needs to be done to find the causes to this performance drop. I imagine that by tweaking the caching settings, so that more of the site is cached by Liferay, should improve the performance significantly.

 

Analysis of the test results

It is not an exact science to relate the number of threads running in JMeter to the corresponding real-life users, but I tend to say that multiplying the thread count in a test by 10 gives a realistic figure for the number of users it represents. The main reason for this big factor, is that people need time to

  1. think about where to move the mouse cursor and clicking a link
  2. moving the mouse and clicking the mouse button

The JMeter test contains no such "think-act time", and a new request is issued as soon as the previous request has finished.

The tests first of all suggests that using the JRE setting "-X:incgc" can increase the performance by 30%. The test also clearly indicates that when pages are cached by Liferay, the performance is several order of magnitudes better than when the pages needs to be created by Liferay on each request. When Liferay is able to serve the pages from cache, the Liferay/Tomact server is able to serve 2.200 requests per seconds, when 1.000 to 5.000 users are constantly hitting the server. If the pages cannot be be served from cache, then the performance drops so that the server is only able to handle about 26 requests per second.

 

Areas needing further investigation

  • The effect of the "maxThreads" in Tomcat. The default value is 40, I plan to run some tests with that default value, I have only used 300 as the value so far.
  • The effect of setting the "session-timeout" in Tomcat. This should reduce the memory load on Tomcat. I have not changed the default value, and I also need to investigate what the default value is.
  • Is there really a performance difference between Open JDK 1.6.0b11 and Sun JDK 1.6.0_6? My tests seems to suggest that the Sun JDK is performing better, but when using "-X:incgc" for OpenJDK, then OpenJDK is best.
  • The effect of Tomcat native libraries. My tests seems to suggest that there is only a minimal effect, I am not sure if I even found a performance difference.
  • The caching configuration in Liferay. Ehcache is used, I need to investigate more on how it works, and check the performance impact.
  • Check how much data is put onto the session object for a user just browsing, not logging in to, a Liferay site
  • Check if there is a way to turn of session creation on Tomcat, and also check if Liferay needs a session object for each user if is serving a public, non-login, web site.
  • Find causes for the significant drop in performance when all the different URLs for the site is accessed. Find out how increase the amount of caching that Liferay does, it is probably just a matter of configuring EhCache.
  • Test Liferay 5.1.1, to see if there are any performance differences.

 

Feedback

Comments, tips, tricks and suggestions for areas investigate are appreciated, drop me an email at This email address is being protected from spambots. You need JavaScript enabled to view it..

 

 

Additional information