Performance testing of Liferay

[12年前 に Pat McBennett によって更新されました。]

We are running an internal Proof-of-Concept on Liferay, and as part of that we need to generate performance statistics.

As a first step, we simply wrote a very simple servlet that we deployed to both Tomcat (as a servlet) and Liferay (as a simple Portlet). We used JMeter to load-test this simple servlet - it just displays a button which when clicked fires a XmlHttpRequest (below) to a servlet that just returns a small text message. The result from the XmlHttpRequest is simply displayed in a text field. We can't think of a simpler test really!

The Javascript code (using jQuery) is extremely simple:

function submitClick() {
$.post(getContextPath() + "/SendBackTextMessageServlet", {
"inputDummyData1" : $("#dummyField1").val(),
"inputDummyData2" : $("#dummyField2").val()
}, function(data) {
$("#resultTextField").val(data);
});

We used the latest versions of Tomcat and Liferay (the Liferay version bundled with Tomcat). We removed all the sample portlets deployed with the out-of-the-box Liferay installation, and so just deployed our simple portlet described above. Our initial tests showed a lot of embedded resources with Liferay (e.g. Liferay logo image, CSS files, Liferay button images (minimising Portlet, closingPortlet, etc.) - and results that showed CPU usage at peak being 4 times higher than with the same load on Tomcat, and response times being 100 times higher at peak load. We configured JMeter to not request all these embedded resources (since our pure Tomcat servlet didn't request any of these resource), but the CPU difference stayed high at 2.4 times higher for Liferay, and the response times remained at 30 times higher.

We want to investigate these differences a bit more, but were hoping for some guidance for anyone who may have run similar performance tests. We've seen the Performance whitepaper from Liferay (2011), but is it possible to get the actual source code for the tests run in that study (or a packaged set of Portlets to run our tests against)? Or can we run the Liferay tests on our servers here to verify the results? We know comparing 'Tomcat' to 'Liferay deployed on top of Tomcat' is comparing two different things, but really we just want to understand the performance degradation that using a Liferay container introduces when working with a simple standalone Portlet that could in theory be implemented as just HTML and servlets?

[12年前 に David H Nebinger によって更新されました。]

Okay, as you've seen Liferay renders a whole page, regardless of situation. Testing a page w/ just a button compared to Liferay rendering a whole page will not be fair.

Also jQuery is not part of the default Liferay 6.x environment, AUI/YUI is. That means that now your Liferay environment is loading not only AUI/YUI loaded, but also jQuery on top of it. And Liferay uses javascript for the WYSIWYG editors and many other widgets, Liferay is still doing more.

So to get things a little more equal, you should:

a) create a simple theme that removes all images, javascript, and css requirements. Actually pretty easy to do.
b) read the performance whitepaper on further performance tuning you can do.
c) refactor the javascript on both sides to use YUI, and drop the jquery.

[12年前 に Pat McBennett によって更新されました。]

Hi David,

Thanks for the quick and thoughtful reply.

You raise perfectly valid points of course, and I understand where you're coming from - but let me address each one from our perspective:

a) '...create a simple theme...' - yep, good point, this would be a fairer test, although we did configure JMeter to *not* pull back all those embedded resources of images, CSS, etc (and of course we configured our Liferay instance to only have our single test portlet deployed, so none of the sample portlets were taking up memory). So I think our test is very close to what you suggest, but I still think your suggestion is worth doing and comparing the results.

b) '...reading the whitepaper...'. We have read the whitepaper, and it's very good. But that paper gives performance stats for Liferay running on given hardware - which is fine of course, and valuable information. But what it doesn't do is compare Liferay to a more lightweight approach (e.g. just using Tomcat to serve up the same simple UI). I fully understand that Liferay's whole reason for existing is that it offers far more power and flexibility that just Tomcat - all I want to understand at this point is what is the performance cost of all that extra power when all you want to do initially is serve up a simple UI. In other words, as per the whitepaper, if Liferay can support 11,000 concurrent users with a mean login time of <500ms, how many users and what response times could the same hardware support if we just used Tomcat and HTML + Javascript???

c) '...refactor the javascript to use YUI...' - well, we are loading the same jQuery library on both our Liferay portlet and our Tomcat servlet, so we're still comparing like-with-like (and jQuery is our standard Javascript framework, so we can't really change that). I know Liferay is *also* loading YUI, and that our sample web app isn't leveraging that, so in effect Liferay is just wasting memory by loading YUI that isn't needed by our Portlet, but that is part of the extra 'bloat' of Liferay (and again I know that Liferay is more 'powerful' and has more 'features' as a result), but it is valid in our case simply because jQuery is our company standard Javascript library, and both the solutions we are comparing are using it in the exactly the same way.

Again, thanks for your feedback David - I'd certainly appreciate any further thoughts you'd have on the above.

Pat.

[12年前 に David H Nebinger によって更新されました。]

If you're comparing a simple servlet in tomcat, one that has a basic form, a button, and the jquery stuff, it still will not compare to the Liferay 'equivalent'.

The Liferay side still has the theme and it's javascript requirements, etc. On the browser side, for example, you're still loading just jQuery on the servlet side but jQuery+AUI+ on the Liferay side, so the memory usage per connection is still going to be different, and an apple-to-orange comparison.

There are other performance tuning aspects that can be implemented. Adjusting the filters used (or not used) in web.xml and portal-ext.properties can have a big impact on performance. On the servlet side you're probably not invoking any filters, but the Liferay side is probably invoking all of the default filters.

...

Basically, Pat, I can tell you that you will not see the same performance metrics of your servlet to your Liferay portlet. They will not be the same, although through a lot of trial and effort you could get them close.

But in the end, the Liferay environment that you end up will will not be a usable portal environment - theme would be completely minimized, javascript and filters gutted, etc.

[12年前 に Pat McBennett によって更新されました。]

Hi David,

Thanks again for your input - very helpful.

I agree completely with all your comments - naturally Liferay will introduce an overhead compared to a simple Servlet component (because Liferay needs a way to introduce all it's extra whiz-bang features). All I'm trying to address is how much of a performance impact it introduces.

For example, hypothetically speaking, if invoking a simple Liferay portlet resulted in a response time of 500 milliseconds, when accessing a 'similar' Tomcat servlet took just 5 milliseconds (at high load), I need to be able to justify that discrepancy. I certainly would never expect there to be *no* discrepancy - I just need to determine what that discrepancy is so that we can say 'Over basic servlets we loss X degree of performance, but X is small enough, and acceptable enough, for us because we get all this lovely extra reusablility for it'. Currently we're seeing a difference of 5ms(Tomcat) to 180ms (Liferay) at not-particularly-high load, and I'm just trying to understand where that's coming from (some of our guys here are already happy that the discrepancy is only 180ms, I'm just more pedantic!).

You've certainly given me food for thought though, for example we can easily add the AUI download to our Tomcat example to improve the apples-to-oranges comparison a bit, and I'll investigate removing some of the 'default' Liferay filters that my simple tests wouldn't need (e.g. maybe Liferay uses a number of 'Social Media' or 'Blog' filters by default that might be skewing our results).

Thanks again David!

Pat.

[12年前 に Olaf Kock によって更新されました。]

I don't understand what kind of performance test you're running: Do you try how many concurrent users a servlet can handle vs. a portal? Do you compare the answering time for a single request?

The background is that I consider the first scenario (how many concurrent users can be handled) to be the approach to go for, but also you need to be careful to configure your test environment in a meaningful way: Browsers will request javascript, css, images and icons only once (typically) and cache them for later use. Plus, in any load test, you typically want to compare "warm" environments, e.g. resources and database content already being largely in cache.

Also, remember that 100 concurrent requests are not the same as 100 concurrent users. This is what I most frequently see in load tests that don't match the expectations. You typically want to go through the hassle of defining realistic scenarios (will your users log in? will they do other things than just accessing your homepage application?).

If you're only interested in your portlet's performance, there's a few options that you could configure in portal-ext.properties to include some individual portlet's execution time in the resulting page - it's a while since I used it, but if you find these settings: The result will be at the end of the page in a HTML comment. This might provide valuable data too.

And, finally, you should not worry about the portal being slower than a servlet, because it most likely will be. But, as you already say, you already get a great deal of services within it, that you'd have to write for yourself otherwise (eating performance in CPU- and in developer-cycles). You should define what your concurrent user scenario (and projected application use) is and if the site (whatever technology used) can handle it - and identify the scaling opportunities should you ever need to go way beyond those initial requirements.

To answer one of your earlier questions, I believe that the scripts used for the performance whitepaper are not available - I expect them to be specific to the environment our team uses to conduct those tests and probably not easily extracted from there. (but this is only a guess)