Liferay JVM Tuning

When people say JVM tuning, most of the time they mean tuning GC. Before we start to tune JVM for Liferay Portal, let us talk about some basic conceptions of GC.

As we all know jvm can handle useless memory block automaticly, this release us from manually free useless memory blocks which is the NO.1 burden for all C and C++ programmers. With the help from GC our life is much easier, we don't need to worry about memory leak for every single line of our code. But this does not mean you can ignored freeing memory.

So let me ask you a very common question: Manually Memory Management VS. Automatic Memory Management, which one is better?

The answer will change depending on how you define better, and who is making the definition.
Let us see the first part, how do you define better?
If easy is better, of course, Automatic Memory Management is better. You don't need to do anything explicit memory management, JVM takes care of everything.
If flexible is better, of course, Manually Memory Management is better. Everything is under your control.

For the second part, who is making the definition?
If you are C and C++ guru, of course, Manually Memory Management is better.
If you are not, most people whom are reading this blog should be this case:), Automatic Memory Management is better.

If you are not very good at Manually Memory Management, you will be very easy to make following mistakes:

Dangling references: It is possible to deallocate the space used by an object to which some other object still has a reference. If the object with that (dangling) reference tries to access the original object, but the space has been reallocated to a new object, the result is unpredictable and not what was intended.

Memory leak:These leaks occur when memory is allocated and no longer referenced but is not released. For example, if you intend to free the space utilized by a linked list but you make the mistake of just deallocating the first element of the list, the remaining list elements are no longer referenced but they go out of the program’s reach and can neither be used nor recovered. If enough leaks occur, they can keep consuming memory until all available memory is exhausted.

Another big problem for memory management is the memory fragments. With Manually Memory Management it is very difficult to fix this problem, but with Automatic Memory Management it becomes very easy.
Suppose you have 4MB heap and you allocate three objects which are 1MB, 1MB and 1MB, so your heap should looks like this:

Then you free the second object, so your heap should looks like this:

Finally you need to allocate the 4th object which is 1.5MB, at this point you have 2MB free heap space, but you can't fit a 1.5MB object into it.

The memory fragments can cause fake out of memory, and can also make memory allocating very slow. Because you will need a link list to record all the free blocks, and do a search for every allocate request.

Ok, that is enough. For more information about JVM memory management you can read memorymanagement_whitepaper
Let us start our real topic:Tune JVM for Liferay Portal.

1)Check your portal server's CPU and memory. Our portal server has 2 4-cores cpu, which means 8 cpus for JVM, and we have 8GB physical memory.
Because we have multiple cpus, we should try to use multi-thread to speed up gc.
In Sun JDK5, there are 4 build-in gc types: Serial Collector, Parallel Collector, Parallel Compacting Collector and Concurrent Mark-Sweep (CMS) Collector.
For our server, we choose Concurrent Mark-Sweep (CMS) Collector, because we have 8 cpus and we need a short gc pause time.(For more detail, please read memorymanagement_whitepaper). Use -XX:+UseConcMarkSweepGC to turn on this option.

2)Fix your Xms and Xmx to a same value. Because our server is a dedicated server for Liferay Portal, once you know the suitable heap size for the whole JVM, there is no reason to define a range then let jvm to group the heap size. We should directly set the heap size to the suitable value. Use -Xms2048m -Xmx2048m to set the heap size.(i will explain why is 2048 later.)

3)Set the young generation size(Don't know what is young generation? again read memorymanagement_whitepaper). For the same reason as above, we should fix the young generation size directly to the suitable size. Experimentally we make the young generation size 1/3 of the whole heap. Use -XX:NewSize=700m -XX:MaxNewSize=700m to set the young generation size.

4)Set PermSize. PermSpace is used to store Class Object, so the size depends on how many classes do you have. For our case 128MB is big enough. If you get a OutOfMemory error complain for there is no more space in PermSpace, you should increase this value. Use -XX:MaxPermSize=128m to set the PermSize.

5)Set the SurvivorRatio to young generation. There is no particular rule for this, the value is from watching(By VisualVM). You should try to make survivor size bigger than the peak memory use size. Here we set the ratio to 20. Use -XX:SurvivorRatio=20 to set the ratio.



Ok, now let us talk about why set the heap size to 2048MB, we are using 64bit JVM, we can support more than that heap size. The rule for choosing heap size is that fit is the best, never making it too big. The algorithm for gc tells us as the heap size increasing by a linear way, the time for gc will grow much more faster than linear. So we should just set the heap size to meet our need. In our case, by testing with VisualVM, we choose 2048MB for the heap size. From the gc time for young generation and old generation you can tell whether this heap size is suitable. Generally the average young generation gc time should around 20ms, the old generation gc time should around 200ms~400ms.

The finally startup parameters should look like this:
JAVA_OPTS="$JAVA_OPTS -XX:NewSize=700m -XX:MaxNewSize=700m -Xms2048m -Xmx2048m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=10"

For more info about jvm startup parameters, please read jvm-options-list

[Gavin Wan]

awesome！

[Michael Young]

Great post, Shuyang!

[ravi kumar gupta]

nicely written.. following u frm now..

[Fernando Nubla Durango]

awesome！ thanks!

[Riccardo Rotondo]

Hi, nice guide. Thank you. I have a question. I have read many guides to optimize tomcat. Usually they suggest to add option -server in catalina.sh but you didn't metion. Why?
Thanks for help
Bye

[Shuyang Zhou]

This supposed to be general, not tie to any app-server. But since you are asking about tomcat, here are some tomcat only tricks you can do to improve GC.

1)In conf/web.xml, set Jasper property genStrAsCharArray = false. This may seem a little odd, most people will tell you set it to true. But if you take a look at our StringBundler and JspFactorySwapper, you will see why.

2)In conf/server.xml, add socketBuffer="-1" to your connector setting. This will trun off that connector's out going socket buffer. The upper logic has done all buffer, no need to buffer again.

3)If you are using a newer version of tomcat which support Servlet Sepc2.5, in webapps/ROOT/WEB-INF/web.xml change the root element define to <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd" version="2.5" metadata-complete="true">. This tells tomcat to skip JavaEE standard annotations dependency lookup, since liferay is not using them.

With these 3 settings, you can see a significant GC difference. Try it yourself

[Neon Liu]

Great post on jvm tuning

[Riccardo Rotondo]

Thank you, I set parameter as you suggested. Despite the fact my server has only 4 GB RAM now it works better. Thank you very much for your help!

[Riccardo Rotondo]

Hi all. Can this option be used for Liferay 6.0.5 ?? I have just tried but I get this error:

INFO: Server startup in 22550 ms
11:35:45,699 ERROR [IncludeTag:231] Current URL / generates exception: java.lang.OutOfMemoryError: PermGen space
11:35:45,703 ERROR [IncludeTag:154] java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass(ClassLoader.java:632)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:134)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:66)
at org.apache.jasper.JspCompilationContext.load(JspCompilationContext.java:628)
at org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:144)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:329)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at com.liferay.taglib.util.IncludeTag.include(IncludeTag.java:175)

11:35:49,196 ERROR [IncludeTag:231] Current URL / generates exception: java.lang.OutOfMemoryError: PermGen space
11:35:49,197 ERROR [IncludeTag:154] java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass(ClassLoader.java:632)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:134)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:66)
at org.apache.jasper.JspCompilationContext.load(JspCompilationContext.java:628)
at org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:144)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:329)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at com.liferay.taglib.util.IncludeTag.include(IncludeTag.java:175)

11:35:53,370 ERROR [IncludeTag:231] Current URL / generates exception: java.lang.OutOfMemoryError: PermGen space
11:35:53,372 ERROR [IncludeTag:154] java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass(ClassLoader.java:632)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:134)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:66)
at org.apache.jasper.JspCompilationContext.load(JspCompilationContext.java:628)
at org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:144)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:329)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at com.liferay.taglib.util.IncludeTag.include(IncludeTag.java:175)

11:35:57,071 ERROR [IncludeTag:231] Current URL / generates exception: java.lang.OutOfMemoryError: PermGen space
11:35:57,072 ERROR [IncludeTag:154] java.lang.OutOfMemoryError: PermGen space

11:35:59,656 ERROR [IncludeTag:231] Current URL / generates exception: java.lang.OutOfMemoryError: PermGen space
11:35:59,657 ERROR [IncludeTag:154] java.lang.OutOfMemoryError: PermGen space

[Shuyang Zhou]

You should increase your perm space size to 256mb, liferay 60x has larger code base.
-XX:MaxPermSize=256m

[Lulu Liu]

Nice! Thanks!

[DO Hung Thuan]

Hi
I have poblem, i run liferay 6.0.6, when i run about 3 days, PS old gen memory to full 100%, i run GC anytime but not reduced
I run: -XX:UseParallelOldGC.

[Cosco SCJ]

Good article.
I need to increase my PermGen parameter to 256m.
Now, sometimes i have this error: "Attempt to allocate stack guard pages failed newsize"

Could you please help me? thanks

[Muhammad Umer Zubair]

Thanks for the very informative article. I have tried the settings with -Xmx 2048m, but apache does not startup. I have 8 gb ram and corei7 machine. Server starts up with 1048 memory settings.

[(你)]

[...] 5 tips for proper Java Heap size. Young Generation. Liferay JVM Tuning. How to Monitor Java Garbage Collection. Liferay Tips: Liferay Performance Tuning. Tuning Tomcat Performance For Optimum Speed... [...] Read More

[Vishal Panchal]

Excellent explanation, thanks for sharing
- Vishal