Liferay 6.1.1-CE-GA2 OutofMemory heap space hibernate cause

[Tiuser Lassei, modifié il y a 7 années.]

Greetings,

Our portal is encountering an OutofMemoryError recently and a heap dump was generated.
server details:
Java 1.7
IBM AIX server
2GB heap size
Liferay 6.1.1-CE-GA2

The server is in production (for several years) and we only encountered this recently (we are not sure why this is happening all of the sudden)

The heap dump seems to be pointing the leak suspects to be hibernate /ehcache related objects.
31.45% about half is from HibernateServiceStatistics and another is from ehcache
17.21% hibernate sessionfactoryImpl
14.06% ehcache again
8.94% ehcache again

Can anyone advise if HibernateServiceStatistics is on by default in Liferay (or is it turned on when jmx setting is enabled for tomcat)? (not really familiar with hibernate and did not find any resources about this in liferay) and may be causing the memory leak?

I attached the pictures of the 4 most largest leak suspects below (heap dump analyzed by IBM heap analyzer)








Note that we have custom portlets but they are not using liferay DB except for user and preferences (and afaik there is no recent change for them which may have been causing above recent outofmemoryerror)

Thanks

[David H Nebinger, modifié il y a 7 années.]

It is always possible that after years of use your data volume has finally pushed past what could be supported in a 2G partition. Do you have any details on your DB size growth?

It's certainly not normal for ehcache to cause such problems, but it can be easy to miss a gradual growth of data/volume and miss the approach to the memory ceiling. In some cases this can be exacerbated by an uptick in GC calls in order to make memory available because the system was running out of memory.





Come meet me at the LSNA!

[Olaf Kock, modifié il y a 7 années.]

What David says.

And:

Note that there's no AlmostOutOfMemoryError - so you might have been running very close to the limits all the time (unless you monitored carefully) and OutOfMemoryError always occurs suddenly (in the event of memory depletion). The data you post might or might not be totally normal for the system - that's impossible to say from a snapshot. ehcache might just do what it's supposed to do, but due to increased utilization, other components might now have grown beyond their previous uses.

[Tiuser Lassei, modifié il y a 7 années.]

Thanks for your replies.

It seems above heap dump seems to be incomplete. Apparently our server heap size setting (-Xmx) is -Xmx16G. I'm not sure why the heap dump only contains 2G (note that there are several heap dumps created so maybe they were separated).

Fortunately a complete heap dump was generated in one instance and we found the actual leak which is not related to liferay (and above classes also have an entry in this dump but have the same size so they might be expected).

Thanks