Memory leak problem

[6年前 に Micael Ericsson によって更新されました。]

Our current site has increased its popularity. We have now 6.000 visitors each day compares to the same amount over 6 month previously.

Since we knew this was going to happen we made all performance testing we could before the release with good success.
In production however we have detected a memory leak. After running 2 full days the heap is to big and GC release is not enough. This makes the GC process take all CPU leaving the users with long long wait time. User count when this happens is 400-500 current users according to Google analytics.
Restarting Liferay (6.2 on Tomcat 7) every morning is a workaround that helps us through the day, but not a permanent solution.

On the last problem day we managed to make a memory snapshot (heap dump). It is 7GB large.
Opening the dump in Eclipse memory analyser and launching dominator_tree gave us the included image.
I am no expert in memory dump analysing but as far as I can see:
- faces.usitl.FactoryExtensionFinder
- faces.bridge.internal.BridgeConfigFactoryImpl
are both involved and causing retained heap a value of 3GB.
In the dominator tree after BridgeConfigFactoryImpl there are 3.690.623 entries of ConcurrentHashMap. This cant be good .

Our site is not that big: 10 portlets and a couple more pages on one main site and two subsites. Portlets are built with JSF 2.2 and Primefaces 5.

What is a good next step in order to find the root for this problem?

[6年前 に Juan Gonzalez によって更新されました。]

Hi Micael.

Seems your issue should be fixed by a recent fix made under https://issues.liferay.com/browse/FACES-2311.

Can you post here all your Liferay Faces library versions from your pom.xml? Then I will provide you new versions where you can check that supposed fix.

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

I think Juan is correct. Specifically, it might be commit 9649fd3f (part of the fix for FACES-2311) that might help the memory leak problem. When you get an opportunity, please provide the pom.xml dependency info that Juan asked for and we will try to get you some SNAPSHOT jars to try in your environment.

Neil

[6年前 に Micael Ericsson によって更新されました。]

Supplied is parent pom with all version information. Let me know if something is missing.
Regarding com.liferay.faces.bridge.impl with version 4.0.1-SNAPSHOT. This is not the latest SNAPSHOT. We had a problem with a release of the SNAPSHOT in october/november a couple of days before our release so we went for the latest working version.

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

Do you know what causes the memory leak to increase? For example, does the leak get worse for each login? Or does it get worse with every request?

Also, are you using Liferay Portal 6.2.5 CE or 6.2 EE?

In any case, please try the following dependencies in your TEST environment:

<repositories>
    <repository>
        <id>liferay-faces-snapshots</id>
        <name>Liferay Faces Snapshots</name>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupid>com.liferay.faces</groupid>
        <artifactid>com.liferay.faces.bridge.ext</artifactid>
        <version>3.0.1-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupid>com.liferay.faces</groupid>
        <artifactid>com.liferay.faces.bridge.impl</artifactid>
        <version>4.1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupid>com.liferay.faces</groupid>
        <artifactid>com.liferay.faces.util</artifactid>
        <version>3.1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupid>com.liferay.faces</groupid>
        <artifactid>com.liferay.faces.portal</artifactid>
        <version>2.0.1-SNAPSHOT</version>
    </dependency>
</dependencies>

Thank you,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

We are running liferay-portal-6.2-ce-ga6.
I can't say if it's the number of people logged in or the number of requests that's causing this problem. I know from the time when have this problem with long wait times the system recover it self when the number of current users drops from 300+ to below100.

The user scenario is pretty straight forward:
1) User logs on
2) User starts our survey made up of 60 questions
3) User logs out
Sometimes the user looks at other stuff.

We have between 2000 and 5000 surveys answered each day.
And as I said erlier one day works fine but two days and the heap cannot recover itself.

I included 2 images from jconsole showing how the heap memory usage has evolved over time.
lr_nem_1: short time period and in this GC worked. Around 17:32 GC could clean up and the site worked fine.
lr_nem_2: The red 'X' is where we restarted the site. In the top two red 'X' (above 6GB.) we had to restart due to long wait time for the user. The third was a restart to prevent problems to occur.

Thanks for the included pom.xml with version info. Have you a suggestion how to monitor if this is working.
At current we only have this problem in production. In our testbench we are using LoadComplete to simulate 250 virtual users at a time but not the continous load that exists in production. And for the moment our testbench is occupied for the next release.
Is it possible to monitor a single class in my local dev evironment to see if this is working?

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

I recommend trying jvisualvm (ships with JDK8) in order to monitor the heap. It will also let you take a heap dump snapshots in order to monitor all object references and potential leaks.

The SNAPSHOT jars that I mentioned in the previous post contain a significant refactor, storing factories and other objects in the PortletContext (ServletContext under the hood) rather than ClassLoader singletons. I can't say for sure whether or not it fixes the memory leak you are observing, but if you continue to observe leaks with the SNAPSHOT jars, we can most easily fix it in the latest codebase.

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

I rebuilt all portlets with supplied SNAPSHOT versions and deployed.

When running sites first page with a portlet I get an error.

java.lang.ClassCastException: org.apache.catalina.core.StandardContext$NoPluggabilityServletContext cannot be cast to javax.portlet.PortletContext
	at com.liferay.faces.bridge.internal.BridgeFactoryFinderImpl.getFactoryInstance(BridgeFactoryFinderImpl.java:44)
	at com.liferay.faces.bridge.BridgeFactoryFinder.getFactory(BridgeFactoryFinder.java:30)
	at com.liferay.faces.bridge.filter.BridgePortletConfigFactory.getPortletConfigInstance(BridgePortletConfigFactory.java:37)
	at com.liferay.faces.bridge.internal.BridgeImpl.doFacesRequest(BridgeImpl.java:157)
	at javax.portlet.faces.GenericFacesPortlet.doView(GenericFacesPortlet.java:673)
	at javax.portlet.GenericPortlet.doDispatch(GenericPortlet.java:359)
	at javax.portlet.faces.GenericFacesPortlet.doDispatch(GenericFacesPortlet.java:589)
	at javax.portlet.GenericPortlet.render(GenericPortlet.java:262)
	at com.liferay.portlet.FilterChainImpl.doFilter(FilterChainImpl.java:103)
	at com.liferay.portlet.ScriptDataPortletFilter.doFilter(ScriptDataPortletFilter.java:55)
	at com.liferay.portlet.FilterChainImpl.doFilter(FilterChainImpl.java:100)
	at com.liferay.portal.kernel.portlet.PortletFilterUtil.doFilter(PortletFilterUtil.java:64)
	at com.liferay.portal.kernel.servlet.PortletServlet.service(PortletServlet.java:112)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

I included full log from startup.

What can be the problem ?

Regards,
Micael

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

Did you delete tomcat/webapps/VisaPdf before attempting to redeploy? It might be that you have old-versions of jars alongside new versions in tomcat/webapps/VisaPdf/WEB-INF/lib.

Kind Regards,
Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

I tried it all again:
- stopped Liferay
- clean temp, work and webapps from all my portlets
- rebuilt portlets with snapshot-versions
- copied only 3 war-files to deploy (excluding among others visa-pdf)
- started Liferay
Lunached the site and ...error again. Same error as last time.

Next step was to see if I could build and deploy my TestPortlet (with no connections to backend etc) with new SNAPSHOT-version.
Now I get and build error:

constituent[28]: file:/C:/Program%20Files/NetBeans%208.2/java/maven/lib/wagon-http-shared-2.10.jar
constituent[29]: file:/C:/Program%20Files/NetBeans%208.2/java/maven/lib/wagon-provider-api-2.10.jar
---------------------------------------------------
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
	at org.apache.maven.wagon.providers.file.FileWagon.resolveDestinationPath(FileWagon.java:206)
	at org.apache.maven.wagon.providers.file.FileWagon.resourceExists(FileWagon.java:265)
	at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:577)
	at org.sonatype.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:60)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils
	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230)
	... 7 more

For including StringUtils in maven I added the following dependency:

 <dependency>
	  <groupid>org.apache.commons</groupid>
	  <artifactid>commons-lang3</artifactid>
	  <version>3.5</version>
	 </dependency>

Now the TestPortlet could build, but upon deploy I get the same error as for my production portlets.

I included the TestPortlet. It should be possible to build outside my environment.
I also included the server log.

Regards,
Micael

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

After I build the test portlet that you attached, I see the following incorrect jars embedded within the war:
com.liferay.faces.alloy-2.0.0.jar
com.liferay.faces.bridge.api-4.0.0.jar

Instead, you should have:
com.liferay.faces.alloy-2.0.1-SNAPSHOT.jar
com.liferay.faces.bridge.api-4.1.0-SNAPSHOT.jar

Normally com.liferay.faces.bridge.api is not specified in pom.xml -- instead, Maven resolves it as a transitive dependency of com.liferay.faces.bridge.impl so that the correct version is always used.

Attached to this post is a "snapshot" version of our PrimeFaces Applicant Portlet that should work in your environment. (Note that I removed the "-SNAPSHOT" file the filename).

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

Change to SNAPSHOT version of api and alloy did the trick. (it wasn't in your suggested pom.xml in previous message.
Now I can compile and test.

Thanks for the help and I continue memory leak test.

Regards,
Micael

[6年前 に Micael Ericsson によって更新されました。]

Closing in on test and releasecycle it's time to decide on version of liferay faces libs.
Currently I have the following libs for Liferay 6.2:

• faces alloy 2.0.1-SNAPSHOT
• faces bridge api 4.1.0-SNAPSHOT
• faces bridge ext 3.0.1-SNAPSHOT
• faces bridge imple 4.1.0-SNAPSHOT
• faces portal 2.0.1-SNAPSHOT
• faces util 3.1.0-SNAPSHOT

Since I am not that keen on going into production with SNAPSHOT-versions when are release-versions coming?

Regarding the problem I had with pre-snapshot version:
- memoryleak problem. I have not been able to produce the amount of erquest over time needed for testing this
- code completion problem i NetBeans. Solved with snapshot-version of alloy if I remeber correct
No new problems have come to my knowledge during development testing.

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

I'm glad to learn that you have not observed any new problems. The Liferay Faces team is currently in a bugfix sprint and we plan to release production quality (non-SNAPSHOT) versions of the dependencies you listed later this month.

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

That is great news. Release at the end of this months works perfect with our release plan.
Thanks for all help.

Regards,
Micael

[6年前 に Micael Ericsson によって更新されました。]

Neil Griffin:

Hi Micael,

I'm glad to learn that you have not observed any new problems. The Liferay Faces team is currently in a bugfix sprint and we plan to release production quality (non-SNAPSHOT) versions of the dependencies you listed later this month.

Kind Regards,

Neil

Later this month What is the release status on these libs?

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

We only have three tickets left that we are fixing before we start our comprehensive testing: FACES-3173, FACES-2970, and FACES-2602. I'm going to try and reproduce your portal:captcha validatorMessage issue today and if that's a bug, then get that fixed too.

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

What's the release status for SNAPSHOT I've testing. We are closing in on release.
Is captcha bug included in these releases?

Regards,
Micael

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

Sorry for the delay -- we have been slowed down in part due to preparing for Hurricane Irma. If we don't encounter any last minute problems in our tests, then it will be our goal to start releasing artifacts to Maven Central starting today, hopefully completing the release by Saturday.

Kind Regards,

Neil

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

Forgot to mention that the portal:captcha bug is fixed in this upcoming release.

Kind Regards,

Neil

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

We just finished releasing the artifacts. They should all be in Maven Central soon. You can expect a formal release announcement sometime next week.

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Neil Griffin:

Hi Micael,

We just finished releasing the artifacts. They should all be in Maven Central soon. You can expect a formal release announcement sometime next week.

Kind Regards,

Neil

Great work. Is it the following artifact and versions that I should expect appear in maven repository?

• faces alloy 2.0.1
• faces bridge api 4.1.0
• faces bridge ext 3.0.1
• faces bridge imple 4.1.0
• faces portal 2.0.1
• faces util 3.1.0

Looking today I couldn't find them. Do you have an exact date? I have this week and 3-4 days next week before code freeze.

Is this place the best place to find info on released artifacts?
https://mvnrepository.com/artifact/com.liferay.faces

[6年前 に Juan Gonzalez によって更新されました。]

It's showing fine in maven central:

https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.liferay.faces%22

[6年前 に Micael Ericsson によって更新されました。]

Juan Gonzalez:

It's showing fine in maven central:

https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.liferay.faces%22

Thanks Juan.

Rebuilt my projects with new releases. No problem and working fine in my development environment.
But when I release to central development and test environment I got a runtime error:

Uncompilable source code - package com.liferay.faces.portal.context does not exist

In my code I am using

com.liferay.faces.portal.context.LiferayFacesContext

.
No problem compiling, deploying. No problem running on local Liferay 6.2 GA6. But why a problem in the other environments. They "should" all have same configuration.

[6年前 に Juan Gonzalez によって更新されました。]

Just be sure to undeploy all of your previous apps before redeploying them. If not they would have multiple Liferay Faces jar files.

[6年前 に Micael Ericsson によって更新されました。]

I thought so too at first. Normally I use a script to remove old and deploy new (with restart inbetween). I did this manually to really verifiy that no old was present. Problem remains. tmp and work dirs are also cleared.

I didn't remove my hooks since they have no dependency to liferay faces libs.

I included liferay logfil and images of:
- liferay webapps dir after I removed my portlets
- login portlet war libs - portlet where runtime exception occurs
- hook war libs - the one hook with 3rd party libs

Please let me know if you can spot something wrong.

[6年前 に Juan Gonzalez によって更新されました。]

Hmm sounds weird. Are there any JDK difference between both environments?

[6年前 に Micael Ericsson によって更新されました。]

Juan Gonzalez:

Hmm sounds weird. Are there any JDK difference between both environments?

All are running on java 1.8.0_91.

[6年前 に Micael Ericsson によって更新されました。]

I rebuild my login-portlet with SNAPSHOT versions. Deployed in test with same procedure as with RELEASE version. It worked.
Something is different in these two versions.

[6年前 に Micael Ericsson によって更新されました。]

Including two images:
- login portalets WEB-INF/lib dir SNAPSHOT vs RELEASE
- faces portal lib SNAPSHOT vs RELEASE

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

Regarding the "Uncompilable source code" error message, my search of the web indicates that it is typically associated with projects that were built with NetBeans. Do you happen to use NetBeans in order to perform builds of your WAR project(s)?

The only difference that I can think of between liferay-faces-portal-3.0.1.jar and liferay-faces-portal-3.0.1-SNAPSHOT.jar is that the former was built with a Java 7 JDK (since the source level is 1.7) and the later is built with a Java 8 JDK due to a limitation of the CI server.

Since a Java 8 JVM is backward compatible with .class files that are compiled with older JDKs, it should be working in your environment.

The fact that it works in your development environment but not in your test environment leads be to believe that there is a problem in your test environment. Here are some things you can check:
1. Make sure that tomcat/lib and tomcat/lib/ext (global classpath dirs) do not contain a copy of liferay-faces-portal.jar
2. Make sure that the same exact .war artifacts that were built and deployed to in your development environment are deployed in your test environment. In other words, you want to use the same binary artifacts in both dev and test.

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

I am using NetBeans to build my portlets. Since the project is maven based I'll tomorrow try to build my portlets outside NetBeans with maven to see if that can change the picture.

I have checked and double checked that there is no liferay-faces-portal.jar in tomcats lib and lib/ext and also in javas lib and lib/ext.

But still, last try of the day was to first build my login portlet with RELEASE and then with SNAPSHOT versions of liferay faces libs.
Then one at a time time deploy my two login portlets in test env. The SNAPSHOT version of login worked like a charm, but the RELEASE version ended up with the error message, not working. Regardsless of both are working in my environment it's obvious that there are a difference between the SNAPHOT and the RELEASE version of liferay faces since one is working and the other is not working in test environment.

Is it possible to print the classpath my login portlet is using at runtime? then I could compare both scenarios and see what differs.
Another thought I had now (remember it's late here in sweden could it be that the liferay faces portal lib is corrupt and thats the reason why the package is not found?

My BaseController class which all my portlets Controller classes inherits from has:

import com.liferay.faces.portal.context.LiferayFacesContext;

Since liferay faces portal jar is listed in war-file under WEB-INF/lib it should be available, unless it has a dependency which not is present in WEB-INF/lib. Do you know of any such dependency which differs between SNAPSHOT and RELEASE version?

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

We can explore the possibility of the jar being corrupt, but if it is, then it should fail in both your development and test environments.

We're here to help -- please let us know how it goes when you build your portlets with Maven from the command line, rather than in NetBeans. For more information about this problem in relation to NetBeans, see NetBeans Issue ID#182009.

Kind Regards,

Neil

[6年前 に Micael Ericsson によって更新されました。]

Hi Neil

Building my portlets with maven outside NetBeans solved the problem. I have now working portlets with latest release of liferay faces.

According to the bug report on NetBeans (from 2010!) it could be a cache problem. I'll try to clear the NetBeans cache and rebuild my portlets next week to see if that solves the problem.

Thanks for the help,
Micael

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

Thanks for letting us know that it is working in your test environment, and also thanks for your patience in waiting for the release.

Kind Regards,

Neil

[6年前 に Neil Griffin によって更新されました。]

Hi Micael,

I also wanted to mention that LiferayFacesContext is deprecated -- for more information, see: FACES-2502.

Kind Regards,

Neil