留言板

LR 6.1 not responding with no aparent errors

Marcos Laurito,修改在9 年前。

LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
I'm using Liferay Portal Community Edition 6.1.1 CE GA2 that comes bundled with Tomcat 7.0.27. I'ts running on Ubuntu Server 12.04 64 bits (VM on VMware ESXi).
It's running perfectly for some days and then suddenly it's not responding. I've checked the logs and no aparent error are present. I've monitored the tomcat with JConsole even when liferay its not responding, and the tomcat seems to be running perfectly (more than half of the total memory of Tomcat is free and no error aparent in Jconsole).
I realize when this happens by my liferay not loading (i try to go open liferay with the browser and it keeps loading an no errors showing, eventually i get to the login page but it seems to have lost all css style and it never ends loading, it only keeps loading).

How can i troubleshoot this? I don't think that is a memory problem because checking on LR Control Panel ---> Server Administration --> Resources , the visual memory indicator never get past a quarter of total memory.

Anybody can help me?
thumbnail
David H Nebinger,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
I've seen lockups actually tied to how database backups are handled. If the database is shut down for backup, Liferay may not be aware of it and the db connections are stale. Then it looks locked cuz it's trying to use the bad connections to query the db.

Sometimes you can resolve this by defining the test query that the cxn pool will use to validate the connection before handing it out for use, and the pool can recover from the database being shut down independently from Liferay.
Marcos Laurito,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
David H Nebinger:
I've seen lockups actually tied to how database backups are handled. If the database is shut down for backup, Liferay may not be aware of it and the db connections are stale. Then it looks locked cuz it's trying to use the bad connections to query the db.

Sometimes you can resolve this by defining the test query that the cxn pool will use to validate the connection before handing it out for use, and the pool can recover from the database being shut down independently from Liferay.


Hi, thanks for the response. I've actually tried to check if the database was running properly, even tried to connect to it and it works ok. In fact, in the same mysql server that hosts the liferay db i have others databases that works with other systems, and they run all just fine while Liferay it's not responding...
thumbnail
Juan Gonzalez,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 3089 加入日期: 08-10-28 最近的帖子
Marcos Laurito:

I realize when this happens by my liferay not loading (i try to go open liferay with the browser and it keeps loading an no errors showing, eventually i get to the login page but it seems to have lost all css style and it never ends loading, it only keeps loading).


Hi Marcos,

when loading page, open a developer console (F12 Chrome) or Firebug, and check in Network section which network resources are getting stuck. Get their URL and post here.
Marcos Laurito,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
Juan Gonzalez:
Marcos Laurito:

I realize when this happens by my liferay not loading (i try to go open liferay with the browser and it keeps loading an no errors showing, eventually i get to the login page but it seems to have lost all css style and it never ends loading, it only keeps loading).


Hi Marcos,

when loading page, open a developer console (F12 Chrome) or Firebug, and check in Network section which network resources are getting stuck. Get their URL and post here.


Ok i will do it and post it here. Just have to wait that the problem happen again. I'ts random, and i can't reproduce it myself. It could happen twice in a day or twice in a week.
By the way, thanks and i will be back with that URL...
Marcos Laurito,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
Marcos Laurito:
Juan Gonzalez:
Marcos Laurito:

I realize when this happens by my liferay not loading (i try to go open liferay with the browser and it keeps loading an no errors showing, eventually i get to the login page but it seems to have lost all css style and it never ends loading, it only keeps loading).


Hi Marcos,

when loading page, open a developer console (F12 Chrome) or Firebug, and check in Network section which network resources are getting stuck. Get their URL and post here.


Ok i will do it and post it here. Just have to wait that the problem happen again. I'ts random, and i can't reproduce it myself. It could happen twice in a day or twice in a week.
By the way, thanks and i will be back with that URL...


Ok man. Its happening again right now. I've checked Network tab on Firebug the resources that keeps loading until timeout and these are the following:


http://myLRip/so-welcome-theme/css/main.css?browserId=firefox&themeId=sowelcome_WAR_sowelcometheme&minifierType=css&languageId=es_ES&b=6101&t=1375971764000
http://myLRip/so-welcome-theme/images/spacer.png
http://myLRip/so-welcome-theme/images/common/openid.gif
http://myLRip/so-welcome-theme/js/main.js?browserId=firefox&minifierType=js&languageId=es_ES&b=6101&t=1375971764000


Being myLRip my intranet liferay ip...

What could you tell me about this resources?
thumbnail
David H Nebinger,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
two are static (odd that they're blocked), the css and js are pulling in the theme's js and css.
Marcos Laurito,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
Is there any other test that i can try while my liferay is locked up?
This keep happening to me and i need to solve it cause its a production environment...
thumbnail
Juan Gonzalez,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 3089 加入日期: 08-10-28 最近的帖子
Mmm seems weird.

Are you using any load balancer/proxy? (apache, etc)

Can you check if the loaded resources URL are different from these ones?

Maybe you can try to stop server, remove work and temp folders from tomcat, and startup again...
Marcos Laurito,修改在9 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
Juan Gonzalez:
Mmm seems weird.

Are you using any load balancer/proxy? (apache, etc)

Can you check if the loaded resources URL are different from these ones?

Maybe you can try to stop server, remove work and temp folders from tomcat, and startup again...



Yes, i'm using Apache2 webserver in front of Tomcat/Liferay to redirection port 80 to 8080. But i understand that if the problem were the apache, while my liferay hangs i could access directly to my tomcat (8080 port) and it has to work. This is not happenig because during the hangs if i try to acces bypassing the apache, it doesn't work either.

What loaded resources do you mean?

Stop the server is the only thing that works for me to have my LR working for me. I could try deleting those folder next time the lockup happens...
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
I'ts been two years and i'm still facing this problem.
I've configured an auto reboot to the liferay server that runs on a daily base (every night my liferay server, that is where the liferay db is as well, reboots and start automatically).
I'm still having the lockups but they appear mostly after several hours of work. The server start to receive the work load around the 7 a.m. and the lockups appears around 7 or 8 p.m.
Anyone can help me?
Thanks and sorry for my english.
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
Ok, made somre progress on this.
My problem is caused by OLD Gen Heap Space filling up. It starts to grow and GC runs ok but its not releasing enough memory, so OLD GEN keeps growing until its full, and then Tomcat stops responding requests.
I've made several memory dumps and tried to analyse it with Eclipse MAT. What I've found is this:

One instance of "com.liferay.portal.kernel.concurrent.ThreadPoolExecutor" loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x740079b30" occupies 1,295,340,320 (66.77%) bytes.


Anyone could tell me if this is a memory leak or is normal that "ThreadPoolExecutor" uses that much memory and won't release it?

Thanks
thumbnail
David H Nebinger,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
Do you restart tomcat after a deployment? Tomcat has a nasty issue of not being able to clean up class loader references when classes/jars are unloaded, typically a result of redeployment activities.

It is always a great idea that, once your artifacts have deployed correctly on the node, that you restart the node to ensure the class loaders are clean and cruft-free.







Come meet me at the LSNA!
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
David H Nebinger:
Do you restart tomcat after a deployment? Tomcat has a nasty issue of not being able to clean up class loader references when classes/jars are unloaded, typically a result of redeployment activities.

It is always a great idea that, once your artifacts have deployed correctly on the node, that you restart the node to ensure the class loaders are clean and cruft-free.







Come meet me at the LSNA!

David, thanks for your response.

What do you mean by "Do you restart tomcat after a deployment? "??
My server is restarted automatically once per day, at early morning, before users starts to login in. This auto restart was setup up trying that the server can handle 1 day long run without filling up the memory and without having to restart it.

What are "redeployment activities"?

Thanks and sorry for my ignorance.
thumbnail
David H Nebinger,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
No problem. Normally w/ tomcat to deploy your plugins you drop them into the $LIFERAY_HOME/deploy folder. Liferay will pick them up and process them into tomcat, removing old versions and deploying the upgraded version, etc.

After the deployment completes, I recommend restarting (your production and test environs) because tomcat cannot clean up all class loader stuff.

Restarting your instance every day is a little extreme and likely symptomatic of a larger problem, one typically originating with custom code. Liferay itself, running under tomcat, without customizations and regardless of configuration, will honestly run forever. If you need to restart daily because of memory leaks, it's not going to be from core code.






Come meet me at the LSNA!
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子

No problem. Normally w/ tomcat to deploy your plugins you drop them into the $LIFERAY_HOME/deploy folder. Liferay will pick them up and process them into tomcat, removing old versions and deploying the upgraded version, etc.

After the deployment completes, I recommend restarting (your production and test environs) because tomcat cannot clean up all class loader stuff.

I understand but i don't do deploys. Did them at the early days after liferay installation to install some portlets, but not anymore.



Restarting your instance every day is a little extreme and likely symptomatic of a larger problem, one typically originating with custom code. Liferay itself, running under tomcat, without customizations and regardless of configuration, will honestly run forever. If you need to restart daily because of memory leaks, it's not going to be from core code.

I know its extreme but its the only way i can think of for avoiding old gen memory filling up, at least while i search for the root cause of the problem. I have to determinate yet if the fact that old gen heap space is filing up, is due to a memory leak or it's normal because of the traffic and load, caused by the normal use of liferay with many users (around 200-300 users).



Is there a way i can figure if this ThreadPoolExecutor is causing a memory leak or is working ok??

This is the old gen consumption history for today. I can see, because of the way it's growing up, that in no more than 2 or 3 hours from now, it's going to be filled up and I will have to restart tomcat.

OLD Gen Consumption History
thumbnail
David H Nebinger,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
Marcos Laurito:
I know its extreme but its the only way i can think of for avoiding old gen memory filling up, at least while i search for the root cause of the problem. I have to determinate yet if the fact that old gen heap space is filing up, is due to a memory leak or it's normal because of the traffic and load, caused by the normal use of liferay with many users (around 200-300 users).


300 users? That's nothing. Liferay is running Liferay and there are thousands of users online at the same time. I've worked on sites where they have millions of users on Liferay.

You're going to have to start collecting diagnostics. Do some heap dumps and dig through them to find out where the memory is going, find out what is allocating it and why it is not being released.

Is there a way i can figure if this ThreadPoolExecutor is causing a memory leak or is working ok??


TPE is not going to leak memory on it's own, but if it's running some custom code the leaks will come from there and appear to be in TPE.








Come meet me at the LSNA!
thumbnail
David H Nebinger,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
Yes, the chart shows the symptom of what is going on, but it's not going to expose the cause. You need to see what it is that is in memory and not being released, that's how you'll solve this problem.







Come meet me at the LSNA!
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
David H Nebinger:
Yes, the chart shows the symptom of what is going on, but it's not going to expose the cause. You need to see what it is that is in memory and not being released, that's how you'll solve this problem.







Come meet me at the LSNA!

Thank.
I know by analysing heap dumps, that TPE is allocating and retaining that memory. Now i have to figure out what is inside TPE that is not releasing that memory.
Will continue investigating.
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
Ok, got some more clues.
It appears that my problem has to do with the Private Messaging Portlet of Social Office.

That portlet is used here at my company as if it were an internal corporate mail. Users are sendind messages to multiple other users, sometimes this multiple messages includes attached files.

Can this be causing the memory not releasing??

I'm using Eclipse MAT to analyse a heap dump and apparently the TaskQueue is filled up with private messaging tasks...

Any help on this?
thumbnail
David H Nebinger,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Liferay Legend 帖子: 14914 加入日期: 06-9-2 最近的帖子
Moved it to the SO thread.

Please provide version details for Liferay and SO.







Come meet me at the LSNA!
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
David H Nebinger:
Moved it to the SO thread.

Please provide version details for Liferay and SO.







Come meet me at the LSNA!

I'm using Liferay Portal Community Edition 6.1.1 CE GA2 that comes bundled with Tomcat 7.0.27. I'ts running on Ubuntu Server 12.04 64 bits (VM on VMware ESXi). Using Liferay Social Office CE 2.0.0.
Marcos Laurito,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Junior Member 帖子: 99 加入日期: 13-4-18 最近的帖子
David H Nebinger:
Moved it to the SO thread.

Please provide version details for Liferay and SO.







Come meet me at the LSNA!

Need some more help with this problem.
Anyone knows how PrivateMessaging portlet works?
Incorrect use of it could cause this memory leak i'm having?

Thanks in advance.
thumbnail
Vishal Panchal,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Expert 帖子: 289 加入日期: 12-5-20 最近的帖子
Dear Marcos,

I am currently running with the same issue. Did you find any solution for this?

I am using Jboss EAP 7.2 with Liferay 6.2 GA6 on HP unix 64-bit machine. Server is running fine for around 50 hours and suddenly becomes unresponsive. I did checked jboss server and Liferay logs but nothing I found form it.

After lots of goggling I've taken gc.log and during GC log analysis I came to conclusion that my Old Generation is full. During server up time GC is not freeing up much amount of memory.

Once the Old Gen mem is full server starts Full GC repetitively. That results in long running GC threads and server becomes completely unresponsive.
Further I checked database was running properly. Currently I am still analyzing heap dumps and will share the results soon.

From below link I have applied -XX:+UseCompressedOops, but that sis not solved, in fact no behavior changed.
http://stackoverflow.com/questions/7513185/what-are-reservedcodecachesize-and-initialcodecachesize

Below are the current JVM memory tuning I did.

JAVA_OPTS="-server -XX:-RewriteBytecodes -d64 -Xss2m -Xms4096m -Xmx8192m -XXemoticonermSize=500m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=512m -Dfile.encoding=UTF-8 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.server.default.config=standalone.xml -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:NewSize=700m -XX:MaxNewSize=700m -XX:NewRatio=4 -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XXemoticonarallelGCThreads=20 -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseCompressedOops -XX:+DisableExplicitGC"

Anyone can point me any further changes or what the exact issue is?

Thanks in advance,
Vishal
thumbnail
Vishal Panchal,修改在7 年前。

RE: LR 6.1 not responding with no aparent errors

Expert 帖子: 289 加入日期: 12-5-20 最近的帖子
Further Updates.

I have also started separate thread for the same at below link.
https://web.liferay.com/community/forums/-/message_boards/message/84471496