80% CPU usage

[Pawel Osak, modificado hace 15 años.]

Hi,
I have a problem with my Liferay instance. I use 4.3.3 running on top of Tomcat 6.0.13 and liferay is the only application that runs on this webserver.
Generally it works OK - the site is used at most by about 50 users, so I don't have performance problems - the CPU usage is at a few percent level. But sometimes CPU usage goes on 80% level and it stays there until I restart tomcat. It seems that it doesn't have a bad impact on overall performance - average user can't see the difference. The only problem is, when another demanding process is started - in my case once a day a stored procedure in MySQL is executed and then performance is worse. However, the most important for me is to collect statistics and they are useless if for most of the time the CPU usage is at 80% level, even if nobody uses the portal.
My question is: does anybody know how to track the portlet causing problems with CPU usage? Maybe some JVM monitor tool? I customized Liferay a bit, developed my own portlets and now it's hard when you don't know where to look...

Paweł

[Artur Linhart, modificado hace 15 años.]

Hello,

Imcould imagine this could happen if the garbage collector is not properly configured or if the system is out of memory so the garbage collector is executed in very short times... I would try to change the JVM, change the memory amount alocated to JVM or change the garbage collector settings... Or - change the operating system, if it is Windows ;-)

with regards, Archie

[Pawel Osak, modificado hace 15 años.]

Thanks for input. (Un)fortunately this is Linux Debian so replacing Windows is not an option here
I'll try to watch the garbage collector, however as far as I understand its activity, I would rather observe frequent peaks on CPU usage, if it was the reason for my problems. In my case when CPU usage goes high, it remains at 80% level all the time, without any interruption.
Maybe this is not the best place to ask the question, but is there a way to tell which method (class, thread) is stealing CPU time?
I feel that high CPU usage may be related to some thread that goes on infinite loop in some circumstances or maybe something like active waiting for a resource that is never released, but I have no idea how to proove it.

Best reagards,
Pawel

[Artur Linhart, modificado hace 15 años.]

Hello,

unfortunatelly I have no idea how to simply identify the thread or class which could cause the problems... You could connect Liferay for example to eclipse and then to try to track the activity - at least you could be able to see if the problem is really in the portal application or not... Maybe then also some profiling tool can be bound to the debugging output - but I do not have exact experiences about it.

Of course if it is Your productive system, the turning on of debugging can slow it down, but maybe for some short time this could be not such terrible condition.

According to the DB - generally maybe it is not a bad idea to put the DB on some other machine - what You probably know and what will not solve the high CPU load problem of liferay :-).

with regards, Archie

[Kevin J Butler, modificado hace 15 años.]

The easiest way to get a picture of the threads that are active in a JVM is to generate a stack dump.

You can do this on a Linux system by sending a kill -QUIT to the JVM process.

See http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/ for more information.

This won't tell you what thread is using the CPU, but it will tell you what threads are active (including background/system threads), and you can do a couple of dumps to identify threads that are remaining active, and you can try doing it when there is no other activity on the system.

kb

[Pawel Osak, modificado hace 15 años.]

Thanks, I'll try that if the JVM goes on high usage again. I think I could find the reason by chance - I noticed that my tomcat was set up to trace database connections from the pool and to close them, if they were not released by the application. I turned it off and for now CPU usage is stable. Anyway, I have to wait a week or two to confirm my assumption.

[Sanya Presich, modificado hace 14 años.]

Hi guys.
What the results of your examinations?

I ran into such problem but my Liferay are using all 100 % on all 4 CPU's)))

I have more than 10.000 users with near 100 open sessions in one time and more than 20.000 organizations.

And randomly my Liferay goes to use all the CPU's.

How can I examine what the reason for such behaviour?

[Pawel Osak, modificado hace 14 años.]

In fact I have no't found the reason. My Liferay installation is not mission critical, so I am able to run a shell script that stops and starts tomcat again once a night. Because Liferay had been starting to consume a lot of CPU after a few days of running this workaround solved the problem.

[Sanya Presich, modificado hace 14 años.]

Help please)))

I have 4-cores ( each 2,4 GHz) server with 8 Gb memory.

There are only Glassfish 2.1 with LifeRay 5.2.2 working.
На 4-х ядерном серваке по 2,4 Ггц с 8 Гб памяти крутится Glassfish 2.1 + LifeRay 5.2.2

After starting glassfish all 4 cores have 100 load!!! without any reasons....because there are only few open sessions at that time...

But, in general I have 15,000 users and 30,000 organizations with custom attributes.

I've been trying to change my JVM and Garbage Collector options but nothing helps me...

+ one thing: here is snapshot of my VisualGC, because I'm interesting why young generation collects each 5-10 seconds ...


Thanks...

[Victor Zorin, modificado hace 14 años.]

Sanya, I had 8-core CPU 100% load when lucene mapped to database. Can you disable your indexing and see if it is the same?

[Sanya Presich, modificado hace 14 años.]

Thanks, I'll try...but this load don't stop so there is only way to decrease it is to reboot glassfish((

[Sanya Presich, modificado hace 14 años.]

Sorry for such question but I can't at once find how to completely disable indexing...

And one more: our site is in production so it must be a careful testing)))

[Victor Zorin, modificado hace 14 años.]

you better take a copy of a system and test this in sandbox.

To disable indexing, check same good old portal.properties file.

[Sanya Presich, modificado hace 14 años.]

We are doing like that now.

There is a block in portal.properties file as Lucene Search.

All the options for our LR is default now.

What else options should I change to disable indexing?...because I have index.on.startup=false and that's all I see about turning indexing on/off...

Sorry for annoying questions)))

[Luca Preziati, modificado hace 14 años.]

What OS? 32 bit or 64 bit? What machine? What version of JVM do you use? What are your compilence level?
What are JVM starting option?
Have you tried to do stress test?
Do you implements some custom element? If yes, do u think use correct algorithm? This happen always or on some particular page? In your custom element do u think you have tuned correctly your cache policy?

[Victor Zorin, modificado hace 14 años.]

Try also index.read.only=true. Delete lucene folders and make sure that they are not created when system is not running. However I am not sure whether this is possible in a version that you are using.

I also read your parallel thread in Russian forum, what do you mean by 10 connections? You also may have some issues not related to portal, did you try to separate your components by boxes, e.g. by leaving liferay on one box and running app.servers on another?

[Sanya Presich, modificado hace 14 años.]

Sorry guys, that was an error in our custom portlets )))

But I understood a lot of debugging and moitoring of Liferay))) and something about Lucene))

So, thank you guys very much)))

[Lisa Simpson, modificado hace 14 años.]

This is for tomcat but some of it will apply to Glassfish as well....

I wrote a tutorial about tuning Liferay