High CPU usage on 6.2 GA4 with Postgres

[William Gosse, modificado 8 Anos atrás.]

Ok I've upgraded to 6.2 GA4 Tomcat Bundle and switched to Postgres but I'm still having the similar issues with high cpu. I've attached some data that I've gathered from visualvm, including a thread dump.

[William Gosse, modificado 8 Anos atrás.]

In yourkit I'm seeing a consistent pattern leading up to CPU going out of control for Tomcat. I see a lot of unsafe parked threads like the following:
ajp-apr-8009-exec-9 [WAITING] [DAEMON]
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:186
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() AbstractQueuedSynchronizer.java:2043
java.util.concurrent.LinkedBlockingQueue.take() LinkedBlockingQueue.java:442
org.apache.tomcat.util.threads.TaskQueue.take() TaskQueue.java:104
org.apache.tomcat.util.threads.TaskQueue.take() TaskQueue.java:32
java.util.concurrent.ThreadPoolExecutor.getTask() ThreadPoolExecutor.java:1068
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1130
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:615
java.lang.Thread.run() Thread.java:745

I also see a lot of blocker threads like the following:

[William Gosse, modificado 8 Anos atrás.]

I've attached the stacktrace from yourkit:

[David H Nebinger, modificado 8 Anos atrás.]

So what custom plugins do you have? What are the types of things they are doing (or not doing)?

Since no one else is seeing this and you've eliminated the DB as a source, eventually the finger has to be pointed somewhere.

The things you've captured from visualvm don't really point at anything. com.liferay.portal.kernel.concurrent.TaskQueue.poll() may indicate that it's checking on a lot of background threads and the C3P0PooledConnectionPoolManager blocks may indicate your connection pool is saturated or overwhelmed with requests, but they're certainly not pointing at a specific problem.

I believe you said in a previous thread that you're not a developer (forgive me if I've confused you w/ someone else). For me, I would start removing any custom code deployed to the portal. When you remove the part that leads to this situation, you'll have some idea what plugin is the likely cause. Once you know that you can focus in on what it is doing and determine what it's problem is.

[William Gosse, modificado 8 Anos atrás.]

I am a developer and I do have 3 custom portlets in this app at this time. I feel fairly confident that they are not involved because I've run the portal without them and the issue still persists. The only other thing I do have is a custom theme in place from a 3rd party which maybe suspect. I'm about ready to back it out and put in the classic theme as test. I was hoping that my analysis would help me to better Identify where the fault lays. We are not really getting a whole lot of real traffic but I do see that we are getting botted. I guess the only other thing I can say is that this whole thing is running in AWS EC2 on a Win 2008 R2 server with 16Gb of memory and a 500gb drive.

[David H Nebinger, modificado 8 Anos atrás.]

Well I'd probably still start removing plugins until the culprit was identified. If you think the theme is the suspect, that might be a good place to start.

Sorry for confusing you w/ another poster that said they weren't a developer, hope I didn't offend.

C3P0 config would be either in the JNDI connection for the database or there's portal-ext.properties you can set if you're using the JDBC parameters.

[William Gosse, modificado 8 Anos atrás.]

David H Nebinger:

Well I'd probably still start removing plugins until the culprit was identified. If you think the theme is the suspect, that might be a good place to start.

Sorry for confusing you w/ another poster that said they weren't a developer, hope I didn't offend.

C3P0 config would be either in the JNDI connection for the database or there's portal-ext.properties you can set if you're using the JDBC parameters.

Do you have an example of which properties would be involved or an example.

[David H Nebinger, modificado 8 Anos atrás.]

#
# Liferay can use C3PO, DBCP, or Tomcat for connection pooling. See
# com.liferay.portal.dao.jdbc.util.DataSourceFactoryBean for the actual
# implementation. It is important to understand the strengths and weaknesses
# of each provider so that you can choose the best one that fits your
# deployment scenario. Provider specific properties can also be passed along
# directly to the provider. For example, the property
# "jdbc.default.acquireIncrement" is read by C3PO, and the property
# "jdbc.default.maxActive" is read by DBCP.
#
# The default provider is C3PO.
#
jdbc.default.liferay.pool.provider=c3po
#jdbc.default.liferay.pool.provider=dbcp
#jdbc.default.liferay.pool.provider=tomcat

#
# The following properties will be read by C3PO if Liferay is configured to
# use C3PO in the property "jdbc.default.liferay.pool.provider". See
# http://www.mchange.com/projects/c3p0/index.html#configuration for a list
# of additional fields used by C3PO for configuring database connections.
#
jdbc.default.acquireIncrement=5
jdbc.default.acquireRetryAttempts=3
jdbc.default.acquireRetryDelay=1000
jdbc.default.connectionCustomizerClassName=com.liferay.portal.dao.jdbc.pool.c3p0.PortalConnectionCustomizer
jdbc.default.idleConnectionTestPeriod=60
jdbc.default.maxIdleTime=3600
jdbc.default.maxPoolSize=100
jdbc.default.minPoolSize=10
jdbc.default.numHelperThreads=10
#jdbc.default.transactionIsolation=1

#
# The following properties will be read by both DBCP and Tomcat JDBC
# Connection Pool if Liferay is configured to use DBCP or Tomcat JDBC
# Connection Pool in the property "jdbc.default.liferay.pool.provider". See
# http://commons.apache.org/dbcp/configuration.html and
# http://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html for a list of
# additional fields used by DBCP or Tomcat JDBC Connection Pool for
# configuring database connections.
#
#jdbc.default.defaultTransactionIsolation=READ_UNCOMMITTED
jdbc.default.maxActive=100
jdbc.default.minIdle=10
jdbc.default.removeAbandonedTimeout=3600

#
# The following properties will be read by Tomcat JDBC Connection Pool if
# Liferay is configured to use Tomcat JDBC Connection Pool in the property
# "jdbc.default.liferay.pool.provider". See
# http://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html for a list of
# additional fields used by Tomcat JDBC Connection Pool for configuring
# database connections.
#
jdbc.default.jdbcInterceptors=org.apache.tomcat.jdbc.pool.interceptor.ConnectionState
jdbc.default.jmxEnabled=true
jdbc.default.fairQueue=true
jdbc.default.timeBetweenEvictionRunsMillis=30000
jdbc.default.useEquals=false

[William Gosse, modificado 8 Anos atrás.]

Thanks for the properties. Are those the defaults and is c3p0 a good pick for a connection pool with liferay?

[David H Nebinger, modificado 8 Anos atrás.]

Yes, those are the defaults that come from portal.properties. Override them in portal-ext.properties to control the settings.

Although I wouldn't think these settings would matter much to a custom theme and portlets which only invoke a remote web service...

[William Gosse, modificado 8 Anos atrás.]

Yes they look pretty acceptable for what we are doing. Thanks for your help with this.

[Juan Gonzalez, modificado 8 Anos atrás.]

Did you already check if removing that custom theme the performance improves?

[Jack Bakker, modificado 8 Anos atrás.]

Which version of PostgreSQL ?

Is your JDBC driver in tomcat*/lib/ext correct ? (as per https://jdbc.postgresql.org/download.html)

You also might try doubling number of postgresql connections available in postgresql.conf

max_connections = 100                   # (change requires restart)
# Note:  Increasing max_connections costs ~400 bytes of shared memory per
# connection slot, plus lock space (see max_locks_per_transaction).

[William Gosse, modificado 8 Anos atrás.]

I did try doubling the max connections in postgres but I hadn't updated the driver in tomcat. I just did so now so we'll see what that brings us. Thanks

I'm on Postgres 9.4 so yes I just installed the the postgresql-9.4-1201.jdbc41 driver because I am on Java 7.

[Jack Bakker, modificado 8 Anos atrás.]

William, plz let me know if driver or other updates help you ; might help me too. Thanks.

[William Gosse, modificado 8 Anos atrás.]

I still had the issue even after the driver upgrade. The only thing left is a third party theme which I've back out. Waiting to see if this may be the cause. Will know in a few more days.

[Jack Bakker, modificado 8 Anos atrás.]

William Gosse:

I still had the issue even after the driver upgrade. The only thing left is a third party theme which I've back out. Waiting to see if this may be the cause. Will know in a few more days.

Thanks for sharing. I've been using postgresql since way back to v7 but am into v9.4 pg lately, investigating the jsonb type (which gets indexed by pg) and doing searches on such. The idea of mixing sql with nosql(key/value) is kinda cool, especially when speedy searches might be done.

[William Gosse, modificado 8 Anos atrás.]

I've been using MongoDB for several years now in both my web services and in Liferay. I've used the Morphia API to access Mongo from Java.

[William Gosse, modificado 8 Anos atrás.]

No problem on the confusion. This is like my fourth Liferay I've developed going back to 6.0 and 6.1 but its definitely been my most problematic. It is the first attempt with Postgres though.

[William Gosse, modificado 8 Anos atrás.]

David H Nebinger:

The things you've captured from visualvm don't really point at anything. com.liferay.portal.kernel.concurrent.TaskQueue.poll() may indicate that it's checking on a lot of background threads and the C3P0PooledConnectionPoolManager blocks may indicate your connection pool is saturated or overwhelmed with requests, but they're certainly not pointing at a specific problem.

How would I increase the connection pool size. The only thing using the database is liferay. My custom portlets are only restful clients that talk to a separate web service. Again at this time they are hardly used.

[Juan Gonzalez, modificado 8 Anos atrás.]

William Gosse:

My custom portlets are only restful clients that talk to a separate web service.

Maybe this is the key. Is there a chance that the response from that web service spends too much time and hence let many http threads idle waiting for that response while other client requests are being made?

[Juan Gonzalez, modificado 8 Anos atrás.]

BTW are those web services being used somehow in your theme?

[William Gosse, modificado 8 Anos atrás.]

Again I have removed the portlets and the problem persists and the web service is not used by theme.

[Alexey Kakunin, modificado 8 Anos atrás.]

Hi William!

Just tested on my local machine: Ubuntu 14.10 + Liferay 6.2 CE4 + PostgreSQL 9.4.1 on Oracle Java 1.7.0_80
Everything works fine - I've used default bundle

Could you:
1. On your machine there you have server try to test clean liferay bundle with clean postgres database?
2. Try to copy environment to some other server and test where?

Porlbem looks very strange and it is important first - understand is it problem in some specific component on your ssytem or not.

====
Alexey Kakunin
Liferay Experts in Russia

[Сергей К, modificado 8 Anos atrás.]

hello. did you resolve that issue? I have same problem after update from liferay 6.1 to 6.2 GA4. I'm using Liferay on tomcat and Postgres 9.4

[Marc-Andre Gauthier, modificado 8 Anos atrás.]

I have cpu usage problems too.

Setup: 6.2 GA5 Tomcat Bundle / Postgres


What I have noticed: CPU is high when a programmer uses eclipse and connects with Liferay IDE’s Remote Server Adapter. Remote Tomcat's CPU usage stays at 75% and doesn't drop. Usage drops to 0% as soon as wee close eclipse on the developers machine.

Maybe it's not the same problem. I don't have a solution.
Someone experienced this problem?

[William Gosse, modificado 8 Anos atrás.]

Sorry I hadn't look at this post in a while but my issues with 6.2 GA4 and Postgres seem to have been reslovled.

I'm not 100% sure of the fix but i think it may have been an issue with an apache httpd I was using as my front door.

After upgrading and re configuring it my other issues seem to have vanished. Been running a couple of months now with no cpu or memory problems at all.

I'll try to give some further detalis to this as well.

Things did stabilize for us while we were still running on AWS. We since migrated to IBM softslate too take advantage of a free server deal. No problems Softslate as well.

Been using 6.2 GA4 and Postgres on some other sites as well and have been happy with this combination so far.

I was a big fan of MySQL, which is less strict then Postgres, but we were having some very awful issue with it in combination with Liferay.