Foren

c3p0 Connection Pool -- unreturned connections?

thumbnail
Andew Jardine, geändert vor 11 Jahren.

c3p0 Connection Pool -- unreturned connections?

Liferay Legend Beiträge: 2416 Beitrittsdatum: 22.12.10 Neueste Beiträge
Hi,

I have a real puzzle I am trying to figure out right now.A quick summary of what I think is going wrong and how I am patching the holes in a sinking ship.

1. 2 node cluster -- 10GB RAM and 500 DB connections per node
2. 10 - 15 minute window until all resources are exhausted and the server hangs
3. I enabled DEBUG level logging for com.mchange and can see the pool steadily increasing
4. I enable several JDBC loggers at DEBUG and can see a global prepared statement count steadily increasing

I suspect that the two coincided and that the prepares statement was not returning the connection to the pool. I turned on 3 additional c3p0 settings in the portal-ext file.

a. jdbc.default.testConnectionOnCheckout=true
b. jdbc.default.unreturnedConnectionTimeout=300 #5 minutes
c. jdbc.default.debugUnreturnedConnectionStackTraces=true

I was hoping that c. would lead me to the source of the leak. Unfortunately, I don't see the stacktraces being pushed out. On top of that I am occasionally having a problem saving a simple WebForm portlet's information. The stack trace that coincides with this error is --


12:33:27,794 ERROR [JDBCExceptionReporter:101] No operations allowed after connection closed.
12:33:27,799 ERROR [WebFormPortlet:331] The web form data could not be saved to the database
com.liferay.portal.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: could not load an entity: [com.liferay.counter.model.Counter#com.liferay.counter.model.Counter]
	at com.liferay.portal.service.persistence.impl.BasePersistenceImpl.processException(BasePersistenceImpl.java:91)
	at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:162)
	at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:121)
	at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:117)
	at com.liferay.counter.service.persistence.CounterUtil.increment(CounterUtil.java:42)
	at com.liferay.counter.service.impl.CounterLocalServiceImpl.increment(CounterLocalServiceImpl.java:44)
	at sun.reflect.GeneratedMethodAccessor596.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy82.increment(Unknown Source)
	at com.liferay.portlet.expando.service.impl.ExpandoValueLocalServiceImpl.addValue(ExpandoValueLocalServiceImpl.java:66)
	at com.liferay.portlet.expando.service.impl.ExpandoValueLocalServiceImpl.addValue(ExpandoValueLocalServiceImpl.java:507)
	at sun.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy98.addValue(Unknown Source)
	at com.liferay.portlet.expando.service.ExpandoValueLocalServiceUtil.addValue(ExpandoValueLocalServiceUtil.java:259)
	at com.liferay.webform.portlet.WebFormPortlet.saveDatabase(WebFormPortlet.java:323)
	at com.liferay.webform.portlet.WebFormPortlet.saveData(WebFormPortlet.java:183)
	at sun.reflect.GeneratedMethodAccessor730.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at com.liferay.portal.kernel.portlet.LiferayPortlet.callActionMethod(LiferayPortlet.java:111)
	at com.liferay.portal.kernel.portlet.LiferayPortlet.processAction(LiferayPortlet.java:69)
	at com.sun.portal.portletcontainer.appengine.filter.FilterChainImpl.doFilter(FilterChainImpl.java:98)
	at com.liferay.portal.kernel.portlet.PortletFilterUtil.doFilter(PortletFilterUtil.java:57)
	at com.liferay.portal.kernel.servlet.PortletServlet.service(PortletServlet.java:100)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
	at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:535)
	at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:472)
	at com.liferay.portlet.InvokerPortletImpl.invoke(InvokerPortletImpl.java:618)
	at com.liferay.portlet.InvokerPortletImpl.invokeAction(InvokerPortletImpl.java:664)
	at com.liferay.portlet.InvokerPortletImpl.processAction(InvokerPortletImpl.java:359)
	at com.liferay.portal.action.LayoutAction.processPortletRequest(LayoutAction.java:604)
	at com.liferay.portal.action.LayoutAction.processLayout(LayoutAction.java:428)
	at com.liferay.portal.action.LayoutAction.execute(LayoutAction.java:195)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
	at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:157)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
	at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:608)
	at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
	at com.liferay.portal.servlet.FriendlyURLServlet.service(FriendlyURLServlet.java:143)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:142)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:140)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:282)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.servlet.filters.cache.CacheFilter.processFilter(CacheFilter.java:425)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:191)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.servlet.filters.threadlocalcache.ThreadLocalCacheFilter.processFilter(ThreadLocalCacheFilter.java:55)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:91)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:154)
	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:94)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
	at java.lang.Thread.run(Thread.java:595)
Caused by: com.liferay.portal.kernel.dao.orm.ORMException: could not load an entity: [com.liferay.counter.model.Counter#com.liferay.counter.model.Counter]
	at com.liferay.portal.dao.orm.hibernate.ExceptionTranslator.translate(ExceptionTranslator.java:41)
	at com.liferay.portal.dao.orm.hibernate.SessionImpl.get(SessionImpl.java:131)
	at com.liferay.counter.service.persistence.CounterPersistence.increment(CounterPersistence.java:145)
	... 130 more


I'm not sure but i suspect that for some reason the unreturned connections are being used in this scenario. When I first boot I don't see the issue but eventually this starts to show up and the "Portet is unavailable message" is displayed to me after submitting the form.

I'm beating my brains trying to figure this problem out. Can anyone help?
Igor Barma, geändert vor 11 Jahren.

RE: c3p0 Connection Pool -- unreturned connections?

New Member Beiträge: 16 Beitrittsdatum: 30.09.09 Neueste Beiträge
Hi,
I've sometimes exactly the same problem with my database c3p0 connector.
Have you find a solution to it? It's very frustating for me to not find the answer.
My site works well for a time (a couple of days) and then my pool become busy on all my connections for a node, i need to restart it and sometimes restart my DB too.
I turned on unreternedConnectionTimeout like you to monitor leak in my code but without result cause the error:
ERROR [JDBCExceptionReporter:101] No operations allowed after connection closed.

It will be very nice if you have found something on this problem....

Thanks,
Igor
thumbnail
Andew Jardine, geändert vor 11 Jahren.

RE: c3p0 Connection Pool -- unreturned connections?

Liferay Legend Beiträge: 2416 Beitrittsdatum: 22.12.10 Neueste Beiträge
Hi Igor,

To be honest I never fully solved the issue. I managed to get it to the point where the server only ran out of resources once a month which seemed a suitable solution for my client (at the time). In the end I simply mucked with the pool settings as well as the hibernate and mysql settings until the results seemed reasonable. I think I also set some pretty aggressive garbage collection settings on the JVM.

I did try a new install which didn't seem to have an issue until I moved the code over. That seemed to me to be an indication of a problem with the custom code (written by someone else for the record! emoticon ) rather than a problem with liferay (core).

Hope that helps.