留言板
Blocked threads + slow performance
Hi All,
We are using Liferay 6.1.2 EE and almost everyday we have some strange behavior of our portal.
Starting from 10:00 AM to 11:30 AM the portal is almost unusable - the users start getting timeout in the browsers, switching/logging in/opening new pages is very slow or impossible.
The used ram is ~2-3GB from 10GB available and the CPUs are at 30-40%.
Few months ago we made some tuning on the portal as we had the same user problems but then the problem was in the RAM, CPU and a bug in one of our libraries.
In the thread dump there are lots of BLOCKED threads (~116) and all of them are related to the struts library where the portal is trying to sync a HashMap which is used to collect the Actions.
I have tried to find some more info about this problem but till now no luck.
I would appreciate if someone can give me hints on this problem.
Here is the dump:
Thanks in advance.
We are using Liferay 6.1.2 EE and almost everyday we have some strange behavior of our portal.
Starting from 10:00 AM to 11:30 AM the portal is almost unusable - the users start getting timeout in the browsers, switching/logging in/opening new pages is very slow or impossible.
The used ram is ~2-3GB from 10GB available and the CPUs are at 30-40%.
Few months ago we made some tuning on the portal as we had the same user problems but then the problem was in the RAM, CPU and a bug in one of our libraries.
In the thread dump there are lots of BLOCKED threads (~116) and all of them are related to the struts library where the portal is trying to sync a HashMap which is used to collect the Actions.
I have tried to find some more info about this problem but till now no luck.
I would appreciate if someone can give me hints on this problem.
Here is the dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.7-b02 mixed mode):
"ajp-bio-8009-exec-2195" daemon prio=10 tid=0x00007ff1088e2800 nid=0x75b3 waiting for monitor entry [0x00007ff041e5f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:278)
- waiting to lock <0x0000000576fef4c8> (a java.util.HashMap)
at com.liferay.portal.struts.PortalRequestProcessor.processActionCreate(PortalRequestProcessor.java:500)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:230)
at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:176)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:560)
at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:537)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:335)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:123)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:294)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.sso.ntlm.NtlmPostFilter.processFilter(NtlmPostFilter.java:83)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:216)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:187)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:167)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:167)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:187)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:73)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
- locked <0x0000000563000a20> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"ajp-bio-8009-exec-2194" daemon prio=10 tid=0x00007ff1088e0800 nid=0x75b2 runnable [0x00007ff041f62000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:309)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:364)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:128)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
- locked <0x000000056323d7c0> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"ajp-bio-8009-exec-2193" daemon prio=10 tid=0x00007ff1088de800 nid=0x75b1 waiting for monitor entry [0x00007ff04205f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:278)
- waiting to lock <0x0000000576fef4c8> (a java.util.HashMap)
at com.liferay.portal.struts.PortalRequestProcessor.processActionCreate(PortalRequestProcessor.java:500)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:230)
at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:176)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:560)
at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:537)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:294)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:73)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at com.liferay.portal.servlet.FriendlyURLServlet.service(FriendlyURLServlet.java:138)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:335)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:123)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:294)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.i18n.I18nFilter.processFilter(I18nFilter.java:241)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.etag.ETagFilter.processFilter(ETagFilter.java:56)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:253)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.sso.ntlm.NtlmPostFilter.processFilter(NtlmPostFilter.java:83)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:216)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:187)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:167)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:167)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:187)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:73)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
- locked <0x000000056323dbb8> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
......
"ajp-bio-8009-exec-966" daemon prio=10 tid=0x00007ff10809a800 nid=0x631e runnable [0x00007ff08629e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.newInstance0(Class.java:326)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.struts.util.RequestUtils.applicationInstance(RequestUtils.java:143)
at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:292)
- locked <0x0000000576fef4c8> (a java.util.HashMap)
at com.liferay.portal.struts.PortalRequestProcessor.processActionCreate(PortalRequestProcessor.java:500)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:230)
at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:176)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:560)
at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:537)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.strip.StripFilter.processFilter(StripFilter.java:335)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.gzip.GZipFilter.processFilter(GZipFilter.java:123)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.secure.SecureFilter.processFilter(SecureFilter.java:294)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:253)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.sso.ntlm.NtlmPostFilter.processFilter(NtlmPostFilter.java:83)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:163)
at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:216)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:57)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:187)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:206)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:108)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:167)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:167)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:187)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:95)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:73)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
- locked <0x0000000608894a58> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
.....
Heap
par new generation total 645120K, used 574244K [0x0000000540000000, 0x000000056bc00000, 0x000000056bc00000)
eden space 573440K, 90% used [0x0000000540000000, 0x000000055fd33d18, 0x0000000563000000)
from space 71680K, 73% used [0x0000000563000000, 0x0000000566395360, 0x0000000567600000)
to space 71680K, 0% used [0x0000000567600000, 0x0000000567600000, 0x000000056bc00000)
concurrent mark-sweep generation total 7672960K, used 2974695K [0x000000056bc00000, 0x0000000740120000, 0x00000007c0000000)
concurrent-mark-sweep perm gen total 1048576K, used 760147K [0x00000007c0000000, 0x0000000800000000, 0x0000000800000000)
Thanks in advance.
Alexey Kakunin，修改在8 年前。
RE: Blocked threads + slow performance
Liferay Master 帖子: 621 加入日期: 08-7-7 最近的帖子
Lubomir, hi!
Since you are Enterprise User it is highly recommended to contact Liferay Support (finally - you paid for it!), especially for such complex questions.
From my expereince support engineers has really great expereince and I hope they will help you!
====
Alexey Kakunin
Liferay Experts in Russia
Since you are Enterprise User it is highly recommended to contact Liferay Support (finally - you paid for it!), especially for such complex questions.
From my expereince support engineers has really great expereince and I hope they will help you!
====
Alexey Kakunin
Liferay Experts in Russia
Lubomir Georgiev，修改在8 年前。
RE: Blocked threads + slow performance
New Member 帖子: 23 加入日期: 13-10-29 最近的帖子
Hi,
Thanks for the answer.
This was my first reaction as well.
The problem is that our client (they have bought the portal) don't want to contact the support as the answer will be to update to the latest LR version.
Currently our client don't want to update LR as it will be related to some code changes.
The only thing they want is to try to fix this bug if it is possible.
Thanks for the answer.
This was my first reaction as well.
The problem is that our client (they have bought the portal) don't want to contact the support as the answer will be to update to the latest LR version.
Currently our client don't want to update LR as it will be related to some code changes.
The only thing they want is to try to fix this bug if it is possible.
Patrick Wolf，修改在8 年前。
RE: Blocked threads + slow performance
Regular Member 帖子: 127 加入日期: 10-9-15 最近的帖子
Hi,
If I were you I would also ask the experts of Liferay to shed a light on your client's problem. As Alexey answered, your client paid for an enterprise licence and may require some advice when such a problem happens. I don't think Liferay would force your client to do an upgrade, which would not solve the problem. Moreover, if development best practices, i.e. hooks to override some services instead of the « ext » model, were applied, a future upgrade should « theoretically » go smoothly.
If I were you I would also ask the experts of Liferay to shed a light on your client's problem. As Alexey answered, your client paid for an enterprise licence and may require some advice when such a problem happens. I don't think Liferay would force your client to do an upgrade, which would not solve the problem. Moreover, if development best practices, i.e. hooks to override some services instead of the « ext » model, were applied, a future upgrade should « theoretically » go smoothly.
Alexey Kakunin，修改在8 年前。
RE: Blocked threads + slow performance
Liferay Master 帖子: 621 加入日期: 08-7-7 最近的帖子
Is problem happens regullary (you said every day from 10 AM till 11-30 AM)? How problem "gone" - you restart server or it just started to work well after some period of time.
If problem happens periodically at specific time - I think these "blocked" threads are only side-effect - original reason in some operations done in background in same time. First you need to find what happens at this time and how it may affect general functionality.
Some other questions:
1. What is state of connection pool? Is it ok?
2. Do you have any dead-locks in database? (we had situations very similar to this then had dead-locks in database).
Anyway - yes, upgrading to new version, even in theory it should be simple, in practise sometimes not too simple (especially if you had to change some core classes via using ext-plugin or activelly used hooks to chnage defult jsp files and actions). And anyway - still it make sense to contact Liferay - they may help with diagnostics.
====
Alexey Kakunin
Liferay Experts in Russia
If problem happens periodically at specific time - I think these "blocked" threads are only side-effect - original reason in some operations done in background in same time. First you need to find what happens at this time and how it may affect general functionality.
Some other questions:
1. What is state of connection pool? Is it ok?
2. Do you have any dead-locks in database? (we had situations very similar to this then had dead-locks in database).
Anyway - yes, upgrading to new version, even in theory it should be simple, in practise sometimes not too simple (especially if you had to change some core classes via using ext-plugin or activelly used hooks to chnage defult jsp files and actions). And anyway - still it make sense to contact Liferay - they may help with diagnostics.
====
Alexey Kakunin
Liferay Experts in Russia
Patrick Wolf，修改在8 年前。
RE: Blocked threads + slow performance
Regular Member 帖子: 127 加入日期: 10-9-15 最近的帖子
If your client gets custom portlets and services and prefer not to call Liferay's experts, did you try to remove or undeploy all the stuff that is not part of the default portal in order to get a bare Liferay that should then work properly, and deploy one after the other your in-house portlets and services, wait for the time when it gets stuck, in order to pinpoint what is the culprit?
This may take some time but you would at least get a better idea of what is going wrong.
This may take some time but you would at least get a better idea of what is going wrong.
Lubomir Georgiev，修改在8 年前。
RE: Blocked threads + slow performance
New Member 帖子: 23 加入日期: 13-10-29 最近的帖子
Hi,
A little bit more info on the problem - during this interval 10:00 AM-11:30 AM the number of users which are using the portal is bigger (~230) in comparison with the rest of the day (~130).
@Alexey - What do you mean by the state of the connection pool? Are you talking about the connection between the LR and its DB? Is there an easy way to check it or I have to use JConsole?
@Patrick - Our client is using mainly custom portlets. During the weekend the portal is ok and stable as the number of users is much lower - so basically If we remove all the custom portlets then there will be no users in the portal.
Here are the start parameters:
Regarding the support - I am trying to persuade our client to open a ticket for this problem.
A little bit more info on the problem - during this interval 10:00 AM-11:30 AM the number of users which are using the portal is bigger (~230) in comparison with the rest of the day (~130).
@Alexey - What do you mean by the state of the connection pool? Are you talking about the connection between the LR and its DB? Is there an easy way to check it or I have to use JConsole?
@Patrick - Our client is using mainly custom portlets. During the weekend the portal is ok and stable as the number of users is much lower - so basically If we remove all the custom portlets then there will be no users in the portal.
Here are the start parameters:
JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8 -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false -Duser.timezone=Europe/Sofia -Xms10240m -Xmx16384m -XX:MaxPermSize=300m -XX:NewSize=700m -XX:MaxNewSize=700m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled"Regarding the support - I am trying to persuade our client to open a ticket for this problem.
Patrick Wolf，修改在8 年前。
RE: Blocked threads + slow performance
Regular Member 帖子: 127 加入日期: 10-9-15 最近的帖子
Hi,
Don't you have a pre-production Liferay which matches the production environment so that you can spot which portlet is lagging and do some stress tests? If you get a production environment only, then it would be difficult for you as you may have to stop and restart Liferay.
230 users is nothing and Liferay should respond in a flash.
Regarding the configuration of the JVM, something looks odd to me. It is recommended that the size of the minimum and maximum heap (-Xms and -Xmx parameters) is set to the same value. These values depend on the amount of RAM of the server where your Tomcat is installed. If your server gets less that 8 GB RAM, try to start with a heap of 700m, if the RAM is greater or equal to 8 GB, set the heap at 3072m.
It is the same for the -XX : PermSize and -XX : MaxPermSize parameters, which define the size of the permanent generation space (PermGen). When this space gets full, the JVM will try to do a full garbage collection to reclaim space, and this can often be a source of problems for applications which dynamically create or load a large number of classes. You get a load of information on that topic through Google.
Liferay uses the database connection pool to perform the database connection. The size of the database connection pool is important. By default this is configured in the portal-ext.properties file, unless you configured the database connection through Tomcat using JNDI. In that case this should be configured in the ROOT.xml file. You may use the c3p0 connection pool. Start with the default connection pool settings which should fit.
Don't you have a pre-production Liferay which matches the production environment so that you can spot which portlet is lagging and do some stress tests? If you get a production environment only, then it would be difficult for you as you may have to stop and restart Liferay.
230 users is nothing and Liferay should respond in a flash.
Regarding the configuration of the JVM, something looks odd to me. It is recommended that the size of the minimum and maximum heap (-Xms and -Xmx parameters) is set to the same value. These values depend on the amount of RAM of the server where your Tomcat is installed. If your server gets less that 8 GB RAM, try to start with a heap of 700m, if the RAM is greater or equal to 8 GB, set the heap at 3072m.
It is the same for the -XX : PermSize and -XX : MaxPermSize parameters, which define the size of the permanent generation space (PermGen). When this space gets full, the JVM will try to do a full garbage collection to reclaim space, and this can often be a source of problems for applications which dynamically create or load a large number of classes. You get a load of information on that topic through Google.
Liferay uses the database connection pool to perform the database connection. The size of the database connection pool is important. By default this is configured in the portal-ext.properties file, unless you configured the database connection through Tomcat using JNDI. In that case this should be configured in the ROOT.xml file. You may use the c3p0 connection pool. Start with the default connection pool settings which should fit.
Alexey Kakunin，修改在8 年前。
RE: Blocked threads + slow performance
Liferay Master 帖子: 621 加入日期: 08-7-7 最近的帖子
Hi! Yes, connection of of connections between Liferay to database. It is usually possible to control via JXM (JConsole for example). Another way to control (but I think available only for 6.1.30) is via Liferay Connected Services. If you have at this time deadlocks in database, connection pool will have big size and have many "used" connections. That will mean you need monitor database to check for possible deadlocks.
====
Alexey Kakunin
Liferay Experts in Russia
====
Alexey Kakunin
Liferay Experts in Russia
Lubomir Georgiev，修改在8 年前。
RE: Blocked threads + slow performance
New Member 帖子: 23 加入日期: 13-10-29 最近的帖子
Hi,
We still do not have any solution to our problem but here is something that we found during the research/monitoring:
There are some constantly growing connections between the LR and our backend (Web Services). These connections are in CLOSE_WAIT state and at the end of the day are about 10K connections. In the same time the number of blocked threads is zero, so I guess that there is no relation between the Blocked Threads and these connections.
Any ideas?
We still do not have any solution to our problem but here is something that we found during the research/monitoring:
There are some constantly growing connections between the LR and our backend (Web Services). These connections are in CLOSE_WAIT state and at the end of the day are about 10K connections. In the same time the number of blocked threads is zero, so I guess that there is no relation between the Blocked Threads and these connections.
Any ideas?
Matthias Delbar，修改在6 年前。
RE: Blocked threads + slow performance
New Member 帖子: 2 加入日期: 17-9-15 最近的帖子
Hello, sorry to gravedig this post, but we recently have encountered the exact same issue on our Production environment and this is the only post I've found that describes our exact issue.
At random moments (though only with medium-high user load), the application slows down to a crawl.
Every action takes more time (not just a specific group), but CPU is always around 30-40%.
Thread dump shows nothing special, and we have plenty of JVM and stack memory left.
We have a JMX statistics page that investigates the DB pool connections, and there are still many connections available so it's not like the pool is saturated either.
We use the community edition so we can't call a support line, and we can't change LR versions because of the cost involved.
Lubomir, if you're still here, did you ever find a fix for your problem?
Does anyone else have any ideas?
At random moments (though only with medium-high user load), the application slows down to a crawl.
Every action takes more time (not just a specific group), but CPU is always around 30-40%.
Thread dump shows nothing special, and we have plenty of JVM and stack memory left.
We have a JMX statistics page that investigates the DB pool connections, and there are still many connections available so it's not like the pool is saturated either.
We use the community edition so we can't call a support line, and we can't change LR versions because of the cost involved.
Lubomir, if you're still here, did you ever find a fix for your problem?
Does anyone else have any ideas?
Christoph Rabel，修改在6 年前。
RE: Blocked threads + slow performance
Liferay Legend 帖子: 1554 加入日期: 09-9-24 最近的帖子
Do you also see a constantly growing amount of connections to some backend services? (See last post)
Then you need to find out which service opens more and more connections.
I had such issues several times. It is quite common in Java that developers forget to properly close connections. The garbage collector then closes them some time in the future.
I once upgraded a server from 4 GB -> 16 GB of RAM. Suddenly we had thousands of "open" connections to an ldap server. Since the server had more RAM now, it practically never garbage collected and never cleaned up -> Connections grew and grew and at some point the backend started to decline connections.
I also had an issue with network storage. Heavy load -> file access slowed to a crawl.
Look at vmstat, iostat, netstat, maybe you can find a bottleneck with those tools. And you should also look into the database. Try to find slow queries. Most databases have tools to help you to find slow queries. Even if it isn't a connection pool issue, it might still be that each connection takes longer to complete when there is more traffic.
Then you need to find out which service opens more and more connections.
I had such issues several times. It is quite common in Java that developers forget to properly close connections. The garbage collector then closes them some time in the future.
I once upgraded a server from 4 GB -> 16 GB of RAM. Suddenly we had thousands of "open" connections to an ldap server. Since the server had more RAM now, it practically never garbage collected and never cleaned up -> Connections grew and grew and at some point the backend started to decline connections.
I also had an issue with network storage. Heavy load -> file access slowed to a crawl.
Look at vmstat, iostat, netstat, maybe you can find a bottleneck with those tools. And you should also look into the database. Try to find slow queries. Most databases have tools to help you to find slow queries. Even if it isn't a connection pool issue, it might still be that each connection takes longer to complete when there is more traffic.
Matthias Delbar，修改在6 年前。
RE: Blocked threads + slow performance
New Member 帖子: 2 加入日期: 17-9-15 最近的帖子
Hi Christoph, thanks for answering!
Sadly, we don't see such behaviour. Our connections to backend services are properly closed as needed, that was one of the first things we checked (for the same reasons you mention).
We have around 200 JVM threads for ~100 concurrent users, with ~80-90 in a RUNNABLE state (i.e. actually working) and the others either WAITING or TIMED_WAITING (mostly worker threads or connection threads waiting for new user input).
We investigated what the threads were doing (via jstack thread-dumps) and both their number and their stacktraces are in line with expectations for that amount of users (either as connection thread performing application logic, or a background worker thread used by JVM/JBoss).
I can also confirm that garbage collections still take place and are "relevant" collections (there is actually stuff being cleaned). We have this info through our JMX-based monitoring page.
We've already been looking at vmstat/netstat/iostat and they don't show any discrepancies or high load. DB queries also take an acceptable time (never more than a couple 100ms for the heaviest queries), until the mystery moment where everything slows down at the same time.
One thing to note is that I don't know if the slow-down persists up to the DB layer, since we don't have access to the DB server (managed by the client). That said, our metrics include time measurements right before/after making a DB call and these also show slowdowns. The DB admin has also told us that the DB and its server did not experience any high load issues, so I think it's safe to assume that the problem isn't in the DB (unless you have reason to believe otherwise).
From what we can tell, there is no specific user interaction that triggers the problem either. We're not even sure if it's due to the load, since our load tests simulate a higher load (more users doing things more frequently) and can find no issues in Dev/Acceptance environments (we can't execute these tests on Production because of live data).
I'm not sure if this helps you at all, as you can see we've tried many things but are still stumped (which is why I'm hoping for the miraculous comeback of Lubomir, unlikely as it is).
Sadly, we don't see such behaviour. Our connections to backend services are properly closed as needed, that was one of the first things we checked (for the same reasons you mention).
We have around 200 JVM threads for ~100 concurrent users, with ~80-90 in a RUNNABLE state (i.e. actually working) and the others either WAITING or TIMED_WAITING (mostly worker threads or connection threads waiting for new user input).
We investigated what the threads were doing (via jstack thread-dumps) and both their number and their stacktraces are in line with expectations for that amount of users (either as connection thread performing application logic, or a background worker thread used by JVM/JBoss).
I can also confirm that garbage collections still take place and are "relevant" collections (there is actually stuff being cleaned). We have this info through our JMX-based monitoring page.
We've already been looking at vmstat/netstat/iostat and they don't show any discrepancies or high load. DB queries also take an acceptable time (never more than a couple 100ms for the heaviest queries), until the mystery moment where everything slows down at the same time.
One thing to note is that I don't know if the slow-down persists up to the DB layer, since we don't have access to the DB server (managed by the client). That said, our metrics include time measurements right before/after making a DB call and these also show slowdowns. The DB admin has also told us that the DB and its server did not experience any high load issues, so I think it's safe to assume that the problem isn't in the DB (unless you have reason to believe otherwise).
From what we can tell, there is no specific user interaction that triggers the problem either. We're not even sure if it's due to the load, since our load tests simulate a higher load (more users doing things more frequently) and can find no issues in Dev/Acceptance environments (we can't execute these tests on Production because of live data).
I'm not sure if this helps you at all, as you can see we've tried many things but are still stumped (which is why I'm hoping for the miraculous comeback of Lubomir, unlikely as it is).