S3 Data Store Timeout Issue

[Sai Mohan Gadamsetty, modifié il y a 6 années.]

Hi,

We are using S3 store as data repository (Liferay 7.0 CE GA3) for our production environment.

We are having a wiki page to load html content, html will initiate more than 300 requests (document thumbnails) to S3 Bucket to display logos on page. In some scenarios/cases observing "Connection Time out" issue and from that time site performance slows down drastically and loading each page will take minutes of time and console (catalina.out) keep on throw below error logs,

We are using below configs to connect S3 bucket
* httpClientMaxConnections = 50
* connectionTimeout = 10000

Right now we have 15 to 20 users (beta users) are visiting to site and performing site activities.

I want to understand why this error occurring, because of initiating 300 requests causing issue or other? Do we have any monitor/log mechanism (like visual vm) to evaluate how connection pool getting utilized by tomcat/Liferay? What are the best S3 data store configuration/setup for 500 to 1000 users?

Please help me to solve this issue.

Error Log

10:01:31,778 WARN [http-nio-8443-exec-8][PortalImpl:6647] com.liferay.portal.kernel.exception.SystemException: Unable to execute HTTP request: Timeout waiting for connection from pool
com.liferay.portal.kernel.exception.SystemException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.liferay.portal.store.s3.S3Store.transform(S3Store.java:709)
at com.liferay.portal.store.s3.S3Store.getS3ObjectSummaries(S3Store.java:603)
at com.liferay.portal.store.s3.S3Store.getHeadVersionLabel(S3Store.java:513)
at com.liferay.portal.store.s3.S3Store.getS3Object(S3Store.java:545)
at com.liferay.portal.store.s3.S3Store.getFileAsStream(S3Store.java:192)
at com.liferay.document.library.kernel.store.BaseStore.getFileAsStream(BaseStore.java:322)
at com.liferay.portlet.documentlibrary.store.DLStoreImpl.getFileAsStream(DLStoreImpl.java:290)
at com.liferay.document.library.kernel.store.DLStoreUtil.getFileAsStream(DLStoreUtil.java:383)
at com.liferay.document.library.kernel.util.DLPreviewableProcessor.doGetThumbnailAsStream(DLPreviewableProcessor.java:424)
at com.liferay.portlet.documentlibrary.util.ImageProcessorImpl.getThumbnailAsStream(ImageProcessorImpl.java:124)
at com.liferay.document.library.kernel.util.ImageProcessorUtil.getThumbnailAsStream(ImageProcessorUtil.java:117)
at com.liferay.portal.webserver.WebServerServlet.sendFile(WebServerServlet.java:961)
at com.liferay.portal.webserver.WebServerServlet.service(WebServerServlet.java:276)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:119)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:142)
at com.liferay.portal.security.sso.ntlm.internal.servlet.filter.NtlmPostFilter.processFilter(NtlmPostFilter.java:107)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:188)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:188)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:188)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:142)
at com.liferay.portal.servlet.filters.uploadservletrequest.UploadServletRequestFilter.processFilter(UploadServletRequestFilter.java:93)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:142)
at com.liferay.portal.servlet.filters.jsoncontenttype.JSONContentTypeFilter.processFilter(JSONContentTypeFilter.java:42)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:142)
at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:268)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:142)
at com.liferay.portal.sharepoint.SharepointFilter.processFilter(SharepointFilter.java:88)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:142)
at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:316)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:188)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
at com.liferay.portal.servlet.filters.urlrewrite.UrlRewriteFilter.processFilter(UrlRewriteFilter.java:65)
at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:48)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:207)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:112)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:168)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:168)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:188)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:115)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:500)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3608)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3561)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)
at com.liferay.portal.store.s3.S3Store.getS3ObjectSummaries(S3Store.java:582)
... 89 more
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)
at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
at sun.reflect.GeneratedMethodAccessor1702.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
at com.amazonaws.http.conn.$Proxy758.getConnection(Unknown Source)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
... 94 more

[Andrew Jardine, modifié il y a 6 années.]

The html code that you are referencing as "on a wiki page" -- is this code that you have written? If it is, then do you really need to grab 300 items for every user that loads the page?

[Sai Mohan Gadamsetty, modifié il y a 6 années.]

Hello Andrew,

Thanks for your quick reply,

Yes, we are loading 300 items through html script (document thumbnail link) via wiki Front Page. I did some workaround to load only certain entries to load page properly for all users.

But I'm looking for monitoring connection pool behavior of Liferay while communicating with S3 as like database connection pool monitoring (Visual VM), do we have any way to do this? Please advise.

[Andrew Jardine, modifié il y a 6 années.]

Hi Sai,

I can see in the configuration class for the S3 store several options around the connection pools / sizes.

	@Meta.AD(
		deflt = "50", description = "http-client-max-connections-help",
		name = "http-client-max-connections", required = false
	)
	public int httpClientMaxConnections();

	@Meta.AD(
		deflt = "5", description = "http-client-max-error-retry-help",
		name = "http-client-max-error-retry", required = false
	)
	public int httpClientMaxErrorRetry();

	@Meta.AD(
		deflt = "7", description = "core-pool-size-help",
		name = "core-pool-size", required = false
	)
	public int corePoolSize();

	@Meta.AD(
		deflt = "20", description = "max-pool-size-help",
		name = "max-pool-size", required = false
	)
	public int maxPoolSize();

.. so I would guess that you should be able to see the details in VisualVM. Have you checked?