Liferay hanging while starting up

[Om P Singh，修改在14 年前。]

Hello,

We have recently done a liferay implementation and are facing a very serious issue on production.

1. Liferay version - 5.1.2 [ standard edition]
2. Database - MySQL
3. Container - Tomcat 5.5.X

Liferay is deployed in a cluster. [ two node cluster ], using Jackrabbit and SOlr as well

Every time we are starting up the server the server hangs while starting up, last time in fact it took 10 minutes for the server to start up.

The thread dump shows this

"main" prio=10 tid=0x09576000 nid=0x423 runnable [0xb7fc9000..0xb7fcc1f8]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)
        at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)
        at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)
        - locked <0x756a4658> (a com.mysql.jdbc.util.ReadAheadInputStream)
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1994)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)
        - locked <0x756a4920> (a java.lang.Object)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1585)
        - locked <0x756a4920> (a java.lang.Object)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1500)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1485)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
        at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.deleteVolatileFiredTriggers(StdJDBCDelegate.java:3210)
        at com.liferay.portal.scheduler.quartz.DynamicDriverDelegate.deleteVolatileFiredTriggers(DynamicDriverDelegate.java:226)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.cleanVolatileTriggerAndJobs(JobStoreSupport.java:733)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$1.execute(JobStoreSupport.java:697)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3628)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3662)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3624)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.cleanVolatileTriggerAndJobs(JobStoreSupport.java:693)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.initialize(JobStoreSupport.java:555)
        at org.quartz.impl.jdbcjobstore.JobStoreTX.initialize(JobStoreTX.java:59)
        at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:1204)
        at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1355)
        at com.liferay.portal.scheduler.quartz.QuartzSchedulerEngineImpl.<init>(QuartzSchedulerEngineImpl.java:84)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:100)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:61)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:877)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:839)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
        - locked &lt;0x75445e40&gt; (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:479)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:162)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:925)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:835)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221)
        - locked &lt;0x75445e40&gt; (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
        - locked &lt;0x753e7f08&gt; (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
        - locked &lt;0x753e7fa0&gt; (a java.lang.Object)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)
        at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:49)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3764)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4216)
        - locked &lt;0x74fe1b40&gt; (a org.apache.catalina.core.StandardContext)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760)
        - locked &lt;0x74fe1a78&gt; (a java.util.HashMap)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:740)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:626)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1149)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022)
        - locked &lt;0x74fe19e8&gt; (a org.apache.catalina.core.StandardHost)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:736)
        - locked &lt;0x74fe19e8&gt; (a org.apache.catalina.core.StandardHost)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014)
        - locked &lt;0x7500bf18&gt; (a org.apache.catalina.core.StandardEngine)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
        at org.apache.catalina.core.StandardService.start(StandardService.java:448)
        - locked &lt;0x7500bf18&gt; (a org.apache.catalina.core.StandardEngine)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
        - locked &lt;0x75047b80&gt; (a [Lorg.apache.catalina.Service;)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
</init>

I have attached the thread dumps from both the server along with this ticket

Any help or suggestion will be really appreciated.

[Lisa Simpson，修改在14 年前。]

Well, I see one thing that jumps out at me right away... Lines 8 and 9 indicate that there is some locking going on with your database. During start up, that's not usually a good thing.

Go find your my.cnf for mysql and set the following:

key_buffer should be at least 10% of your available physical RAM
max_allowed_packet should be 20% of key_buffer
max_connections should be equal to what tomcat is set to
table_cache = 64

restart mysql and restart Liferay and see if that doesn't help you.

[Om P Singh，修改在14 年前。]

thanks lisa,

Sorry for updating so late on this forum.

It did improve the performance

Regards
Om