Clustering - Quartz

[7年前 に Sebastian Ulrich によって更新されました。]

Hello,

We had a problem with the Quartz clustering functionality. at some point (not since the beginning, it started going bad Sunday morning at 5am) the errors below occurred.
We have now disabled the clustering of Quartz, and we have two questions :
- why does this error occur ?
- are there any potential issues with disabling this functionality ?

Thanks forward

04:54:35,310 ERROR [QuartzScheduler_PersistedQuartzSchedulerEngineInstance-fr0-vsiaas-22941464958288839_ClusterManager][PortalJobStore:120] Couldn't rollback jdbc connection. Communications link failure during rollback(). Transaction resolution unknown.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
at com.mysql.jdbc.Util.getInstance(Util.java:383)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1023)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:997)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:983)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:928)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5097)
at org.apache.tomcat.dbcp.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:491)
at org.apache.tomcat.dbcp.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:491)
at sun.reflect.GeneratedMethodAccessor1202.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.rollback(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1202.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73)
at com.sun.proxy.$Proxy317.rollback(Unknown Source)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.rollbackConnection(JobStoreSupport.java:3652)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3277)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3870)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3907)

04:54:35,311 ERROR [QuartzScheduler_PersistedQuartzSchedulerEngineInstance-fr0-vsiaas-22941464958288839_ClusterManager][PortalJobStore:120] ClusterManager: Error managing cluster: Couldn't commit jdbc connection. Deadlock found when trying to get lock; try restarting transaction
org.quartz.JobPersistenceException: Couldn't commit jdbc connection. Deadlock found when trying to get lock; try restarting transaction [See nested exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3674)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3254)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3870)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3907)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
at com.mysql.jdbc.Util.getInstance(Util.java:383)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4226)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4158)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834)
at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1751)
at org.apache.tomcat.dbcp.dbcp2.DelegatingConnection.commit(DelegatingConnection.java:367)
at org.apache.tomcat.dbcp.dbcp2.DelegatingConnection.commit(DelegatingConnection.java:367)
at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.commit(Unknown Source)
at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73)
at com.sun.proxy.$Proxy317.commit(Unknown Source)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3672)
... 3 more

[7年前 に Olaf Kock によって更新されました。]

Sebastian Ulrich:

We had a problem with the Quartz clustering functionality. at some point (not since the beginning, it started going bad Sunday morning at 5am) the errors below occurred.
We have now disabled the clustering of Quartz, and we have two questions :
- why does this error occur ?
- are there any potential issues with disabling this functionality ?

• Which version are you running?
• What happens around the time that this error shows up first?
• How does the database look like - do you have enough resources, connections etc. for it?

What can happen if you are running a cluster but don't run Quartz in cluster mode? You might run scheduled jobs at the same time on all machines. If you're lucky This is only bad for performance. But it might also lead to a concentration of race conditions, concurrent locks etc as the same jobs are firing at roughly the same time. Typically not what you want.