Connection object is closed - Database error on CentOS 7

[Peter Helgren, modifié il y a 6 années.]

Not exactly sure why I am getting this error. I have an installation on LR 7 GA4 CE on Windows 10 which is my development machine. Our pre-production server is running the same version of Liferay, connecting to the same database (MS SQL using JTDS driver plus the SQL Server plugin that Antonio Musarra wrote). Except for the OS, I think the installations are identical but because they were also set up independently it IS possible I missed something.

I checked the firewall to make sure the server was accessible. I reviewed the server.xml, context.xml, portal-ext.properties files to make sure that they matched between the Windows 10 development machine and our pre-production server. It all looks the same to these eyes and yet every portal that has a database component (though a service builder module) experiences the exact same error listed here below:

12:47:35,966 ERROR [http-nio-8080-exec-2][JDBCExceptionReporter:234] Invalid state, the Connection object is closed.
12:47:35,968 ERROR [http-nio-8080-exec-2][JDBCTransaction:228] Could not toggle autocommit
java.sql.SQLException: Invalid state, the Connection object is closed.
        at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1699)
        at net.sourceforge.jtds.jdbc.ConnectionJDBC2.setAutoCommit(ConnectionJDBC2.java:2212)
        at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
        at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
        at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)
        at com.sun.proxy.$Proxy476.setAutoCommit(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
        at com.sun.proxy.$Proxy7.setAutoCommit(Unknown Source)
        at org.hibernate.transaction.JDBCTransaction.toggleAutoCommit(JDBCTransaction.java:224)
        at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:216)
        at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:192)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:682)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:849)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:826)
        at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.rollback(DefaultTransactionExecutor.java:79)
        at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.execute(DefaultTransactionExecutor.java:57)
        at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:58)
        at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:137)
        at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
        at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:137)
        at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:169)
        at com.sun.proxy.$Proxy482.findLanguageByLocaleCd(Unknown Source)

It is the "autocommit" references that have me scratching my head along with the "rollback". This may be normal behavior when a connection attempt fails but I don't see any of this on my development machine. Very strange. Extensive searching and reading hasn't turned up anything useful so far. I'd be grateful for ANY suggestions no matter how far-fetched they would seem.....

[David H Nebinger, modifié il y a 6 années.]

Peter Helgren:

MS SQL using JTDS driver plus the SQL Server plugin that Antonio Musarra wrote

You'll really need to reach out to Antonio for help on that since it's his connector.

Clearly something has closed the underlying connection w/o going through the proxy returned by the pool implementation, but it's hard to say where that might originate.

Our pre-production server is running the same version of Liferay, connecting to the same database

Do you mean the exact same database, or just both are using SQL server? Because when you point at the same database, you're basically defining a cluster and that is not going to fly for LR7CE, and even on LR6 you would only have a partial, broken cluster...







Come meet me at Devcon 2017 or 2017 LSNA!

[Peter Helgren, modifié il y a 6 années.]

Thanks.

I'll ping Antonio although I am also going to try to run a CentOS 7 VM and see if the same weirdness occurs there.

Do you mean the exact same database, or just both are using SQL server? Because when you point at the same database, you're basically defining a cluster and that is not going to fly for LR7CE, and even on LR6 you would only have a partial, broken cluster...

There is one SQL server instance designated for Pre-Production and all our preproduction LR instances connect to it for testing. This hasn't been an issue for our 6.0.6 instances and right now my development box and a LR 6.0.6 instance are both connected without any errors in either environment.

Just to fill in whatever blanks there might be...our Liferay instance is about 95% "select only" on the MS SQL server instance. That is, we use the MSSQL server DB to retrieve and display data, we have other non-LR apps that maintain the data. And again, its the same server that our multiple LR 6.0.6 instances connect to as well as my LR 7.0 development instance connection to so I don't think there is any access contention going on. Oh, and as a quick addition: This is ONLY for our MSSQL database and ONLY used by SB modules for getting "external" data. Liferay's database is still the "vanilla" MySQL/MariaDB that ships out of the box...

Lemme do some more poking around.

[David H Nebinger, modifié il y a 6 années.]

Peter Helgren:

There is one SQL server instance designated for Pre-Production and all our preproduction LR instances connect to it for testing. This hasn't been an issue for our 6.0.6 instances and right now my development box and a LR 6.0.6 instance are both connected without any errors in either environment.

Okay, so there's a couple of problems with this:

• Unless you are storing docs in the DB, they go to the data folder and each preprod instance will have it's own data folder, and therefore docs will be completely out of sync.
• When you make DB changes, in a cluster the node making the change will message the others. This allows each node to purge the cache (to prevent serving stale data) and update the search indices (since those are normally hit before the db).

As these are separate nodes not really in a proper cluster, all kinds of things are going to be out of whack. I'd be surprised if one could test anything with so much potential to be out of sync.

Just to fill in whatever blanks there might be...our Liferay instance is about 95% "select only" on the MS SQL server instance. That is, we use the MSSQL server DB to retrieve and display data, we have other non-LR apps that maintain the data. And again, its the same server that our multiple LR 6.0.6 instances connect to as well as my LR 7.0 development instance connection to so I don't think there is any access contention going on. Oh, and as a quick addition: This is ONLY for our MSSQL database and ONLY used by SB modules for getting "external" data. Liferay's database is still the "vanilla" MySQL/MariaDB that ships out of the box...

Okay, well that makes all the difference. All of your nodes are standalone nodes, they each have their own liferay database and are not a broken cluster.

Yes, I got a little verbose here, but I thought it was important to drag out for those that happen upon this thread...







Come meet me at Devcon 2017 or 2017 LSNA!

[Peter Helgren, modifié il y a 6 années.]

So, finally sorted this out with the help of Rackspace. I had suspected a network issue but originally the network guys waved it off as an SQL problem. After an hour of working with the SQL folks, they passed me back to the network folks (ping...pong). The bottom line was that the routing on the Linux VM was incorrect. There were many red herring on this that made it *seem* like it was a DB issue but the issue was network.

Running fine now...