Problem just after getting into production

[Alfredo del Castillo, modificado hace 12 años.]

After nearly a year of development, my client has decided to upload the new Intranet to production, and the results have been disappointing.

We have 2 machines with "Liferay 6.0.5," working in a cluster on an Oracle database. The problem is that a seemingly random way, users can not access the server, getting a 503 error from apache that is above every tomcat.

The error, which varies but is usually the version "NoSuchFileException" makes any community page is not loading so much to look, we were unable to find the fault. Here are some examples:

13:17:44,956 ERROR [jsp:686] User ID 301309
13:17:44,964 ERROR [jsp:687] Current URL /c/portal/login
13:17:44,965 ERROR [jsp:688] Referer null
13:17:44,967 ERROR [jsp:689] Remote address 10.47.32.136
13:17:44,968 ERROR [jsp:691] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portal.kernel.exception.SystemException: javax.jcr.InvalidItemStateException: Item cannot be saved because it has been modified externally: node /
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portal.kernel.exception.SystemException: javax.jcr.InvalidItemStateException: Item cannot be saved because it has been modified externally: node /
	at com.liferay.portal.events.ServicePreAction.run(ServicePreAction.java:147)
--
13:28:21,641 ERROR [jsp:686] User ID 273937
13:28:21,643 ERROR [jsp:687] Current URL /c/portal/login
13:28:21,645 ERROR [jsp:688] Referer null
13:28:21,646 ERROR [jsp:689] Remote address 10.47.5.189
13:28:21,648 ERROR [jsp:691] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portal.events.ServicePreAction.run(ServicePreAction.java:147)
--
13:46:04,153 ERROR [jsp:686] User ID 243862
13:46:04,155 ERROR [jsp:687] Current URL /c/portal/login
13:46:04,156 ERROR [jsp:688] Referer null
13:46:04,158 ERROR [jsp:689] Remote address 10.1.38.70
13:46:04,159 ERROR [jsp:691] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portal.events.ServicePreAction.run(ServicePreAction.java:147)
--
13:46:08,918 ERROR [jsp:686] User ID 243862
13:46:08,920 ERROR [jsp:687] Current URL /c/portal/login
13:46:08,922 ERROR [jsp:688] Referer null
13:46:08,923 ERROR [jsp:689] Remote address 10.1.38.70
13:46:08,924 ERROR [jsp:691] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
	at com.liferay.portal.events.ServicePreAction.run(ServicePreAction.java:147)

As you will see this does not give too much information. The error appears in the class:

com.liferay.portal.struts.PortalRequestProcessor - Line 459

Do you have any idea? Thanks in advance

[Sandeep Nair, modificado hace 12 años.]

You are using Jackrabbit to save files in database?

Regards,
Sandeep

[Alfredo del Castillo, modificado hace 12 años.]

Sandeep Nair:

You are using Jackrabbit to save files in database?

Regards,
Sandeep

Yes, this is from my portal-ext.properties:

 # Set the name of a class that implements
    # com.liferay.documentlibrary.util.Hook. The
    # document library server will use this persist documents.
    #
    #dl.hook.impl=com.liferay.documentlibrary.util.AdvancedFileSystemHook
    #dl.hook.impl=com.liferay.documentlibrary.util.CMISHook
    #dl.hook.impl=com.liferay.documentlibrary.util.FileSystemHook
    dl.hook.impl=com.liferay.documentlibrary.util.JCRHook
    #dl.hook.impl=com.liferay.documentlibrary.util.S3Hook
	
	#
    # JCRHook
    #
    dl.hook.jcr.fetch.delay=500
    dl.hook.jcr.fetch.max.failures=5

   image.hook.impl=com.liferay.portal.image.DLHook

[Sandeep Nair, modificado hace 12 años.]

Can you please attach repository.xml of both the nodes? Any reason why you are not using AdvancedFileSystemHook?

Regards,
Sandeep

[Alfredo del Castillo, modificado hace 12 años.]

Sure, thanks for the answers, this is my repository.xml from Jackrabbit directory:

<!--?xml version="1.0"?-->
<repository>
	<filesystem class="org.apache.jackrabbit.core.fs.db.OracleFileSystem">
		<param name="driver" value="oracle.jdbc.driver.OracleDriver">
                <param name="url" value="jdbc:oracle:thin:@********************************">
                <param name="user" value="********************************">
                <param name="password" value="********************************">
                <param name="schema" value="oracle">
		<param name="schemaObjectPrefix" value="J_R_FS_">
	</filesystem>

	<security appname="Jackrabbit">
		<accessmanager class="org.apache.jackrabbit.core.security.SimpleAccessManager" />
		<loginmodule class="org.apache.jackrabbit.core.security.SimpleLoginModule">
			<param name="anonymousId" value="anonymous">
		</loginmodule>
	</security>
	<workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="liferay" />
	<workspace name="${wsp.name}">
		<persistencemanager class="org.apache.jackrabbit.core.persistence.db.OraclePersistenceManager">
			<param name="driver" value="oracle.jdbc.driver.OracleDriver">
	                <param name="url" value="jdbc:oracle:thin:@********************************">
	                <param name="user" value="********************************">
	                <param name="password" value="********************************">
	                <param name="schema" value="oracle">
	                <param name="tableSpace" value="">
			<param name="schemaObjectPrefix" value="J_PM_${wsp.name}_">
			<param name="externalBLOBs" value="false">
		</persistencemanager>
		<filesystem class="org.apache.jackrabbit.core.fs.db.OracleFileSystem">
			<param name="driver" value="oracle.jdbc.driver.OracleDriver">
	                <param name="url" value="jdbc:oracle:thin:@********************************">
	                <param name="user" value="********************************">
	                <param name="password" value="********************************">
	                <param name="schema" value="oracle">
			<param name="schemaObjectPrefix" value="J_FS_${wsp.name}_">
		</filesystem>
	</workspace>
	<versioning rootpath="${rep.home}/version">
		<filesystem class="org.apache.jackrabbit.core.fs.db.OracleFileSystem">
			<param name="driver" value="oracle.jdbc.driver.OracleDriver">
	                <param name="url" value="jdbc:oracle:thin:@********************************">
	                <param name="user" value="********************************">
	                <param name="password" value="********************************">
	                <param name="schema" value="oracle">
			<param name="schemaObjectPrefix" value="J_V_FS_">
		</filesystem>
		<persistencemanager class="org.apache.jackrabbit.core.persistence.db.OraclePersistenceManager">
			<param name="driver" value="oracle.jdbc.driver.OracleDriver">
	                <param name="url" value="jdbc:oracle:thin:@********************************">
	    		<param name="user" value="********************************">
	                <param name="password" value="********************************">
	                <param name="tableSpace" value="">
			<param name="schemaObjectPrefix" value="J_V_PM_">
			<param name="schema" value="oracle">
			<param name="externalBLOBs" value="false">
		</persistencemanager>
	</versioning>

	<!--
	Cluster Configuration
	-->

	<cluster id="intranet_bundle" syncdelay="5">
		<journal class="org.apache.jackrabbit.core.journal.OracleDatabaseJournal">
			<param name="revision" value="${rep.home}/revision.log">
			<param name="driver" value="oracle.jdbc.driver.OracleDriver">
	                <param name="url" value="jdbc:oracle:thin:@********************************">
	                <param name="user" value="********************************">
	                <param name="password" value="********************************">
	                <param name="schema" value="oracle">
			<param name="schemaObjectPrefix" value="J_C_">
		</journal>
	</cluster>
</repository>

About why we aren't using AdvancedFileSystemHook, the answer is simple, with two nodes in the cluster, physical access to a repository would give us files concurrency issues. When taken into the oracle database these problems go away, why you see this wrong?

[Sandeep Nair, modificado hace 12 años.]

Just to confirm are both repository.xml identical? If yes there is a part at the end where you have to change the node id. (I have marked it in bold). For both nodes the id should be different.

 <cluster[b] id="intranet_bundle[/b]" syncdelay="5">
        <journal class="org.apache.jackrabbit.core.journal.OracleDatabaseJournal">
            <param name="revision" value="${rep.home}/revision.log">
            <param name="driver" value="oracle.jdbc.driver.OracleDriver">
                    <param name="url" value="jdbc:oracle:thin:@********************************">
                    <param name="user" value="********************************">
                    <param name="password" value="********************************">
                    <param name="schema" value="oracle">
            <param name="schemaObjectPrefix" value="J_C_">
        </journal>
    </cluster[b]>

More on JCR configuration and why AdvancedFileSystemHook(we normally recommend this on SAN and never run into concurrancy issue). Please see the last section

Regards,
Sandeep

[Alfredo del Castillo, modificado hace 12 años.]

Now we have the exactly xml on both nodes, with the same cluster id, I'm going to change it and hope that was the problem...

Anyway, thanks for the link, we will study it and see what happen.

Thanks again,

[Alfredo del Castillo, modificado hace 12 años.]

Bad move, after changing the cluster id on one node, I cannot download any documents. This is what I tried:

upload and link a document in web content, all of this in [node 1]
publish web content
try to download on [node 2]
Result:

16:47:12,521 INFO  [PortalImpl:3832] Current URL /c/document_library/get_file?uuid=e6c9fbdc-a163-45e1-a1d0-bf49508e04dc&groupId=10156 generates exception: null
16:47:12,958 ERROR [jsp:998] java.lang.NullPointerException
java.lang.NullPointerException
	at com.liferay.portlet.documentlibrary.action.GetFileAction.getFile(GetFileAction.java:202)

[Hitoshi Ozawa, modificado hace 12 años.]

When clustering, there are 3 things that needs to be set to be common.
1. Liferay database
2. Indexing - default to use Lucene
3. Document/Image library

What is your system configuration on these? Are you using a single cluster database? Are you using external Solr? Are you using single external JCR?

[Alfredo del Castillo, modificado hace 12 años.]

Hi Hitoshi, thanks for your answer, this is my cluster configuration:

1. we are using an oracle database, and JCRHook for store there the documents, anythig related to jackrabbit actually.
2. We use the default Lucene indexing service, with JChannel to get each node the correct indexes updated.
3. As I said before, we use the JCRHook for this.

As for your answer, Sandeep Nair, we have the same portal-ext.properties, both of them using JCRHook...

And today we keep getting errors like this:

09:19:31,016 ERROR [jsp:686] User ID 255834
09:19:31,017 ERROR [jsp:687] Current URL /c/portal/login
09:19:31,018 ERROR [jsp:688] Referer null
09:19:31,019 ERROR [jsp:689] Remote address 10.1.40.31
09:19:31,020 ERROR [jsp:691] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
[...]
Caused by: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portal.lar.PortletImporter.importPortletData(PortletImporter.java:649)
[...]
Caused by: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portal.kernel.lar.BasePortletDataHandler.importData(BasePortletDataHandler.java:67)
[...]
Caused by: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portlet.documentlibrary.service.impl.DLFileVersionLocalServiceImpl.getLatestFileVersion(DLFileVersionLocalServiceImpl.java:71)
	at sun.reflect.GeneratedMethodAccessor983.invoke(Unknown Source)
[...]
Caused by: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portal.lar.PortletImporter.importPortletData(PortletImporter.java:649)
	at com.liferay.portal.lar.PortletImporter.importPortletData(PortletImporter.java:545)
[...]
Caused by: com.liferay.portal.kernel.lar.PortletDataException: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portal.kernel.lar.BasePortletDataHandler.importData(BasePortletDataHandler.java:67)
	at com.liferay.portal.lar.PortletImporter.importPortletData(PortletImporter.java:642)
	... 123 more
Caused by: com.liferay.portlet.documentlibrary.NoSuchFileVersionException
	at com.liferay.portlet.documentlibrary.service.impl.DLFileVersionLocalServiceImpl.getLatestFileVersion(DLFileVersionLocalServiceImpl.java:71)

[Arno Broekhof, modificado hace 12 años.]

It looks like an out of sync JCR, i also advice on using AdvancedFileSystemHook are you running on windows or linux?

What setup did you use for the AdvancedFileSystemHook ( what kind of File Sharing protocol ) i have used it with GFS and NFS en never had problems with it.

[Sandeep Nair, modificado hace 12 años.]

Thats really bad,

I do hope you were doing that in a Staging environment which mimics the production one. I had configured JCR as mentioned in the wiki and it was working properly. Though i dont use it any more because of performance issue and AdvancedFileSystem is much more easier to setup and configure and works very well performance wise.
I believe both or ur portal-ext.properties are identical and that one is not pointing to jcr and other one is using default(filesystemhook)

Regards,
Sandeep