How do I cluster Liferay with Terracotta?

That's a question I've head many times and in this post I will show you  just how to do that. These instructions are for Liferay 6 CE GA3 Tomcat 6.0 bundle however you can use any app server supported by Terracotta but the location and some configuration might be slightly different. So to get started you need to download:

Next step is to install Liferay and Terracotta. For the purposes of this post I won't go into great detail with the installation as both Terracotta and Liferay has good documentation. Basically the installation consist of unpacking the packages to a directory. From now on I will refer to those locations as LIFERAY_HOME and TERRACOTTA_HOME and inside LIFERAY_HOME we will have tomcat directory which I will refer as TOMCAT_HOME. Normally you would also install Liferay and Terracotta in separate servers but I will post a separate post addressing the recommended architecture. For now we can install everything on the same machine and run Terracotta with default configuration for development purposes.

Normally when clustering Liferay you need to address following components: EhCache and Hibernate, Quartz Scheduler, Document Library, Search Engine and optionally Session Replication. For Document Library and Search Engine Terracotta doesn't offer anything new so you make those centrally available the same way as before. For example SAN for DL and SOLR for Search and Indexing. So we are left with EhCache and Hibernate, Quartz and Session Replication that we can address with Terracotta. 

EhCache and Hibernate Second Level Cache

  1.  Remove ehcache.jar that is bundled with Liferay (located in TOMCAT_HOME/webapps/ROOT/WEB-INF/lib)
  2. Copy all jars in TERRACOTTA_HOME/ehcache/lib to TOMCAT_HOME/webapps/ROOT/WEB-INF/lib
  3. Copy TERRACOTTA_HOME/common/terracotta-toolkit-1.0-runtime-<version>.jar to TOMCAT_HOME/webapps/ROOT/WEB-INF/lib
  4. Create my-ehcache folder to TOMCAT_HOME/webapps/ROOT/WEB-INF/classes
  5. Create a hibernate-terracotta.xml and a liferay-multi-vm-terracotta.xml.
  6. Adjust terracottaConfig in hibernate-terracotta.xml and liferay-multi-vm-terracotta.xml to point to your Terracotta servers. Like this: <terracottaConfig url="localhost:9510"/>
  7. Add following properties to your portal-ext.properties file:
    ehcache.multi.vm.config.location=/my-ehcache/liferay-multi-vm-terracotta.xml
    
net.sf.ehcache.configurationResourceName=/my-ehcache/hibernate-terracotta.xml
    
hibernate.cache.region.factory_class=net.sf.ehcache.hibernate.EhCacheRegionFactory

Quartz

  1. Remove quartz.jar that is bundled with Liferay (located in TOMCAT_HOME/webapps/ROOT/WEB-INF/lib)
  2. Copy TERRACOTTA_HOME/quartz/quartz-terracotta-<version>.jar and quartz-all-<version>.jar to TOMCAT_HOME/webapps/ROOT/WEB-INF/lib
  3. Add following properties to your portal-ext.properties:
    org.quartz.jobStore.class = org.terracotta.quartz.TerracottaJobStore
    
org.quartz.jobStore.tcConfigUrl = localhost:9510
  4. Extract portal.properties from portal-impl.jar and place it in TOMCAT_HOME/webapps/ROOT/WEB-INF/classes
  5. Comment out following properties in portal.properties
    #org.quartz.jobStore.dataSource=ds

    #org.quartz.jobStore.isClustered=false
    
#org.quartz.jobStore.misfireThreshold=60000
    
#org.quartz.jobStore.tablePrefix=QUARTZ_
    
#org.quartz.jobStore.useProperties=false

Session Replication

This is highly container specific so refer to Terracotta documentation for specific instructions. Following steps are for Tomcat 6.0.

  1. Copy TERRACOTTA_HOME/sessions/terracotta-session-<version>.jar to TOMCAT_HOME/lib
  2. Copy TERRACOTTA_HOME/common/terracotta-toolkit-1.0-runtime-<version>.jar to TOMCAT_HOME/lib
  3. Edit TOMCAT_HOME/conf/Catalina/localhost/ROOT.xml and add following line right after <Context>
    
<Valve className="org.terracotta.session.TerracottaTomcat60xSessionValve" tcConfigUrl="localhost:9510"/>

Testing The Configuration

Testing your configuration is simple:

  1. Startup your Terracotta Server
    TERRACOTTA_HOME/bin/start-tc-server.sh
  2. Startup your Tomcat
    TOMCAT_HOME/bin/startup.sh
  3. Before Tomcat has fully started you should see following lines in your Tomcat console log:
    2010-09-01 21:35:40,059 INFO - Terracotta 3.3.0, as of 20100716-140706 (Revision 15922 by cruise@rh5mo0 from 3.3)
    2010-09-01 21:35:40,566 INFO - Successfully loaded base configuration from server at 'localhost:9510'.
  4. Now browse http://localhost:8080 to verify that your portal is running. 
  5. Now launch Terracotta Developer Console to verify that EhCache, Hibernate, Quartz and Sessions are seen by Terracotta. You can launch dev console with following command:
    TERRACOTTA_HOME/bin/dev-console.sh
  6. Once you are connected to your Terracotta you should see Ehcache, Hibernate, Quartz and Sessions under My application which indicates that all of them are connected and recognized by Terracotta. Now you can use Dev Console to see what's inside your cache or session. 

Closing Remarks

Now as you can see it is quite easy the cluster Liferay with Terracotta express installation. Now if you want to use the DSO approach it is whole another beast as it involves tedious instrumentation. If you are a Liferay EE customer and want to get supported version of both Liferay and Terracotta contact your Liferay sales rep and ask about Liferay Terracotta Edition. 

Blogues
Great post but ... I ran into a problem that hopefully you help with.

Followed instructions as above except I have the terracotta server running on a different box. Made the ip address changes as appropriate and created the tc-config.xml file on the terracotta server.

Now when I fire up Liferay, everything looks good through the terracotta startup and then I start getting exceptions:

2010-09-02 18:52:11,606 INFO - Terracotta 3.3.0, as of 20100716-140706 (Revision 15922 by cruise@rh5mo0 from 3.3)
2010-09-02 18:52:12,034 INFO - Successfully loaded base configuration from server at '10.109.206.10:9510'.
2010-09-02 18:52:12,080 INFO - Successfully loaded base configuration from file at '/usr/local/liferay/liferay-portal-6.0.5/tomcat-6.0.26/temp/tc-config4088533534494940638.xml'.
2010-09-02 18:52:12,208 INFO - Log file: '/root/terracotta/client-logs/terracotta-client.log'.
2010-09-02 18:52:13,956 INFO - Connection successfully established to server at 10.109.206.10:9510
18:52:40,171 INFO [PortalImpl:277] Global lib directory /usr/local/liferay/liferay-portal-6.0.5/tomcat-6.0.26/lib/ext/
18:52:40,175 INFO [PortalImpl:297] Portal lib directory /usr/local/liferay/liferay-portal-6.0.5/tomcat-6.0.26/webapps/ROOT/WEB-INF/lib/
Starting Liferay Portal Community Edition 6.0.5 CE (Bunyan / Build 6005 / August 16, 2010)
18:52:55,917 ERROR [velocity:151] ResourceManager.getResource() load exception
18:52:55,919 ERROR [velocity:151] net.sf.ehcache.CacheException: java.io.NotSerializableException: org.apache.velocity.Template
at org.terracotta.modules.ehcache.store.ValueModeHandlerSerialization.createTimestampedValue(ValueModeHandlerSerialization.java:56)

etc..... any suggestions?
Looks like you configured ehcache.single.vm.config.location=/ehcache/liferay-single-vm.xml to use Terracotta. This is definite no no! Single VM cache is not distributable as the objects are not Serializable.
Hello again Mika. I'm sorry to ask this question again but I have followed these instructions to the letter with two completely fresh installs and I'm still getting the same errors I indicated earlier. I agree with your assessment but as far as I can tell, I haven't done anything to tell single-vm cache to use Terracotta. Can you give me some more guidance on how this might have happened and how to isolate it as it seems improbable that this would happen without me messing it up somehow but I can't figure out how? To be clear this is a fresh install and the only things in portal-ext.properties are the mysql parameters and the entries from your post. The my-ehcache folder only contains the two files downloaded from your post with the terracottaConfig url parameter updated so.
Hello do you successfully run clustered liferay portal 6 without terracotta but with "basic" configuration of ehcache and jgroups ?
So the first time around I had a working ehcache cluster for sessions and document library when Mika's posting came out so I converted that and ran into the problem I originally posted. After Mika's response, I assumed there might be some complicating factor from my original config so I created new installs from scratch but ran into the same issue. The common element in both configurations was the Terracotta server.
Have you done this setup with everything on one physical server or do you have separate servers for each Liferay and Terracotta? I will do some experimenting to try to reproduce that issue.
These were separate servers - two Centos/Tomcat/Liferay - one Centos/Terracotta.
James, did you manage to resolve the java.io.NotSerializableException issue ? I have been trying various configurations without success and I am approaching the point where I will need to choose a different solution.

Any feedback would be useful. Thanks, Stuart
No. I still haven't resolved it and have moved onto other parts of my configuration hoping to get an update from Mika's test in the interim. Hoping to see an update soon as I am sure this must be something simple in the cache configuration and I too need to get this working or choose an alternative. If I uncover anything new I will post it.
Hey Guys, sorry it took so long but I've now been able to reproduce your issue. It's caused by the use of MultiVMPoolUtil in com.liferay.portal.velocity.LiferayResourceCacheUtil. If you switch it to SingleVMPoolUtil you should be able to get past that error. Now this will reintroduce the issue described in http://issues.liferay.com/browse/LPS-8105.
Thanks for looking into this Mika.

To make sure I'm understanding this correctly. I run an 'out-of-the-box' Liferay instalation so I can't use Terracotta with Liferay 6 unless I customize the portal core as suggested above and run it in the EXT environment. Is that right?
I am also experiencing the same problem. I orginally had timeout issues with Terracotta which I resolved with Mika's help.

I have tried this with both Liferay and Terracotta running both on a single host and have tried with Terracotta server on a separate host with the same results. The output from catalina.out shows :
Starting Liferay Portal Community Edition 6.0.5 CE (Bunyan / Build 6005 / August 16, 2010)
09:54:22,225 ERROR [velocity:151] ResourceManager.getResource() load exception
09:54:22,227 ERROR [velocity:151] net.sf.ehcache.CacheException: java.io.NotSerializableException: org.apache.velocity.Template
at org.terracotta.modules.ehcache.store.ValueModeHandlerSerialization.createTimestampedValue(ValueModeHandlerSerialization.java:56)
etc.......
You write "Extract portal.properties from portal-impl.jar and place it in TOMCAT_HOME/webapps/ROOT/WEB-INF/classes Comment out following properties in portal.properties"

So just to confirm. there is something that normal portal-ext.properties override does not work?
Yeah, that is correct. You must disable those properties and you can't do that in portal-ext.properties. When using TerracottaJobStore it will throw errors if those properties are present.
Thanks for the excellent post.

I have come across a problem where the Terracotta client log shows:


2010-09-09 14:14:29,874 [ConnectionEstablisher] WARN com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(1.50c199aa5ce84e0ca28d03b4e6e5d99d): Timeout of 10000 milliseconds occured [lossy interval: 10000ms]
2010-09-09 14:14:29,907 [L1_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:53896)] WARN com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(1.50c199aa5ce84e0ca28d03b4e6e5d99d): CLOSE EVENT : com.tc.net.core.TCConnectionJDK14@775009342: connected: false, closed: true local=127.0.0.1:60631 remote=127.0.0.1:9510 connect=[Thu Sep 09 14:14:19 GMT 2010] idle=10015ms [0 read, 84 write]. STATUS : START
2010-09-09 14:14:29,908 [L1_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:53896)] WARN com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(1.50c199aa5ce84e0ca28d03b4e6e5d99d): closing down connection - com.tc.net.core.TCConnectionJDK14@775009342: connected: false, closed: true local=127.0.0.1:60631 remote=127.0.0.1:9510 connect=[Thu Sep 09 14:14:19 GMT 2010] idle=10038ms [0 read, 84 write]
2010-09-09 14:14:30,881 [ConnectionEstablisher] INFO com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(1.50c199aa5ce84e0ca28d03b4e6e5d99d): Attaching new connection: com.tc.net.core.TCConnectionJDK14@425034011: connected: true, closed: false local=127.0.0.1:60633 remote=127.0.0.1:9510 connect=[Thu Sep 09 14:14:30 GMT 2010] idle=5ms [0 read, 0 write]

and the Terracotta console show:

2010-09-09 15:23:22,431 INFO - "Client Cannot Reconnect ConnectionID(0.50c199aa5ce84e0ca28d03b4e6e5d99d) not found. Connection attempts from the Terracotta node at 127.0.0.1:48718 are being rejected by the Terracotta server array. Restart the Client to allow it to rejoin the cluster. Many Client reconnection failures can be avoided by configuring the Terracotta server array for "permanent-store" and tuning reconnection parameters. For more information, see http://www.terracotta.org/kit/reflector?kitID=default&pageID=HA"

I have tried it with persistence set to "permanent-store" and it made no difference.

Any advice would be much appreciated.
Hmm. I've usually seen this if Terracotta server is restarted while Liferay is running. Like the error message suggest you might need to tune the reconnection timeout settings. If you have not restarted your Terracotta server then there might have been some sort of network outage. If the problem persists I would ask help from Terracotta forums describing your setup and providing your tc-config.xml.
Mika, Thanks for your advice.

I have managed to address the problem by adding the following into tc-properties in the terracotta config:

<property name="l2.l1reconnect.enabled" value="true" />
<property name="l2.l1reconnect.timeout.millis" value="600000" />

Unfortunatly I am now experiencing the same problem as James Denmark below. I will add to that issue next.

Thanks again Stuart
I can't get this to work. Using a plain liferay-tomcat-6.0.5 bundle with mysql and the latest build of terracotta. I don't know if we have to configure terracotta but we have just started as described above and it seems to start ok but when we start liferay we keep getting the following in the logs.

2011-02-08 10:42:03,847 INFO - Terracotta 3.4.1, as of 20110130-210115 (Revision 17072 by cruise@su10mo4 from 3.4)
2011-02-08 10:42:04,716 INFO - Successfully loaded base configuration from server at 'localhost:9510'.
2011-02-08 10:42:04,773 INFO - Successfully loaded base configuration from file at '/liferay/liferay-portal-6.0.5/tomcat-6.0.26/temp/tc-config29604913327303$
2011-02-08 10:42:04,993 INFO - Log file: '/root/terracotta/client-logs/terracotta-client.log'.
2011-02-08 10:42:07,242 INFO - Connection successfully established to server at 192.168.1.169:9510
10:42:36,467 INFO [PortalImpl:277] Global lib directory /liferay/liferay-portal-6.0.5/tomcat-6.0.26/lib/ext/
10:42:36,469 INFO [PortalImpl:297] Portal lib directory /liferay/liferay-portal-6.0.5/tomcat-6.0.26/webapps/ROOT/WEB-INF/lib/
10:42:57,869 ERROR [QuartzSchedulerEngineImpl:73] Unable to initialize engine
java.lang.RuntimeException: Problem obtaining node id from TerracottaJobStore.
at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:1180)
at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1465)
at com.liferay.portal.scheduler.quartz.QuartzSchedulerEngineImpl.afterPropertiesSet(QuartzSchedulerEngineImpl.java:70)

Can any help.
Thanks
Keith
Hi Mika,

I am getting following error while integrating liferay 6 sp1 to terracotta 3.6 :

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'counterHibernateSessionFactory' defined in class path resource [META-INF/hibernate-spring.xml]: Invocation of init method failed; nested except
heManager in the config or do one of following:
1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary
2. Shutdown the earlier cacheManager before creating new one with same name.

Please help me to resolve this.
To solve the above problem, in portal-ext.properties ("EhCache and Hibernate Second Level Cache", step 7)
replace

hibernate.cache.region.factory_class=net.sf.ehcache.hibernate.EhCacheRegionFactory
with
hibernate.cache.region.factory_class=net.sf.ehcache.hibernate.SingletonEhCacheRegionFactory
Storing the session in terracotta works fine but the cache doesnt work.


08:01:44,701 ERROR [ContextLoader:220] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'asyncAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'asyncAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'threadLocalCacheAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'threadLocalCacheAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'bufferedIncrementAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bufferedIncrementAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'transactionAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'liferayTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]: Cannot resolve reference to bean 'liferayHibernateSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayHibernateSessionFactory' defined in class path resource [META-INF/hibernate-spring.xml]: Invocation of init method failed; nested exception is org.hibernate.cache.CacheException: net.sf.ehcache.CacheException: Error configuring from null. Initial cause was null
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1325)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1086)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
Are there any plans to update/post instructions for integrating Liferay 6.1 with Terracotta? I tried to execute using the instructions above with Liferay 6.1 EE, however, I wound up getting the same error that Ankit reported above. It seems that the hibernate-spring initialization attempts to create the same caches that are created by the hibernate-terracotta Ehcache configuration. For reference, I used the XML files stored in the 6.1 EE portal-impl.jar rather than the files referenced above; they seem to be largely similar, but I figured that it'd be safer to stick with the ones that are distributed within EE.
[...] I have 2 server. - Server1: OS is Ubuntu server 10.04 LTS. Run Liferay Portal 6.1 CE. (IP: 192.168.2.60) java version "1.6.0_20" OpenJDK Runtime Environment (IcedTea6 1.9.13)... [...] Read More
Hi All

I am facing the issue when integrating terracotta with liferay-portal-6.2.10.1-ee-ga1

I am followed the Mika Blog http://www.liferay.com/web/mika.koivisto/blog/-/blogs/how-do-i-cluster-liferay-with-terracotta-

The below is the error I am getting, Request you to please help me here.

2014-05-17 16:48:54,417 INFO – Connection successfully established to server at 192.168.0.100:9510
16:48:55,350 INFO [localhost-startStop-1][ValueModeHandlerFactory:125] Hibernate types found on the classpath : Enabling Hibernate value mode optimizations
16:48:55,912 INFO [localhost-startStop-1][ClusteredStore:125] Cache [com.liferay.portal.model.impl.UserImpl] using concurrency: 256
16:49:09,032 ERROR [localhost-startStop-1][ContextLoader:206] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘com.liferay.portal.kernel.cache.MultiVMPool’ defined in class path resource [META-INF/util-spring.xml]: Cannot resolve reference to bean ‘com.liferay.portal.kernel.cache.MultiVMPortalCacheManager’ while setting bean property ‘portalCacheManager’; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘com.liferay.portal.kernel.cache.MultiVMPortalCacheManager’ defined in class path resource [META-INF/util-spring.xml]: Cannot create inner bean ‘com.liferay.portal.cache.ehcache.EhcachePortalCacheManager#704fb2′ of type [com.liferay.portal.cache.ehcache.EhcachePortalCacheManager] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘com.liferay.portal.cache.ehcache.EhcachePortalCacheManager#704fb2′ defined in class path resource

Caused by: java.lang.NoSuchFieldException: statisticsExecutor
at java.lang.Class.getDeclaredField(Class.java:1948)
at com.liferay.portal.kernel.util.ReflectionUtil.getDeclaredField(ReflectionUtil.java:66)
at com.liferay.portal.cache.ehcache.CacheManagerUtil.<clinit>(CacheManagerUtil.java:90)
… 52 more
Hi there,

I currently try to integrate terracotta-4.3.0 with liferay-portal-tomcat-6.2-ce-ga2 following this post:
* I've upgraded ehcache-2.10.0.jar slf4j and added terracotta-toolkit-runtime-4.3.0.jar to the ROOT/WEB-INF/lib
* configured liferay hibernate-clustered.xml + liferay-multi-vm-clustered.xml for terracotta
* modified portal-ext.properties to use terracotta cache configuration

The terracotta client connexion works fine:

2015-05-27 17:12:25,973 INFO - Terracotta 4.3.0, as of 20150402-130831 (Revision 26225 from 4.3.0)
2015-05-27 17:12:26,533 INFO - Successfully loaded base configuration from server at 'localhost:9510'.
2015-05-27 17:12:26,566 INFO - Log file: '/home/foo/terracotta/client-logs/terracotta-client.log'.
2015-05-27 17:12:27,882 INFO - Connection successfully established to server at 127.0.1.1:9510
2015-05-27 17:12:45,049 INFO - Terracotta 4.3.0, as of 20150402-130831 (Revision 26225 from 4.3.0)
2015-05-27 17:12:45,414 INFO - Successfully loaded base configuration from server at 'localhost:9510'.
2015-05-27 17:12:45,438 INFO - Log file: '/home/foo/terracotta/client-logs/terracotta-client.log'.
2015-05-27 17:12:45,954 INFO - Connection successfully established to server at 127.0.1.1:9510

but I have a classloader issue in deserialization process on a cached class in a portlet (not the ROOT webapp), here is the stack:

Caused by: net.sf.ehcache.CacheException: Uncaught exception in get() - java.lang.ClassNotFoundException: foo.bar.liferay.model.impl.StructureSiteCacheModel
at org.terracotta.modules.ehcache.store.ClusteredSafeStore.get(ClusteredSafeStore.java:236)
at org.terracotta.modules.ehcache.store.nonstop.NonStopStoreWrapper.get(NonStopStoreWrapper.java:792)
at net.sf.ehcache.Cache.get(Cache.java:1751)
at net.sf.ehcache.Cache.get(Cache.java:1724)
at com.liferay.portal.cache.ehcache.EhcachePortalCache.get(EhcachePortalCache.java:64)
at com.liferay.portal.cache.transactional.TransactionalPortalCache.get(TransactionalPortalCache.java:66)
at com.liferay.portal.dao.orm.common.EntityCacheImpl.loadResult(EntityCacheImpl.java:188)
at com.liferay.portal.kernel.dao.orm.EntityCacheUtil.loadResult(EntityCacheUtil.java:59)
at com.liferay.portal.dao.orm.common.FinderCacheImpl._primaryKeyToResult(FinderCacheImpl.java:273)
at com.liferay.portal.dao.orm.common.FinderCacheImpl.getResult(FinderCacheImpl.java:135)
at com.liferay.portal.kernel.dao.orm.FinderCacheUtil.getResult(FinderCacheUtil.java:47)
at foo.bar.liferay.service.persistence.StructureSitePersistenceImpl.fetchByUai(StructureSitePersistenceImpl.java:686)
at foo.bar.liferay.service.persistence.StructureSitePersistenceImpl.fetchByUai(StructureSitePersistenceImpl.java:667)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:320)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at com.liferay.portal.dao.shard.advice.ShardPersistenceAdvice.invoke(ShardPersistenceAdvice.java:54)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy473.fetchByUai(Unknown Source)
at foo.bar.liferay.service.impl.StructureSiteLocalServiceImpl.fetchByUai(StructureSiteLocalServiceImpl.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.execute(DefaultTransactionExecutor.java:62)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
at com.sun.proxy.$Proxy472.fetchByUai(Unknown Source)
at foo.bar.liferay.service.StructureSiteLocalServiceUtil.fetchByUai(StructureSiteLocalServiceUtil.java:278)
at foo.bar.utils.PersonUtils.getStructuresSites(PersonUtils.java:187)
at foo.bar.utils.AtriumPortletUtils.canCreateSubSite(AtriumPortletUtils.java:277)
at org.apache.jsp.html.portlet.dockbar.view_jsp._jspService(view_jsp.java:1224)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
... 182 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: foo.bar.liferay.model.impl.StructureSiteCacheModel
at com.terracotta.toolkit.collections.map.ServerMap.deserialize(ServerMap.java:389)
at com.terracotta.toolkit.collections.map.ServerMap.deserialize(ServerMap.java:369)
at com.terracotta.toolkit.collections.map.ServerMap.getNonExpiredValue(ServerMap.java:324)
at com.terracotta.toolkit.collections.map.ServerMap.doGet(ServerMap.java:264)
at com.terracotta.toolkit.collections.map.ServerMap.get(ServerMap.java:1274)
at com.terracotta.toolkit.collections.map.AggregateServerMap.get(AggregateServerMap.java:395)
at com.terracotta.toolkit.collections.map.ToolkitCacheImpl.get(ToolkitCacheImpl.java:251)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.terracotta.toolkit.nonstop.NonStopInvocationHandler.invokeMethod(NonStopInvocationHandler.java:154)
at com.terracotta.toolkit.nonstop.NonStopInvocationHandler.invoke(NonStopInvocationHandler.java:54)
at com.sun.proxy.$Proxy19.get(Unknown Source)
at org.terracotta.modules.ehcache.store.ClusteredStore.get(ClusteredStore.java:316)
at org.terracotta.modules.ehcache.store.ClusteredSafeStore.get(ClusteredSafeStore.java:229)
... 225 more

Does anyone experienced this before ? any Idea ?

thanx.
Hello Sébastien: have you been able to solve your issue?
In my architecture, I have a Liferay 6.2 EE cluster (probably 6 nodes) that I want to connect to a Terracotta 4.3.1 cluster. However, because of the previous feedbacks I read in this blog, I just wonder if it is relevant to propose a Terracotta cluster.
Any idea?
The correct answer is it depends. For example if they are geographically distributed in different data centers it might be relevant. If they are all in the same DC then Liferay ClusterLink enabled cache is probably sufficient. In order to get supported Terracotta + Liferay EE you'll need Liferay EE Terracotta Edition.
Thank you Mika for this quick answer.
The issue of using ClusterLink is that my caches will be replicated what decreases the horizontal scalability of my architecture. I found it interesing to have a central cache solution in order to prevent caches and sessions replications.
Are there alternatives cache solutions to Terracotta ?
Don't use cache replication for Liferay caches. It won't give you any performance increase but rather causes issues with stale data that is unless you use Terracotta. So far I've seen Terracotta is the best solution where you need cohesive replicated cache.

Sticky session and cache invalidation instead of replication is the best way to go as far as Liferay is concerned. For your custom apps it might be different.
Hi Van Tuan, I finaly use ClusterLInk in my CE cluster (6.2GA5) multicast mode with stickySession and cache invalidation over 2 nodes according to Mika recommendations. And it works fine ;)