Problems Synchronizing Nodes

[Robert Revnic, geändert vor 11 Jahren.]

Hi All!

I'm having some problems with my Liferay cluster. The cluster consists of two Tomcat nodes which have Liferay 6.1.1 deployed to them. The problem is that the changes made in Liferay on node1 do not reflect on node2 for a significant amount of time (I believe until the ehcache gets invalidated). If I browse to the control panel and hit clear the content cached across the cluster, the changes appear correctly on both nodes. Is there a configuration change that I have overlooked? My portal-ext.properties file can be found below.

lucene.replicate.write=true


################################################################################################
################################################################################################
################################################################################################



##
## Hibernate
##

#
# Input a list of comma delimited Hibernate configurations.
#
hibernate.configs=\
META-INF/mail-hbm.xml,\
META-INF/portal-hbm.xml,\
META-INF/ext-hbm.xml

#
# Liferay will automatically detect the Hibernate dialect in
# com.liferay.portal.spring.PortalHibernateConfiguration. Set this property
# to manually override the automatically detected dialect.
#
#hibernate.dialect=

#
# Set the Hibernate connection release mode. You should not modify this
# unless you know what you're doing. The default setting works best for
# Spring managed transactions. See the method buildSessionFactory in class
# org.springframework.orm.hibernate3.LocalSessionFactoryBean and search for
# the phrase "on_close" to understand how this works.
#
#hibernate.connection.release_mode=on_close

#
# Set the Hibernate cache region factory. Ehcache is recommended in a
# clustered environment. See the property
# "net.sf.ehcache.configurationResourceName" for detailed configuration.
#
hibernate.cache.region.factory_class=com.liferay.portal.dao.orm.hibernate.region.SingletonLiferayEhcacheRegionFactory

#
# This property is used if Hibernate is configured to use Ehcache's region
# factory.
#
net.sf.ehcache.configurationResourceName=/myehcache/hibernate-clustered.xml

#
# Configure this property if using Ehcache's default replication method for

# Hibernate's region cache.
#
net.sf.ehcache.configurationResourceName.peerProviderProperties=peerDiscovery=automatic,multicastGroupAddress=${multicast.group.address["hibernate"]},multicastGroupPort=${multicast.group.port["hibernate"]},timeToLive=1

#
# Set other Hibernate cache settings.
#
hibernate.cache.use_query_cache=true
hibernate.cache.use_second_level_cache=true
hibernate.cache.use_minimal_puts=true
hibernate.cache.use_structured_entries=false


#
# Uncomment these properties to disable Hibernate caching.
#
#hibernate.cache.use_query_cache=false
#hibernate.cache.use_second_level_cache=false

#
# Set the JDBC batch size to improve performance.
#
# If you're using Hypersonic, you SHOULD set the batch size to 0 as a
# workaround for a logging bug in the Hypersonic driver. See LPS-5426 for
# more information.
#
# If you're using Oracle 9i, you MUST set the batch size to 0 as a
# workaround for a hanging bug in the Oracle driver. See LEP-1234 for more
# information.
#
hibernate.jdbc.batch_size=20
#hibernate.jdbc.batch_size=0

#
# Set other miscellaneous Hibernate properties.
#
hibernate.jdbc.use_scrollable_resultset=true
hibernate.bytecode.use_reflection_optimizer=true
hibernate.show_sql=false

#
# Use the classic query factory until WebLogic and Hibernate 3 can get
# along. See http://www.hibernate.org/250.html#A23 for more information.
#
hibernate.query.factory_class=org.hibernate.hql.classic.ClassicQueryTranslatorFactory

#
# Set this property to true to enable Hibernate cache monitoring. See
# LPS-2056 for more information.
#
hibernate.generate_statistics=false




################################################################################################
################################################################################################
################################################################################################

##
## Ehcache
##

#
# Set the classpath to the location of the Ehcache config file for internal
# caches. Edit the file specified in the property
# "ehcache.multi-vm.config.location" to enable clustered cache.
#
ehcache.single.vm.config.location=/ehcache/liferay-single-vm.xml
ehcache.multi.vm.config.location=/myehcache/liferay-multi-vm-clustered.xml

#
# Uncomment the following in a clustered environment.
#
ehcache.multi.vm.config.location.peerProviderProperties=peerDiscovery=automatic,multicastGroupAddress=${multicast.group.address["multi-vm"]},multicastGroupPort=${multicast.group.port["multi-vm"]},timeToLive=1

#
# Input a class name that extends
# net.sf.ehcache.bootstrap.BootstrapCacheLoaderFactory. This class is used
# by Ehcache to determine how caches communicate with each other in a
# clustered environment.
#
ehcache.bootstrap.cache.loader.factory=net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory
#ehcache.bootstrap.cache.loader.factory=com.liferay.portal.cache.ehcache.JGroupsBootstrapCacheLoaderFactory

#
# Input a class name that extends
# net.sf.ehcache.event.CacheEventListenerFactory. This class is used by
# Ehcache to determine how caches communicate with each other in a clustered
# environment.
#
ehcache.cache.event.listener.factory=net.sf.ehcache.distribution.RMICacheReplicatorFactory
#ehcache.cache.event.listener.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory

#
# Input a class name that extends
# net.sf.ehcache.distribution.CacheManagerPeerProviderFactory. This class is
# used by Ehcache to determine how caches communicate with each other in a
# clustered environment.
#
ehcache.cache.manager.peer.provider.factory=net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory
#ehcache.cache.manager.peer.provider.factory=net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory

#
# Set this to true to enable JMX integration in
# com.liferay.portal.cache.EhcachePortalCacheManager. Set this to false if
# the portal needs to be reloaded under WebSphere.
#
ehcache.portal.cache.manager.jmx.enabled=true


#
# Set this to true to allow Ehcache to use blocking caches. This improves
# performance significantly by locking on keys instead of the entire cache.
# The drawback is that threads can hang if the cache is not used properly.
# Make sure that all queries that return a miss also immediately populate
# the cache, or else other threads that are blocked on a query of that same
# key will continue to hang. Reference Ehcache's BlockingCache for more

# information. The blocking cache is no longer implemented by Ehcache's
# BlockingCache, but by Liferay's BlockingPortalCache for better safety and
# faster performance.
#
ehcache.blocking.cache.allowed=true

#
# Set this to true to enable Ehcache statistics.
#
ehcache.statistics.enabled=false



################################################################################################
################################################################################################
################################################################################################



cluster.link.enabled=true
ehcache.cluster.link.replication=true


Any tips/help on this is greatly appreciated.

[Juan Gonzalez, geändert vor 11 Jahren.]

Hi Robert,

You only have to set this properties:

cluster.link.enabled=true
lucene.replicate.write=true

and then check if the default multicast IP are accesible in your network from one node to the others:

multicast.group.address["cluster-link-control"]=239.255.0.1
    multicast.group.port["cluster-link-control"]=23301

    #
    # See the properties "cluster.link.channel.properties.transport.0" and
    # "cluster.link.channel.system.properties".
    #
    multicast.group.address["cluster-link-udp"]=239.255.0.2
    multicast.group.port["cluster-link-udp"]=23302

    #
    # See the property "cluster.link.channel.system.properties".
    #
    multicast.group.address["cluster-link-mping"]=239.255.0.3
    multicast.group.port["cluster-link-mping"]=23303

    #
    # See the properties "net.sf.ehcache.configurationResourceName" and
    # "net.sf.ehcache.configurationResourceName.peerProviderProperties".
    #
    multicast.group.address["hibernate"]=239.255.0.4
    multicast.group.port["hibernate"]=23304

    #
    # See the properties "ehcache.multi.vm.config.location" and
    # "ehcache.multi.vm.config.location.peerProviderProperties".
    #
    multicast.group.address["multi-vm"]=239.255.0.5
    multicast.group.port["multi-vm"]=23305

[Robert Revnic, geändert vor 11 Jahren.]

Thanks for the support Juan.

When commenting out ehcache.cluster.link.replication.enabled=true, the following errors are thrown when the second node is started:

14:33:06,980 INFO [pool-3-thread-1][LiferayCacheManagerPeerProviderFactory:76] portalPropertyKey net.sf.ehcache.configurationResourceName.peerProviderProperties has value peerDiscovery=automatic,multicastGroupAddress=239.255.0.4,multicastGroupPort=23304,timeToLive=1
14:33:08,423 ERROR [pool-3-thread-1][ContextLoader:227] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.liferay.portal.spring.aop.ServiceBeanAutoProxyCreator#0' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'serviceAdvice' while setting bean property 'methodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'asyncAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'asyncAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'threadLocalCacheAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'threadLocalCacheAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'bufferedIncrementAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bufferedIncrementAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'indexableAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexableAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'transactionAdvice' while setting bean property 'nextMethodInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'liferayTransactionManager' while setting bean property 'platformTransactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]: Cannot resolve reference to bean 'liferayHibernateSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayHibernateSessionFactory' defined in class path resource [META-INF/hibernate-spring.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread

followed by several out of memory exceptions.

The nodes are running on the same physical box. Any insight on how I could proceed? Does this still seem like an issue with the multicast?


Thanks again,

Rob

[Juan Gonzalez, geändert vor 11 Jahren.]

Can you try shutting down all nodes and then ensure there aren't any Liferay process running? (for example: ps -ef |grep java and kill all process that should be stopped) ?

After that, start one node, see logs, and then start second node and see its logs....

[Robert Revnic, geändert vor 11 Jahren.]

I should have mentioned that I also have multiple stand alone nodes running on the box (dev, qastg) in addition to the clustered nodes (qa prd). Shutting everything down and restarting one by one fixed the OOM issue. The clustered nodes are in sync now . Removing the ehcache.cluster.link.replication.enabled=true property worked. Thanks for the help.

[Juan Gonzalez, geändert vor 11 Jahren.]

Glad to hear that.

The property

ehcache.cluster.link.replication.enabled=true

shouldn't be used unless you have EE license, and hence you can install ehcache cluster web plugin.

[Sumit Kumari, geändert vor 4 Jahren.]

Hi , I'm new to lIferay , Can you please help me to sync data b/w all nodes.My setup includes: 1) 2 liferay nodes 2) one database 3) apache webserver for  load balancer with mod_jk.My clustering is working and now I want to deploy  on one server so that it is synced with all other nodes.For this, I have followed https://liferay.dev/blogs/-/blogs/liferay-6-1-simple-cluster, where ndeploy.sh script is given to deploy from one node to others but i'm not able to do so.
I want to know if I'm deploying any portlet on one liferay/deploy folder it get s deployed to ndeploy /deploy folder but does not sync with other node to have same data.

[Olaf Kock, geändert vor 4 Jahren.]

Sumit Kumari:

Hi , I'm new to lIferay , Can you please help me to sync data b/w all nodes.My setup includes: 1) 2 liferay nodes 2) one database 3) apache webserver for  load balancer with mod_jk.My clustering is working and now I want to deploy  on one server so that it is synced with all other nodes.For this, I have followed https://liferay.dev/blogs/-/blogs/liferay-6-1-simple-cluster, where ndeploy.sh script is given to deploy from one node to others but i'm not able to do so.
I want to know if I'm deploying any portlet on one liferay/deploy folder it get s deployed to ndeploy /deploy folder but does not sync with other node to have same data.

You're asking this in 2019 on a thread on Liferay 6.1, linking to documentation for that version. Does this mean that you're actually on 6.1? (If the answer is yes, I hope your upgrade-project is coming along quite well, and you're on the way to an updated version)

Anyway: The script and methodology documented in the article will enable you to copy files across your network (and grant one machine shell access along the way), something that you could also do any other way you fancy. Try it manually first, then go to automation.

[Sumit Kumari, geändert vor 4 Jahren.]

Olaf Kock:

Sumit Kumari:

Hi , I'm new to lIferay , Can you please help me to sync data b/w all nodes.My setup includes: 1) 2 liferay nodes 2) one database 3) apache webserver for  load balancer with mod_jk.My clustering is working and now I want to deploy  on one server so that it is synced with all other nodes.For this, I have followed https://liferay.dev/blogs/-/blogs/liferay-6-1-simple-cluster, where ndeploy.sh script is given to deploy from one node to others but i'm not able to do so.
I want to know if I'm deploying any portlet on one liferay/deploy folder it get s deployed to ndeploy /deploy folder but does not sync with other node to have same data.

You're asking this in 2019 on a thread on Liferay 6.1, linking to documentation for that version. Does this mean that you're actually on 6.1? (If the answer is yes, I hope your upgrade-project is coming along quite well, and you're on the way to an updated version)

Anyway: The script and methodology documented in the article will enable you to copy files across your network (and grant one machine shell access along the way), something that you could also do any other way you fancy. Try it manually first, then go to automation.

No, I'm using 6.2 . I didn't find anything related to 6.2 so I  implemented it on 6.2 version. Basically, I want to deploy portlets  across  liferay nodes . I read about farms deployment , can you please tell how I can do that.

[Olaf Kock, geändert vor 4 Jahren.]

Sumit Kumari:

No, I'm using 6.2 . I didn't find anything related to 6.2 so I  implemented it on 6.2 version. Basically, I want to deploy portlets  across  liferay nodes . I read about farms deployment , can you please tell how I can do that.

Farm deployment is a Tomcat feature (maybe other servers as well)
That is an option, though Liferay might add some dependencies when deploying from its "deploy" directory into tomcat's "webapps" directory.
You can build your plugins in a way that this deployment is done already and the deployed WAR file can be directly deployed to tomcat. If memory serves right, the Ant target is "direct-deploy" (but validate, I might be wrong)
Another option is to make sure that some component in your deployment infrastructure copies all plugins to all nodes' deploy directory. You can do that with a script like the one you link to, or with any other solution. Not: Don't share a single deploy directory, because Liferay will delete a processed file from the directory - and then it typically ends up only on one server (the one that found it first)