Liferay Read-Write Database Splitting

The Beauty of Sponsored Development

As a company that likes to work very closely with our community, we're often given the opportunity to work on some very interesting (and many times very challenging) features. The following is one such feature being sponsored by one of our marquee partners: Read-Write database splitting. They are planning a Liferay deployment that scales to several millions of users. 

Because of the dynamic data-intensive nature of this deployment, they've identified that they're likely to to database-bound during peak load.  In order to scale database throughput they wanted to us to implement a mechanism within Liferay that would direct transactional (write) and read-only operations to separate data sources.

How was this done?

First, by using one of lesser-known features of the Spring Framework, target sources, we implmented a custom DynamicDataSourceTargetSource to do the job (if you're not familiar with target sources, one of the nifty things you can do is hot-swap a dynamic proxy's target object at runtime). This DynamicDataSourceTargetSource allows us to switch between read and write data sources at the appropriate time.

Secondly, we wanted to take advantage of Spring's transactional boundaries to help us decide when to use the appropriate data source. Before we did that, there were a couple items that needed cleanup and refactoring:

1) Use @Transactional in favor of xml transaction configuration. Annotations are a better fit for transactions because they allow better fine-grained control of transaction rules. Luckily this transition was very easy because all of our services are autogenerated via ServiceBuilder, so very little coding was needed.

2) We created our own @Transactional interface free of Spring dependencies so that it can be used in our plugins environment. This is virtually a mirror of Spring's @Transactional. We decomposed Spring's <tx:annotation-driven /> and replaced it with our own:

<bean id="transactionAdvice" class="org.springframework.transaction.interceptor.TransactionInterceptor">
<property name="transactionManager" ref="liferayTransactionManager" />
<property name="transactionAttributeSource">
<bean class="org.springframework.transaction.annotation.AnnotationTransactionAttributeSource">
<constructor-arg>
<bean class="com.liferay.portal.spring.annotation.PortalTransactionAnnotationParser" />
</constructor-arg>
</bean>
</property>
</bean>

As you can see we had to implement our own PortalTransactionAnnotationParser to read our custom @Transactional annotation.

3) Now that we have the annotations in place, it's very simple to add additional logic to swap data sources. We created a DynamicDataSourceTransactionInterceptor (subclass of Spring's TransactionInterceptor). This interceptor piggybacks spring's transaction boundary logic to help us decide when to use read or write data sources:

<bean id="transactionAdvice" class="com.liferay.portal.dao.jdbc.aop.DynamicDataSourceTransactionInterceptor">
<property name="dynamicDataSourceTargetSource" ref="dynamicDataSourceTargetSource" />
<property name="transactionManager" ref="liferayTransactionManager" />
<property name="transactionAttributeSource">
<bean class="org.springframework.transaction.annotation.AnnotationTransactionAttributeSource">
<constructor-arg>
<bean class="com.liferay.portal.spring.annotation.PortalTransactionAnnotationParser" />
</constructor-arg>
</bean>
</property>
</bean>

Enabling Read-Write Database Splitting

1) Set up your database for master / slave replication. All major databases support at least a master / slave setup. See some sample instructions for the mysql database. Hint: Make sure you create a read-only user for the slave database so that no one can write directly to the slave.

2) Configure your app server / servlet container for each the master and slave databases. Sample ROOT.xml for Tomcat:

<Resource
name="jdbc/LiferayReadPool"
auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3308/lportal..."
username="read_only"
password="password"
maxActive="20"
/>


<Resource
name="jdbc/LiferayWritePool"
auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3307/lportal..."
username="root"
password="password"
maxActive="20"
/>

3) Set portal-ext.properties to enable the Spring configuration for dynamic data sources:

  spring.configs=\
META-INF/base-spring.xml,\
\
META-INF/hibernate-spring.xml,\
META-INF/infrastructure-spring.xml,\
META-INF/management-spring.xml,\
\
META-INF/util-spring.xml,\
\
META-INF/jcr-spring.xml,\
META-INF/messaging-spring.xml,\
META-INF/scheduler-spring.xml,\
META-INF/search-spring.xml,\
\
META-INF/counter-spring.xml,\
META-INF/documentlibrary-spring.xml,\
META-INF/lock-spring.xml,\
META-INF/mail-spring.xml,\
META-INF/portal-spring.xml,\
META-INF/portletcontainer-spring.xml,\
\
META-INF/mirage-spring.xml,\
\
META-INF/ext-spring.xml, \
META-INF/dynamic-data-source-spring.xml


You now have read-write data source setup. Over the next several months we'll post an update with some performance numbers on a read-write configured set up.  Enjoy :)

[James Min]

Nice work!

[Christian Candia]

This configuration seems to work on Liferay 5.1.X, but it doesn't on 5.2.X. Do you have the correct properties that need to be added to portal-ext.properties for 5.2.X?.

Also what will be the configuration if there are multiple replicas ?. That is, if the read pool contains more than one server.

Thanks,

Christian

[atul patel]

For multiple replicas you can use a loadbalancer to route the sql connections to different instances

[Alex Wallace]

This feature is really great!

What i'm needing right now is a more subtle dynamic change, at will... I've created a thread asking for help about it: http://www.liferay.com/web/guest/community/forums/-/message_boards/message/3225657 ... Any advise towards this would be greatly appreciated!

[Christophe Cariou]

Hi!

Is it possible to have a liferay instance with read-only capabilities ?

I want 2 Liferay instances, each with its database, and master-slave repli between them. I don't want any writing on the slave and also no writing to the master from the slave-associated liferay instane?

[Ciprian Pavel]

Hi Christophe,

Did you find a solution for you issue?
I am facing the same on my site.

Regards,
Ciprian