Liferay 7 on AWS

[ajith ratnarajah, modifié il y a 6 années.]

Is it possible for Liferay to develop a white paper on how to install Liferay in AWS especially taking into account AWS products. I have met customers who are giving up on Liferay especially on Liferay 7 where the performance is terrible from startup to running it on AWS. Elasticsearch configuration is close to a nightmare to get it working with endless "watermark errors" even when the most powerful AWS machine is assigned. The auto scaling capabilities cannot be utilised if the startup of a server takes around 5-10 mins.

[Jorge Díaz, modifié il y a 6 années.]

Hi Ajith,

Which Liferay 7 version are you using? Did you try new GA4 version?

About you message:

ajith ratnarajah:

Elasticsearch configuration is close to a nightmare to get it working

What kind of issues are you having with Elasticsearch configuration? Which Elasticsearch version and configuration are you using? Only remote Elasticsearch configuration is supported for production environment (embedded mode is only available for demo and development environments) and Elasticsearch version should be 2.2, 2.2.1 or 2.2.2

ajith ratnarajah:

...with endless "watermark errors" even when the most powerful AWS machine is assigned.

Can you give more detail about that "watermark errors"?

ajith ratnarajah:

The auto scaling capabilities cannot be utilised if the startup of a server takes around 5-10 mins.

Did you review startup log and analyzed which startup points are spending most of the startup time?

About this sentence:

ajith ratnarajah:

I have met customers who are giving up on Liferay especially on Liferay 7 where the performance is terrible from startup to running it on AWS.

If you are selling Liferay 7 projects to customers, did you think about talking with Liferay, get DXP version prices and try selling your project with an enterprise version, with official support?

Regards,
Jorge Díaz

[ajith ratnarajah, modifié il y a 6 années.]

Thanks Jorge for your reply.

I am looking at configuring Liferay in a distributed architecture (if possible). Would like to use AWS cloudfront, AWS Elasticsearch, AWS RDS, AWS S3, AWS EFS and use EC2 instances mostly as a memory cache. We want to start and shut down EC2 instances in an autoscaling environment. We want to reduce the startup and time to serve pages on the liferay portal. Performance is no 1 priority. We want pages to come up within 1-2 seconds. I doubt we will ever get that with CE or DXP so we are looking at other alternatives. However I am a big liferay fan so want to get it working with CE first.

Right now I have installed Liferay in AWS EFS and trying to get instances to start up to connect to that liferay AWS EFS. if we upgrade liferay then we do it on the EFS then all instances can be upgraded. This is our hope.

On the watermark errors, I am trying to avoid it by configuring AWS Elasticsearch as an independent service for Liferay to access. We want to keep Liferay as a thin GUI layer that co-ordinate services from multiple services in AWS.

I hope this helps.

Regards

Ajith

[ajith ratnarajah, modifié il y a 6 années.]

sorry btw the errors I get are is the following

[elasticsearch[Architect][management][T#2]][decider:157] [Architect] high disk watermark [90%] exceeded on [TkX6_4OYT521JEyAA8fTkQ][Architect][/home/ubuntu/efs/software/liferay/data/elasticsearch/indices/LiferayElasticsearchCluster/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node
12:53:31,322 INFO [elasticsearch[Architect][management][T#2]][decider:218] [Architect] rerouting shards: [high disk watermark exceeded on one or more nodes]
12:54:01,458 WARN [elasticsearch[Architect][management][T#3]][decider:157] [Architect] high disk watermark [90%] exceeded on [TkX6_4OYT521JEyAA8fTkQ][Architect][/home/ubuntu/efs/software/liferay/data/elasticsearch/indices/LiferayElasticsearchCluster/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node
12:54:15,836 INFO [localhost-startStop-1][BaseAutoDeployListener:50] Themes for /home/ubuntu/efs/software/liferay/tomcat-8.0.32/temp/20170624125232415GLDNWFEI/classic-theme.war copied successfully
12:54:31,509 WARN [elasticsearch[Architect][management][T#1]][decider:157] [Architect] high disk watermark [90%] exceeded on [TkX6_4OYT521JEyAA8fTkQ][Architect][/home/ubuntu/efs/software/liferay/data/elasticsearch/indices/LiferayElasticsearchCluster/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node
12:54:31,512 INFO [elasticsearch[Architect][management][T#1]][decider:218] [Architect] rerouting shards: [high disk watermark exceeded on one or more nodes]
12:54:41,755 INFO [localhost-startStop-1][BaseAutoDeployListener:42] Copying themes for /home/ubuntu/efs/software/liferay/tomcat-8.0.32/temp/20170624125441593HBGBTHWI/user-profile-theme.war
12:54:42,897 INFO [localhost-startStop-1][BaseDeployer:873] Deploying user-profile-theme.war
12:55:01,648 WARN [elasticsearch[Architect][management][T#2]][decider:157] [Architect] high disk watermark [90%] exceeded on [TkX6_4OYT521JEyAA8fTkQ][Architect][/home/ubuntu/efs/software/liferay/data/elasticsearch/indices/LiferayElasticsearchCluster/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node

I am using GA3 version

[Christoph Rabel, modifié il y a 6 années.]

Those warnings are from Elasticsearch and not from Liferay. They tell you that since the harddisk is pretty full new shards will not be created in this instance when you have clustered elasticsearch. I guess, they can be configured somehow in elasticsearch.
And they absolutely make no sense in your case. It doesn't matter that 90% of the volume is already used when 8000pb are still free ...

Also you should not use the embedded Elasticsearch in production. It's bad for performance. You should setup a dedicated elasticsearch instance.

We want pages to come up within 1-2 seconds. I doubt we will ever get that with CE or DXP so we are looking at other alternatives.

1-2 seconds isn't that ambitious.

Example: Virtual machine, 1 Core, and that one shared with other instances, 4GB Ram. So, pretty weak. :-)

The page itself should load in less than 100 ms, well, most of the time.
https://www.dccs.at/loesungen

Here's the same page without some brutal optimization/cheating:
https://www.dccs.at/loesungen_orig

[Jorge Díaz, modifié il y a 6 années.]

About "high disk watermark" traces:

ajith ratnarajah:

[elasticsearch[Architect][management][T#2]][decider:157] [Architect] high disk watermark [90%] exceeded on [TkX6_4OYT521JEyAA8fTkQ][Architect][/home/ubuntu/efs/software/liferay/data/elasticsearch/indices/LiferayElasticsearchCluster/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node
12:53:31,322 INFO [elasticsearch[Architect][management][T#2]][decider:218] [Architect] rerouting shards: [high disk watermark exceeded on one or more nodes]
12:54:01,458 WARN [elasticsearch[Architect][management][T#3]][decider:157] [Architect] high disk watermark [90%] exceeded on [TkX6_4OYT521JEyAA8fTkQ][Architect][/home/ubuntu/efs/software/liferay/data/elasticsearch/indices/LiferayElasticsearchCluster/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node

They are written by embedded Elasticsearch that it is included in Liferay only for demo and development purposes.
You must not use Liferay with default embedded Elasticsearh. You have to install a external Elasticsearch 2.2.x server in other machine and change Liferay configuration from "embedded" to "remote", more information see:

• Preparing to Install Elasticsearch
• Configuring Elasticsearch for Liferay Portal
• Installing Elasticsearch

About:

ajith ratnarajah:

I am using GA3 version

Try installing Liferay 7 GA4, there are a lot of throughput improvements in that new version

[ajith ratnarajah, modifié il y a 6 années.]

Thanks Jorge and Christophe for both of your replies. Appreciate your support

1. Is there anyway I can not run elasticsearch ? I assume not but worth asking. Quite frankly fed up trying to set it up. The new version seems to have a bug too in identifying remote hosts. I am getting a lot of errors in Liferay as it is unable to communicate to the server. I am trying to setup Liferay instance on EC2 with t2.large.

2. Is it possible to configure to use AWS Elasticsearch ? if so is there any specific gotcha's i need to be aware of ?

Hi Christophe, I did go to your site but it took me 7 sec to upload the first page its obviously the network connection here that is bad in Australia but unfortunately thats why I have been given 1-2 sec.

Regards

Ajith

[Christoph Rabel, modifié il y a 6 années.]

ajith ratnarajah:

1. Is there anyway I can not run elasticsearch ?

No. Elasticsearch is used extensively.

ajith ratnarajah:

The new version seems to have a bug too in identifying remote hosts. I am getting a lot of errors in Liferay as it is unable to communicate to the server. I am trying to setup Liferay instance on EC2 with t2.large.

What do you mean with "new version". You MUST use Elasticsearch 2.2 with Liferay. You MUST NOT use a newer version.

ajith ratnarajah:

2. Is it possible to configure to use AWS Elasticsearch ? if so is there any specific gotcha's i need to be aware of ?

I don't know. AWS only offers some specific versions, relevant is 2.3. Alas, Liferay wants 2.2.2.
I guess Liferay uses the transport client of elasticsearch and it is "somewhat" compatible to other versions of Elasticsearch. No idea if Liferay works with Elasticsearch 2.3 or if you will get weird errors.

2.2.2 or 2.3 or 5 or anything doesn't really make a difference for me, but for people who want to use AWS, 2.3 would be a real treat.

ajith ratnarajah:

Hi Christophe, I did go to your site but it took me 7 sec to upload the first page its obviously the network connection here that is bad in Australia but unfortunately thats why I have been given 1-2 sec.

Yes, Austria (here) is small, Australia is huge. And far away :-)
Since you absolutely have no control over the network latency you can only improve the performance of the server. In our case, the page is delivered by the server in <100ms but in the standard version it needs about 1-2 seconds. The slow network comes obviously on top of that as a further burden.

You should consider using a CDN. It's really a treat if css, images, ... are delivered from a "local source". Since network performance is probably a really issue for you, a CDN might help you the most.

[Jorge Díaz, modifié il y a 6 années.]

ajith ratnarajah:

2. Is it possible to configure to use AWS Elasticsearch ?

It isn't possible to connect Liferay to AWS Elasticsearch because:

• Elasticsearch version 2.2 is not available in AWS
• AWS only allows HTTP port connections, but Java client only connects to transport port.

More information see: https://web.liferay.com/community/forums/-/message_boards/message/74076858

[ajith ratnarajah, modifié il y a 6 années.]

Thanks Jorge but I will now have major issues in now implementing Elasticsearch with Liferay from an internal compliance point of view.

1. Liferay supports a version that is well behind Elasticsearch current versions and seems like we cannot upgrade Elasticsearch independently even when its an independent server / service ?. This is a compliance issue for me too.
2. AWS Elasticsearch as a service is our preferred approach as AWS does upgrades etc much better. Infact we can even consume the latest version of Elasticsearch and security updates etc.
3. The total cost of ownership of Liferay now is going to be significantly higher as we now have to justify why we need additional servers in AWS when there is already Elasticsearch as a service. That too with an older version. Who is going to maintain, upgrade etc etc.

I think some of the issues raised in the thread by the other participants are quite valid for me too. Anyway again I thank you for your reply and providing me a solution to fix my current problem. Much appreciated your solution atleast fixes my immediate issues. But in the medium term this is going to cause a serious issue for me internally.

We would be looking to use the new Elasticsearch or ELK which is 2.4 and beyond.

Regards

Ajith