Fast and Furious Search Enterprise Configuration

This article aims to deliver what you're really looking for, configure Search Enterprise. If you are looking for the smallest details and deeper information, my suggestion is the Elasticserach Documentation and the Liferay Help Center.

Well, for you, who understands a bit about the subject and knows what you're looking for, here is your place, below I'll show you how you can configure/install your Elasticsearch Cluster/Enterprise.

I would like to remind you that this configuration was not made in a production environment in fact but rather in my own notebook, simulating a production environment, so if you are going to do this configuration in a real production environment, you may find stones in the path that maybe I have not found.

Lets go! To begin with, the lab I've created to simulate this setup was as follows:
  • A Liferay DXP 7.1 SP1 Instance
  • Two Elasticsearch 6.5.1 Instances 

Before we actually begin, this article assumes as a prerequisite the installation of Liferay already done, if you have not yet installed your Liferay DXP 7.1, you can do through this link

Configuring Liferay 7.1 SP1 + Elasticsearch 6.5.1

Elasticsearch versions supported

Liferay DXP 7.1 supports two versions of Elasticsearch:
  • 6.1
  • 6.5 (from the liferay-fix-pack-dxp-5)

Preparing to Install Elasticsearch

Before starting the Elasticsearch installation, there are a few things to worry about:

  • CPU:
    • Recommended: Eight total CPU cores to the Elasticsearch engine, assuming that only one Elasticsearch JVM is running on the machine.
  • Memory:
    • Recommended: At least 16 GB of memory, with 64 GB preferred. Accurate memory allocation depends on the amount of indexed data. For index sizes from 500 GB to 1 TB, 64 GB of memory is sufficient.
  • Disk:
    • Recommended: SSD, or make use of the total performance of your HD with a 15k RPM. Keep 25% more disk capacity than the total size of your indexes. If your index is 60 GB, make sure you have at least 75 GB of available disk space.
    • Not Recommended: Do not use NAS.
  • Cluster Size:
    • The minimum cluster size recommended by Elastic for fault tolerance is three nodes.
    • It makes a known problem called a split brain not happen.

Installing Elasticsearch

For installation, follow the following steps:

  1. Download Elasticsearch from Elastic’s web site
  2. Install Elasticsearch by extracting its archive to the system where you want it to run.
  3. Install some required Elasticsearch plugins.
  4. Name your Elasticsearch cluster.
  5. Configure discovery properties
  6. Configurar a propriedade network.host
  7. Configure Liferay DXP to connect to your Elasticsearch cluster.
  8. Restart Liferay DXP and reindex your search indexes.

Step 1: Download Elasticsearch from Elastic’s web site

The version of Elasticsearch that you're going to download should be the same version of Elasticserach that you are embedding in your Liferay DXP 7.1, to validate which version you are running, just access the URL: http://localhost:9200
 
The return should be as follows:
 
{
  "name" : "g0m223N",
  "cluster_name" : "LiferayElasticsearchCluster",
  "cluster_uuid" : "Ii6STs04Tg-XzTVV5h7M2Q",
  "version" : {
    "number" : "6.5.1",
    "build_hash" : "af51318",
    "build_date" : "2018-01-26T18:22:55.523Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
 
Where number is the version of Elasticsearch you should download from the Elastic website.
 

Step 2: Installing Elasticsearch

Install the elasticsearch by unzipping the Elasticsearch zip into your opreional system in the desired directory.

Step 3: Install some required Elasticsearch plugins.

Install the following required Elasticsearch plugins:

  • analysis-icu
  • analysis-kuromoji
  • analysis-smartcn
  • analysis-stempel

To install the plugins, go to your [Elastic_Home] and run the following command:

./bin/elasticsearch-plugin install [plugin-name]

Step 4: Name your Elasticsearch cluster.

A cluster in Elasticsearch is a collection of nodes (servers) identified as a cluster by a shared cluster name. The nodes work together to share data and workload.
 
Access the elasticsearch.yml file in your Elasticsearch installation directory:
[Elasticsearch Home]/config/elasticsearch.yml
 
Uncomment the cluster .name property and place the name of your cluster.
 

Step 5: Configure discovery properties

There are two important discovery settings that should be configured.
 
discovery.zen.ping.unicast.hosts
 
By default, without any network configuration, Elasticsearch connects to available loopback addresses and scans ports 9300 to 9305 to attempt to connect to other nodes running on the same server. This provides an automatic clustering experience without having to do any configuration.
 
When it's time to create a cluster with nodes on other servers, you'll need to provide a list of IPs from other nodes in the cluster that are likely to be active and contactable. For my lab, it was not necessary to configure this property.
 
discovery.zen.minimum_master_nodes
 
To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.
 
 

Step 6. Configurar a propriedade network.host

network.host: To form a cluster with nodes on other servers, your node will need to bind to a non-loopback address. Although there are many network configurations, generally all you need to configure is network.host.

For my environment, I left the defaults values configured.

NOTE: If you will configure X-Pack Security, the IP configured here should be used to create the certificates and keys.

Step 7: Configure Liferay DXP to connect to your Elasticsearch cluster.

Through the Liferay control panel, let's configure it to connect to your remote Elasticsearch, go to:
 
Control panel → Settings → System Settings → Search.
 
Click Elasticsearch 6 in the list of settings. Now you can configure it.
 
Here are the configuration options to change:
 
Cluster Name: Place the name defined in Elasticsearch in the cluster.name property of the file elasticsearch.yml
Operation Mode: Switch to REMOTE to connect to Elasticsearch.
Transport Addresses: Enter a delimited list of transport addresses for the Elasticsearch nodes. Here, you enter the transport address of the Elasticsearch server that you started. The default value is localhost:9300, which will work, if your elasticsearch is running locally.
 

Step 8: Restart Liferay DXP and reindex your search indexes.

  • Stop Liferay
  • Start Elasticsearsh:
    • ./bin/ elasticsearch
  • Start Liferay
  • Reindex all search indexes:
    • Control Panel → Setup → Search → Execute Reindex all Search Index.

Installing Liferay Enterprise Search

The X-Pack is an extension of Elasticsearch to protect and monitor Elasticsearch clusters. If you use Elasticsearch, X-Pack's security features include Elasticsearch cluster data access authentication and Elasticsearch internal and external communications encryption.

A Liferay Enterprise Search Premium subscription lets you access the two connectors, the X-Pack Connector and the Monitoring Connector. A Liferay Enterprise Search Standard subscription provides the monitoring integration, you only have access to the Monitoring Connector.

NOTE: The X-Pack comes out-of-the-box within the elasticsearch bundle.

  • Download the connectors according to your subscription:
    • Enterprise Search Standard
    • Enterprise Search Premium
  • Deploy connectors in Liferay at:
    • [Liferay_Home]/Deploy
  • Restart Liferay

Enabling X-Pack Security

Before start the X-pack security enablement process you need the licenses, if you do not already have a subscription, you can generate a 30-day trial license.

Follow the steps in the link below to generate a 30-day trial license:

https://www.elastic.co/guide/en/elasticsearch/reference/6.5/start-trial.html

The first thing to do is enable X-Pack security.

  1. Add the following property to the elasticsearch.yml file of each node in your elasticsearch cluster:
    • xpack .security.enabled: true
  2. Restart the elasticsearch nodes

Setting Up X-Pack Users

 On a system that uses X-Pack Security and X-Pack Monitoring, these internal X-Pack users are important:

  • kibana
  • elastic

To create the passwords for these users run the following command:

./bin/ elasticsearch -setup-passwords interactive

NOTE: If a message that the elastic user password has already been changed appears, you can see this link to resolve.

Enabling Transport Layer Security

  • Generate Node Certificates
    • Create a certificate authority, using X-Pack’s certutil command:

      • ./bin/elasticsearch-certutil ca --pem --ca-dn CN=localhost
      • This generates a ZIP file. Extract the contents in the [Elasticsearch Home]/config folder.

    • Generate X.509 certificates and private keys using the CA created:
      • ./bin/elasticsearch-certutil cert --pem --ca-cert /path/to/ca.crt --ca-key /path/to/ca.key --dns localhost --ip 127.0.0.1 --name localhost
      • This generates another ZIP file. Extract the contents in the [Elasticsearch Home]/config folder.

NOTE: Use the IP that was configured in the network.host property of the elasticsearch.yml file

  • Enable TLS

On each node in the elasticsearch.yml file add the following properties:

xpack.ssl.certificate: /path/to/[Elasticsearch Home]/config/localhost.crt
xpack.ssl.key: /path/to/[Elasticsearch Home]/config/localhost.key
xpack.ssl.certificate_authorities: ["/path/to/ca.crt"]
  • Enable transport layer TLS (on each node in the file elasticsearch.yml add the following properties):
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
  • Enable TLS on the HTTP layer to encrypt client communication (on each node in the file elasticsearch.yml add the following property):

xpack.security.http.ssl.enabled: true

Configure the Liferay Connector to X-Pack Security

  • Create the following file in [Liferay_Home]/osgi/configs:

com.liferay.portal.search.elasticsearch6.xpack.security.internal.configuration.XPackSecurityConfiguration.config

  • Add the following properties:
sslKeyPath="/path/to/[Elasticsearch Home]/config/localhost.key"
sslCertificatePath="/path/to/[Elasticsearch Home]/config/localhost.crt"
certificateFormat="PEM"
requiresAuthentication="true"
username="elastic"
password="liferay"
sslCertificateAuthoritiesPaths="/path/to/[Elasticsearch Home]/config/ca.crt"
transportSSLVerificationMode="certificate"
transportSSLEnabled="true"
 
NOTA: Remember to change the path of certificates and keys
 

Configure Liferay Enterprise Search Monitoring

Monitoring is enabled in Elasticsearch by default, but data collection is not. Enable data collection by adding this line to elasticsearch.yml:
  • xpack .monitoring.collection.enabled: true

Install Kibana

  • Download Kibana according to the version of elasticsearch you are using;
  • Unzip Kibana in the desired directory;
  • Tell Kibana which elastic you will monitor through the kibana.yml file;
    • elasticsearch.url: "http://localhost:9200"

NOTE: If SSL is enabled on Elasticsearch, this is an https URL.

If you’re using X-Pack’s security features on the Elasticsearch server.

CONFIGURE KIBANA WITH AUTHENTICATION

If X-Pack requires authentication to access the Elasticsearch cluster, follow these steps:

  • Set the password for the built-in kibana user in [Kibana Home]/config/kibana.yml:
    • elasticsearch.username: "kibana"
    • elasticsearch.password: "liferay"

NOTES:

  1. The password used in this step was created before the topic above Setting Up X-Pack Users.
  2. Once Kibana is installed, you can change the built-in user passwords from the Management user interface.

CONFIGURING KIBANA WITH ENCRYPTION

Add these settings to kibana.yml:

xpack.security.encryptionKey: "xsomethingxatxleastx32xcharactersx"
xpack.security.sessionTimeout: 600000
 
elasticsearch.ssl.verificationMode: certificate
elasticsearch.url: "https://localhost:9200"
elasticsearch.ssl.certificateAuthorities: [ "/path/to/ca.crt" ]
 
server.ssl.enabled: true
server.ssl.certificate: /path/to/[Elasticsearch Home]/config/localhost.crt
 
server.ssl.key: /path/to/[Elasticsearch Home]/config/localhost.key
 
NOTE: Remember to change the path of certificates and keys

Configuring the Liferay Connector to X-Pack Monitoring

  • Once the connector is installed and Kibana and Elasticsearch are securely configured, create a configuration file named:

com.liferay.portal.search.elasticsearch6.xpack.monitoring.web.internal.configuration.XPackMonitoringConfiguration.config

  • Place these settings in the .config file:
kibanaPassword="liferay"
kibanaUserName="elastic"
kibanaURL="http://localhost:5601"
 
NOTE: The values depend on your Kibana configuration. For example, use a secure URL such as kibanaURL="https://localhost:5601" if you’re using X-Pack Security features.
  • Deploy this configuration file to [Liferay Home]/osgi/configs, and your running instance applies the settings. There’s need to restart the server.

IF YOU WANT TO USE LIFERAY X-PACK MONITORING PORTLET INSTEAD THE KIBANA ADMIN DASHBOARD, CONFIGURE THE FOLLOWING SETTINGS

  • There are two more settings to add to Kibana itself. The first forbids Kibana from rewriting requests prefixed with server.basePath. The second sets Kibana’s base path for the Monitoring portlet to act as a proxy for Kibana’s monitoring UI. Add this to kibana.yml:
    • server.rewriteBasePath: false
    • server.basePath: "/o/portal-search- elasticsearch - xpack -monitoring/ xpack -monitoring-proxy"
NOTE: Once you set the server.basePath, you cannot access the Kibana UI through Kibana’s URL (e.g., https://localhost:5601). All access to the Kibana UI is through the Monitoring portlet.
  • Because you’re using the Monitoring portlet in Liferay DXP as a proxy to Kibana’s UI, if you are using X-Pack Security, you must configure the application server’s startup JVM parameters to recognize a valid truststore and password.
  • Navigate to Elasticsearch Home and generate a PKSC#12 certificate from the CA you created when setting up X-Pack security:

./bin/elasticsearch-certutil cert --ca-cert /path/to/ca.crt --ca-key /path/to/ca.key --ip 127.0.0.1 --dns localhost --name localhost --out /path/to/Elasticsearch_Home/config/localhost.p12

NOTE: If the network.host property of the elasticsearch.yml file has been configured, you must use it here.

  • Next  use the  keytool  command to generate a  truststore :

keytool -importkeystore -deststorepass liferay -destkeystore /path/to/truststore.jks -srckeystore /path/to/Elasticsearch_Home/config/localhost.p12 -srcstoretype PKCS12 -srcstorepass liferay

  • Add the trustore path and password to your application server’s startup JVM parameters. Here are example truststore and path parameters for appending to a Tomcat server’s CATALINA_OPTS:

-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=liferay

Monitoring in Liferay DXP

Once Kibana and X-Pack are successfully installed and configured and all the servers are running, add the X-Pack Monitoring portlet to a page:
 
  1. Open the Add menu on a page and choose Widgets

  2. Search for monitoring and drag the X-Pack Monitoring widget from the Search category onto the page.

References

https://help.liferay.com/hc/en-us/articles/360017896852-Installing-Liferay-Enterprise-Search-
https://www.elastic.co/guide/en/elasticsearch/reference/6.5/index.html
https://qbox.io/blog/split-brain-problem-elasticsearch

Blogs

Thank you so much for such a blog full of information.Thank you