Do your own Analytics in Liferay with Elastic Search and Kibana

Company Blogs May 31, 2018 By Carlos Hernandez Staff

Liferay integrates out of the box with Piwik and Google Analytics.

However, doing analytics with ElasticSearch, Logstash and Kibana, is not much harder:


https://youtu.be/os5gqpnC0GA

How to do it?
Easywink

First, we need to get the data:

Offloading work to the user browsers using Ajax seems to be the most logical way to do it (imagine hundreds of concurrent users clicking on things and moving their mouse at the same time), to not affect our server's performance. That's how Piwik, Google Analytics, and Omniture work.

Something similar to this (https://raw.githubusercontent.com/roclas/analytics-storage-server/logstash/javascript_data_collection/hover_and_clicks.js) could do the job.

It would also make sense to not give the burden of receiving these Ajax requests to our Liferay server. Our application server is a bit heavy, and something smaller (and more scalable) would do a better job for this simple task.


What about a pool of threads that work asynchronously?enlightened

This project (https://github.com/roclas/analytics-storage-server/tree/filesystem) basically is that. A pool of asynchronous threads that act as an HTTP server. It is scalable and fast. It could get all our Ajax events and store them in files (so that they can be later analyzed in batch using machine learning, big data, etc).

 

What about visualization? Where is the data analysis here?

 

This is where the second part of our problem starts; we are able to collect the data, but now we have to analyze it and show graphs and pie charts.

In this other branch (https://github.com/roclas/analytics-storage-server/tree/logstash) what the server is doing is piping all the events into Logstash, which will insert them into Elastic Search.


Once our data is in Elastic Search, we just have to point Kibana to our Index and start creating dashboards and playing with charts.yes

Only if you are interested in the details, this video shows how everything works more in detail (it is also probably a bit too long and boring): https://youtu.be/NMPWR2vdnio 

 

Customize Elastic Search and Search by Synonyms in Liferay

Technical Blogs February 24, 2018 By Carlos Hernandez Staff

Q-Have you ever wondered if you can customize your Elastic Search, so that on your searches in Liferay, not only the words you're searching for come up in the results but also synonyms of these words?

A-Yes, you can!

Below, I'll show you an example of how I did it (by customizing my indexes and mappings settings).

Let's say that I have a web content article containing the word "small" and I search for the word "tiny".

 
  1. Navigate to Control Panel → Configuration → System Settings → Foundation
     
  2. Search for the com.liferay.portal.search.elasticsearch.configuration.ElasticsearchConfiguration system setting.
     
  3. Go to "Aditional index configurations" and add your own.

    You can start by copying the contents of your index-settings.json file there. Your index-settings.json file is packed in the Elastic Search module of your Liferay bundle.

    Now, modify it as this document describes: https://www.elastic.co/guide/en/elasticsearch/guide/current/using-synonyms.html .

    It should look like the image below (note the "synonyms" part of the json, where we wrote our list of synonyms):
      Inline image 1
     
  4. Now go to "Override type mappings" and copy into the text area input the contents of the file "liferay-type-mappings.json", also packed in the Elastic Search module (a jar that you will find in your Liferay bundle) into it.

    Modify it, also following the steps this document describes: https://www.elastic.co/guide/en/elasticsearch/guide/current/using-synonyms.html (as in the previous step).

    For example, you could add something like this:
      "content": {
         "index": "analyzed",
         "store": "yes",
                   "search_analyzer" : "my_synonym_analyzer",
                   "analyzer" : "my_synonym_analyzer",
                   "term_vector": "with_positions_offsets",
                   "type": "string"
      },
    and something like this:
     "title": {
    "index": "analyzed",
    "store": "yes",
                   "search_analyzer" : "my_synonym_analyzer",
                   "analyzer" : "my_synonym_analyzer",
    "term_vector": "with_positions_offsets",
    "type": "string"
    },
    to it:
    Inline image 2
      
     
  5. Save your changes
     
  6. Navigate to Control Panel → Configuration → Server Administration and execute "Reindex all search indexes" under the section "Index Actions"
    Inline image 3

     
  7. Perform a search and... Voila!, the magic happens:
    Inline image 4

    Easy, right?

    If Elastic Search can do it, Liferay will do it too (since it leverages on Elastic Search for indexing its documents). You just need to know it can be done, and where in the control panel you can configure it.
     

Liferay DXP and machine learning; Liferay as an integration platform

Company Blogs July 3, 2017 By Carlos Hernandez Staff

Wouldn't it be nice to take pictures with my mobile phone and let Liferay categorize /  organise these pictures for me?

 

Imagine how powerful it would be if I am on a business trip and I take lots pictures with my phone (pictures of the places I see to show them to my friends, pictures of food so that I can show my mom that I'm eating well, and my receipts so that I can expense them later ) and when I'm back home, I go to my document repository and I search for the word "RECEIPT", and since all my receipts have been magically tagged as "receipt", they will all come up on my search result.

 

Does it sound like science fiction?

Now, look at this video because this is exactly the principle we use for this kind of problem:  https://youtu.be/ShAUafF2yfw (the video quality is not perfect but you get the idea. Right?).

 

Liferay is the perfect platform for this problem because:

  1. It comes with a document repository
  2. It can be virtually integrated with anything
  3. It is omnichannel and allows you to use your mobile phone to create documents
  4. It is highly customizable (below I'll show you how I did it)

 

Just to mention,  Liferay Sync is a document sharing functionality for your Liferay system; in this case, it is what allows me to share my mobile pictures with my Liferay server.

 

 

How did I do it?

Easy!!

You can take a look at the code here: https://github.com/roclas/liferay7ClarifaiDocumentClassifierModelListener , but just to make it even easier for you, I will summarise: what I am doing is using a model listener:

@Component(immediate = true, service = ModelListener.class)
public class ClassifyingDocumentListener extends BaseModelListener<AssetEntry>{...

Every time an AssetEntry is created (every time an asset is created), the code will check if it is a document, and if it is, it will extract the bytes of this document and send them to the external API (Clarifai: https://www.clarifai.com/ ). Clarifai will give us information about the picture in an object, and the model listener will use tags to classify my document.

As simple as that!

Please ask if you have any questions.

 

As you can see, the hardest part was to think of an interesting use case, and not about the implementation itself (and the idea was not even mine; thank you, John Feeney and Filipe Afonso; please keep sharing more cool ideas like this one).

 

 

 

What next?

Do you think it would be interesting for some people to put this in the marketplace?

Liferay is an integration platform and makes integration with virtually any library / API possible. We could almost say that: "if it runs on Java it can be put into an OSGi module and deployed into Liferay".

The possibilities for integration use cases are endless.

Do you have any similar idea? What would you like to see Liferay integrating with?

Showing 3 results.