Do your own Analytics in Liferay with Elastic Search and Kibana

Liferay integrates out of the box with Piwik and Google Analytics.

However, doing analytics with ElasticSearch, Logstash and Kibana, is not much harder:


https://youtu.be/os5gqpnC0GA

How to do it?
Easywink

First, we need to get the data:

Offloading work to the user browsers using Ajax seems to be the most logical way to do it (imagine hundreds of concurrent users clicking on things and moving their mouse at the same time), to not affect our server's performance. That's how Piwik, Google Analytics, and Omniture work.

Something similar to this (https://raw.githubusercontent.com/roclas/analytics-storage-server/logstash/javascript_data_collection/hover_and_clicks.js) could do the job.

It would also make sense to not give the burden of receiving these Ajax requests to our Liferay server. Our application server is a bit heavy, and something smaller (and more scalable) would do a better job for this simple task.


What about a pool of threads that work asynchronously?enlightened

This project (https://github.com/roclas/analytics-storage-server/tree/filesystem) basically is that. A pool of asynchronous threads that act as an HTTP server. It is scalable and fast. It could get all our Ajax events and store them in files (so that they can be later analyzed in batch using machine learning, big data, etc).

 

What about visualization? Where is the data analysis here?

 

This is where the second part of our problem starts; we are able to collect the data, but now we have to analyze it and show graphs and pie charts.

In this other branch (https://github.com/roclas/analytics-storage-server/tree/logstash) what the server is doing is piping all the events into Logstash, which will insert them into Elastic Search.


Once our data is in Elastic Search, we just have to point Kibana to our Index and start creating dashboards and playing with charts.yes

Only if you are interested in the details, this video shows how everything works more in detail (it is also probably a bit too long and boring): https://youtu.be/NMPWR2vdnio