Creating a Google Like Search Part V: Finale

Company Blogs November 27, 2017 By Petteri Karttunen Staff

Previous parts of this series can be found here (part 1),  here (part 2), here (part 3) and here (part 4).

In the final part of this blog series few more interesting features are added to the previously created search portlet: possibility to use Liferay Audience Targeting to make segmented content more relevant, possibility to configure sort and facet fields (to any indexed fields) and fully configure search fields and their boosts. There’s also a possibility to make non-Liferay content findable through this search portlet.

There were also quite a few generic improvements I made along the way so in the end of the day, we have a custom Liferay search portlet with following features:

  • Google like appearance 
  • Completely ajaxed interface (no page transitions)
  • 3 selectable search result layouts (image card layout available for files / images)
  • Sortable search results (not available in default Liferay search)
  • Bookmarkable searches with short urls which can easily be collected to Google Analytics
  • Autocompletion & query suggestions
  • Automatic alternate search
  • Support for Boolean operators and Lucene syntax
  • Configurables:
    • Asset types to search for
    • Facets to retrieve
    • Sort fields
    • Fields to search for and their boosting
    • etc.
  • Audience targeting support to boost segment matching contents
  • Ability to include non-Liferay resources in the search results

I also added there few notes how to make this work on CE. Depending on the interest that could  be on the roadmap anyways. I also splitted the application in separate modules for a cleaner architecture. So for example, If you’d just like to use the backend and build your own UI that’s also possible now.

Screenshots

Basic Functionality

Results Image Layout

Non-Liferay Assets in the  Search Results

Configuration

Customizing the Liferay Elasticsearch Adapter

Why would you want to modify Liferay search adapter? Search adapter implementation in Liferay is responsible for implementing the portal search API for a specific search engine. It takes care for example about communication link between the portal and search engine, about implementation of index searchers and writers and about translating the portal search queries to native engine specific queries. Following diagram illustrates the layering roughly:

 

Following diagram on the other hand, illustrates the physical placement of search functionalities in bundles and modules:

 

In this project I had to customize the adapter to get a full support of the most versatile and powerful single query type, Elasticsearch QueryStringQuery.

Liferay search API evolves all the time, but at the moment, support for the QueryStringQuery is sparse. It allows only setting the query string but doesn’t let you to control any of the other, about 30 parameters. Using that query type you cannot thus control for example boosting or fields to target the query to, fuzziness etc.

So, for this purpose I did two things. First I created a new search query type QueryStringQuery extending Liferay’s standard query type StringQuery. This new query type is introduced in gsearch-query-api package. You can think that as an Liferay search API extension.

The other thing I did, was extending the Elasticsearch adapter. If creating a new query type was super easy, this was not. Sure you can write your own adapter but how about just extending a standard one even just a little? Extending search adapter is currently not flexible in a way it could be. When you take a look at the Elasticsearch adapter source code, especially the ElasticsearchQueryTranslator you see there service references to the invidual query translators. The first thought would be to use the extension points, to just create an alternative implementation of StringQuery translator with higher service ranking to replace the standard translator. That would be the way I would like to do that. That way, we wouldn’t have to modify the adapter at all and we would keep the maintainability of our portlet and portal as it is. That’s however not possible, at least at the moment, because of two reasons. First the  references in the ElasticsearchQueryTranslator are by default using STATIC and RELUCTANT injection.

The other problem is, that the Elasticsearch adapter module is only exposing the com.liferay.search.elasticsearch.settings subpackage. So, if you need to reference any package inside the adapter in your custom service, you will get an import-package error at deploy time because those referenced packages are private. David Nebinger wrote an excellent writing of overcoming package access restrictions but actually there’s a third minor problem: that adapters dependencies are not all OSGI compliant but included in the module. So using for example Elasticsearch classes from you custom service, let’s say from a fragment module, leads currently to class loading problems. Hopefully these limitations will be taken care of in the future but at the moment, customization of the adapter source code seems not to be avoidable for extending practically anything of search adapters functionality.

You can see the details of my adapter customization implementation in Github. Basically I created a custom StringQuery translator implementation which, in case of our extended StringQuery type ”QueryStringQuery” will use a specialized translator for that an in case of the standard StringQuery falls to the default implementation.

As there will be an official Liferay support for Elasticsearch 6 in the near future, I decided not to upgrade the Elasticsearch adapter for 5.6.

Making Search More Intelligent

I was not completely satisfied with the relevancy of portal standard search and wanted to play with the idea of improving hits relevancy by means of making field options, like boosting, configurable and implementing some machine learning features.  I tried to make the API easily customizable so you can implement there your own features.

So one of the first thoughts was that it would be great to integrate Audience Targeting feature to search. That way you could boost contents segmented to your user segments but even better, you would have a dynamic way to control search hits relevancy.  As this is a DXP feature only you can enable and disable it in the configuration options. By default, it’s disabled.

How does it work? If current user is segmented to any user segment it adds a condition to the query giving the configure boost for any contents matching that segment. 

Making non Liferay Resources Findable

When starting this project, I planned to do some experiments on getting non Liferay assets to search results, in conjunction to search engine federation, but decided to leave federation out of the scope as it’s something that usually should be transparent to the client (portal in this case). I just mention here that both Elasticsearch and SOLR have means to make that possible.

How to make search to find things in the index that are not Liferay assets. For example, you might have several integration points in your portal, having resources that should be findable by Liferay. Usually the options are: make those resources portal assets so that they can be found by portal search or, create a dedicated, custom search for these external resources. Both of these solutions have lots of challenges or usability issues so how about getting everything on the same result list.

In this simple imaginary scenario external resources are being indexed to the portal search index.  So basically what you have to do, is to take care that these external documents just have the fields needed for our custom search to find them. You can find an example and more information on the GitHub page.

Just a reminder here that this solution only works with our custom portlet here, not with the standard Liferay portlet. Also, in a real world scenario indexing documents in the Liferay portal index would not be recommendable by any meansTo make our custom portlet to find resources in custom indexes, you would just need to customize the Elasticsearch search adapter a little bit further, mainly the ElasticsearchIndexSearcher class where the indexes to search for are being determined.

So that’s it for this blog series. For more information and details please see the project Github page.

Thanks,,,

 

Creating a Google Like Search Part IV: Query Tuning and Lucene Syntax

Technical Blogs October 30, 2017 By Petteri Karttunen Staff

(Previous parts of this series can be found here (part 1),  here (part 2) and here (part 3)

As a random, Liferay custom search developer you might want to get support for Boolean operators and|or Lucene syntax back. Also, it would be nice to have more control of hits relevancy and about the logic, the queries are being built in the end. To get those, we are going a little deeper than before.

Liferay SearchContext is a wonderful transport assistant made for you if you want to implement your custom search easily. You just put your keywords and filtering options there, throw it to, typically FacetedSearcher, and that’s it. But there’s a counter side for this easiness, the amount of control.

When you do like just described, you don’t have control which fields are being searched for in different assets. Nor can you control which query type to use. Especially the query type matters. You may have wondered how standard search gives sometimes even surprising results. Well, it’s because of three things. First, it relies mostly on Elasticsearch MatchQuery (for more information please see the Elasticsearch Query DSL documentation). MatchQuery is the all-rounder of the query types matching, for my likings, too broadly everything. The other thing is that when you execute search, fields are being targeted with multiple different types of SHOULD queries: a WildCard, Match, Phrase and Phrase Prefix –queries. The third thing is that queries are targeted to fields that you, as a guest user, might not be even aware about, like assetTagNames and assetCategoryNames.

So for example a search with phrase “lorem ipsum” transforms through standard search portlet into something like this (2229 lines):


{
  "from": 0,
  "size": 20,
  "query": {
    "bool": {
      "must": {
        "bool": {
          "must": [
            {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "assetCategoryTitles_en_US": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "assetCategoryTitles_en_US": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "assetTagNames": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "assetTagNames": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "comments": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "comments": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "comments": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "content": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "content": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "content": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "match": {
                          "description": {
                            "query": "lorem ipsum",
                            "type": "boolean"
                          }
                        }
                      },
                      "should": [
                        {
                          "match": {
                            "description": {
                              "query": "lorem ipsum",
                              "type": "phrase",
                              "slop": 50
                            }
                          }
                        },
                        {
                          "match": {
                            "description": {
                              "query": "lorem ipsum",
                              "type": "phrase",
                              "boost": 2
                            }
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "properties": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "properties": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "properties": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "title": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "title": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "title": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "url": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "url": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "url": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "userName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "userName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "ddmContent": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "extension": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "extension": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "extension": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "fileEntryTypeId": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "fileEntryTypeId": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "fileEntryTypeId": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "path": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "path": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "city": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "city": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "city": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "country": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "country": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "country": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "emailAddress": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "emailAddress": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "firstName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "firstName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "firstName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "fullName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "fullName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "fullName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "lastName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "lastName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "lastName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "middleName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "middleName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "middleName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "region": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "region": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "region": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "screenName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "screenName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "street": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "street": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "street": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "zip": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "zip": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "zip": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "userName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "userName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "ddmContent": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "articleId": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "articleId": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "articleId": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "classPK": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "classPK": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "classPK": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "content_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "content_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "content_en_US": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "description_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "description_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "description_en_US": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "entryClassPK": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "entryClassPK": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "entryClassPK": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "title_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "title_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "title_en_US": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "userName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "userName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "term": {
                "groupId": "20143"
              }
            }
          ]
        }
      },
      "filter": {
        "bool": {
          "must": {
            "term": {
              "companyId": "20116"
            }
          },
          "should": [
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.wiki.model.WikiPage"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.document.library.kernel.model.DLFileEntry"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.portal.kernel.model.User"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.bookmarks.model.BookmarksFolder"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.blogs.kernel.model.BlogsEntry"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.document.library.kernel.model.DLFolder"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.dynamic.data.lists.model.DDLRecord"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.journal.model.JournalArticle"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.bookmarks.model.BookmarksEntry"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.journal.model.JournalFolder"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.calendar.model.CalendarBooking"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.message.boards.kernel.model.MBMessage"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
  },
  "post_filter": {
    "bool": {
      "must": [
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.wiki.model.WikiPage"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "hidden": "false"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.document.library.kernel.model.DLFileEntry"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "status": "0"
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.portal.kernel.model.User"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.bookmarks.model.BookmarksFolder"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.blogs.kernel.model.BlogsEntry"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "hidden": "false"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.document.library.kernel.model.DLFolder"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "status": "0"
                      }
                    },
                    {
                      "term": {
                        "recordSetScope": "0"
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.dynamic.data.lists.model.DDLRecord"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "head": "true"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.journal.model.JournalArticle"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.bookmarks.model.BookmarksEntry"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.journal.model.JournalFolder"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": {
                    "bool": {
                      "must": [
                        {
                          "bool": {
                            "should": {
                              "term": {
                                "entryClassName": "com.liferay.calendar.model.CalendarBooking"
                              }
                            }
                          }
                        },
                        {
                          "bool": {
                            "should": [
                              {
                                "term": {
                                  "userId": "20120"
                                }
                              },
                              {
                                "terms": {
                                  "roleId": [
                                    "20123"
                                  ]
                                }
                              }
                            ]
                          }
                        }
                      ]
                    }
                  }
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "discussion": "false"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.message.boards.kernel.model.MBMessage"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "terms": {
                  "groupId": [
                    "20143"
                  ]
                }
              },
              {
                "terms": {
                  "scopeGroupId": [
                    "20143"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  },
  "fields": "*",
  "track_scores": true,
  "aggregations": {
    "assetTagNames.raw": {
      "terms": {
        "field": "assetTagNames.raw",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "assetCategoryIds": {
      "terms": {
        "field": "assetCategoryIds",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "entryClassName": {
      "terms": {
        "field": "entryClassName",
        "min_doc_count": 1
      }
    },
    "groupId": {
      "terms": {
        "field": "groupId",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "modified": {
      "range": {
        "field": "modified",
        "ranges": [
          {
            "from": "20171025080000",
            "to": "20171025100000"
          },
          {
            "from": "20171024090000",
            "to": "20171025100000"
          },
          {
            "from": "20171018090000",
            "to": "20171025100000"
          },
          {
            "from": "20170925090000",
            "to": "20171025100000"
          },
          {
            "from": "20161025090000",
            "to": "20171025100000"
          }
        ]
      }
    },
    "userName": {
      "terms": {
        "field": "userName",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "folderId": {
      "terms": {
        "field": "folderId",
        "size": 10,
        "min_doc_count": 1
      }
    }
  },
  "highlight": {
    "pre_tags": [
      ""
    ],
    "post_tags": [
      ""
    ],
    "require_field_match": true,
    "fields": {
      "description": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "description_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "title": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "title_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "assetCategoryTitles": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "assetCategoryTitles_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "content": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "content_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      }
    }
  }
}

Run through our custom portlet it's 140 rows and looks like:


{
   "from":0,
   "size":10,
   "query":{
      "bool":{
         "must":{
            "bool":{
               "must":{
                  "query_string":{
                     "query":"lorem ipsum"
                  }
               }
            }
         },
         "filter":{
            "bool":{
               "must":[
                  {
                     "term":{
                        "companyId":"20116"
                     }
                  },
                  {
                     "term":{
                        "stagingGroup":"false"
                     }
                  },
                  {
                     "term":{
                        "status":"0"
                     }
                  },
                  {
                     "bool":{
                        "should":[
                           {
                              "bool":{
                                 "must":[
                                    {
                                       "term":{
                                          "entryClassName":"com.liferay.journal.model.JournalArticle"
                                       }
                                    },
                                    {
                                       "term":{
                                          "head":"true"
                                       }
                                    }
                                 ],
                                 "should":[
                                    {
                                       "range":{
                                          "displayDate_sortable":{
                                             "from":"-9223372036854775808",
                                             "to":"1509368280059",
                                             "include_lower":true,
                                             "include_upper":true
                                          }
                                       }
                                    },
                                    {
                                       "range":{
                                          "expirationDate_sortable":{
                                             "from":"1509368280059",
                                             "to":"9223372036854775807",
                                             "include_lower":true,
                                             "include_upper":true
                                          }
                                       }
                                    }
                                 ]
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.document.library.kernel.model.DLFileEntry"
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.message.boards.kernel.model.MBMessage"
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.blogs.kernel.model.BlogsEntry"
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.wiki.model.WikiPage"
                              }
                           }
                        ]
                     }
                  },
                  {
                     "bool":{
                        "should":[
                           {
                              "term":{
                                 "roleId":"20123"
                              }
                           },
                           {
                              "term":{
                                 "userId":"20120"
                              }
                           }
                        ]
                     }
                  }
               ]
            }
         }
      }
   },
   "fields":"*",
   "sort":[
      {
         "_score":{

         }
      },
      {
         "modified_sortable":{
            "order":"asc",
            "unmapped_type":"string"
         }
      }
   ],
   "track_scores":true,
   "aggregations":{
      "entryClassName":{
         "terms":{
            "field":"entryClassName"
         }
      }
   }
}

So, where did over 2000 rows go? Ok, this is not completely a fair comparison. I'm using here StringQuery which basically does on one row all the dirty work and searches for all the fields (this topic will be revisited in the next episode). Also, the standard search portlet and this one have very different goals, different emphasis and also different functionalities.

What this exercise shows however, is the strength of Liferay as development platform. The search framework is robust and flexible allowing you to self define the level of control you want to gain.

What was done?

The key things in this solution were:

  • Handling the query building manually i.e. not pushing keywords to SearchContext object
  • Building the search filters manually
  • Using IndexSearchHelper service instead of FacetedSearcher to do the work. It gives you more control.

For the rest of the stuff I'm encouraging you to see the code in Github.

What's still coming

In this part of the series we took more control over building the queries but there's more to do. At the moment the ElasticSearch StringQuery type, we are using, is not implemented fully in portal search.That's why we cannot yet control field level boosting in combination with it. 

For those purposes, in the next episode I'm going to customize the Elasticsearch adapter to support ES 5.6 and implement String Query properly. Also I'm going to show you how to use Audience Targeting in Liferay search.

The final planned episode will be about search federation. About using external systems index data through Liferay search.

Creating a Google Like Search Part III: Autocompletion / Suggestions

Technical Blogs October 30, 2017 By Petteri Karttunen Staff

(Previous parts of this series can be found here (part 1) and here (part 2)

This time we add an autocomplete / keyword suggester to the search field and query suggestions with automatic alternative search mechanism for the queries not giving any results.

First, a few words about the semantics and definitions. Autocomplete, keywords and query suggestions and spellchecking are, in many cases, mixed with each other in spoken language. So, before going to the task I’d prefer to make a slight distinction between those terms. Autocompletion, in my interpretation, is an inline completion component, like the predictive text input in mobile phones. An UI feature. Suggestions are those what an autocomplete component is offering and the challenge is getting those right. Keyword suggestions are terms or phrases offered in a form of a list to you when you type. Query suggestions are alternatives offered to you when your search doesn’t give any results. Spellchecking tends to work like keyword suggestions but its’ sole purpose is to do spell checking. From UI perspective all share a lot, of course.

How do suggesters work?

There are multiple approaches to making a suggesters.

Probably the easiest and most manageable is to use self-defined dictionaries. This is possible with Liferay even with the standard search portlet. If you take a look at portal.properties you can see there:

#index.search.query.suggestion.dictionary[en_US]=\
     com/liferay/portal/search/dependencies/querysuggestions/en_US.txt

If you define these language bound dictionaries and enable query indexing there, queries get indexed automatically to the querySuggestion Elasticsearch type and are ready for the use as query suggestions in standard Liferay search portlet.

Another simple approach would be just to make searches as you type and return suggestions in the preferred form back to UI. Obviously, this would probably kill Elasticsearch servers under heavy traffic unless you had a good caching mechanism and proper delays.

The approach we are using here for both keyword and query suggestions, and what Liferay offers out of the box, is query indexing. You make a succesful search and if the result count is above a defined threshold value the query gets indexed to the querySuggestions type in Elasticsearch. This is also the principle Google is doing it –certainly having a lots of other kind of intelligence and filtering in providing those suggestions to UI.

The best thing in this option is manageable relevancy. You can configure (in portlet configuration) the threshold level, when queries get indexed. You could then for example say that a search phrase returning only 2 results is not relevant enough to get into suggestions. Also as you have the suggestions as a dedicated index type (or in dedicated index) you can do there index level tuning to adjust analyzers and improve the relevancy for you use case. In this exercise here we are using the standard Liferay QueryIndexer with standard index settings but I’ll revisit this topic in the coming parts to show some options how you could improve the suggestions relevancy.

Doing a good keywords suggester is by no means a trivial task and different scenarios would probably need different kind on fine tuning. The solution created here is just a starting point where you can build on.

About the Solution

There are few things to keep in mind in this solution.

First, the suggestions are not persisted, they live only in index. If you reindex, they are lost and portal has to start learning suggestions again. With time however, they’ll get to the same level.

Suggestions are language bound. If you have a multilingual portal, users of different languages are not sharing same suggestions.

Suggestions management. The solution here doesn’t do any filtering, at least yet. If you search with a not nice search phrase and get results above the threshold, it gets indexed.

Query Indexing

How are the queries getting indexed? In this solution I’m bypassing much of the automation that using SearchContext and FacetedSearcher brings, to get a better low-level control of things happening. That’s one of the reasons why I had to implement trigger query indexing here by myself, using however the standard com.liferay.portal.search.internal.hits.QueryIndexingHitsProcessor.

If you are already worried about this app messing up with Liferay indexing: I’m just working here around hits processors and query indexing which are bound to the search interface.

Query indexer processor is triggered after search results have been returned, in fi.soveltia.liferay.gsearch.web.search.internal.GSearchImpl service in method getResults():

_queryIndexerProcessor.process(searchContext, _gSearchDisplayConfiguration, _queryParams, hits);

Processing happens in QueryIndexerProcessorImpl service:


@Override
public boolean process(
	SearchContext searchContext,
	GSearchDisplayConfiguration gSearchDisplayConfiguration,
	QueryParams queryParams, Hits hits)
	throws Exception {

	if (_log.isDebugEnabled()) {
		_log.debug("Processing QueryIndexer");
	}

	if (!gSearchDisplayConfiguration.enableQuerySuggestions() &&
		!gSearchDisplayConfiguration.enableAutoComplete()) {
		return true;
	}

	if (_log.isDebugEnabled()) {
		_log.debug("QueryIndexer is enabled");
	}
	
	if (hits.getLength() >= gSearchDisplayConfiguration.queryIndexingThreshold()) {

		if (_log.isDebugEnabled()) {
			_log.debug("QueryIndexing threshold exceeded. " + 
                              Indexing keywords: " + queryParams.getKeywords());
		}

		addDocument(
			queryParams.getCompanyId(), queryParams.getKeywords(),
			queryParams.getLocale());
	} else {
		if (_log.isDebugEnabled()) {
			_log.debug("QueryIndexing threshold wasn't exceeded." +
                                 " Not indexing keywords.");
		}
	}
	return true;
}

Keyword Suggester

The easiest part in doing the keyword suggester was adding the autocompletion functionality to the searchfield. I’m using here my own override of the Metal.JS autocomplete class but this override just removes the SPACE key from making a suggestion selection.

The autocomplete class GSearchAutoComplete:


import Autocomplete from 'metal-autocomplete/src/Autocomplete';

const DOWN = 40;
const ENTER = 13;
const UP = 38;

/*
 * GSearch autocomplete component extending Metal.JS autocomplete.
 */
class GSearchAutocomplete extends Autocomplete {

	/**
	 * This is an override for the original Metal.js Autocomplete.
	 * It simply removes SPACE from the select keys.
	 *
	 * @param {!Event} event
	 * @protected
	 */
	handleKeyDown_(event) {
		
		if (this.visible) {
			switch (event.keyCode) {
				case UP:
					this.activateListItem_(this.decreaseIndex_());
					event.preventDefault();
					break;
				case DOWN:
					this.activateListItem_(this.increaseIndex_());
					event.preventDefault();
					break;
				case ENTER:
					this.handleActionKeys_();
					event.preventDefault();
				break;
			}
		}
	}
}
export default GSearchAutocomplete;

Binding the autocomplete component to the search field is done in the search fields component class  GSearchFields.es.js


initAutocomplete() {
	
	let _self = this;
	
	let autocomplete = new GSearchAutocomplete ({
		elementClasses: 'gsearch-autocomplete-list',
		inputElement:document.querySelector('#' + this.portletNamespace + 'SearchField'),
		data: function(keywords) {
			if (keywords.length >= _self.getQueryParam('queryMinLength') && 
                           !_self.isSuggesting && keywords.slice(-1) != ' ') {
				return _self.getSuggestions(keywords);
			} else {
				return;
			}
		},
		select: function(keywords, event) {
			$('#' + _self.portletNamespace + 'SearchField').val(keywords.text);
		}
	});
}

Autocomplete request is being made in getSuggestions():


getSuggestions(keywords) {

	// Set this flag to manage concurrent suggest requests (delay between requests).
	
	this.isSuggesting = true;
	
	let _self = this;
	
	let params = new MultiMap();
	
	params.add(this.portletNamespace + 'q', keywords);
	
	return Ajax.request(
		this.suggestionsURL,
		'GET',
		null,
		null,
		params,
		this.requestTimeout
	).then((response) => {
			let suggestions = JSON.parse(response.responseText);

			_self.releaseSuggesting();

			return suggestions;

	}).catch(function(error) {

		_self.releaseSuggesting();

		console.log(error);
	});
}

I added there a const delay of 150ms between subsequent requests. You can change it to your likings in the class constants.

The autocomplete resource url is put to the SOY template context in fi.soveltia.liferay.gsearch.web.portlet.action.ViewMVCRenderCommand


template.put(
	GSearchWebKeys.SUGGESTIONS_URL,
	createResourceURL(renderResponse, GSearchResourceKeys.GET_SUGGESTIONS));

Next thing to do was to implement a resource command action fi.soveltia.liferay.gsearch.web.portlet.action.GetSuggestionsMVCResourceCommand. In that class I’m injecting the suggester service:


@Reference
protected GSearchKeywordSuggester _gSearchSuggester;

...and doing the suggestions:


JSONArray response = null;

try {
	response = _gSearchSuggester.getSuggestions(
		resourceRequest,
		_gSearchDisplayConfiguration);
}
catch (Exception e) {

	_log.error(e, e);

	return;
}

The real work is then done in the fi.soveltia.liferay.gsearch.web.search.internal.suggest. GSearchKeywordSuggesterImpl service.

For the suggestions I chose to use phrase suggester to be able to suggest complete search phrases. Among other options are TermSuggester, which gives results in a single term array and Aggregate suggester which allows you to combine different kind of suggestions.

Query suggestions

For the query suggestions i.e. search phrase suggestions after getting no results, I again used slightly customized version of the standard com.liferay.portal.search.internal.hits.QuerySuggestionHitsProcessor - which is using the same phrasesuggester as the searchfield suggester. Of the same reasons mentioned in section Query Indexing suggestions are processed manually fi.soveltia.liferay.gsearch.web.search.internal.query.processor.QuerySuggestionsProcessorImpl

What QuerySuggestionsProcessor service does is basically finding viable alternative search queries, making an alternative search based on one of those and additionally, if configured, offering other possible alternatives to the UI


public boolean process(
	PortletRequest portletRequest, SearchContext searchContext,
	GSearchDisplayConfiguration gSearchDisplayConfiguration,
	QueryParams queryParams, Hits hits)
	throws Exception {

	if (_log.isDebugEnabled()) {
		_log.debug("Processing QuerySuggestions");
	}
	
	if (!gSearchDisplayConfiguration.enableQuerySuggestions()) {
		return true;
	}

	if (_log.isDebugEnabled()) {
		_log.debug("QuerySuggestions are enabled.");
	}
		
	if (hits.getLength() >= gSearchDisplayConfiguration.
             querySuggestionsHitsThreshold()) {
		
		if (_log.isDebugEnabled()) {
			_log.debug("Hits threshold was exceeded. Returning.");
		}

		return true;
	}

	if (_log.isDebugEnabled()) {
		_log.debug("Below threshold. Getting suggestions.");
	}
	
	// Have to put keywords here to searchcontext because
	// suggestKeywordQueries() expects them to be there

	searchContext.setKeywords(queryParams.getKeywords());

	if (_log.isDebugEnabled()) {
		_log.debug("Original keywords: " + queryParams.getKeywords());
	}
	
	// Get suggestions
	
	String[] querySuggestions = _gSearchSuggester.
          getSuggestionsAsStringArray(portletRequest, gSearchDisplayConfiguration);

	querySuggestions =
		ArrayUtil.remove(querySuggestions, searchContext.getKeywords());

	if (_log.isDebugEnabled()) {
		_log.debug("Query suggestions size: " + querySuggestions.length);
	}
	
	// Do alternative search based on suggestions (if found)
	
	if (ArrayUtil.isNotEmpty(querySuggestions)) {

		if (_log.isDebugEnabled()) {
			_log.debug("Suggestions found.");
		}
		
		// New keywords is plainly the first in the list.

		queryParams.setOriginalKeywords(queryParams.getKeywords());

		if (_log.isDebugEnabled()) {
			_log.debug("Using querySuggestions[0] for alternative search.");
		}

		queryParams.setKeywords(querySuggestions[0]);
		
		Query query = _queryBuilder.buildQuery(portletRequest, queryParams);

		BooleanClause booleanClause = BooleanClauseFactoryUtil.create(
			query, BooleanClauseOccur.MUST.getName());

		searchContext.setBooleanClauses(new BooleanClause[] {
			booleanClause
		});

		Hits alternativeHits =
			_indexSearcherHelper.search(searchContext, query);
		hits.copy(alternativeHits);
	}

	hits.setQuerySuggestions(querySuggestions);

	return true;
}

About the Configuration Options

Configuration options matter a lot for the suggestions. I added there links to related Elasticsearch documents in the configuration options but saying here already, especially the confidence level is important. Basically, lower you put that, more suggestions you get. But error margin grows.

The code, again, can be found on Github https://github.com/peerkar/liferay-gsearch. Please see the Requirements section in Readme for this module to work.

Creating a Google Like Search Part II: Filter by Structure and Document Type

Technical Blogs October 29, 2017 By Petteri Karttunen Staff

Creating a Google Like Search Part II: Filter by Structure and Document Type

(Previous part of the series can be found here)

In the second part of this blog series I’ll be adding new filtering capabilities to the portlet created in the first part. Filters added are filter by web content structure and filter by file document type and extension. That way, if you have for example defined a web content structure “News” or a document type, let’s say “Contract”, you can search only for those. Filtering by document extension is also there so you can search only for “Contracts” of type PDF for example.

You may notice on the project Github page that the codebase looks a little different now. I did some streamlining and refactored the code to make more use of OSGI declarative services. I also added there a search support for Wiki pages and made the selection of supported types configurable.

How it was done?

Starting here with the UI, I first added there in GSearchFilters.soy the Twitter Bootstrap dropdown components having the dropdown skeletons:

<div class="dropdown gsearch-dropdown filter wcfilter hide">
	<button 
      aria-expanded="true" 
      aria-haspopup="true" 
      class="btn btn-link dropdown-toggle" 
      data-toggle="dropdown" 
      id="{$portletNamespace}WebContentStructureFilter" 
      type="button">
						
      <span class="selection"></span>
      <span class="caret"></span>
   </button>

   <ul
      aria-labelledby="{$portletNamespace}WebContentStructureFilter" 
      class="dropdown-menu" 
      id="{$portletNamespace}WebContentStructureFilterOptions">
						
      <li class="selected">
                <a data-value="" href="#">{msg desc=""}any-web-content-structure{/msg}</a>
      </li>
						
      {call .options}
         {param options: $webContentStructureOptions /}
      {/call}
   </ul>
</div>		

There’s also a closure template in the same file for rendering the options items which are coming from the template context:

/**
 * Print dropdown options.
 *
 * @param options
 */
{template .options}
	{foreach $item in $options}
		{if $item.scope == 'all'}
			<li class="all">		
		{else}
			<li>
		{/if}
		
		{if $item.facet}
			<a data-facet="{$item.facet}" data-value="{$item.key}" href="#">
		{else}
			<a data-value="{$item.key}" href="#">
		{/if}

		{$item.name}

		{if $item.groupName}
			<span class="groupname">({$item.groupName})</span>
		{/if}				
		<span class="count"></span></a></li>
	{/foreach}
{/template}

In the Corresponding component class GSearchFilters.es.js I simply added creation of the dropdown option item click events, settings the selected item name and triggering the visibility of these, based on the asset and type selections. Parameter keys are added to the GSearchQuery.es.js class and new language keys to the Language.properties files.

On the backend side, in GSearchDisplayConfiguration, I added a configuration option for document type extensions. Well, it’s not nice configuration syntax I’m using there but for the sake of this exercise, it works. I generally tried to add there enough help texts for all the options.

Next in the ViewMVCRenderCommand class I’m adding there the options in the SOY template context:

template.put(GSearchWebKeys.WEB_CONTENT_STRUCTURE_OPTIONS,
_webContentStructureOptions.getOptions(renderRequest, _gSearchDisplayConfiguration));

In the same class, in setInitialParameters() I’m also adding the handling of these new parameters, if they are coming from the page calling url. For all the filter options there are dedicated services which are referenced like:

@Reference
protected WebContentStructureOptions _webContentStructureOptions;

The service has a declaring interface and at least one implementation. For example, for the web content structure options there’s the interface fi.soveltia.liferay.gsearch.web.search.menuoption.WebContentStructureOptions:

public interface WebContentStructureOptions {

	/**
	 * Get options.
	 * 
	 * @param portletRequest
	 * @param gSearchDisplayConfiguration
	 * @return options JSON array
	 * @throws Exception
	 */
	public JSONArray getOptions(
		PortletRequest portletRequest,
		GSearchDisplayConfiguration gSearchDisplayConfiguration)
		throws Exception;
}

..and the implementation, in this case  fi.soveltia.liferay.gsearch.web.search.internal.menuoption.WebContentStructureOptionsImpl. In the actual implementation I’ll collect the global group, all the users site groups and all public site groups and then using DDMStructureService, get the accessible document type options.

List structures = _ddmStructureService.getStructures(
	themeDisplay.getCompanyId(), groupIds, classNameId,
	WorkflowConstants.STATUS_APPROVED);

For the resulting JSON I’m tagging the options with “scope” all, if they are not from the current scope group. That way I can switch the visibility of available types by scope in the UI.

The last things to do is to pipe these new parameters from search request to the query filters.

For that I added the new parameters in fi.soveltia.liferay.gsearch.web.search.internal.queryparams.QueryParams pojo, handling of parameters in fi.soveltia.liferay.gsearch.web.search.internal.queryparams.QueryParamsBuilder.impl. Finally I add the new parameters to the query filters in fi.soveltia.liferay.gsearch.web.search.internal.query.filter.QueryFilterBuilderImpl:

/**
 * Add web content structure condition.
 * 
 * @throws ParseException
 */
 protected void buildWebContentStructureCondition()
	throws ParseException {

	String structureKey = _queryParams.getWebContentStructureKey();

	if (structureKey != null) {
	   _filter.addRequiredTerm("ddmStructureKey", structureKey);
	}
}	

That’s it for this time. The code is available at https://github.com/peerkar/liferay-gsearch

Creating a Google Like Search

Company Blogs October 5, 2017 By Petteri Karttunen Staff

More than once I've been asked to customise Liferay search to be "simpler" and more "Google like".
 
In the first part of this blog series I'm going to create a custom search portlet from scratch and in the second part I’ll be discussing options to tune search behaviour like relevance.
 
First task in creating a custom search portlet was the choice of UI technology. This time I wanted to try something new. As far as Javascript framework was concerned, I was used to work with Alloy UI but as it's deprecated as of version 7 I decided to choose something else. Using Metal.js and SOY seemed attractive and interesting but there was not too much documentation or full bodied tutorials. On that basis, for someone like me who didn't have experience of closure templates, the task seemed somewhat challenging but I decided to take the challenge. It took a while to get the idea but in the end I think it was definitely worth it. Metal.js and SOY is a great and powerful combination. They enable and facilitate creating cleaner UI code than before by almost forcing you to separate presentation and logic layers.  See more information about Metal.js on https://metaljs.com
 
I wanted this new portlet to be lightweight, simple, support sorting and facets. It should be working with GET instead of POST parameters to make it bookmarkable and easier to work with Google Analytics. The parameters on address line should be readable. Above all I wanted to be fast. I didn't try to make it configurable for all the situations or support all the asset types like users but still it to be easily extensible and customisable. As a basis for something I could build on later. So I added there support for Web content, Blogs and Message boards. Support for other types can be easily added. There's no configuration option for the asset types in this example but basically there are just three things to do: add your new type in the menu, handle the form post QueryParamsBuilder.setTypeParam() and overload BaseResultBuilder for your asset type if needed. 
 
The code was done on DXP FP30 but it should work on CE too. I didn’t use the embedded ElasticSearch engine but a standalone one as debugging the index is much easier that way. There's by the way a great free  ElasticHQ plugin for ElasticSearch http://www.elastichq.org/. Instructions for setting up a standalone ElasticSearch are here https://dev.liferay.com/discover/deployment/-/knowledge_base/7-0/installing-elasticsearch
 
I tried to make code and comments easily readable. Have a look at it at GitHub.
 
 

Installation and configuration

Clone the git repo, build with Gradle and deploy to your portal. After deploying there's just one thing you need to do: put an AssetPublisher on the site you plant the Search portlet to. That’s needed to show any Web contents that do not have a display page but are in the CMS only. By default the portlet searches for a page with friendlyURL "/viewasset" but you can change that in the portlet configuration which can be found along the other few options  in Control Panel -> Configuration -> System Settings -> Other -> Gsearch display configuration

Code is available here https://github.com/peerkar/liferay-gsearch. Hopefully you'll find it useful!

 

Flexible Liferay 7 - Tomcat Setup

Technical Blogs December 16, 2016 By Petteri Karttunen Staff

Flexible Liferay 7 - Tomcat Setup

The challenge

If you want to run Liferay 7 on Tomcat using the bundle provided by Liferay or packages provided in your Linux distribution are convenient ways to get Liferay up and running fast.

In many cases you may however want to have a better control on which Tomcat version you are running on and also want to have an easy upgrade path. You may also want to run multiple Tomcat instances in one server like one for Liferay and one for SOLR and want to reduce administrative tasks and avoid installing Tomcat multiple times. If you are a developer you could also want to run different versions of the portal and even run the portal on different JREs in the same server. It would also be convenient to be able to setup database easily for the Tomcat instances.

Last but not least the setup created by the scripts provided here shouldn’t be anything proprietary or dependent of the scripts.

About this setup

To fulfill these requirement this setup aims at the following objectives:

  • Automation scripts provided here should be readable and easily modifiable
  • Installation structure should be simple and readable
  • Tomcat installation folder should remain unmodified if possible
  • Invidual Tomcat instances should be easily configurable
  • There should be a fast way to create new Tomcat instances from templates
  • Database creation and configuration for a new Tomcat Liferay instance should be easy

By default this setup creates following structure:

There are three scripts in this package:

  • install-environment.sh: creates Tomcat environment, sets up MySQL and Java
  • create-liferay-instance.sh: creates a new Tomcat instance based on defined template
  • manage-instance.sh: starts and stops instances

This setup has been tested in Ubuntu Linux 16.04 LTS.

Yes, the setup and scripts here are not a Swiss knife solution as they are for example relying on MySQL. Hopefully someone will find them useful anyways!

The scripts and more instructions: https://github.com/peerkar/multi-instanced-tomcat-liferay-setup

 

Showing 6 results.
Items 20
of 1