Creating a Google Like Search Part IV: Query Tuning and Lucene Syntax

(Previous parts of this series can be found here (part 1),  here (part 2) and here (part 3)

As a random, Liferay custom search developer you might want to get support for Boolean operators and|or Lucene syntax back. Also, it would be nice to have more control of hits relevancy and about the logic, the queries are being built in the end. To get those, we are going a little deeper than before.

Liferay SearchContext is a wonderful transport assistant made for you if you want to implement your custom search easily. You just put your keywords and filtering options there, throw it to, typically FacetedSearcher, and that’s it. But there’s a counter side for this easiness, the amount of control.

When you do like just described, you don’t have control which fields are being searched for in different assets. Nor can you control which query type to use. Especially the query type matters. You may have wondered how standard search gives sometimes even surprising results. Well, it’s because of three things. First, it relies mostly on Elasticsearch MatchQuery (for more information please see the Elasticsearch Query DSL documentation). MatchQuery is the all-rounder of the query types matching, for my likings, too broadly everything. The other thing is that when you execute search, fields are being targeted with multiple different types of SHOULD queries: a WildCard, Match, Phrase and Phrase Prefix –queries. The third thing is that queries are targeted to fields that you, as a guest user, might not be even aware about, like assetTagNames and assetCategoryNames.

So for example a search with phrase “lorem ipsum” transforms through standard search portlet into something like this (2229 lines):


{
  "from": 0,
  "size": 20,
  "query": {
    "bool": {
      "must": {
        "bool": {
          "must": [
            {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "assetCategoryTitles_en_US": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "assetCategoryTitles_en_US": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "assetTagNames": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "assetCategoryTitles": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "assetTagNames": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "assetTagNames": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "comments": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "comments": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "comments": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "content": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "content": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "content": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "match": {
                          "description": {
                            "query": "lorem ipsum",
                            "type": "boolean"
                          }
                        }
                      },
                      "should": [
                        {
                          "match": {
                            "description": {
                              "query": "lorem ipsum",
                              "type": "phrase",
                              "slop": 50
                            }
                          }
                        },
                        {
                          "match": {
                            "description": {
                              "query": "lorem ipsum",
                              "type": "phrase",
                              "boost": 2
                            }
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "properties": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "properties": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "properties": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "title": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "title": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "title": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "url": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "url": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "url": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "userName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "userName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "ddmContent": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "extension": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "extension": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "extension": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "fileEntryTypeId": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "fileEntryTypeId": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "fileEntryTypeId": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "path": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "path": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "city": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "city": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "city": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "country": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "country": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "country": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "emailAddress": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "emailAddress": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "firstName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "firstName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "firstName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "fullName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "fullName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "fullName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "lastName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "lastName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "lastName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "middleName": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "middleName": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "middleName": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "region": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "region": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "region": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "screenName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "screenName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "street": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "street": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "street": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "zip": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "zip": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "zip": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "userName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "userName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "ddmContent": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "ddmContent": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "articleId": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "articleId": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "articleId": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "classPK": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "classPK": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "classPK": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "content_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "content_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "content_en_US": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "description_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "description_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "description_en_US": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "entryClassPK": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "entryClassPK": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "entryClassPK": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "must": {
                        "bool": {
                          "should": [
                            {
                              "match": {
                                "title_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "boolean"
                                }
                              }
                            },
                            {
                              "match": {
                                "title_en_US": {
                                  "query": "lorem ipsum",
                                  "type": "phrase_prefix"
                                }
                              }
                            }
                          ]
                        }
                      },
                      "should": {
                        "match": {
                          "title_en_US": {
                            "query": "lorem ipsum",
                            "type": "phrase",
                            "boost": 2
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "wildcard": {
                            "userName": "*lorem*"
                          }
                        },
                        {
                          "wildcard": {
                            "userName": "*ipsum*"
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "term": {
                "groupId": "20143"
              }
            }
          ]
        }
      },
      "filter": {
        "bool": {
          "must": {
            "term": {
              "companyId": "20116"
            }
          },
          "should": [
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.wiki.model.WikiPage"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.document.library.kernel.model.DLFileEntry"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.portal.kernel.model.User"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.bookmarks.model.BookmarksFolder"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.blogs.kernel.model.BlogsEntry"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.document.library.kernel.model.DLFolder"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.dynamic.data.lists.model.DDLRecord"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.journal.model.JournalArticle"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.bookmarks.model.BookmarksEntry"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.journal.model.JournalFolder"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.calendar.model.CalendarBooking"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "bool": {
                      "should": {
                        "term": {
                          "entryClassName": "com.liferay.message.boards.kernel.model.MBMessage"
                        }
                      }
                    }
                  },
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "userId": "20120"
                          }
                        },
                        {
                          "terms": {
                            "roleId": [
                              "20123"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
  },
  "post_filter": {
    "bool": {
      "must": [
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.wiki.model.WikiPage"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "hidden": "false"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.document.library.kernel.model.DLFileEntry"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "status": "0"
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.portal.kernel.model.User"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.bookmarks.model.BookmarksFolder"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.blogs.kernel.model.BlogsEntry"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "hidden": "false"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.document.library.kernel.model.DLFolder"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "status": "0"
                      }
                    },
                    {
                      "term": {
                        "recordSetScope": "0"
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.dynamic.data.lists.model.DDLRecord"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "head": "true"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.journal.model.JournalArticle"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.bookmarks.model.BookmarksEntry"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.journal.model.JournalFolder"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "must": {
                    "bool": {
                      "must": [
                        {
                          "bool": {
                            "should": {
                              "term": {
                                "entryClassName": "com.liferay.calendar.model.CalendarBooking"
                              }
                            }
                          }
                        },
                        {
                          "bool": {
                            "should": [
                              {
                                "term": {
                                  "userId": "20120"
                                }
                              },
                              {
                                "terms": {
                                  "roleId": [
                                    "20123"
                                  ]
                                }
                              }
                            ]
                          }
                        }
                      ]
                    }
                  }
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "discussion": "false"
                      }
                    },
                    {
                      "terms": {
                        "status": [
                          "0"
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "should": {
                                "term": {
                                  "entryClassName": "com.liferay.message.boards.kernel.model.MBMessage"
                                }
                              }
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "term": {
                                    "userId": "20120"
                                  }
                                },
                                {
                                  "terms": {
                                    "roleId": [
                                      "20123"
                                    ]
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must": [
              {
                "terms": {
                  "groupId": [
                    "20143"
                  ]
                }
              },
              {
                "terms": {
                  "scopeGroupId": [
                    "20143"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  },
  "fields": "*",
  "track_scores": true,
  "aggregations": {
    "assetTagNames.raw": {
      "terms": {
        "field": "assetTagNames.raw",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "assetCategoryIds": {
      "terms": {
        "field": "assetCategoryIds",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "entryClassName": {
      "terms": {
        "field": "entryClassName",
        "min_doc_count": 1
      }
    },
    "groupId": {
      "terms": {
        "field": "groupId",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "modified": {
      "range": {
        "field": "modified",
        "ranges": [
          {
            "from": "20171025080000",
            "to": "20171025100000"
          },
          {
            "from": "20171024090000",
            "to": "20171025100000"
          },
          {
            "from": "20171018090000",
            "to": "20171025100000"
          },
          {
            "from": "20170925090000",
            "to": "20171025100000"
          },
          {
            "from": "20161025090000",
            "to": "20171025100000"
          }
        ]
      }
    },
    "userName": {
      "terms": {
        "field": "userName",
        "size": 10,
        "min_doc_count": 1
      }
    },
    "folderId": {
      "terms": {
        "field": "folderId",
        "size": 10,
        "min_doc_count": 1
      }
    }
  },
  "highlight": {
    "pre_tags": [
      ""
    ],
    "post_tags": [
      ""
    ],
    "require_field_match": true,
    "fields": {
      "description": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "description_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "title": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "title_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "assetCategoryTitles": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "assetCategoryTitles_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "content": {
        "fragment_size": 80,
        "number_of_fragments": 3
      },
      "content_en_US": {
        "fragment_size": 80,
        "number_of_fragments": 3
      }
    }
  }
}

Run through our custom portlet it's 140 rows and looks like:


{
   "from":0,
   "size":10,
   "query":{
      "bool":{
         "must":{
            "bool":{
               "must":{
                  "query_string":{
                     "query":"lorem ipsum"
                  }
               }
            }
         },
         "filter":{
            "bool":{
               "must":[
                  {
                     "term":{
                        "companyId":"20116"
                     }
                  },
                  {
                     "term":{
                        "stagingGroup":"false"
                     }
                  },
                  {
                     "term":{
                        "status":"0"
                     }
                  },
                  {
                     "bool":{
                        "should":[
                           {
                              "bool":{
                                 "must":[
                                    {
                                       "term":{
                                          "entryClassName":"com.liferay.journal.model.JournalArticle"
                                       }
                                    },
                                    {
                                       "term":{
                                          "head":"true"
                                       }
                                    }
                                 ],
                                 "should":[
                                    {
                                       "range":{
                                          "displayDate_sortable":{
                                             "from":"-9223372036854775808",
                                             "to":"1509368280059",
                                             "include_lower":true,
                                             "include_upper":true
                                          }
                                       }
                                    },
                                    {
                                       "range":{
                                          "expirationDate_sortable":{
                                             "from":"1509368280059",
                                             "to":"9223372036854775807",
                                             "include_lower":true,
                                             "include_upper":true
                                          }
                                       }
                                    }
                                 ]
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.document.library.kernel.model.DLFileEntry"
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.message.boards.kernel.model.MBMessage"
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.blogs.kernel.model.BlogsEntry"
                              }
                           },
                           {
                              "term":{
                                 "entryClassName":"com.liferay.wiki.model.WikiPage"
                              }
                           }
                        ]
                     }
                  },
                  {
                     "bool":{
                        "should":[
                           {
                              "term":{
                                 "roleId":"20123"
                              }
                           },
                           {
                              "term":{
                                 "userId":"20120"
                              }
                           }
                        ]
                     }
                  }
               ]
            }
         }
      }
   },
   "fields":"*",
   "sort":[
      {
         "_score":{

         }
      },
      {
         "modified_sortable":{
            "order":"asc",
            "unmapped_type":"string"
         }
      }
   ],
   "track_scores":true,
   "aggregations":{
      "entryClassName":{
         "terms":{
            "field":"entryClassName"
         }
      }
   }
}

So, where did over 2000 rows go? Ok, this is not completely a fair comparison. I'm using here StringQuery which basically does on one row all the dirty work and searches for all the fields (this topic will be revisited in the next episode). Also, the standard search portlet and this one have very different goals, different emphasis and also different functionalities.

What this exercise shows however, is the strength of Liferay as development platform. The search framework is robust and flexible allowing you to self define the level of control you want to gain.

What was done?

The key things in this solution were:

  • Handling the query building manually i.e. not pushing keywords to SearchContext object
  • Building the search filters manually
  • Using IndexSearchHelper service instead of FacetedSearcher to do the work. It gives you more control.

For the rest of the stuff I'm encouraging you to see the code in Github.

What's still coming

In this part of the series we took more control over building the queries but there's more to do. At the moment the ElasticSearch StringQuery type, we are using, is not implemented fully in portal search.That's why we cannot yet control field level boosting in combination with it. 

For those purposes, in the next episode I'm going to customize the Elasticsearch adapter to support ES 5.6 and implement String Query properly. Also I'm going to show you how to use Audience Targeting in Liferay search.

The final planned episode will be about search federation. About using external systems index data through Liferay search.

Blogs