Fórum
Asynchronous Search Indexing
Karl Garske, modificado 14 Anos atrás.
Asynchronous Search Indexing
New Member Postagens: 5 Data de Entrada: 14/04/09 Postagens Recentes
Hello,
The current document library implementation constructs a search indexing entry synchronously. While this works fine, it slows down the rate at which documents can be added to the library. It can also contribute to a large amount of redundant processing as updates on a document occur. For example, modifying permission on a document will cause re-indexing.
Ideally, there would be a pluggable solution allowing for administrators to fine tune how to dispatch indexing. In my case, I'd like to flag a document as "dirty" and submit it to a queue for processing. Once the document has sat in my queue for a specific amount of time, it is then processed. This allows bulk operations on a document to be performed quickly, while still showing up in searches within a reasonable amount of time.
Realizing that some may still benefit from a synchronous model (for example, transaction safe inserts), I'd propose the following. Using dependency injection, one can inject an "IndexingStrategy" which defines the algorithm used to dispatch indexing. In the attached UML diagram, I've implemented an AsynchronousIndexingStrategy which passes a concrete IndexingTask to an implementation of IndexingQueue. My queue uses the JDK's DelayQueue to wait 30 seconds before processing the task. As we assign several roles immediately after a document is inserted, the performance gains have been significant.
Unfortunately, it appears the only safe way to integrate my changes is by modifying the core rather than EXT. WRT the document library, modifications to addFile and upateFile in com.liferay.documentlibrary.util.Indexer were necessary because these methods are static. My modifications build a DLFileEntryIndexingTask in addFile and updateFile methods, then hand it off to the IndexingStrategy implementation supplied to Indexer.
Attached is my source and UML diagrams. I've never committed to Liferay before, so I'd appreciate some assistance if these modifications sound good.
Karl
The current document library implementation constructs a search indexing entry synchronously. While this works fine, it slows down the rate at which documents can be added to the library. It can also contribute to a large amount of redundant processing as updates on a document occur. For example, modifying permission on a document will cause re-indexing.
Ideally, there would be a pluggable solution allowing for administrators to fine tune how to dispatch indexing. In my case, I'd like to flag a document as "dirty" and submit it to a queue for processing. Once the document has sat in my queue for a specific amount of time, it is then processed. This allows bulk operations on a document to be performed quickly, while still showing up in searches within a reasonable amount of time.
Realizing that some may still benefit from a synchronous model (for example, transaction safe inserts), I'd propose the following. Using dependency injection, one can inject an "IndexingStrategy" which defines the algorithm used to dispatch indexing. In the attached UML diagram, I've implemented an AsynchronousIndexingStrategy which passes a concrete IndexingTask to an implementation of IndexingQueue. My queue uses the JDK's DelayQueue to wait 30 seconds before processing the task. As we assign several roles immediately after a document is inserted, the performance gains have been significant.
Unfortunately, it appears the only safe way to integrate my changes is by modifying the core rather than EXT. WRT the document library, modifications to addFile and upateFile in com.liferay.documentlibrary.util.Indexer were necessary because these methods are static. My modifications build a DLFileEntryIndexingTask in addFile and updateFile methods, then hand it off to the IndexingStrategy implementation supplied to Indexer.
Attached is my source and UML diagrams. I've never committed to Liferay before, so I'd appreciate some assistance if these modifications sound good.
Karl