Error when reindex elasticsearch : bytes can be at most 32766 in length

[7年前 に kuong knight によって更新されました。]

I configuartion to use remote elasticsearch Guide here
When i reindex from control panel, see log many error : original message: bytes can be at most 32766 in length;
Can i fix this?

[7年前 に Jorge Díaz によって更新されました。]

Hi Kuong,

Can you post the log file will full exception or full error traces?

Regards,
Jorge Díaz

[7年前 に kuong knight によって更新されました。]

10:50:56,385 ERROR [liferay/background_task-1][SearchEngineInitializer:116] Error encountered while reindexing
com.liferay.portal.kernel.search.SearchException: com.liferay.portal.kernel.exception.SystemException: java.lang.NoSuchMethodError: org.apache.poi.hwpf.model.ListLevel.getRestart()S
        at com.liferay.portal.kernel.search.BaseIndexer.reindex(BaseIndexer.java:583)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer$3.performAction(DLFileEntryIndexer.java:634)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer$3.performAction(DLFileEntryIndexer.java:620)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.performAction(DefaultActionableDynamicQuery.java:396)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery$1.call(DefaultActionableDynamicQuery.java:305)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery$1.call(DefaultActionableDynamicQuery.java:269)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.doPerformActions(DefaultActionableDynamicQuery.java:325)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.performActions(DefaultActionableDynamicQuery.java:85)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer.reindexFolders(DLFileEntryIndexer.java:639)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer.doReindex(DLFileEntryIndexer.java:521)
        at com.liferay.portal.kernel.search.BaseIndexer.reindex(BaseIndexer.java:577)
        at com.liferay.portal.search.internal.SearchEngineInitializer.reindex(SearchEngineInitializer.java:135)
        at com.liferay.portal.search.internal.SearchEngineInitializer.doReIndex(SearchEngineInitializer.java:106)
        at com.liferay.portal.search.internal.SearchEngineInitializer.reindex(SearchEngineInitializer.java:57)
        at com.liferay.portal.search.internal.SearchEngineInitializer.reindex(SearchEngineInitializer.java:53)
        at com.liferay.portal.search.internal.background.task.ReindexPortalBackgroundTaskExecutor.reindex(ReindexPortalBackgroundTaskExecutor.java:48)
        at com.liferay.portal.search.internal.background.task.ReindexBackgroundTaskExecutor.execute(ReindexBackgroundTaskExecutor.java:54)
        at com.liferay.portal.kernel.backgroundtask.SerialBackgroundTaskExecutor.execute(SerialBackgroundTaskExecutor.java:54)
        at com.liferay.portal.kernel.backgroundtask.DelegatingBackgroundTaskExecutor.execute(DelegatingBackgroundTaskExecutor.java:43)
        at com.liferay.portal.kernel.backgroundtask.ThreadLocalAwareBackgroundTaskExecutor.execute(ThreadLocalAwareBackgroundTaskExecutor.java:56)
        at com.liferay.portal.background.task.internal.messaging.BackgroundTaskMessageListener.doReceive(BackgroundTaskMessageListener.java:123)
        at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:26)
        at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
        at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
        at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
        at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.liferay.portal.kernel.exception.SystemException: java.lang.NoSuchMethodError: org.apache.poi.hwpf.model.ListLevel.getRestart()S
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.doPerformActions(DefaultActionableDynamicQuery.java:341)
        at com.liferay.portal.kernel.dao.orm.IndexableActionableDynamicQuery.doPerformActions(IndexableActionableDynamicQuery.java:98)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.performActions(DefaultActionableDynamicQuery.java:85)
        at com.liferay.portal.kernel.dao.orm.IndexableActionableDynamicQuery.performActions(IndexableActionableDynamicQuery.java:73)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer.reindexFileEntries(DLFileEntryIndexer.java:611)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer.doReindex(DLFileEntryIndexer.java:529)
        at com.liferay.portal.kernel.search.BaseIndexer.reindex(BaseIndexer.java:577)
        ... 26 more
Caused by: java.lang.NoSuchMethodError: org.apache.poi.hwpf.model.ListLevel.getRestart()S
        at org.apache.tika.parser.microsoft.ListManager.buildTuple(ListManager.java:108)
        at org.apache.tika.parser.microsoft.ListManager.getFormattedNumber(ListManager.java:86)
        at org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:277)
        at org.apache.tika.parser.microsoft.WordExtractor.handleParagraph(WordExtractor.java:252)
        at org.apache.tika.parser.microsoft.WordExtractor.parse(WordExtractor.java:168)
        at org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:146)
        at org.apache.tika.parser.microsoft.OfficeParser.parse(OfficeParser.java:117)
        at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
        at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
        at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
        at org.apache.tika.Tika.parseToString(Tika.java:465)
        at org.apache.tika.Tika.parseToString(Tika.java:540)
        at com.liferay.portal.util.FileImpl.extractText(FileImpl.java:429)
        at com.liferay.portal.kernel.util.FileUtil.extractText(FileUtil.java:217)
        at com.liferay.portal.kernel.search.DocumentImpl.addFile(DocumentImpl.java:171)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer.doGetDocument(DLFileEntryIndexer.java:386)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer.doGetDocument(DLFileEntryIndexer.java:98)
        at com.liferay.portal.kernel.search.BaseIndexer.getDocument(BaseIndexer.java:172)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer$2.performAction(DLFileEntryIndexer.java:594)
        at com.liferay.portlet.documentlibrary.util.DLFileEntryIndexer$2.performAction(DLFileEntryIndexer.java:589)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.performAction(DefaultActionableDynamicQuery.java:396)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery$1.call(DefaultActionableDynamicQuery.java:305)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery$1.call(DefaultActionableDynamicQuery.java:269)
        at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.doPerformActions(DefaultActionableDynamicQuery.java:325)
        ... 32 more
10:50:58,211 ERROR [liferay/search_writer/SYSTEM_ENGINE-21][ElasticsearchUpdateDocumentCommandImpl:50] failure in bulk execution:_[1453]: index [liferay-20202], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_46621], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="ddm__keyword__30008__content_en_US" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[66, -31, -70, -94, 78, 32, 84, 73, 78, 32, 84, 72, 85, -31, -70, -66, 44, 32, 66, -31, -70, -94, 79, 32, 72, 73, -31, -69, -126, 77]...', original message: bytes can be at most 32766 in length; got 52310]_[1455]: index [liferay-20202], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_46625], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="ddm__keyword__30008__content_en_US" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[66, -31, -70, -94, 78, 32, 84, 73, 78, 32, 84, 72, 85, -31, -70, -66, 44, 32, 66, -31, -70, -94, 79, 32, 72, 73, -31, -69, -126, 77]...', original message: bytes can be at most 32766 in length; got 52310] [Sanitized]

[7年前 に kuong knight によって更新されました。]

Help me

[7年前 に Jorge Díaz によって更新されました。]

Which Liferay version are you using?
- ¿7.0.0 GA1 or 7.0.1 GA2?

[7年前 に kuong knight によって更新されました。]

i use 7.0.0 GA1

[7年前 に Thirumal Reddy によって更新されました。]

Add this user environment variable,
Name=JVM_ARGS
Value=-Xms1024m -Xmx1024m
restart the server, and reindex again.
regards,
Thirumal.

[7年前 に kuong knight によって更新されました。]

Where can i add this>?

[7年前 に Thirumal Reddy によって更新されました。]

Right click on MyComputer/properties/Advanced system settings /Environment variables/
there you will get user variables and system variables,
In User varaibles click on new and add following variables,
varaible name : JVM_ARGS
variable value : -Xms1024m -Xmx1024m

[7年前 に Amos Fong によって更新されました。]

These links may help you:
https://stackoverflow.com/questions/24019868/utf8-encoding-is-longer-than-the-max-length-32766

https://dev.liferay.com/develop/tutorials/-/knowledge_base/7-0/introduction-to-liferay-search

Essentially the problem is the mapping for your field matches this:

				"template_ddm_keyword": {
					"mapping": {
						"index": "not_analyzed",
						"store": "true",
						"type": "string"
					},
					"match": "ddm__keyword__*",
					"match_mapping_type": "string"
				}

You can see it's not analyzed so it's treated as one really long string and elasticsearch has a limit not analyzed fields. I'm not too familiar with 7 yet, so I'm not sure whether liferay is incorrectly naming your field or if you can configure your web content structure so your content field is not named "ddm__keyword*".

[7年前 に kuong knight によって更新されました。]

@Thirumal Reddy: memory is not a proplem.
@Amos Fong: when add a web-content with long content have this error, and webcontent is not showing. This is a bug? can i fix this.

[7年前 に Jorge Díaz によって更新されました。]

In your log you are having two errors:

• elasticsearch: bytes can be at most 32766 in length
• java.lang.NoSuchMethodError: org.apache.poi.hwpf.model.ListLevel.getRestart()S

About first error, I think Amos Fong is right, you have to check mapping configuration

About second error, It is fixed in 7.0.1 GA2, see: LPS-64958, you should upgrade from 7.0.0 GA1 to 7.0.1 GA2

[7年前 に kuong knight によって更新されました。]

1. I use default Liferay mapping configuration, so this is bug of 7.0.0?

[7年前 に Amos Fong によって更新されました。]

Are using a web content structure? Can you post it here? Or post some reproduction steps?

[7年前 に kuong knight によって更新されました。]

I use default Structure: Basic Web Content and default Template: Basic Web Content.
Step: ControlPanel -> Webcontent -> Add -> Basic Web Content -> publish

[7年前 に Jorge Díaz によって更新されました。]

Hi all,

First error: elasticsearch: bytes can be at most 32766 in length
I was able to reproduce your issue, it is produced in case of creating a Basic Web Content with more than 32766 characters (html code) see LPS-67339.

As a workaround you can try changing index configuration of Basic Web Content:

• Open control panel => webcontent section
• Open "Structures" page
• Edit "Basic Web Content" structure
• Select "content" box and change Indexation configuration from "Indexable - Keyword" to "Indexable - Text"

Second error: java.lang.NoSuchMethodError: org.apache.poi.hwpf.model.ListLevel.getRestart()S
As I explained in my previous message, this issue was resolved in LPS-64958. You should upgrade from 7.0.0 GA1 to 7.0.1 GA2 in order to fix it.

Regards,
Jorge Díaz

[7年前 に Benjamin Bini によって更新されました。]

I had the same problem with the "immense field error", it is indeed the solution, thank you.

I think the default indexType for WYSIWYG / HTML fields should be text.

[7年前 に Jorge Díaz によって更新されました。]

Hi Benjamin,

That issue was solved in LPS-67339 and solution was to change HTML fields from "keyword" type to "text", see:

• https://github.com/brianchandotcom/liferay-portal/pull/42651
• https://github.com/brianchandotcom/liferay-portal/pull/43791

LPS-67339 will be included in future Liferay 7.0 GA4

[6年前 に Yury Uzvards によって更新されました。]

Upgraded Liferay 7 GA3 to Liferay 7 GA4, got same error:

10:04:43,756 ERROR [liferay/search_writer/SYSTEM_ENGINE-6][ElasticsearchUpdateDocumentCommandImpl:50] failure in bulk execution:_[265]: index [liferay-20116], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_57540], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="ddm__keyword__29253__content_lt_LT" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[77, 101, 100, 105, 106, 117, 32, 97, 112, 115, 107, 97, 116, 115, 32, 80, 105, 114, 109, 100, 105, 101, 110, 44, 32, 54, 46, 32, 109, 97]...', original message: bytes can be at most 32766 in length; got 35897]_[275]: index [liferay-20116], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_57576], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="ddm__keyword__29253__content_l...

It seems that this fix was lost on his way to GA4 release

Good point is that proposed workaround is still working, I'll rewrite it for Liferay 7 newbies like I am
As a workaround you can try changing index configuration of Basic Web Content:
- Log in as administrative user to portal and open Content - Web Content section (available under Your site);
- Open "Options - Structures" (Options - 3 vertical dots in right upper corner of Web Content main page)
- Click on "Basic Web Content" (or select "Basic Web Content" Options - Edit)
- Scroll down to Fields | Settings - on the right side there will be written "Content" and empty content placeholder - click on it
- Configuration will be opened on the left side, like
name: value
Type: ddm-text-html
...
Indexable: Keyword
...
- Double-click on "Indexable: Keyword" and select radio-button "Indexable - Text", than click "Save"
- Than "Save" "Basic Web Content"

NB!: Remenber to reindex:
- log in as an administrative user and click on Control Panel → Configuration → Server Administration and click the "Execute" button for "Reindex all search indexes".

[6年前 に Jorge Díaz によって更新されました。]

Hi Yury,

I think LPS-67339 only solves the issue for new Liferay installations in 7.0 GA4, creating the basic webcontent structure with the correct index type.

If you have a 7.0 GA3 installation, basic webcontent object already exists in your database and upgrading it from 7.0 GA3 to 7.0 GA4 doesn't modify existing definition.

So for existing Liferay installations, the best solution is to apply the workaround.

Regards,
Jorge

[6年前 に Yury Uzvards によって更新されました。]

Thanks for advice!