lucene.store.type = jdbc issues

[Rakesh Sinha, geändert vor 14 Jahren.]

Hi,

Can someone please let me know why Liferay is behaving like this.

I have arnd 30K users in the system. The file based lucene index size is arnd 10MB once I index all the users.

But when I changed the store type to jdbc and try to reindex the users it gave exception that cannot extend tablespace size.

Before indexing the liferay instance was using arnd 350MB of tablespace size and the tablespace size limit was 12GIG.

But only after around 10K users were indexed, all the rest of 11.5 GIG of space in the tablespsace was being used.

Apart from this another weird thing is, in jdbc type store index, I can see thousands of records in the LUCENE table. But I think the lucene process merge the index data once in a while during optimizing the index

FYI, I am using Oracle 10.2.0.2 as my database server.

I am really stumped why this is not working right. It seems such a trivial thing. Does anybody has any idea about the space required to store the index in the database and are there any specific properties to set in portal-ext.properties file.

Please help me in fixing this issue. Let me know if you need more information.

Thanks,
Rak.

[Lisa Simpson, geändert vor 14 Jahren.]

I don't know the answer to it. We've got about half your number of users but we use MySQL without any problems.

[Victor Zorin, geändert vor 14 Jahren.]

we had exactly the same issue on Win MSSQL platform, had to switch back to file storage,
see details in Since switch to lucene.store.type=jdbc I'm getting lucene errors

[Marcelo Gore, geändert vor 14 Jahren.]

I am not anywhere near the amount of users you are but I'm still having problems with Lucene using JDBC store. I've got 100 users and barely any content and my Lucene index weighs 500MB. It's Liferay 5.2.2, BTW.

I've found a couple of interesting things while trying to solve this issue:

- It's not Lucene storing indexes in the database, it's Compass (this came in handy when looking for documentation and code excerpts).
- Nearly every BLOB in Lucene's tables is marked as deleted, which means nearly all database's weight comes from unused BLOBs.
- Liferay never uses org.apache.lucene.store.jdbc.JdbcDirectory's deleteMarkDeleted() method, which deletes entries marked as, well, dah, deleted.

As I needed this issue solved ASAP, I did something awful that, still, worked: I created a DB trigger that deletes rows as soon as they are marked for removal. This is, by no means, a good thing to do and I know it, but at least my DB won't die every now and then, starving for tablespace.

I think the proper way to solve this is to use deleteMarkDeleted() method somewhere in Liferay's code to keep the DB clean but I haven't had time to further look into this. Any ideas?

All input is greatly appreciated.

MG

[Jeffrey Handa, geändert vor 14 Jahren.]

Hi Marcelo,

I am not sure if it's related to what you're experiencing, but have you set this property in the portal.properties file?

#
# Lucene's storage of indexes via JDBC has a bug where temp files are not
# removed. This can eat up disk space over time. Set this to true to
# automatically clean up the temporary files once a day. See LEP-2180.
#
lucene.store.jdbc.auto.clean.up=true

[Marcelo Gore, geändert vor 14 Jahren.]

Jeffrey Handa:

Hi Marcelo,

I am not sure if it's related to what you're experiencing, but have you set this property in the portal.properties file?

#
# Lucene's storage of indexes via JDBC has a bug where temp files are not
# removed. This can eat up disk space over time. Set this to true to
# automatically clean up the temporary files once a day. See LEP-2180.
#
lucene.store.jdbc.auto.clean.up=true

Thank you, Jeffrey. Yes, I've already tried that. I also looked at the code and, as I understand it, what it does is to delete all temporary files (filesystem files, that is) created by Lucene, once a day. I don't believe this will have any impact on JDBC store, will it? Maybe I misinterpreted it.

Thanks again,
MG

[Manish K Choudhary, geändert vor 14 Jahren.]

Thanks a lot .
Writing a trigger to delete marked deleted records solved my problem

[Juan Antonio Girón, geändert vor 11 Jahren.]

Hi Manish,
what's the value of deleted_ column in LUCENE tables to mark deleted rows? b'1' or b'2'?
LUCENE_communityid table contains "write.lock" row with b'0' value into delete_ column, is that ok?