Optimizing reindex

[suresh reddy k, modificado 14 Anos atrás.]

Hi All,

I'm adding users thru batch load (sql script) every night to create users. Once the users are created I'm going through
Control Panel-->Server Admin --> Reindex
so that, the users are listed when I login as 'administrator' and click on 'Users'

This all works fine.

But, the problem is I've 3000-4000 users added on every batch load and seems reindexing is considering the whole lot every-time. I'm expecting the reindexing process would be incremental but, not.
It end up taking couple of hours to complete reindex process.

Is there any alternative?

Any pointers would be highly appreciated.

Thanks,
Suresh

[Lisa Simpson, modificado 14 Anos atrás.]

Ok, let's see...

Adding 3-4K users per night

Over one week that's 24,500 users - give or take....

Reindexing isn't incremental - it reindexes EVERYTHING and not just your users..... content, navigation.... everything...

[Victor Zorin, modificado 14 Anos atrás.]

If you use webservice of direct API calls you would not have to do reindexing.
And it is more architecturally sound and safer to use services layer rather than direct sql scripts.

[Lisa Simpson, modificado 14 Anos atrás.]

There's only one problem with that theory... Most of the index managment is in the LIFERAY APPLICATION and not the database. If you don't beleive me, check out the incrementation on the Counter table....

[Lisa Simpson, modificado 14 Anos atrás.]

As near as I can tell, it's kept in a voodoo file somewhere and not in the database.

[David Jefferson, modificado 14 Anos atrás.]

Has anyone identified where this magic index file resides?

[Wilson Man, modificado 14 Anos atrás.]

by default, liferay uses Lucene out of the box and the file resides in

~/liferay/data/lucene

, assuming you haven't configured it to reside somewhere else.

[David Jefferson, modificado 14 Anos atrás.]

Great..., thanks for the quick response.

[Wilson Man, modificado 14 Anos atrás.]

i think there's a confusion here. Suresh is talking about the solr/lucene index, not the database table index. Since he inserts data directly into the database, Liferay doesn't know to auto kick off a lucene/solr search indexing for those new users only.

[Lisa Simpson, modificado 14 Anos atrás.]

There's not any confusion here. You see, kicking off SOLR or Lucene is not all the "Reindex all indexes" button does. We have issues where portlets suddenly set themselves to "javax.portlet_56_whatever" or "javax.portlet_24_whatever". Nor am I the only one seeing this behavior.

Even in the control panel, the web content portlet will become javax.portlet56. Reindex and restart Liferay and it's fixed. When you reindex, it doesn't just rebuild the stuff in the database. It's pulling from some voodoo file somewhere, too. I just haven't had enough time to pick through the source to find it. So I don't know if its static or if the application is writing to it over time.

[suresh reddy k, modificado 14 Anos atrás.]

Apologize for the delay.

Well; first of all, thanks to you people for valuable suggestions.

As I mentioned earlier, we have huge data flowing in everyday and reindexing is taking too long (min 8hrs)... we can not afford that.

I went ahead and implemented webservices approach to add users. this approach seems fine to me. I could see all the users listed on creation (no need to run reindex).
however, I have some issues w.r.t null/blank email values. addUser(...) in UserService doesn;t allow null email values; I have to do some extra handling there; but, still waiting for a better approach on storing null/blank emailIds. I did post a new thread on that issue.

thanks,
Suresh

[Lisa Simpson, modificado 14 Anos atrás.]

Liferay requires a unique email address in order to create teh user. You might try to see if you can write something that will create a fake email address in a unique way.

[e-mail] or something similar

[Brad Shannon, modificado 14 Anos atrás.]

suresh reddy k:

Hi All,

I'm adding users thru batch load (sql script) every night to create users. Once the users are created I'm going through
Control Panel-->Server Admin --> Reindex
so that, the users are listed when I login as 'administrator' and click on 'Users'

This all works fine.

But, the problem is I've 3000-4000 users added on every batch load and seems reindexing is considering the whole lot every-time. I'm expecting the reindexing process would be incremental but, not.
It end up taking couple of hours to complete reindex process.

Is there any alternative?

Any pointers would be highly appreciated.

Thanks,
Suresh

Suresh,

Do you have SQL Script examples that you could provide. I'm working on some of my own and would appreciate some examples to follow.

Thanks!

[Lisa Simpson, modificado 14 Anos atrás.]

I think we've conculed in another thread that the right way to do this was through the API so that there isn't any reindexing required.

[Brad Shannon, modificado 14 Anos atrás.]

Lisa Simpson:

I think we've conculed in another thread that the right way to do this was through the API so that there isn't any reindexing required.

I agree, however, I am restricted on what I am able to use to bulk insert new users. Right now, the SQL Script route is the easiest for me. I would really like to be able to get this working. I got records inserted, re-indexed, and now no users are showing up

[David Jefferson, modificado 14 Anos atrás.]

We were running into the same problem (e.g. inserting new users with SQL script and the users were not appearing in the app, even after re-indexing).

Don't know what your SQL scripts look like of course but we solved it by making sure that when we added a user to the User_ table we also had to add appropriate entries in the Group_ and LayoutSet tables. As soon as we did this, and then re-indexed, the users showed up in the app.

[Brad Shannon, modificado 14 Anos atrás.]

David Jefferson:

We were running into the same problem (e.g. inserting new users with SQL script and the users were not appearing in the app, even after re-indexing).

Don't know what your SQL scripts look like of course but we solved it by making sure that when we added a user to the User_ table we also had to add appropriate entries in the Group_ and LayoutSet tables. As soon as we did this, and then re-indexed, the users showed up in the app.

Thanks! That's what I was missing!