Speeding up LDAP import process and Liferay authentication

Liferay 5.2 EE and 6 improved the capabilities of LDAP integration in many areas (refer to blogs posts LDAP Enhancements and Keeping user password secure with LDAP integration):

• synchronize user custom attributes between Liferay and LDAP
• support LDAP chains and LDAP pagination
• create a role for each LDAP group
• override LDAP import and export processes via Spring
• secure LDAP users' password

As you know, base DN is used as a base to search users and groups. When the number of users and groups is small, you would not meet any performance issue when searching users and groups. But if the number of users and groups is huge (like 500K users and 50K groups in LDAP), you would meet any performance issue when searching users and groups, since each user may be part of 50 groups.

Use case A (as shown in following screenshot): 500K users and 50K groups. Each user may be part of 50 groups. When logged in as a user from LDAP, it took 20-30 seconds by default. It should take less than 1 second.

The portal does introduce following property to search groups.

ldap.import.group.search.filter.enabled=true

As shown in above code, if set above property to true, the group filter will be applied, but only to groups in the specified base DN. If set to false, the filter will not be applied and all groups that are associated with the imported users will be imported regardless of the base DN.

The above workaround proposed would not be an option. Each user may belong to over 50 groups and it would just clutter the system with useless data if you were to import all groups for every user.

This article will address how to speed up LDAP import process and Liferay authentication.

Solution Overview

In two steps, you should be able to speed up LDAP import process and Liferay authentication.

1) Set up custom group base DN in portal-ext.properties

# Set this to true to enable custom group based DN settings.
# Set this to false to disable custom group based DN settings.
ldap.import.group.base.dn.enabled=true

# set up group base DN when the property ldap.import.group.basedn.enabled is set to true
# You can add group base DN against your LDAP here
ldap.import.group.base.dn.default=ou=groups,ou=system

2) Use custom group base DN in LDAP import process

Results

LDAP import process and Liferay authentication just took less than 1 second.

1) Set up base DN as users base DN like

2) Test LDAP users

3) Test LDAP groups

Is this feature useful? your comments / suggestions?

Summary

As you can see, LDAP import process and Liferay authentication can be improved a lot by using base DN as users base DN for users search, and custom group base DN as groups base DN for groups search. Ideally, we should divide base DN ldap.base.dn into ldap.users.base.dn and ldap.groups.base.dn for users search and group searchs, respectively. Refer to LPS-14322.

Last but not least, I'd like to send special thanks to Eduardo Carneiro and Jenny Chen who did a great job to narrow down the issue and to identify solutions.

[Jan Gregor]

Very nice feature, looking forward to test the performance increase !

[l b]

good,I like it!

[Jonas X. Yuan]

@jan and @lb, thanks. Here I want to share test results as follows.

"That’s great news! The patch also worked successfully. In this test, I was able to log-in very quickly and I see that only one group that matched the filter was imported. "

[l b]

I'm trying !

[Jonas X. Yuan]

@lb, thanks. This feature is not available in 6.0.5 CE and 6.0 EE and current alpha version 6.1 so far for now. But a fix patch is available for 6.0 EE already, and, especially, 6.0.5 CE on demand.

[Jack Bakker]

useful info, thanks and I will look at this

so far for me for 6.0.5 for large LDAP imports where multiple subsystems authenticate against LDAP (LR only one such subsystem) ; I have been using "ldap.import.method=group", and involve group naming to delineate by subsystem ( I use LR_* group prefixes in LDAP ) where I only reference groups needed in group import filter

moving to AD from OpenLDAP for me has introduced new challenges ; in OpenLDAP, all was quite flat , now in AD ou hierarchies are more intense

[Jonas X. Yuan]

Hi Jack, Thanks.

Here are fix patch testing:

Following features got fixed.
a. LDAP user temporal password should be auto-generated.
b. Make ‘Import Enabled’ and ‘Import on Startup Enable’ working.
c. Super fast sign-in
d. No password imported

How to apply the patch?

You can apply the updated patch in following steps:

1) Shutdown the portal
2) add JAR lpe9000-ldap-ee6010-portal-impl-jdk5.jar.
3) restart the portal

make sure adding following lines at the end of portal-ext.properties

ldap.import.method=group

ldap.import.create.role.per.group=true

ldap.import.user.password.enabled=false

ldap.import.user.password.autogenerated=true

ldap.import.user.password.default=test

ldap.import.group.base.dn.enabled=true

ldap.import.group.base.dn.default=ou=groups,ou=system

Note that you need to use your own LDAP group base DN.

Last but not least, check ‘Import Enabled’ and ‘Import on Startup Enable’ in LDAP UI.

Before testing, it would be better to start from scratch or remove exiting users, user groups, and roles in Liferay portal.

[Jack Bakker]

Thanks Jonas

[Jonas X. Yuan]

Hi Jack, the following is testing result (tested AD has more than 500K users and 50K groups) based on above settings plus patch:

"The new patch seems to work really well. Great Job! It imported the groups on startup and loaded my test users. The login performance is great and the password is being auto generated.

I’m still running a few tests but things are looking good."

[Henrique Simoes de Andrade]

Hi,
Is there some avaiable patch for 6.0.5 CE?

tks

[Jonas X. Yuan]

Hi Henrigue, thanks. Yes, it is available for 6.0.5 CE. You can drop me email: [e-mail].

[ankit yakkundi]

Hi..
I am using Liferay 6.I am not able to import the user.While testing LDAP groups,it shows me the groups,but when i try to see in usergroup its empty.

please help.

[Jonas Yuan]

HI Ankit, which version are you using? EE or CE?

[Danny Francisco]

Hi Jonas,

I'm new with LDAP. I just successfully installed OpenLDAP in our server. I'm still trying to figure out the structure of my LDAP tree and this article really helped. where can I get lpe9000-ldap-ee6010-portal-impl-jdk5.jar patch? I think I will need it.

[Jonas Yuan]

HI Danny, sorry to miss your post.

Which version are you using? CE or EE?

Thanks

Jonas

[Vamsi Tekuru]

Hi Jonas,

Nice post,We are using Liferay Portal Enterprise Edition 6.0 EE SP1 and we are facing a strange issue, LDAP import is failing for the usergroup having users greater tha 17k.import process is being aborted from then.No errors being written to the logs.

[Jonas Yuan]

Hi Vamsi, thanks.

You may drop an email to [e-mail], where we could discuss more details.

Best regards,

Jonas

[Vamsi Tekuru]

Hi i Jonas,

I have identified and fixed this issue.Thanks

[Danny Francisco]

Hi Jonas,

I'm using 6.0.5 CE.

Best regards,

Danny