Importing users, roles and organizations into Liferay 7

[Thomas Kellerer, modificado 7 Anos atrás.]

We need to upgrade our Liferay 6.0 to Liferay 7

As the automatic upgrade of the database does not work (neither incrementally 6.0 > 6.1 > 6.2 > 7.0, nor directly to 7.0), we need to manually copy the data from the old installation to the new one.

We have already put up with the fact that we can't migrate the content, but we have about a 500 users and 200 organizations with several roles and permissions attached to them, so it's pretty unfeasible to re-create all that manually in the new installation (without missing something)

Is there some way of exporting the necessary information that can be imported automatically into the new installation?

The respective tables in the database have changed quite substantially (what is this new mvccversion column anyway?) so I guess I can't just export the tables from 6.0 into a CSV file and import that into the new database? So I was looking for a Java API that would let me create users and everything attached to them through code, but wasn't very successful as the Java classes are not really documented.

[Jorge Díaz, modificado 7 Anos atrás.]

You can export and import users, roles and organizations using LAR files, but it is only possible to do it between same Liferay version, so you have to first upgrade to Liferay 7.

Thomas Kellerer:

We need to upgrade our Liferay 6.0 to Liferay 7
As the automatic upgrade of the database does not work (neither incrementally 6.0 > 6.1 > 6.2 > 7.0, nor directly to 7.0), we need to manually copy the data from the old installation to the new one.

What kind of error are you detecting during upgrade? You can directly execute upgrade process from 6.0 to 7.0

Thomas Kellerer:

The respective tables in the database have changed quite substantially (what is this new mvccversion column anyway?) so I guess I can't just export the tables from 6.0 into a CSV file and import that into the new database?

It is not a good idea to manually modify Liferay database, you can break your system in case you forget to set or update any new data / relation added during upgrade process.

[Thomas Kellerer, modificado 7 Anos atrás.]

Jorge Díaz:

What kind of error are you detecting during upgrade?

The upgrade tool complained that I need to first migrate the permission algorithm.
But when doing that (e.g. through the control panel in 6.0) I get this error:

16:35:23,623 FATAL [ConvertProcessMessageListener:35] Unable to process message {destinationName=liferay/convert_process, responseDestinationName=null, responseId=null, payload=com.liferay.portal.convert.ConvertPermissionAlgorithm, values=null}
com.liferay.portal.convert.ConvertException: java.sql.BatchUpdateException: Batch entry 20 INSERT INTO ResourcePermission (resourcePermissionId, companyId, name, scope, primKey, roleId, actionIds) VALUES (473, 10112, '103', 4, '15201_LAYOUT_103', 101
20, 0) was aborted: ERROR: duplicate key value violates unique constraint "ix_8d83d0ce"
  Detail: Key (companyid, name, scope, primkey, roleid)=(10112, 103, 4, 15201_LAYOUT_103, 10120) already exists.  Call getNextException to see other errors in the batch.
        at com.liferay.portal.convert.ConvertProcess.convert(ConvertProcess.java:53)
        at com.liferay.portal.convert.messaging.ConvertProcessMessageListener.doReceive(ConvertProcessMessageListener.java:47)
        at com.liferay.portal.convert.messaging.ConvertProcessMessageListener.receive(ConvertProcessMessageListener.java:32)
        at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:63)
        at com.liferay.portal.kernel.messaging.SerialDestination$1.run(SerialDestination.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.BatchUpdateException: Batch entry 20 INSERT INTO ResourcePermission (resourcePermissionId, companyId, name, scope, primKey, roleId, actionIds) VALUES (473, 10112, '103', 4, '15201_LAYOUT_103', 10120, 0) was aborted: ERROR: duplica
te key value violates unique constraint "ix_8d83d0ce"
  Detail: Key (companyid, name, scope, primkey, roleid)=(10112, 103, 4, 15201_LAYOUT_103, 10120) already exists.  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleCompletion(BatchResultHandler.java:163)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:480)
        at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:791)
        at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1547)
        at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
        at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
        at com.liferay.portal.upgrade.util.Table.populateTableRows(Table.java:532)
        at com.liferay.portal.upgrade.util.Table.populateTable(Table.java:494)
        at com.liferay.portal.upgrade.util.Table.populateTable(Table.java:437)
        at com.liferay.portal.convert.ConvertPermissionAlgorithm.convertToBitwise(ConvertPermissionAlgorithm.java:169)
        at com.liferay.portal.convert.ConvertPermissionAlgorithm.doConvert(ConvertPermissionAlgorithm.java:660)
        at com.liferay.portal.convert.ConvertProcess.convert(ConvertProcess.java:44)
        ... 7 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "ix_8d83d0ce"
  Detail: Key (companyid, name, scope, primkey, roleid)=(10112, 103, 4, 15201_LAYOUT_103, 10120) already exists.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:469)
        ... 17 more

The workaround I found was to drop that unique index, let Liferay do the migration, then clean up the duplicates and re-create the index.

Jorge Díaz:

You can directly execute upgrade process from 6.0 to 7.0

Apparently, not because the migration tool stops with

15:49:58,044 INFO [main][StartupHelper:149] Upgrading from 6006 to 7002 is not supported

It has been a while since I tried this, and back then I also tried first migrating to 6.1 then to 6.2 then to 7.0 and I recall many other strange database errors during that process that I didn't want to investigate further.

But now we are forced to migrate to Liferay 7 and it seems I need to find a solution.

But if we can't migrate the database or import the users, groups, organizations and permissions in an automated way, then migrating to Liferay 7 is not an option.

[Thomas Kellerer, modificado 7 Anos atrás.]

So going from 6.0 to 6.1 (after fixing the duplicate key errors during the algorithm migration) then going to 6.2 does work. There were many error during the migration of the images (we use image.hook.impl=com.liferay.portal.image.DatabaseHook).
But we can live with that.

The 6.0 to 6.1 upgrade also reported the following error:

 [java] 07:30:59,598 INFO  [main][BaseDB:96] create unique index IX_431A3960 on VirtualHost (hostname);
[java] Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Duser.language=en -Dswing.boldMetal=false
[java] 07:30:59,611 WARN  [main][BaseDB:104] ERROR: could not create unique index "ix_431a3960"
[java]   Detail: Key (hostname)=(localhost) is duplicated.: create unique index IX_431A3960 on VirtualHost (hostname);

But that doesn't stop the process from finishing. The same error was reported again when doing the 6.1 to 6.2 migration

However the migration from 6.2 to 7.0 now fails with different errors.
The first one is:

07:48:16,475 WARN  [main][UpgradeProcess:350] Fallback to recreating the table
org.postgresql.util.PSQLException: ERROR: cannot drop index listtype_pkey because constraint listtype_pkey on table listtype requires it_  Hint: You can drop constraint listtype_pkey on table listtype instead. [Sanitized]
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301)
	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264)
	at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244)
	at com.liferay.portal.dao.db.BaseDB.runSQL(BaseDB.java:291)
	at com.liferay.portal.dao.db.BaseDB.runSQL(BaseDB.java:261)
	at com.liferay.portal.kernel.dao.db.BaseDBProcess.runSQL(BaseDBProcess.java:52)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.alter(UpgradeProcess.java:314)
	at com.liferay.portal.upgrade.v7_0_0.UpgradeListType.doUpgrade(UpgradeListType.java:27)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:88)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:115)
	at com.liferay.portal.upgrade.UpgradeProcess_7_0_0.doUpgrade(UpgradeProcess_7_0_0.java:83)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:88)
	at com.liferay.portal.kernel.upgrade.util.UpgradeProcessUtil._upgradeProcess(UpgradeProcessUtil.java:175)
	at com.liferay.portal.kernel.upgrade.util.UpgradeProcessUtil.upgradeProcess(UpgradeProcessUtil.java:143)
	at com.liferay.portal.kernel.upgrade.util.UpgradeProcessUtil.upgradeProcess(UpgradeProcessUtil.java:125)
	at com.liferay.portal.events.StartupHelper.upgradeProcess(StartupHelper.java:164)
	at com.liferay.portal.events.StartupHelperUtil.upgradeProcess(StartupHelperUtil.java:81)
	at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:157)
	at com.liferay.portal.tools.DBUpgrader.main(DBUpgrader.java:103)

But that doesn't stop the migration process.

However the upgrade tool fails further down with this exception:

07:48:21,529 INFO  [main][UpgradeProcess:97] Completed upgrade process com.liferay.portal.upgrade.UpgradeProcess_7_0_1 in 218ms
com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.upgrade.UpgradeException: java.lang.NullPointerException
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:91)
	at com.liferay.portal.kernel.upgrade.util.UpgradeProcessUtil._upgradeProcess(UpgradeProcessUtil.java:175)
	at com.liferay.portal.kernel.upgrade.util.UpgradeProcessUtil.upgradeProcess(UpgradeProcessUtil.java:143)
	at com.liferay.portal.kernel.upgrade.util.UpgradeProcessUtil.upgradeProcess(UpgradeProcessUtil.java:125)
	at com.liferay.portal.events.StartupHelper.upgradeProcess(StartupHelper.java:164)
	at com.liferay.portal.events.StartupHelperUtil.upgradeProcess(StartupHelperUtil.java:81)
	at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:157)
	at com.liferay.portal.tools.DBUpgrader.main(DBUpgrader.java:103)
Caused by: com.liferay.portal.kernel.upgrade.UpgradeException: java.lang.NullPointerException
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:91)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:115)
	at com.liferay.portal.upgrade.UpgradeProcess_7_0_1.doUpgrade(UpgradeProcess_7_0_1.java:41)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:88)
	... 7 more
Caused by: java.lang.NullPointerException
	at com.liferay.util.Encryptor.serializeKey(Encryptor.java:256)
	at com.liferay.portal.upgrade.v7_0_1.UpgradeCompany.updateCompanyKey(UpgradeCompany.java:49)
	at com.liferay.portal.upgrade.v7_0_1.UpgradeCompany.doUpgrade(UpgradeCompany.java:34)
	at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:88)
	... 10 more

[Alberto Chaparro, modificado 7 Anos atrás.]

Hi Thomas,

Please read the upgrade documentation:
https://dev.liferay.com/discover/deployment/-/knowledge_base/7-0/upgrading-to-liferay-7

To answer this:

15:49:58,044 INFO [StartupHelper:149] Upgrading from 6006 to 7002 is not supported

You can find there the following explanation:

To upgrade to the latest release directly, you must be coming from Liferay Portal 6.0.12 or higher.

If you’re on Liferay Portal 6.0.11 or below, you should upgrade to Liferay Portal 6.2 before approaching an upgrade to the Liferay Portal CE 7.0 platform. Please see the Upgrading to Liferay Portal 6.2 article for information on upgrading to Liferay Portal 6.2 first.

Best regards.

[Thomas Kellerer, modificado 7 Anos atrás.]

As I said: the migration tool does run once I have migrated the database to 6.2 but fails with the NullPointerException I have shown in my last post. I don't know what the reason for that is.

[Thomas Kellerer, modificado 7 Anos atrás.]

Jorge Díaz:

You can export and import users, roles and organizations using LAR files, but it is only possible to do it between same Liferay version, so you have to first upgrade to Liferay 7.

I finally managed to upgrade the database to 7.0 but I can't find any menu item to export users, organizations, roles and privileges. It used to be in the "context menu" of the User's portlet in the control panel, but there is no export option anywhere to be found.
The "Export/Import" tasks that is available, can can only export content (pages, web content)

[Thomas Kellerer, modificado 6 Anos atrás.]

Is there any way to get the users out of the database in a format that Liferay can import?

The export feature is completely broken in the migrated Liferay 7 instance, see my other thread here:
https://web.liferay.com/community/forums/-/message_boards/message/91017212

I was playing around with using the JSON Web Services but I can't get the curl command that is generated to work (because of a security exception even with supplying a valid p_auth parameter) - apparently those WebService only work within the browser, but not externally.

And if I can't automate the import the JSON WebService is no help (it would be faster to enter those 500 users manually through the Liferay UI then)

[Thomas Kellerer, modificado 6 Anos atrás.]

I tried the export feature with a "Vanilla" Liferay 7 installation.

When checking the contents of the LAR File only Organizations are exported, but not the users. How can I include the users?

In the configuration section for "Content" I can only check "Organizations".



This is with Liferay 7.0 GA4

[Thomas Kellerer, modificado 6 Anos atrás.]

Judging by the fact that I got no answers, I have to assume that preserving user accounts through a migration over 3 versions is not something that is supposed to work.

[Christoph Rabel, modificado 6 Anos atrás.]

I just skimmed the text, so maybe I am off:
You might try to export them to ldap and import them again (I hope, 6.0 can do that, I forgot)

Another thing I did once was using the scripting console. Print all users on the screen, but in the format:
userLocalService.addUser(....);

Copy those lines to the Liferay 7 scripting console and do a couple of hundred adds.