missing images after migration to 6.2

[Marco Ferretti，修改在7 年前。]

Hi

In the attempt to migrate to version 7 I have first migrated an instance of 6.0.6 (community) to latest 6.2 (GA5).

While the migration did not show any error in the logs, I am facing a strange issue: I have some images that have not been migrated. Checking the old instance (currently in production) I can see all of them, while the new instance has some holes.

The old instance used the database hook impl for the image gallery and, according to the migration manual I only needed to set

image.hook.impl=com.liferay.portal.image.DatabaseHook

during upgrade in order for the library to be moved.

Now, when I try to get one of the images that are not showing I get

http://fishnet-dev.jrc.it:9280/documents/10186/10891/3043/f1c60ba2-2197-45ff-a195-745bbb627b5c?version=1.0&t=1316094208892&download=true

while if I try to download one of the images that are correctly showing (inspecting the network traffic) I can see that the url generated is

http://fishnet-dev.jrc.it:9280/documents/10186/10891/3037/f2d726f5-dd65-4769-bffa-600e1c6298e1?version=1.0&t=1316094209287&download=true

I tried to check the database in order to understand if there was something missing.
The table dlfileentry has an entry for both the uuid in the two test url and they point to an entry in dlfileversion and dffolder (since they are actually in a folder within the image gallery).
At this point I tried to reconstruct the file system path from the database entries.
*for the link that works,the database field "name" is 11355 which corresponds to the path in the file system :

<liferay_data>/document_library/10112/10891/11/35/11355.afsh
</liferay_data>

*for the file that is not showing the database field "name" is 11340 which should correspond to

<liferay_data>/document_library/10112/10891/11/34/11340.afsh
</liferay_data>

but the path is not there.

I then checked to see how many images should have been migrated (just for this folder, within the site) by issuing

select * from dlfileentry where groupid=10186 and companyid=10112 and name like '11%'

and tried to match what I have in the document_library folder on the file system.
Well, something must have gone terribly wrong during the migration as I had nothing in the logs but I am randomly missing files within folders (eg, I have a database entry "11344" but I don't have a "<LIFERAY_DATA>/document_library/10112/10891/11/34/11344.afsh" folder while I have a <LIFERAY_DATA>/document_library/10112/10891/11/34/11345.afsh and a "11345" entry) and whole folders (eg. I have "11336","11337","11338" and "11339" entries but I do not have a "<LIFERAY_DATA>/document_library/10112/10891/11/33")

Downloading and uploading images (from the old instance to the new one) is hardly acceptable as there are 1200 entries in the table images ...

Can anyone suggest where I should start digging ?
Thanks in advance,

Marco

[Alberto Chaparro，修改在7 年前。]

Hi Marco,

You can check the logic for converting IGImages and Images into DLFileEntries and files in filesystem here:
com.liferay.portal.upgrade.v6_1_0.UpgradeImageGallery.doUpgrade()

As you can see there are two kind of images:
1- Those ones which were in IGImage, for them the old IGImage.name is now DLFileEntry.title. You can check them in the filesystem by the DLFileEntry.name, there should be something like this /document_library/companyId/repositoryId/**/DLFileEntry.name/version
2- Those ones which weren't in IGImage (just the file in Image table). They should be in the filesystem in this way:
/document_library/companyId/repositoryId/**/(image.getImageId() + StringPool.PERIOD + image.getType())/version

You can also take into account that they this can change depending of the value for:

dl.store.impl=com.liferay.portlet.documentlibrary.store.FileSystemStore
dl.store.file.system.root.dir=${liferay.home}/data/document_library

(For AdvancedFileSystemStore is similar but I bit more complex)

I hope it helps.

Regards.

[Marco Ferretti，修改在7 年前。]

Hi Alberto,

thanks for the reply.
I tried following the code in com.liferay.portal.upgrade.v6_1_0.UpgradeImageGallery.

As far as I understand :

• It adds the entry for the IGImage type
• Update (and adds) the values from IGFolder in DLFolder
• Drops IGFolder
• Converts IGImage into DLFileEntry
• Drops IGImage
• Fix permissions

Now, if my understanding is correct, there's a place where it fails (in my installation) and silently swallows the exception : in updateIGImageEntries it does the following :

		try {
			con = DataAccess.getUpgradeOptimizedConnection();

			ps = con.prepareStatement(
				"select fileEntryTypeId, companyId from  " +
					"where name = ?");

			ps.setString(1, DLFileEntryTypeConstants.NAME_IG_IMAGE);

			rs = ps.executeQuery();
...

The SQL statement should expand in

select fileEntryTypeId, companyId from DLFileEntryType where name = 'Image Gallery Image'

but in my database (after the migration) the latter returns no rows . Checking the table I have the following :

uuid_                                               fileentrytypeid       groupid               companyid             userid                username                                            createdate                     modifieddate                   name                                                description                                         fileentrytypekey                                    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
03f29bec-156c-4142-ab9c-e6a9fd16b03a                1003501               345944                10112                 10115                                                                     2016-10-10 07:44:31.978        2016-10-10 07:44:31.978        <!--?xml version='1.0' encoding='UTF-8'?--><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa image gallery 575c8d05-c821-410d-9d37-547b80db0ed6 0 2016-10-10 07:45:31.202 <?xml version="1.0" encoding="UTF-8" ?><root availa basic-document 0cfc64ab-efdd-4fd0-8916-b315c05f89b2 1009253 345944 10112 10115 2016-10-10 07:49:31.525 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa contract 95388c13-75c5-4cfb-8d3b-63678678a4c7 1009255 345944 10112 10115 2016-10-10 07:49:31.558 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa marketing banner 5b947034-9197-48f6-a2ed-f6e8f739d213 1009257 345944 10112 10115 2016-10-10 07:49:31.58 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa online training 469def87-705d-4b80-8d7d-dbd416561b57 1009259 345944 10112 10115 2016-10-10 07:49:31.606 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa sales presentation < code></root></root></root></root></root></root></root></root></root></root></root>

<br><br>Expanding the value for name I get : <br><pre><code> <!--?xml version='1.0' encoding='UTF-8'?--><root available-locales="en_US" default-locale="en_US"><name language-id="en_US">Image Gallery Image</name></root> </code></pre><br>What puzzles me is that the upgrade process expects a string value in name while in the database there's an XML entry and looking at the function that creates the entries ( addIGImageDLFileEntryType) it seems that it should add DLFileEntryTypeConstants.NAME_IG_IMAGE both in name and description : <br><pre><code> sb.append("insert into DLFileEntryType (uuid_, groupId, "); sb.append("companyId, userId, userName, createDate, "); sb.append("modifiedDate, name, description, fileEntryTypeId) "); sb.append("values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"); ps = con.prepareStatement(sb.toString()); ps.setString(1, PortalUUIDUtil.generate()); ps.setLong(2, groupId); ps.setLong(3, companyId); ps.setLong(4, userId); ps.setString(5, userName); ps.setTimestamp(6, createDate); ps.setTimestamp(7, modifiedDate); ps.setString(8, DLFileEntryTypeConstants.NAME_IG_IMAGE); ps.setString(9, DLFileEntryTypeConstants.NAME_IG_IMAGE); ps.setLong(10, increment()); </code></pre><br>could it be that there is some jdbc parameter that HAS to be set or by default is set to the wrong type ?<br><br>Will check the PG documentation and update this thread. <br>In the meantime, if you have any idea, please let me know<br><br>Regards, <br><br>Marco F.

[Marco Ferretti，修改在7 年前。]

Marco Ferretti:

but in my database (after the migration) the latter returns no rows . Checking the table I have the following :

uuid_                                               fileentrytypeid       groupid               companyid             userid                username                                            createdate                     modifieddate                   name                                                description                                         fileentrytypekey                                    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
03f29bec-156c-4142-ab9c-e6a9fd16b03a                1003501               345944                10112                 10115                                                                     2016-10-10 07:44:31.978        2016-10-10 07:44:31.978        <!--?xml version='1.0' encoding='UTF-8'?--><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa image gallery 575c8d05-c821-410d-9d37-547b80db0ed6 0 2016-10-10 07:45:31.202 <?xml version="1.0" encoding="UTF-8" ?><root availa basic-document 0cfc64ab-efdd-4fd0-8916-b315c05f89b2 1009253 345944 10112 10115 2016-10-10 07:49:31.525 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa contract 95388c13-75c5-4cfb-8d3b-63678678a4c7 1009255 345944 10112 10115 2016-10-10 07:49:31.558 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa marketing banner 5b947034-9197-48f6-a2ed-f6e8f739d213 1009257 345944 10112 10115 2016-10-10 07:49:31.58 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa online training 469def87-705d-4b80-8d7d-dbd416561b57 1009259 345944 10112 10115 2016-10-10 07:49:31.606 <?xml version="1.0" encoding="UTF-8" ?><root availa <?xml version="1.0" encoding="UTF-8" ?><root availa sales presentation < code></root></root></root></root></root></root></root></root></root></root></root>

<br>

<br><br>ok, that table gets updated AFTER the document library is migrated. <br><br>I have debugged the upgrade process in eclipse (attached to local upgrade) but could not find any flaw there ...

[Alberto Chaparro，修改在7 年前。]

Hi Marco,

Maybe you could increase the log for that upgrade process and check it.

Anyway, apart from checking the images urls, have you tried to access to them in the document library after the upgrade? Maybe the images are there store in a different path in the filesystem.

[Marco Ferretti，修改在7 年前。]

Hi Alberto,

I noticed that the latest stable is not GA5 but GA6 so I started from there.
Then, I have found out that the image gallery/ document library had been migrated in the past and that the migration had some leftovers (tables that should have been empty in after the migration actually were not ) so I manually cleaned them before starting the upgrade process.

As far as I can tell, at the moment the migration went fine but we're still manually checking that the document library and the image gallery has been correctly migrated. Once that proves to be ok we'll migrate the themes and hand over the test platforms to the site owners in order to check the content and the permissions.

Hopefully I will be able to update this thread within the week with the results.

Regards,

Marco

[Alberto Chaparro，修改在7 年前。]

That's great Marco. Looking forward your news.

Thanks!