Forums de discussion

Thumbnail generation workaround

thumbnail
Andrea Nicolini, modifié il y a 7 années.

Thumbnail generation workaround

New Member Publications: 16 Date d'inscription: 31/03/16 Publications récentes
Hi all,

I'd like to share a solution I adopted for a problem that I think is quite common among Liferay users.
Our site relies heavily on Liferay generated thumbnail and we often faced problems of misgenerated (eg: CMYK) or missing thumbnails, so I started to think about an alternative.

My first try was to write a webservice which, using liferay api, triggered thumbnail generation on demand, so that if you delete all server thumbnail you can regenerate them in one shot, with no need to open documents in Documents And Media.
It worked, but this approach was still not flexible enough and it put an heavy load on liferay instance while running; so I decided to follow another track.

I ended up writing a simple web service which is only responsible of returning the list of DL paths to source images and DL paths to thumbnail images. Then I wrote a simple bash script that uses these paths to generate thumbnails the way I want.

Here's my webservice:

 
public List<string> getThumbnailsPaths(int totDocs){
                List<string> results = new ArrayList<string>();
                List<string> extsToExt = new ArrayList<string>();
                List<string> extsToPng = new ArrayList<string>();
                List<dlfileentry> files = null;
                DLFileVersion fileVer = null;
                DLFolder fldr = null;
                Date modDate = null;
                String filePath = null;
                String ext = "";
                int j = 0;


                //Valid extensions
                extsToPng.add("eps");
                extsToPng.add("pdf");
                extsToPng.add("tif");
                extsToExt.add("gif");
                extsToExt.add("png");
                extsToExt.add("jpg");
                extsToExt.add("bmp");

                try{
                        if(totDocs == 0)
                                totDocs = DLFileEntryLocalServiceUtil.getDLFileEntriesCount();

                        //fetch documents from DL
                        files = DLFileEntryLocalServiceUtil.getDLFileEntries(0,totDocs);

                        for(DLFileEntry file: files){
                                j++;
                                ext = file.getExtension();
                                if(!extsToExt.contains(ext) &amp;&amp; !extsToPng.contains(ext))
                                        continue;

                                StringBuffer outPath = new StringBuffer("/");

                                outPath.append(file.getCompanyId() + "/");
                                outPath.append(file.getFolderId() + "/");
                                outPath.append(file.getName() + "/");
                                outPath.append(file.getVersion() + "|");
           
                                 fileVer = file.getFileVersion();
                           
                                outPath.append(file.getGroupId());
                                outPath.append(DLUtil.getDividedPath(file.getFileEntryId()));

                                if (fileVer.getFileVersionId() &gt; 0) {
                                        outPath.append("/");
                                        outPath.append(fileVer.getFileVersionId());
                                }

                                if(extsToPng.contains(ext)){
                                        outPath.append(".png");
                                }else{
                                        outPath.append("." + ext);
                                }
                               results.add(outPath.toString());
                        }
                }catch(SystemException | PortalException e){
                        e.printStackTrace();
                }

                return results;

        }

</dlfileentry></string></string></string></string></string></string></string>


And this is my bash script used for actual conversion:


#!/bin/sh
renice -n 10 $$

DLBASEPATH="/opt/liferay/data/document_library"
TNBASEPATH="$DLBASEPATH/10157/0/document_thumbnail"
THUMBSIZE="128x128"
THUMBSIZE1="240x310"
GREPATH="$1"
MAXDBAGE=3600
MAXCONV=100
CURLCMD="curl -s http://localhost:8080/api/jsonws/yourwshere/get-thumbnails-paths -u user:pass -d totDocs=0"

if [ -f "/tmp/thumbs.db.tmp" ]; then
        CURSTAMP=$(date +'%s')
        TDBSTAMP=$(stat -c %Y "/tmp/thumbs.db.tmp")
        if [ $((CURSTAMP - TDBSTAMP)) -gt $MAXDBAGE ]; then
                echo "Querying Liferay DB..."
                $CURLCMD | jq -r '.[]' &gt; /tmp/thumbs.db.tmp
        fi
else
        echo "Querying Liferay DB..."
        #I'm using jq to parse json list of strins
        $CURLCMD | jq -r '.[]' &gt; /tmp/thumbs.db.tmp
fi

#Skip if source file doesn't match filter
if [ -n "$GREPATH" ]; then
        echo "Active filter: $GREPATH"
        grep "$GREPATH" /tmp/thumbs.db.tmp &gt; /tmp/thumbs.db
else
        cp /tmp/thumbs.db.tmp /tmp/thumbs.db
fi


while read LINE; do

        if [ $MAXCONV -eq 0 ]; then
                break
        fi

        IFS='|' read -r -a DOCPATHS &lt;&lt;&lt; $LINE
        SRCPATH="$DLBASEPATH${DOCPATHS[0]}"
        TNPATH="$TNBASEPATH/${DOCPATHS[1]}"
        TNPATH1=$(echo $TNPATH | sed 's/\(^[a-z0-9/_]\+\)\(\.[a-z]\+\)/\1-1\2/') #This is custom thumbnail path: add "-1" in dirname.


        if [ -f $SRCPATH ]; then
                mkdir -p $TNPATH
                mkdir -p $TNPATH1

                #Process only out-of-date files
                if [ -f "$TNPATH/.tstamp" -a -f "$TNPATH/1.0" ]; then
                        SRCSTAMP=$(stat -c %Y "$SRCPATH")
                        TNTSTAMP=$(stat -c %Y "$TNPATH/1.0")
                        TSTSTAMP=$(stat -c %Y "$TNPATH/.tstamp")
                        if [ $TNTSTAMP -ge $TNTSTAMP -a $SRCSTAMP -le $TSTSTAMP ]; then
                                continue
                        fi
                fi

                echo -n "Processing: $SRCPATH " 

                DBLPATH=$(echo $TNPATH | sed 's/\(^[a-z0-9/_]\+\).[a-z]\+/\1\*/')
                DBLCONT=$(ls -d $DBLPATH | wc -l)

                if [ $DBLCONT -gt 2 ]; then
                        echo "Double dir error: $TNPATH"
                        echo $TNPATH &gt;&gt; /opt/liferay/data/doubledir.log
                        continue
                fi

                MIME=$(file -b --mime-type $SRCPATH)
                case "$MIME" in

                        image/*)
                                echo "-&gt; IMG:$TNPATH/1.0"
                                convert "$SRCPATH" -thumbnail $THUMBSIZE1 "png:$TNPATH1/1.0"
                                convert "$TNPATH1/1.0" -thumbnail $THUMBSIZE "$TNPATH/1.0"
                                touch "$TNPATH/.tstamp"
                                MAXCONV=$((MAXCONV-1))
                        ;;
                        application/*)
                                echo "-&gt; PDF:$TNPATH/1.0"
                                #Imagemagick hangs on some pdfs, so I generate png first using gs.
                                gs -q -sDEVICE=png16m -dNOPAUSE -dBATCH -dFirstPage=1 -dLastPage=1 -sOutputFile="/tmp/thumbnail.png" "$SRCPATH"
                                convert "/tmp/thumbnail.png" -thumbnail $THUMBSIZE1 "png:$TNPATH1/1.0"
                                convert "$TNPATH1/1.0" -thumbnail $THUMBSIZE "$TNPATH/1.0"
                                touch "$TNPATH/.tstamp"
                                MAXCONV=$((MAXCONV-1))
                        ;;
                        *)
                                echo "-&gt; unmanaged mime type: $MIME"
                        ;;
                esac
        #else
        #       echo "Not found: $SRCPATH"
        fi


done &lt; /tmp/thumbs.db



With this simple solution I managed to work with thumbnails in a much more comfortable way and I hope this can be of some help to someone who is facing similar problems.

Bye.
thumbnail
David H Nebinger, modifié il y a 7 années.

RE: Thumbnail generation workaround

Liferay Legend Publications: 14914 Date d'inscription: 02/09/06 Publications récentes
Andrea, thanks for the contribution, but what version of Liferay are you having these issues with?

Have you enabled the external tools (open office, image magick, ghostscript and xuggler)? These tools, if enabled, will be used to create thumbnails in lieu of the built-in Liferay code.

I always recommend enabling these tools to offload the work out of Liferay and generate better, consistent results.
thumbnail
Andrea Nicolini, modifié il y a 7 années.

RE: Thumbnail generation workaround

New Member Publications: 16 Date d'inscription: 31/03/16 Publications récentes
David,

I'm using Liferay CE 6.2 GA6 and yes, I enabled Imagemagick but after looking at the code and at some bug report like this:
https://issues.liferay.com/browse/LRDOCS-1616, I came to the conclusion that it is not used the way one would expect.
Actually it seems to be used only for CYMK to RGB conversion, but I might be wrong.

Furthermore, using an external script I can run conversion whenever I want, and I can also generate thumbnail for files that aren't normally thumbnailed.

I know, this is a quick and dirty solution, but it meets my needs.

Thank you for your interest.

Bye.