Updating and deploying your code made easy!

Are you developing a large amount of plugins?

Ever found yourself frustrated with having to manually deploy war-files to development, test and production (clustered?) environments?  When performing Quality Assurance on a release it is important that it is the same build / release that is eventually deployed to production. This might sound like an easy task but there are many traps out there if you're not careful.

 

This blog post is about how we improved our release system with the help from Maven and a custom portlet. Performing releases is now just a click of a button!

 

Our system

Jenkins - builds our source code, polling subversion regularly. Previously also contained jobs to deploy war-files into various systems.

Artifactory - A Maven repository where artefacts built by Jenkins are stored.

Liferay - different environments using Liferay EE. Environments like development, test, production (live) and production (backup)

Update portlet - a portlet written by us that enable us to update to specific release versions

 

Earlier

In our environments we used a job in Jenkins that retrieved the war-file (from appropriate source, more on that later on) and transferred the file to a specified environment. The job was configurable and you had to choose:

• What plugin (one of 30'ish plugins available)
• What environment to deploy to (test, production live, production backup, development etc)

That meant that if we wanted to deploy 5 plugins in a release we had to run the job 5 times for each server, in our case with test and two production machines that meant 15 times. With 10 plugins 30 clicks, where 10 plugins isn't very uncommon for us. Doesn't sound like a big of a problem but the crucial part is that you have to deploy the exact same version for each machine, anything else may prove disastrous.

We did that for a couple of years.

 

Improvement

At first we handled all build artefacts within Jenkins and the job mentioned to deploy war files always accessed the latest artefacts from the release build job. That had a couple of drawbacks, for instance it was very difficult to work in feature branches / doing critical patches based on earlier releases.

As the first step towards a solution was to start using Maven. With Maven we were always working on a major version and other minor versions could be in progress in parallel branches. For instance if a critical patch had to be done, a change to the release-branch and a bump in minor version solved that, without affecting the current major version change being done and evaluated in the main trunk.

 

Just to clarify. By using Maven, artefacts were being published to a central (internal) repository. Each published release version is stored there, forever, and cannot be changed.

 

What did we gain by this?

Earlier we didn't really know what artefact was being released to a system since the version was always latest built artefact from the Jenkins release job. In the case of a developer triggering the release job in Jenkins - a new version would be generated and not necessarily the same as previously was installed in test.

Now we have full control over the versions being used. There were also other benefits from turning towards Maven (we won't go back, ever). A choice of many IDE's for instance (since most support Maven fully).

 

But we're still stuck using the Jenkins job to deploy each artefact to each system, the only change is that the job is downloading the artefact from our repository with a specific version in mind.

 

Sunshine, beaches and happy days

(Or. Enter UpdatePortlet.)

 

So. After much thinking (ahem), pondering, donuts and coffees we had an idea. What if ... there was a way ... to see if there was an update available?

(Yes, the new cloud services from Liferay has stolen our idea regarding patches)

Stretching it even further .. what if we could actually update to that version, with just a click of a button? 

 

Said and done.

 

A portlet listing all the currently installed plugins, accessing the version information from each plugin and displaying related information in a pretty UI (datatables really rock) was created.

Using the portlet information (like artefact id and group) we access our Maven repository and pull the latest version information, comparing that to the version installed locally in Liferay.

If a version differs from the latest available we also display a button for updating the plugin and mark the row in a distinct colour to make it easier for the administrator to find any stale plugins.

 

Since our Maven repository also provides meta data regarding SNAPSHOT versions (that is, versions that have not yet been marked as a release version) we could also detect if there was a newer bleeding-edge-version available.

 

Deploying a bundle

We're still stuck with clicking 15 times. Or 30. The more obvious risk now is that we might click 6 times instead of 5 if there was a plugin in a newer release version but was not meant to be deployed at this point in time. Administrators are of course trigger-happy when seeing that updates exists even if these shouldn't be updated quite yet.

 

The need for deploying a fixed set of plugins with specific versions became apparent, and by using the excellent architecture of Maven the solution was pretty neat.

By creating a maven artefact without any real source, just a pom.xml-file, we could list dependencies towards other plugins (and their specific version). The pom (named inhouse to deploy-pom) could then be processed by the Update Portlet which installs each plugin (even in the specified order) with just one-click-interaction with the administrator. The deploy-pom also has its own version which comes in handy, we're using it to describe the sprint we're deploying (we're using 3-week sprints, currently named 15.05, thus is the deploy-pom's version is named 15.05).

 

 

End result

By selecting a release (deploy-pom's version) and clicking install we're done. 6 clicks (including install-button-click) is all we need for all our environments and we're guaranteed that exactly the same amount of plugins have been deployed and also the same version!

 

[Krzysztof Gołębiowski]

Very interesting post! There are very few articles about how to do CI for Liferay properly (actually I don't know any). Custom portlet for managing liferay-maven plugins is quite an interesting idea (do you plan to release it on the marketplace?).

I still use jenkins together with ssh plugin and always had a problem with checking if the deployment is successfull. At the moment there is a bash script waiting for certain log messages (eg. "x portlet has been deployed") and if it's not found, build is marked as FAILED. This is a very error prone solution and gives lots of false negatives. Did you managed to solve this problem?

[Marcus Hjortzén]

No plans on marketplace. To be honest, the code is in no state to be shown to anyone Still, it would be a neat app to put there, if we ever rewrite the portlet we will probably make it open source and available on marketplace.

We had the same problem with determining if a plugin was installed correctly when using Jenkins. In one of the minor releases of 6.1 the text saying ".. portlets has been deployed" even disappeared, so, no, we ended up with just doing a fire-and-forget, having to manually check the log for actual results.

With the plugin however we're getting other options. I believe we're polling the portal asking for the plugin until it appears, with a timeout indicating failure.

Also (not a problem lately) there was a problem where deployment got stuck halvway through and you had to redeploy the same war-file again. To get around that problem we try to install the plugin 3 times before failing completely.

[Krzysztof Gołębiowski]

Didn't you try to utilize the message that is sent on Liferay messagebus when the plugin is successfully deployed? This was my recent idea that could solve problem with verifying plugin deployment status. I planned to install a hook containing message listener that would somehow notice Jenkins when the plugin is deployed.

[Marcus Hjortzén]

I did not. Had no idea!
That's perfect, thank you Krzysztof!