Demystifying Liferay's Open Development

"What is Liferay working on?"

This question is probably the most popular high-level question asked by the our community.  Whether you are an enthusiast, open source Liferay developer, partner, or potential or long-time customer, you want to know what's coming up and what's being worked on.  On my first day on the job, in my first blog post as your community manager, I promised to "Engage with you on the roadmap for 6.1 and beyond".  Today marks a significant step towards closing the gap between what you think the Liferay Community is doing and what we are actually doing.  This information has sort of been there all along, in the form of individual tickets in JIRA, but our process is evolving and becoming even more transparent and easier to understand. 

A Brief History of Liferay's Development Process

In the early days, when the community was very small, pretty much everyone was in tune with what the community leaders were doing.  There were twice-yearly meetups in LA, the source code base was small, and there wasn't a ton of people issuing requests and complaints and suggestions for improvements.

As the popularity and user base of Liferay grew, and as companies started using it more and more for mission critical applications, the need for knowing what was being worked on (and therefore what could be expected in the next release) grew proportionally.  Since Liferay (and liferay.com) was already a social collaboration tool, and had amazing features like a Wiki, The Roadmap was enshrined as a wiki page (here's the one for 5.0) for all to see and collaborate on, with the expectation that it would be nurtured and updated frequently.

Unfortunately, the wiki turned out to be the place where the roadmap went to die a quiet, ignominious death.  The problem was not that wikis are inherently bad, it's just not the right tool for a quickly evolving project with lots and lots of details.  No one was willing to babysit the wiki page and ensure its accuracy on a daily basis during product release cycles, because it was not much more than an electronic whiteboard (and not very powerful or fun to use).   So, we were left with an afterimage of what the project was supposed to be at the beginning of the release cycle.

In the meantime, Liferay Program Management was becoming more adept at using JIRA to its fullest potential, to manage the huge activity occurring on issues.liferay.com.  What was once essentially a large, flat list of bugs and improvements (with the same workflows and same metadata - name, description, assignee, etc) was slowly transforming into a powerful issue management system with customized metadata, workflows, new filters, and OpenSocial visualizations built on top of the "flat list of bugs", giving the community never-before-seen ways of visualizing and managing issues.

In addition, Liferay itself was changing - trying out new development models and seeing which ones best fit the company's and community's development style.  We're not pioneers here - effective development teams have been using various models like XP, Scrum, Kanban, etc for years, so we were just picking those that seemed to match our development culture best, modifying as needed, and marching on.  In October 2011 I visited our Madrid office, and was pleasantly surprised to see the whiteboard in the office look something like this:

(   Some rights reserved  by  Plutor)

I was like "ahhh.. low tech solutions for low tech problems.  Genius!".  If I could set up a camera in this office, and take a picture every hour and upload it to liferay.com, then we'd all have a pretty clear idea of what was currently being worked on and we'd have a constantly updating picture of what Liferay thought should be in the next release.  But a camera is too fragile, and we can't all be in Spain (though that would be nice...).  What they were practicing was a form of Kanban, prototyping it with the development team there.  We have now taken it from pushpins and post-it notes to JIRA and one of its many plugins called "Greenhopper".

This post isn't meant to educate you on what Kanban is or how to use Greenhopper.   Instead, I'll explain how it can be used to visualize the work of the Liferay developer community (Side note: I tend to lump Liferay employees with the rest of the community!).

The Liferay Activity Board

Many product presentations end with a Roadmap Slide - this is the slide where high level features are paired with expected due dates, and is usually accompanied by some kind of disclaimer which absolves the presenter of all responsibility for the dates and content presented.

This is not a roadmap.

Rather, what we now have at our fingertips is a view onto what was formerly a post-it note board maintained in Liferay's development centers. It's better and more useful than a static roadmap.  At this point you may be asking yourself "So what, I don't want to see the 50 bugs that were fixed today, and I don't care that Ryan was blocked on LPS-23133 and Julio's average cycle time is 7.5 days.  I want to know what's coming in the next version of Liferay!"  Fair enough.  The nice thing about how we are using JIRA and Greenhopper is that it is now possible to get exactly that - a view onto the current, major, high-level themes being worked on for the next version of Liferay, ignoring all of the boring details.  With the new Community Rapid Board, you can do exactly that:

It's called a Rapid Board and is essentially a 2D table showing Stories flowing through various states.  A Story is a supporting artifact for a set of requirements.  The labeled rows (Quick Wins, Collaboration Calendar, etc, also called Swimlanes) represent groups of related stories under the same feature area that are currently being worked.  Columns represent current state of the stories being worked.  So, for example, for Liferay 6.2, you can see that the following high level items are planned or being worked on:

Of course, the set of visible feature areas for 6.2 is evolving, so this list will change over time.  You can click on individual stories and drill down to your heart's content.  Eventually you'll arrive at one or more JIRA tickets representing the lowest level of development subtasks assigned to individual developers, which can ultimately be mapped to source code changes at github.  Following this rabbit hole is very instructive and helpful in the future when unraveling issues :)

Quick Wins

Quick Wins allow any Liferay Developer to work on whatever they want under certain conditions.  Often times, community contributions that you make are handled as Quick Wins.  These are shortcuts to the development process - things that can typically be handled generally within one day, without undue process.  Liferay has always benefited from these and we want to encourage them to keep happening.

How is this different?

Liferay is not only an open source product, its development process is also very open, and now we are making it even easier to keep track of what's going on, which helps you get involved earlier and provide feedback on how you want to see the product evolve.

Of course, it's just as easy to neglect a fancy tool like JIRA as it is to neglect a wiki, so how is this any better?  The difference is that this tool is now in the critical path for daily (and sometimes) hourly work of our community of developers.  Neglect on anyone's part will cause pain for many other people, so it's in everyone's best interest to keep things accurate and use the tool for its intended purpose.  Not only that, it's easy and fun to use (it can be very therapeutic to weed through a complex board every day and see the progress of tickets).  And in the end, it provides a rich set of tools to measure progress and find roadblocks and places where development process can improve.  Of course, there is a higher learning curve compared to a wiki page, but in the end it is worth it.

What does this mean for me?

Right now, these JIRA-based tools are used internally by Liferay's development teams.  Outside of these teams, the process for the community remains the same (for now) - file bugs at issues.liferay.com, discuss new features and improvements via the forums and Proposals Wiki, and then file new feature requests via issues.liferay.com.  These will in time be promoted to  Stories (often times Quick Wins).  As a separate effort, your Community Leadership Team and Liferay Product Management teams are  working on improving the "ideation" experience within our community, to make it very easy to crowd-source new ideas and promote community-born ideas into reality much faster, but that's a separate blog post in the near future (I promise!).

I am sure that "Roadmap Slides" won't be going away anytime soon - you'll probably see several of them in the upcoming Liferay Symposiums - but the extra granularity and daily (and sometimes hourly) reflection of the current state of Liferay within these Rapid Board views gives everyone a realtime view onto the what is happening with Liferay development.

[jelmer kuperus]

So somehow you went from

"I was like "ahhh.. low tech solutions for low tech problems. Genius!""

to

"lets install a plugin that has more knobs you can turn than the starship enterprise"

[Hitoshi Ozawa]

I personally like the post-it. Have it all around me. :-)

So, when in the next release going to come out? lol

[James Falkner]

@jelmer - - at least we have an OCD Captain Kirk at the helm

@hitoshi - *groan* and

[Aniceto P Madrid]

Nice! Marvelous! Sensitive! Open!
...
But 6.1 CE has a lot of bugs. When will be available 6.1.1?

Thanks

[Jorge Ferrer]

Very nicely explained James!

[Brian Kim]

The Rapid Board stuff is pretty nice.

[Sampsa Sohlman]

I did look the board and it is awesome. Just got curious some new things that you are building on .. have to take update from "trunk"

[Corné Aussems]

Strangely enough i can find new features like;
- AP velocity templates; http://issues.liferay.com/browse/LPS-13699
- HTML5 Audio Support for Document Library http://issues.liferay.com/browse/LPS-22919

and Roadmap issues like;
- Portal Management Console
- Monitoring Real-time Application Tuning
- Monitoring Cluster-wide Management

[James Falkner]

@corne They are all there, represented as stories.. For AP velocity variables, it is part of the LPS-26808 "story" .

As for the roadmap issues, #1 this is NOT a roadmap and #2 where did you get that list of items?

[Michael Louis Guerra]

This is great, I will share with customers and prospects.