OSGi Subsystems and Why You Want Them

So last week I'm sitting in an Unconference session at DevCon in a group talking about OSGi. I don't remember how it came up, but we got on a discussion about deployment issues and someone asked about creating an LPKG file (the format Liferay uses to distribute a single artifact containing many bundles). I explained that it might be possible to create a file, but the problem was that the format (outside of being a zip file) is not documented and subject to change at any moment.

That's when Ray Auge jumped in and stated that we didn't want to use LPKG files anyway. Instead we should be using Subsystems, an OSGi specification for packaging many bundles in a single artifact.

Well I had not heard of Subsystems before, so I jotted down a note to myself to do some research on them to see just what they were and how I could use them...

Introduction

OSGi Subsystems is part of the R5 specification for OSGi.

"So What?" you might ask. Well, it turns out they are really useful.

For example, Liferay actually distributes all of their bundles as .lpkg files because it can be hard to distribute and deploy over 500 bundles, but it's actually pretty easy to distribute and deploy 7 .lpkg files.

The problem for us, though, is that the .lpkg file is undocumented and generally not for our use as developers and deployers.

Fine, but when it comes time for you to deploy your own app that consists of, say, 30 bundles, that deployment process can easily become a point of failure. It is all too easy to deploy 29 of the 30 files (without even knowing that one has been missed) or using the wrong version of one of the bundles...

As soon as the number of deployment artifacts grows that large, the risk of deployment failure or issue rises along with the artifact count.

What we need is an .lpkg-like mechanism to package all of our own custom artifacts into one or a small number of deployment artifacts.  This way we can keep the level of modularity we want with the bundles, but we can package them and deploy them together in a managable number of artifacts.

Enter the OSGi Subsystem Specification...

OSGi Subsystems

In case you haven't guessed it, subsystems represent a package or container of other bundles, fragments or even other subsystems.

Subsystems break down into three different types:

  1. Feature - A Feature subsystem is the simplest type and represents basically a container of bundles. All of the bundles in the feature subsystem are accessible from outside of the feature, and all of the bundles inside the feature can access outside bundles.
  2. Application - An Application subsystem is a container of bundles, but these bundles can only access bundles outside of the application; no outside bundles can use or leverage bundles or services inside of the application.
  3. Composite - A Composite subsystem is the most complex type, it is a container of bundles with fine-grained access control for bundles inside and outside of the subsystem.

While each of these types represent a container of bundles, the differences lie in accessibility inside of the subsystem or outside of the subsystem.

Although Subsystems has a lot of available features and functionality, I'm going to limit scope here to a discussion of feature subsystems. I see great value in being able to create a single artifact out of multiple bundles for deployment, but I tend to question the value in limiting or controlling access to the bundles. For most projects, that kind of (micro)management seems to be overkill; I'm guessing that these other types start to show their benefits as the project size increases.

That said, this blog post will help you get started with the simplest of subsystems, the Feature subsystem, but you'll have all of the necessary stuff installed as a foundation for your self-education on the other subsystem types.

You can find out about the full Subsystems specs and usage here:

Benefits of Feature Subsystems

Why do I see value in Feature Subsystems? Because they bring the following benefits to the table:

  • Simplifies deployment by reducing bundle count. Instead of pushing out 30 bundles for deployment, maybe I can reduce that to 5 or even 1.
  • Defines a named and semanticaly-versioned grouping of the contained bundles. Want to release version 1.1 of your portlet? Fine, it may consist of 1.3 of an api, 1.22 of an impl bundle and 1.8 of the portlet module, but they can all version together inside the subsystem.

These are the benefits that are clear to me. There may be others related to management (and scope control if you build an Application or Composite subsystem), plus others that you might see that I'm missing.

Installing Subsystems into Liferay

Okay, so there are a number of bundles you'll need to download and drop into your Liferay deploy folder:

Group ID Artifact ID Version Link
org.apache.aries.subsystem org.apache.aries.subsystem.api 2.0.6 https://repo1.maven.org/maven2/org/apache/aries/subsystem/org.apache.aries.subsystem.api
/2.0.6/org.apache.aries.subsystem.api-2.0.6.jar
org.apache.aries.subsystem org.apache.aries.subsystem.core 2.0.6 https://repo1.maven.org/maven2/org/apache/aries/subsystem/org.apache.aries.subsystem.core
/2.0.6/org.apache.aries.subsystem.core-2.0.6.jar
org.apache.aries org.apache.aries.util 1.1.1 http://repo1.maven.org/maven2/org/apache/aries/org.apache.aries.util/1.1.1/org.apache.aries.util-1.1.1.jar
org.apache.felix org.apache.felix.coordinator 1.0.0 http://repo1.maven.org/maven2/org/apache/felix/org.apache.felix.coordinator
/1.0.0/org.apache.felix.coordinator-1.0.0.jar
org.eclipse.equinox org.eclipse.equinox.region 1.2.101.v20150831-1342 http://repo1.maven.org/maven2/org/eclipse/equinox
/org.eclipse.equinox.region/1.2.101.v20150831-1342/org.eclipse.equinox.region-1.2.101.v20150831-1342.jar
org.slf4j slf4j-api 1.7.12 http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
org.apache.aries.subsystem org.apache.aries.subsystem.gogo-command 1.0.0 https://repo1.maven.org/maven2/org/apache/aries/subsystem
/org.apache.aries.subsystem.gogo-command/1.0.0/org.apache.aries.subsystem.gogo-command-1.0.0.jar

Note: You may want to check to see if there are newer versions you might want to use.

When using SLF4J, you also need to provide a binding to an actual logging implementation. I want the messages to go to the Liferay logging system, so I created an SLF4J binding for the Liferay logging system. You can get the project here: https://github.com/dnebing/slf4j-liferay.

After they have deployed, you can drop into the Gogo shell to check their status:

g! lb | grep slf4j        
  534|Active     |   10|slf4j-liferay (1.7.12)
  535|Active     |   10|slf4j-api (1.7.12)
true
g! lb | grep Apache       
   25|Active     |    6|Apache Commons FileUpload (1.3.2)
   27|Active     |    6|Apache Felix Bundle Repository (2.0.2.LIFERAY-PATCHED-1)
   28|Active     |    6|Apache Felix Configuration Admin Service (1.8.8)
   29|Active     |    6|Apache Felix Dependency Manager (3.2.0)
   30|Active     |    6|Apache Felix Dependency Manager Shell (3.2.0)
   31|Active     |    6|Apache Felix EventAdmin (1.4.6)
   32|Active     |    6|Apache Felix File Install (3.5.4.LIFERAY-PATCHED-1)
   33|Active     |    6|Apache Felix Gogo Command (0.12.0)
   34|Active     |    6|Apache Felix Gogo Runtime (0.10.0)
   35|Active     |    6|Apache Felix Gogo Shell (0.10.0)
   36|Active     |    6|Apache Felix Declarative Services (2.0.6)
  537|Active     |   10|Apache Aries Util (1.1.1)
  538|Active     |   10|Apache Felix Coordinator Service (1.0.0)
  539|Active     |   10|Apache Aries Subsystem Core (2.0.6)
  540|Active     |   10|Apache Aries Subsystem API (2.0.6)
  542|Active     |   10|Apache Aries Subsystem Gogo command (1.0.0)
true
g! lb | grep region       
  541|Active     |    1|org.osgi.service.subsystem.region.context.0 (1.0.0)
true

I've highlighted the new bundles that we loaded.

Part of the deployed bundles includes a new Gogo command for subsystems:

g! subsystem:list
0	ACTIVE	org.osgi.service.subsystem.root 1.0.0

At this point you've installed subsystems, so lets go on to building and deploying one.

Building an Enterprise Subsystem Archive

Hey, before you start on this step, do yourself a favor and make sure your bundles deploy outside of Subsystems. My first pass at this was based off of using some modules I built out of the  Blade Samples. I used those to create an esa archive and tried to deploy it and got some errors. Thinking they were Subsystem errors, I spent some time trying to resolve them. Eventually I decided just to make sure they would work and deploy them directly and all of the errors I was seeing were still there. The modules I had built were not complete and they wouldn't activate whether as normal bundles or within a Subsystem. I started over with clean projects, got them working by deploying directly and only when that was all working did I proceed to work this step.

Moral of the story - Only build an ESA archive out of working, deployable modules.

So, in case you haven't guessed it yet, we need to build a zip file (it is an archive after all) with the extension .esa, this is our Enterprise Subsystem Archive.

Building an esa archive is actually kind of easy.

If you're using Maven, you're going to be leveraging the ESA Maven Plugin. If you're using Ant or Gradle, you're going to be leveraging the ESA Ant Task. For those of you who are new to Gradle, you might want to check out how to invoke Ant from within your Gradle build script.

I created a rather simple set of modules available via this Github repo for a maven workspace: https://github.com/dnebing/subsystem-sample. There's two ServiceBuilder modules, one an API and one a service jar, and there's also a portlet module to do CRUD operations. The entities represent and extremely simple "event" system for booking events for conference rooms.

Each of the three modules were deployed directly into a local dev environment to ensure they could start and be active in the portal.

Our go at this is going to be based on pulling in dependencies to be part of the esa archive. I created a new submodule in the project, subsystem-events-esa, to pull in the dependencies. I like to use mvn archetype:generate to start a new project, but you can create one however you'd like.

Then we need to update the pom. This is the one I came up with:

<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>com.dnebinger</groupId>
    <artifactId>subsystems-modules</artifactId>
    <version>1.0.0</version>
  </parent>

  <groupId>com.dnebinger</groupId>
  <artifactId>subsystem-events-esa</artifactId>
  <version>1.0.0</version>
  <packaging>esa</packaging>

  <name>subsystem-events-esa</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>com.dnebinger</groupId>
      <artifactId>subsystem-events-api</artifactId>
      <version>1.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.dnebinger</groupId>
      <artifactId>subsystem-events-service</artifactId>
      <version>1.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.dnebinger</groupId>
      <artifactId>subsystem-events-web</artifactId>
      <version>1.0.0</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.aries</groupId>
        <artifactId>esa-maven-plugin</artifactId>
        <version>1.0.2</version>
        <extensions>true</extensions>
        <configuration>
          <generateManifest>true</generateManifest>
          <startOrder>dependencies</startOrder>
          <instructions>
            <Subsystem-Type>osgi.subsystem.feature</Subsystem-Type>
          </instructions>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Now when you build this guy, you will get a foo-1.0.0.esa file in the target folder. Peel him open with your zip tool and you should see something like:

Subsystems ESA Archive Contents

So what did we end up with?

Well, we have our three dependencies that were listed in the pom. There's a couple of Maven pom artifacts that we don't really worry about. The other thing we have is the SUBSYSTEM.MF file:

Subsystem-ManifestVersion: 1
Subsystem-SymbolicName: com.dnebinger.subsystem-events-esa
Subsystem-Version: 1.0.0
Subsystem-Name: subsystem-events-esa
Subsystem-Content: com.dnebinger.subsystem.events.api;version="[1.0.0,1.0.0]";start-order:="1",
 com.dnebinger.subsystem.events.service;version="[1.0.0,1.0.0]";start-order:="2",
 com.dnebinger.subsystem.events.admin;version="[1.0.0,1.0.0]";start-order:="3"
Subsystem-Type: osgi.subsystem.feature

If you check the pom, I asked for the file to be generated. You can just as easily specify your own if you wanted to.

This file defines the subsystem, the modules it contains and has the semantic versioning details for the esa.

Deploying the esa Archive

So we built it, let's deploy it. Note that I'm using a clean bundle here, not one where the foo Blade samples have already been deployed to.

Drop the foo-1.0.0.esa file in the Liferay deploy folder and...

Nothing. Which is kind of what I expected. You see, Liferay has an extension point for the Felix FileInstall module to handle .lpkg files and, well, we need a similar extension to support the esa Archives.

Fortunately for everyone, I've already created it. The whole project is available at https://github.com/dnebing/subsystem-deployer. Build the module and drop it into your Liferay deploy folder. Any .esa file dropped in the Liferay deploy folder will be picked up by the module and moved to the osgi/modules directory where it will be processed by OSGi and the Felix FileInstall service.

With this module in place, the .esa file will be picked up and processed. With our new subsystem Gogo command, we can now see the following:

g! subsystem:list
0	ACTIVE	org.osgi.service.subsystem.root 1.0.0
3	ACTIVE	com.dnebinger.subsystem-events-esa 1.0.0

So the subsystem looks good, but how about our bundles?

g! lb | grep subsystem-events
  548|Active     |    1|subsystem-events-service (1.0.0)
  549|Active     |    1|subsystem-events-api (1.0.0)
  550|Active     |    1|subsystem-events-web (1.0.0)
true

That's exactly what we hope to see! Since the bundles are all valid, you can now log into the portal and place the portlet on a page and try it out.

Conclusion

So that's really it. We can now leverage OSGi Subsystems in our portal. This allows us to build an esa archive file containing multiple files (bundles, fragments or other subsystem esa files) and deploy them as a single unit.

Here I've presented how to use the Feature subsystem type that has no limits on using bundles in the subsystem or bundles outside the subsystem it can use.

The two other subsystem types, the Application and Composite types, I didn't do anything with. If you come up with a valid use case or some example code, please share as I'm sure we'd like to hear about your successes.

博客
Hi David,
thank you for sharing this with us! Seems very promising.
Thanks a lot David, very useful! would be great if we have a blade sample project for esa both for maven and gradle.
Well, the big problem with doing a "blade sample" is that it has a dependency on deploying all of the Subsystems artifacts first. The rest of the blade samples can be built and deployed w/o any other environmental dependencies.
Hi David,
thanks for sharing - but after installing the subsystem bundles I run into problems restarting Liferay.

When I try to start the server (currently DXP 7.0.10 GA1 on Tomcat) the last messages I see in the logfile are
...
...
11:02:41,891 INFO [Thread-8][BundleStartStopLogger:38] STOPPED com.liferay.osgi.util_3.1.0 [557]
11:02:41,891 INFO [Thread-8][BundleStartStopLogger:38] STOPPED com.liferay.osgi.felix.file.install.configuration.cleaner_1.0.1 [556]
11:02:41,891 INFO [Thread-8][BundleStartStopLogger:38] STOPPED com.liferay.util.taglib_2.9.0 [2]
11:02:41,893 INFO [Thread-8][BundleStartStopLogger:38] STOPPED org.osgi.service.subsystem.region.context.0_1.0.0 [547]

To get the Server to start I have to delete the <LIFERAY_HOME>/osgi/state folder.

Is this only on my side or do others also experience this problem. Any hints what causes the problem ?
cheers
Joachim