Liferay OSGI and Shell Access Via Gogo Shell

Liferay has been slowly integrating an OSGi framework into it's core; not in the sense that Liferay will nessecarily run "within" an OSGi runtime itself (there are arguments for doing that, and it's possible to a certain extent but the gain just isn't in doing that just yet), more that Liferay will "host" and OSGi runtime within itself that Liferay will utterly control.

Why do this? Well, as I've stated many times before, Liferay is a complex beast with an intricate network of features, so many features in fact that the ocassionaly have such indistinct lines such as finding the where one feature ends and another begins can be difficult.

OSGi will hopefully play a part in 1) providing us with a lot of re-use of features we had ourselves partially implemented but for which there existed a fully specified implementation our in the larger java eco system. The savings those existing features can bring is one of many benefits. 2) A specified lifecycle for "modules" (a.k.a. bundles) also is a huge benefit, one we can simply use. 3) Dependency resolution and management is another piece we had started to implement but it was clear to me that this had already been done much more effectively than Liferay alone could implement, again as part of an OSGi specification.

The number of benefits is almost too great to list. However, one of the greatest advantages can't be discussed enough: Modularity

Modularity is something that Liferay is now learning to embrace. Hopefully in the future we'll see many more examples like the one I'm about to illustrate.

One of the things I spend a lot of time doing is running and debugging code. There are a number of challenges to doing this with such a large application as Liferay. It takes time to get up and running, the vast number of features sometimes makes it tricky to isolate just the right service calls in order to test some specific interaction. I'm most often running the portal in debug mode executed from my IDE (Eclipse). A favourite thing to do is, place a break point and then use the context evaluation to invoke some specific code (using the Display view). However, my main purpose for doing this is to be able to fire some operation often unrelated with the actual breakpoint position. Effectively I just want to execute some arbitrary code in the JVM.

What would be ideal is if I had a command line from which I could do the same thing.

Enter Gogo

The OSGi echo system provides that out there somewhere a bunch of smart people have specified some behaviors, how to build properly interacting bits of code, and provided I have the appropriate runtime for those bits, I can interconnect them with my own bits and compose a greater feature.

Gogo is an OSGi (hence java) command shell implemented by the Apache Felix community.

"Apache Felix Gogo is a subproject of Apache Felix implementing the OSGi RFC 147, which describes a standard shell for OSGi-based environments."

Installing this shell with Liferay (due to Liferay's new interated OSGi runtime) is almost too simple.

Step 1)

First we'll download the necessary bundles:

 
Place these jars in a common, convenient location (it need not be within the Liferay install).

(Note: We could have even avoided this pre-download step, but due to a small logical bug in Liferay's integration, we'll have to download the bundles ahead of time. I'll be fixing this bug in the days to come.)

Step 2)

Enable Liferay's Module Framework

Place the following in your portal-ext.properties (or equivalent props file):

module.framework.enabled=true

Step 3)

Configure some initial bundles to start when the OSGi runtime starts in Liferay (still in  portal-ext.properties):

extra.bundles=file:/downloaded/bundles

module.framework.initial.bundles=\
    felix-fileinstall.jar@start,\
    ${extra.bundles}/org.apache.felix.gogo.command-0.12.0.jar@start,\
    ${extra.bundles}/org.apache.felix.gogo.runtime-0.10.0.jar@start,\
    ${extra.bundles}/org.apache.felix.gogo.shell-0.10.0.jar@start,\
    ${extra.bundles}/org.eclipse.equinox.console_1.0.0.v20120522-1841.jar@start
 

Step 4)

Set the telnet port on which we'll access the shell  (still in    portal-ext.properties) :
 
module.framework.properties.osgi.console=11311
 

Step 5)

Start Liferay
 

Step 6)

Connect to the telnet service from the command line (or from your favourite telnet client):
 
]$ telnet localhost 11311

You should see something like the following:

We have sheel access!

But what can we do with this?

The first thing I wanted to do was investigate how the shell works and how to customize the configuration.

Passing System Properties to OSGi bundles

One thing I should add is that OSGi bundles often have system properties specific to them, and in order to isolate these from potentially other osgi runtimes in the same JVM, Liferay provided a means of passing these as namespaced portal properties.

An example of this is passing a Gogo specific system property called gosh.home. And so I added that property to my portal-ext.properties (prefixed by module.framework.properties.):

e.g.

module.framework.properties.gosh.home=${module.framework.base.dir}

This places the Gogo shell's home directory within Liferay's osgi directory inside ${liferay.home}.

Gogo is then trained to look inside an /etc directory within it's gosh.home for configuration files.

In this folder I added two files.

gosh_profile (think bash_profile):

https://gist.github.com/rotty3000/4712530#file-gosh_profile

motd (a.k.a. Message of the day):

https://gist.github.com/rotty3000/4712530#file-motd

Now a restart and my shell appears like:

Awesome!

Executing commands

The shell offers many build in commands. Executing help will reveal many of the extra commands namespaced by providers. However these are executable without the namespace (the namespace is used for conflict resolution of two namespaces provide commands of the same name).

 

A few choice commands are:

ss (short bundle status)

diag (diagnose a problem bundle)

 

However, it's possible to add commands dynamically as well as use closures. This is where the real power begins.

You may note that we added a custom set of commands based on the java.lang.System class in the gosh_profile:

addcommand system (loadclass java.lang.System)

What this does is loads the class and assigns the public methods of that class under the system prefix. Executing:

system:getproperties

outputs all the system properties. In a similar fashion any class can be added as a set of shell commands.

Conclusion

Modularity via OSGi allowed me to easily add functionality not previously available in Liferay. I can now build out some commands that will enable me to quickly take actions which previously required me to find round about ways to achieve.

 

Well, that's all for today. It was simple and fun to extend Liferay with shell access, I solved my problem. It also demonstrates the power of OSGi since we didn't have to write a single line of code in order to benefit from it's well defined modularity and lifecycle specifications.

 

I'll talk more about shell commands, closures, and how to add and use custom Liferay commands in more detail in a future blog post. I'll also start using this shell to show how to start building small Liferay extensions using OSGi.

Eventually, more of the Module Framework features (such as the web extender) that Miguel and I have been working on will reach master and we'll start to discuss those as well.

 

Blogs
Hey Ray and Miguel, this is fantastic for developers and sysadmins to introspect into their production systems and very conveniently get at Liferay's guts without having to have a full development environment. I can't wait to see the library of scripts that come from this. It'd be great to see a few examples of how to get and (and possibly modify) the Liferay runtime via Liferay APIs in future blog posts! Also, is it possible to install this without having to restart Liferay?
It's possible to install these osgi bundles in Liferay without a restart. Simply drop them into:

${liferay.home}/data/osgi/deploy

However, since some properties are required to enable them and the management UIs have not been committed it's quite hard to configure without a restart. However, at a later time, when we do have these management UIs, you should have full ability to manage and configure any new bundles you want to install in the system at runtime.
Note below properties work for Liferay 6.2 on windows

extra.bundles=file:///D:/apps/felix-framework-4.4.1/modules-for-liferay
module.framework.initial.bundles=\
felix-fileinstall.jar@start,\
${extra.bundles}/org.apache.felix.gogo.command-0.14.0.jar@start,\
${extra.bundles}/org.apache.felix.gogo.runtime-0.12.1.jar@start,\
${extra.bundles}/org.apache.felix.gogo.shell-0.10.0.jar@start,\
${extra.bundles}/org.eclipse.equinox.console_1.0.0.v20120522-1841.jar@start
module.framework.properties.osgi.console=11311

Ref: https://www.liferay.com/community/forums/-/message_boards/message/34253865
Note below properties work for Liferay 6.2 on windows

extra.bundles=file:///D:/apps/felix-framework-4.4.1/modules-for-liferay
module.framework.initial.bundles=\
felix-fileinstall.jar@start,\
${extra.bundles}/org.apache.felix.gogo.command-0.14.0.jar@start,\
${extra.bundles}/org.apache.felix.gogo.runtime-0.12.1.jar@start,\
${extra.bundles}/org.apache.felix.gogo.shell-0.10.0.jar@start,\
${extra.bundles}/org.eclipse.equinox.console_1.0.0.v20120522-1841.jar@start
module.framework.properties.osgi.console=11311

Ref: https://www.liferay.com/community/forums/-/message_boards/message/34253865
Note below properties work for Liferay 6.2 on windows

extra.bundles=file:///D:/apps/felix-framework-4.4.1/modules-for-liferay
module.framework.initial.bundles=\
felix-fileinstall.jar@start,\
${extra.bundles}/org.apache.felix.gogo.command-0.14.0.jar@start,\
${extra.bundles}/org.apache.felix.gogo.runtime-0.12.1.jar@start,\
${extra.bundles}/org.apache.felix.gogo.shell-0.10.0.jar@start,\
${extra.bundles}/org.eclipse.equinox.console_1.0.0.v20120522-1841.jar@start
module.framework.properties.osgi.console=11311

Ref: https://www.liferay.com/community/forums/-/message_boards/message/34253865
[...] I have done a huge refactor for most of our OSGi related work, moving the majority of its components to the Liferay plugins SDK. Everything except the graphical admin UI is already in the master... [...] Read More
Hi Ray, This it great article. It helps me understand the OSGI implementation in Liferay.

Recently I come across a situation where I need to hooked up whole liferay plateform as OSGI plugin in one of the well define OSGI container. I am using liferay 6.2 CE GA2. Do you have any idea how could we do this?

-Hitesh Panchani.
Liferay can't easily run inside an osgi framework. It's a very complex app and so the strategy we have currently is to work toward that possibility by first starting to internally consume and osgi container as a child of Liferay instead.
That is true, it is very complex and tedious.But that is what the requirement is for my project. I am able to hooked liferay in OSGI container but when I start the server(tomcat) I am getting below error. It will be great if you can help me here.

14:41:58,492 ERROR [stop children - Catalina:j2eeType=WebModule,name=//localhost/Liferay,J2EEApplication=none,J2EEServer=none][PortalContextLoaderListener:94] java.lang.NullPointerException
java.lang.NullPointerException
at com.liferay.portal.kernel.servlet.DirectServletRegistryUtil.clearServlets(DirectServletRegistryUtil.java:27)
at com.liferay.portal.spring.context.PortalContextLoaderListener.contextDestroyed(PortalContextLoaderListener.java:123)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4763)
at org.apache.catalina.core.StandardContext$4.run(StandardContext.java:5473)
at java.lang.Thread.run(Thread.java:662)
14:41:59,564 ERROR [stop children - Catalina:j2eeType=WebModule,name=//localhost/Liferay,J2EEApplication=none,J2EEServer=none][PortalContextLoaderListener:94] java.lang.NullPointerException
java.lang.NullPointerException
at com.liferay.portal.kernel.deploy.hot.HotDeployUtil.reset(HotDeployUtil.java:45)
at com.liferay.portal.spring.context.PortalContextLoaderListener.contextDestroyed(PortalContextLoaderListener.java:130)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4763)
at org.apache.catalina.core.StandardContext$4.run(StandardContext.java:5473)
at java.lang.Thread.run(Thread.java:662)
[INFO] MLog clients using com.mchange.v2.log.FallbackM
It looks like it's trying to reload the portal webapp.

this seems to be much after some other initial error failed to let the portal startup.

this message shows no indication of what that is! Also, we don't support running Liferay inside a pure osgi runtime. You need a supported app server.
Thanks Ray for quick answer. You are right it is because of jar conflict in OSGI container on which I am trying to run the Liferay Portal.

Interesting thing is, I have successfully run Liferay inside external OSGI container but pain point is, it is running extremely slow inside it. It is talking around 20 minutes to load even single page and ext files.

It will be great if you provide some hints here to rectify this issue. You are working on OSGI with liferay since long, you might come across this kind of situation before.
Thanks Ray!

I have read your post on OSGI integration in new Liferay 7. I am curious to know that are you planning to provide support for Eclipse Equinox OSGi container? Are you planning to inject "BuckMinster" support to build liferay as p2 repository like eclipse plugin?
[...] “Liferay is a complex beast with an intricate network of features, so many features in fact that they occasionally have such indistinct lines such as finding the where one feature ends and another... [...] Read More
Hi Ray,

I installed the blade CLI and Gradle plugin for Eclipse. I have created my first portlet through blade cli and imported it in gradle task of eclipse.
When i run the deploy tasks in eclipse I get the below output but unfortunately portlet is not shown in the sample category in portal

Working Directory:
Gradle User Home:
Gradle Distribution: Gradle wrapper from target build
Gradle Version: 3.4
Java Home: C:\Program Files\Java\jdk1.8.0_111
JVM Arguments: None
Program Arguments: None
Gradle Tasks: deploy

:compileJava UP-TO-DATE
:buildCSS NO-SOURCE
:processResources UP-TO-DATE
:transpileJS SKIPPED
:configJSModules SKIPPED
:replaceSoyTranslation NO-SOURCE
:wrapSoyAlloyTemplate SKIPPED
:classes UP-TO-DATE
:jar UP-TO-DATE
:deploy UP-TO-DATE

BUILD SUCCESSFUL

Total time: 0.484 secs

And when i run the "blade deploy" command am getting the below error:
Error
0. deploy: Unable to connect to gogo shell on localhost:11311

I followed your steps to try to resolve my gogo shell error
I am not having telnet connection as am working in my local machine set up.
P.S: Am working on Liferay6.2.

Please help in resolving this issue