How to sync pom.xml and bnd.bnd

[Thomas Kellerer, modifié il y a 7 années.]

It seems with the introduction of OSGi I have to maintain all dependencies of my portlet in two files: pom.xml and bnd.bnd.

If I don't add the necessary jar files to the -includeresource directive in bnd.bnd my portlet won't run. So each time I add a Maven dependency I need to figure out which jar files this dependency includes (recursively) and then add them manually to the bnd.bnd file.

I can't imagine this is the correct way of doing this. Is there any way I can generate the needed includes for the bnd.bnd file from the pom? Otherwise this seems like a maintenance nightmare to me (I am also not sure if I also have to maintain the build.gradle file that was generated by the Maven archtetype)

[David H Nebinger, modifié il y a 7 années.]

It is the correct way.

Compile time dependencies are not the same as runtime dependencies. Manually adding (or excluding) in the bnd file is how the runtime dependencies are satisfied.

[Thomas Kellerer, modifié il y a 7 années.]

Thanks for the answer, although it is extremely disappointing that I have to maintain the bnd.bnd manually each time I change my pom.xml.

That's horribly, and a huge step backwards.
What exactly is the advantage in that?

[David H Nebinger, modifié il y a 7 années.]

Dependency management is the most important thing you're going to do to manage your OSGi environment. You have to manage versions, dependencies and transitive dependencies to ensure a bad reference doesn't get in and introduce instability.

Many jars are not properly constructed to limit transitive dependencies. In maven, for example, you can quite easily declare a dependency on Spring, for example, but unless you list transitive dependencies as "optional", your jar will have dependencies on jUnit, log4j, etc. All things that you may be fine in pulling into OSGi, but generally there's a lot of cruft in there that shouldn't get in.

For example, I was pulling in a dependency on Jackson to handle Java<->XML transforms and it was pulling in Bouncy Castle and a whole bunch of other things that I didn't need nor want.

So yes you are forced to manage dependencies correctly in your BND file. This is not meant to be a step backwards, it is meant to be a management tool for you to control what goes into your OSGi container.

[Thomas Kellerer, modifié il y a 6 années.]

this is not meant to be a step backwards,

That is a (big) step backwards. Having to manually manage dependencies in two different files is everything but a step forward.

However, I finally managed to get rid of the error prone and cumbersome managing of the additional bnd file, by using the Felix Maven Bundle plugin

The entry in my pom.xml then looks something like this:

<plugin>
  <groupid>org.apache.felix</groupid>
  <artifactid>maven-bundle-plugin</artifactid>
  <version>2.5.4</version>
  <extensions>true</extensions>
  <configuration>
    <nicemanifest>true</nicemanifest>
    <archive>
      <addmavendescriptor>false</addmavendescriptor>
    </archive>
    <instructions>
      &lt;_dsannotations&gt;*<!--_dsannotations-->
      &lt;_metatypeannotations&gt;*<!--_metatypeannotations-->
      <embed-dependency>*;scope=compile;inline=false</embed-dependency>
      <embed-transitive>true</embed-transitive>
      <include-resource>{maven-resources},{maven-dependencies}</include-resource>
      <bundle-classpath>.,{maven-dependencies}</bundle-classpath>
      <import-package>*</import-package>
      <export-package>com.kaa</export-package>
      <require-capability>osgi.extender;filter:="(&amp;(osgi.extender=jsp.taglib)(uri=http://java.sun.com/portlet_2_0))",osgi.extender;filter:="(&amp;(osgi.extender=jsp.taglib)(uri=http://liferay.com/tld/aui))",osgi.extender;filter:="(&amp;(osgi.extender=jsp.taglib)(uri=http://liferay.com/tld/portlet))",osgi.extender;filter:="(&amp;(osgi.extender=jsp.taglib)(uri=http://liferay.com/tld/theme))",osgi.extender;filter:="(&amp;(osgi.extender=jsp.taglib)(uri=http://liferay.com/tld/ui))",osgi.ee;filter:="(&amp;(osgi.ee=JavaSE)(version=1.8))"</require-capability>
    </instructions>
  </configuration>
  <executions>
    <execution>
      <id>bundle-manifest</id>
      <phase>process-classes</phase>
      <goals>
        <goal>manifest</goal>
      </goals>
    </execution>
    <execution>
      <id>default-build</id>
      <phase>package</phase>
      <goals>
        <goal>bundle</goal>
      </goals>
    </execution>
  </executions>
</plugin>

It let's you specify everything in the plugin definition and can automatically include the needed dependencies from Maven without the need to maintain the list of jar files in a second file.

The biz.aQute.bnd plugins (and the bnd.bnd file) are no longer needed.

[Andrew Jardine, modifié il y a 6 années.]

Some people might say that using an XML format to declare your build is a step backwards as well (when compared to tools like Gradle). With that said, that is a pretty cool plugin that you found for maven and I am now wondering if the same exists for Gradle.

Question though -- the dependencies are still in two files, but this plugin allows you to manage them all in one place, is that it? But you still have to be sure to make the changes in two places within the same file right?

[Christoph Rabel, modifié il y a 6 années.]

I don't understand the usecase for this plugin.
Maybe I don't understand it correctly, but this approach includes all war file dependencies in my jar file, right? But why would I want that in an OSGI environment?

That's just bloat.

Ok, maybe it is useful in some special cases, when you need some really special library that has no OSGI version. But that seems to be quite the minority. The dependencies should be managed in the OSGI container and not added to my jar file.

[Andrew Jardine, modifié il y a 6 années.]

Agreed. Perhaps I misunderstood the post but I thought what Thomas had found was a plugin that allowed him to manage the content that would be injected in the bnd file when the build was run. So rather than him having to flip back and forth between the two files he could manage it all in one place -- that was the original complaint if I understood correctly.

Whether you make a garbage bnd file by hand or manually, well that is another story

[David H Nebinger, modifié il y a 6 années.]

I get it...

I mean, it is a total pain dealing with transitive dependencies. Just copying your listed dependencies from build.gradle or pom.xml over to the bnd file is not enough; since the bnd doesn't understand transitive dependency, you have to do more legwork to get that bnd in order.

Being able to leverage the automatic transitive dependency resolution in maven/gradle removes that legwork.

My only fear is that transitive dependency resolution can really start including all kinds of unnecessary stuff in. For example, say you are trying to use the LazyConnectionDataSourceProxy. Sure it may have some actual dependencies, but trying to pull this in brings along with it all of spring core, all of its transitive dependencies, etc.

So you end up with this bohemoth module jar for the sake of leveraging a few classes.

Yes doing things manually would be a pain, but it would give you the chance to mark all of the unnecessary stuff for exclusion on the package import.

So while the maven plugin solution will make some things easier, it will not always be the best way.







Come meet me at the 2017 LSNA!

[Thomas Kellerer, modifié il y a 6 années.]

Christoph Rabel:

Ok, maybe it is useful in some special cases, when you need some really special library that has no OSGI version. But that seems to be quite the minority. The dependencies should be managed in the OSGI container and not added to my jar file.

So I create a portlet that needs the Apache commons-lang library. The only way I found I can use it, is to include it with the bundle. Then I need another library, so I need to include that as well. But that one needs other libraries, so I need to find out which ones that are and then add them again to the bnd.bnd file, then I need another library ....

With previous Liferay versions I simply created a standard JavaEE WebApplication package (WAR file) with the necessary portlet XML descriptors and Maven would take care of everything. Then I simply dumped the WAR file into Liferay and everything was fine. This plugin at least reliefs me of managing and figuring out the dependencies for the bundle manually.

The only documentation (that I could find) on how to include third party libraries with a portlet describes exactly this process, so you shouldn't be surprised if people use that solution.

And the Wiki documentation on how to migrate from Liferay 6 to Liferay 7 doesn't explain on how to integrate "real" OSGi modules in a Maven project. I could find some hints on the internet that at least some of the Apache Commons libraries would be available as an OSGi bundle, but I could not find any example on how to include that with a Maven based portlet project.

Maybe moving to OSGi made the life for developing Liferay easier, it certainly has made my life as a portlet developer a lot more complicated (e.g. I can't simply save a modified JSP file to the webapps directory - I always need a full deployment of the bundle)

[David H Nebinger, modifié il y a 6 années.]

Thomas Kellerer:

Maybe moving to OSGi made the life for developing Liferay easier, it certainly has made my life as a portlet developer a lot more complicated (e.g. I can't simply save a modified JSP file to the webapps directory - I always need a full deployment of the bundle)

Well, to be perfectly honest while you might have been able to do that, it was never a supported or prescribed development recommendation by Liferay, that was just a side effect of what the application containers were doing.






Come meet me at the 2017 LSNA!

[Christoph Rabel, modifié il y a 6 années.]

And for me, that never worked. Well, at least not reliably. I tried it a couple of times to "just" edit a jsp, but it worked only sometimes. On the plus side: I never got used to do evil things like that and to always do a deployment.

[Christoph Rabel, modifié il y a 6 années.]

Thomas Kellerer:

So I create a portlet that needs the Apache commons-lang library. The only way I found I can use it, is to include it with the bundle. Then I need another library, so I need to include that as well. But that one needs other libraries, so I need to find out which ones that are and then add them again to the bnd.bnd file, then I need another library ....

Sure, yes. You need to do this once.

Thomas Kellerer:

With previous Liferay versions I simply created a standard JavaEE WebApplication package (WAR file) with the necessary portlet XML descriptors and Maven would take care of everything. Then I simply dumped the WAR file into Liferay and everything was fine. This plugin at least reliefs me of managing and figuring out the dependencies for the bundle manually.

In J2EE each war file has it's on class loader and bring its dependencies. In OSGI you can share those dependencies. It's a different way of doing this. What you are doing there is like saying: "Oh, we have OSGI now. Fantastic! Let's not use it and to everything the J2EE way!"

Sure, you can do that, but you shouldn't.

Thomas Kellerer:

The only documentation (that I could find) on how to include third party libraries with a portlet describes exactly this process, so you shouldn't be surprised if people use that solution.

I fully agree with you, the documentation could be better here. Maybe it is crystal clear for people who work for years with OSGI, but it took me quite a while to figure it out too. We fought with the dependencies too and it was even worse, last year, same time, there was far less documentation available.

But I really like it now. We have several tiny jar files here, our applications. We don't need to deploy 20MB jar files anymore, we just deploy our code. Of course, when we deploy them the first time, we need to resolve the dependencies. But later on, we don't need to do that anymore. There are just a few dependencies in the end, the same dependencies in most of our projects.

Of course, the J2EE approach has it's advantages too. You have different classloaders and you are separated from the others. But in Liferay, when you need to live in the Liferay system, it's annoying. You want to be in the same classloader.

All our applications ended up as huge monoliths since it was difficult to share code. Of course, there were workarounds and tricks like fake entities in service builder, but these were just workarounds to get some relief in a bad situation.

Thomas Kellerer:

And the Wiki documentation on how to migrate from Liferay 6 to Liferay 7 doesn't explain on how to integrate "real" OSGi modules in a Maven project. I could find some hints on the internet that at least some of the Apache Commons libraries would be available as an OSGi bundle, but I could not find any example on how to include that with a Maven based portlet project.

Note: We using "only" gradle here. One of the java guys has configured bamboo to do all that maven/gradle stuff and our packages are somehow added to the nexus. But we use gradle.

In gradle I just need to add the compile time dependency to the build.gradle file. The dependencies in the bnd file are autogenerated. I just need to deploy the necessary osgi bundle of the dependency in Liferay. If you need to add them manually, that would indeed be horrible.

Thomas Kellerer:

Maybe moving to OSGi made the life for developing Liferay easier, it certainly has made my life as a portlet developer a lot more complicated (e.g. I can't simply save a modified JSP file to the webapps directory - I always need a full deployment of the bundle)

For me it's the other way round. Developing in Liferay has become far easier for me. That jsp copying stuff never worked reliable for me. Sometimes the new jsp was loaded, sometimes it wasn't. So, we never did that.

[David H Nebinger, modifié il y a 6 années.]

Christoph Rabel:

I fully agree with you, the documentation could be better here. Maybe it is crystal clear for people who work for years with OSGI, but it took me quite a while to figure it out too. We fought with the dependencies too and it was even worse, last year, same time, there was far less documentation available.

I was in the same boat initially, and I even had training from Liferay before the big release. It wasn't until I worked through some projects that the bulbs started to light and I'm finally at a point where I feel like I understand more than 80%

And also agree on the documentation gap, that was kind of why I did the blog post on OSGi dependencies, https://web.liferay.com/web/user.26526/blog/-/blogs/osgi-module-dependencies. It took me awhile to figure out all of the pieces, all of the alternatives, so I wanted to document it so others wouldn't have to spend time figuring out the same.






Come meet me at the 2017 LSNA!

[Christoph Rabel, modifié il y a 6 années.]

That blog post is a great one! Very, very helpful.

But I realized a few days ago, that a lot of people don't know that they just can deploy dependencies (or at least the osgi versions of those packages). They think it is not a real option anyway and think that Option 4 is "always best". It's a solution, sure, but the OSGI way isn't difficult either:

Oh, xy.jar is missing. Download it from maven central (or copy it from the local store) and deploy it. (Repeat till all dependencies are resolved, true, sometimes a real pain).

It's not obvious that you can do that. I had no idea in the beginning.

[David H Nebinger, modifié il y a 6 années.]

Christoph Rabel:

But I realized a few days ago, that a lot of people don't know that they just can deploy dependencies (or at least the osgi versions of those packages). They think it is not a real option anyway and think that Option 4 is "always best". It's a solution, sure, but the OSGI way isn't difficult either.

The problem you have here is the classic problem of moving all of your dependencies to lib/ext in the old days - the jars become "global" and you start running into issues if you need varying versions. Plus not all dependencies are going to be OSGi-friendly.

Dependencies are definitely one of the hardest things to grasp and even harder to get right, especially since each scenario is likely solved by a different method.





Come meet me at the 2017 LSNA!

[Jack Bakker, modifié il y a 6 années.]

classic problem of moving all of your dependencies to lib/ext in the old days

for me, in v6.* having one main servicebuilder portlet, deploying the portlet, moving the service jar to /lib/ext, restarting container was a good approach when many portlets consume common services

even with the container restart this was better than copying 'the' service jar around

but with many services and also many portlets... well then, the copy to global approach would get awkward

so here we are: clearly on v7/DXP such things need to be OSGi friendly

that said, I am not sure that I am OSGi friendly yet, but community discussions of late are shedding more light

[David H Nebinger, modifié il y a 6 années.]

Jack Bakker:

classic problem of moving all of your dependencies to lib/ext in the old days

for me, in v6.* having one main servicebuilder portlet, deploying the portlet, moving the service jar to /lib/ext

That's a different use case - you're pushing a single jar with no unsatisfied dependencies.

If you had wanted to move, for example, spring's JDBC jar, it is unusable. You'd have to also put in the rest of spring and all of their transitive dependencies. That's when you have version problems with local apps. Try running Liferay after you've put spring 4 jars in the lib/ext directory and you'll quickly be regretting this config....





Come meet me at the 2017 LSNA!

[Jack Bakker, modifié il y a 6 années.]

David H Nebinger:

you're pushing a single jar with no unsatisfied dependencies.

correct, in v6.2 currently

[Christoph Rabel, modifié il y a 6 années.]

Most are osgi friendly nowadays. And you could always wrap them, if necessary.
OSGI ticks differently than J2EE and I think we should embrace that and not work around it. In J2EE each application had its own classloader. That approach has advantages, we still create most of our applications that way. We love our "monoliths" and they are perfectly fine. But Liferay needs to be extensible and we should design our packages that way too.

A while ago, I did some cleanup of exports. I had exported interfaces and implementations and I removed the implementations from the export. But then I decided, while this might be the best approach in the "good ol' days", it's wrong in OSGI. And I reverted the change. All our packages now export interface and implementation.

I even have started to remove the private declarations and make them protected. Who am I to decide, that someone else should not mess with my code. Of course, great power -> great responsibility, but I have come to hate black boxes where I stand outside and can't do anything.

Well, that led me a bit away, but I think people should have all options on the table and decide for themselves. Maybe, when we have done more projects, with some updates, I will change my opinion, but I currently think that it is the OSGI way to put the dependencies in the osgi container.