Including additional jar in OSGI modules

[Peter Helgren, modifié il y a 6 années.]

OK. Yes. I have read and re-read Devid's post here: OSGI jar dependencies and although I am trying to follow his direction, he assumes a great deal of knowledge about Gradle and Blade and I am a relative newbie trying to make the transition from 6.0.6 development to 7.0 development.

The issue is that I have a service builder project that uses JavaMail and DKIM so I need to include the javax.mail.api and the DKIMforJavaMail.jar in my project. This is code that I am migrating from source I have over in my 6.0.6 projects. And, although I can resolve all the import issues by adding the jars to my build path and the service builder gradle task runs successfully, when I go to deploy, it cannot find the packages. It is the "classic" issue that David tried to address in his blog post from last summer.

The issue is a "classic" newbie issue when going to the new build models and structures in LR7.0. Folks will say in a post "Add X to the build.gradle file". OK! WHICH build.gradle ?? (there are three in my SB module project: one under the project, one under the "api" subfolder and one under the "service" subfolder. There is also mention in David's post about adding either "headers" or "directives" to the bnd.bnd file. Again, WHICH bnd.bnd file (there are TWO, again under "api" and "service" folders). I try to be as precise as I can when I post..assuming nothing about my reader's level of understanding so I get a little frustrated when I read through posts that outline things, leaving out assumed details...I was attempting to use David's "Option #4" which *seemed* the easiest but trying to tease out the details was difficult.

So, a couple of questions:
1) When jars are manually downloaded and used in a project, how should they be included for deployment? I understand that gradle can actually download the needed jars (I have seen it work and it is slick) but what if the jars don't exist "out there"? Should they still be referenced in the build.gradle and if so, WHICH build.gradle and with what directives?

2) Is there a clear example of the changes needed to build.gradle and bnd.bnd (again, WHICH ones) with a "before and after" view so we can see exactly what what was needed to get a module with "external" jars to deploy?

Even though I have attended a LR7 training, there are essential build concepts that can be a little fuzzy after a while. OSGI modules are just a bit different than the old war file builds and when you add gradle and other tools, it can be quite a jump. I just need a solid example of an SB module with external jars with all the details explained...is there something that someone has found clear and helpful that I can take a look at?

[David H Nebinger, modifié il y a 6 années.]

Peter Helgren:

The issue is that I have a service builder project that uses JavaMail and DKIM so I need to include the javax.mail.api and the DKIMforJavaMail.jar in my project. This is code that I am migrating from source I have over in my 6.0.6 projects. And, although I can resolve all the import issues by adding the jars to my build path and the service builder gradle task runs successfully, when I go to deploy, it cannot find the packages. It is the "classic" issue that David tried to address in his blog post from last summer.

Okay, my first suggestion is to break this up. OSGi works best when you have an API jar with interfaces and no external dependencies and an implementation jar where you have your code and the 3rd party dependencies.

I would break out the mail/DKIM code to separate it from the service builder code. The reason I'd suggest this is that dependencies (and transitive dependencies) can impact non-related code. To paraphrase Bill Maher, although I don't know that javamail & DKIM might impact the service builder code, I believe it can.

The issue is a "classic" newbie issue when going to the new build models and structures in LR7.0. Folks will say in a post "Add X to the build.gradle file". OK! WHICH build.gradle ??

So if you use the Liferay workspace, you'll have many different build.gradle files. There's one for each individual module and then you'll also find some in parent directories that act kind of like a "parent pom" sort of concept.

When you are adding a jar as a dependency to build.gradle, you're basically declaring a dependency for your code. So you want to add the dependency as close to the code as possible, so that means the build.gradle that is part of the module that has the dependency. When you bleed the dependency elsewhere or higher in the parent levels, those become transitive dependencies that all other users of your code will need to satisfy.

(there are three in my SB module project: one under the project, one under the "api" subfolder and one under the "service" subfolder. There is also mention in David's post about adding either "headers" or "directives" to the bnd.bnd file. Again, WHICH bnd.bnd file (there are TWO, again under "api" and "service" folders).

Same sort of response, the headers and directives control the building of the module, so they need to go into the bnd file for the module.

I try to be as precise as I can when I post..assuming nothing about my reader's level of understanding so I get a little frustrated when I read through posts that outline things, leaving out assumed details...I was attempting to use David's "Option #4" which *seemed* the easiest but trying to tease out the details was difficult.

Sorry about that; I'm sure you know how it gets sometimes, you're busy looking at the trees and simply can't see the forest.

1) When jars are manually downloaded and used in a project, how should they be included for deployment? I understand that gradle can actually download the needed jars (I have seen it work and it is slick) but what if the jars don't exist "out there"? Should they still be referenced in the build.gradle and if so, WHICH build.gradle and with what directives?

So this is actually frowned upon but it still possible. It's frowned upon in that gradle follows the maven process of wanting to get all dependencies from a repository rather than some local jar file if only because it is hard to use the same build script in different environments or in build servers like Jenkins.

However, I know it's possible but don't remember the syntax; suffice it to say that for gradle problems you can usually find gradle solutions, they won't necessarily be liferay-only solutions. The only time they become Liferay is when you are using one of the Liferay tools like service builder.

2) Is there a clear example of the changes needed to build.gradle and bnd.bnd (again, WHICH ones) with a "before and after" view so we can see exactly what what was needed to get a module with "external" jars to deploy?

Not that I know of. I mean, normally you know you need a dependency on javamail so the "before" picture is "my code won't compile" whereas the "after" picture is "my code compiles!"

Even though I have attended a LR7 training, there are essential build concepts that can be a little fuzzy after a while. OSGI modules are just a bit different than the old war file builds and when you add gradle and other tools, it can be quite a jump. I just need a solid example of an SB module with external jars with all the details explained...is there something that someone has found clear and helpful that I can take a look at?

Hey, feel free to post them here, I'm sure you're not the only person with these kinds of questions, and I honestly don't mind chiming in to help clarify whatever is still murky...






Come meet me at the 2017 LSNA!

[Peter Helgren, modifié il y a 6 années.]

Thanks David...you seem to have the patience of Job and all the time in the world...sorry I am abusing both.. :-(

You said: "I would break out the mail/DKIM code to separate it from the service builder code. The reason I'd suggest this is that dependencies (and transitive dependencies) can impact non-related code." OK. I have read your stuff extensively, and read the dev.liferay.com pages extensively as well. I am a relatively intelligent guy and I keep asking myself "WHY is this so HARD"....seems like I don't quite get it. I think I know what the concept is but the devil is in the details....

My first thought was: "Well, if I move the code to a different module, aren't I just going to end up with the same issue all over again, in a different module? " Right now I am down to one dependency that can't be resolved. The "-api" module is complaining about ( you can probably guess):

Unresolved requirement: Import-Package: com.sun.mail.smtp; version="[1.4.0,2.0.0)"_ [Sanitized]

The SMTP dependency is throwing me a little because the only reference I have to com.sun.mail.smtp is pointing to a build.gradle entry for version 1.4.5 so I am not sure which component has a dependency on a version 1.4.0 of com.sun.mail.smtp. Curious. I don't even SEE a 1.4.0 version listed in Maven...

So, back to your original proposal, which may be the only way to resolve this issue with smtp (not sure). Scanning through my SB code, for this particular class, there really isn't much in the way of implementation of the interface. Most of this code could live in a pojo. So, here is the $64,000 question: If I break this out into its own module, what kind of module do I create and then how to I make the SB classes available in this separate module? And if THIS module has a dependency on com.sun.mail.smtp how do I keep that from becoming an issue "downstream"?

Basically the SB project structure is like this:

MyModule
MyModule-api
MyModule-service
(would I add a UTILS modules, or whatever, to this SB project, separating out the email code?)

Right now there are a few methods with different signatures that send mail like sendEmail(String, String, String ...etc) These sendEmail methods use the LR MailEngine class, the DKIM classes from a local jar and InternetAddress classes (that you warned me about). If I pull all this code out of SB modules and into it's own module, are you saying that, at the API level, I won't have to deal with any of this as long as the parameter types referenced in the interface don't have dependencies on "external" modules? (I think that is what you posted to me elsewhere)

So, I think I'll end up with (if I take this piecemeal approach) with a "clean" API (no dependencies), an implementation that will have some dependencies on a third module that will ultimately have dependencies on some external jars? Is that the recommended approach?

I walked though creating a module without the benefit of any tooling when I took the LR 7.0 coursework so I guess I'll have to blow the dust off of that code/instructions and see if I can create this more modular "sendEmail" code without adding to the rats nest....

[David H Nebinger, modifié il y a 6 années.]

Peter Helgren:

My first thought was: "Well, if I move the code to a different module, aren't I just going to end up with the same issue all over again, in a different module? "

Well, you certainly can end up with the same issue, but ultimately your goal should be to isolate all dependencies in the implementation jar/module, and not bleed any into the API/interface module.

Ultimately the best practice is to do "design by interface". You create an interface that has all of the methods you need, that goes to your API module. Then you create the separate implementation of the interface, that goes in your implementation jar. When you follow this pattern, you can isolate the direct and transitive dependencies so they don't bleed into other consumers.

Right now I am down to one dependency that can't be resolved. The "-api" module is complaining about ( you can probably guess):

Unresolved requirement: Import-Package: com.sun.mail.smtp; version="[1.4.0,2.0.0)"

This is a demonstration of the bleed of dependencies. A true API jar will have no dependencies (or limited dependencies to existing modules such as the Liferay kernel module). This way, consumers of the service will only need the API and none of the dependencies. The implementation jar is a different story. For your situation it will have a dependency on javax.mail plus whatever else you need to add to it.

But as soon as you start seeing the dependencies show up in the API module, you should take that as a flare going up signaling trouble ahead. With the current dependency as it is on com.sun.mail.smtp in the API module, any other module that uses the API will also incur the dependency on com.sun.mail.smtp, and this is where problems start. Pretty soon you have every module you deploy dependent upon every 3rd party jar that you've added to take care of one problem or another, and dependency management spirals out of control.

However, if you get a clean API with no dependencies, you get that clean separation where consumers of the API just need the API module, but your implementation is free to use whatever it needs to implement the functionality.

The SMTP dependency is throwing me a little because the only reference I have to com.sun.mail.smtp is pointing to a build.gradle entry for version 1.4.5 so I am not sure which component has a dependency on a version 1.4.0 of com.sun.mail.smtp. Curious. I don't even SEE a 1.4.0 version listed in Maven...

So what you're seeing there is a transitive dependency. One of the direct dependencies you've included (could be javax.mail, could be another one) has an entry in it's MANIFEST.MF file declaring a dependency on com.sun.mail.smtp using a version range of at least 1.4.0 but less than 2.0.0. These are the nefarious little things that will cause you the most problems and the ones that will convince you that the clean API module really is the way to go, because most 3rd party jars won't have just dependencies on com.sun classes, but will also have dependencies upon other 3rd party jars, which themselves will declare dependencies on other jars, ... Pretty soon you find you need everything plus the kitchen sink just to get a module to start.

That can be fine and necessary for your implementation jars, but for your API jar you're now passing that dependency chain to other modules that would consume your service.

So, back to your original proposal, which may be the only way to resolve this issue with smtp (not sure). Scanning through my SB code, for this particular class, there really isn't much in the way of implementation of the interface. Most of this code could live in a pojo. So, here is the $64,000 question: If I break this out into its own module, what kind of module do I create and then how to I make the SB classes available in this separate module? And if THIS module has a dependency on com.sun.mail.smtp how do I keep that from becoming an issue "downstream"?

Yep, that is the 64k question. A purist would tell you to "design by interface", create some interfaces/pojos w/ no external dependencies at all in the API module and an implementation module that uses javax.mail plus whatever else is necessary. The SB service module would then just need a project dependency on the API and all should be good.

A practical path is to keep it inside of the SB service module, just try eliminating dependencies from the API jar. Could be that you have extra dependencies listed in the SB api module that don't really need to be there anymore. And if you find that you do need the dependency in the API jar, that just signifies that one of your SB service methods must have a dependency that it's exposing (like how I mentioned in a previous post not to pass InternetAddress as a parameter because that forces the API and all consumers to now have a dependency on javax.mail...).

Right now there are a few methods with different signatures that send mail like sendEmail(String, String, String ...etc) These sendEmail methods use the LR MailEngine class, the DKIM classes from a local jar and InternetAddress classes (that you warned me about). If I pull all this code out of SB modules and into it's own module, are you saying that, at the API level, I won't have to deal with any of this as long as the parameter types referenced in the interface don't have dependencies on "external" modules? (I think that is what you posted to me elsewhere)

Yeah, so you need to avoid using the classes like InternetAddress directly as that's where it becomes a dependency that all must have. So instead of using an InternetAddress as a dependency, define your own pojo to use or just pass two strings as parameters instead. Inside of the implementation of the method, you can turn that into the InternetAddress to use, you just don't want it to be an official parameter and therefore a dependency.

So, I think I'll end up with (if I take this piecemeal approach) with a "clean" API (no dependencies), an implementation that will have some dependencies on a third module that will ultimately have dependencies on some external jars? Is that the recommended approach?

Yes. I mean, this java mail guy could actually stay in SB service as long as you fix parameters, but a clean approach is the one you just outlined. Besides, it would open other components you create to send email w/o going through your service layer, and that can sometimes be useful.

I walked though creating a module without the benefit of any tooling when I took the LR 7.0 coursework so I guess I'll have to blow the dust off of that code/instructions and see if I can create this more modular "sendEmail" code without adding to the rats nest....

It's not too hard. I did a 6-part blog that created a separate API and implementation module for filesystem access, just to show how to take advantage of the DS services and support service injection using the @Reference annotation. Having gone through the training, you're already at a good spot to pick all of this up, but the better news is that by working through it you should reach a level of clarity in dealing with all of these dependency issues and why it is important to manage them lest they manage you...







Come meet me at the 2017 LSNA!

[Peter Helgren, modifié il y a 6 années.]

Once again, awesome! I might actually be at a point where this is all starting to make some sense.

One last question and then I'll deep dive and check your blog post:

"How do you 'discover' what dependencies you might have introduced into your SB code that could bleed out to the API?" Yes, vigilance when starting from scratch would be the best way, but in my case I have a boatload of 6.0.6 code that I am migrating, step by step. I guess a search by package name in the API source modules would be the best? It would be nice to have a "dependency explorer" tool that could list the packages required in the API source (Maybe the IS an Eclipse tool that does this?) Anyway, first I need to figure out what is lurking in the API source that I need to remove and then I need to figure out how to refactor the code to handle what has been removed.

Again David, thanks for your patience. It has been extremely helpful information.

[David H Nebinger, modifié il y a 6 années.]

Peter Helgren:

"How do you 'discover' what dependencies you might have introduced into your SB code that could bleed out to the API?"

That is easy to determine. When you build a new SB module, the API dependencies are pretty sparse:

dependencies {
	compileOnly group: "biz.aQute.bnd", name: "biz.aQute.bndlib", version: "3.1.0"
	compileOnly group: "com.liferay", name: "com.liferay.osgi.util", version: "3.0.0"
	compileOnly group: "com.liferay.portal", name: "com.liferay.portal.kernel", version: "2.0.0"
	compileOnly group: "org.osgi", name: "org.osgi.core", version: "6.0.0"
}

When you add your service methods and rebuild services, you're not going to need really much beyond these.

When you build and deploy and try to start your module, though, you'll know if you have missing dependencies because the module won't start. Starting via gogo will list those "unresolved references", and that is going to be your indication that you have some dependency bleed.

Yes, vigilance when starting from scratch would be the best way, but in my case I have a boatload of 6.0.6 code that I am migrating, step by step.

That's going to be a challenge for you going forward. Things that might have been easy to do in 6.0.6 will cause you problems under LR7. Refactoring may be painful in the budget/project plan, but it will save you some future headaches if you can get things right out of the gate.

It would be nice to have a "dependency explorer" tool that could list the packages required in the API source (Maybe the IS an Eclipse tool that does this?)

You might submit this as a feature request for the Liferay IDE on issues.liferay.com. I don't know that they've considered this kind of thing.










Come meet me at the 2017 LSNA!