Good localtime community,
First a disclaimer, I am a greenhorn when it comes to Maven and OSGi, though I have read through half a dozen threads on the subject here. We are migrating from our current RESTful architecture to Apache's Fuse (ESB) and of course this requires us to sort out our dependencies and carefully craft our POMS for each bundle/container. Generally speaking each container will contain 3 in-house dependency bundles:
The container specific bundle which requires the next two in house bundles).
A processing bundle that all of our in-house containers will require (though they might require different versions), and which requires the bundle below.
A utility bundle that all of our containers will require (though they might require different versions).
Additionally each container can require one or more (and possibly up to over 100) third party jars, which are often reused between containers (though again no guarantee if versions will be the same).
We have (or will soon have) profiles for each container and for the ESB as a whole. What I would like to know is:
What is the best (cleanest) way to reuse third party jars between containers when we can?
I've read that we should never use if we can avoid it, but we are heading down to jarmageddon when we try to explicitly import the jars/packages we need for our three bundles. I have read that *;resolution:=optional is not the way to go either (and for what it's worth, that doesn't seem to work for us anyway). Any thoughts?
I read on some forums that bundling third party jars is the way to go (though that seems a little overboard), and I have read on others that would kind of defeat the purpose of OSGi. Any thoughts there?
Our in-house bundles often require many of the same third party jars/versions. Is this simply a matter of building/installing our bundles in the correct order (utility, processing, and container specific) and exporting jars that can be reused by the next bundle(s)?
We are in a position where we can get our container working if we do some things we do not want to do (export everything, import *), but we'd like to handle this as cleanly as possible since we will have to repeat the process for many containers (with increasing dependencies) and since we will have to live with supporting/updating our implementation.
Thanks in advance for your guidance.
Related
tl;dr
We have two groups of bundles. Call them Group A and Group B. Bundles in Group A MUST import packages from ONLY other bundles in Group A, regardless of whether some bundle in Group B exports a package which could satisfy the dependency. Bundles in Group B MAY resolve their package imports from any bundle, regardless of whether it is in Group A or Group B.
Longer explanation
My team is trying to find a way to enforce a kind of "safe mode" for a product built using OSGi. Essentially we have our core product, and we allow customers to install their own components on top to extend our functionality. Obviously, this is the type of thing OSGi is made for.
However, we have noticed that if a customer installs a bundle which happens to export a package used by one of our core bundles, there is a chance that the core bundle will get wired up to something installed by a third party. While I understand that semantic versioning implies that this is not a major cause for concern, we have noticed that a significant portion of our core bundles are restarted if/when some third party bundles are refreshed.
What we want to do is ensure that bundles in our core product do not wire up to any bundle installed by a third party. We are using bundle start levels to set our core bundles to a start level before third party bundles. This lets us set the framework start level to exclude all bundles after our core in the event that we need to debug issues with third party code. However, start levels alone are not enough to prevent package level dependencies from wiring up to our core components.
The only way I can think of to do this (which I do not believe is a good solution and we have no plan to implement) is to maintain a list of all third party bundles added to the runtime after our core product is set up. Then, when the framework shuts down, uninstall all bundles in this list (backing up the actual bundle file). On startup, the core product would start and wire up correctly, then we'd automate the re-installation of all the third party bundles that have been installed. This feels to me like a very fragile, hacky, and just plain wrong way to achieve what we want, but I can't think of any other way. So I am turning to the SO community for help!
If you have developed your system around services then sometimes the best approach is to start another framework and run the customer code in the other framework. If you're more classic Java oriented then this is too much work usually.
Then there are a number of possibilities:
Resolve Hooks – With these hooks you can hide bundles and very likely control things exactly as you want.
Security – With the security manager you can limit package imports and exports
Attributes – Export packages from the core with a magic attribute and import only when that attribute is set.
Weaving – Have a weaving hook that adds a non-existent package import to every non-core bundle. Initialize core, synthesize a bundle with the non-existent package and install/resolve it.
Obviously both Resolve Hooks and Weaving require your bundle to run very early. If you use the Bndtools launcher then there is a way to do this.
Edit: After reviewing the play, the example I used below is a tad misleading. I am looking for the case where I have two 3rd party jars (not homegrown jars where I have access to the source code) that both depend on different versions of the same jar.
Original:
So I've recently familiarized myself with what OSGi is, and what ("JAR Hell") problems it addresses at its core. And, as intrigued as I am with it (and plan on migrating somewhere down the road), I just don't have it in me to begin learning what it will take to bring my projects over to it.
So, I'm now lamenting: if JAR hell happens to me, how do I solve this sans OSGi?
Obviously, the solution would almost have to involve writing my own ClassLoader, but I'm having a tough time visualizing how that would manifest itself, and more importantly, how that would solve the problem. I did some research and the consensus was that you have to write your own ClassLoader for every JAR you produce, but since I'm already having a tough time seeing that forest through the trees, that statement isn't sinking in with me.
Can someone provide a concrete example of how writing my own ClassLoader would put a band-aid on this gaping wound (I know, I know, the only real solution is OSGi)?
Say I write a new JAR called SuperJar-1.0.jar that does all sorts of amazing stuff. Say my SuperJar-1.0.jar has two other dependencies, Fizz-1.0.jar and Buzz-1.0.jar. Both Fizz and Buzz jars depend on log4j, except Fizz-1.0.jar depends on log4j-1.2.15.jar, whereas Buzz-1.0.jar depends on log4j-1.2.16.jar. Two different versions of the same jar.
How could a ClassLoader-based solution resolve this (in a nutshell)?
If you're asking this question from an "I'm building an app, how do I avoid this" problem rather than a "I need this particular solution" angle, I would strongly prefer the Maven approach - namely, to only resolve a single version of any given dependency. In the case of log4j 1.2.15 -> 1.2.16, this will work fine - you can include only 1.2.16. Since the older version is API compatible (it's just a patch release) it's extremely likely that Fizz 1.0 won't even notice that it's using a newer version than it expected.
You'll find that doing this will probably be way easier to debug issues with (nothing confuses me like having multiple versions of even classes or static fields floating around! Who knows which one you're dealing with!) and doesn't need any clever class loader hacks.
But, this is exactly what all the appservers out there have to deal with. Pretend that your Fizz and Buzz are web applications (WARs), and Super-Jar is you appserver. Super-Jar will arrange a class loader for each web app that "breaks" the normal delegation model, i.e. it will look locally (down) before looking up the hierarchy. Go read about it in any of the appservers's documentation. For example http://download.oracle.com/docs/cd/E19798-01/821-1752/beade/index.html.
Use log4j-1.2.16. It only contains bugfixes wrt 1.2.15.
If Fizz breaks with 1.2.16, fork and patch it, then submit those patches back to the author of Fizz.
The alternative of creating custom classloaders with special delegation logic is very complex and likely to cause you many problems. I don't see why you would want to do this rather than just use OSGi. Have you considered creating an embedded OSGi framework, so you don't have to convert your whole application?
I'm writing a server application which makes use of external modules. I would like to make them to be upgradeable without requiring server restart. How do I do that? I've found OSGi but it looks very complicated and big for my task.
Simple *.jar files are ok, but once they are loaded, I suppose, I cannot unload them from VM and replace with another version on-the-fly.
What approach can you suggest?
It seems like OSGi is exactly what you're asking for. It can be complex, but there are ways to deal with that. Some of the complexity can be mitigated by using SpringDM or something similar to handle the boilerplate tasks of registering and consuming services in the runtime. Annotation-driven service registration and dependency injection really reduces the amount of code that needs to be written.
Another way to reduce complexity is to deploy the bulk of your application in a single bundle and only deploy the parts that need to be modular into their own bundles. This reduces your exposure to registering and using services from other bundles in the runtime as well as reducing the complexity of deployment. Code running within a bundle can use other code in the same bundle just as in a standard Java app - no need to interact with the OSGi runtime. The opposite of this approach is to break up your application into lots of discrete bundles that export well-defined services to other bundles in the system. While this is a very modular approach, it does come with extra complexity of managing all those bundles and more interaction with the OSGi runtime.
I would suggest taking a look at the book "OSGi in Action" to get a sense of the issues and to see some decent samples.
It would at least require you to define your custom classloader... I don't see how can this be simpler than just using Felix, Equinox, Knoplerfish or any open source Osgi runtime to do the task.
Maybe SpringDM is simpler...
What you're going for is definitely possible. I believe that you can unload classes from memory by loading them in a separate ClassLoader and then disposing that ClassLoader. If you're not wanting to go all out and use OSGI, I'd recommend something like JBoss Microcontainer (http://www.jboss.org/jbossmc) or ClassWorlds (http://classworlds.codehaus.org/). It's not too terribly difficult to write something like this from scratch if your needs are specialized enough.
Hope this helps,
Nate
If you follow the ClassLoader route (is not that difficult, really), I suggest each module to be packaged in its own jar, and use a different ClassLoader to read each jar. that way, unloading a module is the same as "discarding" the ClassLoader.
OSGi is not so complicated - using PAX runner with maven worked as a breeze.
Or implement your own ClassLoader and set it to JVM :
java -Djava.system.class.loader=com.test.YourClassLoader App.class
Our current app runs in a single JVM.
We are now splitting up the app into separate logical services where each service runs in its own JVM.
The split is being done to allow a single service to be modified and deployed without impacting the entire system. This reduces the need to QA the entire system - just need to QA the interaction with the service being changed.
For inter service communication we use a combination of REST, an MQ system bus, and database views.
What I don't like about this:
REST means we have to marshal data to/from XML
DB views couple the systems together which defeats the whole concept of separate services
MQ / system bus is added complexity
There is inevitably some code duplication between services
You have set up n JBoss server configurations, we have to do n number of deployments, n number of set up scripts, etc, etc.
Is there a better way to structure an internal application to allow modular development and deployment while allowing the app to run in a single JVM (and achieving the associated benefits)?
I'm a little confused as to what you're really asking here. If you split your application up into different services running across the network, then data marshalling has to occur somewhere.
Having said that, have you investigated OSGi ? You can deploy different bundles (basically, jar files with additional metadata defining the interfaces) into the same OSGi server, and the server will facilitate communication between these bundles transparently, since everything is running within the same JVM - i.e. you call methods on objects in different bundles as you would normally.
An OSGi server will permit unloading and upgrades of bundles at runtime and applications should run normally (if in a degraded fashion) provided the OSGi bundle lifecycle states are respected.
It sounds like your team has a manual QA process and the real issue is automating regression tests so that you can deploy new releases quickly and with confidence. Breaking up the code into separate servers is a workaround for that.
If you're willing to restart the server then one approach might be to compile the code into separate jar files, and deploy a module by dropping in a new jar and restarting. This is largely a matter of structuring your code base so that bad dependencies don't creep in and the calls between jars are made via interfaces that don't change. (Or alternately, use abstract classes so you can add a new method with a default implementation.) Your build system could help by making sure that separately deployed modules can only depend on common interfaces and anything else is a compile error. But note that your compiler isn't going to help you detect incompatibilities when you're swapping in jars that you didn't compile against, so I'm not sure this really avoids having a good QA process.
If you want to deploy new code without restarting the JVM then OSGI is the standard way to do that. (But one that I know little about.)
I have a large application (~50 modules) using a structure similar to the following:
Application
Communication modules
Color communication module
SSN communication module
etc. communication module
Router module
Service modules
Voting service module
Web interface submodule for voting
Vote collector submodule for voting
etc. for voting
Quiz service module
etc. module
I would like to import the application to Maven and Subversion. After some research I found that two practical approaches exists for this.
One is using a tree structure just as the previous one. The drawback of this structure is that you need a ton of tweaking/hacks to get the multi-module reporting work well with Maven. Another downside is that in Subversion the standard trunk/tags/branches approach add even more complexity to the repository.
The other approach uses a flat structure, where there are only one parent project and all the modules, submodules and parts-of-the-submodules are a direct child of the parent project. This approach works well for reporting and is easier in Subversion, however I feel I lose a bit of the structure this way.
Which way would you choose in the long term and why?
We have a largish application (160+ OSGi bundles where each bundle is a Maven module) and the lesson we learned, and continue to learn, is that flat is better. The problem with encoding semantics in your hierarchy is that you lose flexibility. A module that is 100% say "communication" today may be partly "service" tomorrow and then you'll need to be moving things around in your repository and that will break all sorts of scripts, documentation, references, etc.
So I would recommend a flat structure and to encode the semantics in another place (say for example an IDE workspace or documentation).
I've answered a question about version control layout in some detail with examples at another question, it may be relevant to your situation.
I think you're better off flattening your directory structure. Perhaps you want to come up with a naming convention for the directories such that they sort nicely when viewing all of the projects, but ultimately I don't think all of that extra hierarchy is necessary.
Assuming you're using Eclipse as your IDE all of the projects are going to end up in a flat list once you import them anyway so you don't really gain anything from the additional sub directories. That in addition to the fact that the configuration is so much simpler without all the extra hierarchy makes the choice pretty clear in my mind.
You might also want to consider combining some of the modules. I know nothing about your app or domain, but it seems like a lot of those leaf level modules might be better suited as just packages or sets of packages inside another top level module. I'm all for keeping jars cohesive, but it can be taken too far sometimes.