I have a complex Java project that ultimately boils down to 3 major packages:
com.myapp.client --> client-side classes
com.myapp.shared --> client- and server-side utility classes
com.myapp.server --> server-side classes
I would like to write an Ant task that checks that no com.myapp.client classes show up as dependencies for com.myapp.server classes, and vice versa. I call this concept "drawbridging", because you're putting up a drawbridge between client and server code. The Ant task would fail the build if such violating dependencies are found.
For the life of me I can't figure out how to do this. Preferably, something already exists out there, however Google didn't turn back anything.
Short of an existing open source solution, my next guess would be to loop through all the client classes and check their imports for server classes; however according to this SO question the Java compiler throws away import references at compile-time.
So, I'm at a loss here. Any ideas? Thanks in advance!
As Java bytecode references to classes by their fully qualified names it should be quite feasible (but not easy), using a library like BCEL or ASM.
However, I think this question has a pretty distinctive X-Y ring to it, meaning that you're trying to solve a problem that is actually a symptom of the root cause you should be solving, in this case probably by properly compartmentalizing your code just as Brian Roach already suggested.
AntContrib VerifyDesign is exactly what I was looking for. Thanks me!
Related
Edit: After reviewing the play, the example I used below is a tad misleading. I am looking for the case where I have two 3rd party jars (not homegrown jars where I have access to the source code) that both depend on different versions of the same jar.
Original:
So I've recently familiarized myself with what OSGi is, and what ("JAR Hell") problems it addresses at its core. And, as intrigued as I am with it (and plan on migrating somewhere down the road), I just don't have it in me to begin learning what it will take to bring my projects over to it.
So, I'm now lamenting: if JAR hell happens to me, how do I solve this sans OSGi?
Obviously, the solution would almost have to involve writing my own ClassLoader, but I'm having a tough time visualizing how that would manifest itself, and more importantly, how that would solve the problem. I did some research and the consensus was that you have to write your own ClassLoader for every JAR you produce, but since I'm already having a tough time seeing that forest through the trees, that statement isn't sinking in with me.
Can someone provide a concrete example of how writing my own ClassLoader would put a band-aid on this gaping wound (I know, I know, the only real solution is OSGi)?
Say I write a new JAR called SuperJar-1.0.jar that does all sorts of amazing stuff. Say my SuperJar-1.0.jar has two other dependencies, Fizz-1.0.jar and Buzz-1.0.jar. Both Fizz and Buzz jars depend on log4j, except Fizz-1.0.jar depends on log4j-1.2.15.jar, whereas Buzz-1.0.jar depends on log4j-1.2.16.jar. Two different versions of the same jar.
How could a ClassLoader-based solution resolve this (in a nutshell)?
If you're asking this question from an "I'm building an app, how do I avoid this" problem rather than a "I need this particular solution" angle, I would strongly prefer the Maven approach - namely, to only resolve a single version of any given dependency. In the case of log4j 1.2.15 -> 1.2.16, this will work fine - you can include only 1.2.16. Since the older version is API compatible (it's just a patch release) it's extremely likely that Fizz 1.0 won't even notice that it's using a newer version than it expected.
You'll find that doing this will probably be way easier to debug issues with (nothing confuses me like having multiple versions of even classes or static fields floating around! Who knows which one you're dealing with!) and doesn't need any clever class loader hacks.
But, this is exactly what all the appservers out there have to deal with. Pretend that your Fizz and Buzz are web applications (WARs), and Super-Jar is you appserver. Super-Jar will arrange a class loader for each web app that "breaks" the normal delegation model, i.e. it will look locally (down) before looking up the hierarchy. Go read about it in any of the appservers's documentation. For example http://download.oracle.com/docs/cd/E19798-01/821-1752/beade/index.html.
Use log4j-1.2.16. It only contains bugfixes wrt 1.2.15.
If Fizz breaks with 1.2.16, fork and patch it, then submit those patches back to the author of Fizz.
The alternative of creating custom classloaders with special delegation logic is very complex and likely to cause you many problems. I don't see why you would want to do this rather than just use OSGi. Have you considered creating an embedded OSGi framework, so you don't have to convert your whole application?
Yes i know one alternative to solving this problem is simply to create two source directories from the original. The class path for the GWT compiler would thus be setup to simply only see the compatible source while both would be used for the server portion of your app.
Firstly i find this kind of ugly, because it means i now have two source directories with potential doubles of classes.
refactoring and other structural abilities of the IDE can potentially be problematic as it will get confused.
Sometimes its not possible to put some stuff in separate packages: think client and server packages simply because one would then have to make something public which should really be package private to limit scope accessibility.
is there a library that enables classes or methods to marked as ignored by the GWT compiler ?
Is there a better way ?
You can exclude classes (files actually) from GWT's source path using Ant-like includes/excludes: http://code.google.com/webtoolkit/doc/latest/DevGuideOrganizingProjects.html#DevGuidePathFiltering
You cannot exclude methods or inner classes though, it really is file-based. See http://code.google.com/p/google-web-toolkit/issues/detail?id=3769
Make a shared directory that has the code that both the GWT side and server side can read. Any classes that would be duplicated instead go into this folder, to be accessed (without duplication!) from both client- and server-sides of your app.
The problem scienario is as follows (Note: this is not a cross-jar dependency issue, so tools like JarAnalyzer, ClassDep or Tattletale would not help. Thanks).
I have a big project which is compiled into 10 or more jar artifacts. All jars depend on each other and form a dependency hierarchy.
Whenever I need to modify one of the jars, I would check out the relevant source code and the source code for projects that depend on it. Modify the code, compile, repackage the jars. So far so good.
The problem is: I may forget to check one of the dependent projects, because inter-jar dependencies can be quite long, and may change with time. If this happens some jars may go "out-of-sync" and I will eventually get a NoSuchMethodException or a some other class incompatibility issue at run-time, which is what I want to avoid.
The only solution I can think of, the most straighforward one, is to check out all projects, and recompile the bunch. But this takes time, especially if I re-build it every small change. I do have a continuous integration server, that could do this for me, but it's shared with other developers, so seeing if the build breaks is not an option for me.
However, I do have all the jars so hypothetically it should be possible to verify jars which depend on the code that I modified have an inconsistency in method signature, class names, etc. But how could I perform such check?
Has anyone faced a similar problem before? If so, how did you solve it? Any tools or methodologies would be appreciated.
Let me know if you need clarification. Thanks.
EDIT:
I would like to clarify my question a little bit.
The ultimate goal of this task is to check that the changes that I have made will compile against the whole project. I am looking for a tool/technique that would aid me perform such check.
Consider this example:
You have 2 projects: A and B which are deployed as A.jar and B.jar respectively. A depends on B.
You wish to modify B, so you check it out and modify a method signature that A happens to depend on. You can compile B and run all tests by itself without any problems because B itself does not depend on anything. So you happily commit your changes.
In a few hours the complete project integration fails because A could not be compiled!
How do I avoid this?
The kind of tool I am looking for would retrieve A.jar and check that all dependencies in A on the new modified B are still fine. Like a potential compilation error that would happen if I were to recompile A and B sources together.
Another solution, as was suggested by many of you, is to set up a local continuous integration system that would recompile the whole project locally. I don't mind doing this, but I want to avoid doing it inside my workspace. On the other hand, if I check-out all sources to another temporary workspace, then I need to mirror my local changes to the temporary workspace.
This is quite a big issue in my team, as builds break very often because somebody forgot to check out (or open in Eclipse) the right set of projects. I tried persuading people to check-out source and recompile the bunch before commits, but not only it takes time, it needs running quite a few commands so most people just find it too troublesome to do. If the technique is not easy or automated, then it's unusable.
If you do not want to use your shared continuous integration server you should set up a local one on your developer machine where you perform the rebuild processes on change.
I know Jenkins - it is easy to setup (just start) on a local machine and I would advice to run it locally if no one is provided in the IT infrastructure that fits your needs.
Checking signatures is unfortunately not enough. Having the correct signatures does not mean it'll work. It's all about contracts and not just signatures. I mean what happens if the new version of a library has the same method signature, but accepts an ArrayList parameter now in reversed order? You will run into issues - sooner or later. I guess you maybe consider implementing tools like Ivy or Maven:
http://ant.apache.org/ivy/
http://maven.apache.org/
Yes it can be pain to implement it but once you have it it will "guard" your versions forever. You should never run into such an issue. But even those build tools are not 100% accurate. The only proper way of dealing with incompatible libraries, I know you won't like my answer, is extensive regression testing. For this you need bunch of testing tools. There are plenty of them out there: from very basic unit testing (JUnit) to database testing (JDBC Proxy) and UI testing frameworks like SWTBot (depends if your app is a web app or thick client).
Please note if your project gets really huge and you have large amount of dependencies you always not using all of the code there. Trying to check all interfaces and all signatures is way too much. Its not necessary to test it all when your code use lets say 30 % of the library code. What you need is to test what you really use. And this can be only done with extensive regression testing.
I have finally found a whole treasure box of answers at this post. Thanks for help, everyone!
The bounty goes to K. Claszen for the quickest and most input.
I'm also think that just setup local Jenkins is a best idea. What tool you use for build? Maybe you can improve you situation with switching to Maven as build tool? In more smart and don't recompile full project if you don't ask it directly. But switch to in can be HUGE paint in the neck - it hardly depends on how you project organized now...
And about VCS- exist Mercurial/SVN bridge - so you can use local Mercurial for you development ....
check this link: https://www.mercurial-scm.org/wiki/WorkingWithSubversion
There is a solution, jarjar, which allows to have different versions of the same library to be included multiple times in the dependency graph.
I use IntelliJ, not Eclipse, so maybe my answer is too IDE-specific. But in IntelliJ, I would simply include the modules from B into A, so that when I make changes to A, it breaks B immediately when compiling in the IDE. Modules can belong to multiple projects, so this is not anything like duplication, it's just adding references in the IDE to modules in other projects.
The title speaks for itself. The language is Java.
Yes, there is. This is however a tedious and expensive work. You need to crawl through all class files and all JAR files with help of ClassLoader#getResources() and a shot of java.io.File and load all classes of it with help of Class#forName() and finally check if the method is there by Class#getMethod().
However, there are 3rd party API's which can take the tedious work from hands, but it is still expensive, because loading a class would cause its static initializers being executed.
A cleaner way is to make use of annotations and annotate the methods in question and then make use of libraries which searches for classes/methods/fields based on the annotations, such as Google Reflections.
On the other hand, if the entire package name or the JAR file name is known beforehand, then the work will be less tedious and expensive (no need to do stuff recursively nor to load the all of the classes of entire classpath).
Update: I remember, I ever wrote sample code to achieve something like that, you can find it here. It's good to start with, you only need to change it a bit to check the method.
No, you can't, in general. If you could get a complete list of available classes you could check each of them using reflection - but you can't ask a classloader for a list of everything that's available. (For instance, it may be fetching classes over HTTP, and may not know all the files available.)
If you knew that you were interested in classes in a jar file, however, you could open the jar file, find all the class files within it and ask the classloader for those classes. It would be somewhat fiddly.
What's the bigger picture here? There may be a better way to approach the problem.
Also, in Eclipse, you can simply ask for this :
Clic on the method, and type Ctrl-T.
It may not be best practice but are there ways of removing unsused classes from a third party's jar files. Something that looks at the way in which my classes are using the library and does some kind of coverage analysis, then spits out another jar with all of the untouched classes removed.
Obviously there are issues with this. Specifically, the usage scenario I put it though may not use all classes all the time.
But neglecting these problems, can it be done in principle?
There is a way.
The JarJar project does this AFAIR. The first goal of the JarJar project is to allow one to embed third party libraries in your own jar, changing the package structure if necessary. Doing so it can strip out the classes that are not needed.
Check it out at http://code.google.com/p/jarjar/.
Here is a link about shrinking jars: http://sixlegs.com/blog/java/jarjar-keep.html
There is a tool in Ant called a classfileset. You specify the list of root classes that you know you need, and then the classfileset recursively analyzes their code to find all dependencies.
Alternatively, you could develop a good test suite that exercises all of the functions that you need, then run your tests under a test coverage tool. The tool will tell you which classes (and statement in them) were actually utilized. This could give you an even smaller set of code than what you'd find with static analysis.
I use ProGuard for this. As well as being an excellent obfuscator, it has a code shrinking phase which can combine multiple JARs and then strip out any unused classes or class members. It does an excellent job at shrinking.
At a previous job, I used a Java obfuscator that as well as obfuscating the code, also removed classes and methods that weren't being used. If you were doing "Class.byName" or any other type of reflection stuff, you needed to tell the obfuscator because it couldn't tell by inspecting the code what classes or methods called by reflection.
The problem, of course, is that you don't know if other parts of the third party library are doing any reflection, and so removing an "unused" class might cause things to break in an obscure case that you haven't tested.
jar is just a zip file, so I guess you can. If you could get to the source, it's cleaner. Maybe try disassembling the class?
Adding to this question, can that improve performance? Since the classes not used would not be JIT compiled improving startup time or does the java automatically detect that while compiling to bytecode and do not even deal with the code that is not used?
This would be an interesting project (has anyone done it already?)
I presume you'd give the tool your jar(s) as a starting point, and the library jar to clean up. It could use reflection to determine which classes your jar(s) reference directly, and which are used indirectly down the call tree (this is not trivial at all, but doable). If it encounters any reflection code in any of the two places, it should give a very loud warning.