I am trying to run a map/reduce job and I am getting a java.lang.NoSuchMethodError. I did some research on this and this appears when my code is executed (not compiled). The correct version of the class and methods are there during compilation, but when trying to run, the correct method is not available. The jar file that is causing this is guava. I know this from the stack that is printed. I throws an error when trying to execute the following line of code:
ArrayDeque<Entry<String, String>> a = Queues.newArrayDeque();
This jar is part of the hadoop classpath because it comes with the CDH verson 5.3.0 that I am using. I have tried adding the correct version of guava to the classpath, but the error does not change. My questions are as follows:
I believe that I have correctly identified the issue. Does this seem reasonable to you? I have never come across this error before.
I believe that I need to remove the older version of guava from the classpath and add the new one. However, I really do not know where to begin with correcting this. The command that is issued to hadoop jar does not contain the older version of guava (in the -libjar parm). The jar is part of the hadoop classpath when I issue the command "hadoop classpath". So I am assuming that there is some hadoop config file I could edit to make this go away. Is that the correct way to go, or is there some other thing I need to do?
I am using Java 7, CDH 5.3.0, NetBeans 8.
TIA
At the time that I'm writing this, Hadoop has a dependency on Guava version 11.0.2. It uses the library pretty heavily in its internal implementation.
According to the Guava JavaDocs, the Queues#newArrayDeque method was added in version 12.0. If your code is compiling successfully, then that means that Guava version 12.0 or higher is available on your compilation classpath at build time, but since version 11.0.2 is supplied at runtime by Hadoop, the method doesn't exist, resulting in NoSuchMethodError.
Unfortunately, there is no reliable way to swap out a different Guava version in Hadoop. Specifically, I recommend that you do not attempt to replace the Guava 11.0.2 jar that ships in the Hadoop distro. Replacing this with a different Guava version is untested, and it would risk destabilizing the cluster.
The broader problem is that Hadoop's dependencies "leak" to its clients. HADOOP-11656 is an unimplemented feature request that would isolate Hadoop's internal dependencies away from clients, so that you could more easily use common libraries like Guava at your desired version. Meanwhile, until that feature is implemented, I think your only options are to stick to Guava 11.0.2 APIs, or possibly try inlining some of the Guava code that you really want into your own project directly. The code for Queues#newArrayDeque is visible on GitHub.
public static <E> ArrayDeque<E> newArrayDeque() {
return new ArrayDeque<E>();
}
In this particular case, it looks like it will be easy to replace your code with a direct call to the java.util.ArrayDeque constructor. Thanks to the Java 7 diamond operator, it won't even be much more verbose.
ArrayDeque<Entry<String, String>> a = new java.util.ArrayDeque<>();
Related
I need to know the Java runtime compatibility of all of the dependencies I'm using in my gradle project (as if it were checking each project's "sourceCompatibility" setting so to speak), preferrably without perusing each dependency's documentation to find it, if it's there at all. Is there a way to do this with gradle? Or even some other automating tool?
(Specifically I'm trying to see which dependencies might be using Java 8 features like Streams, since I'm trying to compile for Android with retrolambda, and iOS with RoboVM.)
Such information probably does not exist, and is very tedious to generate.
"sourceCompatibility" does not exist after the source is compiled-- it's just to tell the compiler how to interpret the source syntax. What you're more likely to be interested in is the "targetCompatibility", or the class file format major version: Java 8 is 52, Java 7 is 51, etc.
This tells Java that Java 8 is required to understand the class format and bytecodes contained in the class, so you could download and unpack every dependency in your project, and all of their dependencies, and then look at the version number of every class file, except...
Simply looking at the class file format version doesn't tell you whether the class makes reference to methods and fields that exist only in specific versions of the JDK. I've not tested with Java 8, but in Java 7 it's possible to set the source/target compatibility to 1.6 and still reference new methods that were added in Java 7. A Java 6 JVM will load and run the file, but fail with NoSuchMethodException despite otherwise looking perfectly fine.
The only way to realistically check if a dependency is completely compatible with a different version of Java than the one it was compiled for is to go through the constant pool of every class, find every class and method reference, and then verify that they are valid for the desired JRE.
You will want an automated tool for this (The JVM could do it if you have the desired version of Java installed and 100% coverage in your unit tests...), but I don't know of a standalone tool that does, and neither gradle nor project documentation usually includes this sort of info.
According to the ASM FAQ, to get example ASM code, I should use the ASMifier class, like this:
java -classpath "asm.jar;asm-util.jar" org.objectweb.asm.util.ASMifier org/domain/package/YourClass.class
But that gets me the error:
Error: Could not find or load main class org.objectweb.asm.util.ASMifier
Looking at JAR files, the ASMifier class seems to be missing, though its helper classes are present:
./org/objectweb/asm/util/ASMifierClassVisitor.class
./org/objectweb/asm/util/ASMifierAnnotationVisitor.class
./org/objectweb/asm/util/ASMifierMethodVisitor.class
./org/objectweb/asm/util/ASMifierAbstractVisitor.class
./org/objectweb/asm/util/ASMifierFieldVisitor.class
This is with ASM 3.3.1, as provided by Fedora 20. Is the FAQ for a newer version, and I should be using different instructions? Did Fedora mess up the packaging (even though their bug-tracker shows nothing)? Something else?
When browsing the SVN repository of ASM, you can read up the revision history of the ASMifier: It was formerly known as the ASMifierClassVisitor which was also this utility's name in version 3.1.1.
ASM never had a reputation of maintaining binary or even compilation compatibility. Thus, you might encounter several problems like the one you describe when using non-bleeding edge versions of the library. (The authors promised to improve this after ASM's version four.) You, or the libraries that you use, should however always repackage ASM into a different namespace in order to avoid such issues. This is even recommended in the FAQ to using ASM.
For running your example, you would have to use:
java -classpath "asm.jar;asm-util.jar" \
org.objectweb.asm.util.ASMifierClassVisitor \
org/domain/package/YourClass.class
I am from .net world. I remember .net will immediately complain if you build with one dll but supply a different dll at run time.
I am now adding some hadoop reference to my project and find the following article.
http://answers.mapr.com/questions/364/maven-repository-for-mapr-jar-files
I just don't understand how this happens.
Java can build with one jar but run with a different jar?
Thanks
yes. this is often the case with APIs (you compile the API, but at runtime you may run with a newer version of the API which may be included with the implementation). everything will work out fine as long as the classes/method prototypes referenced in your compiled code are unchanged from the jar you compiled against.
For a specific definition of compatibility, see binary compatibility (thanks to #MiserableVariable for the link).
There is Java tool (it is called Mallet)
http://mallet.cs.umass.edu/download.php
which I want to use in my .NET project.
To convert this tool to .NET library at first I've tried to build it in single .jar file using Apache Ant. I've done everything corresponding to instructions at link above.
Download Developer Release from Mercurial repository.
Download Apache Ant, install JDK, set JAVA_HOME var to use Apache Ant.
Using Ant I've built single mallet.jar file.
And then I would to convert mallet.jar to .NET library using IKVMC.
When converting, I've got a lot of warnings such as:
Warning IKVMC0108: not a class file "cc/mallet/util/tests/TestPriorityQueue$1.cl
ass", including it as resource
(class format error "51.0")
Despite of these warnings, mallet.dll was created. But when I try to reference to it from my .NET project, it looks "empty". It has not any classes or namespaces. I don't forget to reference to IKVM.OpenJDL.Core.
And this is unusual that I can't find any same problems in Google.
I think that problem is in warnings. And I have never worked with Ant and I don't understand all process exactly.
The class format version 51 was introduced with Java 7.
IKVM most likely doesn't support that version yet and the file name you quote (cc/mallet/util/tests/TestPriorityQueue$1.class) points at an anonymous inner class of TestPriorityQueue that certainly is needed for the library to work correctly.
My suggestion: compile Mallet using an older JDK or at least using the -source and -target switches set to 6 (to ensure that it's compile for Java 6).
FYI v8.1 (currently in RC) of IKVM supports Java 8:
http://weblog.ikvm.net/2015/08/26/IKVMNET81ReleaseCandidate0.aspx
http://sourceforge.net/p/ikvm/mailman/message/34502991/
This is a common problem. I'm using 2 libraries A.jar and B.jar and these depend on different versions of the same jar.
Let's say that at runtime I need THIS.x.x.x.jar
MY.jar
-> A.jar -> THIS.1.0.0.jar
-> B.jar -> C.jar -> THIS.5.0.0.jar
I can compile the specific jar (A.jar/B.jar) against its dependency but at runtime I've to load only 1 version. Which one?
Loading only 1 dependency (the latest version) means that my code will probably throw runtime exceptions if the libraries are not Backward Compatible (are there Backward Compatible libraries out there?).
Anyway I know that something like OSGi can fix this issue.
I'm wondering what's the old way to fix this kind of problems...
Thanks a lot
"Old way" you mentioned (and the one OSGI certainly uses under the hood) is to install your own ClassLoader for both branches of your dependencies. That's how, for instance, application servers are able to run both older and newer versions of the same application inside the same JVM.
Read about classloader hierarchy.
In your setup, the tricky part is the joint point, where classes from both branches meet. Neither branches can use classes loaded into another one. The way to make it work is to make sure only classes loaded by boot classloader (JRE classes) or classloader of MY.jar are passed down to both branches.
OSGi can fix this problem. An OSGi bundle is nothing more than a jar with additional metadata detailing versions. A bundle has a version number, and will detail version numbers (or ranges) of dependent jars.
Take a look at this introductory Javaworld article for more information.
To solve this without OSGi means having to ensure manually that you compile and run with compatible jars. As you've discovered that's not necessarily a trivial task. Since jars don't necessarily identify their versions, the only sure way to do this to record/compare checksums or signatures.
Many libraries are backward compatible. But not all..
The old way is to try to depend from only one version.
It is probably safer to compile both with the same version (latest).
At least you get compile-time errors, instead of runtime errors.
If needed, you can modify a little bit your library that works with the old dependency...
This would require access to the source...
Please note that compile-time compatibility will not guarantee correct runtime behavior either. It is one step, then you can:
read the WhatsNew file for the new version of the jar
look on the Internet for users reporting compatibility problems
write JUnits
compare the codes in both jars
As mentioned by KLE, the default approach is to depend on the newer version. There is no guarantee, but most of the time this works. Probably the best way (while being a bloated one) is using OSGI to get over it.
To refer a basic "oldway" implementation checkout https://github.com/atulsm/ElasticsearchClassLoader
This provides an approach to handle non-backward compatible versions of elasticsearch client usage.