Develop Reference

How to use org.apache.httpcomponents inside a spark job on Hadoop/Spark?

I am trying to run a spark job on a Hadoop cluster that also makes an http request to another server. I am using org.apache.httpcomponents to make this request, which works fine locally on my machine. However this fails the moment I submit the job to the cluster (managed by Cloudera) with the following error:
User class threw exception: java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.(SSLConnectionSocketFactory.java:151)
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:977)
at org.apache.http.impl.client.HttpClients.createDefault(HttpClients.java:56)
From all the reading I have done, this error is caused by multiple versions of Apache Http client jar. It appears that Hadoop/Spark engine has it's own dependency to Apache Http client and that is a different version than the one I am using. Because my jar is run as part of the hadoop/spark engine it ends up including both my version of http as well as the one Hadoop requires.
If I add 'compileOnly' for org.apache.httpcomponents in my build.gradle and submit, I get this error instead:
User class threw exception: java.lang.NoClassDefFoundError: org/apache/http/impl/client/HttpClients
Is there a way for me to configure this in gradle so that when I build my jar, it will use the already existing version on Hadoop? ie. A way to declare a temporary dependency (when running locally download and use latest version, but when building UberJar drop the dependency)?
UPDATE
I decided to try swapping to a different http library (okhttp3) to see if that would resolve the issue. However I get a very similar exception when trying to run through the cluster here too:
User class threw exception: java.lang.NoSuchFieldError: Companion
at okhttp3.internal.Util.(Util.kt:70)
at okhttp3.OkHttpClient.(OkHttpClient.kt:959)
Looks like Cloudera also supplies a version of okhttp with it's spark2 client which is unfortunate.

Referencing an external class from a Jersey REST web app

I am trying a simple example of creating a RESTful API. I used the Maven jersey-quickstart-webapp archtype to create a Jersey web app. In a separate project I created a simple class: SimpleClass, which only has one method that returns "hello". I am trying to create an instance of that class in my method in the web app (I am using the default MyResource class) and call its method to get "Hello". I exported the project with the SimpleClass to a JAR file, and added that file to the build path for my Jersey project, and imported the relevant package.
When I run the web app on a local server, and call the MyResource method, I get an error with the root cause (just the top few rows):
java.lang.NoClassDefFoundError: Lorg/test/simple/SimpleClass;
java.lang.Class.getDeclaredFields0(Native Method)
java.lang.Class.privateGetDeclaredFields(Class.java:2583)
java.lang.Class.getDeclaredFields(Class.java:1916)
So, it can't find the SimpleClass class.
What am I doing wrong here? I'd really appreciate any help!
Thanks,
Yariv.
New info: I tried to create a RESTful web service without Maven. I successfully created the project with a simple "Hello World" example, and then I tried to add my own class from another project: I imported the .jar, added it to the build path, and created a class in the web service that creates an instance of my class when a GET is made to the server. My class happens to include (as members) instances of other classes, one of which connects to a MongoDB database, and when I run the service, I again get a HTTP 500 error as follows:
A MultiException has 2 exceptions. They are:
1. java.lang.NoClassDefFoundError: com/mongodb/DBObject
2. java.lang.IllegalStateException: Unable to perform operation: create on org.notes.server.NotesServer
Adding the MongoDB jar to the web project did not help. I must be missing something big here. What should I do to call code I created in another project (with a set of dependencies there) in my web service?
I'll appreciate any help I can get, I am getting nowhere with this...

I was able to resolve all these issues by adding all the relevant JARs to the deployment assembly (right-click on the project, click Properties, then select Deployment Assembly and add all the JARs that my class depends on, as well as its own JAR).
Thanks for your help, radai!

NoClassDefFoundError for com.sun.xml.internal.ws.fault.SOAPFaultBuilder

Our web service client in live environment recently got the exception:
java.lang.NoClassDefFoundError: Could not initialize class com.sun.xml.internal.ws.fault.SOAPFaultBuilder
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:107)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:135)
at com.sun.proxy.$Proxy146.search(Unknown Source)
....
I've done a lot of search online, including a few posts here at StackOverflow:
Catching webservice exception with CXF: NoClassDefFoundError: SOAPFaultBuilder
Could not initialize class com.sun.xml.internal.ws.fault.SOAPFaultBuilder
java.lang.NoClassDefFoundError: Could not initialize class com.sun.xml.internal.ws.fault.SOAPFaultBuilder
And my understanding was our client received a SOAP Fault from the server and it's missing some jar file. So I'm trying to solve the problem by first recreating it.
I created a simple Web Service server project in Eclipse which has a web method throws a simple fault class annotated by #WebFault. Then I created a simple Web Service client project which consumes the web method. The client project doesn't have any additional libraries/jars in its classpath; all it has is the JRE. To my surprise, it didn't throw the NoClassDefFoundError exception! Instead, I got the javax.xml.ws.soap.SOAPFaultException I defined on the server side.
The class SOAPFaultBuilder is indeed in rt.jar in JRE. So the simple web service projects I created probably just work as they should. However, how come the web service client in our live environment throw the NoClassDefFoundError exception? That project definitely has rt.jar in the classpath.
Can anyone please shed some light on this problem? If it's missing some jar files (either from the jaxws RI or Apache CXF or others), why would the super simple client I created didn't throw the error? Both the live environment and my local environment use Java7u51.

I have encountered the same problem and I could subsequently resolve this.
Due to the following error mentioned in the Problem, the underlying cause of the SOAP Fault could not be found.
I followed the steps mentioned in the following link, to identify the reason for NoClassDefFoundError.
http://javarevisited.blogspot.in/2011/06/noclassdeffounderror-exception-in.html
I could find the following errors in my application log:
java.lang.ExceptionInInitializerError
Caused by: java.lang.ClassCastException: com.sun.xml.bind.v2.runtime.JAXBContextImpl cannot be cast to com.sun.xml.internal.bind.api.JAXBRIContext
The reason for the above classcast exception is due to conflicting jar files.
rt.jar (this is present in jre classpath)
jaxb-impl-2.0.1.jar (this is present in my application classpath).
I have removed the file jaxb-impl-2.0.1.jar from my classpath and the actual error is gone.

I had the same erorr and I resolved this issue by deleting the jax-impl.jar from the tomcat lib folder. It was a possible conflict of the jaxb jar versions with one of my webapps installed in tomcat.
http://programtalk.com/java/i-was-running-gwt-application-on-tomca/

Axis2 WS consumer in WebMethods8.2

I've got into a scenario where I have to get an Axis2 based ws consumer working within WebMethods as a java service. I've implemented the ws consumer first in netbeans just to see if it works and thus i found that the minimal amount of jars I'll require are the following:
[ xmlschema-1.4.7.jar, apache-mime4j-core-0.7.2.jar,
axiom-api-1.2.13.jar, axiom-impl-1.2.13.jar, axis2-adb-1.6.2.jar,
axis2-kernel-1.6.2.jar, axis2-transport-http-1.6.2.jar,
axis2-transport-local-1.6.2.jar, commons-codec-1.3.jar,
commons-httpclient-3.1.jar, commons-logging-1.1.1.jar,
httpcore-4.0.jar, mail-1.4.jar, neethi-3.0.2.jar, wsdl4j-1.6.2.jar ]
I've uploaded these jar files under the IS/packages/{package_name}/code/jars folder. Whenever I try to execute the java service that would send the request and process the response I get the following exception:
java.lang.reflect.InvocationTargetException:
org.apache.axiom.om.OMFactory.getMetaFactory()Lorg/apache/axiom/om/OMMetaFactory;
From the IS error log file I found that the actual error message is as follows:
org.apache.axiom.om.OMFactory.getMetaFactory()Lorg/apache/axiom/om/OMMetaFactory;
Caused by: java.lang.reflect.InvocationTargetException: null Caused
by:
java.lang.NoSuchMethodError:org.apache.axiom.om.OMFactory.getMetaFactory()Lorg/apache/axiom/om/OMMetaFactory;
The platform is WebMethods 8.2 under Linux environment. The JDK version is 1.6.0_32 and the application server under WebMethods is Jetty.

Actually the solve of this problem was a bit more tricky. First of all I manually had to configure the manifest file of the package on the IS server to use the jars provided in the package abnd thus it wouldn't get in conflict with the Axis used by the IS itself. On the other hand I had to manually add the ClassLoader because WebMethods can't use META-INF based information from jar files as it seems. To solve this problem simply use:
System.setProperty("org.apache.axiom.om.OMMetaFactory", "org.apache.axiom.om.impl.llom.factory.OMLinkedListMetaFactory");
That solves all the problems.

DWR invoke error in Javascript

I am using Spring 3.05 and DWR 3.0rc1 in my application for making AJAX calls. In local environment, i.e. Eclipse & Tomcat Server, it's working fine but in Clustered Environment I am getting the following errors while invoking DWR and a script error is coming saying DWR is not defined, but I am able to see myApplication/dwr/index.html properly.
Error Log:
Skipping 'script' due to NoClassDefFoundError on org.directwebremoting.create.ScriptedCreator. Cause: org/apache/bsf/BSFException
Skipping 'pageflow' due to ClassNotFoundException on org.directwebremoting.beehive.PageFlowCreator. Cause: Beehive/Weblogic jar file not available.
adding creator type: none = class org.directwebremoting.create.NullCreator
adding creator type: new = class org.directwebremoting.create.NewCreator
Please help me out

Looks to me that jars are not included in the classpath. Can you please check whether it is included?