Develop Reference

remote flink job with query to Hive on yarn-cluster error:NoClassDefFoundError: org/apache/hadoop/mapred/JobConf

env: HDP: 3.1.5(hadoop: 3.1.1, hive: 3.1.0), Flink: 1.12.2
Java code:
public static void main(String[] args) {
EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().build();
TableEnvironment tblEnv=TableEnvironment.create(settings);
String name = "myhive";
String defaultDatabase = "default";
String hiveConfDir = "/etc/hive/conf";
HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir);
tblEnv.registerCatalog("myhive", hive);
tblEnv.useCatalog("myhive");
//tblEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
tblEnv.sqlQuery("SELECT * FROM users").execute().print();
}
Dependency:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hive_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
error 1:
org.apache.flink.util.FlinkException: JobMaster for job 35afe414e1dd861c86130ddd031312f2 failed.
at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:887) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.dispatcher.Dispatcher.dispatcherJobFailed(Dispatcher.java:465) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.dispatcher.Dispatcher.handleDispatcherJobResult(Dispatcher.java:444) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
...
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_292]
...
Caused by: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator Source: HiveSource-zjdev_xiangliang.users -> SinkConversionToTuple2
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:231) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:866) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
...
Caused by: java.lang.NoClassDefFoundError: Lorg/apache/hadoop/mapred/JobConf;
at java.lang.Class.getDeclaredFields0(Native Method) ~[?:1.8.0_292]
at java.lang.Class.privateGetDeclaredFields(Class.java:2583) ~[?:1.8.0_292]
at java.lang.Class.getDeclaredField(Class.java:2068) ~[?:1.8.0_292]
at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1871) ~[?:1.8.0_292]
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_292]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_292]
...
try add dependency
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
get another error
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
at org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.<clinit>(CommandLineOptions.java:27)
at org.apache.flink.runtime.entrypoint.DynamicParametersConfigurationParserFactory.options(DynamicParametersConfigurationParserFactory.java:43)
at org.apache.flink.runtime.entrypoint.DynamicParametersConfigurationParserFactory.getOptions(DynamicParametersConfigurationParserFactory.java:50)
at org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:42)
at org.apache.flink.runtime.entrypoint.ClusterEntrypointUtils.parseParametersOrExit(ClusterEntrypointUtils.java:63)
at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
try to fix conflict about commons-cli:1.3.1 with 1.2:
choose 1.3.1 then error 1;
choose 1.2 then error 2;
add dependency commons-cli 1.4, then error 1.

1、commons-cli choose 1.3.1 or 1.4
2、add $hadoop_home/../hadoop_mapreduce/* to yarn.application.classpath

i got this error too , when i think i maybe a version conflict ,because i used
hive 3.1.2 version is to high than hadoop 2.7.6 version. i always got error such as guava version conflict and so on
so when i used flink-1.15.0 and link hive with jar flink-sql-connector-hive-3.1.2_2.12-1.15.0.jar i got Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
resolve: this error it because not found jars please move your hadoop jars
hadoop-mapreduce-client-core.,hadoop-common,hadoop-mapreduce-client-common
,hadoop-mapreduce-client-jobclient and hive-exec-3.1.2.jar to flink lib path

Elasticsearch - java.lang.NoClassDefFoundError

I'm trying to connect a java client to my elasticsearch server.
The version of elasticsearch is 2.4.0.
This server has a special configuration that's why I use port 10700 instead of 9300 in order to connect.
(When I try "telnet serveraddress 10700" I succeed).
As you'll see below, I have an error and I don't know how to resolve it.
Can you help me, please ?
Thanks.
My Code :
public void connexionToEs() throws UnknownHostException {
String clusterName = "xxx";
String serverAddress = "xxx";
try{
Settings settings = Settings.builder()
.put("cluster.name", clusterName)
.put("client.transport.sniff", true)
.build();
TransportClient client = new PreBuiltTransportClient(settings);
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(serverAddress), 10700));
SearchResponse response = client.prepareSearch().execute().actionGet();
String output = response.toString();
System.out.println(output);
client.close();
}catch(Exception e){
e.printStackTrace();
}
}
My dependencies :
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-web</artifactId>
<version>2.6.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
<version>2.6.2</version>
</dependency>
My error :
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/plugins/ActionPlugin
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.elasticsearch.transport.client.PreBuiltTransportClient.<clinit>(PreBuiltTransportClient.java:54)
at fr.solocal.dao.impl.CentralDAOImpl.connexionToEs(CentralDAOImpl.java:422)
at fr.solocal.App.main(App.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.plugins.ActionPlugin
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more

As #Roma Khomyshyn mentioned, Spring Data Elasticsearch only support transport client currently, but the version of Elasticsearch server and transport client must be the same. That's why you got unexpected error like this one.
A possible solution is another Spring Data implementation using JEST client - https://github.com/VanRoy/spring-data-jest

java.lang.ClassNotFoundException: com.mulesoft.weave.module.writer.Writer

I'm having an error with mule standalone version 3.8.1 when trying to transform a payload into an xml before sending it to a webservice.
My flow config is like this:
<dw:transform-message doc:name="Transform Message" mode="immediate">
<dw:set-payload>
<![CDATA[%dw 1.0
%input payload application/java
%output application/xml
%namespace ns0 urn:INDRA:JIRA:Integracion
---
{
ns0#Create: {
KeyClient: payload.externalIssue,
Accion: payload.accion,
Priority: payload.priority,
Assignee: payload.assignee,
Resolution_Coment: payload.reopenComment,
Solucion: payload.comment,
IssueType: payload.issueType,
ComponentJIRA: payload.component,
No_Rellenar: ""
}
}
]]>
</dw:set-payload>
</dw:transform-message>
When we try to process the message we get:
ERROR 2016-12-19 13:37:52,208 [ActiveMQ Session Task-1] org.mule.exception.RollbackMessagingExceptionStrategy:
********************************************************************************
Message : com.mulesoft.weave.module.Module: Provider com.mulesoft.weave.module.flatfile.FlatFileModule could not be instantiated: java.lang.NoClassDefFoundError: com/mulesoft/weave/module/writer/Writer (java.util.ServiceConfigurationError).
Payload : RemedyMessageBean{accion='TCURSO', comment=' ', empresa='SSII-Administración (CSA)', externalIssue='INC000000004965', projectParameter4='No Aplica', dueDate='31/12/2016 10:17:00', priority='', reopenComment='', projectParameter1='No Aplica', projectParameter11='No Aplica', stopReason='', component='AX-UTES', issueType='', stopComment='null', assignee=''}
Element XML : <dw:transform-message doc:name="Transform Message" mode="immediate">
<dw:set-payload>%dw 1.0%input payload application/java%output application/xml%namespace ns0 urn:INDRA:JIRA:Integracion---{ns0#Create: {KeyClient: payload.externalIssue,Accion: payload.accion,Priority: payload.priority,Assignee: payload.assignee,Resolution_Coment: payload.reopenComment,Solucion: payload.comment,IssueType: payload.issueType,ComponentJIRA: payload.component,No_Rellenar: ""}}</dw:set-payload>
</dw:transform-message>
Payload Type : es.indra.esb.remedy.servicios.RemedyMessageBean
Element : /AsincronoRemedySI/processors/2/5/0/3 # EsbOrigin-a:remedy-config.xml:508 (Transform Message)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.lang.ClassNotFoundException: com.mulesoft.weave.module.writer.Writer
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at org.mule.module.launcher.FineGrainedControlClassLoader.findClass(FineGrainedControlClassLoader.java:175)
at org.mule.module.launcher.MuleApplicationClassLoader.findClass(MuleApplicationClassLoader.java:134)
at org.mule.module.launcher.FineGrainedControlClassLoader.loadClass(FineGrainedControlClassLoader.java:119)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2493)
at java.lang.Class.getConstructor0(Class.java:2803)
at java.lang.Class.newInstance(Class.java:345)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
The dependency in my pom.xml is:
...
<mule.version>3.8.1</mule.version>
...
<dependency>
<groupId>com.mulesoft.weave</groupId>
<artifactId>mule-plugin-weave</artifactId>
<version>${mule.version}</version>
</dependency>
<dependency>
<groupId>com.mulesoft.weave</groupId>
<artifactId>flatfile</artifactId>
<version>1.2.0-M4</version>
</dependency>
Thanks,
Jose.

Add the following dependency
<dependency>
<groupId>com.mulesoft.weave</groupId>
<artifactId>core</artifactId>
<version>1.2.0-M4</version>
</dependency>

As confirmed by #joseluisvl (the OP) in comments, the exception arises because DataWeave is only available for EE (Enterprise Edition) runtimes.

update your pom to mule plugin version 4
<!-- https://mvnrepository.com/artifact/com.mulesoft.weave/mule-plugin-weave -->
<dependency>
<groupId>com.mulesoft.weave</groupId>
<artifactId>mule-plugin-weave</artifactId>
<version>4.0.0-M4</version>
</dependency>
<dependency>
<groupId>com.mulesoft.weave</groupId>
<artifactId>core</artifactId>
<version>1.2.0-M4</version>
</dependency>
SAXParseException: cvc-complex-type.2.4.a: Invalid content was found
starting with element 'dw:transform-message
this means that you did not configure correctly the flow so be sure to declare everything you need in the namespace

connection to Magento SOAP API java

i am trying to connect to the magento soap api in java, i have added almost all the jar files as per the error logs, but i am stuck at this run time exception, please have a look a look at the code and the output.
import com.google.code.magja.soap.MagentoSoapClient;
import com.google.code.magja.soap.SoapConfig;
import org.apache.axis2.AxisFault;
public class testConnection {
public static void main(String[] args) throws AxisFault {
String user = "XXXXXX";
String pass = "XXXXX";
String host = "http://XXX.co.in/magento/index.php/api/soap?wsdl";
SoapConfig soapConfig = new SoapConfig(user, pass, host);
MagentoSoapClient magentoSoapClient = MagentoSoapClient.getInstance(soapConfig);
// configure connection
magentoSoapClient.setConfig(soapConfig);
}
}
OUTPUT
Exception in thread "main" java.lang.RuntimeException: org.apache.axis2.deployment.DeploymentException: org.apache.axis2.transport.http.CommonsHTTPTransportSender
at com.google.code.magja.soap.MagentoSoapClient.<init>(MagentoSoapClient.java:56)
at com.google.code.magja.soap.MagentoSoapClient.getInstance(MagentoSoapClient.java:79)
at magentomanager.testConnection.main(testConnection.java:14)
Caused by: org.apache.axis2.deployment.DeploymentException: org.apache.axis2.transport.http.CommonsHTTPTransportSender
at org.apache.axis2.deployment.AxisConfigBuilder.processTransportSenders(AxisConfigBuilder.java:708)
at org.apache.axis2.deployment.AxisConfigBuilder.populateConfig(AxisConfigBuilder.java:122)
at org.apache.axis2.deployment.DeploymentEngine.populateAxisConfiguration(DeploymentEngine.java:857)
at org.apache.axis2.deployment.FileSystemConfigurator.getAxisConfiguration(FileSystemConfigurator.java:116)
at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:64)
at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContextFromFileSystem(ConfigurationContextFactory.java:210)
at org.apache.axis2.client.ServiceClient.configureServiceClient(ServiceClient.java:151)
at org.apache.axis2.client.ServiceClient.<init>(ServiceClient.java:144)
at org.apache.axis2.client.ServiceClient.<init>(ServiceClient.java:251)
at com.google.code.magja.soap.MagentoSoapClient.login(MagentoSoapClient.java:138)
at com.google.code.magja.soap.MagentoSoapClient.<init>(MagentoSoapClient.java:53)
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.axis2.transport.http.CommonsHTTPTransportSender
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.apache.axis2.util.Loader.loadClass(Loader.java:261)
at org.apache.axis2.deployment.AxisConfigBuilder.processTransportSenders(AxisConfigBuilder.java:683)
Added JAR Files

I just ran into the same issue and added these dependencies to my maven pom
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-transport-http</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-transport-local</artifactId>
<version>1.6.2</version>
</dependency>
Hope this helps.

Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/JsonParseException

I am using Twilio to send SMS messages from my web app and I have almost completed the integration. It's sending the SMS, but an error is being generated and I don't understand why. I have included a JSON dependency from this json and this file link but I'm still getting an error. Which other jar do I need to include?
Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/JsonParseException
at com.twilio.sdk.TwilioRestResponse.getParser(TwilioRestResponse.java:225)
at com.twilio.sdk.TwilioRestResponse.toMap(TwilioRestResponse.java:243)
at com.twilio.sdk.resource.list.SmsList.create(SmsList.java:70)
at Example.main(Example.java:25)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.JsonParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 4 more
screenshot showing json added
Update this is the new screesnshot

You have missed jackson-core-asl.jar 1.9.13 in you project. You can make use of this Maven POM dependency
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.13</version>
</dependency>
Hope this solves your problem. Check this link for more information. Direct link for download
Class is present in the jar Screen below
Remove your java-json.jar and add the jackson-core-asl.jar