Develop Reference

Spark- Exception in thread java.lang.NoSuchMethodError

I am trying to score a model from pmml file using pmml4s library. Every time I submit the job in Spark I get the following error:
20/05/13 23:30:10 ERROR SparkSubmit: org.apache.spark.sql.types.StructType.names().
[Ljava/lang/String;
java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names().
[Ljava/lang/String;
at org.pmml4s.spark.ScoreModel.transform(ScoreModel.scala:56)
at com.aexp.JavaPMML.main(JavaPMML.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
Following is my code sample:
ScoreModel model = ScoreModel.fromFile(args[0]);
SparkConf conf = new SparkConf();
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
Dataset<?> df = spark.read().format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load(args[1]);
Dataset<?> scoreDf = model.transform(df);
Following is the pom file that I am using:
<dependencies>
<dependency>
<groupId>org.pmml4s</groupId>
<artifactId>pmml4s-spark_2.11</artifactId>
<version>0.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.3.2</version>
</dependency>
</dependencies>
I have edited my pom file and made the spark version similar still I face the same issue. When I am using Scala, I am facing the same problem. Is there any dependency that I am missing?

Try to use same version of spark libraries. If spark versions are not matching we will be getting NoSuchMethodError issue in many places as those methods might have modified or removed in latest versions.

The error is caused by the PMML4S-Spark used the method names of StructType, which is introduced since Spark 2.4. Now it has been fixed in the latest PMML4S-Spark 0.9.5. Please, update your pom file to use the new version:
<dependency>
<groupId>org.pmml4s</groupId>
<artifactId>pmml4s-spark_2.11</artifactId>
<version>0.9.5</version>
</dependency>

Spark Cassandra connector base simple program getting spark logging error

I was trying a very basic hello world program for java+spark+cassandra.
Initially I had some mixed versions of libraries which caused the NoSuchMethodError (#5). When I got the versions right, I get the noclassdefound error for spark logging (#4). This is from the cassandra connector code. I have built it from the b2.3 branch of github which is only couple of commits behind the master(using sbt).
All solutions for spark logging issue point to moving to older versions. This is not a practical solution for us as we need to figure this out for future development.
Wonder why the latest stable build of cassandra connector refers to spark logging which is no longer available now??.
Any help is appreciated.
Spark version: 2.3.0
Cassandra: 3.9.0
The relevant code snippet is pasted below.
#1 SparkConf sparkConf = new SparkConf().setAppName("appname")
.setMaster("local");
#2 sparkConf.set("spark.cassandra.connection.host", "127.0.0.1");
#3 JavaSparkContext ctx = new JavaSparkContext(sparkConf);
#4 CassandraConnector connector = CassandraConnector.apply(ctx.getConf()); <<<< org/apache/spark/logging noclassdeffound error
#5 try (Session session = connector.openSession()) { <<< nosuchmethoderror: scala.runtime.objectref.zero()lscala/runtime/objectref
The POM is below
http://maven.apache.org/xsd/maven-4.0.0.xsd">
4.0.0
com.mygroup
apache-spark
1.0-SNAPSHOT
jar
apache-spark
http://maven.apache.org
<dependencies>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.11</artifactId>
<version>1.6.0-M1</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.5.0</version>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.11.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<org.apache.spark.spark-core.version>2.2.1</org.apache.spark.spark-core.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>

First thing to fix is
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.11</artifactId>
<version>1.6.0-M1</version>
</dependency>
Which does not match any of your other build versions. The Java module was merged into the main artifact. You also shouldn't include the java driver module on it's own since this will most likely have issue with guava inclusions.
Take a look at
https://github.com/datastax/SparkBuildExamples/blob/master/scala/maven/oss/pom.xml for example pom files.

Vertx2: org.vertx.java.platform.impl.cli.Starter

I'm writing a "Hello-World" application in Vertx2.
I have the following dependencies in pom.xml:
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-core</artifactId>
<version>2.1.6</version>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-platform</artifactId>
<version>2.0.0-beta1</version>
</dependency>
and run it as IntelliJ application with the following configuration:
I get the error:
Exception in thread "main" java.lang.NoSuchMethodError:
org.vertx.java.core.impl.VertxInternal.getOrAssignContext()Lorg/vertx/java/core/impl/DefaultContext;
Do you know what is the problem here?
Edit:
This project is for Vertx version 2 (not version 3). This is why the solution in this IntelliJ and Vertx: How to run org.vertx.java.deploy.impl.cli.Starter ? doesn't resolve the issue.

I found a solution to this problem. I just changed the version numbers for vertx-core and vertx-platform, and now it works.
The new dependencies in pom.xml are:
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-core</artifactId>
<version>2.1RC3</version>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-platform</artifactId>
<version>2.1.5</version>
</dependency>

Spark on HBase exception class not found (JAVA)

Im trying to communicate with hbase using spark. I´m using this code below:
SparkConf sparkConf = new SparkConf().setAppName("HBaseRead");
JavaSparkContext jsc = new JavaSparkContext(sparkConf);
Configuration conf = HBaseConfiguration.create();
conf.addResource(new Path("/etc/hbase/conf/core-site.xml"));
conf.addResource(new Path("/etc/hbase/conf/hbase-site.xml"));
JavaHBaseContext hbaseContext = new JavaHBaseContext(jsc, conf);
Scan scan = new Scan();
scan.setCaching(100);
JavaRDD<Tuple2<ImmutableBytesWritable, Result>> hbaseRdd = hbaseContext.hbaseRDD(TableName.valueOf("climate"), scan);
System.out.println("Number of Records found : " + hbaseRdd.count());
If I execute this, I get the following error:
Exception in thread "dag-scheduler-event-loop" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/regionserver/StoreFileWriter
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
...
I did not find any solution via google. Has anyone an idea?
--------edit--------
I´m using maven. My Pom looks like:
<dependencies>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.sharegov</groupId>
<artifactId>mjson</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-xml_2.10</artifactId>
<version>0.3.5</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-spark</artifactId>
<version>2.0.0-SNAPSHOT</version>
</dependency>
</dependencies>
Im building my application with dependencies using the maven-assembly-plugin

You are getting the NoClassDefFoundError, because spark is not able to find hbase jars in the classpath, you need to supply the required jars to spark-submit explicitly using --jars parameter while launching job:
${SPARK_HOME}/bin/spark-submit \
--jars ${..add hbase jars comma separated...}
--class ....
.........

HBase Java API for HBaseTable creation shows Error

I am a beginner in Hadoop and Hbase. I am getting error when I try to create an HBase Table using Java API. My project is a maven Project. I am using cloudera machine for running my application. Code sample and error and POM.xml is given below
public static void main(String[] args) throws IOException {
// Instantiating configuration class
Configuration con = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(con);
// Instantiating table descriptor class
HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf("empdata"));
// Adding column families to table descriptor
tableDescriptor.addFamily(new HColumnDescriptor("personal"));
tableDescriptor.addFamily(new HColumnDescriptor("official"));
// Execute the table through admin
admin.createTable(tableDescriptor);
System.out.println(" Table created ");
}
Command I am giving like this :
-bash-4.1$ hadoop jar HBaseTableCreation-1.0-SNAPSHOT.jar com.HBaseTable
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at com.Feathersoft.HBaseTable.main(HBaseTable.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 6 more
-bash-4.1$
POM.xml dependancies
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>0.95.0</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shell</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-thrift</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.4.0a</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>11.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-cdh4.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
How can I set classpath in Cloudera Machine. I am not so much familiar in Linux.
Can any one help, It would be great ....