Invalid class exception in apache spark - java

I am trying to run a spark job using spark-submit. When I run it in eclipse the job runs without any issue. When I copy the same jar file to a remote machine and run the job there I get the below issue
17/08/09 10:19:15 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, ip-10-50-70-180.ec2.internal): java.io.InvalidClassException: org.apache.spark.executor.TaskMetrics; local class incompatible: stream classdesc serialVersionUID = -2231953621568687904, local class serialVersionUID = -6966587383730940799
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1829)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1986)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I saw some other links in SO and tried the below
Changed the version of spark jars to 2.11 from 2.10 which I was using before. Now the dependencies in pom look like this
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.2</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_2.11</artifactId>
<version>2.0.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.0.2</version>
<scope>provided</scope>
</dependency>
I also checked that the version 2.11-2.0.2 exists in the jars folder of spark as suggested in a few links.
I also added provided in the dependencies as suggested in few links
None of the above helped. Any help would be of great help as I am stuck in this issue. Thanks in advance. Cheers
Edit 1: This is the spark-submit command
spark-submit --deploy-mode cluster --class "com.abc.ingestion.GenericDeviceIngestionSpark" /home/hadoop/sathiya/spark_driven_ingestion-0.0.1-SNAPSHOT-jar-with-dependencies.jar "s3n://input-bucket/input-file.csv" "SIT" "accessToken" "UNKNOWN" "bundleId" "[{"idType":"D_ID","idOrder":1,"isPrimary":true},{"idType":"HASH_DEVICE_ID","idOrder":2,"isPrimary":false}]"
Edit 2:
I also tried adding the variable serialVersionUID = -2231953621568687904L; to the related class but that didn't resolve the issue

I finally resolved the issue. I commented out all the dependencies and uncommented them one at a time. First I uncommented spark_core dependency and the issue got resolved. I uncommented another dependency in my project which again brought back the issue. Then on investigation I found that the second dependency was in turn having dependency of a different version(2.10) of spark_core which was causing the issue. I added exclusion to the dependency as below:
<dependency>
<groupId>com.data.utils</groupId>
<artifactId>data-utils</artifactId>
<version>1.0-SNAPSHOT</version>
<exclusions>
<exclusion>
<groupId>javax.ws.rs</groupId>
<artifactId>javax.ws.rs-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
</exclusion>
</exclusions>
</dependency>
This resolved the issue. Just in case someone gets stuck on this issue. Thanks #JosePraveen for your valuable comment which gave me the hint.

We see this issue when slightly different jar versions were being used on the Spark master and 1 or more of the Spark slaves.
I was facing this issue because I had only copied my jar to the master node. Once I copied the jar to all the slave nodes, my application started working just fine.

Related

Couldn't understand managed version under Maven dependency hierarchy

In my pom.xml, I have a SikuliX Jar which has a transitive dependency on jna-platform.
As seen in below image, version 4.5.2 has overrided version 5.4.0.
But i dont understand, how this version is overrided as i have not specified any dependency for jna-platform. I had also verified that no any there dependency is fetching this jar.
Please help me understand why this is happening. Any detailed document is well appreciated.
Related dependencies:-
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixapi</artifactId>
<version>2.0.4</version>
<exclusions>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
</exclusion>
</exclusions>
</dependency>
Thanks
Since you were using spring boot, as suggested here (there's also the reason of this behaviour):
java.lang.NoClassDefFoundError: com/sun/jna/platform/win32/SspiUtil$ManagedSecBufferDesc #882
you can change your order of dependencies, or specify the exact version, like this:
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>5.4.0</version>
</dependency>
or add this property:
<jna.version>5.4.0</jna.version>

JAVA - com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl not found

I'm working on an application that consumes a web service using SOAP requests.
Sometimes I get this error:
filters.LoggerFilter:92 - org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.Error: javax.xml.soap.SOAPException: Unable to create SAAJ meta-factoryProvider com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl not found
The weird part is that I get this error randomly, but I can't seem to figure out the cause.
I even added a new dependency, but it doesn't seem to correct the issue:
<dependency>
<groupId>com.sun.xml.messaging.saaj</groupId>
<artifactId>saaj-impl</artifactId>
<version>1.3</version>
</dependency>
I just had the same problem while using Java 11 to create an application that consumes SOAP-requests.
I added the new dependency and it worked fine for me.
<dependency>
<groupId>com.sun.xml.messaging.saaj</groupId>
<artifactId>saaj-impl</artifactId>
<version>1.5.1</version>
</dependency>
For me, I was using Java 13 and the following worked for me(add these in the pom.xml)
<dependency>
<groupId>org.glassfish.metro</groupId>
<artifactId>webservices-rt</artifactId>
<version>2.4.4</version>
</dependency>
<dependency>
<groupId>org.glassfish.metro</groupId>
<artifactId>webservices-api</artifactId>
<version>2.4.4</version>
</dependency>
For those who face this issue in intellij IDEA using Spring Boot under Java SDK 9+, you have to include explicitly --add-modules java.se.ee in VM parameters (edit configurations -> VM options). This answer may help to resolve other importing issues related to new Java Modules
I had the same problem.
For me, adding saaj-impl was not enough to get rid of the exception
<dependency>
<groupId>com.sun.xml.messaging.saaj</groupId>
<artifactId>saaj-impl</artifactId>
<version>1.5.2</version>
</dependency>
I had to also add saaj-api which fixed it finally:
<dependency>
<groupId>javax.xml.soap</groupId>
<artifactId>saaj-api</artifactId>
<version>1.3.5</version>
</dependency>
Since I realised that my application was using saaj-api 1.3.4 after checking with command, upgrading to 1.3.5 helped
mvn dependency:tree -Dverbose
Change your project sdk as Java 1.8
If you are using import javax.xml.ws library it could be confusing com.sun.xml.messaging.saaj dependency in Java 11. Clearing saaj dependency then using Java8 may be a solution in this issue
With open JDK 17 and spring boot 3.0.1, I was getting the same issue. It got resolved by adding these dependencies in pom.xml
<dependency>
<groupId>org.glassfish.metro</groupId>
<artifactId>webservices-rt</artifactId>
<version>2.4.4</version>
</dependency>
<dependency>
<groupId>org.glassfish.metro</groupId>
<artifactId>webservices-api</artifactId>
<version>2.4.4</version>
</dependency>

java.lang.NoSuchMethodError: com.sun.mail.util.TraceInputStream

I'm trying to send an email through Java Mail API and it works fine on my laptop. When I'm doing exactly the same in Heroku, I'm getting this:
java.lang.NoSuchMethodError: com.sun.mail.util.TraceInputStream.(Ljava/io/InputStream;Lcom/sun/mail/util/MailLogger;)V
at com.sun.mail.smtp.SMTPTransport.initStreams(SMTPTransport.java:2014)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1936)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:654)
at javax.mail.Service.connect(Service.java:291)
at ...
Here is what I have in pom.xml:
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mailapi</artifactId>
<version>1.4.3</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
<version>1.5.3</version>
<scope>runtime</scope>
</dependency>
I guess there is another version of Java Mail API inside Heroku JDK, which doesn't have this constructor... How can this be fixed?
By default, Java apps running on latest Heroku stack use OpenJDK 8.
Your problem does not seems related to the actual JVM implementation but rather due to the missing smtp-1.5.1.jar in classpath . To be sure to load correctly TraceInputStream try this :
java.net.URL classUrl = this.getClass().getResource("com.sun.mail.util.TraceInputStream");
out.println(classUrl.getFile());
I also faced the same issue. It was because I don't have the class TraceInputStream inside the library. I simply downgrade libraries to 1.4.4 and it worked.
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>smtp</artifactId>
<version>1.4.4</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>mailapi</artifactId>
<version>1.4.4</version>
</dependency>
You've mixed different versions of the API and implementation; don't do that. For that matter, you only need the com.sun.mail:javax.mail dependency. If Heroku isn't providing it in the runtime environment, you'll need to package it in your application. Make sure the JavaMail jar file is ending up in the WEB-INF/lib directory of your application.
Getting reference from #Osanda Deshan. Adding below gradle dependencies Worked for me.
implementation 'javax.mail:javax.mail-api:1.4.4'
implementation 'javax.mail:mail:1.4.4'

Missing plugins found while loading a transformation on Kettle

I receive this error whenever I run my extraction from the command line, not in the Spoon UI.
Missing plugins found while loading a transformation
Step : MongoDbInput
at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1200)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:643)
at org.pentaho.di.job.Job.execute(Job.java:714)
at org.pentaho.di.job.Job.execute(Job.java:856)
... 4 more
Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException:
Missing plugins found while loading a transformation
My maven dependencies is as follows.
<dependency>
<groupId>rhino</groupId>
<artifactId>js</artifactId>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-core</artifactId>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-engine</artifactId>
</dependency>
<dependency>
<groupId>pentaho-library</groupId>
<artifactId>libbase</artifactId>
<version>5.1.0.0-752</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>pentaho-big-data-plugin</artifactId>
<version>5.1.0.0-751</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>pentaho-mongodb-plugin</artifactId>
<version>5.1.0.0-751</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>2.11.1</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>metastore</artifactId>
<version>5.1.0.0-751</version>
</dependency>
<dependency>
<groupId>pentaho-library</groupId>
<artifactId>libformula</artifactId>
</dependency>
<dependency>
<groupId>simple-jndi</groupId>
<artifactId>simple-jndi</artifactId>
</dependency>
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-compiler</artifactId>
</dependency>
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-core</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
</dependency>
</dependencies>
Am I missing anything?
before init kettleEnviroment
KettleEnviroment.init();
you should add this code like
StepPluginType.getInstance().getPluginFolders().add(new PluginFolder("your plugins path", false, true));
Having the MongoDB plugin as a dependency only works for compilation. When PDI runs it looks for a plugins/ folder under the current directory, and will load all plugins from there. You can override the location by setting the KETTLE_PLUGIN_BASE_FOLDERS system property. Either way, you'll want a plugins/ folder and to unzip the MongoDB plugin package (not the JAR) into plugins/. That should put a folder called pentaho-mongodb-plugin under plugins/, and if PDI is pointing at that plugins/ folder, it should load and use the plugin successfully at runtime.
You must run Kitchen from Kettle installation directory.
As written here:
http://wiki.pentaho.com/display/EAI/Kitchen+User+Documentation
Please make sure that you are positioned in the Kettle directory
before running the samples below. If you put these scripts into a
batch file or shell script, simply do a change directory to the
installation directory
I had the same error message after an update from version 7 to 8.3. In my case, it was the step 'Abort' and 'Attend' that caused the error. In version 8 these two steps aren't part of the library "kettle-engine-8.3.0.0-371.jar" anymore, they are now part of the library "pdi-core-plugins-impl-8.3.0.0-371.jar" in a new plugin. I had to add the plugin in the maven-dependencies:
<dependency>
<groupId>org.pentaho.di.plugins</groupId>
<artifactId>pdi-core-plugins-impl</artifactId>
<version>8.3.0.0-371</version>
</dependency>
But that was still not everything, I still got the error message. The reason: "To use non-native plugins with an embedded Pentaho Server, you must configure the server to find where the plugins reside" (https://help.hitachivantara.com/Documentation/Pentaho/8.0/Developer_Center/PDI/Embed).
So above the line KettleEnvironment.init() in my java-code, I added this one:
System.setProperty("KETTLE_PLUGIN_CLASSES", "org.pentaho.di.trans.steps.abort.AbortMeta,org.pentaho.di.trans.steps.append.AppendMeta");
KettleEnvironment.init();
This way worked for me. There's also a suggestion using a plugin-folder (see the link or the other answers) and working with KETTLE_PLUGIN_BASE_FOLDER instead.
Unfortunately, the error message doesn't mention which plugin is missing. I deleted and added them (in my local Spoon application) until I found exactly the one that was missing.
Just edit spoon.sh and add
OPT="$OPT -DKETTLE_PLUGIN_BASE_FOLDERS=$BASEDIR/plugins"
right after the line where the OPT variable is set.
This is actually the solution from mattyb. Editing spoon.sh will fix the problem for good.
Another solution is to create a symbolic link in your ${KETTLE_HOME}/.kettle directory which points to the plugins folder.

Cant run feature in Cucumber

Im having issues running a feature in Cucumber, the feature is very basic as it's from a tutorial.
It is not defined and is as follows:
Feature: Proof that my concept works
Scenario: My first test
Given this is my first step
When this is my second step
Then this is my final step
My Cucumber runner class is as follows:
package cucumber;
import org.junit.runner.RunWith;
import cucumber.api.junit.Cucumber;
#RunWith(Cucumber.class)
#Cucumber.Options(
format = {"pretty", "json:target/"},
features = {"src/cucumber/"}
)
public class CucumberRunner {
}
Also the external .jar files that I have in the project are as follows:
The exception that I'm getting is:
Exception in thread "main" cucumber.runtime.CucumberException: Failed
to instantiate public
cucumber.runtime.java.JavaBackend(cucumber.runtime.io.ResourceLoader)
with [cucumber.runtime.io.MultiLoader#75d837b6]
I've tried to look around online for the solution to this problem but have not had any luck.
I've also discussed with the OP of the tutorial and I'm still awaiting feedback but it has been a while.
I ran into a similar issue and got the same error as you did.
Firstly mention the path to the feature file
features = {"src/cucumber/myfile.feature"}
Anyway, that didn't cause the error.
To just run your Cucumber runner class, all the dependencies you need are
cucmber-junit
cucumber-java and
junit.
I had an additional cucumber-guice which was creating the problem and once I removed it, the error went away and runner was executed successfully.
From the link to the image you have mentioned it looks like you are not using cucumber-guice but still I would recommend you remove other unnecessary cucumber dependencies and try again.
1, I ran into this too few days ago, its simple just remove cucumber-Spring from the dependency.
2 If that doesn't work try updating cucumber-core, cucumber-junit, and cucumber-java all version 1.2.3
I believe the issue is that many of the cucumber add-ins, such as cucumber-testng, cucumber-spring, and (in my case) cucumber-guice, expect the corresponding module they link to be included as well. But apparently the cucumber experts decided not to include this dependency in their pom.xml files, so the problem doesn't manifest itself until runtime.
So (to answer Eugene S's question under LING's answer) if you want to actually use guice with cucumber, you need to also add guice itself as a dependency.
This worked for me, I hope it will work for you as well.
Update your Cucumber dependencies in pom.xml
i.e
cucumber-java (1.2.2)
cucumber-jvm (1.2.2)
cucumber-junit (1.2.2)
And update your Junit dependency as well. (4.11).
The only reason for this error is the version of all the cucumber libraries are not same. It should be like this:
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java8</artifactId>
<version>4.2.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-picocontainer -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-picocontainer</artifactId>
<version>4.2.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-testng -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-testng</artifactId>
<version>4.2.6</version>
<exclusions>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
</exclusions>
</dependency>
First Thing : We would request you to use Cucumber v >=4.0.0 as you are using pretty old dependency(v1.2.5) of Cucumber.
Key Point : We shall not mix direct & transitive dependencies specially their versions! Doing so can cause unpredictable outcome.
Solution: Please remove. cucumber-core, cucumber-java, cucumber-jvm-deps, gherkin and cucumber-html. They're transitive dependencies and will be provided by your direct dependencies.
You can add below set of cucumber minimal dependencies.
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>4.2.6</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-picocontainer</artifactId>
<version>4.2.6</version>
<scope>test</scope>
</dependency>
After spending a lot of time on this issue, most of the errors I was receiving were due to dependencies and dependencies versions mismatch. Adding these dependencies to pom.xml file worked for me:
<!-- https://mvnrepository.com/artifact/junit/junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-scala_2.11</artifactId>
<version>4.7.1</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-jvm -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-jvm</artifactId>
<version>4.8.1</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-junit -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-java8 -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java8</artifactId>
<version>4.8.1</version>
</dependency>

Categories

Resources