Java exception when running pyspark on Macbook [duplicate] - java

Description:
I installed spark on my MacBook following using Homebrew. I followed the instruction process from: https://www.tutorialkart.com/apache-spark/how-to-install-spark-on-mac-os/.
The step by step process included installing Java, followed by Scala and then Spark. Java and Scala got installed successful. Spark got installed successfully as well.
When I tried to verify spark installation using below input command, I ran into an error.
Input command: spark-shell
Expected Behavior: Expect Spark to start on terminal
Actual Behavior: I get this below error:
Unable to find any JVMs matching version "1.8".
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/2.4.5/libexec/jars/spark-unsafe_2.11-2.4.5.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:348)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:348)
at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
at scala.Option.map(Option.scala:146)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:355)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3756)
at java.base/java.lang.String.substring(String.java:1902)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)
What I tried:
I tried to change JAVA_HOME using below commands:
export JAVA_HOME=/usr/local/opt/java
Previous JAVA_HOME path was /opt/anaconda3. I can see that JAVA_HOME got changed to usr/local/opt/java.
I am still getting the error. Appreciate your answers/feedback. Thanks!!!

Follow these steps for macOS
Step 1. install java 8 as Spark 2.2 onwards requires java8. See Spark Documentation! for details.
brew install openjdk#8
Then set update java path
export JAVA_HOME=/usr/local/opt/openjdk#8/libexec/openjdk.jdk/Contents/Home

I was searching across the web and saw instructions to install pyspark.
I ran this command pip install pyspark on Terminal
After I installed pyspark, both spark and pyspark are running.
Not sure what happened, but I am able to run spark now.
Thanks #Elliott for interacting and giving some directions!

Related

'RunTime:loadLib: libopencv_java430.so (failed) probably dependent libs missing' error while running sikuli test script in Red had 7.6

Used maven project:
My Automation sikuli script:
package Sikuli1.DesktopProject;
Error logs:
OpenJDK 64-Bit Server VM warning: You have loaded library /home/rajatesh/.Sikulix/SikulixLibs/libopencv_java430.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
[error] RunTime:loadLib: libopencv_java430.so (failed) probably dependent libs missing:
/home/rajatesh/.Sikulix/SikulixLibs/libopencv_java430.so: /home/rajatesh/.Sikulix/SikulixLibs/libopencv_java430.so: file too short
[error] Save your work, correct the problem and restart the IDE!
[error] see: https://github.com/RaiMan/SikuliX1/wiki/macOS-Linux:-Support-Libraries-for-OpenCV-4
TERMINATING: problem with native library: libopencv_java430.so
Exception in thread "main" java.lang.ExceptionInInitializerError at org.sikuli.script.Pattern.(Pattern.java:128) at Sikuli1.DesktopProject.SikuliProject.main(SikuliProject.java:11)
Caused by: org.sikuli.script.SikuliXception: fatal: problem with native library: libopencv_java430.so at org.sikuli.script.support.RunTime.terminate(RunTime.java:1140) at org.sikuli.script.support.RunTime.libsLoad(RunTime.java:1285) at org.sikuli.script.support.RunTime.loadLibrary(RunTime.java:1434) at org.sikuli.script.Finder$Finder2.(Finder.java:538)
... 2 more
You need to install opencv Java library. Note that you are using Redhat while Sikuli 2 works out of the box with Ubuntu flavors. So you need install required libs by yourself.

sbt test error: java.lang.RuntimeException: /packages cannot be represented as URI

To start, I have reviewed the following response and am still struggling to resolve this issue:
Scala: packages cannot be represented as URI
I still see the following error output when I run sbt test:
error: error while loading String, class file '/modules/java.base/java/lang/String.class' is broken
(class java.lang.NullPointerException/Cannot invoke "scala.tools.nsc.Global$Run.typerPhase()" because the return value of "scala.tools.nsc.Global.currentRun()" is null)
[error] java.io.IOError: java.lang.RuntimeException: /packages cannot be represented as URI
[error] at java.base/jdk.internal.jrtfs.JrtPath.toUri(JrtPath.java:176)
[error] at scala.tools.nsc.classpath.JrtClassPath.asURLs(DirectoryClassPath.scala:204)
[error] at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$asURLs$1(AggregateClassPath.scala:55)
Here is what I'm seeing when I look and my build settings:
I checked my "/Library/ directory and I actually don't see a Java folder at all.
Appears to be a compatibility issue with sbt version and java version.
I ran into a similar error when trying to import a sbt project using sbt version 1.2.7 (in project/build.properties) using openjdk 15
I found that project specified to use if I updated project/build.properties to sbt.version=1.4.7 the error would go away.
I also found I could use older jdk version openjdk 11 with sbt 1.2.7.
Screenshot seems to indicate you are using older version of java 8 and I think latest sbt requires at java 11+, so in your case I suggest upgrading java and trying again.

Trying to open spark after installation and getting an error: Unable to find any JVMs matching version "1.8"

Description:
I installed spark on my MacBook following using Homebrew. I followed the instruction process from: https://www.tutorialkart.com/apache-spark/how-to-install-spark-on-mac-os/.
The step by step process included installing Java, followed by Scala and then Spark. Java and Scala got installed successful. Spark got installed successfully as well.
When I tried to verify spark installation using below input command, I ran into an error.
Input command: spark-shell
Expected Behavior: Expect Spark to start on terminal
Actual Behavior: I get this below error:
Unable to find any JVMs matching version "1.8".
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/2.4.5/libexec/jars/spark-unsafe_2.11-2.4.5.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:348)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:348)
at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
at scala.Option.map(Option.scala:146)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:355)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3756)
at java.base/java.lang.String.substring(String.java:1902)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)
What I tried:
I tried to change JAVA_HOME using below commands:
export JAVA_HOME=/usr/local/opt/java
Previous JAVA_HOME path was /opt/anaconda3. I can see that JAVA_HOME got changed to usr/local/opt/java.
I am still getting the error. Appreciate your answers/feedback. Thanks!!!
Follow these steps for macOS
Step 1. install java 8 as Spark 2.2 onwards requires java8. See Spark Documentation! for details.
brew install openjdk#8
Then set update java path
export JAVA_HOME=/usr/local/opt/openjdk#8/libexec/openjdk.jdk/Contents/Home
I was searching across the web and saw instructions to install pyspark.
I ran this command pip install pyspark on Terminal
After I installed pyspark, both spark and pyspark are running.
Not sure what happened, but I am able to run spark now.
Thanks #Elliott for interacting and giving some directions!

How to fix "Could not initialize class org.apache.ignite.IgniteJdbcThinDriver" error in Apache Ignite?

The Problem
I'm trying to connect to a Apache Ignite server with Apache Ignite built-in tool, SQLLine. I get the error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver
Background
I have Apache Ignite running in a container and CentOS7 running in another container. Both containers running in the same network (pinging works both ways). The tried connection is happening from CentOS7 to Apache Ignite.
Apache Ignite seems to be running fine with just the default configuration. In the CentOS7 container, I have installed Oracle JDK 12.0.1 and I have Apache Ignite 2.7.0 binary files in a folder. I have also set the IGNITE_HOME environmental variable.
Here, (https://apacheignite-sql.readme.io/docs/sqlline), it says I can connect to my cluster with just: ./sqlline.sh --verbose=true -u jdbc:ignite:thin://127.0.0.1/. However, this throws the previously mentioned error.
SQLLine should come with Ignite JDBC drivers. I have tried downloading them manually (https://apacheignite-sql.readme.io/docs/jdbc-driver#section-multiple-endpoints). When I downloaded the driver, which is said to be the ignite-core-{version}.jar, I put it in the same folder as sqlline.jar files.
Output
[root#bc72c4fbf47e bin]# ./sqlline.sh
sqlline version 1.3.0
sqlline> !connect jdbc:ignite:thin://172.19.0.2/
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ignite.internal.util.GridUnsafe$2 (file:/var/tmp/apache-ignite/libs/ignite-core-2.7.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.ignite.internal.util.GridUnsafe$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:415)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:555)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:547)
at java.sql/java.sql.DriverManager.getDrivers(DriverManager.java:449)
at java.sql/java.sql.DriverManager.getDrivers(DriverManager.java:426)
at sqlline.SqlLine.findRegisteredDriver(SqlLine.java:1568)
at sqlline.SqlLine.scanForDriver(SqlLine.java:1542)
at sqlline.Commands.connect(Commands.java:1074)
at sqlline.Commands.connect(Commands.java:1001)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:791)
at sqlline.SqlLine.begin(SqlLine.java:668)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
Conclusion
I should be able to connect to my Ignite server with !connect jdbc:ignite:thin://172.19.0.2/ command in sqlline.
This does not work, and throws Could not initialize class org.apache.ignite.IgniteJdbcThinDriver
IgniteJDBCThinDriver is installed/available.
Adding JVM argument
--add-opens java.base/java.nio=ALL-UNNAMED
solved the problem for me.
It is recommended to Java 8, as Apache Ignite 2.7.0 does not have full Java 12 support. Otherwise you can try tinkering with JVM options.
Thank you #alamar, that worked!
I uninstalled JDK12 that i had installed with RPM.
Check the package name:
rpm -qa | grep jdk.
Delete the package:
rpm -e jdk-12.0.1-12.0.1-ga.x86_64.
I'm working on a isolated system, so I downloaded and transferred JDK8.rpm from another machine.
Install JDK8:
rpm -ihv jdk-8u211-linux-x64.rpm.
Now when I run:
./sqlline.sh --verbose=true -u jdbc:ignite:thin://172.19.0.2,
I get:
issuing: !connect jdbc:ignite:thin://172.19.0.2/ '' '' org.apache.ignite.IgniteJdbcTh
Connecting to jdbc:ignite:thin://172.19.0.2/
Connected to: Apache Ignite (version 2.7.0#20181130-sha1:256ae401)
Driver: Apache Ignite Thin JDBC Driver (version 2.7.0#20181130-sha1:256ae401)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0.
I can now query my database.
I got this error in the following very strange situation:
A dependency of my maven java project had Apache-Ignite as a dependency. The moment I opened a connection to a SQLite database at an invalid path (e.g. ./result/r.sqlite where the folder ./result does not exist) this error occurred (using th driver org.xerial.sqlite-jdbc:v3.30.1.)So this call
Class.forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite:result.sqlite")
would result in this stacktrace:
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:555)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:674)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251)
at MyClass.myMethod(MyClass.java:154)
So the solution would obviously be to make sure the folder exsits where the sqlite database should be created in.

Ionic error on android emulate

I have installed java JRE but i keep on getting. Am using windows 10 for development
After running ionic run android or ionic emulate android i get this error:
Exception in thread "main" java.lang.RuntimeException: java.util.zip.ZipException: error in opening zip file
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:78)
at org.gradle.wrapper.Install.createDist(Install.java:47)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:48)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:215)
at java.util.zip.ZipFile.<init>(ZipFile.java:145)
at java.util.zip.ZipFile.<init>(ZipFile.java:159)
at org.gradle.wrapper.Install.unzip(Install.java:160)
at org.gradle.wrapper.Install.access$400(Install.java:29)
at org.gradle.wrapper.Install$1.call(Install.java:70)
at org.gradle.wrapper.Install$1.call(Install.java:47)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65)
... 3 more
I have also set JRE_HOME in the environment variables
Probably it is related to something missing or corrupted.
You can find a similar question here, hope it helps.
First of all you have to install
node.js
apache ANT
JDK & JRE
Android SDK & Eclipse
After installation, you have to set software paths into environmental- variable. Then fire your npm install -g cordova. Without this PhoneGap will not be created.

Categories

Resources