Develop Reference

Why does spark-submit fail with “Error executing Jupyter command”?

When trying to run Spark locally on my Mac (which used to work) ...
/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java \
-cp /usr/local/Cellar/apache-spark/2.4.0/libexec/conf/:/usr/local/Cellar/apache-spark/2.4.0/libexec/jars/* \
-Xmx1g org.apache.spark.deploy.SparkSubmit \
--packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.0 \
/Users/crump/main.py
I'm now getting the following error:
Error executing Jupyter command '/Users/crump/main.py': [Errno 2] No such file or directory
The file is there. Since I know this used to work, I must have installed something recently that changed a library, sdk, etc.

Ok, I found the answer finally: PYSPARK_DRIVER_PYTHON=jupyter in my environment. I set this up to launch Jupyter/Spark notebooks with just the pyspark command, but it causes spark-submit to fail.
The solution is set the variable to use python, not jupyter: PYSPARK_DRIVER_PYTHON=python.

Maven specify settings file location via MAVEN_OPTS

I need to use maven with a settings file in a specific location, normally you can give MAVEN_OPTS env variable but they are passed to JVM so the following will yield:
$ MAVEN_OPTS="-s /settings.xml"
$ mvn clean
Unrecognized option: -s
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I searched a lot but found two keys, org.apache.maven.user-settings and org.apache.maven.global-settings which is explained here but it seemed it was working with Maven 2 only. Aliasing mvn to mvn -s /settings.xml would probably work but I do not like it.

From the mvn shell script:
# -----------------------------------------------------------------------------
# Apache Maven Startup Script
#
# Environment Variable Prerequisites
#
# JAVA_HOME Must point at your Java Development Kit installation.
# MAVEN_OPTS (Optional) Java runtime options used when Maven is executed.
# MAVEN_SKIP_RC (Optional) Flag to disable loading of mavenrc files.
# -----------------------------------------------------------------------------
so MAVEN_OPTS contains JVM arguments, not Maven arguments (which is consistent with the error message indicating the JVM doesn't like your arguments).
The actual invocation is
exec "$JAVACMD" \
$MAVEN_OPTS \
$MAVEN_DEBUG_OPTS \
-classpath "${CLASSWORLDS_JAR}" \
"-Dclassworlds.conf=${MAVEN_HOME}/bin/m2.conf" \
"-Dmaven.home=${MAVEN_HOME}" \
"-Dlibrary.jansi.path=${MAVEN_HOME}/lib/jansi-native" \
"-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \
${CLASSWORLDS_LAUNCHER} "$#"
so there is nowhere to put it. I would therefore suggest that you write your own ? mvn script which in turn calls the real maven command with the arguments you like (in my experience scripts are more robust than aliases). Additionally I have recently found myself that the Java versions later than 8 have ... interesting issues... so I really need to have mvn8, mvn11 (and perhaps more) commands anyway.
Another approach that I only started using recently is the Maven wrapper (https://github.com/takari/maven-wrapper) where a ./mvnw command is placed in your project which then downloads Maven when needed. This is very useful. To get started use
mvn -N io.takari:maven:wrapper
after which ./mvnw should be directly usable instead of mvn. The interesting part here is that the generated Maven command looks like
exec "$JAVACMD" \
$MAVEN_OPTS \
-classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \
"-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \
${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$#"
and MAVEN_CONFIG is not set earlier in the script. So for mvnw you can set MAVEN_CONFIG to your "-s /settings.xml" string.

Maven 4
The MAVEN_ARGS environment variable is supported and can be used.
Maven 3
There was a feature request MNG-5824: Support MAVEN_ARGS environment variable as a way of supplying default command line arguments. This was closed unimplemented with a suggestion to use the .mvn/maven.config in project directory

OpenJDK JDK11 not having JMC- Java Mission Controller- FlightRecorder

I was hoping JMC would be available with OpenJDK, JDK11 binaries as this has been opensourced from Java 11 by oracle, but could not locate this in Oracle and AdoptOpenJDK Java-11 binaries under bin folder.
I have also tried this https://jdk.java.net/jmc/ as some article said its being releases separately.
Does anyone know how to get JMC for OpenJDK-11.

I am editing this answer since builds are now available, and have been available, from multiple vendors for quite some time. The list is available in the readme for the JMC GitHub repo:
https://github.com/openjdk/jmc
Don't forget to give the project a star if you like it! :)
Here is the original answer:
Normally the builds will be available here:
https://jdk.java.net/jmc/
See http://hirt.se/blog/?p=1007 for more information on the new delivery format.
The builds have been (temporarily) pulled because a switch from the old javax.mail coordinates to the new coordinates at jakarta-ee has not yet gotten the proper third-party approval. A new build, with plenty of fixes and with all the approvals properly in place (or a revert of the change), should be along within the next few weeks.
Up until then it is possible (also not hard) to build JMC 7, by pulling the official JMC repo from here:
https://hg.openjdk.java.net/jmc/jmc7/
You can also build and pull the mainline mirror from the inofficial GitHub repo:
https://github.com/JDKMissionControl/jmc
For more information on building JMC, see:
http://hirt.se/blog/?p=947 (or simply read the README.md in the repository root)
Good luck!

Since Java11, JMC is not part of the JDK any more. It is a separate project, as you already noticed.
https://www.oracle.com/technetwork/java/javase/11-relnote-issues-5012449.html

Azul provides free, unsupported builds of Zulu Mission Control available for download at http://azul.com/products/zulu-mission-control
Builds are available for clients running Windows, Linux, and Mac OS X.
There are now binaries produced by Oracle as well:
https://jdk.java.net/jmc/

I believe they're still evaluating GitHub to moving to an independent repository with current progress being still under jmc7.
You can currently download standalone versions from Oracle's website for mission-control.

BellSoft provides Liberica Mission Control: https://bell-sw.com/pages/lmc/
As per their documentation, it's free to use in production environments, and there is a commercial support included as part of Support Subscription for Liberica JDK.

Update
JMC 8 available when compiling it from source. Here is a single command for linux users to build your own copy of jmc
mkdir ~/jmcToDelete && \
cd ~/jmcToDelete && \
git clone https://github.com/openjdk/jmc.git && \
cd jmc/releng/third-party && \
mvn p2:site && \
runJetty="mvn jetty:run" && \
bash -c "$runJetty &" && \
cd ~/jmcToDelete/jmc/core && \
mvn clean install && \
cd ~/jmcToDelete/jmc && \
mvn package -Dmaven.test.skip=true && \
kill $(jps | grep Launcher | awk '{print $1}') && \
sudo mkdir -p /opt/java/jmc && \
sudo tar xzf $(find ~/jmcToDelete/jmc/target -name '*.jmc-linux*') -C /opt/java/jmc && \
sudo ln -s /opt/java/jmc/jmc /usr/local/bin/jmc && \
rm -rf ~/jmcToDelete
typing jmc in a terminal should start it.
Outdated
As stated by Hirt you can compile it from http://hg.openjdk.java.net/jmc
The jmc available in jdk8 (v5.5) requires the special flags -XX:+UnlockCommercialFeatures -XX:+FlightRecorder to be present in the JVM process and it will not retrieve "flight records" if they are not there, so you can only use it with java1.8. If you would like to add those flags on openjdk-11 it will fail with Unrecognized VM option 'UnlockCommercialFeatures' meaning that you don't need them as they are enabled by default ( FlightRecorder ).
I compiled jmc-7.1.0 without issues by downloading the gz archive from http://hg.openjdk.java.net/jmc/jmc/
Follow the steps present in the README file:
Make sure the compilation is done with jdk1.8
In one terminal :
cd releng/third-party
mvn p2:site
mvn jetty:run
And in the second terminal:
cd core
mvn clean install
cd ..
mvn package
On completion you should have all your artefacts in the target folder.
The default jmc start script has a lot of flags present and it will not start with all of them, hence you can start the intended jar using
java -jar ./jmc-[...]/target/products/jmc/plugins/org.eclipse.equinox.launcher_[...].jar

jni.h No such file or directory during cmake linux?

I've been trying to make opencv for linux, I used the cmake parameters:
cmake -D CMAKE_BUILD_TYPE=RELEASE -D WITH_OPENCL=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_SHARED_LIBS=OFF -D JAVA_INCLUDE_PATH=$JAVA_HOME/include -D JAVA_AWT_LIBRARY=$JAVA_HOME/jre/lib/amd64/libawt.so -D JAVA_JVM_LIBRARY=$JAVA_HOME/jre/lib/arm/server/libjvm.so -D CMAKE_INSTALL_PREFIX=/usr/local
and it generated the files fine.
It was then into around the 81% when it was trting to generate the opencv-jar it opped up with
/home/pi/Desktop/opencv-3.1.0/modules/java/generator/src/cpp/common.h:8:17 fatal error jni.h No such file or directory
So I'm not sure what to be doing now with it. openjdk is installed properly too
Edit: I tried using the -I flag, by doing the command
make -I/usr/lib/jvm/java-8-openjdk-armhf/includes
to no avail

the -I flag on make(1) command only affects the files included in the makefile by the .include directive, not the directories searched for by the compiler. For that purpose, just pass the -I flag to each compilation. One way to do this is
$ make CFLAGS="-I/usr/lib/jvm/java-8-openjdk-armhf/includes"
you can also pass the CFLAGS from the environment, as in
$ export CFLAGS=\""-I/usr/lib/..."\" # escaped double quotes make them to be included in the string.
$ make

Please check: https://stackoverflow.com/a/67154438/1290868
FindJNI
find_package(JNI)
if (JNI_FOUND)
message (STATUS "JNI_INCLUDE_DIRS=${JNI_INCLUDE_DIRS}")
message (STATUS "JNI_LIBRARIES=${JNI_LIBRARIES}")
endif()

Installing JDK on Linux

I’m trying to install JDK on Centos, but I’m getting an error:
[root#www opt]# rpm -ivh /opt/jdk-7u51-linux-x64.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
error: unpacking of archive failed on file /usr/java/jdk1.7.0_51/db/lib/derbyLocale_zh_CN.jar;533ab42a: cpio: read
Where can be problem?
Thanks

Can you please check the md5sum of the downloaded file against these values here: http://www.oracle.com/technetwork/java/javase/downloads/java-se-binaries-checksum-1956892.html ?
If the md5sum does not check, it must be a broken download.

A short answer: try downloading and installing again.
A longer answer:
Even though you are getting this, the jdk is getting installed. Sort of. This seems to be an error in the packaging but it is not fatal. Here is the output from my machine. In the example I cat'd the release file so you can see what CentOS I'm using and did an md5sum on the rpm I'm using. You'll notice that it doesn't match the md5sum from the Oracle site link (provided above by Kristof).
Next, I did an ls on the target directory for the jdk /usr/java/ so you can see that it is empty. Then ran the rpm install and did another ls to show that some files are there.
Of course, this defeats the purpose of using rpm, because the install information is not going to go into the rpm database. If I query the database it doesn't come up and it's missing the links that usually get installed pointing to 'latest' and 'default'
So I took a guess that maybe there is a mirror that might have a bad rpm and went to the Oracle site and downloaded the file again. I gave it a different name so that I can compare the two files. This time it was successful.
In the screen shot below you can see the two files. The newest download being the one I called jdk-7u51-linux-x86_64.rpm. You can see the md5sum shows the files are different, but on the new file, the md5 matches the Oracle site. I removed the directory that was created by the failed install and then ran the rpm -ivh command and it completed successfully. The ls on the /usr/java directory shows all the correct stuff and querying the rpm database shows it was installed. Hope this helps!

You can do it with simple script which you can find here
It's for Centos rpm, but you can edit script and change rpm to tar.gz for example
Also you can change version or env in variables bellow
BASE_URL="technetwork/java/javase/downloads"
BASE_URL_OUTPUT="$(curl -s -L0 http://www.oracle.com/${BASE_URL}/)"
JAVA_ENVIRONMENT="JDK"
JAVA_BASE_VERSION=8
DOWNLOAD_SITE="$(echo $BASE_URL_OUTPUT | grep -m 1 -io "${JAVA_ENVIRONMENT}${JAVA_BASE_VERSION}-downloads-[0-9]*.html" -- | tail -1)"
echo "DOWNLOAD_SITE="$DOWNLOAD_SITE
DOWNLOAD_LINK_OUTPUT="$(curl -s -L -j -H "Cookie: oraclelicense=accept-securebackup-cookie" http://www.oracle.com/${BASE_URL}/${DOWNLOAD_SITE} | grep -io "filepath.*${JAVA_ENVIRONMENT}-[${JAVA_BASE_VERSION}].*linux[-_]x64[._].*\(rpm\)" -- | cut -d '"' -f 3 | tail -1)"
echo "DOWNLOAD_LINK_OUTPUT="$DOWNLOAD_LINK_OUTPUT
curl -L -o java_rpm_packet.rpm -b "oraclelicense=a" $DOWNLOAD_LINK_OUTPUT
and install with
yum localinstall -y java_rpm_packet.rpm