Develop Reference

Jenkins in Azure VM "This site can’t be reached" at 8080

I attempted to add the nvm-wrapper plugin (https://plugins.jenkins.io/nvm-wrapper/) to jenkins and reboot and am no longer able to reach the site (This site can’t be reached). It is deployed as an azure VM. The VM is running, I can ssh in and systemctl status jenkins.service is showing:
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Mon 2022-04-04 21:04:43 UTC; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2530 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Apr 04 21:04:42 pipeline jenkins[2530]: Correct java version found
Apr 04 21:04:42 pipeline jenkins[2530]: * Starting Jenkins Automation Server jenkins
Apr 04 21:04:42 pipeline su[2587]: Successful su for jenkins by root
Apr 04 21:04:42 pipeline su[2587]: + ??? root:jenkins
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session closed for user jenkins
Apr 04 21:04:43 pipeline jenkins[2530]: ...fail!
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Control process exited, code=exited status=7
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Failed with result 'exit-code'.
Apr 04 21:04:43 pipeline systemd[1]: Failed to start LSB: Start Jenkins at boot time.
I had previously been running into Incorrect Java version due to having installed Java 12, so I modified the /etc/init.d/jenkins like so:
# Which Java versions can be used to run Jenkins
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
# Work out the JAVA version we are working with:
JAVA_VERSION=$($JAVA -version 2>&1 | sed -n ';s/.* version "\([0-9]*\)\.\([0-9]*\)\..*".*/\1\2/p;')
to allow for Java 12. Any thoughts on how/why either the plugin addition or Java version and/or /etc/init.d/jenkins edit could be impacting things? My sense is that the initial reboot failed due to the Incorrect Java version issue, but not sure how I can resolve things and get it back up and running. It should, by default, be available at 8080 and that is where I am seeing This site can’t be reached.
I also have the networking set up like so:
and port 8080 should allow traffic. I have attempted to restart, hard start and stop/start to no avail as well. "Resource health" says the VM is available which should be obvious since it is running and I can ssh in. Do I need to redeploy perhaps?

First, please delegate the service to a non root user.
Second, azure, means in the net, please use SSL, the times without SSL are gone.
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
you are trying java 12, but its configured java 120, i think you should fix that to
JAVA_ALLOWED_VERSIONS=( "18" "11" "12" )

tomcat 8 stopped working after ubuntu update

I have recently updates my ubuntu deskto to 18.04. After this my tomcat is failing at startup. Each time I restart tomcat using the command service tomcat8 restart the log file give the following error -
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture.get(LRUHybridCache.java:164)
at org.glassfish.hk2.utilities.cache.LRUHybridCache.compute(LRUHybridCache.java:303)
... 135 more
Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3119)
at java.base/java.lang.Class.getDeclaredMethods(Class.java:2268)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperUtilities$3.run(ClassReflectionHelperUtilities.java:108)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperUtilities$3.run(ClassReflectionHelperUtilities.java:104)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperUtilities.secureGetDeclaredMethods(ClassReflectionHelperUtilities.java:104)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperUtilities.getDeclaredMethodWrappers(ClassReflectionHelperUtilities.java:133)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperUtilities.getAllMethodWrappers(ClassReflectionHelperUtilities.java:192)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperUtilities.getAllMethodWrappers(ClassReflectionHelperUtilities.java:193)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperImpl$3.compute(ClassReflectionHelperImpl.java:84)
at org.glassfish.hk2.utilities.reflection.internal.ClassReflectionHelperImpl$3.compute(ClassReflectionHelperImpl.java:80)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:115)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:111)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
I then installed jdk8 and run update-alternatives command like this -
dimension:bin$ sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/javac 1101 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/javac 1101 manual mode
* 2 /usr/lib/jvm/java-8-openjdk-amd64/bin/javac 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number:
dimension:bin$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 manual mode
* 2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
Press <enter> to keep the current choice[*], or type selection number:
When I again restarted the tomcat, I saw the same error. Here are the contents to my jvm folder -
ls -lh /usr/lib/jvm/
total 8.0K
lrwxrwxrwx 1 root root 25 Apr 8 18:46 default-java -> java-1.11.0-openjdk-amd64
lrwxrwxrwx 1 root root 21 Aug 24 23:06 java-1.11.0-openjdk-amd64 -> java-11-openjdk-amd64
drwxr-xr-x 9 root root 4.0K Oct 4 10:56 java-11-openjdk-amd64
lrwxrwxrwx 1 root root 20 Oct 28 2016 java-1.8.0-openjdk-amd64 -> java-8-openjdk-amd64
drwxr-xr-x 7 root root 4.0K Oct 4 16:03 java-8-openjdk-amd64
My JAVA_HOME env is currently set to empty.
What should I do to fix my issue.

The java.base part of exception stack trace at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) suggests that you are still running with JDK 11.
Ensure that Tomcat 8 is running with JDK 8. Try debugging the Tomcat startup script with set -x to see how is the Java location determined, maybe it's set in /etc/default/tomcat8?

Spark saveAsTextFile() results in Mkdirs failed to create for half of the directory

I am currently running a Java Spark Application in tomcat and receiving the following exception:
Caused by: java.io.IOException: Mkdirs failed to create file:/opt/folder/tmp/file.json/_temporary/0/_temporary/attempt_201603031703_0001_m_000000_5
on the line
text.saveAsTextFile("/opt/folder/tmp/file.json") //where text is a JavaRDD<String>
The issue is that /opt/folder/tmp/ already exists and successfully creates up to /opt/folder/tmp/file.json/_temporary/0/ and then it runs into what looks like a permission issue with the remaining part of the path _temporary/attempt_201603031703_0001_m_000000_5 itself, but I gave the tomcat user permissions (chown -R tomcat:tomcat tmp/ and chmod -R 755 tmp/) to the tmp/ directory. Does anyone know what could be happening?
Thanks
Edit for #javadba:
[root#ip tmp]# ls -lrta
total 12
drwxr-xr-x 4 tomcat tomcat 4096 Mar 3 16:44 ..
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 file.json
drwxrwxrwx 3 tomcat tomcat 4096 Mar 7 20:01 .
[root#ip tmp]# cd file.json/
[root#ip file.json]# ls -lrta
total 12
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 _temporary
drwxrwxrwx 3 tomcat tomcat 4096 Mar 7 20:01 ..
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 .
[root#ip file.json]# cd _temporary/
[root#ip _temporary]# ls -lrta
total 12
drwxr-xr-x 2 tomcat tomcat 4096 Mar 7 20:01 0
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 ..
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 .
[root#ip _temporary]# cd 0/
[root#ip 0]# ls -lrta
total 8
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 ..
drwxr-xr-x 2 tomcat tomcat 4096 Mar 7 20:01 .
The exception in catalina.out
Caused by: java.io.IOException: Mkdirs failed to create file:/opt/folder/tmp/file.json/_temporary/0/_temporary/attempt_201603072001_0001_m_000000_5
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:799)
at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more

saveAsTextFile is really processed by Spark executors. Depending on your Spark setup, Spark executors may run as a different user than your Spark application driver. I guess the spark application driver prepares the directory for the job fine, but then the executors running as a different user have no rights to write in that directory.
Changing to 777 won't help, because permissions are not inherited by child dirs, so you'd get 755 anyways.
Try running your Spark application as the same user that runs your Spark.

I suggest to try changing to 777 temporarily . See if it works at that point. There have been bugs/issues wrt permissions on local file system. If that still does not work let us know if anything changed or precisely same result.

I also had the same problem, and my issue has been resolved by using full HDFS path:
Error
Caused by: java.io.IOException: Mkdirs failed to create file:/QA/Gajendra/SparkAutomation/Source/_temporary/0/_temporary/attempt_20180616221100_0002_m_000000_0 (exists=false, cwd=file:/home/gajendra/LiClipse Workspace/SpakAggAutomation)
Solution
Use full HDFS path with hdfs://localhost:54310/<filePath>
hdfs://localhost:54310/QA/Gajendra/SparkAutomation

Could it be selinux/apparmor that plays you a trick? Check with ls -Z and system logs.

So, I've been experiencing the same issue, with my setup there is no HDFS and Spark is running in stand-alone mode. I haven't been able to save spark dataframes to an NFS share using the native Spark methods. The process runs as a local user, and I try to write to the users home folder. Even when creating a subfolder with 777 I cannot write to the folder.
The workaround for this is to convert the dataframe with toPandas() and after that to_csv(). This magically works.

I have the same issue as yours.
I also did not want to write to hdfs but to a local memory share.
After some research, I found that for my case the reason is: there are several nodes executing, however, some of the nodes has no access to the directory where you want to write your data.
Thus the solution is to make the directory available to all nodes, and then it works~

We need to run the application in local mode.
val spark = SparkSession
.builder()
.config("spark.master", "local")
.appName("applicationName")
.getOrCreate()

Giving the full path works for me.
Example:
file:/Users/yourname/Documents/electric-chargepoint-2017-data

this is tricky one, but simple to solve. One must configure job.local.dir variable to point to working directory. Following code works fine with writing CSV file:
def xmlConvert(spark):
etl_time = time.time()
df = spark.read.format('com.databricks.spark.xml').options(rowTag='HistoricalTextData').load(
'/home/zangetsu/proj/prometheus-core/demo/demo-1-iot-predictive-maintainance/dataset/train/')
df = df.withColumn("TimeStamp", df["TimeStamp"].cast("timestamp")).groupBy("TimeStamp").pivot("TagName").sum(
"TagValue").na.fill(0)
df.repartition(1).write.csv(
path="/home/zangetsu/proj/prometheus-core/demo/demo-1-iot-predictive-maintainance/result/",
mode="overwrite",
header=True,
sep=",")
print("Time taken to do xml transformation: --- %s seconds ---" % (time.time() - etl_time))
if __name__ == '__main__':
spark = SparkSession \
.builder \
.appName('XML ETL') \
.master("local[*]") \
.config('job.local.dir', '/home/zangetsu/proj/prometheus-core/demo/demo-1-iot-predictive-maintainance') \
.config('spark.driver.memory','64g') \
.config('spark.debug.maxToStringFields','200') \
.config('spark.jars.packages', 'com.databricks:spark-xml_2.11:0.5.0') \
.getOrCreate()
print('Session created')
try:
xmlConvert(spark)
finally:
spark.stop()

Unable to find a javac compiler even when JDK present

I'm trying to run an Apache Hive web interface, so I downloaded the lastest version of ant (Ant 1.9.4), and when I execute the command to produce the hive.war file
ANT_LIB=/opt/ant/lib bin/hive --service hwi
I get the following error:
cp=/tmp/Jetty_0_0_0_0_9999_hive.hwi.0.10.0.war__hwi__ae9cmk/jsp
cp=null
work dir=/tmp/Jetty_0_0_0_0_9999_hive.hwi.0.10.0.war__hwi__ae9cmk/jsp
extension dir=/usr/lib/jvm/jdk1.7.0_65/jre/lib/ext:/usr/java/packages/lib/ext
srcDir=/tmp/Jetty_0_0_0_0_9999_hive.hwi.0.10.0.war__hwi__ae9cmk/jsp
compilerTargetVM=1.5
compilerSourceVM=1.5
include=org/apache/jsp/index_jsp.java
15/02/23 09:56:59 ERROR compiler.Compiler: Error compiling file: /tmp/Jetty_0_0_0_0_9999_hive.hwi.0.10.0.war__hwi__ae9cmk/jsp//org/apache/jsp/index_jsp.java [javac] warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
15/02/23 09:56:59 ERROR mortbay.log: /hwi/
Unable to find a javac compiler;
com.sun.tools.javac.Main is not on the classpath.
Perhaps JAVA_HOME does not point to the JDK.
It is currently set to "/usr/lib/jvm/jdk1.7.0_65/jre"
at org.apache.tools.ant.taskdefs.compilers.CompilerAdapterFactory.getCompiler(CompilerAdapterFactory.java:130)
at org.apache.tools.ant.taskdefs.Javac.findSupportedFileExtensions(Javac.java:984)
at org.apache.tools.ant.taskdefs.Javac.scanDir(Javac.java:961)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:932)
at org.apache.jasper.compiler.AntCompiler.generateClass(AntCompiler.java:220)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:298)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:277)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:265)
I'm executing that code as root. My JAVA_HOME is correctly set at the .bashrc:
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65
export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec
export HADOOP_CONF_DIR=/etc/hadoop/conf
export PATH=$PATH:$JAVA_HOME/bin
and .bash_profile files:
.bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export PATH
I want to know why is Ant referencing the javac at "/usr/lib/jvm/jdk1.7.0_65/jre". Moreover, here is my JDK structure:
[root#ip-10-32-205-19 ~]# cd $JAVA_HOME
[root#ip-10-32-205-19 jdk1.7.0_65]# ls -l
total 19760
drwxr-xr-x. 2 root root 4096 Feb 23 04:09 bin
-r--r--r--. 1 root root 3339 Jun 17 2014 COPYRIGHT
drwxr-xr-x. 4 root root 4096 Jun 17 2014 db
drwxr-xr-x. 3 root root 4096 Jun 17 2014 include
drwxr-xr-x. 5 root root 4096 Jun 17 2014 jre
drwxr-xr-x. 5 root root 4096 Feb 23 04:09 lib
-r--r--r--. 1 root root 40 Jun 17 2014 LICENSE
drwxr-xr-x. 4 root root 4096 Jun 17 2014 man
-r--r--r--. 1 root root 114 Jun 17 2014 README.html
-rw-r--r--. 1 root root 499 Jun 17 2014 release
-rw-r--r--. 1 root root 19902785 Jun 17 2014 src.zip
-rw-r--r--. 1 root root 110114 Jun 16 2014 THIRDPARTYLICENSEREADME-JAVAFX.txt
-r--r--r--. 1 root root 173559 Jun 17 2014 THIRDPARTYLICENSEREADME.txt
which shows I have the JDK, and not only the JRE installed. I also took a look at similar questions, but none could solve my problem. The Java is 64bits, plus I also ran source for both .bashrc and .bash_profile.

The error message says:
Perhaps JAVA_HOME does not point to the JDK. It is currently set to
"/usr/lib/jvm/jdk1.7.0_65/jre"
However, your .bashrc has:
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65
You likely need to run source ~/.bashrc in your Bash shell to reset JAVA_HOME to the root of your JDK installation. Then run Ant again.

GSSException: [..] Encryption type AES256CTS mode with HMAC SHA1-96 is not supported/enabled

After setting our domain users to support AES encryption for Kerberos tokens (Windows Server 2008R2), on a web-application server side we get the following exception:
GSSException: Failure unspecified at GSS-API level (Mechanism level:
Encryption type AES256CTS mode with HMAC SHA1-96 is not
supported/enabled)
Strangely we have Java 6 (1.6.0_27) , which means that AES should be supported, according to this document: http://docs.oracle.com/javase/6/docs/technotes/guides/security/jgss/jgss-features.html
Any ideas what's missing in our web-application or Java, or third parties? We are using Spring security Kerberos extension (with minimal code modifications to fit into our current Spring 2.x version and additional authentication requirements).

EDIT (2017-05-06): upcoming JDK versions will have this included. Only a config parameter needs to be set, see JDK-8157561.
Follow this link - Java SE Downloads, scroll down and download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for your specific JDK version and follow the process in this tutorial titled: 5.4.2. Kerberos and Unlimited Strength Policy.
The basic steps are as follows:
locate your JDK's security directory (showing Unix below):
$ locate 'jre/lib/security' | grep 'lib/security$'
/usr/java/jdk1.7.0_17/jre/lib/security
/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre/lib/security
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/security
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/lib/security
Noting the above, we need to add the downloaded JCE .jar files to /usr/java/jdk1.7.0_17/jre/lib/security.
The JCE .zip file includes the following (showing JDK 1.7's JCE):
$ ls -l UnlimitedJCEPolicy
total 16
-rw-rw-r-- 1 root root 2500 May 31 2011 local_policy.jar
-rw-r--r-- 1 root root 7289 May 31 2011 README.txt
-rw-rw-r-- 1 root root 2487 May 31 2011 US_export_policy.jar
These are the bundled versions with the JDK (again 1.7):
$ ls -l /usr/java/jdk1.7.0_17/jre/lib/security/*.jar
-rw-r--r--. 1 root root 2865 Mar 1 2013 /usr/java/jdk1.7.0_17/jre/lib/security/local_policy.jar
-rw-r--r--. 1 root root 2397 Mar 1 2013 /usr/java/jdk1.7.0_17/jre/lib/security/US_export_policy.jar
We need to move these out of the way and replace them with the included versions in the JCE .zip file. I typically do the following:
$ pushd /usr/java/jdk1.7.0_17/jre/lib/security/
/usr/java/jdk1.7.0_17/jre/lib/security ~
$ mkdir limited
$ mv *.jar limited/
$ cp ~/UnlimitedJCEPolicy/*.jar .
$ ls -l *.jar
-rw-r--r-- 1 root root 2500 Jun 25 12:50 local_policy.jar
-rw-r--r-- 1 root root 2487 Jun 25 12:50 US_export_policy.jar
Restart anything that's making use of JDK (Tomcat, etc.).