Develop Reference

Jenkins in Azure VM "This site can’t be reached" at 8080

I attempted to add the nvm-wrapper plugin (https://plugins.jenkins.io/nvm-wrapper/) to jenkins and reboot and am no longer able to reach the site (This site can’t be reached). It is deployed as an azure VM. The VM is running, I can ssh in and systemctl status jenkins.service is showing:
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Mon 2022-04-04 21:04:43 UTC; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2530 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Apr 04 21:04:42 pipeline jenkins[2530]: Correct java version found
Apr 04 21:04:42 pipeline jenkins[2530]: * Starting Jenkins Automation Server jenkins
Apr 04 21:04:42 pipeline su[2587]: Successful su for jenkins by root
Apr 04 21:04:42 pipeline su[2587]: + ??? root:jenkins
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session closed for user jenkins
Apr 04 21:04:43 pipeline jenkins[2530]: ...fail!
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Control process exited, code=exited status=7
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Failed with result 'exit-code'.
Apr 04 21:04:43 pipeline systemd[1]: Failed to start LSB: Start Jenkins at boot time.
I had previously been running into Incorrect Java version due to having installed Java 12, so I modified the /etc/init.d/jenkins like so:
# Which Java versions can be used to run Jenkins
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
# Work out the JAVA version we are working with:
JAVA_VERSION=$($JAVA -version 2>&1 | sed -n ';s/.* version "\([0-9]*\)\.\([0-9]*\)\..*".*/\1\2/p;')
to allow for Java 12. Any thoughts on how/why either the plugin addition or Java version and/or /etc/init.d/jenkins edit could be impacting things? My sense is that the initial reboot failed due to the Incorrect Java version issue, but not sure how I can resolve things and get it back up and running. It should, by default, be available at 8080 and that is where I am seeing This site can’t be reached.
I also have the networking set up like so:
and port 8080 should allow traffic. I have attempted to restart, hard start and stop/start to no avail as well. "Resource health" says the VM is available which should be obvious since it is running and I can ssh in. Do I need to redeploy perhaps?

First, please delegate the service to a non root user.
Second, azure, means in the net, please use SSL, the times without SSL are gone.
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
you are trying java 12, but its configured java 120, i think you should fix that to
JAVA_ALLOWED_VERSIONS=( "18" "11" "12" )

Tomcat 9 no longer starting using systemctl but will start manually

Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat

So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.

ubuntu server 16.04 - tomcat7 failed to start

For the past 2 months I run tomcat7 perfectly on my ubuntu server. Today tomcat7 failed to start.
I started tomcat 7
sudo systemctl start tomcat 7
Job for tomcat7.service failed because the control process exited with error code.
See "systemctl status tomcat7.service" and "journalctl -xe" for details.
Then I checked for the status
sudo systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat7; generated)
Active: failed (Result: exit-code) since Sat 2020-05-16 15:47:00 HKT; 8min ago
Docs: man:systemd-sysv-generator(8)
Process: 6043 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=1/FAILURE
May 16 15:46:55 www.example.com systemd[1]: Starting LSB: Start Tomcat....
May 16 15:46:55 www.example.com tomcat7[6043]: * Starting Tomcat servlet engine
May 16 15:47:00 www.example.com tomcat7[6043]: ...fail!
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Control process exit
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Failed with result '
May 16 15:47:00 www.example.com systemd[1]: Failed to start LSB: Start Tomcat..
Then, I checked for the log error and I have no idea how to fix this
sudo tail /etc/var/log/tomcat7/catalina.out
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Somebody please help ....

System environment variables in Jetty application

How to configure system environment variables inside one Jetty application?
e.g. For database connection details, putting it in file and checking it into cvs is bad idea. For that reason using system environment is one way to go. While the system environment variables are defined in /etc/environments file or .bashrc/.zshrc file , in Jetty application, doing System.getenv("variable_name") won't give anything. It will result in null.
I have read this question: Configuring a Jetty application with env variables which concludes that which tells that Jetty doesn't support System.getenv() and not even in start.ini file.
And jetty and etc environment on ubuntu 12.10 which says In the jetty.sh script you can source the /etc/environment file and they will be present. which I tried and didn't get the values as expected meaning it gave me only null.
If I can't use the default System.getenv() or in any .ini file then how to specify credentials such as Database connection etc ?

Not supporting System.getenv() is not a Jetty thing, but a Java thing.
There are ton of restrictions around System.getenv() and your environment, making it nearly useless in all but the most naive and basic test case. (eg: multiline values are not supported. multiline entries can break parsing. keys without values are not supported. keys without values can often merge with next key during parsing. entries with non US-ASCII characters are not supported. entries with control characters are not supported.)
The common technique when using System Environment variables with Java programs is to use the shell specific techniques to obtain the values and inject them either on the command line, or into a ini file format for Jetty to then use.
Depending on your technique, these values would either show up as Jetty properties, or as Java System Properties.
Just created a project to demonstrate 4 ways to accomplish this at
https://github.com/jetty-project/jetty-external-config
External Configuration Properties with Jetty
Demonstration of how to configure simple properties that can be accessed
by Servlets within Jetty.
This demonstration shows 4 different ways to configure a property
at runtime, that can then be read by the Servlet running within
Jetty.
The props.war
This is a simple war file with a single HttpServlet and a WEB-INF/web.xml
[jetty-external-config]$ jar -tvf target/props.war
0 Mon Feb 23 09:02:14 MST 2015 META-INF/
131 Mon Feb 23 09:02:14 MST 2015 META-INF/MANIFEST.MF
0 Mon Feb 23 09:02:14 MST 2015 WEB-INF/
0 Mon Feb 23 09:02:14 MST 2015 WEB-INF/classes/
0 Mon Feb 23 09:02:14 MST 2015 WEB-INF/classes/org/
0 Mon Feb 23 09:02:14 MST 2015 WEB-INF/classes/org/eclipse/
0 Mon Feb 23 09:02:14 MST 2015 WEB-INF/classes/org/eclipse/demo/
2188 Mon Feb 23 09:02:12 MST 2015 WEB-INF/classes/org/eclipse/demo/PropsServlet.class
572 Mon Feb 23 08:45:22 MST 2015 WEB-INF/web.xml
See PropsServlet.java for details of behavior.
Just compile the top level and the war file will be built and placed in all of the demo jetty.base locations for this project.
Example #1: Basic Command Line
The /base-command-line project contains a simple start.ini which starts jetty on port 9090, and deploys the webapp. no extra configuration is done by the on-disk configuration.
If you start it up like so ...
[base-command-line]$ java -jar /path/to/jetty-distribution-9.2.7.v20150116/start.jar
2015-02-23 09:15:46.088:INFO::main: Logging initialized #290ms
2015-02-23 09:15:46.222:INFO:oejs.Server:main: jetty-9.2.7.v20150116
2015-02-23 09:15:46.235:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/home/joakim/code/stackoverflow/jetty-external-config/base-command-line/webapps/] at interval 1
2015-02-23 09:15:46.325:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /props, did not find org.eclipse.jetty.jsp.JettyJspServlet
2015-02-23 09:15:46.343:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext#6e7f61a3{/props,file:/tmp/jetty-0.0.0.0-9090-props.war-_props-any-27537844855769703.dir/webapp/,AVAILABLE}{/props.war}
2015-02-23 09:15:46.353:INFO:oejs.ServerConnector:main: Started ServerConnector#67cd35c5{HTTP/1.1}{0.0.0.0:9090}
2015-02-23 09:15:46.353:INFO:oejs.Server:main: Started #555ms
you'll see that it has started up and deployed to the /props context path.
From here you can test for properties in the servlet via tooling like wget or curl.
Example:
$ curl http://localhost:9090/props/props
[java.runtime.name] = Java(TM) SE Runtime Environment
[sun.boot.library.path] = /home/joakim/java/jvm/jdk-7u75-x64/jre/lib/amd64
[java.vm.version] = 24.75-b04
[java.vm.vendor] = Oracle Corporation
[java.vendor.url] = http://java.oracle.com/
...
[file.separator] = /
[java.vendor.url.bug] = http://bugreport.sun.com/bugreport/
[sun.io.unicode.encoding] = UnicodeLittle
[sun.cpu.endian] = little
[sun.desktop] = gnome
[sun.cpu.isalist] =
You can even request a specific property ..
$ curl http://localhost:9090/props/props/user.timezone
[user.timezone] = America/Phoenix
Lets stop the server and run it with a system property of our choice.
Notice the -Dfoo=bar ?
[base-command-line]$ java -Dfoo=bar -jar /path/to/jetty-distribution-9.2.7.v20150116/start.jar
2015-02-23 09:15:46.088:INFO::main: Logging initialized #290ms
2015-02-23 09:15:46.222:INFO:oejs.Server:main: jetty-9.2.7.v20150116
2015-02-23 09:15:46.235:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/home/joakim/code/stackoverflow/jetty-external-config/base-command-line/webapps/] at interval 1
2015-02-23 09:15:46.325:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /props, did not find org.eclipse.jetty.jsp.JettyJspServlet
2015-02-23 09:15:46.343:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext#6e7f61a3{/props,file:/tmp/jetty-0.0.0.0-9090-props.war-_props-any-27537844855769703.dir/webapp/,AVAILABLE}{/props.war}
2015-02-23 09:15:46.353:INFO:oejs.ServerConnector:main: Started ServerConnector#67cd35c5{HTTP/1.1}{0.0.0.0:9090}
2015-02-23 09:15:46.353:INFO:oejs.Server:main: Started #555ms
and look for it via curl ...
$ curl http://localhost:9090/props/props/foo
[foo] = bar
That demonstrates access of a property that was specified via the command line, now lets look at the other choices.
Example #2: Using start.ini
The /base-startini project contains a simple start.ini which starts jetty on port 9090, and deploys the webapp.
This start.ini also contains a foo.ish property.
Lets start up Jetty and try our props servlet access again ...
[base-startini]$ java -jar /path/to/jetty-distribution-9.2.7.v20150116/start.jar
2015-02-23 09:16:46.088:INFO::main: Logging initialized #290ms
2015-02-23 09:16:46.222:INFO:oejs.Server:main: jetty-9.2.7.v20150116
and request it via curl ...
$ curl http://localhost:9090/props/props/foo.ish
[foo.ish] = bar
Example #3: Using start.d optional ini
The /base-startd project contains a simple start.ini which starts jetty on port 9090, and deploys the webapp.
This start.ini also contains no extra properties that we are interested in.
The start.d/myconf.ini contains a property called foo.d that we are interested in.
Lets start up Jetty and try our props servlet access again ...
[base-startd]$ java -jar /path/to/jetty-distribution-9.2.7.v20150116/start.jar
2015-02-23 09:19:46.088:INFO::main: Logging initialized #290ms
2015-02-23 09:19:46.222:INFO:oejs.Server:main: jetty-9.2.7.v20150116
and request it via curl ...
$ curl http://localhost:9090/props/props/foo.d
[foo.d] = over here
Example #4: Using --include-jetty-dir optional config
The /base-jettyinclude project contains a new start.ini which starts jetty on port 9090, and deploys the webapp.
This start.ini also contains no extra properties that we are interested in.
However, the start.ini uses the --include-jetty-dir=../jettydir optional configuration that points to an entirely new interrim jetty.base configuration source.
The ../jettydir/start.ini contains a property called foo.jetty.dir that we are interested in.
Lets start up Jetty and try our props servlet access again ...
[base-jettyinclude]$ java -jar /path/to/jetty-distribution-9.2.7.v20150116/start.jar
2015-02-23 09:24:46.088:INFO::main: Logging initialized #290ms
2015-02-23 09:24:46.222:INFO:oejs.Server:main: jetty-9.2.7.v20150116
and request it via curl ...
$ curl http://localhost:9090/props/props/foo.jetty.dir
[foo.jetty.dir] = more of the same

neo4j cannot find JAVA_HOME when checking service status

I installed neo4j 1.8.2 on opensuse 12.2/64. To do so I had to add the JAVA_HOME path in the /etc/profile file as:
export JAVA_HOME=/opt/java/64/jdk1.7.0_21/jre/:
export PATH=$PATH:/opt/java/64/jdk1.7.0_21/jre/bin/;
Now when I try to check the server status I get the following error
>service neo4j-service status
neo4j-service.service - LSB: The Neo4J graph database server. See http://neo4j.org
Loaded: loaded (/etc/init.d/neo4j-service)
Active: failed (Result: exit-code) since Fri, 26 Apr 2013 17:13:56 +0200; 10s ago
Process: 7234 ExecStart=/etc/init.d/neo4j-service start (code=exited, status=1/FAILURE)
CGroup: name=systemd:/system/neo4j-service.service
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/...bin)
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: Error: JAVA_HOME is not defined correctly.
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: We cannot execute
It's quite puzzling considering neo4j-service links to ./bin/neo4j, namely the file used at installation time with
./bin/neo4j install
Some ideas on what is going on here?
Thanks
SOLVED
Actually I was using jdk 7 instead of jdk 6
EDIT 2
According to the official neo4j page one runs the server using neo4j start. But I got into troubles when trying to run service neo4j start/status/stop as suggested in the Installing Neo4j in Linux how-to.

Try changing them to this:
export JAVA_HOME=/opt/java/64/jdk1.7.0_21/:
export PATH=$PATH:/opt/java/64/jdk1.7.0_21/bin/;

Not exactly for the question, but I reached here because I also though Neo4j could not find the environment variable JAVA_HOME when
sudo neo4j start
My problem was that OS reset environment variables when using sudo.
sudo -E neo4j start
solved my problem.