Develop Reference

Running Tomcat9 with dedicated tomcat user: Startup fails with JAVA_HOME or 500 in localhost:8080

I have just installed Apache Tomcat9 on my machine, following mostly these instructions, but I am really stuck with user management here.
My enviroment: Ubuntu 20.04, Java 11 (There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-11-oracle/bin/java).
Few exceptions: I neither have JAVA_HOME, nor the java home directory in my PATH environment variable.
I also have created a symlink at /opt/tomcat/latest/ for pointing to the right folder for all tomcat files.
Now I want to let tomcat run as a service as user 'tomcat'.
This is how my tomcat.service file looks like:
[Unit]
Description=Tomcat 9 servlet container
After=network.target
[Service]
Type=forking
User=tomcat
Group=tomcat
Environment="JAVA_HOME=/usr/lib/jvm/java-11-oracle/bin/java"
# Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
Environment="CATALINA_BASE=/opt/tomcat/latest"
Environment="CATALINA_HOME=/opt/tomcat/latest"
Environment="CATALINA_PID=/opt/tomcat/latest/tomcat.pid"
# Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
ExecStart=/opt/tomcat/latest/bin/startup.sh
ExecStop=/opt/tomcat/latest/bin/shutdown.sh
[Install]
WantedBy=multi-user.target
Now, when I comment out the line for the JAVA_HOME environment variable, the startup of the tomcat service fails (journalctl -xe):
Okt 27 17:43:52 my-user startup.sh[7714]: Tomcat started.
Okt 27 17:43:52 my-user shutdown.sh[7737]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Okt 27 17:43:52 my-user sudo[7690]: pam_unix(sudo:session): session closed for user root
So there is something running as root, what I don't get.
But when I comment out this line, tomcat runs happily, but on localhost:8080 I am getting this 500:
Message org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.index_jsp
or this 404
Message JSP file [/index.jsp] not found
It is similar like here and here.
I also did sudo chown -R tomcat /opt/tomcat/latest/work/Catalina/localhost, but this did not help.
Any help is appreciated.

Now it works and I'm going to list my changes here, but I'm not sure, which one was crucial and which was unnecessary.
JAVA_HOME in tomcat.service changed to: Environment="JAVA_HOME=/usr/lib/jvm/java-11-oracle"
CATALINA_PID in tomcat.service changed to: Environment="CATALINA_PID=/opt/tomcat/latest/temp/tomcat.pid"
give write access to other users to folders work, temp, logs, webapps. Sources here and here.
using this guide: https://www.interserver.net/tips/kb/install-apache-tomcat-on-ubuntu-18-04/

Enabling jmx authentication on java app run as a service using procrun

I am running a java app as a service using procrun. The app is running fine when the JMX authentication is not enabled, but when I add an access and password files, restrict permissions on the password file and enable the JMX authentication, the app is not starting and giving 1074: The process terminated unexpectedly error.
Below is the script I am using to register the app as a service using the latest version of procrun.
cd E:\MyApp
MyApp.exe //IS//MyApp --Install=%cd%\MyApp.exe --Jvm="C:\Program Files\Java\jdk1.8.0_131\jre\bin\server\jvm.dll" --JvmMx=2048 --JvmMs=2048 --Startup=auto --StartMode jvm --StartClass com.MyApp --StartParams start --StartMethod start --StopMode jvm --StopClass com.MyApp --StopParams stop --StopMethod stop --Classpath=%cd%\MyApp.jar --LibraryPath=%cd%\bin --JvmOptions="-Duser.dir=%cd%" ++JvmOptions=-XX:+RelaxAccessControlCheck ++JvmOptions=-Dcom.sun.management.jmxremote ++JvmOptions=-Dcom.sun.management.jmxremote.port=1099 ++JvmOptions=-Dcom.sun.management.jmxremote.ssl=false ++JvmOptions=-Dcom.sun.management.jmxremote.authenticate=true ++JvmOptions=-Dcom.sun.management.jmxremote.access.file=E:\MyApp\Conf\jmxremote.access ++JvmOptions=-Dcom.sun.management.jmxremote.password.file=E:\MyApp\Conf\jmxremote.password --JvmOptions=-XX:+UseConcMarkSweepGC --JvmOptions=-XX:CMSInitiatingOccupancyFraction=40 ++JvmOptions=-Dorg.xml.sax.parser="com.sun.org.apache.xerces.internal.parsers.SAXParser" ++JvmOptions=-Djavax.xml.parsers.DocumentBuilderFactory="com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl" ++JvmOptions=-Djavax.xml.parsers.SAXParserFactory="com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl" --LogPath=E:\Logs --LogLevel=DEBUG --StdOutput=auto --StdError=auto --StartPath=%cd% --StopPath=%cd% --DisplayName=MyApp
Any clues on what could be wrong?

I think there is a problem with your password file. At first just use a simple jmx configuration like mine. Here is my simple JMX configuration for startup:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=11099
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=localhost
-Dcom.sun.management.jmxremote.rmi.port=11098

How to grab a particular log file and show its content in jenkins Console output

I have the following Jenkins post-build shell script:
ssh user#my_server <<EOF
service my_service stop
service my_service start
tail -f /opt/services/my_service/logs/current
exit
EOF
This script restarts my_service on a remote host (my_server).
My problem is: command service my_service start just makes a request to RUNIT to run a my_service, i.e service my_service start returns immediately after execution.
But service my_service start runs a SpringBoot java web application that writes all log info into .../logs/current log file. To catch this log info I've added command tail -f /opt/services/my_service/logs/current but in this case Jenkins build is never ends)) such as tail -f command never stops.
Is there a way to execute my post-build script (which only start my web app on a remote server) and grabbing the .../logs/current log file during 2 minutes or until this log has the line "Web app MyApplication has been Started".
I wanna see the content of .../logs/current log file right in Jenkins's Console output and kill tail -f after 2 minutes

tail -f will not end until it gets interrupted, so your script will never finish running.
what you can do is use grep -q on your log, which will exit with 0 exit status when it finds it's pattern:
grep -q 'Web app MyApplication has been Started' <(tail -f /opt/services/my_service/logs/current)

How to start Glassfish service in Windows using batch file that will pause execution untill domain is started?

At this moment I am using command below as a part of my batch script to boot domain1:
asadmin start-domain domain1
however I have recently installed domain1 as a service so now when I use this command the domain is starting under my user process instead of booting as a service. So after I logout, the domain is gone. I used:
net start domain1
and
sc start domain1
However both of these seem to return as soon as signal[or whatever else] is dispatched toward service, and they do not wait untill domain1 is actually started. "asadmin start-domain" did return after it started the domain...
I have to wait as in my script I am undeploying/deploying new app shortly after domain start. So is there any way to start Glassfish as service using batch command and wait untill it is started?

Install:
sc create ServiveName binpath= <PATH_TO_SERVICE>.exe
net start ServiveName
PAUSE
Start:
net start ServiceName
PAUSE
Stop:
net stop ServiceName
PAUSE
Uninstall:
net stop ServiceName
sc delete ServiceName
PAUSE

One of the solutions I am using:
#echo off
SETLOCAL enableextensions enabledelayedexpansion
set GLASSFISH_HOME=c:\glassfish
set DOMAIN=domain1
net start %DOMAIN%
:loop
call timeout /t 1 /NOBREAK > NUL
echo Still waiting for domain to start
for /f "tokens=1,2 delims= " %%A IN ( '"%GLASSFISH_HOME%\bin\asadmin.bat" list-domains' ) DO IF "%%A"=="%DOMAIN%" SET GLASSFISH_RUNNING=%%B
if not "%GLASSFISH_RUNNING%"=="running" (
goto loop
)

I modified the above version a little bit for better understanding:
#echo off
SETLOCAL enableextensions enabledelayedexpansion
set GLASSFISH_HOME=D:\glassfish
set DOMAIN=domainName
set SERVICE_NAME="name of your service"
net start %SERVICE_NAME%
:loop
call timeout /t 1 /NOBREAK > NUL
echo Still waiting for domain to start
for /f "tokens=1,2 delims= " %%A IN ( '"%GLASSFISH_HOME%\bin\asadmin.bat" list-domains' ) DO IF "%%A"=="%DOMAIN%" SET GLASSFISH_RUNNING=%%B
if not "%GLASSFISH_RUNNING%"=="running" (
goto loop
)
Some applicatios already provide possiblities to automatically create Windows services. However every .exe can be configured this way.
GUI: http://www.sevenforums.com/tutorials/2495-services-start-disable.html
Console: https://support.microsoft.com/de-de/kb/137890
Automatic startup before logon: https://serverfault.com/questions/227862/run-a-program-without-user-being-logged-on

Type the following code in notepad and save [name].bat (for windows)
cd C:\glassfish3\glassfish3\bin
asadmin start-domain
PAUSE

Solr Configuration

I tried to installed Solr using:
java -jar start.jar
However I downloaded the source code and didn't compile it (Didn't pay attention). And the error was:
http://localhost:8983/solr/admin/
HTTP ERROR: 404
Problem accessing /solr/admin/. Reason:
NOT_FOUND
Then I downloaded the compiled version of solr but when trying to run the example configuration I'm getting exception:
java.net.BindException: Address already in use
Is there a way to revert solr configuration and start from scratch? Looks like the configuration got messed up. I don't see anything related to it in the manual.
Here is the error:
2011-07-10 22:41:27.631:WARN::failed SocketConnector#0.0.0.0:8983: java.net.BindException: Address already in use
2011-07-10 22:41:27.632:WARN::failed Server#c4e21db: java.net.BindException: Address already in use
2011-07-10 22:41:27.632:WARN::EXCEPTION
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)
at java.net.ServerSocket.<init>(ServerSocket.java:150)
at org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
at org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Jul 10, 2011 10:41:27 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher#5b6b9e62 main

This means you already have an application running on that particular port.
Run:
$ lsof -i :8983
This gives you a list of any application running on that port. In my case, Solr is already running, and I get back:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 10289 patricia 111u IPv6 399410 0t0 TCP *:8983 (LISTEN)
Kill this process by filling in your PID:
$ kill 10289
And then try running Solr again.

The java.net.BindException means that you are attempting to restart solr while an earlier instance continues to run, or less probably that you have something else running on port 8983. You should find that process, kill it, and then start solr again.

Its bound to a some other application. In case if its a important app you can change jetty default port using following:
java -Djetty.port=8181 -jar start.jar

ps aux | grep solr
OR
ps aux | grep start.jar
AND the get then process id and kill it:
kill -9 #PID#
example : kill -9 4989
UPDATE:
after killing the process if you want to reinstall solr you soul first uninstall it, the following is one of the solutions to uninstall it:
sudo service solr stop
sudo rm -r /var/solr
sudo rm -r /opt/solr-5.3.1
sudo rm -r /opt/solr
sudo rm /etc/init.d/solr
sudo deluser --remove-home solr
sudo deluser --group solr
An now you can reinstall it with no problem.

If sudo lsof -i:8983 won't help finding application running on the same port, the common mistake is Tomcat misconfiguration (if you're using it).
For example by default Tomcat listens on port 8005 for SHUTDOWN command and if you set another Connector to listen on the same port, you'll get port conflict.
So please double check in server.xml if these ports are different:
<Server port="8005" shutdown="SHUTDOWN">
<Connector port="8983" protocol="HTTP/1.1"

Maybe some crazy idea is to use docker to read a complete extended step by step and repeatable installation:
Here dockerhub to select the specific version tu run in docker Docker Hub Solr
And here github to read the docker recipe Solr docker Recipe