Docker jetty application does not see files from volume - java

I have two dockerized applications and additional volume container:
dbdata volume container has one volume: /home/git/data
dockerized Gitlab - runned with --volumes-from dbdata to mount volume from volume container,
my own application https://github.com/tunguski/gitlab-java-event-listener - runned with --volumes-from dbdata to mount volume from volume container (Dockerfile).
My application tries to clone git repository from /home/git/data/repositories. This directory is owned by user git from gitlab container. But after adding
RUN groupadd --gid 1000 git && usermod -a -G git jetty
to Dockerfile, user jetty sees files in it. I've tested it executing su - jetty and ls -la /home/git/data/repositories.
JGit reports that /home/git/data/repositories/<user>/<repo.git> does not exist. I modified app to log what parent directory actually exist, and it's /home/git/data/repositories. My java application does not see any subdirectories.
But when I connect to my app's container using:
docker exec -it my-app bash
change user to jetty:
su - jetty
then listing directory:
ls -la /home/git/data/repositories/<user>/<repo.git>
shows that repository exist!
To confirm that server runs on this particular user:
$ ps auxwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 72 0.0 0.0 21920 3556 ? Ss 13:54 0:00 bash
root 77 0.0 0.0 46432 2920 ? S 13:54 0:00 \_ su - jetty
jetty 78 0.0 0.0 4328 700 ? S 13:54 0:00 \_ -su
jetty 81 0.0 0.0 17488 2072 ? R+ 13:54 0:00 \_ ps auxwf
jetty 1 0.5 5.7 5627056 464960 ? Ssl 12:56 0:18 /usr/bin/java -Djetty.logging.dir=/usr/local/jetty/logs -Djetty.home=/usr/local/jetty -Djetty.base=/var/lib/jetty -Djava.io.tmpdir=/tmp/jett[...]
I really don't know what is the problem here.
Do you know what may be the cause of this?

Related

Can't start JIRA by source

OS: CentOS
Installed JIRA by official guide:
https://confluence.atlassian.com/adminjiraserver/installing-jira-applications-on-linux-from-archive-file-938846844.html
When start JIRA, got error:
[jira#node1 bin]$ ./start-jira.sh
To run JIRA in the foreground, start the server with start-jira.sh -fg
executing as current user
`sMMMMMMMMMMMMMM+
MMMMMMMMMMMMMM
:sdMMMMMMMMMMM
MMMMMM
`sMMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sMMMMMMMMMMM MMMMM
MMMMMM `UOJ
`sMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sdMMMMMMMMMM MMMMM
MMMMMM `UOJ
MMMMMM
+MMMMM
MMMMM
`UOJ
Atlassian Jira
Version : 8.5.1
If you encounter issues starting or stopping JIRA, please see the Troubleshooting guide at https://docs.atlassian.com/jira/jadm-docs-085/Troubleshooting+installation
Using JIRA_HOME: /home/jira/jirasoftware-home
Server startup logs are located in /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/logs/catalina.out
Using CATALINA_BASE: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone
Using CATALINA_HOME: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone
Using CATALINA_TMPDIR: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/temp
Using JRE_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
Using CLASSPATH: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/bootstrap.jar:/home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/tomcat-juli.jar
Using CATALINA_PID: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/work/catalina.pid
/home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/check-java.sh: line 31: [: -ne: unary operator expected
/home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/set-gc-params.sh: line 5: [: -ge: unary operator expected
Existing PID file found during start.
Unable to read PID file. Start aborted.
Check JAVA_HOME:
[jira#node1 bin]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
Check PID:
[jira#node1 bin]$ ps aux | grep jira
root 8185 0.0 0.4 241148 4552 pts/0 S 06:47 0:00 sudo su - jira
root 8186 0.0 0.2 191748 2300 pts/0 S 06:47 0:00 su - jira
jira 8187 0.0 0.2 116096 2812 pts/0 S 06:47 0:00 -bash
jira 8299 0.0 0.1 155364 1876 pts/0 R+ 07:04 0:00 ps aux
jira 8300 0.0 0.0 112712 960 pts/0 R+ 07:04 0:00 grep --color=auto jira
No PID exists in the jira user process. Why aborted?
Use script to stop the jira and then start it again. As you can see in the log you have already PID file:
Existing PID file found during start.
Or you can remove the PID file by hand and try to start again

java NIO memory map a docker mounted file fail

I recently run into docker and deploy my java application into a tomcat docker container. But I met a very specific error about NIO memory mapping a file:
File mark = new File("/location/to/docker/mounted/file");
m_markFile = new RandomAccessFile(mark, "rw");
MappedByteBuffer m_byteBuffer = m_markFile.getChannel().map(MapMode.READ_WRITE, 0, 20);
And the last function call failed as:
Caused by: java.io.IOException: Invalid argument
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:906)
at com.dianping.cat.message.internal.MessageIdFactory.initialize(MessageIdFactory.java:127)
at com.dianping.cat.message.internal.DefaultMessageManager.initialize(DefaultMessageManager.java:197)
... 34 more
I don't know what happened. I tested it in my local Mac environment, it's ok. And within the tomcat docker container, I change the file location to a normal file path, it's ok too. Seems which happens only on docker mounted file.
Other information:
root#4520355ed3ac:/usr/local/tomcat# uname -a
Linux 4520355ed3ac 4.4.27-boot2docker #1 SMP Tue Oct 25 19:51:49 UTC 2016 x86_64 GNU/Linux
Mounted a folder in Mac Users to /data
root#4520355ed3ac:/usr/local/tomcat# df
Filesystem 1K-blocks Used Available Use% Mounted on
none 18745336 6462240 11292372 37% /
tmpfs 509832 0 509832 0% /dev
tmpfs 509832 0 509832 0% /sys/fs/cgroup
Users 243924992 150744296 93180696 62% /data
/dev/sda1 18745336 6462240 11292372 37% /etc/hosts
shm 65536 0 65536 0% /dev/shm
docker version
huanghaideMacBook-Pro:cat huanghai$ docker --version
Docker version 1.12.3, build 6b644ec
huanghaideMacBook-Pro:cat huanghai$ docker-machine --version
docker-machine version 0.8.2, build e18a919

Spark saveAsTextFile() results in Mkdirs failed to create for half of the directory

I am currently running a Java Spark Application in tomcat and receiving the following exception:
Caused by: java.io.IOException: Mkdirs failed to create file:/opt/folder/tmp/file.json/_temporary/0/_temporary/attempt_201603031703_0001_m_000000_5
on the line
text.saveAsTextFile("/opt/folder/tmp/file.json") //where text is a JavaRDD<String>
The issue is that /opt/folder/tmp/ already exists and successfully creates up to /opt/folder/tmp/file.json/_temporary/0/ and then it runs into what looks like a permission issue with the remaining part of the path _temporary/attempt_201603031703_0001_m_000000_5 itself, but I gave the tomcat user permissions (chown -R tomcat:tomcat tmp/ and chmod -R 755 tmp/) to the tmp/ directory. Does anyone know what could be happening?
Thanks
Edit for #javadba:
[root#ip tmp]# ls -lrta
total 12
drwxr-xr-x 4 tomcat tomcat 4096 Mar 3 16:44 ..
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 file.json
drwxrwxrwx 3 tomcat tomcat 4096 Mar 7 20:01 .
[root#ip tmp]# cd file.json/
[root#ip file.json]# ls -lrta
total 12
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 _temporary
drwxrwxrwx 3 tomcat tomcat 4096 Mar 7 20:01 ..
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 .
[root#ip file.json]# cd _temporary/
[root#ip _temporary]# ls -lrta
total 12
drwxr-xr-x 2 tomcat tomcat 4096 Mar 7 20:01 0
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 ..
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 .
[root#ip _temporary]# cd 0/
[root#ip 0]# ls -lrta
total 8
drwxr-xr-x 3 tomcat tomcat 4096 Mar 7 20:01 ..
drwxr-xr-x 2 tomcat tomcat 4096 Mar 7 20:01 .
The exception in catalina.out
Caused by: java.io.IOException: Mkdirs failed to create file:/opt/folder/tmp/file.json/_temporary/0/_temporary/attempt_201603072001_0001_m_000000_5
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:799)
at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
saveAsTextFile is really processed by Spark executors. Depending on your Spark setup, Spark executors may run as a different user than your Spark application driver. I guess the spark application driver prepares the directory for the job fine, but then the executors running as a different user have no rights to write in that directory.
Changing to 777 won't help, because permissions are not inherited by child dirs, so you'd get 755 anyways.
Try running your Spark application as the same user that runs your Spark.
I suggest to try changing to 777 temporarily . See if it works at that point. There have been bugs/issues wrt permissions on local file system. If that still does not work let us know if anything changed or precisely same result.
I also had the same problem, and my issue has been resolved by using full HDFS path:
Error
Caused by: java.io.IOException: Mkdirs failed to create file:/QA/Gajendra/SparkAutomation/Source/_temporary/0/_temporary/attempt_20180616221100_0002_m_000000_0 (exists=false, cwd=file:/home/gajendra/LiClipse Workspace/SpakAggAutomation)
Solution
Use full HDFS path with hdfs://localhost:54310/<filePath>
hdfs://localhost:54310/QA/Gajendra/SparkAutomation
Could it be selinux/apparmor that plays you a trick? Check with ls -Z and system logs.
So, I've been experiencing the same issue, with my setup there is no HDFS and Spark is running in stand-alone mode. I haven't been able to save spark dataframes to an NFS share using the native Spark methods. The process runs as a local user, and I try to write to the users home folder. Even when creating a subfolder with 777 I cannot write to the folder.
The workaround for this is to convert the dataframe with toPandas() and after that to_csv(). This magically works.
I have the same issue as yours.
I also did not want to write to hdfs but to a local memory share.
After some research, I found that for my case the reason is: there are several nodes executing, however, some of the nodes has no access to the directory where you want to write your data.
Thus the solution is to make the directory available to all nodes, and then it works~
We need to run the application in local mode.
val spark = SparkSession
.builder()
.config("spark.master", "local")
.appName("applicationName")
.getOrCreate()
Giving the full path works for me.
Example:
file:/Users/yourname/Documents/electric-chargepoint-2017-data
this is tricky one, but simple to solve. One must configure job.local.dir variable to point to working directory. Following code works fine with writing CSV file:
def xmlConvert(spark):
etl_time = time.time()
df = spark.read.format('com.databricks.spark.xml').options(rowTag='HistoricalTextData').load(
'/home/zangetsu/proj/prometheus-core/demo/demo-1-iot-predictive-maintainance/dataset/train/')
df = df.withColumn("TimeStamp", df["TimeStamp"].cast("timestamp")).groupBy("TimeStamp").pivot("TagName").sum(
"TagValue").na.fill(0)
df.repartition(1).write.csv(
path="/home/zangetsu/proj/prometheus-core/demo/demo-1-iot-predictive-maintainance/result/",
mode="overwrite",
header=True,
sep=",")
print("Time taken to do xml transformation: --- %s seconds ---" % (time.time() - etl_time))
if __name__ == '__main__':
spark = SparkSession \
.builder \
.appName('XML ETL') \
.master("local[*]") \
.config('job.local.dir', '/home/zangetsu/proj/prometheus-core/demo/demo-1-iot-predictive-maintainance') \
.config('spark.driver.memory','64g') \
.config('spark.debug.maxToStringFields','200') \
.config('spark.jars.packages', 'com.databricks:spark-xml_2.11:0.5.0') \
.getOrCreate()
print('Session created')
try:
xmlConvert(spark)
finally:
spark.stop()

AWS Elastic Beanstalk option_settings not applied

I am deploying a Java SE app to Elastic Beanstalk and want to ensure that the option_settings specified in my .ebextensions/otions.config file are applied as described in the docs. I want to adjust the ELB and ASG settings:
option_settings:
- namespace: aws:elb:policies
option_name: ConnectionSettingIdleTimeout
value: 120
- namespace: aws:autoscaling:launchconfiguration
option_name: InstanceType
value: t2.nano
I include this file in the artifact I am deploying to beanstalk. I am deploying a app.zip file with the following structure:
$ unzip -l app.zip
...
72 12-17-15 18:17 Procfile
83199508 12-17-15 18:17 app.jar
0 12-17-15 18:17 .ebextensions/
209 12-17-15 18:17 .ebextensions/options.config
I am using the eb CLI to create/terminate/update my EBS app:
$ eb terminate
$ # .. create app.zip file
$ eb create
$ eb deploy
However, when I create the EBS environment and/or update it, neither the ELB nor the ASG configuration get applied. Is the file at the wrong place? Do you need to change the way I am deploying this to EBS to apply the config properly? My app it

Trouble running UPNP on Docker

I trying to run an UPnP service on my docker container using the Cling UPNP library (http://4thline.org/projects/cling/). There is a simple program that creates a device (in software) that hosts some service. This is written in Java and when I try to run the program I get the following exception (Note: This runs perfectly fine directly on my Ubuntu machine):
Jun 5, 2015 1:47:24 AM org.teleal.cling.UpnpServiceImpl <init>
INFO: >>> Starting UPnP service...
Jun 5, 2015 1:47:24 AM org.teleal.cling.UpnpServiceImpl <init>
INFO: Using configuration: org.teleal.cling.DefaultUpnpServiceConfiguration
Jun 5, 2015 1:47:24 AM org.teleal.cling.transport.RouterImpl <init>
INFO: Creating Router: org.teleal.cling.transport.RouterImpl
Exception occured: org.teleal.cling.transport.spi.InitializationException: Could not discover any bindable network interfaces and/or addresses
org.teleal.cling.transport.spi.InitializationException: **Could not discover any bindable network interfaces and/or addresses
at org.teleal.cling.transport.impl.NetworkAddressFactoryImpl.<init>(NetworkAddressFactoryImpl.java:99)**
For anyone that finds this and needs the answer.
Your container is obscuring your external network. In other words, by default containers are isolated and cannot see the outer network which is of course required in order to open the ports in your IGD.
You can run your container as a "host" to make it non isolated, simply add --network host to your container creation command.
Example (taken from https://docs.docker.com/network/network-tutorial-host/#procedure):
docker run --rm -d --network host --name my_nginx nginx
I have tested this using docker-compose.yml which looks a bit different:
version: "3.4"
services:
upnp:
container_name: upnp
restart: on-failure:10
network_mode: host # works while the container runs
build:
context: .
network: host # works during the build if needed
version 3.4 is very important and the network: host will not work otherwise!
My upnp container Dockerfile looks like so:
FROM alpine:latest
RUN apk update
RUN apk add bash
RUN apk add miniupnpc
RUN mkdir /scripts
WORKDIR /scripts
COPY update_upnp .
RUN chmod a+x ./update_upnp
# cron to update each UPnP entries every 10 minutes
RUN echo -e "*/10\t*\t*\t*\t*\tbash /scripts/update_upnp 8080 TCP" >> /etc/crontabs/root
CMD ["crond", "-f"]
# on start, open needed ports
ENTRYPOINT bash update_upnp 80 TCP && bash update_upnp 8080 TCP
update_upnp script is simply using upnpc (installed as miniupnpc in the Dockerfile above) to open the ports.
Hopefully this will help somebody.
Edit: Here is how update_upnp script may look like:
#!/bin/bash
port=$1
protocol=$2
echo "Updating UPnP entry with port [$port] and protocol [$protocol]"
gateway=$(ip route | head -1 | awk '{print $3}')
echo "Detected gateway is [$gateway]"
# port - e.g. 80
# protocol - TCP or UDP
upnpc -e 'your-app-name' -r $port $protocol
echo "Done updating UPnP entry with port [$port] and protocol [$protocol]"

Categories

Resources