I am starting too many processes in Java using:
Runtime.getRuntime().exec("java -ja myJar.jar")
When it reaches around 350 processes, I get an IO exception:
Cannot run program "java": java.io.IOException: error=24, Too many open files
Exception in creating process
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at java.lang.Runtime.exec(Runtime.java:610)
at java.lang.Runtime.exec(Runtime.java:448)
at java.lang.Runtime.exec(Runtime.java:345)
In each process I am using one database connection.
I am running ubuntu 32 bit OS. But when I run:
ulimit -u
I can see that process limit is unlimited. What could be the problem?
All systems have their limits - sounds like you've hit your system's limit.
In linux, creating new processes consumes lots of inodes (like windows handles), which is a lot like a file handle. The only way around it is to allocate more via kernel settings (I don't know how offhand).
Have you considered starting lots of java Threads instead? They would consume a lot less system resources.
The problem is that you have too many files open, not too many processes in operation. To check the file limit do:
ulimit -n
It will commonly be 1024.
Check http://www.puschitz.com/TuningLinuxForOracle.shtml and search for ulimit for good instructions on changing this limit.
Running 350 JVM instances is not the normal way. Can you redesign and run 350 "main threads" within the same JVM. That is how servlet container works. All web applications are running in the same JVM.
PD. ulimit man pages says to see max. open files is ulimit -n.
Related
I need to run a program after a Linux EC2 machine is provisioned on AWS. The following code will get "Too many open file" error. my_program will open a lot of files, maybe around 5000.
string cmd = "my_program";
Process process = new ProcessBuilder()
.inheritIO()
.command(cmd)
.start();
However, running my_program in the console can finish without any error. What's the ulimit when running the program using ProcessBuilder()...start()?
ulimit -n output 65535 in bash terminal.
First find out the limits your app has when running:
ps -ef | grep <<YOUR-APP-NAME>>
then:
cat /proc/<<PID-of-your-APP>>/limits
Here the problem is that you app. starts under X or Y user and these users have a different ulimit setup.
Check:
cat /etc/security/limits
... I think and increase those values.
Just my 2 cents...
You need to ensure that you close() the files after use. They will be closed by the garbage collector (I'm not completely sure about this, as it can differ on each implementation) but if you process many files without closing them, you can run out of file descriptors right before the garbage collector has any chance to execute.
Another thing you can do is to use a resource based try statement, that will ensure that every resource you declare in the parenthesis group is a Closeable resource that will be forcibly close()d on exit from the try.
Anyway, if you want to rise the value of the maximum number of open files per process, just look at your shell's man page (bash(1) most probably) and search in it about the ulimit command (there's no ulimit manual page as it is an internal command to the shell, the ulimit values are per process, so you cannot start a process to make your process change it's maximum limits)
Beware that linux distributions (well, the most common linux distributions) don't have normally a way to configure an enforced value for this in a per user way (there's a full set of options to do things like this in BSD systems, but linux's login(8) program has not implemented the /etc/login.conf feature to do this) and rising arbitrarily this value can be a security problem in your system if it runs as a multiuser system.
I am trying to create 10000 concurrent udp flows from one computer to another using java. It works for around 3000-4000 but afterwards it gives me the following error.
java.net.SocketException: Too many open files
at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
at java.net.AbstractPlainDatagramSocketImpl.create(AbstractPlainDatagramSocketImpl.java:82)
at java.net.DatagramSocket.createImpl(DatagramSocket.java:322)
at java.net.DatagramSocket.<init>(DatagramSocket.java:179)
at SendThread.<init>(SendThread.java:20)
at Client.main(Client.java:9)
After looking up several other links on the same error it appears that linux has a limit on how many files can be concurrently opened on a single app. That limit on my computer appears to be unlimited.
>ulimit
unlimited
This is my code.
https://copy.com/9YwpvpScPDEUITQ8
Check ulimit -n and update it using ulimit -n <some high number>.
Short description of my problem: I start up Tomcat with my deployed Wicket application. When I want to shut down tomcat I get this error message:
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.ref.Reference.<clinit>(Reference.java:145)
I am running the following setup:
Ubuntu Linux: 10.04 (lucid) with a 2.6.18-028stab094.3 kernel
Java Version: "1.6.0_26" Java HotSpot(TM) 64-Bit Server VM
Tomcat Version: 7.0.23
jvm_args: -Xms512m -Xmx512m -XX:MaxPermSize=205m (these are added via CATALINA_OPTS, nothing else)
Wicket 1.5.1
Tomcat is configured with two virtual hosts on subdomains with ModProxy
My application is deployed as ROOT.war in the appbase directory (it makes no difference if I deploy one or both applications)
'''No application deployed does not result in OOM on shutdown''', unless I mess around with the jvm args
The size of the war is about 500k, all libraries are deployed in tomcat/common/lib (directory which I added to common.loader in conf/catalina.properties)
ulimit -u -> unlimited
When I check the Tomcat manager app it says the following about the JVM memory:
Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB
(http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1
'top' or 'free -m' is similar:
Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers
20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java
I tried to start jmap to get a dump of the heap, it also fails with an OutOfMemoryError. Actually as long as one or both of my applications are deployed any other java process fails with the same OOM Error (see top).
The problem occurs while the application is deployed. So something is seriously wrong with it. However the application is actually running smoothly for quite a while. But I have seen OOMs in the application as well, so I don't trust the calm.
My application is using a custom filter class? Could that be it?
For completeness (hopefully), here's the list of libraries in my common/lib:
activation-1.1.jar
antlr-2.7.6.jar
antlr-runtime-3.3.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
c3p0-0.9.1.1.jar
commons-collections-3.1.jar
commons-email-1.2.jar
dependencies-provided.tgz
dom4j-1.6.1.jar
ejb3-persistence-1.0.2.GA.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
hibernate-annotations-3.4.0.GA.jar
hibernate-commons-annotations-3.1.0.GA.jar
hibernate-core-3.3.0.SP1.jar
hibernate-entitymanager-3.4.0.GA.jar
hibernate-search-3.1.0.GA.jar
javassist-3.4.GA.jar
joda-time-1.6.2.jar
jta-1.1.jar
log4j-1.2.16.jar
lombok-0.9.3.jar
lucene-core-2.4.0.jar
mail-1.4.1.jar
mysql-connector-java-5.1.14.jar
persistence-api-1.0.jar
quartz-2.1.1.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
stringtemplate-4.0.2.jar
wicket-auth-roles-1.5.1.jar
wicket-core-1.5.1.jar
wicket-datetime-1.5.1.jar
wicket-extensions-1.5.1.jar
wicket-request-1.5.1.jar
wicket-util-1.5.1.jar
xml-apis-1.0.b2.jar
I appreciate any hint or even speculation that gives me additional ideas what to try.
Update: I tested some more and found that this behaviour only occurs while one or both of my applications are deployed. The behaviour does not occur on "empty" tomcat (that was a mistake on my part messing with jvm args)
Update2: I am currently experimenting trying to reproduce this behaviour in a virtual box, I want to debug this with a profiler. I am still not convinved that it should be impossible to run my setup on 2GB RAM.
Update3 (10/01/12): I am trying to run jenkins instead of my own application. Same behaviour, so it is definitely server configuration issues. Jenkins jobs fail when maven is called, so I need not even try the shutdown hack suggested below because I need a second java process while running Jenkins. It was suggested to me that because this is a Virtual Server ulimits may be imposed from outside and I would not be able to see them. I think I'll ask a new question regarding this. Thx all.
Update4 (02/05/12): see below for the answer that contains the hint. I'll clarify some more up here: I am now 95% sure that the errors occur because I am reaching my thread limit. However because this is a virtual server the method described below would not work to check this value because it is not visible with ulimit, that was what was confusing me and only today I found out that this is the "numproc" value that I can see in the Parallels Power Panel that I can log into for my virtual server. There were Resource Alerts for numproc but I did not see those either until just now. The value has a hard limit of 96 which I cannot change of course. The current value of numproc corresponds to the number of processes I see with "top" after toggling "H" to see threads. I had a very hard time finding this because this numproc value is hidden deep inside the panel. Sadly 96 is a rather low number if you want to run a tomcat with apache and mysql. I am also very sad that I cannot even find this value in the small print of my hosting contract and it is rather relevant to my application I dare say. So I guess I'll need a server upgrade.
Thanks all for your helpful answers in the end everyone helped me a bit to find out what the problem was.
The tomcat shutdown procedure consits of sending an command/word via a tcp port to the running tomcat VM. This port is configured in the server.xml (if I remember corretly, writting on my phone right now). So far so good.
Unfortunately, the shutdown script does this by starting a 2. VM using the same java options used for the tomcat. Your system simply has not enough memory for this.
As a sollution you could write your own stop script using telnet or something.
I could help with later if needed.
Hope that helps.
Viele grüsse Bert
Seems you have too many threads open.
Use this command :
ulimit -u
What is the result ?
Should be something like :
max user processes (-u) 100
If this is correct, you can edit this file :
/etc/security/limits.conf
and the the following modifications :
#<domain> <type> <item> <value>
user soft nproc 10000
user hard nproc 10000
You can probably survive for a while like this. All you need to do is kill the tomcat process whenever you need to restart it. It is not a nice approach, but the main concern is that your application runs correctly.
It seems to me though, that on the long run, you might need to order a hosting plan with more RAM available.
I was having a similar problem with a tomcat installation just last week. I managed to fix it by giving tomcat a smaller heap. Something like this:
export CATALINA_OPTS=”-Xms256m -Xmx512m”
Before starting Tomcat may help. In the meantime you'll have to kill it the old fashioned way, with a kill -9 ;)
EDIT: you could also take look here, it appears tomcat automatically creates a bunch of "spare" threads, but you can limit those as well as your max thread count in the config. Hope it helps.
When I run PowerShell in a remote session (etsn {servername}), I sometimes can't seem to run Java processes, even the most simple:
[chi-queuing]: PS C:\temp> java -cp .\hello.jar Hello
Error occurred during initialization of VM
Could not reserve enough space for object heap
Hello.jar is an "Hello, world!" application that should just print "Hello" to standard output.
So, the question is, is there something special about running processes on the other side of a PowerShell session? Is there something special about how the Java VM works that might not allow treatment like this? The memory is allocated on the remote computer, right? Here is a readout on the physical memory available:
[chi-queuing]: PS C:\temp> $mem = Get-wmiobject -class Win32_OperatingSystem
[chi-queuing]: PS C:\temp> $mem.FreePhysicalMemory
1013000
But, when I remote desktop to the server and ask the OS how much free memory there is, it says 270 MB physical memory free. Let me know what you think!
According to this:
http://msdn.microsoft.com/en-us/library/aa384372(VS.85).aspx
MaxMemoryPerShellMB
Specifies the maximum amount of memory allocated per shell, including the shell's child processes. The default is 150 MB.
Increase Max Memory Per Shell MB
winrm set winrm/config/winrs '#{MaxMemoryPerShellMB="1000"}'
I have a different answer to share with you guys. I found myself in the same situation and increasing memory min/max for Java.exe or using winrm did NOT solve my issue.
I compared two servers: one working and one not working.
I used this link https://technet.microsoft.com/en-us/library/ff520073%28v=ws.10%29.aspx to check my Windows Management Foundation wich is needed to run WINRS and also remote powershell.
the result: Both servers running Windows Server 2008 R2. One server running WMF 2.0, one running WMF 3.0.
To my surprise, the server running 2.0 was working and the one running 3.0 was NOT!
My solution: I upgraded the 3.0 WMF to 4.0!
Just a fyi: we suffered the same symptoms, and had an endless investigation based on the other two answers.
The actual solution for us was changing jdk1.8.0_31 to jdk1.8.0_51.
I'm trying to debug a file descriptor leak in a Java webapp running in Jetty 7.0.1 on Linux.
The app had been happily running for a month or so when requests started to fail due to too many open files, and Jetty had to be restarted.
java.io.IOException: Cannot run program [external program]: java.io.IOException: error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)
at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:246)
At first I thought the issue was with the code that launches the external program, but it's using commons-exec and I don't see anything wrong with it:
CommandLine command = new CommandLine("/path/to/command")
.addArgument("...");
ByteArrayOutputStream errorBuffer = new ByteArrayOutputStream();
Executor executor = new DefaultExecutor();
executor.setWatchdog(new ExecuteWatchdog(PROCESS_TIMEOUT));
executor.setStreamHandler(new PumpStreamHandler(null, errorBuffer));
try {
executor.execute(command);
} catch (ExecuteException executeException) {
if (executeException.getExitValue() == EXIT_CODE_TIMEOUT) {
throw new MyCommandException("timeout");
} else {
throw new MyCommandException(errorBuffer.toString("UTF-8"));
}
}
Listing open files on the server I can see a high number of FIFOs:
# lsof -u jetty
...
java 524 jetty 218w FIFO 0,6 0t0 19404236 pipe
java 524 jetty 219r FIFO 0,6 0t0 19404008 pipe
java 524 jetty 220r FIFO 0,6 0t0 19404237 pipe
java 524 jetty 222r FIFO 0,6 0t0 19404238 pipe
when Jetty starts there are just 10 FIFOs, after a few days there are hundreds of them.
I know it's a bit vague at this stage, but do you have any suggestions on where to look next, or how to get more detailed info about those file descriptors?
The problem comes from your Java application (or a library you are using).
First, you should read the entire outputs (Google for StreamGobbler), and pronto!
Javadoc says:
The parent process uses these streams
to feed input to and get output from
the subprocess. Because some native
platforms only provide limited buffer
size for standard input and output
streams, failure to promptly write the
input stream or read the output stream
of the subprocess may cause the
subprocess to block, and even
deadlock.
Secondly, waitFor() your process to terminate.
You then should close the input, output and error streams.
Finally destroy() your Process.
My sources:
http://stuffthathappens.com/blog/2007/11/28/crash-boom-too-many-open-files/
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4
http://kylecartmell.com/?p=9
As you are running on Linux I suspect you are running out of file descriptors. Check out ulimit. Here is an article that describes the problem: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Aside from looking into root cause issues like file leaks, etc. in order to do a legitimate increase the "open files" limit and have that persist across reboots, consider editing
/etc/security/limits.conf
by adding something like this
jetty soft nofile 2048
jetty hard nofile 4096
where "jetty" is the username in this case. For more details on limits.conf, see http://linux.die.net/man/5/limits.conf
log off and then log in again and run
ulimit -n
to verify that the change has taken place. New processes by this user should now comply with this change. This link seems to describe how to apply the limit on already running processes but I have not tried it.
The default limit 1024 can be too low for large Java applications.
Don't know the nature of your app, but I have seen this error manifested multiple times because of a connection pool leak, so that would be worth checking out. On Linux, socket connections consume file descriptors as well as file system files. Just a thought.
You can handle the fds yourself. The exec in java returns a Process object. Intermittently check if the process is still running. Once it has completed close the processes STDERR, STDIN, and STDOUT streams (e.g. proc.getErrorStream.close()). That will mitigate the leaks.
This problem comes when you are writing data in many files simultaneously and your Operating System has a fixed limit of Open files. In Linux, you can increase the limit of open files.
https://www.tecmint.com/increase-set-open-file-limits-in-linux/
How do I change the number of open files limit in Linux?