We have a Java application running on Solaris which makes a connection to Oracle and checks the database for work to perform, and it runs just fine. We tried running the same code on a standalone Fedora system, its performance is good too. However, when we move it to its home on a Fedora VMWare virtual machine, it can take upwards of five minutes for the application to make the connection to the database. It ultimately DOES make the connection - it's just snail-slow. We suspect it's a configuration issue somewhere but can't find it. So far as we can tell, the two Fedora boxes have nearly identical configurations. Has anyone run into this problem before? If so, how did you get around it?
Thanks in advance for your help.
Mike Preston
Found it! When we are running under Solaris, we are running a 32-bit JVM with 32-bit extensions. We are executing through a Korn shell script and had an added -d64 flag to coerce 64-bit processing. On the Linux boxes we removed the -d64 flag from the shell script and everybody's happy. Thanks Alex for your thoughts and assistance.
Here is the solution which settled the issue...our headless development server was only occasionally getting any keyboard activity to fill the entropy pool (please read the article - I won't try to explain it here) and I assume it was blocking until there was enough "noise" to generate the requisite random numbers.Since there is only one other developer working on the system, it might take a couple of minutes to fill the buffer. Once the buffer was full it went ahead and executed the connection as expected. That also explains why we sometimes would see crisp performance followed by slow. In a nutshell, we added the string "-Djava.security.egd=file:///dev/urandom" to the Korn shell script between the call to java and the jar file name and now it works like a champ. Here's the full command string:
/usr/bin/java -Xms64m -Xmx1024m -Djava.security.egd=file:///dev/urandom -jar $1 $2 $PID
If you DO read the article, be sure to read the comments below. One of them is really funny!
Related
I am running a simple script in Groovy on an Ubuntu 11.10 machine, which takes key/value pairs and adds them to a JDBM map in a loop. Every ~3 minutes the script hangs for a couple of minutes and then resumes. When I look at the resource monitor I see that there is no CPU or Memory activity and the process is in futex_wait_queue_me().
Please suggest means to overcome this, on a Windows machine by the way the application runs without the hangs.
Could this be an OS issue? (found many similar threads about similar futex_wait_queue_me() problems in Ubuntu0
Thanks
Please check the version of the kernel. I ran into a similar problem (java and other multithreaded applications) on Centos6 and upgrading the kernel to version 2.6.32-504.16.2.el6.x86_64 solved the issue.
See the centos bug report: https://bugs.centos.org/view.php?id=8703 which contains this pointer to an explanation of the problem:
https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85beb03c75121419a7dec52f0 [^]
My stacktrace was:
cat /proc/23199/stack
[<ffffffff810b226a>] futex_wait_queue_me+0xba/0xf0
[<ffffffff810b33a0>] futex_wait+0x1c0/0x310
[<ffffffff810b4c91>] do_futex+0x121/0xae0
[<ffffffff810b56cb>] sys_futex+0x7b/0x170
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
For anyone interested I used those parameters when running java:
-Xms16384M -Xmx16384M
You can find additional GC optimization tips at http://randomlyrr.blogspot.it/2012/03/java-tuning-in-nutshell-part-1.html
When I run PowerShell in a remote session (etsn {servername}), I sometimes can't seem to run Java processes, even the most simple:
[chi-queuing]: PS C:\temp> java -cp .\hello.jar Hello
Error occurred during initialization of VM
Could not reserve enough space for object heap
Hello.jar is an "Hello, world!" application that should just print "Hello" to standard output.
So, the question is, is there something special about running processes on the other side of a PowerShell session? Is there something special about how the Java VM works that might not allow treatment like this? The memory is allocated on the remote computer, right? Here is a readout on the physical memory available:
[chi-queuing]: PS C:\temp> $mem = Get-wmiobject -class Win32_OperatingSystem
[chi-queuing]: PS C:\temp> $mem.FreePhysicalMemory
1013000
But, when I remote desktop to the server and ask the OS how much free memory there is, it says 270 MB physical memory free. Let me know what you think!
According to this:
http://msdn.microsoft.com/en-us/library/aa384372(VS.85).aspx
MaxMemoryPerShellMB
Specifies the maximum amount of memory allocated per shell, including the shell's child processes. The default is 150 MB.
Increase Max Memory Per Shell MB
winrm set winrm/config/winrs '#{MaxMemoryPerShellMB="1000"}'
I have a different answer to share with you guys. I found myself in the same situation and increasing memory min/max for Java.exe or using winrm did NOT solve my issue.
I compared two servers: one working and one not working.
I used this link https://technet.microsoft.com/en-us/library/ff520073%28v=ws.10%29.aspx to check my Windows Management Foundation wich is needed to run WINRS and also remote powershell.
the result: Both servers running Windows Server 2008 R2. One server running WMF 2.0, one running WMF 3.0.
To my surprise, the server running 2.0 was working and the one running 3.0 was NOT!
My solution: I upgraded the 3.0 WMF to 4.0!
Just a fyi: we suffered the same symptoms, and had an endless investigation based on the other two answers.
The actual solution for us was changing jdk1.8.0_31 to jdk1.8.0_51.
I'm on Windows Vista 64 bit, with a 64 bit jvm installed. I'm trying to use jstack and jmap -- two utilities that come with the JDK -- to peek into an application server's guts. This works fine on a windows xp machine, 32 bit.
However, when I run these commands against the processid for a ColdFusion application server on this vista64 machine, I get the error message in the title of this post.
All I'm doing is running jstack , where pid is the processid of my CF server, and I'm getting this
this machine has plenty of available memory, but I highly doubt it's a memory problem. The reason I say that is that if I start JBoss, which is taking up just as much memory as CF, I can run jstack against that process.
Thanks for advice
Figured it out. The problem was that ColdFusion was running as a windows service. By stopping the service and running from the command line (jrun start cfusion) , I was able to successfully use the JDK tools
this posting provides details on how to execute jstack when the process is running as a windows service. basically, use the psexec command.
Jstack and Not enough storage is available to process this command
I'm trying to run jstack command on my java application. Application is rather big, running inside jboss AS occupying about 4gb of memory. OS is Windows Server 2003 Standard edition. Every time i get an error "Not enough storage is available to process this command". There is enough ram, 16gb, and disk space. So, any ideas?
I ran into this recently on Win2008r2 and thought I'd share my solution since it took a while to figure out. Rob's comment about psexec -s is what did it for me.
It appears that on Vista and later jstack doesn't work against services because of the user context. It has nothing to do with memory. I suspect this is the same reason people have seen this problem on 2003 via remote desktop, unless you use the /admin or /console switch on mstsc. As of Vista the tightened security is probably what broke it.
Starting my app from a cmd window worked fine, but that doesn't help me debug our standard install. Enabling the java debug port (for VisualVM, Eclipse or most any Java debugger) requires an app restart, so you lose the state you're probably trying to capture if you don't already have debugging enabled. Starting the service under my user credentials did not work - I was a little surprised at that. But psexec -s runs jstack from the system context, which worked like a charm. Oh, and you'll need to run psexec from an elevated cmd prompt, if UAC is on.
In the past I have seen this when the JVM is running as a Windows Service on Windows 2003.
First, check to see if this is an issue with the TMP directory.
Second, jstack (or the other utilities like jconsole) will not connect to the local process unless it is running in the same session. If the service is running as a specific user, you may be able to connect by logging into the same session. If you are using Remote Desktop, you can connect using "mstsc /admin" (used to be /console) and try to run jstack again. Definitely check to make sure the TMP directory is set properly if this doesn't fix the problem.
If the service is running as LocalSystem, the above procedure probably will not help much. I don't know if there is a way to log into the same session as LocalSystem.
Some other alternatives may be to set the process up for remote monitoring and use jvisualvm (from the server itself or another machine) to connect over a port and do a thread dump.
We had problems running JStack on a Windows machine with even a modest application (1GB). We ended up doing our stack and heap analysis using Netbeans. This seemed to cope with the parsing of dump files a lot better. YMMV.
Give Netbeans a try for profiling - its very good. Note that VisualVM is a cutdown NB profiler and comes with 6u7.
psexec -s jstack PID >> c:\jstack.log perfectly works on the same machine. For the first time it took some time but again I executed with the redirect to file option, it completed with in few seconds.
This is an error message from the underlying O/S. There's not much you can do in your code to deal with this other than catch the exception which is thrown. Boo to Windows for being so limited.
http://technet.microsoft.com/en-us/library/cc978735.aspx
anyone have any experience using this?
if so, is it worth while?
I just used jdb for the first time yesterday and am really pleased with the results. You see, I program in Eclipse on my laptop, then deploy to a VM to make sure the whole shebang still works. Very occasionaly, I'll have to work on something that gets executed standalone, as a commandline. These things sometimes need debugging.
This has always been a problem, because I don't want to go to the trouble of installing Eclipse on the VM (it's slow enough already!), yet I don't know of an easy way to get it to connect to my commandline-running class before it finishes running.
jdb to the rescue! It works a treat - small and functional, almost to the point where it is bare... this forces you to apply your mind more than you apply the tool (like I said here).
Make sure to print out the reference (solaris, windows, java 1.5 - I think they're all about the same, really) and have your source code open and browsable on your second screen. I hope you have a second screen, or you'll be alt-tabbing a lot.
Assume your program is started by the following command:
java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=<port> <class>
You can attach to this process by jdb:
jdb -attach <port>
In some cases you need to use the following command .
jdb -sourcepath \.src -connect com.sun.jdi.SocketAttach:hostname=localhost,port= <port>
JDB is incredibly difficult to use. Placing System.outs or using an IDE debugger will produce better results. And for the more interesting features (e.g. tracking threads, heap size, etc.), you can get the information graphically with the JConsole tool.