I have a java application that I run from eclipse 3.5.
My OS is WinXP(SP2) and the JRE version is 6.05.
I run the application on two identical computers (or so I think) but the application behaves differently on each computer.
The computers are the same Dell Optiplex model with the same amount of memory and have the same GPU.
On the first computer, the application runs flawlessly. However, on the second one the application freezes for a couple of minutes and then returns to run normally.
The strange thing is that the CPU usage on the second computer is not high at all. It seems as though my application does not receive any CPU for no apparent reason.
Computers should be deterministic so I assume there must be some difference between the machines but I don't know where to look.
I would love some ideas on where the problem might be.
Thanks,
Yoav.
I've found the problem.
The application that was unresponsive was run in debug mode.
Sorry to have wasted your time...
It may help you to get a Thread Dump when the app freezes. This will hopefully tell you exactly what is holding you up (i.e. waiting for IO somewhere).
Well, I would first update your JRE version as there are newer versions now.
As for both computers being identical, are they really identical? I find it difficult to believe that both have the same exact software and setup and that anything you have done to one, you have always done to the other. If this is indeed the case, you may want to try to debug your application on the second machine (the one that hangs) and find out specifically where it hangs.
It may also help us if you give more information about your application. The problem may not be your computer at all if the application is doing things like web access, network access, etc.
So both computers have nearly identical hardware. A few other things to check
Do they both have Eclipse 3.5, WinXP(SP2) and JRE 6.05 installed?
And behave differently when run from within Eclipse (on both machines or on one run from command-line)?
Is this reproducible? If yes When does it happen? On startup? Or on some specific action?
Does the program have a GUI?
Is there maybe some kind of virus scanner or another comparable software installed on one of the machines which could delay the program
Is networking, file acccess, multithreading involved?
I can think of two non-application possibilities:
Memory Paging. There's something extra happening on the slow machine, so your JVM is not getting a fair share of CPU time. A large daemon process or some such.
Network access. Your app is making some kind of network call and it's glitching or timeing out. Perhaps going and fetching some XML schema, perhaps a disk acesss to a mounted drive.
I've seen all manner of weirdness when apps attempt to access hosts by name and DNS is not well. One machine has an etc/host entry the other does not. Even each machine might want to resolve itself.
Related
I have a number of Java processes using OpenJDK 11 running on Windows Server 2019. The server has two physical processors and 36 total cores; it is an HP machine. When I start my processes, I see work allocation in Task Manager across all the cores. This is good. However after the processes run for some period of time, not a consistent amount of time, the machine begins to only utilize only half the cores.
I am working off a few theories:
The JDK has some problem that is preventing it from consistently accessing all the cores.
Something with Windows Server 2019 is causing a problem, limiting Java from accessing all the cores.
There is a thermal management problem and one processor is getting too hot and the OS is directing all the processing to the other processor.
There is some issue with hyper-threading and the 'logical' processors that is causing the process to not be able to utilize all the cores.
I've tried searching for JDK issues and haven't found anything like this mentioned. I went down to the server and while it's running a little warm, it didn't appear excessively hot. I have not yet tried disabling hyper-threading. I have tried a number of parameters to force the JVM to use all the cores and indeed the process initially does use all the cores; I can see the activity in Task Manager.
Anyone have any thoughts? This is a really baffling problem and I'd appreciate any ideas.
UPDATE: I am able to make it use the other processor by using the Task Manager to assign one of the java.exe processes to the other processor. This is also working from the java invocation on the command line as well with an argument for which socket to use.
Now that said, this feels like a hack. I don't see why I should have to manually assign a socket to each of my java processes; that job should be left to the OS. I'm still not sure exactly where the problem is, if it's the OS or what.
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
Scenario 1: I have a test server that gets OS reinstalls on a frequent basis. Is there any way to add a program to the server that will remain and execute even if the OS is reinstalled? (I know it's a stretch, but had to ask)
Scenario 2: I have another server running ESXi 5.1 (which I admit, I know nothing about) how (or can) I run a program at the OS level (not as a VM)? Reason being, I need to get information specific to the server and not the VM's such as ip,MAC address, etc that my program gathers with Runtime.exec().
I have a PXE server setup with kickstart files that work great for linux, but not sure if I can do it with ESX or not, anyone ever try to PXE boot ESX like this? On linux, I run my program via crontab, and in the past did it with rc.local. Any suggestions would be appreciated, even if it is a link to potential resources you have had luck with in similar situations.
1) The program must run within an OS, or the JVM will have been designed to run without an OS. I don't believe there is a JVM which will use an OS if it is there but not care if it is not.
You can do this with virtual machines. You can have the OS run/stop/start/reinstall in a virtual machine, while an application is running on the bare machine or in another virtual machine.
2) When you application runs, it is at the OS level. The distinction is largely an illusion. You can get the IP and MAC addresses in normal Java. If you nee to get something else, you can use JNA/JNI/JNR.
I have not heard of ESX before.
I have been developing a Java application using J2EE and a Derby database. My boss does most of the testing and I do most of the coding, but he has come to me with a strange problem. He claims that occasionally the Java application "crashes his computer".
To mention a few details, first let me say that I am currently working remotely, so I can't be around when these "crashes" happen. Second, I use OS X 10.6 and he uses Windows XP (SP3 I believe). The Java application in question does not use any JNI or anything weird except for an embedded Derby database. Lastly, he said it freezes everything in Windows (his mouse doesn't even move) -- it doesn't show up in the console like an uncaught exception would.
So, is it possible that my Java program is crashing his computer? I didn't think that Java code could have any system-wide effects outside of the JVM. Is this something that could possibly be the fault of my program, or should I just ignore it and attribute it to some problem with his computer?
For a Java application to crash the OS it runs on, there must be a bug in the JVM. That said there are situations that can give the same impression:
the Java application can grow its heap far enough that the OS starts to swap and other applications appear to slow down to a halt
the Java application can grab all CPU by one or more threads in a tight busy loop
If you can setup your testers' machine so that a heap dump can be triggered when the problem occurs, you can analyse that dump remotely. For instance with IBM's Java heap analyzer found on alphaworks.
There were cases before of crashes under windows on IBM Thinkpads (and other machines I am sure) due to a bad graphics driver. I'd suggest doing the usual thing of making sure the drivers are up to date just to be on the safe side.
While your code may not use JNI directly a lot of what happens under the hood can (anything that integrates with the underlying OS essentially). Which means drivers can be a big problem.
The other thing would to be sure that the most recent version of the Java is being used (if 1.6_17 isn't possible at last the latest version of the version of Java being used).
One other thing that has fixed random crashes for me is to re-seat the memory (unplug it and plug it back in).
he said it freezes everything in Windows (his mouse doesn't even move)
A user-mode application — whether Java or otherwise — can't do that against a modern OS like WinNT.
He either has a hardware problem, or a bad driver.
To crash the entire computer, try this:
public void crashComputer() {
while(true)
new Thread(new Runnable() {
#Override
public void run() {
while(true) {
crashComputer();
}
}
}).start();
}
public void crashJVM() {
while(true)
crashJVM();
}
The crashComputer function takes about 2 seconds to crash the entire computer. You can stop it from crashing by holding power button.
The crashJVM function crashed only the JVM by overloading the stack, causing a stack overflow (That's where the name of this website comes from).
WARNING: Use at your own risk. This will not damage your computer, but I'm not taking the blame if you forget to click save on a document or something.
Also check the page file space on the client (Windows/XP) system.
Point one: On my XP, SP3 system, when any program runs full tilt, it almost locks up the computer. When my anti-virus program checks for updates, everything else stops for all practical purposes, and it uses very little CPU (seems to be writing to disk all the time). My own infinite loops, using 100% CPU, have a similar effect. (Why the task manager does not get priority over a user program with "normal" priority I do not know.)
Point two: On any computer I've ever used, heavy paging will bring a program to an almost complete stop.
So start with an XP computer with 512Mb of real memory and 2000Mb of virtual memory. Write a Java program with 1400Mb worth of arrays and other data structures. Put in a loop that repeats several billion times and reads or writes to each byte in that 1400Mb on each execution. This program will not finish until long after the universe collapses in on itself. The computer will do nothing. I have not tried this and I do not intend to, but I will bet anything even the mouse won't move. Depending on the make of the computer, the only fix may be pulling the power plug out of the wall socket. (Note that technically, the computer has not crashed. Indeed, it is working just fine. But you are going to need to be patient. Move the mouse the day before you plan to click.)
The moral of this story is to avoid both XP and virtual memory, but if you have to deal with either, be aware of these problems.
Is it possible to dump an image of a running JVM and later restore the previous state by loading the image into the JVM? I'm fairly certain the answer is negative, but would love to be wrong.
With all the dynamic languages available for the JVM comes an increase in interactivity, being able to save a coding session would help save time manually restoring the VM to a previous session.
There was a JSR 323 proposed for this a while back but it was rejected. You can find some links in those articles about the research behind this and what it would take. It was mostly rejected as an idea that was too immature.
I have heard of at least one startup (unfortunately don't recall the name) that was working on a virtualization technology over a hypervisor (probably Xen) that was getting pretty close to being able to move JVMs, including even things like file system refs and socket endpoints. Because they were at the hypervisor level, they had access to all of that stuff. By hooking that and the JVM, they had most of the pieces. I think they might have gone under though.
The closest thing you can get today is Terracotta, which allows you to cluster a portion of your JVM heap, storing it in a server array, which can be made persistent. On JVM startup, you connect to the cluster and can continue using whatever portions of your heap are specified as clustered. The actual objects are faulted in on an as-needed basis.
Not possible at present. In general, pausing and restarting a memory image of a process in a different context is incredibly hard to achieve: what are you going to do with open OS resources? Transfers to machines with different instruction sets? database connections?
Also images of the running JVM are probably quite large - maybe much larger than the subset of the state you are actually interested in. So it's not a good idea from a performance perspective.
A much better strategy is to have code that persists and recreates the application state: this is relatively feasible with most JVM dynamic languages. I do so similar stuff in Clojure, where you have an interactive environment (REPL) and it is quite possible to create and run a sequence of operations that rebuild the application state that you want in another JVM.
This is currently not possible in any of the JVMs I know. It would not be very difficult to implement something like this in the JVM if programs run disconnected from their environments. However, many programs have hooks into their environment (think file handles, database connections) which would make implementing something like this very hairy.
As of early 2023, there's some progress in this space and it seems a lot of things can at least be tried, even if without claims for their production readiness.
One such feature is called CRaC. You can check their docs or even get an OpenJDK build that includes the feature. The project has its own repo under OpenJDK and looks quite promising.
Another vendors/products to check:
Azul ReadyNow!
OpenJ9 InstantOn
What's also really exciting, is AWS Lambda SnapStart. It doesn't give you full snapshoting capabilities, and is intrinsically vendor-specific, but it's what a ton of Java engineering who use AWS Lambda were waiting for so long.