I'm currently hacking on a new project, a web-app. But something's wrong, and I think it's Vistas fault, when I'm stress-testing the app, not all of the requests are answered.
The only thing I can think of is that the queue of incomming requests is getting too long, I've googled around, but can't find out how long the queue is, only that it depends on OS. Tough this may simply be because I don't know the "real" name for it;)
The project is written in Java SE.
If someone knows the answer I'd be really happy as I can't find it myself anywhere;)
UPDATE: The app is just a pet-project, and I won't be running it on a Vista machine when it's done. I've even been thinking of running it on a Solaris box, just to try Solaris:)
There are no errors or anything, but the request counter is way too low. Testing is done from Opera, with 30 tabs on auto-refresh every one second. I know, it's not the right way to do it, but it works:)
I don't use any frameworks, EE or anything, just pure Java SE.
If you're writing a server app, but running it on Vista, you may be hitting the hard limit for inbound network connections on the Vista OS. I've seen numbers from 5 (for Home Basic) up to 25 (for Business Pro and Ultimate) simultaneous connections allowed.
Vista is not designed as a Server OS. If you want more simultaneous network connections, and you want to run Windows, get a license for one of the Windows Server products. Alternately, you can run on some *NIX system.
Related
I have a number of Java processes using OpenJDK 11 running on Windows Server 2019. The server has two physical processors and 36 total cores; it is an HP machine. When I start my processes, I see work allocation in Task Manager across all the cores. This is good. However after the processes run for some period of time, not a consistent amount of time, the machine begins to only utilize only half the cores.
I am working off a few theories:
The JDK has some problem that is preventing it from consistently accessing all the cores.
Something with Windows Server 2019 is causing a problem, limiting Java from accessing all the cores.
There is a thermal management problem and one processor is getting too hot and the OS is directing all the processing to the other processor.
There is some issue with hyper-threading and the 'logical' processors that is causing the process to not be able to utilize all the cores.
I've tried searching for JDK issues and haven't found anything like this mentioned. I went down to the server and while it's running a little warm, it didn't appear excessively hot. I have not yet tried disabling hyper-threading. I have tried a number of parameters to force the JVM to use all the cores and indeed the process initially does use all the cores; I can see the activity in Task Manager.
Anyone have any thoughts? This is a really baffling problem and I'd appreciate any ideas.
UPDATE: I am able to make it use the other processor by using the Task Manager to assign one of the java.exe processes to the other processor. This is also working from the java invocation on the command line as well with an argument for which socket to use.
Now that said, this feels like a hack. I don't see why I should have to manually assign a socket to each of my java processes; that job should be left to the OS. I'm still not sure exactly where the problem is, if it's the OS or what.
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
I have a scenario with these particular demands:
Production ready & stable.
Point to point connection, with the producer behind a firewall and a consumer in the cloud. It might be possible to split the traffic between a couple of producers\consumers, but all the traffic still has to traverse a single WAN connection which will probably be the bottleneck.
High throughput - something along the order of 300 Mb/sec (may be up to 1Gb!). Message sizes vary from ~1KB to possibly several MBs.
Guaranteed delivery a must - every message has to arrive at the consumer eventually, so we need to start saving messages to disk in the event of a momentary network outage or risk running out of memory.
Message order is not important, messages are timestamped and can be re-arranged at the consumer.
Highly preferable but not as important - should run on both linux & windows (JVM seems the obvious choice)
I've been looking at so many MQs lately, and I don't have any hands-on experience with any.
Thought it will be a better idea to ask someone with experience.
We're considering mostly Kafka, but I'm not sure it's the best for our use case, seems to be tailored to distributed deployment & mutliple topics\consumers\producers. Also, definitely not production ready on windows.
What about Apache ActiveMQ or Apollo\Artemis? RabbitMQ seems not to be a good fit for our performance requirements. Or maybe there's some Java library that has the features we need without a middleman broker?
Any help making sense of this kludge would be greatly appreciated.
If anyone comes across this, we went with Kafka in the end. Its performance is impressive and so far it's very stable on linux. No attempt yet to run it on windows in production deployments.
UPDATE 12/3/2017:
Works fine and very stable on Linux, but on Windows this is not usable in production. Old data never gets deleted due to leaky file handles, the relevant Jira is being ignored since 2013: https://issues.apache.org/jira/browse/KAFKA-1194
So basically the problem is described in the title.
The server works in the following way:
Listens to a new connection
Once connection is requested - adds the request to the Q,
Continues listening to a new connection
Separate process takes care of a Q and spawns a new thread to deal with the clients' requests.
The server code is similar to this tutorial (everything is in try / catch, unfortunately I cant show the source-code - company policy)
It seems to work very well, until the number of clients exceeds ~ 50, Then it just hangs with no exceptions / warnings / etc. There is a cpu thread limit of 32k, no limits on the number of open files / open sockets / etc. OS = CentOS 5.5 (same seems to happen in ubuntu tho). The server logs data to MySQL using ODBC. Separate stress tests of both showed that I can have up to 32k java processes (limited by /proc/sys/kernel/threads-max ) and MySQL can perform up to 20k simple operations / second, so Im assuming the problem is with the sockets.
So the question really is:
What is the limiting factor in socket connections and how can I make it bigger?
OR am I looking in the wrong place?
The chances are that you have induced a deadlock somewhere in the code. The key indicator here is if by 'hang' you mean the CPU usage of the server drops to nothing and no futher activity is seen in the server.
When the server hangs run jdk tool: jstack against it's process. This should show you what is waiting on what lock. Also in the tool kit is jvisualvm and if on a unix box a simple kill -3 pid will do a thread dump to stderr.
With out the code or at least a reproducable sample I'm afraid I can't help much more. One thing you might want to look at is using jetty as your embedded server instead of a hand roled one, they have already been through the deadlock/threading pain so you don't have to.
Don´t know if this will help you and if your are using it, but try to run your socket server with java switch "-server",this will select the Java HotSpot Server VM.The -server turns on the optimizing JIT along with a few other "server-class" settings. Generally you get the best performance out of this setting. The default VM is -client.
Also check your other params, so your socket server don´t run with minimal resources
Have a nice day
I have a java application that I run from eclipse 3.5.
My OS is WinXP(SP2) and the JRE version is 6.05.
I run the application on two identical computers (or so I think) but the application behaves differently on each computer.
The computers are the same Dell Optiplex model with the same amount of memory and have the same GPU.
On the first computer, the application runs flawlessly. However, on the second one the application freezes for a couple of minutes and then returns to run normally.
The strange thing is that the CPU usage on the second computer is not high at all. It seems as though my application does not receive any CPU for no apparent reason.
Computers should be deterministic so I assume there must be some difference between the machines but I don't know where to look.
I would love some ideas on where the problem might be.
Thanks,
Yoav.
I've found the problem.
The application that was unresponsive was run in debug mode.
Sorry to have wasted your time...
It may help you to get a Thread Dump when the app freezes. This will hopefully tell you exactly what is holding you up (i.e. waiting for IO somewhere).
Well, I would first update your JRE version as there are newer versions now.
As for both computers being identical, are they really identical? I find it difficult to believe that both have the same exact software and setup and that anything you have done to one, you have always done to the other. If this is indeed the case, you may want to try to debug your application on the second machine (the one that hangs) and find out specifically where it hangs.
It may also help us if you give more information about your application. The problem may not be your computer at all if the application is doing things like web access, network access, etc.
So both computers have nearly identical hardware. A few other things to check
Do they both have Eclipse 3.5, WinXP(SP2) and JRE 6.05 installed?
And behave differently when run from within Eclipse (on both machines or on one run from command-line)?
Is this reproducible? If yes When does it happen? On startup? Or on some specific action?
Does the program have a GUI?
Is there maybe some kind of virus scanner or another comparable software installed on one of the machines which could delay the program
Is networking, file acccess, multithreading involved?
I can think of two non-application possibilities:
Memory Paging. There's something extra happening on the slow machine, so your JVM is not getting a fair share of CPU time. A large daemon process or some such.
Network access. Your app is making some kind of network call and it's glitching or timeing out. Perhaps going and fetching some XML schema, perhaps a disk acesss to a mounted drive.
I've seen all manner of weirdness when apps attempt to access hosts by name and DNS is not well. One machine has an etc/host entry the other does not. Even each machine might want to resolve itself.