Good day everybody!
I'm developing java GWT web application. Yesterday it was working fine - task manager was showing netbeans process and ONE java process - definetely it was tomcat. But today I'm observing netbeans process, java process of tomcat and some unknown java process which causes java heap space error. This strange process eats a lot of memory and it's memory consumption grows dramatically in time.
Probably useful information: the only thing I changed in my app is dropping database and creating it again from some backup. I suspect java JDBC driver can't connect to DB because of probable incorrect user privileges - it is not a problem, queries are performing successfully but strange java process is exists.
Question: How to define host of this unknown java-process? What application, netbeans or tomcat or something else creates it?
On a Unix platform, ps has several options that show more than just the process name ("java") - e.g. on Linux try ps ax | grep java and you'll see the whole command line that was used to start the java process. It's easy to determine from there what process is running and what they're supposed to do.
On Windows you'll have to find an equivalent - if you're lucky the user executing the process will help you as well - e.g. if it's you or SYSTEM (for services), but the full commandline definitely beats it.
OK, I found the reason - I select a lot of data from my DB. It seems JDBC driver was loading incoming data continuously until memory was enough.
Related
I've made several java software's using hibernate + Mysql/Sqlite and I have noticed a significant difference in fetch time (from database) when using both launchers.
When I start my application using java.exe, of course a console is displayed but the application is faster than when I use javaw.exe .
My application is a windows based application, and I don't want java.exe to load the console when the application starts, so I've been using javaw.exe, but it lacks in fetch time.
What's the explanation of that issue? And how can I either start my application using java.exe without starting a console, Or start it using javaw.exe and have the same performances?
Thanks in advance.
Found the problem using VisualVM (profiler).
Found out that there was logging instructions which were taking much time.
Removed all logs as below:
LogManager.getLogManager().getLogger("").setLevel(Level.OFF);
Thanks to fl0w.
I am using Cassandra DB in my java application. Am using Thrift client to connect Cassandra from my java application. If the Cassandra disk get full means it automatically terminates. So from my java program i could not find the correct error why the Cassandra is down.
So how to avoid the auto termination of Cassandra or is their any way to identify the disk full error ?
Also i dont have physical access to cassandra drive. Its running in some other remote machine.
Disk errors and, in general, generic hardware/system errors are not usually properly handled in any application. The database should only provide as much durability as possible in such scenarios and it is the correct behavior - shut down and break as little as possible.
As for your application - if you can not connect to the database, there is no difference as to what caused an error. You app will not work anyway.
There are special tools that can monitor your machine, i.e. Nagios. If you are the administrator of that server, use such applications. When the disk is getting filled up you will receive an email or text. Use such tools and don't break an open door by implementing several hundred of lines of code to handle random and very rare situations.
Setup ssh access to Casandra machine and use some ssh client like JSch to run df /casandra/drive (if Linux) or fsutil volume diskfree c:\casandra\drive (if Windows) from your Java client. Capture output that is simple and parse to obtain the free disk space. That way your application will monitor that is happening there and probably should alert the user and refuse to add data if there is an out of disk space threat.
You can also use standard monitoring tools or setup server side script to send the message if the disk space low. However this will not stop your application from crashing, you need to take actions after you see that the disk space is low.
Yesterday evening I left the office with a running Java program written by me. It should insert a lot of records into our company database (Oracle) using a JDBC connection. This morning when I came back to work I saw this error (caught by a try-catch):
java.sql.SQLRecoverableException: I/O Exception: Connection reset
The program wrote almost all records before getting this problem, but what if it happens early (just minutes after I leave the office at evening)? I cannot understand what happened, I contacted my database admin and he said there was no particular issue on the database.
Any idea on what happened and what can I do do to avoid it?
The error occurs on some RedHat distributions. The only thing you need to do is to run your application with parameter java.security.egd=file:///dev/urandom:
java -Djava.security.egd=file:///dev/urandom [your command]
I want to produce a complementary answer of nacho-soriano's solution ...
I recently search to solve a problem where a Java written application (a Talend© ELT job in fact) want to connect to an Oracle database (11g and over) then randomly fail. OS is both RedHat Enterprise and CentOS. Job run very quily in time (no more than half a minute) and occur very often (approximately one run each 5 minutes).
Some times, during night-time as work-time, during database intensive-work usage as lazy work usage, in just a word randomly, connection fail with this message:
Exception in component tOracleConnection_1
java.sql.SQLRecoverableException: Io exception: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:229)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:458)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:465)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
and StackTrace follow ...
##Problem explanation:##
As detailed here
Oracle connection needs some random numbers to assume a good level of security. Linux random number generator produce some numbers bases keyboard and mouse activity (among others) and place them in a stack. You will grant me, on a server, there is not a big amount of such activity. So it can occur that softwares use more random number than generator can produce.
When the pool is empty, reads from /dev/random will block until additional environmental noise is gathered. And Oracle connection fall in timeout (60 seconds by default).
##Solution 1 - Specific for one app solution##
The solution is to give add two parameters given to the JVM while starting:
-Djava.security.egd=file:/dev/./urandom
-Dsecurerandom.source=file:/dev/./urandom
Note: the '/./' is important, do not drop it !
So the launch command line could be:
java -Djava.security.egd=file:/dev/./urandom -Dsecurerandom.source=file:/dev/./urandom -cp <classpath directives> appMainClass <app options and parameters>
One drawback of this solution is that numbers generated are a little less secure as randomness is impacted. If you don't work in a military or secret related industry this solution can be your.
##Solution 2 - General Java JVM solution##
As explained here
Both directives given in solution 1 can be put in Java security setting file.
Take a look at $JAVA_HOME/jre/lib/security/java.security
Change the line
securerandom.source=file:/dev/random
to
securerandom.source=file:/dev/urandom
Change is effective immediately for new running applications.
As for solution #1, one drawback of this solution is that numbers generated are a little less secure as randomness is impacted. This time, it's a global JVM impact. As for solution #1, if you don't work in a military or secret related industry this solution can be your.
We ideally should use "file:/dev/./urandom" after Java 5 as previous path will again point to /dev/random.
Reported Bug : https://bugs.openjdk.java.net/browse/JDK-6202721
##Solution 3 - Hardware solution##
Disclamer: I'm not linked to any of hardware vendor or product ...
If your need is to reach a high quality randomness level, you can replace your Linux random number generator software by a piece of hardware.
Some information are available here.
Regards
Thomas
This simply means that something in the backend ( DBMS ) decided to stop working due to unavailability of resources etc.
It has nothing to do with your code or the number of inserts.
You can read more about similar problems here:
http://kr.forums.oracle.com/forums/thread.jspa?threadID=941911
http://forums.oracle.com/forums/thread.jspa?messageID=3800354
This may not answer your question, but you will get an idea of why it might be happening. You could further discuss with your DBA and see if there is something specific in your case.
Solution
Change the setup for your application, so you this parameter[-Djava.security.egd=file:/dev/../dev/urandom] next to the java command:
java -Djava.security.egd=file:/dev/../dev/urandom [your command]
Ref :- https://community.oracle.com/thread/943911
We experienced these errors intermittently after upgraded from 11g to 12c and our java was on 1.6.
The fix for us was to upgrade java and jdbc from 6 to 7
export JAVA_HOME='/usr/java1.7'
export CLASSPATH=/u01/app/oracle/product/12.1.0/dbhome_1/jdbc/libojdbc7.jar:$CLASSPATH
Several days later, still intermittent connection resets.
We ended up removing all the java 7 above. Java 6 was fine. The problem was fixed by adding this to our user bash_profile.
Our groovy scripts that were experiencing the error were using /dev/random on our batch VM server. Below forced java and groovy to use /dev/urandom.
export JAVA_OPTS=" $JAVA_OPTS -Djava.security.egd=file:///dev/urandom "
Your exception says it all "Connection reset".
The connection between your java process and the db server was lost, which could have happened for almost any reason(like network issues). The SQLRecoverableException just means that its recoverable, but the root cause is connection reset.
I had a similar situation when reading from Oracle in a Spark job. This connection reset error was caused by an incompatibility between the Oracle server and the JDBC driver used. Worth checking it.
add java security in your run command
java -jar -Djava.security.egd="file:///dev/urandom" yourjarfilename.jar
When I run PowerShell in a remote session (etsn {servername}), I sometimes can't seem to run Java processes, even the most simple:
[chi-queuing]: PS C:\temp> java -cp .\hello.jar Hello
Error occurred during initialization of VM
Could not reserve enough space for object heap
Hello.jar is an "Hello, world!" application that should just print "Hello" to standard output.
So, the question is, is there something special about running processes on the other side of a PowerShell session? Is there something special about how the Java VM works that might not allow treatment like this? The memory is allocated on the remote computer, right? Here is a readout on the physical memory available:
[chi-queuing]: PS C:\temp> $mem = Get-wmiobject -class Win32_OperatingSystem
[chi-queuing]: PS C:\temp> $mem.FreePhysicalMemory
1013000
But, when I remote desktop to the server and ask the OS how much free memory there is, it says 270 MB physical memory free. Let me know what you think!
According to this:
http://msdn.microsoft.com/en-us/library/aa384372(VS.85).aspx
MaxMemoryPerShellMB
Specifies the maximum amount of memory allocated per shell, including the shell's child processes. The default is 150 MB.
Increase Max Memory Per Shell MB
winrm set winrm/config/winrs '#{MaxMemoryPerShellMB="1000"}'
I have a different answer to share with you guys. I found myself in the same situation and increasing memory min/max for Java.exe or using winrm did NOT solve my issue.
I compared two servers: one working and one not working.
I used this link https://technet.microsoft.com/en-us/library/ff520073%28v=ws.10%29.aspx to check my Windows Management Foundation wich is needed to run WINRS and also remote powershell.
the result: Both servers running Windows Server 2008 R2. One server running WMF 2.0, one running WMF 3.0.
To my surprise, the server running 2.0 was working and the one running 3.0 was NOT!
My solution: I upgraded the 3.0 WMF to 4.0!
Just a fyi: we suffered the same symptoms, and had an endless investigation based on the other two answers.
The actual solution for us was changing jdk1.8.0_31 to jdk1.8.0_51.
I'm trying to run jstack command on my java application. Application is rather big, running inside jboss AS occupying about 4gb of memory. OS is Windows Server 2003 Standard edition. Every time i get an error "Not enough storage is available to process this command". There is enough ram, 16gb, and disk space. So, any ideas?
I ran into this recently on Win2008r2 and thought I'd share my solution since it took a while to figure out. Rob's comment about psexec -s is what did it for me.
It appears that on Vista and later jstack doesn't work against services because of the user context. It has nothing to do with memory. I suspect this is the same reason people have seen this problem on 2003 via remote desktop, unless you use the /admin or /console switch on mstsc. As of Vista the tightened security is probably what broke it.
Starting my app from a cmd window worked fine, but that doesn't help me debug our standard install. Enabling the java debug port (for VisualVM, Eclipse or most any Java debugger) requires an app restart, so you lose the state you're probably trying to capture if you don't already have debugging enabled. Starting the service under my user credentials did not work - I was a little surprised at that. But psexec -s runs jstack from the system context, which worked like a charm. Oh, and you'll need to run psexec from an elevated cmd prompt, if UAC is on.
In the past I have seen this when the JVM is running as a Windows Service on Windows 2003.
First, check to see if this is an issue with the TMP directory.
Second, jstack (or the other utilities like jconsole) will not connect to the local process unless it is running in the same session. If the service is running as a specific user, you may be able to connect by logging into the same session. If you are using Remote Desktop, you can connect using "mstsc /admin" (used to be /console) and try to run jstack again. Definitely check to make sure the TMP directory is set properly if this doesn't fix the problem.
If the service is running as LocalSystem, the above procedure probably will not help much. I don't know if there is a way to log into the same session as LocalSystem.
Some other alternatives may be to set the process up for remote monitoring and use jvisualvm (from the server itself or another machine) to connect over a port and do a thread dump.
We had problems running JStack on a Windows machine with even a modest application (1GB). We ended up doing our stack and heap analysis using Netbeans. This seemed to cope with the parsing of dump files a lot better. YMMV.
Give Netbeans a try for profiling - its very good. Note that VisualVM is a cutdown NB profiler and comes with 6u7.
psexec -s jstack PID >> c:\jstack.log perfectly works on the same machine. For the first time it took some time but again I executed with the redirect to file option, it completed with in few seconds.
This is an error message from the underlying O/S. There's not much you can do in your code to deal with this other than catch the exception which is thrown. Boo to Windows for being so limited.
http://technet.microsoft.com/en-us/library/cc978735.aspx