We have a java application which use JBoss 4.2.0, EJB, Hibernate.
In one of our production server, getting OutOfMemory error at regular interval.
We are seeing user defined exceptions in the jboss and server log files. we intentionally throwing this user defined exception from our code when the user hit the particular piece of code. But the number of users and usage is more for this customer. so it throws 400-500 exceptions. After that we are getting a
WARN org.jboss.mq.SpyJMSException: No pong received;
We have already implemented the ExceptionListener to reconnect the topicConnection in case of failure. And also some of the socketexception as below.
2012-10-12 10:33:57,824 ERROR [org.apache.tomcat.util.net.PoolTcpEndpoint] (http-0.0.0.0-8880-Processor1436) Socket error caused by remote host /125.236.40.205 java.net.SocketException: Connection reset
Have analysed the heap dump generated. 4.5 GB memory is occupied by jmsspyobject exception.
We have got the thread dump from jmx console. Do we need to load the copied thread dump to TDA(Thread Dump Analyzer) tool?
Is OOM because of user defined exception in so many numbers? do we need to reduce it ? or user defined exceptions are not cause for the OOM error?
Please find the attached thread log. I am very new to analyze the jboss thread logs. Could anyone help me to do this?
http://www.4shared.com/file/toRiqF1n/perf-info.html
Thanks.
Related
I see these exception
org.hibernate.queryexception could not resolve property
in Dynatrace exception logs thrown from a specific Hibernate query fired through an action performed. I am trying to replicate this error in my local workspace (Eclipse Mars with Websphere 8.5) in order to debug and fix this issue but I don't get this error in my server logs. I have made hibernate.show_sql = true in hibernate.cfg.xml, but this only prints the HQL statements. Is there some other properties that I would have to set in order see this exception in my server logs?
Dynatrace will also capture Exceptions that dont make it to your log files as Dynatrace captures Exceptions when Exception objects are created and not when they are logged to disk. This is why you typically see more Exceptions in Dynatrace than in log files.
What you could do is to use Dynatrace on your local workstation. There is a free for life version for local workstations - https://www.dynatrace.com/en/products/dynatrace-personal-license.html?utm_medium=blog&utm_source=dynatrace&utm_campaign=devops&utm_term=agrabner
Andi
I have a java1.6 gateway application runs on aix. During the day millions of messages passing through successfully but when I look at logs I see a few logs like stacktrace below.
log1:
java.net.SocketException: Invalid argument
at java.io.DataOutputStream.write(DataOutputStream.java:119)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:921)
at org.mule.providers.http.HttpServerConnection.writeResponse(HttpServerConnection.java:223)
at com.ibtech.smg.esb.providers.esb.ESBHttpMessageReceiver$NewHttpWorker.run(ESBHttpMessageReceiver.java:162)
at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
log2:
java.net.SocketException: Invalid argument
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:125)
at java.net.SocketOutputStream.write(SocketOutputStream.java:171)
at java.io.DataOutputStream.write(DataOutputStream.java:119)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:304)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:308)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:154)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:288)
at java.io.BufferedWriter.flush(BufferedWriter.java:266)
at java.io.FilterWriter.flush(FilterWriter.java:112)
at org.mule.providers.http.ResponseWriter.flush(ResponseWriter.java:75)
I have searched internet but none of the trace or case seem to be fit with mine.
I have tried to generate situation in my development environment (on windows.) Tried big messages, closed socket situations, 0 size messages, but no luck. Can not see the same error. Do anybody has any clue why this error can occur ? This is a bug in application or os bug ?
The problem (java.net.SocketException: Invalid argument) occurs when you try to write a buffer larger than 64K using SocketOutputStream.write().
When you use SocketOutputStream.write(), remenber use SocketOutputStream.flush() to clear SocketOutputStream.
Another situation, connection has closed by server.
(use command 'netstat' to check status of your connection, will be 'CLOSE_WAIT')
When you using SocketOutputStream.write(), log also dispaly "java.net.SocketException: Invalid argument".
REF_DOC
I am making load test for my web application, and I set the connect timeout and response timeout to be 20 seconds, and sometimes I am getting exceptions like:
Non HTTP response code: java.net.SocketTimeoutException: Non HTTP response message: Read timed out
I get the above exception in JMeter test result, and there are no errors thrown in my application, and no stacktrace available to trace the exception.
I want to discover the cause of the message: is the app waiting for something to process, or slow SQL, or hung thread, or application is refusing connections because maxed out?
How can I find why this exception is thrown, and how can I fix it?
I would recommend you improve debugging activities (in jMeter) in the following way:
try to add debug postprocessor to your http sampler
launch log viewer (jMeter log viewer before any http
sampler(-s)/script(-s) execution
probably this could help you to figure out why you're facing socket timeout.
Hope this helps you.
OK, somehow in the log of the server I was not getting the proper error, but after trying different things I got an OutOfMemory:PermGenSpace error, and for that one stackoverflow already has a solution.
Dealing with "java.lang.OutOfMemoryError: PermGen space" error
I use the solution there and my problem is solved :)
Thanks #maximdim for your help
I've been struggling with an issue for two weeks.
I am connecting to test.salesforce.com through a web service in a Java web application using jdk7.
I generated the stubs with JAX-WS wsimport.
I am using STS with VMWare vFabric tc Server v2.6 in my local environment, here connection works fine.
The problem is when I deploy to the test server which is SpringSource tc Runtime 7.0 with jdk 7 I get the following exception after doing the web service call:
Exception in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception in thread "RMI TCP Connection(idle)" Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
I have already incremented the memory parameters in the test server. It is starting with 1 GB or memory. I am giving more detail on the way memory has been increased:
vFabric server has a console, so we have the following configuration there:
Min Heap Size: 1,000MB
Max Heap Size: 16,000 MB
Thread Stack Sie: 192 KB.
I also found the file where these parameters are set (setenv.sh) and they are like this:
JVM_OPTS="-Xms1000m -Xmx16000m -Xss192k -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n"
I also know that is not a connectivity issue because sometimes the connection is made successfully but after that I get the error.
I also got a dump of the memory after the OutOfMemoryError and analyze it with Eclipse plug in and the memory leak is in:
com.sun.xml.internal.ws.client.sei.SEIStub
$Proxy51
Tried to post the image here but as I am a new user stackoverflow didn't allow me.
Please HELP!! Any help will be appreciated.
Alan Robles
How exactly did you 'incremented the memory parameters'? There are few different memory regions in JVM so you might increment one of them but the problem is in another - e.g. PermGen vs. Heap.
OK, somehow in the log of the server I was not getting the proper error, but after trying different things I got an OutOfMemory:PermGenSpace error, and for that one stackoverflow already has a solution.
Dealing with "java.lang.OutOfMemoryError: PermGen space" error
I use the solution there and my problem is solved :)
Thanks #maximdim for your help.
I have a few web services on a Weblogic 10 server. Each of these is part of a larger system. Running locally and on our qa environment the system works flawless, replies fast, and as expected. Everything looks to be okay.
Before going into production we're going to stress test the system, thus see how much load we can have before reply time becomes to large. When testing the web services (e.g. using front end or SOAPUI) we hit a certain load (e.g. to many replies per sec or something like that, I'm not sure what exactly triggers the system to fail) we get the error listed below. I haven't got the slightest clue as to why. Seconds later the system replies flawless again, so I'm guessing that it has something to do with the number of requests...
Any ideas or hints is much appreciated! I'm lost here, so please - anything will help.
We're running: Weblogic 10.3.2, Spring 2.5.6 (for architectural reasons we cannot upgrade), Spring-WS 1.5.9 (for architectural reasons we cannot upgrade) and Stripes 1.5.4
<11-11-2011 08:43:58 CET> <Error> <HTTP> <BEA-101017> <[ServletContext#11242741[app:salesoverview-ws-web module:salesoverview-ws-web path:/salesoverview-ws-web spec-version:2.5], request: weblogic.servlet.internal.ServletRequestImpl#1fbbfc5[POST /salesoverview-ws-web/services HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "" User-Agent: Jakarta Commons-HttpClient/3.1 Content-Length: 425]] Root cause of ServletException.
org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error; nested exception is javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:169)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:97)
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
Truncated. see log file for complete stacktrace
Caused By: javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at weblogic.xml.saaj.SOAPMessageImpl.SOAPPart_writeTo(SOAPMessageImpl.java:1011)
at weblogic.xml.saaj.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:816)
at org.springframework.ws.soap.saaj.Saaj13Implementation.writeTo(Saaj13Implementation.java:292)
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:165)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
Truncated. see log file for complete stacktrace
>
By digging BEA-101017 I found a little info about the from the Weblogic error dok - although this doesn't help me:
Error: [context] Root cause of ServletException.
Description: [context] Root cause of ServletException, which the Web
application container caught while servicing the request.
Cause: The Web application container caught an unexpected exception.
Action: Check the exception for the exact error message.
Assuming that the web service from your example doesn't access other web services (and therefore the above trace corresponds to your web service sending the response):
It seems that your web service, via SAAJ, is trying to write to a disconnected (or otherwise unavailable) socket. An usual cause for this is that the client has disconnected while waiting for the server reply.
I'd suggest to:
Check if your client was waiting for too long before receiving the response, that could have caused it to disconnect.
Check if the operating system might be having issues allocating sockets. Use 'netstat' or other monitoring tool (like TCPView on Windows) to check how many sockets are open (most operating system impose limits on the number of sockets allowed per user or globally).
Ensure there are absolutely no network errors during your tests (shouldn't be the case if you are testing on localhost, but otherwise you need to ensure your network devices (routers, switches, other computers) are not dropping connections or packets. Perhaps this is happening when traffic load is high.
Make sure you have no threading conflicts that could cause your web service to use or close other requests' sockets (this would be a rare situation especially if you are using Spring).
Check this thread Official reasons for "Software caused connection abort: socket write error" and other possible causes of "Software caused connection abort" (note that the issue could be specific to your application server and operating system).
Hope that helps.
After debugging a lot I found out that the problem happened due to DB2 issues - we hit a corner of our database, which triggered an internal stack overflow, which then probagated to the Dao and onwards to the SOAP-part (only making it harder to detect due to Spring JDBC templates in the Dao).
A long story short and the issue was an uncaught exception, which by Spring-WS resulted in a "SaajSoapMessageException". The hint came from "Software caused connection abort: socket write error", but happened on the WS side (not client nor the communication between client/server).
Hint: Surround your database with try/catch and catch Exception thus being able to find the exact exception thrown. In my case it threw a DB2 exception ("SQLCode -1218") and this is normally used when you run out of resources (e.g. data source connections). I my case it was the SQL which DB2 didn't like - and really didn't like under load. I can't explain it, but it has to do with DB2s own internal resources - gah, go figure! :)
Thank you jjmontes, for hints and pointers, but it was not the problem in this case.