I'm upgrading our application to MQ7 (7.5.0.5) and I'm seeing some odd behavior in a small test application that I have written.
My application uses Springs CachingConnectionFactory and is configured to use only one thread.
I can see that by debugging through the code 2 tcp connections are created, one for the initial connection and one for the JMS session. Every 60 seconds, the 2 tcp connections that are used by my message sink are broken and replaced with 2 new connections.
The following error is present in the error logs on the queue manager.
05/16/2016 09:38:26 AM - Process(1609.14) User(mqm) Program(amqrmppa)
Host(xxxxxxxxx) Installation(Installation1)
VRMF(7.5.0.2) QMgr(xxxxx)
AMQ9271: Channel 'XX.XXXX.X' timed out.
EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'XX.XXX.X'. The address of the remote end of the connection was '57.4.4.145'.
ACTION:
The return code from the (recv) [TIMEOUT] 60 seconds call was 0 (X'0').
Record these values and tell the systems administrator.
I have the following settings on my channel: DISCINT(60), SHARECNV(1), the exceptions are linked to the DISCINT time, changing that changes the frequency of the exceptions, also the
exceptions disappear with a SHARECNV value >1
Can anyone tell my why the connections are broken even when the channel is active and messages are being sent and received?
Thanks!
This sounds like APAR IV62728 which describes the symptoms you're seeing:
http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg1IV62728
Fixed in 7.5.0.6. Try upgrading to that level and see if it solves the problem.
I managed to find a solution to this issue. When using the CachingConnectionFactory with an underlying IBM connection factory, an initial connection is created in a stopped state. That connection in then used to create JMSSessions.
The issue was that the initial common connection was timing out.
I managed to keep the connection active by adjusting the HBINT value to 5. It appears that a number of heatbeats are required to keep the connection open and my initial value of 20 was too high.
Related
I am working on a web application which runs in pcf environment and it has approximately 100 users. I am using Hikari CP library to manage databae connections and customized connectionTimedout property by setting it to 1 sec in the application code. Connection pool size is set to 100.
In one scenario, making a call to stored procedure where I am explicitly creating
Connection = DriverManager.getConnection()
object as ArrayDescriptor() is expecting connection object.
I am using ArrayDescriptor as input for stored procedure requires array of object.
However this code is throwing Socket Read Timed Out error randomly
The same code was working fine when configured with dbcp library managed connection pool.
Can anyone help? What's the problem with Hikari CP library?
As per compliance rules I can't post code on public domains.
connectionTimeout
This property controls the maximum number of milliseconds that a
client (that's you) will wait for a connection from the pool. If this
time is exceeded without a connection becoming available, a
SQLException will be thrown. Lowest acceptable connection timeout is
250 ms. Default: 30000 (30 seconds)
I am using withRemote to connect my java application to gremlin server running in AWS with dynamodb storage backend. I am getting connection timeout after few seconds (~3.3 seconds):
org.apache.tinkerpop.gremlin.process.remote.RemoteConnectionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException]]
I need to figure out how to reconnect which means detecting if the connection is closed. I am not sure how to detect that. I get the above exception when I use the graph traversal, is there a way to discover it before and reconnect or is there an option in configuration that allows reconnecting automatically (like create new connection before this one closes) so my application is always connected?
In case you need, this is how I am doing connection - currently connection part is singleton when the application starts:
this.graph = EmptyGraph.instance();
GryoMessageSerializerV1d0 gryoMessageSerializerV1d0 = new GryoMessageSerializerV1d0(
GryoMapper.build().addRegistry(JanusGraphIoRegistry.getInstance()));
this.cluster = Cluster.build().serializer(gryoMessageSerializerV1d0)
.addContactPoint(configuration.getString("graphDb.host", "localhost"))
.port(configuration.getInt("graphDb.port", 8182)).create();
this.graphTraversalSource = this.graph.traversal().withRemote(DriverRemoteConnection.using(cluster));
I feel like this problem is already solved with connection.keepAlive configuration option. It defaults to 180 seconds so it's longer than your timeout of 60 seconds in your load balancer which is why it gives up.
That said, the driver should be reconnecting on its own. It's constantly trying to do that given the connectionPool.reconnectInterval but perhaps there is a condition where you're quickly exhausting all the connections to the point of getting that error....not sure. Either way, hopefully the
I have been handling a application which uses wicket+JPA+springs technologies.Recently we got many 5XX error in logs(greater than threshhold).During that time,There were some general problems due to unstable response times of the mainframe db2 which is backend for our application.
But after that once the mainframe is OK this application servers did not come to normal again.
There are a lot of hanging transactions (from my appplication).
There are many threads in the server that may be hung.
As users will go on keeping login or will access the links in aplication during that time the situation becomes worse.
When I look at webspehere logs I found following exceptions:
00000035 ThreadMonitor W WSVR0605W: Thread "WebContainer : 88" (000005ac)
has been active for 637111 milliseconds and may be hung.
There is/are 43 thread(s) in total in the server that may be hung.
In application logs i found following exceptions:
-->CouldNotLockPageException: Could not lock page 4. Attempt lasted 3 minutes
-->DefaultExceptionMapper - Connection lost, give up responding.
org.apache.wicket.protocol.http.servlet.ResponseIOException:
com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during
write.
--> JDBCExceptionReporter - [jcc][t4][2030][11211][3.67.27] A communication error occurred
during operations on the connection's underlying socket, socket input stream,
or socket output stream.
Error location: Reply.fill() - socketInputStream.read (-1). Message:
Connection reset. ERRORCODE=-4499, SQLSTATE=08001DSRA0010E: SQL State = 08001, Error Code = - 4.499
Now we are working on the solutions to this problem.The follwing are two solutions that we are thinking as of now.
1.I have gone through many forums and found that whenever we get CouldNotLockPageException then it would be better to invaidate the session and force user to login page.Currently We do not have session invalidation (logout) mechanism.So we will implement that one.
2.We need to implement transaction timeouts so that we can stop hanging transactions.
I need solution for this problem from java or server side.Here we are using wicket,jpa and springs frameworks.I have few queries.
1.How can we implement transaction timeouts in the above frameworks?
2.Will invalidating session can stop hanging transaction or threads that may hung?
Since you are already using Spring, it's as simple as that:
#Transactional(timeout = 300)
The Transaction annotation allow you to supply a timeout value(in seconds) and the transaction manager will forward it to the JTA transaction manager or your Data Source connection pool. It works nice with Bitronix Transaction Manager, which automatically picks it up.
You also need to make sure the java.sql.Conenction are always being closed and Transaction are always committed (when all operations succeeded) or rollbacked on failure.
Invalidating the user http session has nothing to do with jdbc connections. Your jdbc connection should always be committed/rollbacked and closed(which in case on connection pooling, will release the connection to the pool).
And make sure the max pool size is not greater than tour db max concurrent connections setting.
I have a problem with apache http client (4.2.1) connection timeouts, if host exists but does not respond in time, the connection is closed by timeout (everything is as expected), but if there is no such host, the client keeps waiting longer than expected (about 12 seconds instead of 5 specified in configuration). Eventually this results in NoRouteToHostException, probably because of the specific network issues (when I was trying to reproduce this in another network, I've got socket read timeout exception after 5 sec of waiting, as expected).
I'm using the following timeout settings:
http.socket.timeout = 5 sec
http.connection.timeout = 5 sec
Any thoughts are appreciated.
Update
If someone ever has the same issue, it is probably caused by the connection retries, performed by the client. I'll update this post when I solve the problem
Update2
Eventually I've been able to fix the issue. The problem was caused by the connection retries performed by DefaultHttpRequestRetryHandler, used by AbstractHttpClient (that is the parent of DefaultHttpClient), if there is no request retry handler specified explicitly. So, if you want to get rid of it, just specify the request retry handler with smaller number of retries
Eventually I've been able to fix the issue. The problem was caused by the connection retries performed by DefaultHttpRequestRetryHandler, used by AbstractHttpClient (that is the parent of DefaultHttpClient), if there is no request retry handler specified explicitly. So, if you want to get rid of it, just specify the request retry handler with smaller number of retries
I'm using the IA92 Java implementation for MQTT, which allows me to connect to a MQTT broker. In order to establish the connection, I'm doing something like this:
// Create connection spec
String mqttConnSpec = "tcp://the_server#the_port";
// Create the client and connect
mqttClient = MqttClient.createMqttClient(mqttConnSpec, null);
mqttClient.connect("the_id", true, 666);
The problem is that sometimes the server takes too much time to send a response, and it throws a timeout exception:
org.apache.harmony.luni.platform.OSNetworkSystem.connectStreamWithTimeoutSocket(OSNetworkSystem.java:130)
at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:246)
at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:533)
at java.net.Socket.connect(Socket.java:1055)
at com.ibm.mqtt.j2se.MqttJava14NetSocket.<init>((null):-1)
at com.ibm.mqtt.j2se.MqttJavaNetSocket.setConnection((null):-1)
at com.ibm.mqtt.Mqtt.tcpipConnect((null):-1)
at com.ibm.mqtt.MqttBaseClient.doConnect((null):-1)
at com.ibm.mqtt.MqttBaseClient.connect((null):-1)
at com.ibm.mqtt.MqttClient.connect((null):-1)
at com.ibm.mqtt.MqttClient.connect((null):-1)
What I need to do is setting a timeout manually, instead of letting the mqtt client decide that. The documentation says: There are also methods for setting attributes of the MQ Telemetry Transport connection, such as timeouts and retries.
But, honestly, I haven't found anything about it. I have taken a look at the whole javadoc reference and there's no evidence of timeout configuration. I can't see the source code since it's not open source.
So how can I set the timeout for the Mqtt connection?
If you have confusion you can go to MqttConnectionOptions for detail.
String userName="Ohelig";
String password="Pojke";
MqttClient client = new MqttClient("tcp://192.168.1.4:1883","Sending");
MqttConnectOptions authen = new MqttConnectOptions();
authen.setUserName(userName);
authen.setPassword(password.toCharArray());
authen.setKeepAliveInterval(30);
authen.setConnectionTimeout(300);
client.connect(authen);
I don't know anything about ia92, but I'd imagine that the 666 in the connect() call is what you're trying to set the timeout to?
The timeout the documentation is referring to is probably the keepalive timeout. This is the maximum number of seconds (chosen by the client) that can elapse without communication between the server and client. I think this is what you're most interested in.
Retries on the other hand are most likely to refer to the retrying of messages that seem to have gone astray when sending messages with QoS>0. This will be something handled by the client library code though, rather than the broker. This is something that comes into play only after you've connected though, so I very much doubt it's your problem.
To be sure that the keepalive timeout is being set correctly, I'd try pointing your client at a modified mosquitto broker. You can modify mqtt3_handle_connect() in src/read_handle_server.c to print out the keepalive value when you connect. This will ensure it's doing what you think, but won't help with the actual problem I'm afraid!
What broker do you use? Really Small Message Broker V1.1 Alpha, Mosquitto, the broker that comes with IBM WebSphere? You need to set this timeout value in your server configuration. Because the system works that way. You set a keep alive value in your broker and send a ping from the client before that interval expires, in order not for the broker to close the client-server connection, and the process restarts. Actually, even if that interval expires, server will still not close the connection until the 'grace period' ends. See http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/mqtt-v3r1.html#connect