The setup:
-2-node Cassandra 2.0.7.31 cluster
-replicas=1
-With default configuration
-Using DataStax java driver 1.0
Activity
Simple insert query using QueyBuilder class
Result
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.181.13.239 ([/10.181.13.239] Unexpected exception triggered))
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
at com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:214)
at com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:169)
at com.jpmc.es.rtm.storage.impl.EventExtract.main(EventExtract.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.181.13.239 ([/10.181.13.239] Unexpected exception triggered))
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:98)
at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:165)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
Problem was that at my end I have created one thread one connection model. Which is quite synchronous. But Datstax driver works asynchronously(its using netty for that i guess) and taking multiple request on single connection. so in my case I one session for each request and guess what? I have left with pool of open connection b/w Driver and Cassandra. Server got choked so does the Driver. Problem was solved simply by letting Driver manage its pool.
Datastax java driver by default maintain minimum amount of connection for handling Certain amount of simultaneous Request.
Spec says that driver have a limitation of handling at the most 128 request per connection.so When we driver find that more than 128 request is coming Then it only open connection. So Driver is beautifully managing connection pool.[ http://www.datastax.com/drivers/java/2.0/com/datastax/driver/core/PoolingOptions.html] This link was helpful
Go to your cassandra.yaml file. It will be present in the conf folder of your cassandra installation.
For listen_address give the IP address of the cassandra server. eg.
listen_address: 10.181.13.239
and then restart the cassandra server. Also check the port for native transport is not commented. There should be an entry like
native_transport_port: 9042
Related
I am trying to parse a CSV file and create nodes and edges into Cosmos db via the help of gremlin-java client ( https://github.com/Azure-Samples/azure-cosmos-db-graph-java-getting-started ).
Some queries I have changed in this program:
For adding data :
Submitting this Gremlin 87246 query:
g.V().has('id','1120-L CONSOLIDATED SCHEDULE L').fold().coalesce(unfold(), addV('sub_schedule').property('id', '1120-L CONSOLIDATED SCHEDULE L').property('name', '1120-L CONSOLIDATED SCHEDULE L'))
For Creating edge:
g.V().hasLabel('parent').has('id', '1120').addE('has').to(g.V().hasLabel('state').has('id', 'FD'))
After running it almost for 6 hours successfully it gives weird error:
Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: An existing connection was forcibly closed by the remote host
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at GetStarted.WriteWithOriginalCSVData.main(WriteWithOriginalCSVData.java:158)
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at java.lang.Thread.run(Thread.java:745)
Can somebody please tell me the possible causes for this?
To mention I am invoking this java client remotely from my laptop via open connection and hitting azure services ( in different location ).
Firstly I thought it might be because of my internet connectivity . But it exactly throws error when it has completed some 87-88k gremlin queries.
Can someone please let me know where I am going wrong here?
I have a deployed Cassandra DB that I've been trying to connect to but I get the below error message:
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: <ipaddress_here>:<port_here> (com.datastax.driver.core.exceptions.TransportException: [<ipaddress_here>:<port_here> ] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:232)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1631)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1549)
at com.datastax.driver.core.Cluster.init(Cluster.java:160)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:342)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:317)
at com.datastax.driver.core.Cluster.connect(Cluster.java:259)
at org.springframework.cassandra.config.CassandraCqlClusterFactoryBean.executeSpecsAndScripts(CassandraCqlClusterFactoryBean.java:327)
at org.springframework.cassandra.config.CassandraCqlClusterFactoryBean.afterPropertiesSet(CassandraCqlClusterFactoryBean.java:243)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
The <ipaddress_here>:<port_here> snippet in the error is actually my valid ip and port that I'm expecting but was removed for obvious reasons. I've had Cassandra working on my local and the permissions on the remote Cassandra are correct since a colleague can successfully connect to the keyspace.
I've seen extending the timeout and this post about the native connections but don't know what else to try.
My Cassandra config is setup similar to this.
The port I am using is different from the default 9042 and 9160 that I've seen on the boards elsewhere. I also used 9042 when I connected on my localhost.
Any help is appreciated including how to get better error logs!
Is there a proxy I would have to set if I'm behind one?
We are using camunda with RDS/MySql as DB. It works fine but then sometimes it says DB is closed and so throws the ProcessEngine Exception.
Here is what I understood from our config and logs:
We have 5active connections at any time in our pool (Specified in datasource config)
There was a scenario where it was closed.
We saw error like:
Request received Context path: /engine-rest Request received Path
Info: /user PathInfo: /user ExceptionHandler:
org.camunda.bpm.engine.ProcessEngineException: Process engine
persistence exception at
org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:148)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:173)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
...... Caused by: org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed. The error may exist in
org/camunda/bpm/engine/impl/mapping/entity/User.xml The error may
involve
org.camunda.bpm.engine.impl.persistence.entity.UserEntity.selectUserByQueryCriteria
The error occurred while executing a query SQL: select distinct RES.*
from ACT_ID_USER RES
order by RES.ID_ asc LIMIT ? OFFSET ? Cause:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed.
Our tomcat props specify: minIdle = 5;
My best guess: Its closed on server but we are maintaining locally due to the above property.
Per tomcat doc (https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html):
testOnBorrow = true;
validationQuery = "select 1";
The two props should fix it as it validates the connection.
Question I am trying to figure out:
How can I repro this issue? Apart from keeping connection it idle for several hours, the scenario where this happened.
Does the AWS RDS server close the connection? If so, can we control it?
As mentioned by #Zelldon, the connection timeout can be reduced and then we could try it. It works as expected.
Just to be sure, I ran two instances of camunda, one with this fix and other without. Could see that the fix worked.
Regarding RDS, it does close the connection but I could not find any documentation on it.
I cannot connect to Microsoft SQL Server v. 2008 in Websphere IIB (formerly, Message Broker) with a JDBC provider.
I've created the provider according to the IBM documentation - Setting up a JDBC provider for type 4 connections and specified a valid security identity.
I've also tried using the latest versions of sqljdbc4.jar and sqljdbc41.jar from Microsoft.
I'm trying to get the connection using:
Connection conn = getJDBCType4Connection(THE_NAME_OF_THE_PROVIDER,JDBC_TransactionType.MB_TRANSACTION_AUTO)
I get the following error trace:
Exception details: message: java.lang.ExceptionInInitializerError stack trace: [java.lang.J9VMInternals.initialize(J9VMInternals.java:259)
javax.crypto.KeyAgreement.getInstance(Unknown Source)
com.ibm.jsse2.lb.g(lb.java:44)
com.ibm.jsse2.lb.a(lb.java:108)
com.ibm.jsse2.l$c_.a(l$c_.java:6)
com.ibm.jsse2.l.a(l.java:80)
com.ibm.jsse2.m.a(m.java:15)
com.ibm.jsse2.m.h(m.java:13)
com.ibm.jsse2.qc.a(qc.java:38)
com.ibm.jsse2.qc.<init>(qc.java:632)
com.ibm.jsse2.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:16)
com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1606)
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1324)
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:992)
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:828)
com.microsoft.sqlserver.jdbc.SQLServerDataSource.getConnectionInternal(SQLServerDataSource.java:629)
com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.createNewConnection(SQLServerPooledConnection.java:60)
com.microsoft.sqlserver.jdbc.SQLServerPooledConnection.<init>(SQLServerPooledConnection.java:42)
com.microsoft.sqlserver.jdbc.SQLServerXAConnection.<init>(SQLServerXAConnection.java:30)
com.microsoft.sqlserver.jdbc.SQLServerXADataSource.getXAConnection(SQLServerXADataSource.java:51)
com.ibm.broker.jdbctype4.jdbcdbasemgr.JDBCType4Connection.createXAConnection(JDBCType4Connection.java:244)
com.ibm.broker.jdbctype4.jdbcdbasemgr.JDBCType4DatabaseManager.getThreadOwnedConnections(JDBCType4DatabaseManager.java:308)
com.ibm.broker.jdbctype4.localtrxn.JDBCType4SinglePhaseTrxnHandler.getConnection(JDBCType4SinglePhaseTrxnHandler.java:262)
com.ibm.broker.jdbctype4.localtrxn.JDBCType4SinglePhaseTrxnHandler.getConnection(JDBCType4SinglePhaseTrxnHandler.java:154)
com.ibm.broker.jdbctype4.connfact.JDBCType4ConnectionFactory.getConnection(JDBCType4ConnectionFactory.java:163)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
java.lang.reflect.Method.invoke(Method.java:619)
com.ibm.broker.plugin.MbNode.getJDBCType4Connection(MbNode.java:1575)
ru.cinimex.fr2back.FR2BACK_DB_InsertMessage_JavaCompute.fillInCache(FR2BACK_DB_InsertMessage_JavaCompute.java:115)
ru.cinimex.fr2back.FR2BACK_DB_InsertMessage_JavaCompute.createCache(FR2BACK_DB_InsertMessage_JavaCompute.java:102)
ru.cinimex.fr2back.FR2BACK_DB_InsertMessage_JavaCompute.evaluate(FR2BACK_DB_InsertMessage_JavaCompute.java:38)
com.ibm.broker.javacompute.MbRuntimeJavaComputeNode.evaluate(MbRuntimeJavaComputeNode.java:339)
com.ibm.broker.plugin.MbNode.evaluate(MbNode.java:1491)
I think the SQL server driver is trying to use SSL which comes into a conflict with IBM's Java. Is there a way to prevent it from using SSL at all?
When I'm doing the same thing with an Oracle database, everything works well.
It's a requirement to use JDBC and not IIB's native ODBC drivers.
If you want to disable the Microsoft JDBC Driver's use of SSL, it should be possible to do so by setting the "encrypt" data source property to a value of false.
https://msdn.microsoft.com/en-us/library/bb879920%28v=sql.110%29.aspx
The above would apply if you were programmatically creating & configuring the data source. In the case where there is intermediate code handling that, it's likely there would be somewhere to specify name/value pairs of data source properties, for example, in data source configuration or on an API that you are using. Is there anywhere that you have the opportunity to do that? If this were a WebSphere Application Server data source, it would either be in data source custom properties (traditional) or properties.microsoft.sqlserver element nested under dataSource in server.xml (Liberty).
I have a web application running on JBoss 4.2.3 connecting to a SQL Server 2008 R2 database through JTDS 1.2.2 driver. The Datasource is setup to have minimum 10 connections in the pool. This is the ds file jtds-ds.xml: (I have replaced the real jndi name, server name, database name, user name and password).
<datasources>
<local-tx-datasource>
<jndi-name>MyDS</jndi-name>
<connection-url>jdbc:jtds:sqlserver://MyServer/MyDatabase;tds=8.0;lastupdatecount=true</connection-url>
<driver-class>net.sourceforge.jtds.jdbc.Driver</driver-class>
<user-name>MyUserName</user-name>
<password>MyPassword</password>
<set-tx-query-timeout>true</set-tx-query-timeout>
<!--
sql to call on an existing pooled connection when it is obtained from pool
-->
<check-valid-connection-sql>SELECT 1</check-valid-connection-sql>
<!-- Pooling Parameters -->
<prepared-statement-cache-size>500</prepared-statement-cache-size>
<min-pool-size>10</min-pool-size>
<max-pool-size>40000</max-pool-size>
</local-tx-datasource>
</datasources>
I run the SQL Server Profiler. After the JBoss starts in the SQL Server Profiler, I can see 10 database connections to the MyDatabase database from the web application . Whenever any user works on the web application, I can see it in SQL Server Profiler. Everything looks good.
I start a process which is a very long process and insert million of rows in database. The process inserts 1000 rows at a time i.e. it gets connection from pool, insert 1000 rows in database, then close connection. I can see it in SQL Server Profiler and it is working ok.
While the process was running I got a warning in the log file and I think it was because the connection was lost to the database server
2015-03-26 11:50:35,611 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Unable to fill pool
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Network error IOException: Connection timed out: connect)
at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:190)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:619)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:534)
at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLException: Network ` IOException: Connection timed out: connect
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:385)
at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:182)
at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:172)
... 4 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:304)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:255)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:310)
... 7 more
After this error, the SQL Server Profiler is not showing any connections for the web application. JBoss is still running. Why JBoss would destroy its connection pool in this case?
UPDATE:
When another user's request comes in the web application, I saw the SQL Server Profiler shows me 4 connections by the web application. So it is at least start building the connection pool again although I would expect it to show 10 connections.
My question still is that when such an error occurs (Connection timed out), why JBoss drop the whole 10 connections in the connection pool? The process was using only 1 connection, the other 9 connections were not being used as there was nobody else using the web application when the process was run. Also is it possible for JBoss to rebuild its connection pool by itself when it was somehow destroyed? I thought check-valid-connection-sql tag does that trick?