I run a java program which is a thread executor program that inserts thousands of documents to a table in mongodb. I get the following error
Exception in thread "pool-1-thread-301" com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded.
at com.mongodb.PooledConnectionProvider.get(PooledConnectionProvider.java:70)
at com.mongodb.DefaultServer.getConnection(DefaultServer.java:73)
at com.mongodb.BaseCluster$WrappedServer.getConnection(BaseCluster.java:221)
at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:508)
at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:456)
at com.mongodb.DBTCPConnector.getPrimaryPort(DBTCPConnector.java:414)
at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:176)
at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:159)
at com.mongodb.DBCollection.insert(DBCollection.java:93)
at com.mongodb.DBCollection.insert(DBCollection.java:78)
at com.mongodb.DBCollection.insert(DBCollection.java:120)
at ScrapResults103$MyRunnable.run(MyProgram.java:368)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
How can I resolve this? Please help me.
You need to check what is the connections per host value which you have given while setting up connection (looking at the exception I think you would have set it to 500).
MongoClientOptions.Builder builder = new MongoClientOptions.Builder();
builder.connectionsPerHost(200);
MongoClientOptions options = builder.build();
mongoClient = new MongoClient(URI, connectionOptions);
An ideal way of setting the connections per host would be by trial and error but you need to make sure that the value which you set should not exceed the number of connections you can have by opening the mongo shell and executing:
db.serverStatus().connections.available
you are in maxWaitQueueSize limit , so increase multiplier ;)
MongoClientOptions options = MongoClientOptions.builder()
.threadsAllowedToBlockForConnectionMultiplier(10)
.build();
MongoClient mongo = new MongoClient("127.0.0.1:27017", options);
//run 2000 threads and use database ;)
waitQueueMultiple is the product of maxConnectionPoolSize and threadsAllowedToBlockForConnectionMultiplier hence you can modify one of these three options to tune your app in MongoClientOptions with corresponding values and consume it to your MongoClient as an argument how it was done above here (marked as an answer) https://stackoverflow.com/a/25347310/2852528
BUT
I strongly recommend analysing your code first (where it communicates to the DB), and if no optimization is available (e.g. caching, using aggregation, paging etc.) then go ahead and change the options
Related
I have project with vertx where I am using a HtttpClient where I configurure a HttpConfigOption object in order to specify host, maxPoolSize, keepAlive, etc.
HttpClientOptions httpClientOptions = new HttpClientOptions()
.setKeepAlive(KeepAlive)
.setDefaultHost(baseUrl)
.setTcpNoDelay(true)
.setVerifyHost(verifyHost)
.setTrustAll(trustAll)
.setMaxPoolSize(50)
.setPoolCleanerPeriod(POOL_CLEANER_PERIOD)
.setDefaultPort(port).setSsl(isSSl);
I start having some timeouts and I want to know if the problem is from my connection pool or the timeouts are coming from the services that I consume.
Is there a way to print in vertx the number of connections that are at the moment in use? So for example let's say that I have a poll with max 50 connections and on timeout it will be nice if I can print the number of connections that are active, in use.
Thanks
You can use the vertx micrometer module to grab vertx_http_client_active_connections
ref: https://vertx.io/docs/vertx-micrometer-metrics/java/
I want to create a "X" number of connections to mongodb on my application startup (i.e before my application starts taking traffic).
MongoDB Version: 4.0.11
Mongo Java Driver Version (maven): 3.4.1
I have tried setting the "minConnectionsPerHost" to the required number, but when i execute the code it barely open 1 or 2 connections. But, when i put load on my application the connection count is slowly going up to accommodate the traffic. I want to create those connections before my application starts taking traffic.
ServerAddress address = new ServerAddress("localhost", 27017);
List<ServerAddress> serverAddresses = Arrays.asList(address);
MongoCredential credential =
MongoCredential.createCredential("XXXX", "XXXX",
"XXXX".toCharArray());
List<MongoCredential> mongoCredentials =
Arrays.asList(credential);
MongoClientOptions clientOptions =
MongoClientOptions.builder().connectionsPerHost(100).
minConnectionsPerHost(50).build();
MongoClient mongoClient = new MongoClient(serverAddresses,
mongoCredentials, clientOptions);
Is there a way to achieve this using the mongo java driver?
You can set a minConnectionsPerHost() in the options builder, and then use a warmup script to create many connections. The connection pool will keep the minConnectionsPerHost connections alive without closing.
The warmup script can have a program which spawns 2*minConnectionsPerHost number of threads, which will connect and do may be a dummy read operation. This way connections will be opened, minimum connections will be kept alive.
This seems to be dirty solution :-) But might work!
What's the easiest way to set a timeout for spark's connection methods, like read and write?
So far I tried to add "spark.network.timeout" with something really low, like "2s", but then I got an exception requesting for "spark.executor.heartbeatInterval" to be lower than the timeout, so I set the heartbeatInterval to "1s".
Setting the timeout:
SparkSession sparkSession = SparkSession.builder().appName("test").master("local[*]").config("spark.network.timeout","2s").config("spark.executor.heartbeatInterval", "1s").getOrCreate();
Reading data:
Dataset<Row> dataset = sparkSession.read().jdbc(url, fromStatement, properties);
Writing data:
dataset.write().mode(SaveMode.Overwrite).jdbc(destinyUrl, tableName, accessProperties);
The read method took 11 seconds to load the dataset, and the write method took 13 seconds to save the dataset into the database, but no actions got stopped after the 2 seconds.
When there is no n/w issues you cant expect to get it stopped. 'spark.network.timeout' is default for all network interactions.
This config will be used in place if below properties are not configured.
spark.core.connection.ack.wait.timeout,
spark.storage.blockManagerSlaveTimeoutMs,
spark.shuffle.io.connectionTimeout
spark.rpc.askTimeout or spark.rpc.lookupTimeout
where as spark.executor.heartbeatInterval is Interval between each executor's heartbeats to the driver. Heartbeats let the driver know that the executor is still alive and update it with metrics for in-progress tasks. spark.executor.heartbeatInterval should be significantly less than spark.network.timeout (default = 10s)
source : https://spark.apache.org/docs/latest/configuration.html#networking
Hey first time using mongo replica sets, and I've configured a set of 3 servers in Java. I'm trying to determine which secondary is being read from by my service (using secondary preferred). Does the mongo driver expose access to this info within Java?
As I'm struggling to find a reliable way to track it. I'm also aware once a connection is made to a secondary it is unlikely to switch to read from the other secondary unless there is failure.
You can use the getAdress method like below, see the documentation for more details.
MongoClient mongoClient = new MongoClient("localhost", 27017);
ServerAddress serverAdress = mongoClient.getAddress()
HttpConnections where not being used efficiently by our code using Axis2 1.5.1 project. By setting a certain limit of max connections per host and stressing the application, responsiveness was not the good I expected according the intentional limits and sometimes connections got stucked indefinitly, so the available connections were each time less till reaching the point that none request was attended by the application.
Configuration:
MultiThreadedHttpConnectionManager connManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams connectionManagerParams = connManager.getParams();
connectionManagerParams.setMaxTotalConnections(httpMaxConnections);
connectionManagerParams.setDefaultMaxConnectionsPerHost(httpMaxConnectionsPerHost);
HttpClient httpClient = new HttpClient(connManager);
ConfigurationContext axisContext;
try {
axisContext = ConfigurationContextFactory.createDefaultConfigurationContext();
} catch (Exception e) {
throw new AxisFault(e.getMessage());
}
axisContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
service = new MyStub(axisContext, url);
ServiceClient serviceClient = service._getServiceClient();
serviceClient.getOptions().setProperty(HTTPConstants.CONNECTION_TIMEOUT, httpConnectionTimeout);
serviceClient.getOptions().setProperty(HTTPConstants.SO_TIMEOUT, httpReadTimeout);
serviceClient.getOptions().setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);
So, as you can see, we're defining max. connections and timeouts.
I have a workaround I will share, hoping to help somebody under hurries as I was. I'll mark my answer as the good one a few days later if there isn't any better answer from experts.
1) PoolTimeout to prevent the connections that got stucked (for any reason)
Next line helped us to prevent Axis2 to lose connections that got stucked forever:
httpClient.getParams().setParameter(HttpClientParams.CONNECTION_MANAGER_TIMEOUT, 1000L);
Let's call it PoolTimeout in this entry. Make sure it's a Long, since an Integer (or int) would raise a ClassCastException that will prevent your service to even be triggered outside your client.
The system you're developing, and that is using Axis, could be in turn a client for another system. And that other system will have for sure an specific ConnectionTimeout. So I suggest
PoolTimeout <= ConnectionTimeout
Example:
serviceClient.getOptions().setProperty(HTTPConstants.CONNECTION_TIMEOUT, httpConnectionTimeout);
httpClient.getParams().setParameter(HttpClientParams.CONNECTION_MANAGER_TIMEOUT, Long.valueOf(httpConnectionTimeout) );
2) Connections release
I was using Amila's suggestion for connection management, but actually the connections were not released as fast as in advance I expected they would be (because I prepared consciously the delay times mocked external system would respond to fit limits accordingly my tunning configuration).
So I found that next lines, in method org.apache.axis2.client.OperationClient.executeImpl(boolean), helped to mark as available the connection in the pool as soon as it's been used:
HttpMethod method = (HttpMethod) getOperationContext().getMessageContext(WSDLConstants.MESSAGE_LABEL_OUT_VALUE)
.getProperty(HTTPConstants.HTTP_METHOD);
method.releaseConnection();
That's what Axis is trying to do when calling serviceClient.cleanupTransport() but it seems the context is not correct.
Now, performance tunning is working in a predictable way, so it's in hands of our integrators to select the tunning configuration that best suits production needs.
A better answer will be highly appreciated.