Problem: MongoDB writes fail with the error -
Timed out after 30000 ms while waiting for a server that matches
PrimaryServerSelector. Client view of cluster state is
{type=REPLICA_SET, servers=[{address=intdb01:27017,
type=REPLICA_SET_SECONDARY, roundTripTime=0.7 ms, state=CONNECTED}]
This happens when the primary has switched from intdb01 to intdb02.
Looks like the client driver is still looking for intdb01
to be primary node.
Our Setup
We have 3 mongoDb nodes in a replicaSet called rs0.
When we connect to it using Java, we give all 3 servers in the connection string as follows:
mongodb://intdb01:27017,intdb02:27017,intdb03:27017/?replicaSet=rs0
db.version --> 3.0.4
Java Driver Version: mongodb-driver-3.0.4.jar, mongodb-driver-core-3.0.4.jar
Client Connection code:
if( mongoClient == null) {
MongoClientURI mcu = new MongoClientURI(mongoConnect);
mongoClient = new MongoClient(mcu);
}
mongoConnect contains the connection string shown above.
Mongo Replicaset Status Info
> rs.status()
{
"set" : "rs0",
"date" : ISODate("2016-07-19T21:14:03.001Z"),
"myState" : 1,
"members" : [{
"_id" : 0,
"name" : "intdb01",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 439786,
"optime" : Timestamp(1468521291, 5),
"optimeDate" : ISODate("2016-07-14T18:34:51Z"),
"lastHeartbeat" : ISODate("2016-07-19T21:14:01.877Z"),
"lastHeartbeatRecv" : ISODate("2016-07-19T21:14:01.611Z"),
"pingMs" : 0,
"configVersion" : 4
},
{
"_id" : 1,
"name" : "intdb02:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2948844,
"optime" : Timestamp(1468521291, 5),
"optimeDate" : ISODate("2016-07-14T18:34:51Z"),
"electionTime" : Timestamp(1468523057, 1),
"electionDate" : ISODate("2016-07-14T19:04:17Z"),
"configVersion" : 4,
"self" : true
},
{
"_id" : 2,
"name" : "intdb03:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 439779,
"optime" : Timestamp(1468521291, 5),
"optimeDate" : ISODate("2016-07-14T18:34:51Z"),
"lastHeartbeat" : ISODate("2016-07-19T21:14:01.294Z"),
"lastHeartbeatRecv" : ISODate("2016-07-19T21:14:01.294Z"),
"pingMs" : 0,
"configVersion" : 4
}
],
"ok" : 1
}
I have checked and rechecked the code/config with the documentation but cannot find what is amiss.
We can simulate this by stepping down the primary.
The Secondary takes over but the app writes start failing. :-(
Any pointers/tips much appreciated!
Adding Stack trace-
Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=intdb01:27017, type=REPLICA_SET_SECONDARY, roundTripTime=0.7 ms, state=CONNECTED}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:141)
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:72)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:482)
at com.mongodb.MongoCollectionImpl.update(MongoCollectionImpl.java:474)
at com.mongodb.MongoCollectionImpl.updateOne(MongoCollectionImpl.java:325)
at com.grid.core.persistence.mongodb.dao.GridEventDao.save(GridEventDao.java:69)
... 6 more
2016-07-19 17:05:01,882 [pool-2-thread-2] INFO org.mongodb.driver.cluster - No server chosen by PrimaryServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=SINGLE, all=[ServerDescription{address=intdb01:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 0, 4]}, minWireVersion=0, maxWireVersion=3, electionId=null, maxDocumentSize=16777216, roundTripTimeNanos=724681, setName='rs0', canonicalAddress=intdb01:27017, hosts=[intdb01:27017, intdb02:27017], passives=[intdb03:27017], arbiters=[], primary='intdb02:27017', tagSet=TagSet{[]}}]}. Waiting for 30000 ms before timing out
I am trying to figure out if it is a coding error or configuration.
The last info Message in the stack trace, says connectionMode is SINGLE. Have a suspicion that it is causing this issue but I cannot find any info about it in the developer documentation site.
All of the server addresses should appear in the error message. For example, try this with 3 arbitrary hostnames what won't resolve:
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=xxx:27117, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: xxx}, caused by {java.net.UnknownHostException: xxx}}, {address=xxx:27118, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: xxx}, caused by {java.net.UnknownHostException: xxx}}, {address=xxx:27119, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: xxx}, caused by {java.net.UnknownHostException: xxx}}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:106)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:58)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:327)
at com.mongodb.DBCollection.insert(DBCollection.java:323)
at com.mongodb.DBCollection.insert(DBCollection.java:314)
at com.mongodb.DBCollection.insert(DBCollection.java:284)
at com.mongodb.DBCollection.insert(DBCollection.java:250)
at com.mongodb.DBCollection.insert(DBCollection.java:187)
at InsertTest2.main(InsertTest2.java:26)
Notice how all 3 server addresses appear. I would suggest that you create a simple test program like this and verify, then work backwards from there. Also verify that the hostnames resolve correctly, you might also try IP addresses.
Related
I'm currently working on a project that will poll an email inbox for a daily report.
I have a working implementation using spring-integration-mail with an imapInboundAdapter when hitting an email user hosted on Amazon Workmail.
When there is a single email in the inbox that is unread and unflagged it receives that email and properly emits only a single message.
However, when I changed the email server to one hosted by outlook I instead receive the email twice in the same poll.
As far as I can tell, it seems like the seen / flag status of the email isn't updated until the second polling attempt when on outlook, but is updated on the firstattempt on workmail.
The second attempt on outlook retrieves an email that should already have been processed.
IntegrationFlows
.from(
Mail.imapInboundAdapter(format("imaps://%s:%s/INBOX", source.getHost(), source.getPort()))
.javaMailAuthenticator(authenticator)
.maxFetchSize(10),
e -> e.poller(... maxMessagesPerPoll = -1, cronTrigger = "*/15 * * ? * *" ...)
).log(INFO, m -> "Received email: " + m)
This is the logging that occurs when hitting workmail:
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#imap.mail.us-west-2.awsapps.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 8
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache : create message number 8
o.s.integration.mail.ImapMailReceiver : found 1 new messages
o.s.integration.mail.ImapMailReceiver : Received 1 messages
o.s.integration.mail.ImapMailReceiver : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#a101a27]
o.s.integration.handler.LoggingHandler : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#a101a27, headers={id=20fb1886-eeda-c1e2-7ce0-8a9aae4f7ebc, timestamp=1624590243455}]
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#imap.mail.us-west-2.awsapps.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 8
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
o.s.integration.mail.ImapMailReceiver : found 0 new messages
o.s.integration.mail.ImapMailReceiver : Received 0 messages
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
This is the logging that occurs when hitting outlook:
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#outlook.office365.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 2
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache : create message number 2
o.s.integration.mail.ImapMailReceiver : found 1 new messages
o.s.integration.mail.ImapMailReceiver : Received 1 messages
o.s.integration.mail.ImapMailReceiver : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#5dffc30b]
o.s.integration.handler.LoggingHandler : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#5dffc30b, headers={id=9d158e52-d46e-fd82-38b7-b438a4899a1e, timestamp=1624589763164}]
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#outlook.office365.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 2
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache : create message number 2
o.s.integration.mail.ImapMailReceiver : found 1 new messages
o.s.integration.mail.ImapMailReceiver : Received 1 messages
o.s.integration.mail.ImapMailReceiver : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#59c04c6c]
o.s.integration.handler.LoggingHandler : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#59c04c6c, headers={id=002e6b83-caf5-86cd-b090-44aa346df119, timestamp=1624589766210}]
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#outlook.office365.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 2
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
o.s.integration.mail.ImapMailReceiver : found 0 new messages
o.s.integration.mail.ImapMailReceiver : Received 0 messages
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
The issue turned out to be directly related to the windows outlook mail client.
If you add an account with imap in the outlook client it will cause duplication to occur if you mark an email as unread through the client (Possibly folder handling related).
I'm not sure of the exact cause, but adding the account as a full outlook account (instead of via imap) resolved the issue.
Otherwise you can mark emails as unread in the web browser instead of in the outlook client.
Sounds like that setFlag() is an async for Outlook. You may consider to not fetch all the messages at once - .maxFetchSize(1). Another way is to use a custom SearchTermStrategy or selector to skip messages which you have read already.
Given a normal user ('simpleROUser', only with the 'read' role on the database), it is throwing error when attempting to list collections.
The error message is:
Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 13 (Unauthorized): 'not authorized on wmMonitoring to execute command { listCollections: 1, cursor: {}, $db: "wmMonitoring", ...' on server xxxxxxx:27001. The full response is {"operationTime": {"$timestamp": {"t": 1614169303, "i": 1}}, "ok": 0.0, "errmsg": "not authorized on wmMonitoring to execute command { listCollections: 1, cursor: {}, $db: \"wmMonitoring\", ...
However, changing only the user credentials to one with 'root' role, it works (lists all the collections under the database 'wmMonitoring'.
I've checked the 'simpleROUser' privileges, the 'listCollections' is there.
rs-dev-00:PRIMARY> grants = db.getUser( "simpleROUser", { showCredentials: true, showPrivileges: true, showAuthenticationRestrictions: true } )
rs-dev-00:PRIMARY> grants.user
simpleROUser
rs-dev-00:PRIMARY> grants.inheritedPrivileges
[
{
"resource" : {
"db" : "wmISMonitoring",
"collection" : ""
},
"actions" : [
...
"listCollections",
...
]
},
...
]
rs-dev-00:PRIMARY>
So... what am I missing?
More info:
MongoDB server: Percona distribution, v4.4.1-3
Mongo Java Driver: v4.2.1
Found th issue.
In Brazilan Portuguese, this is also referred as "dedo gordo".
There was a typo in the grant: the actual database name is wmMonitoring.
Once the database name (in the grant command) has fixed, everything else worked.
We seem to be out of ideas on how to continue troubleshooting this issue. Suddenly we see the exception listed below being hit every few minutes.
This is a 4 Mongo Shard Setup. The 4 MongoS servers are proxied using 3 HAProxies. We aren't having any visible network issues between our application and shards. The mongo logs for the shards, config servers or mongos don't show anything out of the ordinary. The haproxies seem to be doing their job just fine. Any thoughts and leads on this would be most appreciated!
{"#timestamp":"2016-02-10T05:10:23.780+00:00","#version":1,"message":"unable to process event [ Request Id: [ eb8c702c-b3e7-4605-99a7-c3dcb5a076a9 ] - Event: id [ 49e0ae8d-f16e-448c-9b28-a4af59aa2eb0 ] messageType [ bulk ] operationType: [ create ] xpoint [ 10254910941235296908401 ] ]","logger_name":"com.xyz.event.RabbitMQMessageProcessor","thread_name":"SimpleAsyncTaskExecutor-1","level":"WARN","level_value":30000,"stack_trace":"java.io.EOFException: null\n\tat org.bson.io.Bits.readFully(Bits.java:50) ~[mongo-java-driver-2.12.5.jar:na]\n\tat org.bson.io.Bits.readFully(Bits.java:35)\n\tat org.bson.io.Bits.readFully(Bits.java:30)\n\tat com.mongodb.Response.(Response.java:42)\n\tat com.mongodb.DBPort$1.execute(DBPort.java:141)\n\tat com.mongodb.DBPort$1.execute(DBPort.java:135)\n\tat com.mongodb.DBPort.doOperation(DBPort.java:164)\n\tat com.mongodb.DBPort.call(DBPort.java:135)\n\tat c.m.DBTCPConnector.innerCall(DBTCPConnector.java:289)\n\t... 56 common frames omitted\nWrapped by: c.m.MongoException$Network: Read operation to server prod_mongos.internal.xyz.com:27017 failed on database xyz\n\tat c.m.DBTCPConnector.innerCall(DBTCPConnector.java:297) ~[mongo-java-driver-2.12.5.jar:na]\n\tat c.m.DBTCPConnector.call(DBTCPConnector.java:268)\n\tat c.m.DBCollectionImpl.find(DBCollectionImpl.java:84)\n\tat c.m.DBCollectionImpl.find(DBCollectionImpl.java:66)\n\tat c.m.DBCollection.findOne(DBCollection.java:869)\n\tat c.m.DBCollection.findOne(DBCollection.java:843)\n\tat c.m.DBCollection.findOne(DBCollection.java:789)\n\tat o.s.d.m.c.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:2013) ~[spring-data-mongodb-1.6.3.RELEASE.jar:na]\n\tat o.s.d.m.c.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:1997)\n\tat o.s.d.m.c.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1772)\n\t... 47 common frames omitted\nWrapped by: o.s.d.DataAccessResourceFailureException: Read operation to server prod_mongos.internal.xyz.com:27017 failed on database xyz; nested exception is com.mongodb.MongoException$Network: Read operation to server prod_mongos.internal.xyz.com:27017 failed on database xyz\n\tat o.s.d.m.c.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:59) ~[spring-data-mongodb-1.6.3.RELEASE.jar:na]\n\tat o.s.d.m.c.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1946)\n\tat o.s.d.m.c.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1776)\n\tat o.s.d.m.c.MongoTemplate.doFindOne(MongoTemplate.j...","HOSTNAME":"prod-node-09","requestId":"eb8c702c-b3e7-4605-99a7-c3dcb5a076a9","WHAT":"ProcessBulkDiscoveredXYZEvent","host":"172.30.31.155:44243","type":"cloud_service","tags":["_grokparsefailure"]}
I have a java/dropwizard application using elasticsearch (running version 1.3.2 but I've also tested with 1.4.0)
For local development/testing I use elasticsearch in local mode, starting it with the following commands:
Node node = nodeBuilder().local(true).node();
Client client = node.client();
According the documentation (http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.html, section on Node Client) this should be sufficient, but I'm not able to run any queries. Checking the health status I get the following result:
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 10
}
I assume the problem is that I have no active shards. Do anyone know how activate shards for this kind of setup?
I have a replica set setup with 1 primary (mongo1.test.com), 1 secondary(mongo2.test.com) and 1 arbiter(mongo3.test.com). When I use MongoClient to connect to them and print out the ReplicaSetStatus, it shows that mongo1.test.com is unconnected with type=Unknown, mongo2.test.com as type=ReplicaSetSecondary, and mongo3.test.com as type=Unknown. Because it doesn't know which one is the primary, I can do find queries but I cannot insert or update.
Right now I don't know if it is the setup with Mongo or a config issue with the driver. Any advice?
Mongo version 2.6.1
Java Mongo driver version 2.12.1
Mongo installed on 3 seperate Amazon EC2 Linux servers.
Here's the code:
MongoClientOptions.Builder optionsBuilder = MongoClientOptions.builder()
.acceptableLatencyDifference(10000)
.writeConcern(WriteConcern.REPLICA_ACKNOWLEDGED)
.readPreference(ReadPreference.secondaryPreferred())
.connectionsPerHost(10)
.connectTimeout(15000)
.maxWaitTime(30000)
.socketTimeout(60000)
.threadsAllowedToBlockForConnectionMultiplier(1500);
MongoClientOptions options = optionsBuilder.build();
MongoClient mc = new MongoClient(Arrays.asList(
new ServerAddress("mongo1.test.com",27017),
new ServerAddress("mongo2.test.com",27018),
new ServerAddress("mongo3.test.com",27019)),
Arrays.asList(MongoCredential.createMongoCRCredential(xxx, "admin", xxx.toCharArray()))), options);
System.out.println(mc.getRelicaSetStatus());
ReplicaSetStatus printout:
ReplicaSetStatus {
name=eeRS1,
cluster=ClusterDescription{
type=ReplicaSet,
connectionMode=Multiple,
all=[ServerDescription{
address=PRIVATE IP:27017,
type=Unknown,
hosts=[],
passives=[],
arbiters=[],
primary='null',
maxDocumentSize=16777216,
maxMessageSize=33554432,
maxWriteBatchSize=512,
tags={}, setName='null',
setVersion='null',
averagePingTimeNanos=0,
ok=false,
state=Unconnected,
version=ServerVersion{
versionList=[0, 0, 0]
},
minWireVersion=0,
maxWireVersion=0
}, ServerDescription{
address=mongo2.test.com:27018,
type=ReplicaSetSecondary,
hosts=[PRIVATE IP:27017,
mongo2.test.com:27018],
passives=[],
arbiters=[mongo3.test.com:27019],
primary='PRIVATE IP:27017',
maxDocumentSize=16777216,
maxMessageSize=48000000,
maxWriteBatchSize=1000,
tags={},
setName='eeRS1',
setVersion='17',
averagePingTimeNanos=215754657,
ok=true,
state=Connected,
version=ServerVersion{
versionList=[2, 6, 1]
},
minWireVersion=0,
maxWireVersion=2
},
ServerDescription{
address=mongo3.test.com:27019,
type=ReplicaSetArbiter,
hosts=[PRIVATE IP:27017,
mongo2.test.com:27018],
passives=[],
arbiters=[mongo3.test.com:27019],
primary='PRIVATE IP:27017',
maxDocumentSize=16777216,
maxMessageSize=48000000,
maxWriteBatchSize=1000,
tags={},
setName='eeRS1',
setVersion='17',
averagePingTimeNanos=132660144,
ok=true,
state=Connected,
version=ServerVersion{
versionList=[2, 6, 1]
},
minWireVersion=0,
maxWireVersion=2
}]
}
}
Calling insert on any DB gives following error:
com.mongodb.MongoServerSelectionException:
Unable to connect to any server that matches{
serverSelectors=[ReadPreferenceServerSelector{
readPreference=primary
},
LatencyMinimizingServerSelector{
acceptableLatencyDifference=10000 ms
}]
}
I think your issue isn't the Java driver but the ReplSet itself. Fire up a mongo shell and do rs.status(). You'll likely see output almost exactly like the stuff the Java driver is giving you. It seems that your mongo1.test.com is either offline entirely or not starting up. SSH onto mongo1.test.com and see if you can mongo shell from there. My bet is "no". Tail the mongo server log to see what that tells you. Maybe stop the service and start it there to get the log to give you a fresh view of the issue.
The good news is, I think your Java code is actually working, you just have a dorked mongod in the ReplSet configuration. Undork it, the ReplSet will heal itself, and you'll be on your way.