I'm currently working on a project that will poll an email inbox for a daily report.
I have a working implementation using spring-integration-mail with an imapInboundAdapter when hitting an email user hosted on Amazon Workmail.
When there is a single email in the inbox that is unread and unflagged it receives that email and properly emits only a single message.
However, when I changed the email server to one hosted by outlook I instead receive the email twice in the same poll.
As far as I can tell, it seems like the seen / flag status of the email isn't updated until the second polling attempt when on outlook, but is updated on the firstattempt on workmail.
The second attempt on outlook retrieves an email that should already have been processed.
IntegrationFlows
.from(
Mail.imapInboundAdapter(format("imaps://%s:%s/INBOX", source.getHost(), source.getPort()))
.javaMailAuthenticator(authenticator)
.maxFetchSize(10),
e -> e.poller(... maxMessagesPerPoll = -1, cronTrigger = "*/15 * * ? * *" ...)
).log(INFO, m -> "Received email: " + m)
This is the logging that occurs when hitting workmail:
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#imap.mail.us-west-2.awsapps.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 8
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache : create message number 8
o.s.integration.mail.ImapMailReceiver : found 1 new messages
o.s.integration.mail.ImapMailReceiver : Received 1 messages
o.s.integration.mail.ImapMailReceiver : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#a101a27]
o.s.integration.handler.LoggingHandler : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#a101a27, headers={id=20fb1886-eeda-c1e2-7ce0-8a9aae4f7ebc, timestamp=1624590243455}]
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#imap.mail.us-west-2.awsapps.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 8
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
o.s.integration.mail.ImapMailReceiver : found 0 new messages
o.s.integration.mail.ImapMailReceiver : Received 0 messages
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
This is the logging that occurs when hitting outlook:
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#outlook.office365.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 2
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache : create message number 2
o.s.integration.mail.ImapMailReceiver : found 1 new messages
o.s.integration.mail.ImapMailReceiver : Received 1 messages
o.s.integration.mail.ImapMailReceiver : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#5dffc30b]
o.s.integration.handler.LoggingHandler : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#5dffc30b, headers={id=9d158e52-d46e-fd82-38b7-b438a4899a1e, timestamp=1624589763164}]
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#outlook.office365.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 2
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache : create message number 2
o.s.integration.mail.ImapMailReceiver : found 1 new messages
o.s.integration.mail.ImapMailReceiver : Received 1 messages
o.s.integration.mail.ImapMailReceiver : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#59c04c6c]
o.s.integration.handler.LoggingHandler : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage#59c04c6c, headers={id=002e6b83-caf5-86cd-b090-44aa346df119, timestamp=1624589766210}]
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap : IMAPProtocol noop
com.sun.mail.imap.connectionpool : releaseStoreProtocol()
com.sun.mail.imap.connectionpool : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver : opening folder [imaps://[SNIPPED]#outlook.office365.com:993/INBOX]
com.sun.mail.imap : connection available -- size: 1
com.sun.mail.imap.messagecache : create cache of size 2
o.s.integration.mail.ImapMailReceiver : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
o.s.integration.mail.ImapMailReceiver : found 0 new messages
o.s.integration.mail.ImapMailReceiver : Received 0 messages
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap.connectionpool : connection pool current size: 0 pool size: 1
com.sun.mail.imap : added an Authenticated connection -- size: 1
The issue turned out to be directly related to the windows outlook mail client.
If you add an account with imap in the outlook client it will cause duplication to occur if you mark an email as unread through the client (Possibly folder handling related).
I'm not sure of the exact cause, but adding the account as a full outlook account (instead of via imap) resolved the issue.
Otherwise you can mark emails as unread in the web browser instead of in the outlook client.
Sounds like that setFlag() is an async for Outlook. You may consider to not fetch all the messages at once - .maxFetchSize(1). Another way is to use a custom SearchTermStrategy or selector to skip messages which you have read already.
Related
I was able to send an Access-Request to Freeradius server and get Access-Accept, using tinyradius library in a spring boot application. Now I need to send a COA Request and then a Disconnect-Request to end the session.
I tried to send a COA-Request, and I get an error.
> 2019-07-23 08:11:26.383 INFO 1 --- [nio-8080-exec-2]
> org.tinyradius.util.RadiusClient : communication failure,
> retry 1 2019-07-23 08:11:29.387 INFO 1 --- [nio-8080-exec-2]
> org.tinyradius.util.RadiusClient : communication failure,
> retry 2 2019-07-23 08:11:32.391 ERROR 1 --- [nio-8080-exec-2]
> org.tinyradius.util.RadiusClient : communication failure
> (timeout), no more retries
RadiusClient rc = new RadiusClient("172.17.0.2", "testing123");
CoaRequest coaRequest=new CoaRequest();
coaRequest.setAuthenticator(dto.getAuthenticator().getBytes());
RadiusPacket response = rc.communicate(coaRequest,3799);
I get 'Receive timed out' error and the request doesn't seem to hit the server either. I would like to know how to send the COA-Request and Disconnect-Request using Tinyradius library
Problem: MongoDB writes fail with the error -
Timed out after 30000 ms while waiting for a server that matches
PrimaryServerSelector. Client view of cluster state is
{type=REPLICA_SET, servers=[{address=intdb01:27017,
type=REPLICA_SET_SECONDARY, roundTripTime=0.7 ms, state=CONNECTED}]
This happens when the primary has switched from intdb01 to intdb02.
Looks like the client driver is still looking for intdb01
to be primary node.
Our Setup
We have 3 mongoDb nodes in a replicaSet called rs0.
When we connect to it using Java, we give all 3 servers in the connection string as follows:
mongodb://intdb01:27017,intdb02:27017,intdb03:27017/?replicaSet=rs0
db.version --> 3.0.4
Java Driver Version: mongodb-driver-3.0.4.jar, mongodb-driver-core-3.0.4.jar
Client Connection code:
if( mongoClient == null) {
MongoClientURI mcu = new MongoClientURI(mongoConnect);
mongoClient = new MongoClient(mcu);
}
mongoConnect contains the connection string shown above.
Mongo Replicaset Status Info
> rs.status()
{
"set" : "rs0",
"date" : ISODate("2016-07-19T21:14:03.001Z"),
"myState" : 1,
"members" : [{
"_id" : 0,
"name" : "intdb01",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 439786,
"optime" : Timestamp(1468521291, 5),
"optimeDate" : ISODate("2016-07-14T18:34:51Z"),
"lastHeartbeat" : ISODate("2016-07-19T21:14:01.877Z"),
"lastHeartbeatRecv" : ISODate("2016-07-19T21:14:01.611Z"),
"pingMs" : 0,
"configVersion" : 4
},
{
"_id" : 1,
"name" : "intdb02:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2948844,
"optime" : Timestamp(1468521291, 5),
"optimeDate" : ISODate("2016-07-14T18:34:51Z"),
"electionTime" : Timestamp(1468523057, 1),
"electionDate" : ISODate("2016-07-14T19:04:17Z"),
"configVersion" : 4,
"self" : true
},
{
"_id" : 2,
"name" : "intdb03:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 439779,
"optime" : Timestamp(1468521291, 5),
"optimeDate" : ISODate("2016-07-14T18:34:51Z"),
"lastHeartbeat" : ISODate("2016-07-19T21:14:01.294Z"),
"lastHeartbeatRecv" : ISODate("2016-07-19T21:14:01.294Z"),
"pingMs" : 0,
"configVersion" : 4
}
],
"ok" : 1
}
I have checked and rechecked the code/config with the documentation but cannot find what is amiss.
We can simulate this by stepping down the primary.
The Secondary takes over but the app writes start failing. :-(
Any pointers/tips much appreciated!
Adding Stack trace-
Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=intdb01:27017, type=REPLICA_SET_SECONDARY, roundTripTime=0.7 ms, state=CONNECTED}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:141)
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:72)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:482)
at com.mongodb.MongoCollectionImpl.update(MongoCollectionImpl.java:474)
at com.mongodb.MongoCollectionImpl.updateOne(MongoCollectionImpl.java:325)
at com.grid.core.persistence.mongodb.dao.GridEventDao.save(GridEventDao.java:69)
... 6 more
2016-07-19 17:05:01,882 [pool-2-thread-2] INFO org.mongodb.driver.cluster - No server chosen by PrimaryServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=SINGLE, all=[ServerDescription{address=intdb01:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 0, 4]}, minWireVersion=0, maxWireVersion=3, electionId=null, maxDocumentSize=16777216, roundTripTimeNanos=724681, setName='rs0', canonicalAddress=intdb01:27017, hosts=[intdb01:27017, intdb02:27017], passives=[intdb03:27017], arbiters=[], primary='intdb02:27017', tagSet=TagSet{[]}}]}. Waiting for 30000 ms before timing out
I am trying to figure out if it is a coding error or configuration.
The last info Message in the stack trace, says connectionMode is SINGLE. Have a suspicion that it is causing this issue but I cannot find any info about it in the developer documentation site.
All of the server addresses should appear in the error message. For example, try this with 3 arbitrary hostnames what won't resolve:
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=xxx:27117, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: xxx}, caused by {java.net.UnknownHostException: xxx}}, {address=xxx:27118, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: xxx}, caused by {java.net.UnknownHostException: xxx}}, {address=xxx:27119, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: xxx}, caused by {java.net.UnknownHostException: xxx}}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:106)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:58)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:327)
at com.mongodb.DBCollection.insert(DBCollection.java:323)
at com.mongodb.DBCollection.insert(DBCollection.java:314)
at com.mongodb.DBCollection.insert(DBCollection.java:284)
at com.mongodb.DBCollection.insert(DBCollection.java:250)
at com.mongodb.DBCollection.insert(DBCollection.java:187)
at InsertTest2.main(InsertTest2.java:26)
Notice how all 3 server addresses appear. I would suggest that you create a simple test program like this and verify, then work backwards from there. Also verify that the hostnames resolve correctly, you might also try IP addresses.
I'm trying connect hbase standalone(0.94.27) on ec2, from java client on My PC(windows server 2008).
here is my code snippets
HBaseChatConfiguration conf = HBaseChatConfiguration.getInstance();
conf.set("hbase.zookeeper.quorum", "my ec2 IP Address");
HTablePool tablePool = null;
try {
tablePool = new HTablePool(conf.getConfiguration(), 10);
HTableInterface table = tablePool.getTable(Bytes.toBytes("helloTable"));
System.out.println("table initiated");
Put p = new Put(Bytes.toBytes("testRow1"));
p.add(Bytes.toBytes("cf1"), Bytes.toBytes("cfs"), Bytes.toBytes("test"));
table.put(p);
But java application printed out a exception,
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for helloTable,,99999999999999 after 10 tries.
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:265)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:195)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:174)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:213)
at kr.stocktalk.chat.ChatServer.main(ChatServer.java:42)
Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for helloTable,,99999999999999 after 10 tries.
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:955)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:860)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:962)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
... 5 more
But I can access the table on local shell
hbase(main):001:0> describe 'helloTable'
DESCRIPTION ENABLED
'helloTable', {NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCO true
PE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_D
ELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOC
KCACHE => 'true'}
1 row(s) in 0.9700 seconds
What can I do for access from java client application?
edited:
Here is hbase log
2016-03-26 06:37:11,607 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1
2016-03-26 06:37:11,612 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation#6b171477
2016-03-26 06:37:11,615 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2016-03-26 06:38:12,490 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /106.249.235.42:64629
2016-03-26 06:38:12,490 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /106.249.235.42:64629
2016-03-26 06:38:12,495 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x153b183d8e6000c with negotiated timeout 40000 for client /106.249.235.42:64629
2016-03-26 06:42:02,390 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.04 MB, free=245.89 MB, max=247.92 MB, blocks=1, accesses=30, hits=29, hitRatio=96.66%, , cachingAccesses=30, cachingHits=29, cachingHitsRatio=96.66%, , evictions=0, evicted=0, evictedPerRun=NaN
2016-03-26 06:42:11,608 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1
2016-03-26 06:42:11,613 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation#6b171477
2016-03-26 06:42:11,616 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2016-03-26 06:47:02,391 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.04 MB, free=245.89 MB, max=247.92 MB, blocks=1, accesses=30, hits=29, hitRatio=96.66%, , cachingAccesses=30, cachingHits=29, cachingHitsRatio=96.66%, , evictions=0, evicted=0, evictedPerRun=NaN
2016-03-26 06:47:11,608 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1
2016-03-26 06:47:11,614 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation#6b171477
2016-03-26 06:47:11,617 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2016-03-26 06:49:06,196 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x153b183d8e6000c due to java.io.IOException: Connection reset by peer
2016-03-26 06:49:06,196 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /106.249.235.42:64629 which had sessionid 0x153b183d8e6000c
2016-03-26 06:49:46,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x153b183d8e6000c, timeout of 40000ms exceeded
2016-03-26 06:49:46,000 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x153b183d8e6000c
I'm using simple qpid java client which consumes messages from broker and sends it to through SOAP service. On the producer side we put all data to map and then send this map to qpid. Here is a snippet:
QueueSender conBusQueueSender = (QueueSender) Component.getInstance("conBusQueueSender");
QueueSession queueSession = org.jboss.seam.jms.QueueSession.instance();
Map<String,Object> map = new HashMap<String,Object>();
map.put("applicationId", applicationId);
map.put("soapAction", "urn:changeApplicationStatus");
map.put("soapXML", changeApplicationStatusString);
MapMessage message = queueSession.createMapMessage();
message.setObject("map",map);
conBusQueueSender.send(message);
On the client side we receive message and trying to send it to web service through SOAP
while (true) {
MapMessage m = (MapMessage) consumer.receive();
#SuppressWarnings("unchecked")
Map<String, Object> map = (Map<String, Object>) m.getObject("map");
SOAPMessage message = createSoapMessage(map);
while (true) {
try {
SOAPMessage result = makeSoapCall(message, (String)map.get("applicationId"), (String)map.get("soapAction"));
if(result.getSOAPBody().hasFault()){
System.out.println("SOAPFault: "+result.getSOAPBody().getFault().getFaultString());
System.out.println("SOAPMessage tried to send: " + (String)map.get("soapXML"));
}
m.acknowledge();
// if we came here successfully -> break and process next message
break;
} catch (SOAPException e) {
// if we have exception -> sleep and retry
e.printStackTrace();
Thread.sleep(1000L);
continue;
}
}
}
Everything works fine but when consumer tries to send a little bigger message ~66KB it just prints this error:
73 [main] INFO org.apache.qpid.client.security.DynamicSaslRegistrar - Additional SASL providers successfully registered.
87 [main] INFO org.apache.qpid.client.AMQConnection - Connection:amqp://guest:********#localhost/?brokerlist='tcp://localhost:5672'
314 [main] INFO org.apache.qpid.client.protocol.AMQProtocolSession - Using ProtocolVersion for Session:0-10
330 [main] INFO org.apache.qpid.client.handler.ClientMethodDispatcherImpl - New Method Dispatcher:AMQProtocolSession[null]
341 [main] INFO org.apache.qpid.client.AMQConnection - Connecting with ProtocolHandler Version:0-10
414 [IoReceiver - localhost/127.0.0.1:5672] INFO org.apache.qpid.transport.ClientDelegate - The broker does not support the configured connection idle timeout of 120 sec, using the brokers max supported value of 0 sec instead.
420 [main] INFO org.apache.qpid.client.AMQConnection - Connected with ProtocolHandler Version:0-10
Connection established to amqp://guest:guest#localhost/?brokerlist='tcp://localhost:5672'
443 [main] INFO org.apache.qpid.client.AMQSession - Created session:org.apache.qpid.client.AMQSession_0_10#112c3327
Session created...
476 [main] INFO org.apache.qpid.client.AMQSession - Prefetching delayed existing messages will not flow until requested via receive*() or setML().
Consumer initialized...
481 [main] INFO org.apache.qpid.client.AMQSession.Dispatcher - Dispatcher-Channel-0 created
481 [Dispatcher-Channel-0] INFO org.apache.qpid.client.AMQSession.Dispatcher - Dispatcher-Channel-0 started
492 [Dispatcher-Channel-0] ERROR org.apache.qpid.client.BasicMessageConsumer - Caught exception (dump follows) - ignoring...
java.lang.IllegalArgumentException: unknown code: 105
at org.apache.qpid.transport.codec.AbstractDecoder.getType(AbstractDecoder.java:354)
at org.apache.qpid.transport.codec.AbstractDecoder.readMap(AbstractDecoder.java:287)
at org.apache.qpid.transport.codec.BBDecoder.readMap(BBDecoder.java:34)
at org.apache.qpid.transport.codec.AbstractDecoder.read(AbstractDecoder.java:455)
at org.apache.qpid.transport.codec.AbstractDecoder.readMap(AbstractDecoder.java:288)
at org.apache.qpid.transport.codec.BBDecoder.readMap(BBDecoder.java:34)
at org.apache.qpid.client.message.AMQPEncodedMapMessage.populateMapFromData(AMQPEncodedMapMessage.java:96)
at org.apache.qpid.client.message.JMSMapMessage.<init>(JMSMapMessage.java:71)
at org.apache.qpid.client.message.AMQPEncodedMapMessage.<init>(AMQPEncodedMapMessage.java:52)
at org.apache.qpid.client.message.AMQPEncodedMapMessageFactory.createMessage(AMQPEncodedMapMessageFactory.java:36)
at org.apache.qpid.client.message.AbstractJMSMessageFactory.create010MessageWithBody(AbstractJMSMessageFactory.java:135)
at org.apache.qpid.client.message.AbstractJMSMessageFactory.createMessage(AbstractJMSMessageFactory.java:166)
at org.apache.qpid.client.message.MessageFactoryRegistry.createMessage(MessageFactoryRegistry.java:150)
at org.apache.qpid.client.BasicMessageConsumer_0_10.createJMSMessageFromUnprocessedMessage(BasicMessageConsumer_0_10.java:221)
at org.apache.qpid.client.BasicMessageConsumer_0_10.createJMSMessageFromUnprocessedMessage(BasicMessageConsumer_0_10.java:47)
at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:693)
at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:205)
at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:47)
at org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3392)
at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3336)
at org.apache.qpid.client.AMQSession$Dispatcher.access$900(AMQSession.java:3117)
at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3110)
at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:55)
at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3264)
at java.lang.Thread.run(Thread.java:680)
And there is no any errors like "Enqueue capacity threshold exceeded on queue "bus-status-queue"". What is wrong?
This is due to a restriction in AMPQ protocol related to String size. str16 is a 64K String size. The code part where this restriction is placed come into action when data-structures such as Maps and List are used to contain the Strings. With simple TextMessage, it is treated as raw data.
The solution to your question is, use byte[] instead, which is defined as vbin32 which can contain 4GB data.
I have a Java application which uses RMI for client/server communication.
To secure this communication the traffic is tunneled through an ssh connection.
Everything works well, except that the connection keeps getting closed automatically after a few seconds.
I have set the keep alive property true of:
SSHD connection
SSH client connection
ServerSocket server side
ClientSocket client side
A common connection routine of connecting to the register (port 4000) and invoking a method on an object (port 4005) outputs the following log:
INFO org.apache.sshd.server.session.ServerSession - Authentication succeeded
INFO org.apache.sshd.server.session.ServerSession - Received SSH_MSG_CHANNEL_OPEN direct-tcpip
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Receiving request for direct tcpip: hostToConnect=ThinkPad, portToConnect=4000, originatorIpAddress=127.0.0.1, originatorPort=64539
INFO org.apache.sshd.server.session.ServerSession - Received SSH_MSG_CHANNEL_OPEN direct-tcpip
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Receiving request for direct tcpip: hostToConnect=ThinkPad, portToConnect=4005, originatorIpAddress=127.0.0.1, originatorPort=64540
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Received SSH_MSG_CHANNEL_EOF on channel 1
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Send SSH_MSG_CHANNEL_CLOSE on channel 1
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Received SSH_MSG_CHANNEL_CLOSE on channel 1
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Closing channel 1 immediately
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Closing channel 1 immediately
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Send SSH_MSG_CHANNEL_EOF on channel 1
INFO org.apache.sshd.server.session.ServerSession - Closing session
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Closing channel 0 immediately
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Closing channel 0 immediately
INFO org.apache.sshd.server.channel.ChannelDirectTcpip - Send SSH_MSG_CHANNEL_EOF on channel 0
The line ** Received SSH_MSG_CHANNEL_EOF on channel 1 ** suggests that the method invoked on the object has generated an EOF message. This then causes the session to close...
Possible solutions I can think of:
Intercept or prevent the EOF message (but where and how?)
Try to configure the server side sessionfactory to ignore the EOF messages (feels wrong...)
RMI connections are pooled at the client and closed if not reused within 15 seconds. You can adjust this behaviour via system properties: see the Sun system properties page linked from the RMI home page.