Develop Reference

Fetching only message headers from Gmail

I'm writing an small app to generate stats in a GMail inbox to help with the endless task of cleaning our mailbox.
My code is working, but I am unnecessarily download messages's payload data to access only the message headers (From field header).
The method Gmail.Users.Messages.Get.setFields(String) with payload option, but payload is exactly the heaviest part of it. I already tried using values like "payload.header", "payload.headers", but none of these values work.
So the question is, how can I access only message's from field instead of fetching entire payload data in order to improve performance?
Map<String, Integer> emailAddressCountMap = new HashMap<String, Integer>();
for (Message message : messages) {
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload").execute();
Stream<String> fromHeaderValue = m1.getPayload().getHeaders().stream()
.filter(h -> "From".equals(h.getName())).map(h -> h.getValue());
String emailAddress = fromHeaderValue.toArray(String[]::new)[0];
Integer count = emailAddressCountMap.get(emailAddress);
if (count == null) {
count = 0;
}
emailAddressCountMap.put(emailAddress, count + 1);
System.out.println(emailAddress + ": " + count);
}

How about a following modification?
From :
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload").execute();
To :
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload/headers").execute();
If this didn't work, or if I misunderstand your question, I'm sorry.

how to get all uncommitted messages in kafka when manually committing offset

In my application ,i am consuming json messages from kafka topic and Multiple instances are running of my application. I have set kafka prop as: props.put("enable.auto.commit", "false")
So When i consume message ,i push it to my DB and then commit it as :
private static void commitMessage(KafkaConsumer<String, String> kafkaConsumer, ConsumerRecord message, String kafkaTopic) {
long nextOffset = message.offset() + 1;
TopicPartition topicPartition = new TopicPartition(kafkaTopic, message.partition());
OffsetAndMetadata offsetAndMetadata = new OffsetAndMetadata(nextOffset);
Map<TopicPartition, OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>();
offsetAndMetadataMap.put(topicPartition, offsetAndMetadata);
//
log.info("Commiting processed kafka message, topic [" + kafkaTopic + "], partition [" + message.partition() + "], next offset [" + nextOffset + "]");
kafkaConsumer.commitSync(offsetAndMetadataMap);
}
Now it may happen after consuming message(but before pushing it to DB) my Application restarts for some reason. Now i want to consume uncommitted message again from kafka after restart. I am able to do using seek:
private static void seekAllPartitions(KafkaConsumer<String, String> kafkaConsumer, String kafkaTopic) {
List<PartitionInfo> partitionInfos = kafkaConsumer.partitionsFor(kafkaTopic);
println 'Size ofpartition list : ' + partitionInfos.size()
for (PartitionInfo partitionInfo : partitionInfos) {
TopicPartition topicPartition = new TopicPartition(kafkaTopic, partitionInfo.partition());
OffsetAndMetadata committedForPartition = kafkaConsumer.committed(topicPartition);
try {
if (committedForPartition != null) {
println 'Seeking offset...' + committedForPartition.offset()
kafkaConsumer.seek(topicPartition, committedForPartition.offset());
}
} catch (Exception ex) {}
}
}
Now problem is - seek(topicPartition,committedForPartition.offset()) gives me last uncommitted message and not the intermediate uncommitted messages.As i mentioned ,multiple instance are running - i may end up with intermediate uncommitted messages for ex : Instance a -2nd msg was not committed and Instance b - 5 the msg not committed but it gives me 5th message only and not 2nd.

Mapping key and values, adding message in Java

I have a question about mapping, map key and map values.
I am writing a chat program : I have a problem to add a message. I can't add a message. That puts me in a empty web page with an error(can't see the number and reason of error)
Can you tell me where is the problem ?
// add a message to a chatroom
#RequestMapping(value="/addMessageSalon/{salon}/{pseudo}/{message}", method = {RequestMethod.GET, RequestMethod.POST})
public String addMessageSalon(HttpServletRequest request, #PathVariable("salon") String chatroom, #PathVariable("pseudo") String username, #PathVariable("message") String message) {
Message mes = null;
mes.setMessage(message);
mes.setPseudo(username);
GestionMessages addition = (GestionMessages)request.getSession().getServletContext().getAttribute("gestionMessages");
Map<String, ArrayList<Message>> resultat = addition.getMessages();
Iterator<Map.Entry<String, ArrayList<Message>>> entries = resultat.entrySet().iterator();
// iteration
while(entries.hasNext()) {
Map.Entry<String, ArrayList<Message>> entry = entries.next();
if(!entries.hasNext() && !entry.getKey().contains(chatroom)) {
// if chatroom does not exist, we give an error
throw new IllegalArgumentException("Chatroom '" + chatroom + "' doesn't exist");
}
if(entry.getKey().contains(chatroom)){
ControleurPrincipal.getUsersInDataBase().add(username);
addition.getMessagesSalon(chatroom).add(mes);
break;
}
}
resultat = addition.getMessages();
return "redirect:/";
}

First of all:
Message mes = null;
mes.setMessage(message);
This will throw a NullPointerException, every time. So either that's the error you're getting, or that is not your code.
If that's your actual code, then you need to instantiate Message first, like this:
Message mes = new Message();

Instead of doing this
if(!entries.hasNext() && !entry.getKey().contains(monSalon)) {
you might want to do
if(!resultat.contains(monSalon)) {
and do it before the while.

Receiving objects with jms java

could someone help me. Here is my problem :
I try to send an object with jms (this part works) and receive it with jms.
My object is quite simple. 3 String, 3 int, and a boolean.
There are no problem of connexion or anything like this. I receive the object but it's as if I received every things one by one.
Here is my MessageListener :
MessageListener listner = new MessageListener() {
public void onMessage(Message message) {
ObectToSend yo=null;
try {
if (message instanceof ObjectMessage) {
ObjectMessage myMessage = (ObjectMessage) message;
System.err.println("test");
yo = (ObectToSend) myMessage.getObject();
System.err.println("test2");
System.err.println(yo.entite + " " + yo.error + " " + yo.idGloreg + " " + yo.indPerso + " " + yo.nom + " " + yo.prenom + " " + yo.nom);
}
} catch (JMSException e) {
System.out.println("Caught:" + e);
e.printStackTrace();
}
}
};
And that my sending part :
Serializable ObectTest = new ObectToSend("pro", "enc", 134, 10, true, "yayaya", 0);
MessageProducer producer = session.createProducer(topic);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
ObjectMessage message = session.createObjectMessage();
message.setObject(ObectTest);
connection.start();
producer.send(message);
Finally here is what I want to send (in receiver.java and sender.java) :
public static class ObectToSend implements Serializable{
private static final long serialVersionUID = 1L;
String prenom;
String nom;
int idGloreg;
int indPerso;
boolean ok;
String entite;
int error;
ObectToSend(String prenomP, String nomP, int idGloregP, int indPersoP, boolean okP, String entiteP, int errorP){
prenom = prenomP ;
nom= nomP;
idGloreg = idGloregP;
indPerso = indPersoP;
ok = okP;
entite= entiteP;
error = errorP;
}
}
My console :
test
test
test
test
test
test
test
If someone could tell me what's the problem that would be great. I don't get it. My textmessage with topic/queue/sync/async are working so nicely. It comes to object and....
It seems the problem is here :
yo = (ObectToSend) myMessage.getObject();
but.....

For future users of JMS I will answer my own question.
It was really hard to find any information as it's not explained in JMS documentation.
I found a lot of people asking how to do it but never had any answer. That's because it's not releated to JMS but to Java.
So here it goes :
If you want to use a same classe (object) like
ObectToSend yo = (ObectToSend) myMessage.getObject();
My first object (yo) is from the class ObectToSend.java in the package com.test.jms and my second object (myMessage.getObject() ) is from the package com.test.jms2. So I have an exception like "notfoundclass". And cannot cast objects.
The class ObectToSend.java should be in both projects. But you can't just copy paste with the same name (what I stupidly did).
You need to create a jar of the class used in both projects/packages and add it to both projects.
That way you use EXACTLY the same class and not a copy of it. And your 2 objects are exactly the same.

You could also use a map message. Indeed, you only have strings, ints, and booleans. You actually don't need an object. MapMessage is here for you and is much much simpler.
Here is an exemple :
// create mapMessage
message = session.createMapMessage();
// Here insert variables in properties of the message
// This can be filtred with selector
message.setStringProperty("entity", entity);
message.setStringProperty("messageFrom", messageFrom);
// Here insert variables in body of the message
//This CAN'T be filtred (what you want I think)
message.setString("title", title);
message.setString("description", description);
// Get map message
MapMessage mapMessage = (MapMessage) message;
// Here get variables of your message
String title = mapMessage.getString("title");
String description = mapMessage.getString("description");
See how simple it is ?

AWS SQS Java. Not all messages are retrieved from the SQS queue

I have been trying several approaches to retrieve all messages from the SQS queue by using AWS SDK for Java to no avail. I have read about the distributed nature of the AWS SQS and that messages are stored on the different servers. But what I do not understand is why this architecture is not hidden from the end user. What tricks do I have to apply in Java code to retrieve all messages and be 100% sure that no one was missed?
I tried this with the "Long Polling":
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages) {
System.out.println(" Message");
System.out.println(" MessageId: " + message.getMessageId());
System.out.println(" ReceiptHandle: " + message.getReceiptHandle());
System.out.println(" MD5OfBody: " + message.getMD5OfBody());
System.out.println(" Body: " + message.getBody());
for (Entry<String, String> entry : message.getAttributes().entrySet()) {
System.out.println(" Attribute");
System.out.println(" Name: " + entry.getKey());
System.out.println(" Value: " + entry.getValue());
}
}
System.out.println();
And this with Request Batching / Client-Side Buffering:
// Create the basic Amazon SQS async client
AmazonSQSAsync sqsAsync = new AmazonSQSAsyncClient();
// Create the buffered client
AmazonSQSAsync bufferedSqs = new AmazonSQSBufferedAsyncClient(sqsAsync);
CreateQueueRequest createRequest = new CreateQueueRequest().withQueueName("MyTestQueue");
CreateQueueResult res = bufferedSqs.createQueue(createRequest);
SendMessageRequest request = new SendMessageRequest();
String body = "test message_" + System.currentTimeMillis();
request.setMessageBody( body );
request.setQueueUrl(res.getQueueUrl());
SendMessageResult sendResult = bufferedSqs.sendMessage(request);
ReceiveMessageRequest receiveRq = new ReceiveMessageRequest()
.withMaxNumberOfMessages(10)
.withQueueUrl(queueUrl);
ReceiveMessageResult rx = bufferedSqs.receiveMessage(receiveRq);
List<Message> messages = rx.getMessages();
for (Message message : messages) {
System.out.println(" Message");
System.out.println(" MessageId: " + message.getMessageId());
System.out.println(" ReceiptHandle: " + message.getReceiptHandle());
System.out.println(" MD5OfBody: " + message.getMD5OfBody());
System.out.println(" Body: " + message.getBody());
for (Entry<String, String> entry : message.getAttributes().entrySet()) {
System.out.println(" Attribute");
System.out.println(" Name: " + entry.getKey());
System.out.println(" Value: " + entry.getValue());
}
}
But I am still unable to retrieve all messages.
Any idea?
AWS Forum keeps silence on my post.

When receiving messages from an SQS queue, you need to repeatedly call sqs:ReceiveMessage.
On each call to sqs:ReceiveMessage, you will get 0 or more messages from the queue which you'll need to iterate through. For each message, you'll also need to call sqs:DeleteMessage to remove the message from the queue when you're done processing each message.
Add a loop around your "Long Polling" sample above to receive all messages.
for (;;) {
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages) {
System.out.println(" Message");
System.out.println(" MessageId: " + message.getMessageId());
System.out.println(" ReceiptHandle: " + message.getReceiptHandle());
System.out.println(" MD5OfBody: " + message.getMD5OfBody());
System.out.println(" Body: " + message.getBody());
for (Entry<String, String> entry : message.getAttributes().entrySet()) {
System.out.println(" Attribute");
System.out.println(" Name: " + entry.getKey());
System.out.println(" Value: " + entry.getValue());
}
}
System.out.println();
}
Also note that you may receive the same message more than once. So allow your work to "reprocess" the same message, or detect a repeated message.

I too was facing same issue - only one message was getting returned , then i tried
receiveMessageRequest.setMaxNumberOfMessages(10) , which would help me in retrieving 10 messages in a loop,
since my queue has >500 records what i did was
List<String> messagelist = new ArrayList<>();
try
{
AmazonSQS sqs = new AmazonSQSClient(credentials);
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
sqs.setRegion(usWest2);
boolean flag = true;
while(flag)
{
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queuename);
receiveMessageRequest.setMaxNumberOfMessages(number_of_message_);
receiveMessageRequest.withMaxNumberOfMessages(number_of_message_).withWaitTimeSeconds(wait_time_second_);
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages)
{
// System.out.println(" Body: " + message.getBody());
messagelist.add( message.getBody());
String messageReceiptHandle = message.getReceiptHandle();
sqs.deleteMessage(new DeleteMessageRequest().withQueueUrl(queuename).withReceiptHandle(messageReceiptHandle));
}
if(messages.size()==0)
{
flag = false;
}
}
}
catch (AmazonServiceException ase) {
ase.printStackTrace();
} catch (AmazonClientException ace) {
ace.printStackTrace();
}
finally {
return messagelist ;
}
I am reading records from SQS then saving it into a String list and then deletion the record from queue.
so in the end i will have all the data from the queue in a list

An SQS queue is not a database. You can't read all the messages into a list like you are trying to do. There is no beginning and no end to the queue. You poll the queue and ask for some messages, it returns you some messages if they exist.
If you want a method that can return the entire dataset, then sqs is not the right tool - a traditional database might be better in that case.

Long polling will wait if there is no message in Queue. This means that if you call ReceiveMessage with long polling in loop you are guaranteed that you will get all messages. When there is 0 messages received in response, you've already received all messages.
You mentioned that you used also web console. Web console works in same way as calling API with SDK. This means that when you receive and see messages in console, messages are invisible to other clients until visibility timeout expires. That's probably reason why you don't see messages.
See more information about visibility timeout:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html