I'm writing an small app to generate stats in a GMail inbox to help with the endless task of cleaning our mailbox.
My code is working, but I am unnecessarily download messages's payload data to access only the message headers (From field header).
The method Gmail.Users.Messages.Get.setFields(String) with payload option, but payload is exactly the heaviest part of it. I already tried using values like "payload.header", "payload.headers", but none of these values work.
So the question is, how can I access only message's from field instead of fetching entire payload data in order to improve performance?
Map<String, Integer> emailAddressCountMap = new HashMap<String, Integer>();
for (Message message : messages) {
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload").execute();
Stream<String> fromHeaderValue = m1.getPayload().getHeaders().stream()
.filter(h -> "From".equals(h.getName())).map(h -> h.getValue());
String emailAddress = fromHeaderValue.toArray(String[]::new)[0];
Integer count = emailAddressCountMap.get(emailAddress);
if (count == null) {
count = 0;
}
emailAddressCountMap.put(emailAddress, count + 1);
System.out.println(emailAddress + ": " + count);
}
How about a following modification?
From :
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload").execute();
To :
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload/headers").execute();
If this didn't work, or if I misunderstand your question, I'm sorry.
Related
I have gatling script to send HTTP request with an array of 10,000 email addresses, the JSON body is like this one:
{
"userIds": [
"user1#paragonia.com",
"user2#bedlam.com",
"user3#blurrybus.com",
"user4#barkarama.com",
"user5#bullzone.com",
.
.
.
"user10000#miraclis.com"
]
}
So, I generate an ArrayList of 10,000 random email addresses:
val emails = new util.ArrayList[String]
for(i <- 1 to 10000) {
emails.add("\"" + Random.alphanumeric.take(8).mkString.toLowerCase +
"#" + Random.alphanumeric.take(10).mkString.toLowerCase + ".com\"")
}
And I need to feed that ArrayList into my scenario:
val scn = scenario("Add Users")
.exec(
http("AddUsers")
.post(path)
.header("Authorization", apiKey)
.body(StringBody("{" +
"\n\t\"userIds\": " +
userNames +
"\n\t\n" +
"}")).asJson
)
The problem is that the same array sent to all the requests in my scenario, and I need to generate a different array every time.
I guess I need to convert my ArrayList to a feeder or an Iterator but I'm stuck on it.
Is it possible to do such thing in Gatling?
I found the answer.
I created a function to build the ArrayList of random emails:
def getEmailsArray(count: Integer): util.ArrayList[String] = {
val emails = new util.ArrayList[String]
for (i <- 1 to count) {
emails.add("\"" + Random.alphanumeric.take(8).mkString.toLowerCase +
"#" + Random.alphanumeric.take(10).mkString.toLowerCase + ".com\"")
}
emails
}
Then I get the ArrayList into a feeder:
val emailsFeeder = Iterator.continually(Map("emails" -> getEmailsArray(totalEmails)))
I am new to Kafka. I am using Kafka 0.9.0.0 client for java. While consuming the data from a particular topic, I am getting same message every time (Which was posted for the first time ), when I start the producer-consumer java project.
My Requirement is to produce some message and consume it and check if both the messages are same or not.
Below is the code I am using for Kafka Consumer:-
KafkaConsumer<String, String> newConsumer = new KafkaConsumer<String, String>(properties);
newConsumer.subscribe(Collections.singletonList(props.getProperty("monitoring.topic")));
String consumerRecord = "";
ConsumerRecords<String, String> consumerRecords = newConsumer.poll(120000);
for (ConsumerRecord<String, String> record : consumerRecords) {
logger.info("Found message for {} {} {}", adapter, record.key(), record.value());
System.out.println("consumerMessage : " + record.value());
JSONObject jsonConsumerMessage = (JSONObject) (parser.parse(record.value()));
Long offset = record.offset();
System.out.println("Offset of this record is " + offset);
String UUIDProducer = message.get("UUID").toString();
String UUIDConsumer = jsonConsumerMessage.get("UUID").toString();
System.out.println("UUIDProducer : " + UUIDProducer);
System.out.println("UUIDConsumer : " + UUIDConsumer);
if (UUIDProducer.equals(UUIDConsumer)) {
return true;
} else
return false;
}
Note: -I am able to consume the latest messages through command line.
Can anyone please guide me on this ?
It was my silly mistake that I am returning the true and false value inside for loop. Its causing the loop to come out as soon as first message came from the topic.
I have a question about mapping, map key and map values.
I am writing a chat program : I have a problem to add a message. I can't add a message. That puts me in a empty web page with an error(can't see the number and reason of error)
Can you tell me where is the problem ?
// add a message to a chatroom
#RequestMapping(value="/addMessageSalon/{salon}/{pseudo}/{message}", method = {RequestMethod.GET, RequestMethod.POST})
public String addMessageSalon(HttpServletRequest request, #PathVariable("salon") String chatroom, #PathVariable("pseudo") String username, #PathVariable("message") String message) {
Message mes = null;
mes.setMessage(message);
mes.setPseudo(username);
GestionMessages addition = (GestionMessages)request.getSession().getServletContext().getAttribute("gestionMessages");
Map<String, ArrayList<Message>> resultat = addition.getMessages();
Iterator<Map.Entry<String, ArrayList<Message>>> entries = resultat.entrySet().iterator();
// iteration
while(entries.hasNext()) {
Map.Entry<String, ArrayList<Message>> entry = entries.next();
if(!entries.hasNext() && !entry.getKey().contains(chatroom)) {
// if chatroom does not exist, we give an error
throw new IllegalArgumentException("Chatroom '" + chatroom + "' doesn't exist");
}
if(entry.getKey().contains(chatroom)){
ControleurPrincipal.getUsersInDataBase().add(username);
addition.getMessagesSalon(chatroom).add(mes);
break;
}
}
resultat = addition.getMessages();
return "redirect:/";
}
First of all:
Message mes = null;
mes.setMessage(message);
This will throw a NullPointerException, every time. So either that's the error you're getting, or that is not your code.
If that's your actual code, then you need to instantiate Message first, like this:
Message mes = new Message();
Instead of doing this
if(!entries.hasNext() && !entry.getKey().contains(monSalon)) {
you might want to do
if(!resultat.contains(monSalon)) {
and do it before the while.
So I have a collection of emails and what I want to do is use them to output unique triplets (sender email, receiver email, timestamp) like so:
user1#stackoverflow.com user2#stackoverflow.com 09/12/2009 16:45
user1#stackoverflow.com user9#stackoverflow.com 09/12/2009 18:45
user3#stackoverflow.com user4#stackoverflow.com 07/05/2008 12:29
In the above example user 1 sent a single email to multiple recipients (user 2 and user 9). To store the recipients, I created a data structure EdgeWritable(implements WritableComparable)that will hold the Sender and Recipient email addresses as well as a Timestamp.
My mapper looks like this:
private final EdgeWritable edge = new EdgeWritable(); // Data structure for triplets.
private final NullWritable noval = NullWritable.get();
...
#Override
public void map(Text key, BytesWritable value, Context context)
throws IOException, InterruptedException {
byte[] bytes = value.getBytes();
Scanner scanner = new Scanner(new ByteArrayInputStream(bytes), "UTF-8");
String from = null; // Sender's Email address
ArrayList<String> recipients = new ArrayList<String>(); // List of recipients' Email addresses
long millis = -1; // Date
// Parse information from file
while(scanner.hasNext()) {
String line = scanner.nextLine();
if (line.startsWith("From:")) {
from = procFrom(stripCommand(line, "From:")); // Get sender e-mail address.
} else if (line.startsWith("To:")) {
procRecipients(stripCommand(line, "To:"), recipients); // Populate recipients into a list.
} else if (line.startsWith("Date:")) {
millis = procDate(stripCommand(line, "Date:")); // Get timestamp.
if (line.equals("")) { // Empty line indicates the end of the header
break;
}
}
scanner.close();
// Emit EdgeWritable as intermediate key containing Sender, Recipient and Timestamp.
if (from != null && recipients.size() > 0 && millis != -1) {
//EdgeWritable has 2 Text values (ew[0] and ew[1]) and a Timestamp. ew[0] is the sender, ew[1] is a recipient.
edge.set(0, from); // Set ew[0]
for(int i = 0; i < recipients.size(); i++) {
edge.set(1, recipients.get(i)); // Set edge from sender to each recipient i.
edge.setTS(millis); // Set date.
context.write(edge, noval); // Emit the edge as an intermediate key with a null value.
}
}
}
...
My reducer simply formats the date and outputs the edges:
public void reduce(EdgeWritable key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException {
String date = MailReader.sdf.format(edge.getTS());
out.set(edge.get(0) + " " + edge.get(1) + " " + date); // same edge from Mapper (an EdgeWritable).
context.write(noval, out); // same noval from Mapper (a NullWritable).
}
Using EdgeWritable as the intermediate key and NullWritable as the value (in mapper) is a requirement, I'm not permitted to use other methods. This is my first Hadoop / MapReduce program and I just wanted to know that I'm going in the right direction. I have looked at plenty of MapReduce examples online and have never seen key/value pairs being emitted in a for-loop the way I have done it. I feel like I'm missing some sort of trick here, but using a for-loop in this way is the only approach I can think of.
Is this 'bad'? I hope this is clear but please let me know if any further clarification is needed.
Map method gets called for each record, so your array list is having only 1 record for every call. Declare your array list at class level so that u can store values for all records. Then in clean up method you can do the emit logic which you have written inside map. Try this and let me know if that works.
I need to batch elements that have similar client id (String type, but at the moment only numeric values, like "12345", "235134", etc.)
Map<String, List<Client>> _batched = new HashMap<String, List<Client>>();
for (Client c : _Clients)
{
String id = c.getIdClient();
List<Client> clients = _batched.get(id);
if(_clients == null){
clients = new ArrayList<Client>();
_batched.put(id, clients);
}
clients.add(c);
}
The problem is that when I compare this function with the results of Excel (=SUM(IF(FREQUENCY(C2:C618,C2:C618)>0,1))), then I get different results, i.e. 526 and 519.
Is something wrong with my code?
Your problem is here:
String id = c.getIdClient();
List<Client> _clients = _batched.get(id);
if(_clients == null){
pois = new ArrayList<Client>();
_batched.put(id, _clients);
}
_clients.add(c);
You create a new array into a variable called pois but then put the contents of the variable _clients into _batched. What happens with the value put into pois?
I don't understand how that doesn't null pointer exception actually.