I use javamail to read mails from an exchage account using IMAP protocol. Those mails are in plain format and its contents are XMLs.
Almost all those mails have short size (usually under 100Kb). However, sometimes I have to deal with large mails (about 10Mb-15Mb). For example, yesterday I received an email which was 13Mb size. It took more than 50min just to read it. Is it normal? Is there a way to increase its performance?
The code is:
Session sesion = Session.getInstance(System.getProperties());
Store store = sesion.getStore("imap");
store.connect(host, user, passwd);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_WRITE);
Message[] messages = inbox.search(new FlagTerm(new Flags(Flags.Flag.SEEN), false));
for (int i = 0 ; i< messages.length ; i++){
Object contents = messages[i].getContent(); // Here it takes 50 min on 13Mb mail
// ...
}
Method that takes such a long time is messages[i].getContent(). What am I doing wrong? Any hint?
Thanks a lot and sorry for my english! ;)
I finally solved this issue and wanted to share.
The solution, at least the one that worked to me, was found in this site: http://www.oracle.com/technetwork/java/faq-135477.html#imapserverbug
So, my original code typed in my first post becomes to this:
Session sesion = Session.getInstance(System.getProperties());
Store store = sesion.getStore("imap");
store.connect(host, user, passwd);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_WRITE);
// Convert to MimeMessage after search
MimeMessage[] messages = (MimeMessage[]) carpetaInbox.search(new FlagTerm(new Flags(Flags.Flag.SEEN), false));
for (int i = 0 ; i< messages.length ; i++){
// Create a new message using MimeMessage copy constructor
MimeMessage cmsg = new MimeMessage(messages[i]);
// Use this message to read its contents
Object obj = cmsg.getContent();
// ....
}
The trick is, using MimeMessage() copy constructor, create a new MimeMessage and read its contents instead of original message.
You should note that such object is not really connected to server, so any changes you make on it, like setting flags, won't take effect. Any change on message, have to be done on original message.
To sum up: This solution works reading large Plain Text mails (up to 15Mb) connecting to an Exchange Server using IMAP protocol. The times lowered from 51-55min to read a 13Mb mail, to 9seconds to read same mail. Unbelievable.
Hope this helps someone and sorry for English mistakes ;)
It would always be messages[i].getContent() that would be the slowest part of the code. The reason is normally IMAP server would not cache this part of message data. Nevertheless, you can try this:
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add(FetchProfileItem.FLAGS);
fp.add(FetchProfileItem.CONTENT_INFO);
fp.add("X-mailer");
and after you have specified the fetch profile then you do your search/fetch of messages.
Basically the concept is that the IMAP provider fetches the data for a message from the server only when necessary. (The javax.mail.FetchProfile is used to optimize this). The header and body structure information, once fetched, is always cached within the Message object. However, the content of a bodypart is not cached. So each time the content is requested by the client (either using getContent() or using getInputStream()), a new FETCH request is issued to the server. The reason for this is that the content of a message could be potentially large, and if we cache this content for a large number of messages, there is the possibility that the system may run out of memory soon since the garbage collector cannot free the referenced objects. Clients should be aware of this and must hold on to the retrieved content themselves if needed.
By using the above mentioned code snippet you could 'hope' for some speed improvement but it solely depends on your SMTP server if this would work or not. All the big SMTP server do not support this behaviour because of the load issue mentioned in the previous paragraph and hence you may not gain any speed.
Using the Folder.fetch method you can prefetch in one operation the metadata for multiple messages. That will reduce the time to process each message, but won't help that much with a huge message.
The handle huge message parts efficiently, you'll generally want to use the getInputStream method to process the data incrementally, rather than using the getContent method to read all the data in and create a huge String object with all the data.
You can also tune the fetching by specifying the "mail.imap.fetchsize" property, which defaults to 16384. If most of your messages are less than 100K, and you always need to read all of the data in the message, you might set the fetchsize to 100K. That will make small messages much faster and larger message more efficient.
I had a similar issue. Fetching mails via IMAP were very slow. Furthermore I had another issue downloading large attachments. After a look in the JavaMail FAQ I found the solution for the later issue in this question that advises to set the mail.imap.partialfetch (respectively mail.imaps.partialfetch) to false. This not only fixes the download issue but the slow reading of the messages as well.
In the referenced JavaMail notes.txt it is said.
Due to a problem in the Microsoft Exchange IMAP server, insufficient
number of bytes may be retrieved when reading big messages. There
are two ways to workaround this Exchange bug:
(a) The Exchange IMAP server provides a configuration option called
"fast message retrieval" to the UI. Simply go to the site, server
or recipient, click on the "IMAP4" tab, and one of the check boxes
is "enable fast message retrieval". Turn it off and the octet
counts will be exact. This is fully described at
http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q191504
(b) Set the "mail.imap.partialfetch" property to false. You'll have
to set this property in the Properties object that you provide to
your Session.
Certain IMAP servers do not implement the IMAP Partial FETCH
functionality properly. This problem typically manifests as corrupt
email attachments when downloading large messages from the IMAP
server. To workaround this server bug, set the
"mail.imap.partialfetch"
property to false. You'll have to set this property in the Properties
object that you provide to your Session.
Related
I have currently a huge problem for which I need help for it.
Currently I´m not loading all emails at once.
I found here the following function for it:
Message[] messages = emailFolder.getMessages(start, end);
I know I can use SortTerm for sorting the emails:
SortTerm sortTerm[] = new SortTerm[] { SortTerm.REVERSE, SortTerm.DATE };
Message messages = ((IMAPFolder) emailFolder).getSortedMessages(sortTerm);
But than I will load again all emails.
How can I use together:
- search
- sort
- and use getMessages(start, end)
A sample code would be very helpful.
Many thanks
To be clear, when using IMAP no messages are "loaded" when you call getMessages. All that happens is that the JavaMail client creates a Message object that refers to the message on the server, and sets it up so that the Message object will fetch the data for the message on the server when you ask for it.
You could create a SearchTerm that uses a pair of MessageNumberTerms to constrain the message to be within a certain range just as you were doing with "start, end". But you should ask yourself whether you really want to sort all the messages in the mailbox first by message number (effectively a forward sort by received date) and then reverse sort them by sent date. What exactly are you trying to accomplish?
I am using Amazon AWS SES to send my email campaigns. I have around 35,000 subscribers on my list. At present I am using a code something similar to the following.
for (Entry<Integer, String> emailEntry : email_ids.entrySet()) {
MimeMessage msg = getMimeMessage(emailEntry.getKey(), emailEntry.getValue());
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
msg.writeTo(outputStream);
RawMessage rawMessage = new RawMessage(ByteBuffer.wrap(outputStream.toByteArray()));
ses.sendRawEmail(new SendRawEmailRequest(rawMessage));
}
This way I was able to send email to all my subscribers the way I wanted. But there was a huge bill accounting to Data transfer. Each MimeMessage is of 150Kb in size and sending it to 35,000 subscribers resulted in 5.5 GB of data transfer.
So I decided to use BulkTemplateEmail in my application, to create the template once and send it to 35,000 emails. This way the email has to be send to SES only once and there will be significant gain in terms of data transfer.
Can anyone provide me a sample to do this via Java AWS SDK? I want to add List-Unsubscribe header on each Destination. This is where I am actually stuck. Couldn't find any methods to add custom email headers for each Destination. Is this possible with BulkTemplateEmail?
Any info is highly appreciated.
When sending emails using SES, Amazon charges for data transfer out. The current price is $0.12 per GB. For large volumes of emails this can result in serious charges.
Amazon SES pricing
For embedded images, attachments, etc. another solution is to use links instead of embedded objects. This way you can mitigate and reduce data transfers fees. This can have a moderate to high impact for email campaigns where a lot of emails are never opened, thereby saving on the data transfer charges.
If your links reference files on your EC2 instances, remember that you will still be charged for Data-Out from your EC2 instances. S3 will provide a lower cost.
We use java mail api with imap and fetch messages of folders containing millions of messages. There are some rules and limitiations:
We do not have always open connections to mail server and therefore we can not add listeners.
The messages will be stored in a local database with all properties, subject, body, receive date, from etc.
Can not use multiple threads
To keep the performance at acceptable levels and prevent out of memory crashes, I am planning:
1.During inital fetch, where all messages have to be fetched, store only the message headers and bypass body and attachments. Getting the body and attachment of a message will be done when requested by the client. The initialization can take hours, it is not a problem.
2.When fetching all messages at start, use a appropriate fetch profile to make it faster, but process in blocks, for example:
Message m1[] = f.getMessages(1, 10000);
f.fetch(m1, fp);
//process m1 array
Message m2[] = f.getMessages(10001, 20000);
f.fetch(m2, fp);
//process m2 array
instead of
Message m_all[] = f.getMessages(1, NUMALLMESSAGES);
f.fetch(m_all, fp);
//process m_all array, may throw out of memory errors
3.And after we have all the messages, store the UID of recent message in the DB and on the next fetch perform:
f.getMessagesByUID(LASTUIDREADFROMDB, UIDMAX)
Do you have additional suggestions, or see any points we have to care of (memory, performance)
So I am trying to figure out how to get messages with UIDs
IMAPStore store = (IMAPStore) session.getStore("imaps");
store.connect();
IMAPFolder folder = (IMAPFolder) store.getFolder(FOLDER_NAME);
folder.open(Folder.READ_ONLY);
Then I want to search the folder:
Message unreadMessages[] =
folder.search(new FlagTerm( new Flags(Flags.Flag.SEEN), false));
But the messages in the array are returned without UIDs, so how do I pull them out later for processing without an ID to reference them by?
I'm assuming you're talking about IMAP UIDs and you know how IMAP UIDs work. The deleted answer explained this well. Folder.search gives you a bunch of Message objects. Using the UIDFolder.getUID method, you can iterate over each of those Message objects and get its corresponding UID. Using the Folder.fetch method, you can prefetch those UIDs with a single IMAP command so the iteration over each Message to get its UID happens locally. If you save the UIDs, you can later use the UIDFolder methods to get back the corresponding Message objects. Don't forget to check the UIDVALIDITY.
I'm building a client for trading with a remote server using FIX protocol and QuickFix/J API.
I can send order, receive price updates, cancel orders etc...
I'm asked now to "query API for current position of an instrument".
So let's say I can submit an order for buying an instrument, and it doesn't get executed, I would like to receive from the server some information like "you are LONG on intrument X with quantity Y etc".
Is it possible using QuickFix/J API?
I have written a method like this
static void positionReport() throws SessionNotFound{
quickfix.fix50.PositionReport order = new quickfix.fix50.PositionReport();
SessionID sessionId = (SessionID) initiator.getSessions().get(0);
order.set(new Account("1005390"));
order.set(new SecurityID("4663789"));
order.set(new SecurityExchange("XETR"));
order.set(new Symbol("SAP"));
Session.sendToTarget(order, sessionId);
}
which sends FIX messages like this
8=FIX.4.29=9835=AP34=4949=HIQ6_ORDER52=20140324-
15:54:10.14256=HIQFIX1=100539048=466378955=SAP207=XETR10=199
and receives messages like this:
8=FIX.4.29=9935=334=6949=HIQFIX52=20140324-15:54:10.89156=HIQ6_ORDER45=4958=Invalid
MsgType372=AP373=1110=242
As you can see I get "Invalid message" error
Check your counterparty's documentation.
FIX is a fairly "dumb" protocol. It just provides a communication infrastructure. The default message definitions are best thought of as a list of suggested messages that you can use. Even if one message type is supported by two counterparties, it's possible that each of the two counterparties could use it in totally different ways.
Most connection providers only use a subset of these messages. You should check their documentation to see if they support the PositionRequest message, and to see how they want you to set the fields in it.
No you cannot do that using Quickfix, unless and until the counterparty is modelled to give you FIX acknowledgements to your specific liking. That is why you can add your customized FIX fields to the FIX XML config file.
373 tag says 11 -> 11 = Invalid MsgType
58 confirms it for you again.
Check your FIX XML config and check if your message is complete and if your counterparty allows the messages of type AP.