I've been using javamail to retrieve mails from IMAP server (currently GMail). Javamail retrieves list of messages (only ids) in a particular folder from server very fast, but when I actually fetch message (only envelop not even contents) it takes around 1 to 2 seconds for each message. What are the techniques should be used for fast retrieval?
here is my code:
try {
IMAPStore store = null;
if(store!=null&&store.isConnected())return;
Properties props = System.getProperties();
Session sessionIMAP = Session.getInstance(props, null);
try {
store = (IMAPStore) sessionIMAP.getStore("imaps");
store.connect("imap.gmail.com",993,"username#gmail.com","password");
} catch (Exception e) {
e.printStackTrace();
}
IMAPFolder folder = (IMAPFolder) store.getFolder("INBOX");
folder.open(Folder.READ_ONLY);
System.out.println("start");
Message[] msgs = folder.getMessages(1,10);
long ftime = System.currentTimeMillis();
FetchProfile fp=new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
folder.fetch(msgs, fp);
long time = System.currentTimeMillis();
System.out.println("fetch: "+(time-ftime));
for (Message message : msgs) {
System.out.println(message.getSubject());
Address[] from = message.getFrom();
for (Address address : from) {
System.out.println(address);
}
Address[] recipients = message.getAllRecipients();
for (Address address : recipients) {
System.out.println(address);
}
}
long newTime = System.currentTimeMillis();
System.out.println("convert: "+(newTime-time));
}catch (Exception e) {
e.printStackTrace();
}
}
I believe that Gmail throttles the IMAP message reads to one every second or so. You might be able to speed it up with multiple IMAP connections.
Please set the Property mail.imap.fetchsize with the required size. the default is 16k.
In case you increase the size of this property, retrieve speed will go up.
props.put("mail.imap.fetchsize", "3000000");
Note that if you're using the "imaps" protocol to access IMAP over SSL, all the properties would be named "mail.imaps.*".
Good Luck.
Yaniv
I'm not sure if this is a Javamail issue as much as it may be a Gmail issue. I have an application that retrieves mail from a number of sources, including Gmail, and Gmail is definitely the slowest. The Javamail api is pretty straightforward, but it would be hard to make suggestions without seeing what you are currently doing.
I'm running into the same thing. After profiling, I noticed that getBody was being called every time I tried to do a message.getFrom() like you are, even though I was only accessing fields that should be covered by the Envelope flag. See https://java.net/projects/javamail/forums/forum/topics/107956-gimap-efficiency-when-only-reading-headers
Related
I am trying to write a simple code that will read messages from gmail inbox.
I have found some examples, but non of them is working.
Most promising is code I've found on CompilatimEerror.com, BUT whatever I try I get this error:
java.lang.ClassCastException: java.lang.String cannot be cast to javax.mail.Multipart
Here is my code:
import java.util.*;
import javax.mail.*;
public class ReadingEmail {
public static void main(String[] args) {
Properties props = new Properties();
props.setProperty("mail.store.protocol", "imaps");
try {
Session session = Session.getInstance(props, null);
Store store = session.getStore();
store.connect("imap.gmail.com", "yourEmailId#gmail.com","password");
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_ONLY);
Message msg = inbox.getMessage(inbox.getMessageCount());
Address[] in = msg.getFrom();
for (Address address : in) {
System.out.println("FROM:" + address.toString());
}
Multipart mp = (Multipart) msg.getContent(); // here it breaks
BodyPart bp = mp.getBodyPart(0);
System.out.println("SENT DATE:" + msg.getSentDate());
System.out.println("SUBJECT:" + msg.getSubject());
System.out.println("CONTENT:" + bp.getContent());
} catch (Exception mex) {
mex.printStackTrace();
}
}
}
There is no connection error, as it gets the subject, date, and all this stuff, but the email body is a mystery. There will be no attachments, I will get only simple mails(this is a part of greater project)
What I am looking for is to read the unread mail (and then delete this message, so the inbox will be permanently empty(spam will be deleted manually)).
I lack knowledge about web programing/structures, and all that pop's, imaps and stuff is a blank space.
Also keep in mind that I am a novice programmer and this is the first time that I go outside of my computer with my code unfortunately straight into the problems of protocols / authentication / getting things from internet.
I went through a lot of pages, but never found an explanation that would allow me to create it myself...
Are you related to this guy?
The JavaMail FAQ has pointers to lots of helpful information, including the JavaMail project page, sample code, etc. You'll also find pointers to some useful background material and tutorials here.
I am currently trying to use JavaMail to get emails from IMAP servers (Gmail and others). Basically, my code works: I indeed can get the headers, body contents and so on. My problem is the following: when working on an IMAP server (no SSL), it basically takes 1-2ms to process a message. When I go on an IMAPS server (hence with SSL, such as Gmail) I reach around 250m/message. I ONLY measure the time when processing the messages (the connection, handshake and such are NOT taken into account).
I know that since this is SSL, the data is encrypted. However, the time for decryption should not be that important, should it?
I have tried setting a higher ServerCacheSize value, a higher connectionpoolsize, but am seriously running out of ideas. Anyone confronted with this problem? Solved it one might hope?
My fear is that the JavaMail API uses a different connection each time it fetches a mail from the IMAPS server (involving the overhead for handshake...). If so, is there a way to override this behavior?
Here is my code (although quite standard) called from the Main() class:
public static int connectTest(String SSL, String user, String pwd, String host) throws IOException,
ProtocolException,
GeneralSecurityException {
Properties props = System.getProperties();
props.setProperty("mail.store.protocol", SSL);
props.setProperty("mail.imaps.ssl.trust", host);
props.setProperty("mail.imaps.connectionpoolsize", "10");
try {
Session session = Session.getDefaultInstance(props, null);
// session.setDebug(true);
Store store = session.getStore(SSL);
store.connect(host, user, pwd);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_ONLY);
int numMess = inbox.getMessageCount();
Message[] messages = inbox.getMessages();
for (Message m : messages) {
m.getAllHeaders();
m.getContent();
}
inbox.close(false);
store.close();
return numMess;
} catch (MessagingException e) {
e.printStackTrace();
System.exit(2);
}
return 0;
}
Thanks in advance.
after a lot of work, and assistance from the people at JavaMail, the source of this "slowness" is from the FETCH behavior in the API. Indeed, as pjaol said, we return to the server each time we need info (a header, or message content) for a message.
If FetchProfile allows us to bulk fetch header information, or flags, for many messages, getting contents of multiple messages is NOT directly possible.
Luckily, we can write our own IMAP command to avoid this "limitation" (it was done this way to avoid out of memory errors: fetching every mail in memory in one command can be quite heavy).
Here is my code:
import com.sun.mail.iap.Argument;
import com.sun.mail.iap.ProtocolException;
import com.sun.mail.iap.Response;
import com.sun.mail.imap.IMAPFolder;
import com.sun.mail.imap.protocol.BODY;
import com.sun.mail.imap.protocol.FetchResponse;
import com.sun.mail.imap.protocol.IMAPProtocol;
import com.sun.mail.imap.protocol.UID;
public class CustomProtocolCommand implements IMAPFolder.ProtocolCommand {
/** Index on server of first mail to fetch **/
int start;
/** Index on server of last mail to fetch **/
int end;
public CustomProtocolCommand(int start, int end) {
this.start = start;
this.end = end;
}
#Override
public Object doCommand(IMAPProtocol protocol) throws ProtocolException {
Argument args = new Argument();
args.writeString(Integer.toString(start) + ":" + Integer.toString(end));
args.writeString("BODY[]");
Response[] r = protocol.command("FETCH", args);
Response response = r[r.length - 1];
if (response.isOK()) {
Properties props = new Properties();
props.setProperty("mail.store.protocol", "imap");
props.setProperty("mail.mime.base64.ignoreerrors", "true");
props.setProperty("mail.imap.partialfetch", "false");
props.setProperty("mail.imaps.partialfetch", "false");
Session session = Session.getInstance(props, null);
FetchResponse fetch;
BODY body;
MimeMessage mm;
ByteArrayInputStream is = null;
// last response is only result summary: not contents
for (int i = 0; i < r.length - 1; i++) {
if (r[i] instanceof IMAPResponse) {
fetch = (FetchResponse) r[i];
body = (BODY) fetch.getItem(0);
is = body.getByteArrayInputStream();
try {
mm = new MimeMessage(session, is);
Contents.getContents(mm, i);
} catch (MessagingException e) {
e.printStackTrace();
}
}
}
}
// dispatch remaining untagged responses
protocol.notifyResponseHandlers(r);
protocol.handleResult(response);
return "" + (r.length - 1);
}
}
the getContents(MimeMessage mm, int i) function is a classic function that recursively prints the contents of the message to a file (many examples available on the net).
To avoid out of memory errors, I simply set a maxDocs and maxSize limit (this has been done arbitrarily and can probably be improved!) used as follows:
public int efficientGetContents(IMAPFolder inbox, Message[] messages)
throws MessagingException {
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.FLAGS);
fp.add(FetchProfile.Item.ENVELOPE);
inbox.fetch(messages, fp);
int index = 0;
int nbMessages = messages.length;
final int maxDoc = 5000;
final long maxSize = 100000000; // 100Mo
// Message numbers limit to fetch
int start;
int end;
while (index < nbMessages) {
start = messages[index].getMessageNumber();
int docs = 0;
int totalSize = 0;
boolean noskip = true; // There are no jumps in the message numbers
// list
boolean notend = true;
// Until we reach one of the limits
while (docs < maxDoc && totalSize < maxSize && noskip && notend) {
docs++;
totalSize += messages[index].getSize();
index++;
if (notend = (index < nbMessages)) {
noskip = (messages[index - 1].getMessageNumber() + 1 == messages[index]
.getMessageNumber());
}
}
end = messages[index - 1].getMessageNumber();
inbox.doCommand(new CustomProtocolCommand(start, end));
System.out.println("Fetching contents for " + start + ":" + end);
System.out.println("Size fetched = " + (totalSize / 1000000)
+ " Mo");
}
return nbMessages;
}
Do not that here I am using message numbers, which is unstable (these change if messages are erased from the server). A better method would be to use UIDs! Then you would change the command from FETCH to UID FETCH.
Hope this helps out!
You need to add a FetchProfile to the inbox before you iterate through the messages.
Message is a lazy loading object, it will return to the server for each message and for each
field that doesn't get provided with the default profile.
e.g.
for (Message message: messages) {
message.getSubject(); //-> goes to the imap server to fetch the subject line
}
If you want to display like an inbox listing of say just From, Subject, Sent, Attachement etc.. you would use something like the following
inbox.open(Folder.READ_ONLY);
Message[] messages = inbox.getMessages(start + 1, total);
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add(FetchProfileItem.FLAGS);
fp.add(FetchProfileItem.CONTENT_INFO);
fp.add("X-mailer");
inbox.fetch(messages, fp); // Load the profile of the messages in 1 fetch.
for (Message message: messages) {
message.getSubject(); //Subject is already local, no additional fetch required
}
Hope that helps.
The total time includes the time required in cryptographic operations. The cryptographic operations need a random seeder. There are different random seeding implementations which provide random bits for use in the cryptography. By default, Java uses /dev/urandom and this is specified in your java.security as below:
securerandom.source=file:/dev/urandom
On Windows, java uses Microsoft CryptoAPI seed functionality which usually has no problems. However, on unix and linux, Java, by default uses /dev/random for random seeding. And read operations on /dev/random sometimes block and takes long time to complete. If you are using the *nix platforms then the time spent in this would get counted in the overall time.
Since, I dont know what platform you are using, I can't for sure say that this could be your problem. But if you are, then this could be one of reasons why your operations are taking long time. One of the solution to this could be to use /dev/urandom instead of /dev/random as your random seeder, which does not block. This can be specified with the system property "java.security.egd". For example,
-Djava.security.egd=file:/dev/urandom
Specifying this system property will override the securerandom.source setting in your java.security file. You can give it a try. Hope it helps.
Note: added after answer:
Thanks.. Yeah I had tried the Flag.SEEN to true and saveChanges.. I also had read getContent marks it read. I tried using it in the for statement that loops through the messages. But I got the messages again from the folder anyways in the next loop. I was assuming the folder was live, so grabbing the content, then grabbing the messages again from the folder with the filter to not get any seen should work, but I was still getting the same message. I could try closing the folder and reopen as a test to see if it's marked. Also if I go over to my client and click the message, then my code stops seeing it even in the loop, so I was hoping to do the same in the code.
original:
I'm using javamail to get email from a gmail account, it's working great, when I get the message I'd like to mark it as read, can anyone give me some direction? Here is my current code:
Properties props = System.getProperties();
props.setProperty("mail.store.protocol", "imaps");
try {
Session session = Session.getDefaultInstance(props, null);
Store store = session.getStore("imaps");
store.connect("imap.gmail.com", eUserName, ePassWord);
// Get folder
Folder folder = store.getFolder("INBOX");
if (folder == null || !folder.exists()) {
return null;
}
folder.open(Folder.READ_ONLY);
// Only pull unread
FlagTerm ft = new FlagTerm(new Flags(Flags.Flag.SEEN), false);
Message messages[]; // = folder.search(ft);
for(int x = 0; x < timeOutInSeconds; x++) {
log.reportMessage("looking for emails");
try {
folder.getMessages();
messages = folder.search(ft);
if (messages.length > 0) {
for (Message message : messages) {
//log.reportMessage("found message: should not see again, marking read");
// want to mark as read
}
}
Thread.sleep(1000);
}
catch(Exception ex) {
}
}
// Close connection
folder.close(false);
store.close();
return null;
}
catch (NoSuchProviderException ex) {
return null;
}
catch (MessagingException ex) {
return null;
}
}
First of all, you can't mark a message as read if you are using a POP3 server - the POP3 protocol doesn't support that. However, the IMAP v4 protocol does.
You might think the way to do this is to get the message, set the Flags.Flag.SEEN flag to true, and then call message.saveChanges(). Oddly, this is not the case.
Instead, the JavaMail API Design Specification, Chapter 4, section "The Flags Class" states that the SEEN flag is implicitly set when the contents of a message are retrieved. So, to mark a message as read, you can use the following code:
myImapFolder.open(Folder.READ_WRITE);
myImapFolder.getMessage(myMsgID).getContent();
myImapFolder.close(false);
Or another way is to use the MimeMessage copy constructor, ie:
MimeMessage source = (MimeMessage) folder.getMessage(1)
MimeMessage copy = new MimeMessage(source);
When you construct the copy, the seen flag is implicitly set for the message referred to by source.
One liner that will do it WITHOUT downloading the entire message:
single message:
folder.setFlags(new Message[] {message}, new Flags(Flags.Flag.SEEN), true);
all messages:
folder.setFlags(messages, new Flags(Flags.Flag.SEEN), true);
Other methods of calling getContent() or creating a copy with new MimeMessage(original) cause the client to download the entire message, and creates a huge performance hit.
Note that the inbox must be opened for READ_WRITE:
folder.open(Folder.READ_WRITE);
Well this post is old but the easiest solution hasnĀ“t been posted yet.
You are accessing the Message.
message.setFlag(Flag.SEEN, true);
for (Message message : messages) {
message.setFlag(Flags.Flag.SEEN,true);
}
and change the below line
folder.open(Folder.READ_ONLY);
to this
folder.open(Folder.READ_WRITE);
You may also consider having a public static int max_message_number, and storing in it the message[i].getMessageNumber(); as soon as you read a message. Then before reading any message just check if the max_message_number < message[i].getmessageNumber(). If true then don't print this message (as it has been already read)
message.setFlag( Flag.SEEN,true ) give "cannot find symbol"
message.setFlag( Flags.Flag.SEEN,true ) seems good.
If you are using a for loop to read or check a mail one by one, the code can be as follows to mark a gmail message as read:
Message[] unreadMessages = inbox.search(new FlagTerm(new Flags(Flag.SEEN), false));
for (int q = 0; q < unreadMessages.length; q++) {
unreadMessages[q].setFlag(Flag.SEEN, true);
}
What this code does is that it makes it unread one by one.
And also folder/inbox needs to be READ_WRITE, instead of READ_ONLY:
folder.open(Folder.READ_WRITE);
You can also try
head over to the gmail settings > Forwarding and POP/IMAP
from the dropdown of When messages are accessed with POP
select mark Gmail's copy as read and save the changes
To mark the mail as read, you just have to call the mailmessage.getContent() method.
Whether you use IMAP or POP, calling that method on a specific mail mark it as Read
The easiest way to do that is set the folder to be read or written into or from. Means like this...
Folder inbox = null;
inbox.open(Folder.READ_WRITE);
the Folder class should be imported.
I am trying to perform a search of my gmail using Java. With JavaMail I can do a message by message search like so:
Properties props = System.getProperties();
props.setProperty("mail.store.protocol", "imaps");
Session session = Session.getDefaultInstance(props, null);
Store store = session.getStore("imaps");
store.connect("imap.gmail.com", "myUsername", "myPassword");
Folder inbox = store.getFolder("Inbox");
inbox.open(Folder.READ_ONLY);
SearchTerm term = new SearchTerm() {
#Override
public boolean match(Message mess) {
try {
return mess.getContent().toString().toLowerCase().indexOf("boston") != -1;
} catch (IOException ex) {
Logger.getLogger(JavaMailTest.class.getName()).log(Level.SEVERE, null, ex);
} catch (MessagingException ex) {
Logger.getLogger(JavaMailTest.class.getName()).log(Level.SEVERE, null, ex);
}
return false;
}
};
Message[] searchResults = inbox.search(term);
for(Message m:searchResults)
System.out.println("MATCHED: " + m.getFrom()[0]);
But this requires downloading each message. Of course I can cache all the results, but this becomes a storage concern with large gmail boxes and also would be very slow (I can only imagine how long it would take to search through gigabytes of text...).
So my question is, is there a way of searching through mail on the server, a la gmail's search field? Maybe through Microsoft Exchange?
Hours of Googling has turned up nothing.
You can let the server do the search for you, with the appropriate IMAP command. The SEARCH command will only get you so far, what you probably need is the SORT command. SORT isn't implemented in JavaMail but the documentation shows how you can implement it yourself:
http://java.sun.com/products/javamail/javadocs/com/sun/mail/imap/IMAPFolder.html#doCommand(com.sun.mail.imap.IMAPFolder.ProtocolCommand)
(I couldn't figure out how to link to a URL with parentheses)
Connect to the Exchange IMAP store and use javax.mail.search.SearchTerm
I'd like to send mail without bothering with the SMTP-Server which is used for delivery.
So JavaMail API doesn't work for me because I have to specify a SMTP server to connect to.
I'd like the library to find out on its own which SMTP server is responsible for which email address by querying the MX record of the mail address domain.
I'm looking for something like Aspirin. Unfortunately I can't use Aspirin itself because the development stopped 2004 and the library fails to communicate with modern spam hardened servers correctly.
An embeddable version of James would do the task. But I haven't found documentation concerning whether this is possible.
Or does anyone know about other libraries I could use?
One possible solution: get the MX record on your own and use JavaMail API.
You can get the MX record using the dnsjava project:
Maven2 dependency:
<dependency>
<groupId>dnsjava</groupId>
<artifactId>dnsjava</artifactId>
<version>2.0.1</version>
</dependency>
Method for MX record retrieval:
public static String getMXRecordsForEmailAddress(String eMailAddress) {
String returnValue = null;
try {
String hostName = getHostNameFromEmailAddress(eMailAddress);
Record[] records = new Lookup(hostName, Type.MX).run();
if (records == null) { throw new RuntimeException("No MX records found for domain " + hostName + "."); }
if (log.isTraceEnabled()) {
// log found entries for debugging purposes
for (int i = 0; i < records.length; i++) {
MXRecord mx = (MXRecord) records[i];
String targetString = mx.getTarget().toString();
log.trace("MX-Record for '" + hostName + "':" + targetString);
}
}
// return first entry (not the best solution)
if (records.length > 0) {
MXRecord mx = (MXRecord) records[0];
returnValue = mx.getTarget().toString();
}
} catch (TextParseException e) {
throw new RuntimeException(e);
}
if (log.isTraceEnabled()) {
log.trace("Using: " + returnValue);
}
return returnValue;
}
private static String getHostNameFromEmailAddress(String mailAddress) throws TextParseException {
String parts[] = mailAddress.split("#");
if (parts.length != 2) throw new TextParseException("Cannot parse E-Mail-Address: '" + mailAddress + "'");
return parts[1];
}
Sending mail via JavaMail code:
public static void sendMail(String toAddress, String fromAddress, String subject, String body) throws AddressException, MessagingException {
String smtpServer = getMXRecordsForEmailAddress(toAddress);
// create session
Properties props = new Properties();
props.put("mail.smtp.host", smtpServer);
Session session = Session.getDefaultInstance(props);
// create message
Message msg = new MimeMessage(session);
msg.setFrom(new InternetAddress(fromAddress));
msg.setRecipient(Message.RecipientType.TO, new InternetAddress(toAddress));
msg.setSubject(subject);
msg.setText(body);
// send message
Transport.send(msg);
}
This is completely the wrong way to handle this.
Anyone connected to the internet will have some kind of "legit" SMTP server available to them to take the submission of email -- your ISP, your office, etc.
You WANT to leverage because they do several things for you.
1) they take your message and the responsibility to handle that message. After you drop it off, it's not your problem anymore.
2) Any mail de-spamming technologies are handled by the server. Even better, when/if those technologies change (Domain keys anyone?), the server handles it, not your code.
3) You, as a client of that sending mail system, already have whatever credentials you need to talk to that server. Main SMTP servers are locked down via authentication, IP range, etc.
4) You're not reinventing the wheel. Leverage the infrastructure you have. Are you writing an application or a mail server? Setting up mail server is an every day task that is typically simple to do. All of those casual "dumb" users on the internet have managed to get email set up.
Don't.
Sending email is much more complex than it seems. Email servers excel at (or should excel at) reliable delivery.
Set up a separate email server if you need to- that will be essentially the same as implementing one in Java (I doubt you will find libraries for this task- they would be essentially complete mail servers), but much more simpler.