Are there any packages available that will efficiently send emails with large attachments (each file is capped at 10mb, but could include multiple files). If not, any suggestions on an appropriate design that wouldn't result in out of memory exceptions causing issues across applications deployed on the same server?
Files are delivered to the application server by ftp. Once transmission is complete, a web service is invoked (metadata for the transaction). Based on business rules, this service may need need to email the files.
My initial thoughts were a putting the request on a message queue (so the service can return immediately), and having a synchronized method process the request (so multiple requests at or around the same time won't blow up the heap).
updating with code
messageBodyPart = new MimeBodyPart();
FileDataSource fileDataSource =new FileDataSource("locationTo.big.file");
messageBodyPart.setDataHandler(new DataHandler(fileDataSource));
messageBodyPart.setFileName("big.file");
multipart.addBodyPart(messageBodyPart);
<rinse..repeat>
message.setContent(multipart);
Transport.send(msg);
If I attach 5 10mb attachments, 50mb won't be eaten up by the heap all at once?
Why not use an Executor, with a thread pool growing/shrinking within reason. Each task submitted is a Runnable or Callable. The Task sends via JavaMail, which DOES not take much memory if you implement your own DataSource implementations for the attachments and/or message body. (I am assuming you have have InputStream acccess to the attachments)
Adding code as sample (note this code was written many years ago, and is pretty bad for many reasons. But it shows the concept)
public static void sendMailAndThrowException(SMTPParams sparams,String subject, DataSource msgTextSource,DataSource[] fids,boolean debug) throws MessagingException {
Session session=getMailSession(sparams);
PrintStream f = null;
if (debug) {
f= getPrintStream();
}
// null is System.out by javamail api
session.setDebug(debug);
session.setDebugOut(f);
try
{
// create a message
MimeMessage msg = new MimeMessage(session);
msg.setFrom(new InternetAddress(sparams.getFrom()));
// Recipients are comma delimitted
String to_list[] = sparams.getRecipients().split(",");
InternetAddress[] address = new InternetAddress[to_list.length];
for( int i=0; i< to_list.length; i++)
{
// MJB: remove extraneous spaces, sanity check
String temp = to_list[i].trim();
if (temp.length()>0) {
address[i] = new InternetAddress(to_list[i].trim());
}
}
// Addresses are always TO, never CC or BCC in this library
msg.setRecipients(Message.RecipientType.TO, address);
if ((msg.getAllRecipients() == null) || (msg.getAllRecipients().length==0)) {
throw new MessagingException("No valid recipients");
}
// Set the subject
msg.setSubject(subject,"UTF-8");
// create the Multipart and add its parts to it
Multipart mp = new MimeMultipart();
if (msgTextSource != null) {
// create and fill the first message part
MimeBodyPart mbp1 = new MimeBodyPart();
mbp1.setDataHandler(new DataHandler(msgTextSource));
mp.addBodyPart(mbp1);
}
if( fids != null)
{
for (int i=0;i<fids.length;i++) {
// create the second message part
if (fids[i]==null) continue;
MimeBodyPart mbp2 = new MimeBodyPart();
// attach the file to the message
mbp2.setDataHandler(new DataHandler(fids[i]));
mbp2.setFileName(fids[i].getName());
mp.addBodyPart(mbp2);
}
}
// add the Multipart to the message
msg.setContent(mp);
// set the Date: header
msg.setSentDate(new java.util.Date());
// Connect to SMTP server
smtpSend(session, msg, sparams);
}
catch (MessagingException mex)
{
throw mex;
} finally {
closeDebug(f);
}
}
JavaMail allows easily for sending mails with large attachments if given enough RAM to hold the various pieces.
Any particular reason you cannot use that?
Related
In our system we have mailmessages saved and a function where we can forward these messages.
I know want to be able to add text to the top of the message when we forward it.
This has proven to be surprisingly difficult.
Some of the code.
private void addMessageStartOfMail(MimeMessage mail, String forwardMailBody) throws Exception{
Object content = mail.getContent();
if (content.getClass().isAssignableFrom(MimeMultipart.class)) {
MimeMultipart mimeMultipart = (MimeMultipart) content;
for (int i = 0; i < mimeMultipart.getCount(); i++) {
BodyPart bodyPart = mimeMultipart.getBodyPart(i);
if (bodyPart.getContentType().startsWith("text/plain")) {
String cnt = forwardMailBody;
cnt = MailUtil.toPlainText(cnt);
cnt = cnt + (String)bodyPart.getContent();
bodyPart.setContent(cnt, bodyPart.getContentType());
}
......
This works but unfortunatelly not all mails are text/plain, some are text/html and worse, some are multipart which is a total mess.
The trouble code
}else if(bodyPart.getContentType().startsWith("multipart")) {
Multipart mp = (Multipart) bodyPart.getContent();
int count = mp.getCount();
for (int j = 0; j < count; j++) {
BodyPart bp = mp.getBodyPart(j);
if (bp.getContentType().startsWith("text/html")) {
String cnt = form.getForwardMailBody();
cnt = cnt + (String)bp.getContent();
bp.setContent(cnt, bp.getContentType());
....
For some reason this turns the contenttype from html to plain which makes the original message a mess.
I feel there must be a smarter way to this.
Can I somehow add a simple bodypart to beginning of a Mimemessage or something.
Any advice?
After VGRs answer I made a new attempt.
private void addMessageStartOfMail(MimeMessage mail, String forwardMailBody) throws Exception{
Object content = mail.getContent();
if (content.getClass().isAssignableFrom(MimeMultipart.class)) {
MimeMultipart mimeMultipart = (MimeMultipart) content;
BodyPart messageBodyPart = new MimeBodyPart();
messageBodyPart.setText("Test");
mimeMultipart.addBodyPart(messageBodyPart, 0);
}
}
Obviously much cleaner but how do I add the new bodypart. I want it at top of the mail but without overwriting the original mail, which this solution did.
The JavaMail FAQ describes two ways to forward a message.
The simplest way is to create a new message and attach the original message as an attachment.
If instead you want to forward the original message "inline", you need to construct a new body for the message based on the body from the original message. The JavaMail FAQ contains code for finding the main message body. Preserving the original structure of the message with the attachments and so on is much more difficult, especially when you consider all the possible cases of multipart/related, multipart/alternative, etc.
The technique described in the comments above of inserting a new MimeBodyPart into the original message will work in some cases, but it will still be a challenge to find out where exactly to insert it, and the end result may not display as you would like.
Am using apache camel, With Polling consumer, when poll my mail is mark as read.
options : delete=false&peek=false&unseen=true
After polling , when i am processing the attachment, if any error occurs , i want to make the mail as "unread". So that i can pool again later.
public void process(Exchange exchange) throws Exception {
Map<String, DataHandler> attachments = exchange.getIn().getAttachments();
Message messageCopy = exchange.getIn().copy();
if (messageCopy.getAttachments().size() > 0) {
for (Map.Entry<String, DataHandler> entry : messageCopy.getAttachments().entrySet()) {
DataHandler dHandler = entry.getValue();
// get the file name
String filename = dHandler.getName();
// get the content and convert it to byte[]
byte[] data =
exchange.getContext().getTypeConverter().convertTo(byte[].class, dHandler.getInputStream());
log.info("Downloading attachment, file name : " + filename);
InputStream fileInputStream = new ByteArrayInputStream(data);
try {
// Processing attachments
// if any error occurs here, i want to make the mail mark as unread
} catch (Exception e) {
log.info(e.getMessage());
}
}
}
}
I noticed the option peek, by setting it to true, It will not make the mail mark as read during polling, in that case is there any option to make it mark as read after processing.
To get the result that you want you should have options
peek=true&unseen=true
The peek=true option is supposed to ensure that messages remain the exact state on the mail server as they where before polling even if there is an exception. However, currently it won't work. This is actually a bug in Camel Mail component. I've submitted a patch to https://issues.apache.org/jira/browse/CAMEL-9106 and this will probably be fixed in a future release.
As a workaround you can set mapMailMessages=false but then you will have to work with the email message content yourself. In Camel 2.15 onward you also have postProcessAction option and with that you could probably remove the SEEN flags from messages with processing errors. Still, I would recommend waiting for the fix though.
We can set the mail unread flag with the following code
public void process(Exchange exchange) throws Exception {
final Message mailMessage = exchange.getIn(MailMessage.class).getMessage();
mailMessage.setFlag(Flag.SEEN, false);
}
Hi i have scanned an application using Fortify tool, in the generated reports i got path manipulation issue in the following method.
Note: In the report it is not showing the error line no. can anyone suggest me how to resove it?
private MimeMessage prepareMessage(EmailMessage req) throws EmailProviderException {
long start=System.currentTimeMillis(),finish=0;
try {
MimeMessage message = emailSender.createMimeMessage();
// create a multipart message
MimeMessageHelper helper = new MimeMessageHelper(message, true);
// set email addresses
helper.setFrom(convertAddress(req.getFromAddress()));
helper.setTo(convertAddress(req.getToAddress()));
helper.setCc(convertAddress(req.getCcAddress()));
helper.setBcc(convertAddress(req.getBccAddress()));
// set subject and body
helper.setSubject(req.getEmailSubject());
String emailBody = req.getEmailBody();
String emailMime = req.getEmailMimeType();
MimeBodyPart messagePart = new MimeBodyPart();
DataSource bodyDataSource = new ByteArrayDataSource(emailBody, emailMime);
messagePart.setDataHandler(new DataHandler(bodyDataSource));
helper.getMimeMultipart().addBodyPart(messagePart);
// add attachments
List<EmailAttachment> lAttach = req.getEmailAttachment();
if (lAttach != null) {
for (EmailAttachment attachMnt: lAttach) {
DataSource dSource = new ByteArrayDataSource(attachMnt
.getContent(), attachMnt
.getMimeType());
helper.addAttachment(attachMnt.getFileName(), dSource);
}
}
finish=System.currentTimeMillis();
statsLogger.info(new FedExLogEntry("prepareMessage took {0}ms",new Object[]{finish-start}));
return message;
} catch (Exception e) {
// covers MessagingException, IllegalStateException, IOException, MailException
String emsg = new StringBuilder("Unable to prepare smtp message.")
.append("\n").append(req.toString()).toString();
logger.warn(emsg, e);
throw new EmailProviderException(emsg, e);
}
}
Hmm. If Fortify is having issues trying to show you the correct line where the issue exists, then it's possible that fortify ran in to a parsing error when it was scanning and rendering the results to your FPR. One thing you could try is to rescan your application under a different build-id and generate a new FPR. Beyond that, I don't know. Sorry.
Something else that I'd recommend would be to inspect your log file to see if there were any errors or warnings during translation/scan.
But after looking at your code sample, I'm thinking that Fortify is tainting the parameter req and flagging the the operation that is occurring when it's trying to add the file as an attachment. Most likely your sink is going to be at
helper.addAttachment(attachMnt.getFileName(), dSource);
You'd want to validate the file names of the attachment themselves prior to trying to save them to disk.
I have a requirement where I need to process the first line in the email message and, possibly, forward it.
But the problem happens when this message has attachments. And I need to forward them as well. I just can't find a good example of processing email messages with java.mail in a safe way that would cater for multiple message structures. Also, the forwarding example is a problem.
Can anyone point me to a good resource with some code examples?
Thank you
The code of getting the first line of the email message, forwarding I don't have working:
private String getMessgaeFirstLine(Message msg) throws IOException, MessagingException{
String result = null;
Object objRef = msg.getContent();
Multipart mp = (Multipart) objRef;
int count = mp.getCount();
for (int i = 0; i < count; i++)
{
BodyPart bp = mp.getBodyPart( i );
if (bp instanceof MimeBodyPart )
{
MimeBodyPart mbp = (MimeBodyPart) bp;
if ( mbp.isMimeType( "text/plain" )) {
result = (String) mbp.getContent();
result = result.replaceAll("(\\r|\\n)", "");
break;
}
}
}
return result;
}
The simplest way will be to forward the original message as an attachment to the new message. See the JavaMail FAQ.
I've been using javamail to retrieve mails from IMAP server (currently GMail). Javamail retrieves list of messages (only ids) in a particular folder from server very fast, but when I actually fetch message (only envelop not even contents) it takes around 1 to 2 seconds for each message. What are the techniques should be used for fast retrieval?
here is my code:
try {
IMAPStore store = null;
if(store!=null&&store.isConnected())return;
Properties props = System.getProperties();
Session sessionIMAP = Session.getInstance(props, null);
try {
store = (IMAPStore) sessionIMAP.getStore("imaps");
store.connect("imap.gmail.com",993,"username#gmail.com","password");
} catch (Exception e) {
e.printStackTrace();
}
IMAPFolder folder = (IMAPFolder) store.getFolder("INBOX");
folder.open(Folder.READ_ONLY);
System.out.println("start");
Message[] msgs = folder.getMessages(1,10);
long ftime = System.currentTimeMillis();
FetchProfile fp=new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
folder.fetch(msgs, fp);
long time = System.currentTimeMillis();
System.out.println("fetch: "+(time-ftime));
for (Message message : msgs) {
System.out.println(message.getSubject());
Address[] from = message.getFrom();
for (Address address : from) {
System.out.println(address);
}
Address[] recipients = message.getAllRecipients();
for (Address address : recipients) {
System.out.println(address);
}
}
long newTime = System.currentTimeMillis();
System.out.println("convert: "+(newTime-time));
}catch (Exception e) {
e.printStackTrace();
}
}
I believe that Gmail throttles the IMAP message reads to one every second or so. You might be able to speed it up with multiple IMAP connections.
Please set the Property mail.imap.fetchsize with the required size. the default is 16k.
In case you increase the size of this property, retrieve speed will go up.
props.put("mail.imap.fetchsize", "3000000");
Note that if you're using the "imaps" protocol to access IMAP over SSL, all the properties would be named "mail.imaps.*".
Good Luck.
Yaniv
I'm not sure if this is a Javamail issue as much as it may be a Gmail issue. I have an application that retrieves mail from a number of sources, including Gmail, and Gmail is definitely the slowest. The Javamail api is pretty straightforward, but it would be hard to make suggestions without seeing what you are currently doing.
I'm running into the same thing. After profiling, I noticed that getBody was being called every time I tried to do a message.getFrom() like you are, even though I was only accessing fields that should be covered by the Envelope flag. See https://java.net/projects/javamail/forums/forum/topics/107956-gimap-efficiency-when-only-reading-headers