Develop Reference

How to extract body from a text file containing an e-mail [Enron Data Set]

I have enron e-mail data set as a folder, which contains e-mails in the form of text files, and I want to extract "body" part of those e-mails
The problem is, fields like sender's email, receiver's email are specified by To:, From: etc.
But Body does not start with any heading, it just starts after all the other fields have been specified.
now, a text file can contain many bodies (in case of email threads/conversation).
I want to extract the body(ies) from these files. Can javamail api be used, if yes, then how? It is just offline data set, in the form of text files in my hard disk drive, not on internet.
The file is like this-
Message-ID: <16159836.1075855377439.JavaMail.evans#thyme>
Date: Fri, 7 Dec 2001 10:06:42 -0800 (PST)
From: heather.dunton#enron.com
To: k..allen#enron.com
Subject: RE: West Position
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-From: Dunton, Heather </O=ENRON/OU=NA/CN=RECIPIENTS/CN=HDUNTON>
X-To: Allen, Phillip K. </O=ENRON/OU=NA/CN=RECIPIENTS/CN=Pallen>
X-cc:
X-bcc:
X-Folder: \Phillip_Allen_Jan2002_1\Allen, Phillip K.\Inbox
X-Origin: Allen-P
X-FileName: pallen (Non-Privileged).pst
Please let me know if you still need Curve Shift.
Thanks,
Heather
-----Original Message-----
From: Allen, Phillip K.
Sent: Friday, December 07, 2001 5:14 AM
To: Dunton, Heather
Subject: RE: West Position
Heather,
Did you attach the file to this email?
-----Original Message-----
From: Dunton, Heather
Sent: Wednesday, December 05, 2001 1:43 PM
To: Allen, Phillip K.; Belden, Tim
Subject: FW: West Position
Attached is the Delta position for 1/16, 1/30, 6/19, 7/13, 9/21
-----Original Message-----
From: Allen, Phillip K.
Sent: Wednesday, December 05, 2001 6:41 AM
To: Dunton, Heather
Subject: RE: West Position
Heather,
This is exactly what we need. Would it possible to add the prior day for each of the dates below to the pivot table. In order to validate the curve shift on the dates below we also need the prior days ending positions.
Thank you,
Phillip Allen
-----Original Message-----
From: Dunton, Heather
Sent: Tuesday, December 04, 2001 3:12 PM
To: Belden, Tim; Allen, Phillip K.
Cc: Driscoll, Michael M.
Subject: West Position
Attached is the Delta position for 1/18, 1/31, 6/20, 7/16, 9/24
<< File: west_delta_pos.xls >>
Let me know if you have any questions.
Heather

Please provide an example file, the most complex one if possible.
The job would be to programmatically open every file, parse its content, and extract email's bodies.
Where do you want to store it then ?
Which OS are you running ?

If each file is a single message in MIME format, you can use the JavaMail MimeMessage constructor that takes an InputStream. You can then use the JavaMail APIs to extract the contents of the message. See the JavaMail FAQ, javadocs, web site, specification, etc.

taking mail into text boxes

I've written a function in java to get this. On the reception of an e-mail I'm storing the contents of the mail into an external file
MIME-Version: 1.0 Received: by 10.216.237.25 with SMTP id x25mr3166819weq.30.1346350346502; Thu, 30 Aug 2012 11:12:26 -0700 (PDT)Received: by 10.216.123.197 with HTTP; Thu, 30 Aug 2012 11:12:26 -0700 (PDT)Date: Thu, 30 Aug 2012 20:12:26 +0200 Message-ID: <CAEE+wmOpUFC9Nb9wNAVKsCrFRX7GmDwhDya4brB=hvRHXY6q7Q#mail.gmail.com> Subject: ok, lets see ... From: ABCD DEFG <xyz#gmail.com> To: XYX <xyz#yahoo.com>, def#gmail.com Content-Type: multipart/alternative; boundary=000e0cd5f7142581b204c87f9ed5 X-Processed-By: mx3.mail.genotec.ch X-Spam-Checksum: a84a5db7738f685c28343edc93a6546a X-Greylist: Sender IP 209.85.212.179 not delayed X-Spam-Report: ---- Start der SpamAssassin Auswertung -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -0.1 SPF_PASS SPF: sender matches SPF record 0.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
---- Ende der SpamAssassin Auswertung X-GIC-MailScanner-SpamScore: -1.199
X-GIC-MailScanner-SpamCheck: ham, hits=-1.199 required=10 scantime="4.74 seconds" origip=209.85.212.179 X-Scanned-By: "MGate 1.0" on 82.195.224.57
--000e0cd5f7142581b204c87f9ed5 Content-Type: text/plain; charset=ISO-8859-1
hello worldli ... --000e0cd5f7142581b204c87f9ed5 Content-Type: text/html; charset=ISO-8859-1 hello worldli ... --000e0cd5f7142581b204c87f9ed5--
As we can see it not aligned .. I'd like to sort the data according to SENDER, RECEIVER, ACTUAL CONTENTS OF AN EMAIL, DATE (in short I want only SENDER, RECEIVER, DATE, ACTUAL DATA that's all).... how can I achieve this using JAVASCRIPT. Other unnecessary things should be avoided.... and have to display the fetched data in text-boxes ..
Let's say four text-boxes for four fields...

You can use regular expression to finish this task.
You can write 4 different regular expressions for each of the field.

Java POP3 message parser without JavaMail API

Is there any available email parser library or implementation tutorial (without JavaMail API) to parse messages of the fallowing format to get their fields such as FROM / SUBJECT /etc ?
I want to parse a text file of these messages received via the POP3 protocol in my application and finally display them in a html file.
a sample message is :
-------------------- messsage 1, size=4234 --------------------
Return-path: <yyy#gmail.com>
Received: from mail-iy0-f171.google.com (mail-iy0-f171.google.com [209.85.210.171])
by xxxmail.com (xxxmail.com)
(MDaemon PRO v11.0.0)
with ESMTP id md50000000053.msg
for <xx#xxxmail.com>; Mon, 12 Dec 2011 22:55:39 +0330
Authentication-Results: xxxmail.com
spf=pass smtp.mail=haghighi.fa#gmail.com;
x-ip-ptr=pass dns.ptr=mail-iy0-f171.google.com (ip=x.x.x.x);
x-ip-helo=pass smtp.helo=mail-iy0-f171.google.com (ip=x.x.x.x);
x-ip-mail=hardfail smtp.mail=haghighi.fa#gmail.com (does not match x.x.x.x);
x-vbr=hardfail header.vbr-info=gmail.com (domain not recognized);
dkim=pass header.d=gmail.com (b=Cv+42gRZMW; 1:0:good);
Received-SPF: pass (xxxmail.com: domain of yyy#gmail.com
designates x.x.x.x as permitted sender)
x-spf-client=MDaemon.PRO.v11.0.0
receiver=xxxmail.com
client-ip=x.x.x.x
envelope-from=<yyy#gmail.com>
helo=mail-iy0-f171.google.com
X-Spam-Processed: xxxmail.com, Mon, 12 Dec 2011 22:55:39 +0330
(not processed: sender in recipient's private address book)
X-MDPtrLookup-Result: pass dns.ptr=mail-iy0-f171.google.com (ip=x.x.x.x) (xxxmail.com)
X-MDHeloLookup-Result: pass smtp.helo=mail-iy0-f171.google.com (ip=x.x.x.x) (xxxmail.com)
X-MDMailLookup-Result: hardfail smtp.mail=yyy#gmail.com (does not match x.x.x.x) (xxxmail.com)
X-MDDKIM-Result: unapproved (xxxmail.com)
X-MDVBR-Result: not certified (xxxmail.com)
X-MDSPF-Result: unapproved (xxxmail.com)
X-Rcpt-To: xxx#xxxmail.com
X-MDRcpt-To: xxx#xxxmail.com
X-MDRemoteIP: x.x.x.x
X-Return-Path: yyy#gmail.com
X-Envelope-From: yyya#gmail.com
X-MDaemon-Deliver-To: xxx#xxxmail.com
Received: by iaen33 with SMTP id n33so35124825iae.30
for <xxx#xxxmail.com>; Mon, 12 Dec 2011 11:28:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=gamma;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=SZ4ok7NaD3MuwvPJVzJAtsCaqSMeAJJ6Kq3AFI1elK8=;
b=Cv+42gRZMW8gRSzCX11ToY/EnHTqzO2E5cWkJsnwj6JCapz5GXG2iIadjBkHyKuGkE
i5cjPQyZXhHBg9ZfKRaWViSMqiaySvak7WA+yVf65JB2zEHykysFHbqbfPLG2CaRxXHi
PVfJVURd5MHCLpSyCxeW25slIBOBqpWabuWj4=
MIME-Version: 1.0
Received: by 10.42.151.68 with SMTP id d4mr13014719icw.36.1323718132412; Mon,
12 Dec 2011 11:28:52 -0800 (PST)
Received: by 10.43.49.67 with HTTP; Mon, 12 Dec 2011 11:28:52 -0800 (PST)
In-Reply-To: <4ee65516.85310e0a.43bf.238fSMTPIN_ADDED#mx.google.com>
References: <4ee65516.85310e0a.43bf.238fSMTPIN_ADDED#mx.google.com>
Date: Mon, 12 Dec 2011 11:28:52 -0800
Message-ID: <CAPGX6DbXDV-wFmbMcaw1rQVvrZGbBOnqPpJSNkwuhbrBFa=TRQ#mail.gmail.com>
Subject: Re: Hi :D
From: y y <yyy#gmail.com>
To: xxx#xxxmail.com
Content-Type: multipart/alternative; boundary=90e6ba6e872e10b18504b3ea252a
X-Antivirus: avast! (VPS 111228-0, 12/28/2011), Inbound message
X-Antivirus-Status: Clean
--90e6ba6e872e10b18504b3ea252a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
=D8=A7=D9=84=D9=84=D9=87 =D9=88=D8=A7=D9=82=D8=B9=D9=86 =D8=A7=DA=A9=D8=A8=
=D8=B1=D9=87 :=D8=AF=DB=8C
On Mon, Dec 12, 2011 at 12:00 AM, <xxx#xxxmail.com> wrote:
> Hi! this is the main text of the message :)
>
>
>

I've found these examples for both POP3 and SMTP:
http://inetjava.sourceforge.net/lectures/part1_sockets/InetJava-1.8-Email-Examples.html
They don't rely on Java Mail, they use just a plain socket. Look at getMessageHeaders() to obtain the information about the fields FROM and SUBJECT.

Javamail Pop3 Session timeout?

I'm trying to use javamail to download a mailbox of 1000 messages of hotmail.
The problem is after an hour or so, I get a Pop Session timeout exception, and I can't fetch messages anymore.
C: TOP 3210 0
S: +OK 1444 byte(s)
X-Message-Delivery: Vj0zLjQuMDt1cz0wO2k9MDtsPTA7YT0x
X-Message-Status: n:0
X-SID-PRA: Super Wall <apps+ocdlfcez#facebookmail.com>
X-SID-Result: Pass
X-Message-Info: R00BdL5giqoqgO8FeGWl8Lch6n3is6BT1wNitKPj0Jb+fghk1p9MsC+MFGyB2nflerotq/xZ5r8LiguM+3GjEOSj3umkoXeU
Received: from mx-out.facebook.com ([204.15.20.140]) by bay0-mc7-f15.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668);
Tue, 8 Apr 2008 15:14:55 -0700
Received: from api.facebook.com (intlb01-mip1.sctm.tfbnw.net [10.1.240.6])
by mx-out.facebook.com [email018.sctm.facebook.com] (8.13.6/8.13.6) with ESMTP id m38MEtOg030239
for <xgameprogrammer#hotmail.com>; Tue, 8 Apr 2008 15:14:55 -0700
X-Facebook: from zuckmail ([168.143.164.188])
by api.facebook.com with HTTP (ZuckMail);
Date: Tue, 8 Apr 2008 15:14:55 -0700
To: Ahmed Saleh <xgameprogrammer#hotmail.com>
From: Super Wall <apps+ocdlfcez#facebookmail.com>
Reply-to: Facebook <apps+ocdlfcez#facebookmail.com>
Subject: You just received a new photo from Lejla Boric
Message-ID: <03df95f3306af0a88432e7fcca22f7ac#api.facebook.com>
X-Priority: 3
X-Mailer: ZuckMail [version 1.00]
X-Facebook-Notify: platform_email
Errors-To: apps+ocdlfcez#facebookmail.com
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="b1_03df95f3306af0a88432e7fcca22f7ac"
Return-Path: apps+ocdlfcez#facebookmail.com
X-OriginalArrivalTime: 08 Apr 2008 22:14:55.0926 (UTC) FILETIME=[F9D6A560:01C899C5]
.
C: RETR 3210
S: -ERR POP3 session timed out
javax.mail.MessagingException: No inputstream from datasource;
SentDate : Wed Apr 09 01:14:55 AST 2008
nested exception is:

As I said in my comment, this is just a simple case of the session timing out. Like anything, the mail server will have a time limit to prevent excessive access, which is how would describe downloading emails for over an hour!
There is nothing wrong with your code, your just asking too much of hotmail / the mail server!
Though, just some friendly advice: the next time you ask a question here on SO, please don't post irrelevant information like you have done here. In cases like yours, it is helpful to see atleast a snippet of the code that is responsible for that part of the application that is functioning incorrectly and any exceptions - we'd also need to see the line(s) of code mentioned in the stack trace... However, a start would be to actually ask a question instead of this stating a fact, like you have done here!

Send meeting invitation mail over Google app engine

i want to send iCalendar meeting invitations. I am using Google App engine, Java.
I managed to send a mail with an iCalendar file as an attachment, but programs like Outlook do not automatically recognize it as a meeting invitation.
I figure, i have to set the content-type of the attachment to: "text/calendar; method=REQUEST", but it seems to me, GAE is not accepting this?
Update: I was wrong above. I actually found, that i have to send the mail with the iCalendar part directly in the content, not as an attachment! So my problem is, GAE seems not to accept setting the content-type of the message itself.
Has anyone succesfully send a meeting invitation iCalendar element through mail using GAE?
Update:
I understand, i have to be more specific. In fact, i want to send iMip messages. iMip messages are not multipart, their content-type is "text/calendar". And in the case of sending a meeting invitation, it would be "text/calendar;method=REQUEST". So i tried this:
Message msg = new MimeMessage(session);
msg.setContent(iCalendarAsString, "text/calendar;method=REQUEST");
Then i send the message with Transport.send(..);
In the logs in the development server of GAE i then see, that the content type is "text/plain".
that's why i said, i think, GAE is not accepting setting a different content type.
Or am i wrong?
Update 2
OK, here's the complete code:
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
try {
Message msg = new MimeMessage(session);
msg.setFrom(new InternetAddress("ical#someapp.appspotmail.com", "SomeApp"));
msg.addRecipient(Message.RecipientType.TO,
new InternetAddress("me#mydomain.de", "Sven Busse"));
msg.setSubject("meeting invitation!");
msg.setContent(iCalendarAsString, "text/calendar;method=REQUEST");
Transport.send(msg);
} catch (AddressException e) {
log.warning(e.toString());
e.printStackTrace();
} catch (MessagingException e) {
log.warning(e.toString());
e.printStackTrace();
} catch (UnsupportedEncodingException e) {
log.warning(e.toString());
e.printStackTrace();
}
So, i set the content-type to "text/calendar;method=REQUEST". And once i send, this is the logs:
14.09.2011 09:52:59 com.google.appengine.api.mail.dev.LocalMailService log
INFO: MailService.send
INFO: From: SomeApp <ical#someapp.appspotmail.com>
INFO: To: Sven Busse <me#mydomain.de>
INFO: Reply-to: SomeApp <ical#someapp.appspotmail.com>
INFO: Subject: meeting invitation!
INFO: Body:
INFO: Content-type: text/plain
INFO: Data length: 458
So, the logs don't look pretty, but you can see, the content-type has changed to "text/plain".
Update 3:
Delivered-To: me#mydomain.de
Received: by 10.68.59.7 with SMTP id v7cs112540pbq;
Thu, 15 Sep 2011 07:45:55 -0700 (PDT)
Received: by 10.52.95.44 with SMTP id dh12mr1121151vdb.20.1316097954738;
Thu, 15 Sep 2011 07:45:54 -0700 (PDT)
Return-Path: <3oQ9yTgoJCgIdqqmngvqpciockn.eqouxgp.dwuugiqqingockn.eqo#2uix4h7xygsz66weerlq.apphosting.bounces.google.com>
Received: from mail-vw0-f69.google.com (mail-vw0-f69.google.com [209.85.212.69])
by mx.google.com with ESMTPS id o9si1579035vcv.136.2011.09.15.07.45.53
(version=TLSv1/SSLv3 cipher=OTHER);
Thu, 15 Sep 2011 07:45:53 -0700 (PDT)
Received-SPF: pass (google.com: domain of 3oQ9yTgoJCgIdqqmngvqpciockn.eqouxgp.dwuugiqqingockn.eqo#2uix4h7xygsz66weerlq.apphosting.bounces.google.com designates 209.85.212.69 as permitted sender) client-ip=209.85.212.69;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of 3oQ9yTgoJCgIdqqmngvqpciockn.eqouxgp.dwuugiqqingockn.eqo#2uix4h7xygsz66weerlq.apphosting.bounces.google.com designates 209.85.212.69 as permitted sender) smtp.mail=3oQ9yTgoJCgIdqqmngvqpciockn.eqouxgp.dwuugiqqingockn.eqo#2uix4h7xygsz66weerlq.apphosting.bounces.google.com
Received: by vws20 with SMTP id 20so3831617vws.4
for <me#mydomain.de>; Thu, 15 Sep 2011 07:45:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.236.187.1 with SMTP id x1mr5481998yhm.8.1316097953249; Thu, 15
Sep 2011 07:45:53 -0700 (PDT)
Reply-To: SomeApp <ical#someapp.appspotmail.com>
X-Google-Appengine-App-Id: someapp
Message-ID: <20cf305e2551fe38a104acfbee28#google.com>
Date: Thu, 15 Sep 2011 14:45:53 +0000
Subject: meeting invitation!
From: SomeApp <ical#someapp.appspotmail.com>
To: Sven Busse <me#mydomain.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
BEGIN:VCALENDAR
PRODID:-//Ben Fortuna//iCal4j 1.0//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VEVENT
DTSTAMP:20110915T144552Z
UID:20110915T144552Z-1#someapp.appspotmail.com
SUMMARY:tolles projekt\, macht viel spass.
DTSTART;VALUE=DATE:20110919
DTEND;VALUE=DATE:20110926
DESCRIPTION:tolles projekt\, macht viel spass.
ATTENDEE;ROLE=REQ-PARTICIPANT;RSVP=TRUE:mailto:me#mydomain.de
ORGANIZER:mailto:ical#someapp.appspotmail.com
END:VEVENT
END:VCALENDAR

Maybe you should think of using a service like http://context.io/ ?
(not affiliated !)