Develop Reference

How to extract body from a text file containing an e-mail [Enron Data Set]

I have enron e-mail data set as a folder, which contains e-mails in the form of text files, and I want to extract "body" part of those e-mails
The problem is, fields like sender's email, receiver's email are specified by To:, From: etc.
But Body does not start with any heading, it just starts after all the other fields have been specified.
now, a text file can contain many bodies (in case of email threads/conversation).
I want to extract the body(ies) from these files. Can javamail api be used, if yes, then how? It is just offline data set, in the form of text files in my hard disk drive, not on internet.
The file is like this-
Message-ID: <16159836.1075855377439.JavaMail.evans#thyme>
Date: Fri, 7 Dec 2001 10:06:42 -0800 (PST)
From: heather.dunton#enron.com
To: k..allen#enron.com
Subject: RE: West Position
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-From: Dunton, Heather </O=ENRON/OU=NA/CN=RECIPIENTS/CN=HDUNTON>
X-To: Allen, Phillip K. </O=ENRON/OU=NA/CN=RECIPIENTS/CN=Pallen>
X-cc:
X-bcc:
X-Folder: \Phillip_Allen_Jan2002_1\Allen, Phillip K.\Inbox
X-Origin: Allen-P
X-FileName: pallen (Non-Privileged).pst
Please let me know if you still need Curve Shift.
Thanks,
Heather
-----Original Message-----
From: Allen, Phillip K.
Sent: Friday, December 07, 2001 5:14 AM
To: Dunton, Heather
Subject: RE: West Position
Heather,
Did you attach the file to this email?
-----Original Message-----
From: Dunton, Heather
Sent: Wednesday, December 05, 2001 1:43 PM
To: Allen, Phillip K.; Belden, Tim
Subject: FW: West Position
Attached is the Delta position for 1/16, 1/30, 6/19, 7/13, 9/21
-----Original Message-----
From: Allen, Phillip K.
Sent: Wednesday, December 05, 2001 6:41 AM
To: Dunton, Heather
Subject: RE: West Position
Heather,
This is exactly what we need. Would it possible to add the prior day for each of the dates below to the pivot table. In order to validate the curve shift on the dates below we also need the prior days ending positions.
Thank you,
Phillip Allen
-----Original Message-----
From: Dunton, Heather
Sent: Tuesday, December 04, 2001 3:12 PM
To: Belden, Tim; Allen, Phillip K.
Cc: Driscoll, Michael M.
Subject: West Position
Attached is the Delta position for 1/18, 1/31, 6/20, 7/16, 9/24
<< File: west_delta_pos.xls >>
Let me know if you have any questions.
Heather

Please provide an example file, the most complex one if possible.
The job would be to programmatically open every file, parse its content, and extract email's bodies.
Where do you want to store it then ?
Which OS are you running ?

If each file is a single message in MIME format, you can use the JavaMail MimeMessage constructor that takes an InputStream. You can then use the JavaMail APIs to extract the contents of the message. See the JavaMail FAQ, javadocs, web site, specification, etc.

Generating HttpResponse

When creating the HTTP Response manually, how can one get Server and ETag
* HTTP/1.1 200 OK
* Date: Mon, 23 Apr 2012 23:44:52 GMT
* Server: Apache/2.2.3 (Red Hat) <-----
* Last-Modified: Fri, 16 Sep 2005 18:08:50 GMT
* ETag: "421142-2f-400e77c517080" <-----
* Accept-Ranges: bytes
* Content-Length: 47
* Content-Type: text/plain
* Connection: close

"Server" is whatever your HTTP server wants to name/identify itself. I.e. "Zumgto Surver 4.5".
"ETag" identifies "version" of particular item, so as long as your server can reasonable say "this ETag corresponds to current version" you can send pretty much anything. I.e. "v3345", or hash of the item... Totally optional if you don't support "If-None-Match" header in requests.

Neither is required. You can make up your own sever tag using the same format above. Omit the eTag or just generate your own. You could use the current timestamp or a constant. The following formats should work.
Server: Program/version (O/S)
ETag: "Timestamp"

How to control alignment of mail header?

I encounter a situation, when I send a email which email subject more than 80 char, sometimes the email subject will be insert a 'tab' char. I check the invalid email's mail header, they are aligned, if a mail header entry more than 80 char, it will be wrap to a new line, and insert a 'tab'. I don't know which configuration control it, javaMail setting? SMTP setting?
Sample headers:
case one: invaild email subject, inserted a char 'tab'.
Received: from as09.aaaaaaaaa.com ([192.168.1.129]) by aaaaaaaaa.com
([192.168.1.49]) with ESMTP (TREND IMSS SMTP Service 7.0; TLS:
TLSv1/SSLv3,168bits,DES-CBC3-SHA) id ba8afe04001a23ba ; Wed, 1 Feb 2012
16:17:10 +0800
Received: from as09 (localhost [127.0.0.1])by as09.aaaaaaaaa.com
(8.13.8+Sun/8.13.8) with ESMTP id q118HAgi021891;Wed, 1 Feb 2012 16:17:10
+0800 (HKT)
Message-ID: <25866052.1328084230581.JavaMail.oracle#as09>
Date: Wed, 1 Feb 2012 08:17:10 +0000
From: <sss#abc.com>
To: ben#abc.com,
john#abc.com,
jimmy#abc.com
Subject: test Daily report for city pairs of iPhone App enquiries indocument
version Demo Customer/Test/Test/123/1231
CC: sam#abc.com
MIME-Version: 1.0
case two: valid email subject, just wrapped
Received: from as08.aaaaaaaaa.com ([192.168.1.104]) by aaaaaaaaa.com
([192.168.1.49]) with ESMTP (TREND IMSS SMTP Service 7.0; TLS:
TLSv1/SSLv3,168bits,DES-CBC3-SHA) id c356e969001b37a5 ; Fri, 3 Feb 2012
09:16:58 +0800
Received: from as08.aaaaaaaaa.com (localhost [127.0.0.1]) by
as08.aaaaaaaaa.com (8.13.4+Sun/8.13.4) with ESMTP id q131Gw1l006160; Fri, 3
Feb 2012 09:16:58 +0800 (HKT)
Message-ID: <17351450.1328231818282.JavaMail.oracle#as08.aaaaaaaaa.com>
Date: Fri, 3 Feb 2012 01:16:58 +0000
From: <sss#abc.com>
To: ben#abc.com
Subject: test Daily report for city pairs of iPhone App enquiries in
document version Demo Customer/Test/Test/123/1231
CC: sam#abc.com
sample java code:
public void sendEmail(Request req) throws Exception {
Session mailSession;
MimeMessage msg;
Properties mailSessionProp = new Properties();
mailSessionProp.put("mail.smtp.host", "localhost");
mailSessionProp.put("mail.smtp.port", "25");
mailSession = Session.getInstance(mailSessionProp, null);
msg = new MimeMessage(mailSession);
msg.addRecipients("TO", InternetAddress.parse(req.getRecipientAddress()));
msg.setSubject(MimeUtility.encodeText(req.getSubject(), "UTF-8", "B"));
msg.setHeader("X-Mailer", "");
msg.setSentDate(new Date());
msg.setContent(req.getMessage(), req.getMessageContentType().getType());
Transport.send(msg);
}
JavaMail:
Manifest-Version: 1.0
Implementation-Version: 1.3.3_01
Specification-Title: JavaMail(TM) API Design Specification
Specification-Version: 1.3
Extension-Name: javax.mail
Created-By: 1.3.1 (Sun Microsystems Inc.)
Implementation-Vendor-Id: com.sun
Implementation-Vendor: Sun Microsystems, Inc.
Specification-Vendor: Sun Microsystems, Inc.
SCCS-ID: #(#)javamail.mf 1.5 02/03/14

This behaviour is correct according to http://www.faqs.org/rfcs/rfc2822.html
2.2.3. Long Header Fields
Each header field is logically a single line of characters comprising the field name, the colon, and the field body. For convenience however, and to deal with the 998/78 character limitations per line, the field body portion of a header field can be split into a multiple
line representation; this is called "folding". The general rule is that wherever this standard allows for folding white space (not simply WSP characters), a CRLF may be inserted before any WSP.

JavaMail will wrap the header when you call setSubject(), and unwrap it when you call getSubject().

Java POP3 message parser without JavaMail API

Is there any available email parser library or implementation tutorial (without JavaMail API) to parse messages of the fallowing format to get their fields such as FROM / SUBJECT /etc ?
I want to parse a text file of these messages received via the POP3 protocol in my application and finally display them in a html file.
a sample message is :
-------------------- messsage 1, size=4234 --------------------
Return-path: <yyy#gmail.com>
Received: from mail-iy0-f171.google.com (mail-iy0-f171.google.com [209.85.210.171])
by xxxmail.com (xxxmail.com)
(MDaemon PRO v11.0.0)
with ESMTP id md50000000053.msg
for <xx#xxxmail.com>; Mon, 12 Dec 2011 22:55:39 +0330
Authentication-Results: xxxmail.com
spf=pass smtp.mail=haghighi.fa#gmail.com;
x-ip-ptr=pass dns.ptr=mail-iy0-f171.google.com (ip=x.x.x.x);
x-ip-helo=pass smtp.helo=mail-iy0-f171.google.com (ip=x.x.x.x);
x-ip-mail=hardfail smtp.mail=haghighi.fa#gmail.com (does not match x.x.x.x);
x-vbr=hardfail header.vbr-info=gmail.com (domain not recognized);
dkim=pass header.d=gmail.com (b=Cv+42gRZMW; 1:0:good);
Received-SPF: pass (xxxmail.com: domain of yyy#gmail.com
designates x.x.x.x as permitted sender)
x-spf-client=MDaemon.PRO.v11.0.0
receiver=xxxmail.com
client-ip=x.x.x.x
envelope-from=<yyy#gmail.com>
helo=mail-iy0-f171.google.com
X-Spam-Processed: xxxmail.com, Mon, 12 Dec 2011 22:55:39 +0330
(not processed: sender in recipient's private address book)
X-MDPtrLookup-Result: pass dns.ptr=mail-iy0-f171.google.com (ip=x.x.x.x) (xxxmail.com)
X-MDHeloLookup-Result: pass smtp.helo=mail-iy0-f171.google.com (ip=x.x.x.x) (xxxmail.com)
X-MDMailLookup-Result: hardfail smtp.mail=yyy#gmail.com (does not match x.x.x.x) (xxxmail.com)
X-MDDKIM-Result: unapproved (xxxmail.com)
X-MDVBR-Result: not certified (xxxmail.com)
X-MDSPF-Result: unapproved (xxxmail.com)
X-Rcpt-To: xxx#xxxmail.com
X-MDRcpt-To: xxx#xxxmail.com
X-MDRemoteIP: x.x.x.x
X-Return-Path: yyy#gmail.com
X-Envelope-From: yyya#gmail.com
X-MDaemon-Deliver-To: xxx#xxxmail.com
Received: by iaen33 with SMTP id n33so35124825iae.30
for <xxx#xxxmail.com>; Mon, 12 Dec 2011 11:28:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=gamma;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=SZ4ok7NaD3MuwvPJVzJAtsCaqSMeAJJ6Kq3AFI1elK8=;
b=Cv+42gRZMW8gRSzCX11ToY/EnHTqzO2E5cWkJsnwj6JCapz5GXG2iIadjBkHyKuGkE
i5cjPQyZXhHBg9ZfKRaWViSMqiaySvak7WA+yVf65JB2zEHykysFHbqbfPLG2CaRxXHi
PVfJVURd5MHCLpSyCxeW25slIBOBqpWabuWj4=
MIME-Version: 1.0
Received: by 10.42.151.68 with SMTP id d4mr13014719icw.36.1323718132412; Mon,
12 Dec 2011 11:28:52 -0800 (PST)
Received: by 10.43.49.67 with HTTP; Mon, 12 Dec 2011 11:28:52 -0800 (PST)
In-Reply-To: <4ee65516.85310e0a.43bf.238fSMTPIN_ADDED#mx.google.com>
References: <4ee65516.85310e0a.43bf.238fSMTPIN_ADDED#mx.google.com>
Date: Mon, 12 Dec 2011 11:28:52 -0800
Message-ID: <CAPGX6DbXDV-wFmbMcaw1rQVvrZGbBOnqPpJSNkwuhbrBFa=TRQ#mail.gmail.com>
Subject: Re: Hi :D
From: y y <yyy#gmail.com>
To: xxx#xxxmail.com
Content-Type: multipart/alternative; boundary=90e6ba6e872e10b18504b3ea252a
X-Antivirus: avast! (VPS 111228-0, 12/28/2011), Inbound message
X-Antivirus-Status: Clean
--90e6ba6e872e10b18504b3ea252a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
=D8=A7=D9=84=D9=84=D9=87 =D9=88=D8=A7=D9=82=D8=B9=D9=86 =D8=A7=DA=A9=D8=A8=
=D8=B1=D9=87 :=D8=AF=DB=8C
On Mon, Dec 12, 2011 at 12:00 AM, <xxx#xxxmail.com> wrote:
> Hi! this is the main text of the message :)
>
>
>

I've found these examples for both POP3 and SMTP:
http://inetjava.sourceforge.net/lectures/part1_sockets/InetJava-1.8-Email-Examples.html
They don't rely on Java Mail, they use just a plain socket. Look at getMessageHeaders() to obtain the information about the fields FROM and SUBJECT.

Javamail Pop3 Session timeout?

I'm trying to use javamail to download a mailbox of 1000 messages of hotmail.
The problem is after an hour or so, I get a Pop Session timeout exception, and I can't fetch messages anymore.
C: TOP 3210 0
S: +OK 1444 byte(s)
X-Message-Delivery: Vj0zLjQuMDt1cz0wO2k9MDtsPTA7YT0x
X-Message-Status: n:0
X-SID-PRA: Super Wall <apps+ocdlfcez#facebookmail.com>
X-SID-Result: Pass
X-Message-Info: R00BdL5giqoqgO8FeGWl8Lch6n3is6BT1wNitKPj0Jb+fghk1p9MsC+MFGyB2nflerotq/xZ5r8LiguM+3GjEOSj3umkoXeU
Received: from mx-out.facebook.com ([204.15.20.140]) by bay0-mc7-f15.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668);
Tue, 8 Apr 2008 15:14:55 -0700
Received: from api.facebook.com (intlb01-mip1.sctm.tfbnw.net [10.1.240.6])
by mx-out.facebook.com [email018.sctm.facebook.com] (8.13.6/8.13.6) with ESMTP id m38MEtOg030239
for <xgameprogrammer#hotmail.com>; Tue, 8 Apr 2008 15:14:55 -0700
X-Facebook: from zuckmail ([168.143.164.188])
by api.facebook.com with HTTP (ZuckMail);
Date: Tue, 8 Apr 2008 15:14:55 -0700
To: Ahmed Saleh <xgameprogrammer#hotmail.com>
From: Super Wall <apps+ocdlfcez#facebookmail.com>
Reply-to: Facebook <apps+ocdlfcez#facebookmail.com>
Subject: You just received a new photo from Lejla Boric
Message-ID: <03df95f3306af0a88432e7fcca22f7ac#api.facebook.com>
X-Priority: 3
X-Mailer: ZuckMail [version 1.00]
X-Facebook-Notify: platform_email
Errors-To: apps+ocdlfcez#facebookmail.com
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="b1_03df95f3306af0a88432e7fcca22f7ac"
Return-Path: apps+ocdlfcez#facebookmail.com
X-OriginalArrivalTime: 08 Apr 2008 22:14:55.0926 (UTC) FILETIME=[F9D6A560:01C899C5]
.
C: RETR 3210
S: -ERR POP3 session timed out
javax.mail.MessagingException: No inputstream from datasource;
SentDate : Wed Apr 09 01:14:55 AST 2008
nested exception is:

As I said in my comment, this is just a simple case of the session timing out. Like anything, the mail server will have a time limit to prevent excessive access, which is how would describe downloading emails for over an hour!
There is nothing wrong with your code, your just asking too much of hotmail / the mail server!
Though, just some friendly advice: the next time you ask a question here on SO, please don't post irrelevant information like you have done here. In cases like yours, it is helpful to see atleast a snippet of the code that is responsible for that part of the application that is functioning incorrectly and any exceptions - we'd also need to see the line(s) of code mentioned in the stack trace... However, a start would be to actually ask a question instead of this stating a fact, like you have done here!