Develop Reference

How To Detect An Email Address Is Real

IMPORTANT
I have been blocked by hotmail services. There is a control mechanism
called spamhaus which kicked me out. I'm stuck right now.
I am trying to detect an email address is valid and if its valid then check if this email address potentially used (I know that its not certain). For example, lets assume that there is a website with domain myimaginarydomain.com. If I run code below, I guess it won't fail because domain address is valid. But nobody can take an email address with that domain.
Is there any way to find out that email address is valid? (In this case its invalid)
I don't want to send confirmation email
Sending ping may be useful?
public class Application {
private static EmailValidator validator = EmailValidator.getInstance();
public static void main(String[] args) {
while (true) {
Scanner scn = new Scanner(System.in);
String email = scn.nextLine();
boolean isValid = validateEmail(email);
System.out.println("Syntax is : " + isValid);
if (isValid) {
String domain = email.split("#")[1];
try {
int test = doLookup(domain);
System.out.println(domain + " has " + test + " mail servers");
} catch (NamingException e) {
System.out.println(domain + " has 0 mail servers");
}
}
}
}
private static boolean validateEmail(String email) {
return validator.isValid(email);
}
static int doLookup(String hostName) throws NamingException {
Hashtable env = new Hashtable();
env.put("java.naming.factory.initial",
"com.sun.jndi.dns.DnsContextFactory");
DirContext ictx = new InitialDirContext(env);
Attributes attrs =
ictx.getAttributes(hostName, new String[]{"MX"});
Attribute attr = attrs.get("MX");
if (attr == null) return (0);
return (attr.size());
}
}

There is no failsafe way to do this in all cases, but, assuming the server uses SMTP then https://www.labnol.org/software/verify-email-address/18220/ gives quite a good tutorial on one method that may work.
The method used in the tutorial relies on OS tools, so you will need to ensure they exist before using them. a ProcessBuilder may help. Alternatively, you can open a socket directly in code and avoid using OS-dependent tools.
Essentially, you find out what the mail servers are (using nslookup), then telnet to one of the mail servers and start writing an email:
3a: Connect to the mail server:
telnet gmail-smtp-in.l.google.com 25
3b: Say hello to the other server
HELO
3c: Identify yourself with some fictitious email address
mail from:<labnol#labnol.org>
3d: Type the recipient’s email address that you are trying to verify:
rcpt to:<billgates#gmail.com>
The server response for rcpt to command will give you an idea whether an email address is valid or not. You’ll get an “OK” if the address exists else a 550 error

There really is no sensible way except trying to send a notification with a token to the address and ask the other party to confirm it, usually by visiting a web-page:
the recipients MX may be unavailable at the moment but come back online later, so you cannot rely on a lookup in real time;
just because the MX accepts the email doesn't mean that the address is valid, the message could bounce later down the pipe (think UUCP);
if this is some kind of registration service, you need to provide some confirmation step anyway as otherwise it'd become too easy to subscribe random strangers on the internet that do not want your service.

How to learn system (Windows) non-proxy hosts in Java

I'm writting a Java (1.7) application to be running on Windows. The application is accessing additional services running on the same host and other ones running in the Internet. The application can be run in two environments where in one, proxy settings must be specified (there is proxy when accessing the Internet); while in the other environment, the proxy settings must not be specified (there is no proxy).
I want the application to be simple and don't want its users bother with specification of the proxy settings on cmd-line (-Dhttp.proxyHost, etc.) - the application should learn the proxy settings from Windows system settings (IE / Tools / Internet Properties / Connections / LAN Settings).
I have written a piece of code that is supposed to learn that settings, see below. The trouble is that this piece of code does not identify localhost, 127.0.0.1 and my-computer-name (where my-computer-name is the name of my computer) as URLs where proxy should be by-passed when being accessed (yes, I do have 'Bypass proxy server for local addresses' checked). As a result, the application tries to access local services through the proxy which is wrong.
So far I've found out that one way to teach JVM not to use proxy for 'local addresses' is to list the strings (localhost, 127.0.0.1, my-computer-name) in Proxy Settings / Exceptions (Do not use proxy server for addresses beginning with). Obviously, this is not a good solution as usually no one is listing these strings there (the first check-box is enough for non-Java applications).
Second (trivial) solution would be just to count with these strings in my piece of code and do not use proxy settings for them even when JVM thinks otherwise. I don't think this is a good solution and if this is the only solution, IMHO, there is a defect in JVM.
I've found many resources in the Internet how to learn System proxy settings. But how to learn the non-proxy settings?
Thanks,
PP
public static final String HTTP_PROXY_HOST_KEY = "http.proxyHost";
public static final String HTTPS_PROXY_HOST_KEY = "https.proxyHost";
public static final String HTTP_PROXY_PORT_KEY = "http.proxyPort";
public static final String HTTPS_PROXY_PORT_KEY = "https.proxyPort";
public static final String NO_PROXY_HOSTS_KEY = "http.nonProxyHosts";
// provide list of urls which are to be accessed by this application and return proxy and non-proxy settings
private Properties getSystemProxyConfiguration(String[] urls) {
log.debug("Getting system proxy");
Properties properties = new Properties();
SortedSet<String> nonProxyHosts = new TreeSet<>();
for (String url : urls) {
URI uri;
try {
uri = new URI(url);
} catch (URISyntaxException e) {
throw new IllegalStateException(e);
}
InetSocketAddress address = getSystemProxy(uri);
if (address != null) {
if (url.toLowerCase().startsWith("https")) {
properties.put(HTTPS_PROXY_HOST_KEY, address.getHostString());
properties.put(HTTPS_PROXY_PORT_KEY, ""+address.getPort());
//todo verify that all previous URLs in this array are using the same proxy
log.debug("HTTPS proxy: " + address.getHostString() + ":" + address.getPort());
} else {
properties.put(HTTP_PROXY_HOST_KEY, address.getHostString());
properties.put(HTTP_PROXY_PORT_KEY, ""+address.getPort());
//todo verify that all previous URLs in this array are using the same proxy
log.debug("HTTP proxy: " + address.getHostString() + ":" + address.getPort());
}
} else { //todo DEFECT -> this does not find the non-proxy hosts (even though specified in IE Internet settings)
nonProxyHosts.add(uri.getHost());
}
}
if (nonProxyHosts.size() > 0) {
String nonProxyHostsString = nonProxyHosts.first();
nonProxyHosts.remove(nonProxyHostsString);
for (String nonProxyHost : nonProxyHosts) {
nonProxyHostsString = nonProxyHostsString + "|" + nonProxyHost;
}
properties.put(NO_PROXY_HOSTS_KEY, nonProxyHostsString);
log.debug("Non HTTP(S) proxy hosts: "+nonProxyHostsString);
} else {
log.debug("No non HTTP(S) proxy hosts set");
}
return properties;
}
private InetSocketAddress getSystemProxy(URI uri) {
List<Proxy> proxyList;
proxyList = ProxySelector.getDefault().select(uri);
if (proxyList != null && proxyList.size() > 0) { //todo DEFECT - this never returns DIRECT proxy for localhost, 127.0.0.1, my-computer-name strings
Proxy proxy = proxyList.get(0);
if (proxyList.size() > 1) {
log.warn("There is more " + proxy.type() + " proxies available. Use "+PROXY_PROPERTIES_FILE_NAME+" to set the right one.");
}
InetSocketAddress address = (InetSocketAddress) proxy.address();
return address;
}
return null;
}

Google Apps migration API batch doesn't send/insert emails

I'm trying to send emails using Google Apps Migration API v.1, however I fail to send them and Code does not throw any exception or error.
Here is my code:
MailItemService mailItemService = new MailItemService("exampleCo-exampleApp-1");
mailItemService.setUserCredentials("user","password");
MailItemEntry entry = new MailItemEntry();
Rfc822MediaSource mediaSource = new Rfc822MediaSource(mail.getInputStream().toString());
entry.setMediaSource(mediaSource);
entry.addMailProperty(MailItemProperty.STARRED);
entry.addMailProperty(MailItemProperty.UNREAD);
entry.addLabel(new Label("pruebas"));
MailItemFeed feed = new MailItemFeed();
for (int i = 0; i < mailItems.size(); i++) {
BatchUtils.setBatchId(mailItems.get(i), Integer.toString(i));
feed.getEntries().add(mailItems.get(i));
}
mailItemService.batch("domain", "destinationUser", feed);

Can you define what you mean by "send"? It's against the APIs ToS to use it for anything other than migration of old mail to a mailbox (e.g. performing regular new message mailbox inserts is not allowed). If you're trying to send the mailbox new mail, use SMTP and/or IMAP.
Jay

JavaMail IMAP over SSL quite slow - Bulk fetching multiple messages

I am currently trying to use JavaMail to get emails from IMAP servers (Gmail and others). Basically, my code works: I indeed can get the headers, body contents and so on. My problem is the following: when working on an IMAP server (no SSL), it basically takes 1-2ms to process a message. When I go on an IMAPS server (hence with SSL, such as Gmail) I reach around 250m/message. I ONLY measure the time when processing the messages (the connection, handshake and such are NOT taken into account).
I know that since this is SSL, the data is encrypted. However, the time for decryption should not be that important, should it?
I have tried setting a higher ServerCacheSize value, a higher connectionpoolsize, but am seriously running out of ideas. Anyone confronted with this problem? Solved it one might hope?
My fear is that the JavaMail API uses a different connection each time it fetches a mail from the IMAPS server (involving the overhead for handshake...). If so, is there a way to override this behavior?
Here is my code (although quite standard) called from the Main() class:
public static int connectTest(String SSL, String user, String pwd, String host) throws IOException,
ProtocolException,
GeneralSecurityException {
Properties props = System.getProperties();
props.setProperty("mail.store.protocol", SSL);
props.setProperty("mail.imaps.ssl.trust", host);
props.setProperty("mail.imaps.connectionpoolsize", "10");
try {
Session session = Session.getDefaultInstance(props, null);
// session.setDebug(true);
Store store = session.getStore(SSL);
store.connect(host, user, pwd);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_ONLY);
int numMess = inbox.getMessageCount();
Message[] messages = inbox.getMessages();
for (Message m : messages) {
m.getAllHeaders();
m.getContent();
}
inbox.close(false);
store.close();
return numMess;
} catch (MessagingException e) {
e.printStackTrace();
System.exit(2);
}
return 0;
}
Thanks in advance.

after a lot of work, and assistance from the people at JavaMail, the source of this "slowness" is from the FETCH behavior in the API. Indeed, as pjaol said, we return to the server each time we need info (a header, or message content) for a message.
If FetchProfile allows us to bulk fetch header information, or flags, for many messages, getting contents of multiple messages is NOT directly possible.
Luckily, we can write our own IMAP command to avoid this "limitation" (it was done this way to avoid out of memory errors: fetching every mail in memory in one command can be quite heavy).
Here is my code:
import com.sun.mail.iap.Argument;
import com.sun.mail.iap.ProtocolException;
import com.sun.mail.iap.Response;
import com.sun.mail.imap.IMAPFolder;
import com.sun.mail.imap.protocol.BODY;
import com.sun.mail.imap.protocol.FetchResponse;
import com.sun.mail.imap.protocol.IMAPProtocol;
import com.sun.mail.imap.protocol.UID;
public class CustomProtocolCommand implements IMAPFolder.ProtocolCommand {
/** Index on server of first mail to fetch **/
int start;
/** Index on server of last mail to fetch **/
int end;
public CustomProtocolCommand(int start, int end) {
this.start = start;
this.end = end;
}
#Override
public Object doCommand(IMAPProtocol protocol) throws ProtocolException {
Argument args = new Argument();
args.writeString(Integer.toString(start) + ":" + Integer.toString(end));
args.writeString("BODY[]");
Response[] r = protocol.command("FETCH", args);
Response response = r[r.length - 1];
if (response.isOK()) {
Properties props = new Properties();
props.setProperty("mail.store.protocol", "imap");
props.setProperty("mail.mime.base64.ignoreerrors", "true");
props.setProperty("mail.imap.partialfetch", "false");
props.setProperty("mail.imaps.partialfetch", "false");
Session session = Session.getInstance(props, null);
FetchResponse fetch;
BODY body;
MimeMessage mm;
ByteArrayInputStream is = null;
// last response is only result summary: not contents
for (int i = 0; i < r.length - 1; i++) {
if (r[i] instanceof IMAPResponse) {
fetch = (FetchResponse) r[i];
body = (BODY) fetch.getItem(0);
is = body.getByteArrayInputStream();
try {
mm = new MimeMessage(session, is);
Contents.getContents(mm, i);
} catch (MessagingException e) {
e.printStackTrace();
}
}
}
}
// dispatch remaining untagged responses
protocol.notifyResponseHandlers(r);
protocol.handleResult(response);
return "" + (r.length - 1);
}
}
the getContents(MimeMessage mm, int i) function is a classic function that recursively prints the contents of the message to a file (many examples available on the net).
To avoid out of memory errors, I simply set a maxDocs and maxSize limit (this has been done arbitrarily and can probably be improved!) used as follows:
public int efficientGetContents(IMAPFolder inbox, Message[] messages)
throws MessagingException {
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.FLAGS);
fp.add(FetchProfile.Item.ENVELOPE);
inbox.fetch(messages, fp);
int index = 0;
int nbMessages = messages.length;
final int maxDoc = 5000;
final long maxSize = 100000000; // 100Mo
// Message numbers limit to fetch
int start;
int end;
while (index < nbMessages) {
start = messages[index].getMessageNumber();
int docs = 0;
int totalSize = 0;
boolean noskip = true; // There are no jumps in the message numbers
// list
boolean notend = true;
// Until we reach one of the limits
while (docs < maxDoc && totalSize < maxSize && noskip && notend) {
docs++;
totalSize += messages[index].getSize();
index++;
if (notend = (index < nbMessages)) {
noskip = (messages[index - 1].getMessageNumber() + 1 == messages[index]
.getMessageNumber());
}
}
end = messages[index - 1].getMessageNumber();
inbox.doCommand(new CustomProtocolCommand(start, end));
System.out.println("Fetching contents for " + start + ":" + end);
System.out.println("Size fetched = " + (totalSize / 1000000)
+ " Mo");
}
return nbMessages;
}
Do not that here I am using message numbers, which is unstable (these change if messages are erased from the server). A better method would be to use UIDs! Then you would change the command from FETCH to UID FETCH.
Hope this helps out!

You need to add a FetchProfile to the inbox before you iterate through the messages.
Message is a lazy loading object, it will return to the server for each message and for each
field that doesn't get provided with the default profile.
e.g.
for (Message message: messages) {
message.getSubject(); //-> goes to the imap server to fetch the subject line
}
If you want to display like an inbox listing of say just From, Subject, Sent, Attachement etc.. you would use something like the following
inbox.open(Folder.READ_ONLY);
Message[] messages = inbox.getMessages(start + 1, total);
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add(FetchProfileItem.FLAGS);
fp.add(FetchProfileItem.CONTENT_INFO);
fp.add("X-mailer");
inbox.fetch(messages, fp); // Load the profile of the messages in 1 fetch.
for (Message message: messages) {
message.getSubject(); //Subject is already local, no additional fetch required
}
Hope that helps.

The total time includes the time required in cryptographic operations. The cryptographic operations need a random seeder. There are different random seeding implementations which provide random bits for use in the cryptography. By default, Java uses /dev/urandom and this is specified in your java.security as below:
securerandom.source=file:/dev/urandom
On Windows, java uses Microsoft CryptoAPI seed functionality which usually has no problems. However, on unix and linux, Java, by default uses /dev/random for random seeding. And read operations on /dev/random sometimes block and takes long time to complete. If you are using the *nix platforms then the time spent in this would get counted in the overall time.
Since, I dont know what platform you are using, I can't for sure say that this could be your problem. But if you are, then this could be one of reasons why your operations are taking long time. One of the solution to this could be to use /dev/urandom instead of /dev/random as your random seeder, which does not block. This can be specified with the system property "java.security.egd". For example,
-Djava.security.egd=file:/dev/urandom
Specifying this system property will override the securerandom.source setting in your java.security file. You can give it a try. Hope it helps.

Someone knows a mail (SMTP) delivery library for Java?

I'd like to send mail without bothering with the SMTP-Server which is used for delivery.
So JavaMail API doesn't work for me because I have to specify a SMTP server to connect to.
I'd like the library to find out on its own which SMTP server is responsible for which email address by querying the MX record of the mail address domain.
I'm looking for something like Aspirin. Unfortunately I can't use Aspirin itself because the development stopped 2004 and the library fails to communicate with modern spam hardened servers correctly.
An embeddable version of James would do the task. But I haven't found documentation concerning whether this is possible.
Or does anyone know about other libraries I could use?

One possible solution: get the MX record on your own and use JavaMail API.
You can get the MX record using the dnsjava project:
Maven2 dependency:
<dependency>
<groupId>dnsjava</groupId>
<artifactId>dnsjava</artifactId>
<version>2.0.1</version>
</dependency>
Method for MX record retrieval:
public static String getMXRecordsForEmailAddress(String eMailAddress) {
String returnValue = null;
try {
String hostName = getHostNameFromEmailAddress(eMailAddress);
Record[] records = new Lookup(hostName, Type.MX).run();
if (records == null) { throw new RuntimeException("No MX records found for domain " + hostName + "."); }
if (log.isTraceEnabled()) {
// log found entries for debugging purposes
for (int i = 0; i < records.length; i++) {
MXRecord mx = (MXRecord) records[i];
String targetString = mx.getTarget().toString();
log.trace("MX-Record for '" + hostName + "':" + targetString);
}
}
// return first entry (not the best solution)
if (records.length > 0) {
MXRecord mx = (MXRecord) records[0];
returnValue = mx.getTarget().toString();
}
} catch (TextParseException e) {
throw new RuntimeException(e);
}
if (log.isTraceEnabled()) {
log.trace("Using: " + returnValue);
}
return returnValue;
}
private static String getHostNameFromEmailAddress(String mailAddress) throws TextParseException {
String parts[] = mailAddress.split("#");
if (parts.length != 2) throw new TextParseException("Cannot parse E-Mail-Address: '" + mailAddress + "'");
return parts[1];
}
Sending mail via JavaMail code:
public static void sendMail(String toAddress, String fromAddress, String subject, String body) throws AddressException, MessagingException {
String smtpServer = getMXRecordsForEmailAddress(toAddress);
// create session
Properties props = new Properties();
props.put("mail.smtp.host", smtpServer);
Session session = Session.getDefaultInstance(props);
// create message
Message msg = new MimeMessage(session);
msg.setFrom(new InternetAddress(fromAddress));
msg.setRecipient(Message.RecipientType.TO, new InternetAddress(toAddress));
msg.setSubject(subject);
msg.setText(body);
// send message
Transport.send(msg);
}

This is completely the wrong way to handle this.
Anyone connected to the internet will have some kind of "legit" SMTP server available to them to take the submission of email -- your ISP, your office, etc.
You WANT to leverage because they do several things for you.
1) they take your message and the responsibility to handle that message. After you drop it off, it's not your problem anymore.
2) Any mail de-spamming technologies are handled by the server. Even better, when/if those technologies change (Domain keys anyone?), the server handles it, not your code.
3) You, as a client of that sending mail system, already have whatever credentials you need to talk to that server. Main SMTP servers are locked down via authentication, IP range, etc.
4) You're not reinventing the wheel. Leverage the infrastructure you have. Are you writing an application or a mail server? Setting up mail server is an every day task that is typically simple to do. All of those casual "dumb" users on the internet have managed to get email set up.

Don't.
Sending email is much more complex than it seems. Email servers excel at (or should excel at) reliable delivery.
Set up a separate email server if you need to- that will be essentially the same as implementing one in Java (I doubt you will find libraries for this task- they would be essentially complete mail servers), but much more simpler.