Develop Reference

javax.mail error when parsing email address header with '[' character

I'm trying to digest some messages coming in via a gmail account and one or more messages I receive have a formatted name in the header for the sender. Something along the lines of
[hris [hristerson <chrisDoesNotExist#gmail.com>
The name above is fake, but illustrates the scenario I'm encountering, the name is replacing one or more 'C' characters with '['.
The code that triggers the error
String header = "[hris [hristerson <chrisDoesNotExist#gmail.com>";
boolean strict = true; // I also tried using a false value
InternetAddress.parse(header, strict);
The error itself manifests as: javax.mail.internet.AddressException: Missing ']'
Some research I was doing was making me think that using encoded-words per RFC-2047 might be an answer. But unless I was attempting it incorrectly, it didn't appear to work. Additionally, other similarly illegal characters don't give me the same error per my testing, which makes me think that it's incidental, and is more uniquely a java library problem.
The code that I used to attempt a rudimentary encoded-word fix was:
public String encodePrettyEmailHeader(String header, String charset, char encoding) {
Map<String, String> illegalChars = new HashMap<>();
illegalChars.put("\\[", "=5B");
illegalChars.put("]", "=5D");
illegalChars.put("\\(", "=28");
illegalChars.put("\\);", "=29");
illegalChars.put("<", "=3C");
illegalChars.put(">", "=3E");
illegalChars.put("#", "=40");
illegalChars.put(",", "=2C");
illegalChars.put(";", "=3B");
illegalChars.put(":", "=3A");
illegalChars.put("\"", "=22");
illegalChars.put("/", "=2F");
illegalChars.put("\\?", "=3F");
illegalChars.put(" ", "=20");
String email = getEmailFromHeader(header).trim(); // regex to get the <email#address.com> from the header
String name = header.substring(0, header.indexOf(email)).trim();
for (Map.Entry<String, String> anEntry : illegalChars.entrySet()) {
name = name.replaceAll(anEntry.getKey(), anEntry.getValue());
}
return String.format("=?%s?%c?%s?= %s", charset, encoding, name, email);
}
Calling it like: InternetAddress.parse(encodePrettyEmailHeader(header, "UTF-8", 'Q'), strict);
I'm hoping that there is an answer that doesn't involve manually encoding the data into quoted-printable, I believe I should be able to make this janky solution work, however, this will add more overhead since I will need to un-encode it before the data can be usefully displayed.
Does anyone have any advice for me to proceed?
Edit:
I upgraded my version and the same result happened. However, I realized that when I obfuscated the address header, I added an additional set of quotes beyond what arrives from gmail. For brevity:
This is how I incorrectly formatted it:
"[hris [hristerson" <chrisDoesNotExist#gmail.com>
This is the correct format that fails for me:
[hris [hristerson <chrisDoesNotExist#gmail.com>
Edit 2:
For clarity, I'm using group: 'com.sun.mail', name: 'javax.mail', version: '1.5.5' and I've made the upgrade to group: 'com.sun.mail', name: 'javax.mail', version: '1.6.0' with no difference in behavior.
I've also tried setting strict to false, but the exception thrown is no different.

I have two solutions for you.
1) Using constructor:
InternetAddress workingStyle2 = new InternetAddress("chrisDoesNotExist#gmail.com", "[hris [hristerson","UTF-8");
System.out.println(workingStyle2.getPersonal());
System.out.println(workingStyle2.getAddress());
2) Using strict mode = false. Note the different API when you use strict mode, call parseHeader instead of just parse.
To better support the range of "invalid" addresses seen in real messages, this method enforces fewer syntax rules than the parse method when the strict flag is false and enforces more rules when the strict flag is true. If the strict flag is false and the parse is successful in separating out an email address or addresses, the syntax of the addresses themselves is not checked.
boolean strict = false;
InternetAddress[] working = InternetAddress.parseHeader(header, strict);
System.out.println(working[0].getPersonal());
System.out.println(working[0].getAddress());

java regex matcher results != to notepad++ regex find result

I am trying to extract data out of a website access log as part of a java program. Every entry in the log has a url. I have successfully extracted the url out of each record.
Within the url, there is a parameter that I want to capture so that I can use it to query a database. Unfortunately, it doesn't seem that the web developers used any one standard to write the parameter's name.
The parameter is usually called "course_id", but I have also seen "courseId", "course%3DId", "course%253Did", etc. The format for the parameter name and value is usually course_id=_22222_1, where the number I want is between the "_" and "_1". (The value is always the same, even if the parameter name varies.)
So, my idea was to use the regex /^.*course_id[^_]*_(\d*)_1.*$/i to find and extract the number.
In java, my code is
java.util.regex.Pattern courseIDPattern = java.util.regex.Pattern.compile(".*course[^i]*id[^_]*_(\\d*)_1.*", java.util.regex.Pattern.CASE_INSENSITIVE);
java.util.regex.Matcher courseIDMatcher = courseIDPattern.matcher(_url);
_courseID = "";
if(courseIDMatcher.matches())
{
_courseID = retrieveCourseID(courseIDMatcher.group(1));
return;
}
This works for a lot of the records. However, some records do not record the course_id, even though the parameter is in the url. One such example is the record:
/webapps/contentDetail?course_id=_223629_1&content_id=_3641164_1&rich_content_level=RICH&language=en_US&v=1&ver=4.1.2
However, I used notepad++ to do a regex replace on this (in fact, every) url using the regex above, and the url was successfully replaced by the course ID, implying that the regex is not incorrect.
Am I doing something wrong in the java code, or is the java matcher broken?

Java decoding encoded LDAP filters before transmitting? Preventing LDAP injections

I am currently properly escaping my filters, either using Spring LDAP Filter clases, or by going through LdapEncoder.filterEncode().
At the same time, I am using WireShark to capture packets being exchanged between my local machine and the LDAP server.
And I seem to have a problem. Even if I properly escape values (which I have confirmed through debugging), they come out unescaped through the network. I have also confirmed (through debugging) that the value stays encoded all the way until it enters javax.naming.InitialContext.
Here is an example (note that I am using Spring LDAP 1.3.0, and that these happen on both Oracle JDK 6u45 and Oracle JDK 7u45).
In my own code, on the service layer, the call being made is:
String lMailAddress = (String) ldapTemplate.searchForObject("", new EqualsFilter(ldapUserSearchFilterAttribute, principal).encode(), new ContextMapper() {
#Override
public Object mapFromContext(Object ctx) {
DirContextAdapter lContext = (DirContextAdapter) ctx;
return lContext.getStringAttribute("mail");
}});
At this point, I can confirm that the String returned by the encode() method on the filter is "(sAMAccountName=boi\2a)"
The last point I can debug the code is the following one (starts at line 229 of org.springframework.ldap.core.LdapTemplate):
SearchExecutor se = new SearchExecutor() {
public NamingEnumeration executeSearch(DirContext ctx) throws javax.naming.NamingException {
return ctx.search(base, filter, controls);
}
};
When executeSearch() is later invoked, I can also verify that the filter String contains "(sAMAccountName=boi\2a)".
I cannot debug any further, since I do not have the source code to javax,naming.* or com.sun.jndi.ldap.* (since com.sun.jndi.ldap.LdapCtx is being invoked).
However, as soon as the call returns from executeSearch(), WireShark informs me that an LDAP packet containing a searchRequest with the filter "(sAMAccountName=boi*)" has been transmitted (the * is no longer escaped).
I have used similar encoding and used different methods of LdapTemplate that yielded the result I was expecting (I saw the encoded filter being transmitted in WireShark), but I cannot explain why, in the case I just exposed, the value gets decoded before being transmitted.
Please help me understanding the situation. Hpoefully, I am the one who does not properly understand the LDAP protocol here.
Thanks.
Disclaimer: I have posted the same question to Spring LDAP forums.
TL/DR: Why is com.sun.jndi.ldap.LdapCtx decoding LDAP encoded filters (like \2a to *) before transmitting them to the LDAP server?
Update: Tried and observed the same behavior with IBM's J9 JDK7.

Although I'm not familiar with Spring LDAP, it doesn't sound like there's necessarily a reason to be concerned. LDAP filters aren't transmitted as clear text, but rather in a binary encoding, and there is no need for escaping in this mechanism (nor would it be correct to do so).
Let's take "(sAMAccountName=boi*)" as an example. As written, this filter is a substring filter with a subInitial component of "boi". As you point out, if you want it to be an equality filter rather than a substring filter, then the string representation would have to be "(sAMAccountName=boi\2a)". However, the binary encodings for these filters don't use any escaping, but instead use an ASN.1 BER type to differentiate between substring and equality filters.
If you want "(sAMAccountName=boi*)" as a substring filter, then the encoded representation would be:
a417040e73414d4163636f756e744e616d6530058003626f69
On the other hand, if you want "(sAMAccountName=boi\2a)" as an equality filter, the encoding would be:
a316040e73414d4163636f756e744e616d650404626f692a
The full explanation of the encoding isn't something I want to get into, but the "a4" at the beginning of the first one indicates that it's a substring filter, whereas the "a3" at the beginning of the second indicates that it's an equality filter.
You should be able to verify the actual bytes sent in WireShark. It may well be that WireShark doesn't properly escape the filter when generating the string representation, but that would be an issue with WireShark itself. The directory server only gets the binary representation, and it's hard to believe that an LDAP server would misinterpret that.

OWASP suggest to encode strings for searches:
public static final String escapeLDAPSearchFilter(String filter) {
StringBuffer sb = new StringBuffer(); // If using JDK >= 1.5 consider using StringBuilder
for (int i = 0; i < filter.length(); i++) {
char curChar = filter.charAt(i);
switch (curChar) {
case '\\':
sb.append("\\5c");
break;
case '*':
sb.append("\\2a");
break;
case '(':
sb.append("\\28");
break;
case ')':
sb.append("\\29");
break;
case '\u0000':
sb.append("\\00");
break;
default:
sb.append(curChar);
}
}
return sb.toString();
}
DN strings are escaped different. See the link below.
https://www.owasp.org/index.php/Preventing_LDAP_Injection_in_Java

The best way is to use parameterized filter search method, thus the parameter will be properly encoded.
See https://docs.oracle.com/javase/jndi/tutorial/ldap/search/search.html
// Perform the search
NamingEnumeration answer = ctx.search("ou=NewHires",
"(&(mySpecialKey={0}) (cn=*{1}))", // Filter expression
new Object[]{key, name}, // Filter arguments
null); // Default search controls

regular expression for email validation in Java

I am using the follwoing regular expression
(".+#.+\\.[a-z]+")
Bit it accepts ###.com as a valid email. What's the pattern I should use?

You should use apache-commons email validator. You can get the jar file from here.
Here is a simple example of how to use it:
import org.apache.commons.validator.routines.EmailValidator;
boolean isValidEmail = EmailValidator.getInstance().isValid(emailAddress);

Here's a web page that explains that better than I can: http://www.regular-expressions.info/email.html (EDIT: that appears to be a bit out of date since it refers to RFC 2822, which has been superseded by RFC 5322)
And another with an interesting take on the problem of validation: http://www.markussipila.info/pub/emailvalidator.php
Generally the best strategy for validating an email address is to just try sending mail to it.

If somebody wants to enter non-existent email address he'll do it whatever format validation you choose.
The only way to check that user owns email he entered is to send confirmation (or activation) link to that address and ask user to click it.
So don't try to make life of your users harder. Checking for presence of # is good enough.

[A-Z0-9._%+-]+#[A-Z0-9.-]+.[A-Z]{2,4}

I usually use the following one:
([a-zA-Z0-9]+(?:[._+-][a-zA-Z0-9]+)*)#([a-zA-Z0-9]+(?:[.-][a-zA-Z0-9]+)*[.][a-zA-Z]{2,})

import java.util.regex.*;
class ValidateEmailPhone{
public static void main(String args[]){
//phone no validation starts with 9 and of 10 digit
System.out.println(Pattern.matches("[9]{1}[0-9]{9}", "9999999999"));
//email validation
System.out.println(Pattern.matches("[a-zA-Z0-9]{1,}[#]{1}[a-z]{5,}[.]{1}+[a-z]{3}", "abcd#gmail.com"));
}
}

This is my regex for email validation:
(([a-zA-Z0-9]+)([\.\-_]?)([a-zA-Z0-9]+)([\.\-_]?)([a-zA-Z0-9]+)?)(#)([a-zA-Z]+.[A-Za-z]+\.?([a-zA-Z0-9]+)\.?([a-zA-Z0-9]+))
For username it allows ".", "_", "-" for separators.
After "#" allows only "." and "-".
Can be easy modified for more words.

How do I open the default mail program with a Subject and Body in a cross-platform way?

How do I open the default mail program with a Subject and Body in a cross-platform way?
Unfortunately, this is for a a client app written in Java, not a website.
I would like this to work in a cross-platform way (which means Windows and Mac, sorry Linux). I am happy to execute a VBScript in Windows, or AppleScript in OS X. But I have no idea what those scripts should contain. I would love to execute the user's default program vs. just searching for Outlook or whatever.
In OS X, I have tried executing the command:
open mailto:?subject=MySubject&body=TheBody
URL escaping is needed to replace spaces with %20.
Updated On Windows, you have to play all sorts of games to get start to run correctly. Here is the proper Java incantation:
class Win32 extends OS {
public void email(String subject, String body) throws Exception {
String cmd = "cmd.exe /c start \"\" \"" + formatMailto(subject, body) + "\"";
Runtime.getRuntime().exec(cmd);
}
}

In Java 1.6 you have a stardard way to open the default mailer of the platform:
the Desktop.mail(URI) method.The URI can be used to set all the fields of the mail (sender, recipients, body, subject).
You can check a full example of desktop integration in Java 1.6 on Using the Desktop API in Java SE 6

start works fine in Windows (see below). I would use Java's built in UrlEscape then just run a second replacement for '+' characters.
start mailto:"?subject=My%20Subject&body=The%20Body"

Never use Runtime.exec(String) on Mac OS X or any other operating system. If you do that, you'll have to figure out how to properly quote all argument strings and so on; it's a pain and very error-prone.
Instead, use Runtime.exec(String[]) which takes an array of already-separated arguments. This is much more appropriate for virtually all uses.

1. Add a Subject Line
You can prefill the subject line in the email by adding the subject preceded by '?subject=' after the email address.
So the link now becomes:
Email Us
2. Send to Multiple Recipients
Mail can be sent to additional recipients either as carbon copies (cc) or blind carbon copies (bcc).
This is done in a similar way, by placing '?cc=someoneelse#theirsite.com' after the initial address.
So the link looks like this:
Email Us
cc can simply be replaced by bcc if you wish to send blind carbon copies.
This can be very useful if you have links on pages with different subjects. You might have the email on each page go to the appropriate person in a company but with a copy of all mails sent to a central address also.
You can of course specify more than one additional recipient, just separate your list of recipients with a comma.
Email Us
Sourced from Getting More From 'mailto' which now 404s. I retrieved the content from waybackmachine.
3. Combining Code
You can combine the various bits of code above by the addition of an '&' between each.
Thus adding
me#mysite.com?subject=Hello&cc=you#yoursite.com&bcc=her#hersite.com
would send an email with the subject 'Hello' to me, you and her.
4. Write the Email
You can also prefill the body of the email with the start of a message, or write the whole message if you like! To add some thing to the body of the email it is again as simple as above - '?body=' after the email address. However formatting that email can be a little tricky. To create spaces between words you will have to use hex code - for example '%20' between each word, and to create new lines will mean adding '%0D'. Similarly symbols such as $ signs will need to be written in hex code.
If you also wish to add a subject line and send copies to multiple recipients, this can make for a very long and difficult to write bit of code.
It will send a message to three people, with the subject and the message filled in, all you need to do is add your name.
Just look at the code!
<a href="mailto:abbeyvet#outfront.net?CC=spooky#outfront.net
&BCC=thomasbrunt#outfront.net&Subject=Please%2C%20I%20insist
%21&Body=Hi%0DI%20would%20like%20to%20send%20you%20
%241000000%20to%20divide%20as%20you%20see%20fit%20among
%20yourselves%20and%20all%20the%20moderators.%0DPlease%
20let%20me%20know%20to%20whom%20I%20should%20send
%20the%20check.">this link</a>
Note: Original source URL where I found this is now 404ing so I grabbed to content from waybackmachine and posted it here so it doesn't get lost. Also, the OP stated it was not for a website, which is what these examples are, but some of these techniques may still be useful.

I had to re-implement URLencode
because Java's would use + for space
and Mail took those literally.
I don't know if Java has some built-in method for urlencoding the string, but this link http://www.permadi.com/tutorial/urlEncoding/ shows some of the most common chars to encode:
; %3B
? %3F
/ %2F
: %3A
# %23
& %24
= %3D
+ %2B
$ %26
, %2C
space %20 or +
% %25
< %3C
> %3E
~ %7E
% %25

I don't know if Java has some built-in method for urlencoding the string, but this link http://www.permadi.com/tutorial/urlEncoding/ shows some of the most common chars to encode:
For percent-encoding mailto URI hnames and hvalues, I use the rules at http://shadow2531.com/opera/testcases/mailto/modern_mailto_uri_scheme.html#encoding. Under http://shadow2531.com/opera/testcases/mailto/modern_mailto_uri_scheme.html#implementations, there's a Java example that may help.
Basically, I use:
private String encodex(final String s) {
try {
return java.net.URLEncoder.encode(s, "utf-8").replaceAll("\\+", "%20").replaceAll("\\%0A", "%0D%0A");
} catch (Throwable x) {
return s;
}
}
The string that's passed in should be a string with \r\n, and stray \r already normalized to \n.
Also note that just returning the original string on an exception like above is only safe if the mailto URI argument you're passing on the command-line is properly escaped and quoted.
On windows that means:
Quote the argument.
Escape any " inside the quotes with \.
Escape any \ that precede a " or the end of the string with \.
Also, on windows, if you're dealing with UTF-16 strings like in Java, you might want to use ShellExecuteW to "open" the mailto URI. If you don't and return s on an exception (where some hvalue isn't completely percent-encoded, you could end up narrowing some wide characters and losing information. But, not all mail clients accept unicode arguments, so ideally, you want to pass a properly percent-encoded-utf8 ascii argument with ShellExecute.
Like 'start', ShellExecute with "open" should open the mailto URI in the default client.
Not sure about other OS's.

Mailto isn't a bad route to go. But as you mentioned, you'll need to make sure it is encoded correctly.
The main problem with using mailto is with breaking lines. Use %0A for carriage returns, %20 for spaces.
Also, keep in mind that the mailto is considered the same as a URL of sorts and therefore will have the same limitations for length. See
http://support.microsoft.com/kb/208427, note the maximum URL length of 2083 characters. This is confirmed for mailto as well
in this article: http://support.microsoft.com/kb/279460/en-us. Also, some mail clients can also have a limit (I believe older versions of Outlook Express had a limit of something much smaller like 483 characters or something. If you expect to have a longer string than that then you'll need to look at alternatives.
BTW, you shouldn't have to resort to kicking out a script to do that as long as you can shell out a command from Java (I don't know if you can since I don't do Java).

You may use this...
main(string[] args){
String forUri = String.format("mailto:?subject=%s&body=%s", urlEncode(sub), urlEncode(mailBody));
Desktop.getDesktop().mail(new URI(forUri));
}
private static final String urlEncode(String str) {
try {
return URLEncoder.encode(str, "UTF-8").replace("+", "%20");
} catch (UnsupportedEncodingException e) {
throw new RuntimeException(e);
}
}
Also for formatting read A simple way of sending emails in Java: mail-to links

I have implemented this, and it works well on OS X. (Ryan's mention of the max URL length has not been codified.)
public void email(String subject, String body) throws Exception {
String cmd = "open mailto:";
cmd += "?subject=" + urlEncode(subject);
cmd += "&body=" + urlEncode(body);
Runtime.getRuntime().exec(cmd);
}
private static String urlEncode(String s) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < s.length(); i++) {
char ch = s.charAt(i);
if (Character.isLetterOrDigit(ch)) {
sb.append(ch);
}
else {
sb.append(String.format("%%%02X", (int)ch));
}
}
return sb.toString();
}
I had to re-implement URLencode because Java's would use + for space and Mail took those literally. Haven't tested on Windows yet.