Develop Reference

Typesafe ConfigFactory error with reserved characters

Hi I am trying to load configuration from a String in Java as follows:
#Test
public void testIllegalCharacter(){
String input = "prop=\\asd";
Config conf = ConfigFactory.parseString(input);
}
The code above produces the following error:
com.typesafe.config.ConfigException$Parse: String: 1: Expecting a value but got wrong token: '\' (Reserved character '\' is not allowed outside quotes) (if you intended '\' (Reserved character '\' is not allowed outside quotes) to be part of a key or string value, try enclosing the key or value in double quotes, or you may be able to rename the file .properties rather than .conf)
I understand I have an illegal character in my String. Although how do I find the full set of illegal characters?
If I (for example) convert this String into a Properties object and then parse it with ConfigFactory.parseProperties I can see the value "\\asd" in resolved as "asd". So there must be some some sanitising going on in the typesafe library, I wish I could call that sanitisation myself, but I cannot see how. Parsing to Properties is not a viable solution as the configuration could be composed by Objects or Lists too.
Has anyone have any suggestion how to solve this issue?
Alternatively can someone point out all the reserved characters set?
Many thanks

If I understand the error message correctly, you should put quotes around you special characters, e.g. like this:
"prop=\"\\asd\"";
Not sure why you's want a property definition with a backslash a ('\a') in it, but I guess I don't need to know :-)

I think I might have found the answer. I need to set the ConfigParseOptions.defaults().setSyntax(ConfigSyntax.PROPERTIES)
Which works for the test below:
#Test
public void test(){
String input = "prop=C:/MyDocuments/mydir";
Config conf = ConfigFactory.parseString(input, ConfigParseOptions.defaults().setSyntax(ConfigSyntax.PROPERTIES));
assertEquals("C:/MyDocuments/mydir", conf.getAnyRef("prop"));
}
But will not work for the test with backslashes
#Test
public void test(){
String input = "prop=C:\\MyDocuments\\mydir";
Config conf = ConfigFactory.parseString(input, ConfigParseOptions.defaults().setSyntax(ConfigSyntax.PROPERTIES));
assertEquals("C:\\MyDocuments\\mydir", conf.getAnyRef("prop"));
}
Which fails:
org.junit.ComparisonFailure:
Expected :C:\MyDocuments\mydir
Actual :C:MyDocumentsmydir
So I am not sure this is the definitive answer...

non-basic characters in java, how to handle the encoding correctly

when I am trying to call method with parameter using my Polish language f.e.
node.call("ąćęasdasdęczć")
I get these characters as input characters.
Ä?Ä?Ä?asdasdÄ?czÄ
I don't know where to set correct encoding in maven pom.xml? or in my IDE? I tried to change UTF-8 to ISO_8859-2 in my IDE setting, but it didn't work. I was searching similiar questions, but I didn't find the answer.
#Edit 1
Sample code:
public void findAndSendKeys(String vToSet , By vLocator){
WebElement element;
element = webDriverWait.until(ExpectedConditions.presenceOfElementLocated(vLocator));
element.sendKeys(vToSet);
}
By nameLoc = By.id("First_Name");
findAndSendKeys("ąćęasdasdęczć" , nameLoc );
Then in input field I got Ä?Ä?Ä?asdasdÄ?czÄ. Converting string to Basic Latin in my IDE helps, but It's not the solution that I needed.
I have also problems with fields in classes f.e. I have class in which I have to convert String to basic Latin
public class Contacts{
private static final By LOC_ADDRESS_BTN = By.xpath("//button[contains(#aria-label,'Wybór adresu')]");
// it doesn't work, I have to use basic latin and replace "ó" with "\u00f3" in my IDE
}
#Edit 2 - Changed encoding, but problem still exists
1:

Matcher.find() only find the last match in JUnit Test

i have this weird problem. I have this Java method that works fine in my program:
/*
* Extract all image urls from the html source code
*/
public void extractImageUrlFromSource(ArrayList<String> imgUrls, String html) {
Pattern pattern = Pattern.compile("\\<[ ]*[iI][mM][gG][\t\n\r\f ]+.*[sS][rR][cC][ ]*=[ ]*\".*\".*>");
Matcher matcher = pattern.matcher(html);
while (matcher.find()) {
imgUrls.add(extractImgUrlFromTag(matcher.group()));
}
}
This method works fine in my java application. But whenever I test it in JUnit test, it only adds the last url to the ArrayList
/**
* Test of extractImageUrlFromSource method, of class ImageDownloaderProc.
*/
#Test
public void testExtractImageUrlFromSource() {
System.out.println("extractImageUrlFromSource");
String html = "<html><title>fdjfakdsd</title><body><img kfjd src=\"http://image1.png\">df<img dsd src=\"http://image2.jpg\"></body><img dsd src=\"http://image3.jpg\"></html>";
ArrayList<String> imgUrls = new ArrayList<String>();
ArrayList<String> expimgUrls = new ArrayList<String>();
expimgUrls.add("http://image1.png");
expimgUrls.add("http://image2.jpg");
expimgUrls.add("http://image3.jpg");
ImageDownloaderProc instance = new ImageDownloaderProc();
instance.extractImageUrlFromSource(imgUrls, html);
imgUrls.stream().forEach((x) -> {
System.out.println(x);
});
assertArrayEquals(expimgUrls.toArray(), imgUrls.toArray());
}
Is it the JUnit that has the fault. Remember, it works fine in my application.

I think there is a problem in the regex:
"\\<[ ]*[iI][mM][gG][\t\n\r\f ]+.*[sS][rR][cC][ ]*=[ ]*\".*\".*>"
The problem (or at least one problem) us the first .*. The + and * metacharacters are greedy, which means that they will attempt to match as many characters as possible. In your unit test, I think that what is happening is that the .* is matching everything up to the last 'src' in the input string.
I suspect that the reason that this "works" in your application is that the input data is different. Specifically, I suspect that you are running your application on input files where each img element is on a different line. Why does this make a difference? Well, it turns out that by default, the . metacharacter does not match line breaks.
For what it is worth, using regexes to "parse" HTML is generally thought to be a bad idea. For a start, it is horribly fragile. People who do a lot of this kind of stuff tend to use proper HTML parsers ... like "jsoup".
Reference: RegEx match open tags except XHTML self-contained tags

I wish I could comment as I'm not sure about this, but it might be worth mentioning...
This line looks like it's extracting the URLs from the wrong array...did you mean to extract from expimgUrls instead of imgUrls?
instance.extractImageUrlFromSource(imgUrls, html);
I haven't gotten this far in my Java education so I may be incorrect...I just looked over the code and noticed it. I hope someone else who knows more can actually give you a solid answer!

ESAPI XSS prevention for user supplied url property

One of my REST APIs is expecting a property "url" which expects a URL as input from the user. I am using ESAPI to prevent from XSS attacks. The problem is that the user supplied URL is something like
http://example.com/alpha?abc=def&phil=key%3dbdj
The cannonicalize method from the ESAPI encoder throws intrusion exception here claiming that the input has mixed encoding, since it is url encoded and the piece '&phi' is treated as HTML encoded and thus the exception.
I had a similar problem with sanitizing one of my application urls where the second query parameter started with 'pa' or 'pi' and was converted to delta or pi characters by HTML decoding. Please refer to my previous Stackoverflow question here
Now since the problem is that since the entire URL is coming as input from the user, I cannot simply parse out the Query parameters and sanitize them individually, since malicious input can be created combining the two query parameters and sanitizing them individually wont work in that case.
Example: &ltscr comes is last part of first query param value and ipt&gtalert(0); or something comes as first part of the next query param control context.
Has anyone faced a similar problem? I would really like to know what solutions you guys implemented. Thanks for any pointers.
EDIT: The below answer from 'avgvstvs' does not throw the intrusion exception (Thanks!!). However, the cannonicalize method now changes the original input string. ESAPI treats &phi of the query param to be some html encoded char and replaces it to '?' char. Something like my previous question which is linked here. The difference being that was a URL of my application whereas this is user input. Is my only option maintaining a white list here?

The problem that you're facing here, is that there are different rules for encoding different parts of a URL--to memory there's 4 sections in a URL that have different encoding rules. First, understand why in Java, you need to build URLs using the UriBuilder class. The URL specification will help with nitty-gritty details.
Now since the problem is that since the entire URL is coming as input
from the user, I cannot simply parse out the Query parameters and
sanitize them individually, since malicious input can be created
combining the two query parameters and sanitizing them individually
wont work in that case.
The only real option here is java.net.URI.
Try this:
URI dirtyURI = new URI("http://example.com/alpha?abc=def&phil=key%3dbdj");
String cleanURIStr = enc.canonicalize( dirtyURI.getPath() );
The call to URI.getPath() should give you a non-percent encoded URL, and if enc.canonicalize() detects double-encoding after that stage then you really DO have a double-encoded string and should inform the caller that you will only accept single-encoded URL strings. The URI.getPath() is smart enough to use decoding rules for each part of the URL string.
If its still giving you some trouble, the API reference has other methods that will extract other parts of the URL, in the event that you need to do different things with different parts of the URL. IF you ever need to manually parse parameters on a GET request for example, you can actually just have it return the query string itself--and it will have done a decoding pass on it.
=============JUNIT Test Case============
package org.owasp.esapi;
import java.net.URI;
import java.net.URISyntaxException;
import org.junit.Test;
public class TestURLValidation {
#Test
public void test() throws URISyntaxException {
Encoder enc = ESAPI.encoder();
String input = "http://example.com/alpha?abc=def&phil=key%3dbdj";
URI dirtyURI = new URI(input);
enc.canonicalize(dirtyURI.getQuery());
}
}
=================Answer for updated question=====================
There's no way around it: Encoder.canonicalize() is intended to reduce escaped character sequences into their reduced, native-to-Java form. URLs are most likely considered a special case so they were most likely deliberately excluded from consideration. Here's the way I would handle your case--without a whitelist, and it will guarantee that you are protected by Encoder.canonicalize().
Use the code above to get a URI representation of your input.
Step 1: Canonicalize all of the URI parts except URI.getQuery()
Step 2: Use a library parser to parse the query string into a data structure. I would use httpclient-4.3.3.jar and httpcore-4.3.3.jar from commons. You'll then do something like this:
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Iterator;
import java.util.List;
import javax.ws.rs.core.UriBuilder;
import org.apache.http.client.utils.URLEncodedUtils;
import org.junit.Test;
import org.owasp.esapi.ESAPI;
import org.owasp.esapi.Encoder;
public class TestURLValidation
{
#Test
public void test() throws URISyntaxException {
Encoder enc = ESAPI.encoder();
String input = "http://example.com/alpha?abc=def&phil=key%3dbdj";
URI dirtyURI = new URI(input);
UriBuilder uriData = UriBuilder.fromUri(enc.canonicalize(dirtyURI.getScheme()));
uriData.path(enc.canonicalize(enc.canonicalize(dirtyURI.getAuthority() + dirtyURI.getPath())));
println(uriData.build().toString());
List<org.apache.http.NameValuePair> params = URLEncodedUtils.parse(dirtyURI, "UTF-8");
Iterator<org.apache.http.NameValuePair> it = params.iterator();
while(it.hasNext()) {
org.apache.http.NameValuePair nValuePair = it.next();
uriData.queryParam(enc.canonicalize(nValuePair.getName()), enc.canonicalize(nValuePair.getValue()));
}
String canonicalizedUrl = uriData.build().toString();
println(canonicalizedUrl);
}
public static void println(String s) {
System.out.println(s);
}
}
What we're really doing here is using standard libraries to parse the inputURL (thus taking all the burden off of us) and then canonicalizing the parts after we've parsed each section.
Please note that the code I've listed won't work for all url types... there are more parts to a URL than scheme/authority/path/queries. (Missing is the possibility of userInfo or port, if you need those, modify this code accordingly.)

How do I encode URI parameter values?

I want to send a URI as the value of a query/matrix parameter. Before I can append it to an existing URI, I need to encode it according to RFC 2396. For example, given the input:
http://google.com/resource?key=value1 & value2
I expect the output:
http%3a%2f%2fgoogle.com%2fresource%3fkey%3dvalue1%2520%26%2520value2
Neither java.net.URLEncoder nor java.net.URI will generate the right output. URLEncoder is meant for HTML form encoding which is not the same as RFC 2396. URI has no mechanism for encoding a single value at a time so it has no way of knowing that value1 and value2 are part of the same key.

Jersey's UriBuilder encodes URI components using application/x-www-form-urlencoded and RFC 3986 as needed. According to the Javadoc
Builder methods perform contextual encoding of characters not permitted in the corresponding URI component following the rules of the application/x-www-form-urlencoded media type for query parameters and RFC 3986 for all other components. Note that only characters not permitted in a particular component are subject to encoding so, e.g., a path supplied to one of the path methods may contain matrix parameters or multiple path segments since the separators are legal characters and will not be encoded. Percent encoded values are also recognized where allowed and will not be double encoded.

You could also use Spring's UriUtils

I don't have enough reputation to comment on answers, but I just wanted to note that downloading the JSR-311 api by itself will not work. You need to download the reference implementation (jersey).
Only downloading the api from the JSR page will give you a ClassNotFoundException when the api tries to look for an implementation at runtime.

I wrote my own, it's short, super simple, and you can copy it if you like:
http://www.dmurph.com/2011/01/java-uri-encoder/

It seems that CharEscapers from Google GData-java-client has what you want. It has uriPathEscaper method, uriQueryStringEscaper, and generic uriEscaper. (All return Escaper object which does actual escaping). Apache License.

I think that the URI class is the one that you are looking for.

Mmhh I know you've already discarded URLEncoder, but despite of what the docs say, I decided to give it a try.
You said:
For example, given an input:
http://google.com/resource?key=value
I expect the output:
http%3a%2f%2fgoogle.com%2fresource%3fkey%3dvalue
So:
C:\oreyes\samples\java\URL>type URLEncodeSample.java
import java.net.*;
public class URLEncodeSample {
public static void main( String [] args ) throws Throwable {
System.out.println( URLEncoder.encode( args[0], "UTF-8" ));
}
}
C:\oreyes\samples\java\URL>javac URLEncodeSample.java
C:\oreyes\samples\java\URL>java URLEncodeSample "http://google.com/resource?key=value"
http%3A%2F%2Fgoogle.com%2Fresource%3Fkey%3Dvalue
As expected.
What would be the problem with this?