Develop Reference

UrlBuilder query string encoding not working

WebClient urlbuilder query string has changed
String key = "bEyBOCB4SoL%2F1gI%s"
webClient.get()
.scheme("http")
.host("www.localhost:8080")
.queryParam("key", URLEncoder.encode(key, StandardCharsets.UTF_8.toString()))
.build()
... send request
query string has changed "bEyBOCB4SoL%25252F1gI%2525"
origin = "bEyBOCB4SoL%2F1gI%s"

You are triple-encoding the value.
The key string is already encoded, see the %2F which is the encoded value of /.
You then manually encode by calling URLEncoder.encode(...), which will encode % as %25.
The queryParam(...) method will then encode again, which will encode % as %25.
The result is the %25252F you see.
Change the code:
String key = "bEyBOCB4SoL/1gI%s";
.queryParam("key", key)
The result will be: bEyBOCB4SoL%2F1gI%25s

Cannot get '#' symbol in Controller using Spring #RequestParam

I have the following request Url /search?charset=UTF-8&q=C%23C%2B%2B.
My controller looks like
#RequestMapping(method = RequestMethod.GET, params = "q")
public String refineSearch(#RequestParam("q") final String searchQuery,....
and here i have searchQuery = 'CC++'.
'#' is encoded in '%23' and '+' is '%2B'.
Why searchQuery does not contain '#'?
searchQuery in debug

I resolved a similar problem by URL encoding the hash part. We have Spring web server and mix of JS and VueJS client. This fixed my problem:
const location = window.location;
const redirect = location.pathname + encodeURIComponent(location.hash);

The main cause is known as the "fragment identifier". You find more detail for Fragment Identifier right here. It says:
The fragment identifier introduced by a hash mark # is the optional last part of a URL for a document. It is typically used to identify a portion of that document.
When you write # sign, it contains info for clientbase. Put everything only the browser needs here. You can get this problem for all types of URI characters you can look Percent Encoding for this. In my opinion The simple solution is character replacing, you could try replace in serverbase.

Finally i found a problem.In filters chain ServletRequest is wrapped in XSSRequestWrapper with DefaultXSSValueTranslator and here is the method String stripXSS(String value) which iterates through pattern list,in case if value matches with pattern, method will delete it.
Pattern list contains "\u0023" pattern and '#' will be replaced with ""
DefaultXSSValueTranslator.
private String stripXSS(String value) {
Pattern scriptPattern;
if (value != null && value.length() > 0) {
for(Iterator var3 = this.patterns.iterator(); var3.hasNext(); value = scriptPattern.matcher(value).replaceAll("")) {
scriptPattern = (Pattern)var3.next();
}
}
return value;
}

AsyncRestTemplate '#' sign encoding in query parameter

I am using AsyncRestTemplate to make an API call to Google Maps from a Springboot 1.5.2 service. Unfortunately, some of my search strings contain a pound/hashtag sign #
and are not getting encoded properly in my search parameters. I am using the exchange method.
An example below for address 05406, VT, BURLINGTON, 309 College St #10:
#Service
public class ExampleAsyncRestTemplate {
private AsyncRestTemplate asyncRestTemplate;
#Autowired
public ExampleAsyncRestTemplate() {
this.asyncRestTemplate = new AsyncRestTemplate();
}
public ListenableFuture<ResponseEntity<T>> getGeoCodedAddress() {
String googleUrl = "https://maps.googleapis.com/maps/api/geocode/json?address=05406, VT, BURLINGTON, 309 College St #10&key=some_key";
Map<String, String> uriVariables = new HashMap<>();
uriVariables.put("address", "05406, VT, BURLINGTON, 309 College St #10");
uriVariables.put("key", "some_key");
return asyncRestTemplate.exchange(googleUrl, HttpMethod.GET, new HttpEntity<>(), GoogleResponse.class, uriVariables);
}
}
The resulting URL gets encoded as:
https://maps.googleapis.com/maps/api/geocode/json?address=05406,%20VT,%20BURLINGTON,%20309%20College%20St%20#10&key=some_key
Note that the # is still in the address parameter, when it should be encoded as %23 as per the docs.
Digging into the debugger, seems like the string after the # (10&key=some_key) is being taken as the fragment of the URL. Hence why the # never gets encoded.
Has anybody been able to submit # signs in your query parameters using AsyncRestTemplate?
The only thing I've been able to come up with is replacing # with number, which actually works, but feels hacky/suboptimal.
Thanks for your help.

Note that googleUrl is a template where the encoded params get interpolated into. So you cannot provide the actual parameters as part of the url. You need to change the String into a template like this
final String googleUrl = "https://maps.googleapis.com/maps/api/geocode/json?address={address}&key={key}";
This returns the correct encoding:
https://maps.googleapis.com/maps/api/geocode/json?address=05406,%20VT,%20BURLINGTON,%20309%20College%20St%20%2310&key=some_key

Spring's UriComponentsBuilder.queryParam issue

I have recently switched to Spring for consuming REST API calls hosted by ServiceNow.
I am building my URI as below:
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(baseUrl.toString());
logger.info("URI before Query Param: " + builder.build().encode().toUri());
builder.queryParam("sysparm_limit", "2000000");
builder.queryParam("sysparm_offset", "0");
builder.queryParam("sysparm_exclude_reference_link", "true");
//this line is the issue because there is a = sign here
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
logger.info("URI after Query Param: " + builder.build().encode().toUri());
The output of this code is:
INFO: URI before Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=AX0011
The problem is with the final builder.queryParam. I am getting output as this:
sysparm_query%3Duser_name=AX0011
but what I want is:
sysparm_query=user_name=AX0011
So that eventually the final URI looks like this:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name=Z001NR6
So I tried replacing,
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
by:
builder.query("sysparm_query=user_name=" + snUser.getUser_name());
which changed the original output from:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=Z001NR6
to:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name%3DZ001NR6
Notice how sysparm_query%3Duser_name=Z001NR6 changed to sysparm_query=user_name%3DZ001NR6
Is ther anyway to see a = instead of %3D in the output?

The param looks quite strange - however - you can add it manually using the UriComponentsBuilder#query method:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc")
.query("query=username=JOE");
System.out.println(builder.build().toString());
// produces https://example.com/api/?param1=12345&param2=abc&query=username=JOE
System.out.println(builder.build().encode().toString());
// produces https://example.com/api/?param1=12345&param2=abc&query=username%3DJOE
Manual concatenation:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc");
// the parameter has to be properly url-encoded manually (not shown here)
String uri = builder.build().encode().toString() + "&query=username=JOE";
System.out.println(uri);
// produces: https://example.com/api/?param1=12345&param2=abc&query=username=JOE

The query component of a URL is frequently used to carry information in key=value pairs; you could think of this as a Map<String, String>. In this case, = and & are special characters that delimit these pairs, and they must be encoded when they form part of the key or the value to ensure that anything reading the query string this way is able to parse it properly.
In your case, how you use the builder depends on how you would want to retrieve your data later on. There are two options:
// Building the URL:
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
// URL contains ...&sysparm_query%3Duser_name=AX0011
// Reading the parsed query map:
Map<String, String> query = ...
String data = query.get("sysparm_query=user_name");
// value is AX0011
Or
// Building the URL:
builder.queryParam("sysparm_query", "user_name=" + snUser.getUser_name());
// URL contains ...&sysparm_query=user_name%3DAX0011
// Reading the parsed query map:
Map<String, String> query = ...
String value = query.get("sysparm_query");
// value is user_name=AX0011
In a correctly encoded URL, one of the = will always be encoded as %3D. Using a UriComponentsBuilder ensures that your URLs will be correctly encoded and that anything reading your URLs will be able to do so properly without data loss.

How to parse and decode URI in Java to URI components?

I am trying to find a method that would parse an URL, decoded it and returned the decoded components in an unambiguous way.
URLDecoder isn't a right fit, because it may return ambiguous String, e.g.
URLDecoder.decode("http://www.google.com?q=abc%26def", "UTF-8")
returns:
http://www.google.com?q=abc&def
So the information about escaped & is lost.
I'd like to have something like:
DecodedUrlComponents cmp = GreatURLDecoder.decode(url);
Map<String, List<String>> decodedQuery = cmp.getQuery();
decodedQuery.get("q").get(0); //returns "abc&def"
How do I accomplish that?
EDIT:
Thanks for the responses, but my question was a bit different: I would like to get decoded components in an unambiguous way, so neither of the following does what I need:
new URI("http://www.google.com?q=abc%26def").getRawQuery() returns encoded query: q=abc%26def
new URI("http://www.google.com?q=abc%26def").getQuery() returns ambiguous value: q=abc&def
URLDecoder.decode("http://www.google.com?q=abc%26def", "UTF-8") returns ambiguous value: http://www.google.com?q=abc&def
org.springframework.web.util.UriComponentsBuilder.fromUriString("http://www.google.com?q=abc%26def").build(true).getQueryParams() - close, but still not what I want, because it returns a map of encoded params: {q=[abc%26def]}

With spring framework (org.springframework.web.util) you can do the following:
URI uri = <your_uri_here>;
UriComponentsBuilder uriComponentsBuilder = UriComponentsBuilder.fromUri(uri);
UriComponents uriComponents = uriComponentsBuilder.build();
String path = uriComponents.getPath();
MultiValueMap<String, String> queryParams = uriComponents.getQueryParams(); //etc.

You could for example use an implementation of javax.ws.rs.core.UriInfo. One example would be org.jboss.resteasy.spi.ResteasyUriInfo. If you're using maven you only need to add the following to your pom.xml:
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<version>3.0.6.Final</version>
</dependency>
Then the following code should do what you want:
UriInfo ui = new ResteasyUriInfo(new URI("http://www.google.com?q=abc%26def"));
List<String> qValues = ui.getQueryParameters().get("q");
for (String q : qValues) {
System.out.println(q);
}

Use the following:
String url = "http://www.google.com?test=34%3fg";
URL testUrl = new java.net.URL(url);
System.out.println(testUrl.getQuery());
Should print test=34%3fg.

URLDecoder does not split your URL into components, it simply translates the String representation thereof to a specific format, as hinted by it's JavaDoc and its signature, which returns a String. As others mentioned, you should just construct a URL object from your string, which exposes all the functionality you need. See here.

Generate a java.net.URL from your URL-String and then use mwthods like url.getQuery(), url.getProtocol(), url.getHost() etc. - it's all there.