I have recently switched to Spring for consuming REST API calls hosted by ServiceNow.
I am building my URI as below:
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(baseUrl.toString());
logger.info("URI before Query Param: " + builder.build().encode().toUri());
builder.queryParam("sysparm_limit", "2000000");
builder.queryParam("sysparm_offset", "0");
builder.queryParam("sysparm_exclude_reference_link", "true");
//this line is the issue because there is a = sign here
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
logger.info("URI after Query Param: " + builder.build().encode().toUri());
The output of this code is:
INFO: URI before Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=AX0011
The problem is with the final builder.queryParam. I am getting output as this:
sysparm_query%3Duser_name=AX0011
but what I want is:
sysparm_query=user_name=AX0011
So that eventually the final URI looks like this:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name=Z001NR6
So I tried replacing,
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
by:
builder.query("sysparm_query=user_name=" + snUser.getUser_name());
which changed the original output from:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=Z001NR6
to:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name%3DZ001NR6
Notice how sysparm_query%3Duser_name=Z001NR6 changed to sysparm_query=user_name%3DZ001NR6
Is ther anyway to see a = instead of %3D in the output?
The param looks quite strange - however - you can add it manually using the UriComponentsBuilder#query method:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc")
.query("query=username=JOE");
System.out.println(builder.build().toString());
// produces https://example.com/api/?param1=12345¶m2=abc&query=username=JOE
System.out.println(builder.build().encode().toString());
// produces https://example.com/api/?param1=12345¶m2=abc&query=username%3DJOE
Manual concatenation:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc");
// the parameter has to be properly url-encoded manually (not shown here)
String uri = builder.build().encode().toString() + "&query=username=JOE";
System.out.println(uri);
// produces: https://example.com/api/?param1=12345¶m2=abc&query=username=JOE
The query component of a URL is frequently used to carry information in key=value pairs; you could think of this as a Map<String, String>. In this case, = and & are special characters that delimit these pairs, and they must be encoded when they form part of the key or the value to ensure that anything reading the query string this way is able to parse it properly.
In your case, how you use the builder depends on how you would want to retrieve your data later on. There are two options:
// Building the URL:
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
// URL contains ...&sysparm_query%3Duser_name=AX0011
// Reading the parsed query map:
Map<String, String> query = ...
String data = query.get("sysparm_query=user_name");
// value is AX0011
Or
// Building the URL:
builder.queryParam("sysparm_query", "user_name=" + snUser.getUser_name());
// URL contains ...&sysparm_query=user_name%3DAX0011
// Reading the parsed query map:
Map<String, String> query = ...
String value = query.get("sysparm_query");
// value is user_name=AX0011
In a correctly encoded URL, one of the = will always be encoded as %3D. Using a UriComponentsBuilder ensures that your URLs will be correctly encoded and that anything reading your URLs will be able to do so properly without data loss.
Related
Below was the code used to encode uri having query params using UriComponentsBuilder
String uri = "http://hostname/api/items"
// api expected with params --> http://hostname/api/items?filter=IN('123') and id eq '123_&123'
restTemplate.exchange(UriComponentsBuilder.fromUriString(uri).queryParam("filter","IN('123') and id eq '123_&123'").encode().toUriString(), HttpMethod.GET, request, Response_Entity.class)
When above code is called, somehow at api side, i was getting 2 query params with keys -->filter & 123
How to handle it correctly using ?
try encoding query param by using URLEncoder.
String param = "IN('123') and id eq '123_&123'";
String encodedParam = URLEncoder.encode(param, Charset.defaultCharset()));
restTemplate.exchange(UriComponentsBuilder.fromUriString(uri).queryParam("filter",encodedParam).toUriString(), httpMethod, httpEntity, Some_Entity.class)
https://www.baeldung.com/java-url-encoding-decoding
Somehow query params are encoded and at api side, by default these are retrieved correctly after decoding, if i use toURI() of UriComponentsBuilder
Same was not working if i convert it to string using toUriString
Below is the code which worked for me.
URI uri = UriComponentsBuilder.fromUriString(uri)
.queryParam("filter",encodedParam)
.encode()
.build()
.toUri();
restTemplate.exchange(uri, HttpMethod.GET, request, Response_Entity.class)
Based on question Spring's UriComponentsBuilder.queryParam issue, what if a query parameter has a plus ("+") in its value? By default, a plus will not encoded. Why and how can I force this?
In my example, I want to use da ISO-coded timestamp and I need to encode the plus, because the other endpoint will interprete the "+" as a space (" ") and this breaks by timestamp.
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl("https://example.com/api/")//
.queryParam("date1", "2020-10-05T08:20:00.000+02:00") //
.query("date2=2020-10-05T08:20:00.000+02:00");
System.out.println(builder.build().toString());
// produces https://example.com/api/?date1=2020-10-05T08:20:00.000+02:00&date2=2020-10-05T08:20:00.000+02:00
System.out.println(builder.build().encode().toString());
// produces https://example.com/api/?date1=2020-10-05T08:20:00.000+02:00&date2=2020-10-05T08:20:00.000+02:00
The expected result should encode the plus sign like 2020-10-05T08%3A20%3A00.000%2B02%3A00 or at least 2020-10-05T08:20:00.000%2B02:00 with %2B.
You can use DefaultUriBuilderFactory to encode the query values as such:
String baseUrl = "https://example.com";
DefaultUriBuilderFactory uriBuilderFactory = new DefaultUriBuilderFactory(baseUrl);
String uri = uriBuilderFactory.uriString("/api/")
.queryParam("date1", "{date1}")
.build("2020-10-05T08:20:00.000+02:00").toString(); // https://example.com/api/?date1=2020-10-05T08%3A20%3A00.000%2B02%3A00
I am using AsyncRestTemplate to make an API call to Google Maps from a Springboot 1.5.2 service. Unfortunately, some of my search strings contain a pound/hashtag sign #
and are not getting encoded properly in my search parameters. I am using the exchange method.
An example below for address 05406, VT, BURLINGTON, 309 College St #10:
#Service
public class ExampleAsyncRestTemplate {
private AsyncRestTemplate asyncRestTemplate;
#Autowired
public ExampleAsyncRestTemplate() {
this.asyncRestTemplate = new AsyncRestTemplate();
}
public ListenableFuture<ResponseEntity<T>> getGeoCodedAddress() {
String googleUrl = "https://maps.googleapis.com/maps/api/geocode/json?address=05406, VT, BURLINGTON, 309 College St #10&key=some_key";
Map<String, String> uriVariables = new HashMap<>();
uriVariables.put("address", "05406, VT, BURLINGTON, 309 College St #10");
uriVariables.put("key", "some_key");
return asyncRestTemplate.exchange(googleUrl, HttpMethod.GET, new HttpEntity<>(), GoogleResponse.class, uriVariables);
}
}
The resulting URL gets encoded as:
https://maps.googleapis.com/maps/api/geocode/json?address=05406,%20VT,%20BURLINGTON,%20309%20College%20St%20#10&key=some_key
Note that the # is still in the address parameter, when it should be encoded as %23 as per the docs.
Digging into the debugger, seems like the string after the # (10&key=some_key) is being taken as the fragment of the URL. Hence why the # never gets encoded.
Has anybody been able to submit # signs in your query parameters using AsyncRestTemplate?
The only thing I've been able to come up with is replacing # with number, which actually works, but feels hacky/suboptimal.
Thanks for your help.
Note that googleUrl is a template where the encoded params get interpolated into. So you cannot provide the actual parameters as part of the url. You need to change the String into a template like this
final String googleUrl = "https://maps.googleapis.com/maps/api/geocode/json?address={address}&key={key}";
This returns the correct encoding:
https://maps.googleapis.com/maps/api/geocode/json?address=05406,%20VT,%20BURLINGTON,%20309%20College%20St%20%2310&key=some_key
I have an url and would like to parse and extract params from it. My implementation is based on the following stackoverflow post
However my url is more complex than the one used in the post above. It looks like this:
https://example.com/cdscontent/login?initialURI=https%3A%2F%2Fexample.com%2Fdashboard%2F%3Fportal%3Dmyportal%26LO%3D4%26contentid%3D10007.786471%26viewmode%3Dcontent%26variant%3D%2Fmyportal%2F
As you can see it has the param initialURI which is (encoded) url itself and the order of the params in it cannot be changed.
When I run org.apache.http.client.utils.URLEncodedUtils#parse it returns
[initialURI=https://example.com/dashboard/?portal=myportal, LO=4, contentid=10007.786471, viewmode=content, variant=/myportal/]
as you can see it parses every param except portal. It is still bound to https://example.com/dashboard/ In other words I am expecting this:
[initialURI=https://example.com/dashboard/, portal=myportal, LO=4, contentid=10007.786471, viewmode=content, variant=/myportal/]
Am I doing here something wrong or do you think that URLEncodedUtils#parse cannot handle this case?
Do you have any alternative to suggest?
Thx a lot!
Unit test to try
public class UrlParserTest {
#Test
public void testParseUrl() throws UnsupportedEncodingException, URISyntaxException {
String url =
"https://www.example.com/cdscontent/login?initialURI=https%3A%2F%2Fwww.example.com%2Fdashboard%2F%3Fportal%3Dmyportal%26LO%3D4%26contentid%3D10007.786471%26viewmode%3Dcontent%26variant%3D%2Fmyportal%2F";
String decoded = URLDecoder.decode(url, "UTF-8");
List<NameValuePair> params = URLEncodedUtils.parse(new URI(decoded), "UTF-8");
System.out.println(params);
}
}
What are we working with
You have the following url (decoded):
https://www.example.com/cdscontent/login?initialURI=https://www.example.com/dashboard/?portal=myportal&LO=4&contentid=10007.786471&viewmode=content&variant=/myportal/
This url consists of the main url:
https://www.example.com/cdscontent/login
which has 1 query parameter initialURI:
https://www.example.com/dashboard/?portal=myportal&LO=4&contentid=10007.786471&viewmode=content&variant=/myportal/
This url has multiple query parameters (the ones you're looking for):
portal=myportal&LO=4&contentid=10007.786471&viewmode=content&variant=/myportal/
Solution
Step 1:
We first must get the url in the query parameter initialURI:
List<NameValuePair> params = URLEncodedUtils.parse(new URI(url), Charset.forName("UTF-8"));
// Find first NameValuePair where the name equals initialURI
Optional<NameValuePair> initialURI = params.stream()
.filter(e -> e.getName().equals("initialURI"))
.findFirst();
System.out.println(initialURI);
This prints:
Optional[initialURI=https://www.example.com/dashboard/?portal=myportal&LO=4&contentid=10007.786471&viewmode=content&variant=/myportal/]
Step 2:
Now we can get the query parameters of this url and print them:
List<NameValuePair> initialParams = URLEncodedUtils
.parse(new URI(initialURI.get().getValue()), Charset.forName("UTF-8"));
System.out.println(initialParams);
This results in:
[portal=myportal, LO=4, contentid=10007.786471, viewmode=content, variant=/myportal/]
Note
This is not entirely your expected behavior, you expected initialURI=https://example.com/dashboard/ to be in the list aswell. However you can see that this is not a query parameter, the entire url in initialURI (with it's query parameters) is the query parameter.
I am trying to find a method that would parse an URL, decoded it and returned the decoded components in an unambiguous way.
URLDecoder isn't a right fit, because it may return ambiguous String, e.g.
URLDecoder.decode("http://www.google.com?q=abc%26def", "UTF-8")
returns:
http://www.google.com?q=abc&def
So the information about escaped & is lost.
I'd like to have something like:
DecodedUrlComponents cmp = GreatURLDecoder.decode(url);
Map<String, List<String>> decodedQuery = cmp.getQuery();
decodedQuery.get("q").get(0); //returns "abc&def"
How do I accomplish that?
EDIT:
Thanks for the responses, but my question was a bit different: I would like to get decoded components in an unambiguous way, so neither of the following does what I need:
new URI("http://www.google.com?q=abc%26def").getRawQuery() returns encoded query: q=abc%26def
new URI("http://www.google.com?q=abc%26def").getQuery() returns ambiguous value: q=abc&def
URLDecoder.decode("http://www.google.com?q=abc%26def", "UTF-8") returns ambiguous value: http://www.google.com?q=abc&def
org.springframework.web.util.UriComponentsBuilder.fromUriString("http://www.google.com?q=abc%26def").build(true).getQueryParams() - close, but still not what I want, because it returns a map of encoded params: {q=[abc%26def]}
With spring framework (org.springframework.web.util) you can do the following:
URI uri = <your_uri_here>;
UriComponentsBuilder uriComponentsBuilder = UriComponentsBuilder.fromUri(uri);
UriComponents uriComponents = uriComponentsBuilder.build();
String path = uriComponents.getPath();
MultiValueMap<String, String> queryParams = uriComponents.getQueryParams(); //etc.
You could for example use an implementation of javax.ws.rs.core.UriInfo. One example would be org.jboss.resteasy.spi.ResteasyUriInfo. If you're using maven you only need to add the following to your pom.xml:
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<version>3.0.6.Final</version>
</dependency>
Then the following code should do what you want:
UriInfo ui = new ResteasyUriInfo(new URI("http://www.google.com?q=abc%26def"));
List<String> qValues = ui.getQueryParameters().get("q");
for (String q : qValues) {
System.out.println(q);
}
Use the following:
String url = "http://www.google.com?test=34%3fg";
URL testUrl = new java.net.URL(url);
System.out.println(testUrl.getQuery());
Should print test=34%3fg.
URLDecoder does not split your URL into components, it simply translates the String representation thereof to a specific format, as hinted by it's JavaDoc and its signature, which returns a String. As others mentioned, you should just construct a URL object from your string, which exposes all the functionality you need. See here.
Generate a java.net.URL from your URL-String and then use mwthods like url.getQuery(), url.getProtocol(), url.getHost() etc. - it's all there.