Develop Reference

Encoding query parameters in URL using Java with valid charset

I am trying to understand what is the difference and importance of different charsets available while encoding and decoding text.
I have a scenario, where I want to call a RestAPI. The RestAPI has a base URL, for ex: https://myrestapiurl.com. Now to perform a GET request, the URL is formed by appending the id of the entity that I want to fetch, like: https://myrestapiurl.com('id')
id : It has no limitations on valid characters!
I have encountered an id: باقی ریسورس , So before calling the RestAPI, I need to encode it. Using Java's URLEncoder, I tried the following:
String s ="باقی ریسورس";
String encodedID = URLEncoder.encode(s, StandardCharsets.UTF_8.name() )
Using the encodedID, I try to make a request using PostMan. The request fails with 404 or 400 when I use different charset. It only succeeds when I encode using ISO_8859_1 as follows:
String encodedID = URLEncoder.encode(s, StandardCharsets.ISO_8859_1.name());
String URL = "https://myrestapiurl.com('" + encodedID + "')";
This works fine, through code as well as PostMan. My question is:
How can I decide which charset to use before encoding? Or should I have fallbacks? That is if it fails with UTF_8 then try with UTF_16 etc etc...but this is very in-efficient. In case if the entity actually doesn't exist, then, these tries would be overhead
Also, when I visit https://www.w3schools.com/tags/ref_urlencode.ASP and enter the text to be encoded, it provides the valid encoded string with ISO_8859_1 , how does it manage to do so?
How can this be done in Java without using any other extra libraries like apache? We don't have choice to add extra dependencies!

Documenting JSON in URL not possible

In my Rest API it should be possible to retrieve data which is inside a bounding box. Because the bounding box has four coordinates I want to design the GET requests in such way, that they accept the bounding box as JSON. Therefore I need to be able to send and document JSON strings as URL parameter.
The test itself works, but I can not document these requests with Spring RestDocs (1.0.0.RC1). I reproduced the problem with a simpler method. See below:
#Test public void ping_username() throws Exception
{
String query = "name={\"user\":\"Müller\"}";
String encodedQuery = URLEncoder.encode(query, "UTF-8");
mockMvc.perform(get(URI.create("/ping?" + encodedQuery)))
.andExpect(status().isOk())
.andDo(document("ping_username"));
}
When I remove .andDo(document("ping_username")) the test passes.
Stacktrace:
java.lang.IllegalArgumentException: Illegal character in query at index 32: http://localhost:8080/ping?name={"user":"Müller"}
at java.net.URI.create(URI.java:852)
at org.springframework.restdocs.mockmvc.MockMvcOperationRequestFactory.createOperationRequest(MockMvcOperationRequestFactory.java:79)
at org.springframework.restdocs.mockmvc.RestDocumentationResultHandler.handle(RestDocumentationResultHandler.java:93)
at org.springframework.test.web.servlet.MockMvc$1.andDo(MockMvc.java:158)
at application.rest.RestApiTest.ping_username(RestApiTest.java:65)
After I received the suggestion to encode the URL I tried it, but the problem remains.
The String which is used to create the URI in my test is now /ping?name%3D%7B%22user%22%3A%22M%C3%BCller%22%7D.
I checked the class MockMvcOperationRequestFactory which appears in the stacktrace, and in line 79 the following code is executed:
URI.create(getRequestUri(mockRequest)
+ (StringUtils.hasText(queryString) ? "?" + queryString : ""))
The problem here is that a not encoded String is used (in my case http://localhost:8080/ping?name={"user":"Müller"}) and the creation of the URI fails.
Remark:
Andy Wilkinson's answer is the solution for the problem. Although I think that David Sinfield is right and JSONs should be avoided in the URL to keep it simple. For my bounding box I will use a comma separated string, as it is used in WMS 1.1: BBOX=x1,y1,x2,y2

You haven't mentioned the version of Spring REST Docs that you're using, but I would guess that the problem is with URIUtil. I can't tell for certain as I can't see where URIUtil is from.
Anyway, using the JDK's URLEncoder works for me with Spring REST Docs 1.0.0.RC1:
String query = "name={\"user\":\"Müller\"}";
String encodedQuery = URLEncoder.encode(query, "UTF-8");
mockMvc.perform(get(URI.create("/baz?" + encodedQuery)))
.andExpect(status().isOk())
.andDo(document("ping_username"));
You can then use URLDecoder.decode on the server side to get the original JSON:
URLDecoder.decode(request.getQueryString(), "UTF-8")

The problem is that URIs have to be encoded as ACII. And ü is not a valid ASCII character, so it must be escaped in the url with % escaping.
If you are using Tomcat, you can use URIEncoding="UTF-8" in the Connector element of the server.xml, to configure UTF-8 escaping as default. If you do this, ü will be automatically converted to %C3%BC, which is the ASCII representation of the \uC3BC Unicode code-point (which represents ü).
Edit: It seems that I have missed the exact point of the error, but it is still the same error. Curly braces are invalid in a URI. Only the following characters are acceptable according to RFC 3986:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]#!$&'()*+,;=%
So these must be escaped too.

How to extract query string from a URL of a web-page using java

From the following URL in OathCallBack page I want extract access_token and token_type using Java. Any idea how to do it?
http://myserver.com/OathCallBack#state=/profile&access_token=ya29.AHES6ZQLqtYrPKuw2pMzURJtWuvINspm8-Vf5x-MZ5YzqVy5&token_type=Bearer&expires_in=3600
I tried the following, but unable to extract required information.
{
String scheme = req.getScheme(); // http
String serverName = req.getServerName(); // myserver.com
int serverPort = req.getServerPort(); // 80
String contextPath = req.getContextPath();
String servletPath = req.getServletPath();
String pathInfo = req.getPathInfo(); // return null and exception
String queryString = req.getQueryString(); // return null
}
<---------------------------------------------------------->
I am going to edit my question
Thank you every one for nice reply,
google did it,
you can refer to that link by URL
http://developers.google.com/accounts/docs/OAuth2Login
inside above URL page there is following link
http://accounts.google.com/o/oauth2/auth? scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww‌​.googleapis.com%2Fauth%2Fuserinfo.profile& state=%2Fprofile& redirect_uri=https%3A%2F%2Foauth2-login-demo.appspot.com%2Foauthcallback& response_type=token& client_id=812741506391.apps.googleusercontent.com
when you click on above link, then you will get your gmail login account access_token, and that token is after # sign

Some characters cannot be part of a URL (for example, the space) and some other characters have a special meaning in a URL: for example, the character # can be used to further specify a subsection (or fragment) of a document; the character = is used to separate a name from a value.
see http://en.wikipedia.org/wiki/Query_string for more:

It looks like the '#' should be a '?'.
In a normal URL, the parameters are passed as key value pairs following a '?' and multiple parameters chained together using '&'. A URL might look as follows:
http: //someserver.com/somedir/somepage.html?param1=value1&param2=value2&param3=value3.
Normally the Java servlet container would return everything after the '?' when calling getQueryString() but due to the absence of the '?' it returns null.
As #Sandeep Nair has suggested getRequestURL() should return this full URL to you and you could parse it using regular expressions to get the information you want. A possible regular expression to use would be along the lines of:
(?<=access_token=)[a-zA-Z0-9.-]*
However, getRequestURL() does NOT normally return the query string, so using this method is relying on the fact that there is a '#' rather and a '?' and is therefore probably not a great solution. See here.
I would advise that you find out why you are getting a '#' instead of a '?' and try to get this changed, if you can do this then the servlet container should manage the URL parameters for you and call to request.getAttribute("access_token") and request.getAttribute("token_type") (see here) will return both values as strings.

You get query string by calling
String queryString = req.getQueryString();
It correctly returns null in your case, as there is no query string. The characters after "#" are anchor specification, which is only visible to the browser and not sent to server.

How to encode URL to avoid special characters in Java? [duplicate]

This question already has answers here:
HTTP URL Address Encoding in Java
(24 answers)
Closed 5 years ago.
i need java code to encode URL to avoid special characters such as spaces and % and & ...etc

URL construction is tricky because different parts of the URL have different rules for what characters are allowed: for example, the plus sign is reserved in the query component of a URL because it represents a space, but in the path component of the URL, a plus sign has no special meaning and spaces are encoded as "%20".
RFC 2396 explains (in section 2.4.2) that a complete URL is always in its encoded form: you take the strings for the individual components (scheme, authority, path, etc.), encode each according to its own rules, and then combine them into the complete URL string. Trying to build a complete unencoded URL string and then encode it separately leads to subtle bugs, like spaces in the path being incorrectly changed to plus signs (which an RFC-compliant server will interpret as real plus signs, not encoded spaces).
In Java, the correct way to build a URL is with the URI class. Use one of the multi-argument constructors that takes the URL components as separate strings, and it'll escape each component correctly according to that component's rules. The toASCIIString() method gives you a properly-escaped and encoded string that you can send to a server. To decode a URL, construct a URI object using the single-string constructor and then use the accessor methods (such as getPath()) to retrieve the decoded components.
Don't use the URLEncoder class! Despite the name, that class actually does HTML form encoding, not URL encoding. It's not correct to concatenate unencoded strings to make an "unencoded" URL and then pass it through a URLEncoder. Doing so will result in problems (particularly the aforementioned one regarding spaces and plus signs in the path).

I also spent quite some time with this issue, so that's my solution:
String urlString2Decode = "http://www.test.com/äüö/path with blanks/";
String decodedURL = URLDecoder.decode(urlString2Decode, "UTF-8");
URL url = new URL(decodedURL);
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
String decodedURLAsString = uri.toASCIIString();

If you don't want to do it manually use Apache Commons - Codec library. The class you are looking at is: org.apache.commons.codec.net.URLCodec
String final url = "http://www.google.com?...."
String final urlSafe = org.apache.commons.codec.net.URLCodec.encode(url);

Here is my solution which is pretty easy:
Instead of encoding the url itself i encoded the parameters that I was passing because the parameter was user input and the user could input any unexpected string of special characters so this worked for me fine :)
String review="User input"; /*USER INPUT AS STRING THAT WILL BE PASSED AS PARAMTER TO URL*/
try {
review = URLEncoder.encode(review,"utf-8");
review = review.replace(" " , "+");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
String URL = "www.test.com/test.php"+"?user_review="+review;

I would echo what Wyzard wrote but add that:
for query parameters, HTML encoding is often exactly what the server is expecting; outside these, it is correct that URLEncoder should not be used
the most recent URI spec is RFC 3986, so you should refer to that as a primary source
I wrote a blog post a while back about this subject: Java: safe character handling and URL building

Escaping & in a URL

I am using jsps and in my url I have a value for a variable like say "L & T". Now when I try to retrieve the value for it by using request.getParameter I get only "L". It recognizes "&" as a separator and thus it is not getting considered as a whole string.
How do I solve this problem?

java.net.URLEncoder.encode("L & T", "utf8")
this outputs the URL-encoded, which is fine as a GET parameter:
L+%26+T

A literal ampersand in a URL should be encoded as: %26
// Your URL
http://www.example.com?a=l&t
// Encoded
http://www.example.com?a=l%26t

You need to "URL encode" the parameters to avoid this problem. The format of the URL query string is:
...?<name>=<value>&<name>=<value>&<etc>
All <name>s and <value>s need to be URL encoded, which basically means transforming all the characters that could be interpreted wrongly (like the &) into %-escaped values. See this page for more information:
http://www.w3schools.com/TAGS/ref_urlencode.asp
If you're generating the problem URL with Java, you use this method:
String str = URLEncoder.encode(input, "UTF-8");
Generating the URL elsewhere (some templates or JS or raw markup), you need to fix the problem at the source.

You can use UriUtils#encode(String source, String encoding) from Spring Web. This utility class also provides means for encoding only some parts of the URL, like UriUtils#encodePath.