Develop Reference

AsyncRestTemplate '#' sign encoding in query parameter

I am using AsyncRestTemplate to make an API call to Google Maps from a Springboot 1.5.2 service. Unfortunately, some of my search strings contain a pound/hashtag sign #
and are not getting encoded properly in my search parameters. I am using the exchange method.
An example below for address 05406, VT, BURLINGTON, 309 College St #10:
#Service
public class ExampleAsyncRestTemplate {
private AsyncRestTemplate asyncRestTemplate;
#Autowired
public ExampleAsyncRestTemplate() {
this.asyncRestTemplate = new AsyncRestTemplate();
}
public ListenableFuture<ResponseEntity<T>> getGeoCodedAddress() {
String googleUrl = "https://maps.googleapis.com/maps/api/geocode/json?address=05406, VT, BURLINGTON, 309 College St #10&key=some_key";
Map<String, String> uriVariables = new HashMap<>();
uriVariables.put("address", "05406, VT, BURLINGTON, 309 College St #10");
uriVariables.put("key", "some_key");
return asyncRestTemplate.exchange(googleUrl, HttpMethod.GET, new HttpEntity<>(), GoogleResponse.class, uriVariables);
}
}
The resulting URL gets encoded as:
https://maps.googleapis.com/maps/api/geocode/json?address=05406,%20VT,%20BURLINGTON,%20309%20College%20St%20#10&key=some_key
Note that the # is still in the address parameter, when it should be encoded as %23 as per the docs.
Digging into the debugger, seems like the string after the # (10&key=some_key) is being taken as the fragment of the URL. Hence why the # never gets encoded.
Has anybody been able to submit # signs in your query parameters using AsyncRestTemplate?
The only thing I've been able to come up with is replacing # with number, which actually works, but feels hacky/suboptimal.
Thanks for your help.

Note that googleUrl is a template where the encoded params get interpolated into. So you cannot provide the actual parameters as part of the url. You need to change the String into a template like this
final String googleUrl = "https://maps.googleapis.com/maps/api/geocode/json?address={address}&key={key}";
This returns the correct encoding:
https://maps.googleapis.com/maps/api/geocode/json?address=05406,%20VT,%20BURLINGTON,%20309%20College%20St%20%2310&key=some_key

How to deal with dot in an url path in writing service

I am writing a "GET" endpoint looks like following:
#RequestMapping(value = "/{configSetId}/{version}", method = RequestMethod.GET, produces = { "application/json" })
public ResponseEntity<List<Metadata>> getMetadatasByConfigSetIdAndVersion(
#PathVariable("configSetId") final String configSetId,
#PathVariable("version") final String version) {
return ResponseEntity.ok(metadataService.getMetadatasByConfigSetIdAndVersion(configSetId, version));
}
So I can send a "GET" request to localhost:8080/{configSetId}/{version}, for example: localhost:8080/configSet1/v1
But the problem is if the version is "v1.02", then the ".02" will be ignored and the version I got is v1. How can I avoid this behaivor? Thank you!

Since "." is special character so don't use it directly on your request.
Instead of
v1.02
Just try
v1%2E02
Where %2E is URL encoding of ".".
For more information, please refer to this link HTML URL Encoding

Spring's UriComponentsBuilder.queryParam issue

I have recently switched to Spring for consuming REST API calls hosted by ServiceNow.
I am building my URI as below:
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(baseUrl.toString());
logger.info("URI before Query Param: " + builder.build().encode().toUri());
builder.queryParam("sysparm_limit", "2000000");
builder.queryParam("sysparm_offset", "0");
builder.queryParam("sysparm_exclude_reference_link", "true");
//this line is the issue because there is a = sign here
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
logger.info("URI after Query Param: " + builder.build().encode().toUri());
The output of this code is:
INFO: URI before Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=AX0011
The problem is with the final builder.queryParam. I am getting output as this:
sysparm_query%3Duser_name=AX0011
but what I want is:
sysparm_query=user_name=AX0011
So that eventually the final URI looks like this:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name=Z001NR6
So I tried replacing,
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
by:
builder.query("sysparm_query=user_name=" + snUser.getUser_name());
which changed the original output from:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=Z001NR6
to:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name%3DZ001NR6
Notice how sysparm_query%3Duser_name=Z001NR6 changed to sysparm_query=user_name%3DZ001NR6
Is ther anyway to see a = instead of %3D in the output?

The param looks quite strange - however - you can add it manually using the UriComponentsBuilder#query method:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc")
.query("query=username=JOE");
System.out.println(builder.build().toString());
// produces https://example.com/api/?param1=12345&param2=abc&query=username=JOE
System.out.println(builder.build().encode().toString());
// produces https://example.com/api/?param1=12345&param2=abc&query=username%3DJOE
Manual concatenation:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc");
// the parameter has to be properly url-encoded manually (not shown here)
String uri = builder.build().encode().toString() + "&query=username=JOE";
System.out.println(uri);
// produces: https://example.com/api/?param1=12345&param2=abc&query=username=JOE

The query component of a URL is frequently used to carry information in key=value pairs; you could think of this as a Map<String, String>. In this case, = and & are special characters that delimit these pairs, and they must be encoded when they form part of the key or the value to ensure that anything reading the query string this way is able to parse it properly.
In your case, how you use the builder depends on how you would want to retrieve your data later on. There are two options:
// Building the URL:
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
// URL contains ...&sysparm_query%3Duser_name=AX0011
// Reading the parsed query map:
Map<String, String> query = ...
String data = query.get("sysparm_query=user_name");
// value is AX0011
Or
// Building the URL:
builder.queryParam("sysparm_query", "user_name=" + snUser.getUser_name());
// URL contains ...&sysparm_query=user_name%3DAX0011
// Reading the parsed query map:
Map<String, String> query = ...
String value = query.get("sysparm_query");
// value is user_name=AX0011
In a correctly encoded URL, one of the = will always be encoded as %3D. Using a UriComponentsBuilder ensures that your URLs will be correctly encoded and that anything reading your URLs will be able to do so properly without data loss.

IO Exception using java-google-translate-text-to-speech Api

I am having issues using java-google-translate-text-to-speech.Trying to translate a language to another. This is my code:
import com.gtranslate.Language;
import com.gtranslate.Translator;
import java.io.IOException;
public class Main {
public static void main(String[] args){
Translator translate = Translator.getInstance();
String text = translate.translate("Hello", Language.ENGLISH,Language.PORTUGUESE);
System.out.println(text);
}
}
Its giving me an error:
java.io.IOException: Server returned HTTP response code: 503 for URL: http://ipv4.google.com/sorry/IndexRedirect?continue=http://translate.google.com.br/translate_a/t%3Fclient%3Dt%26text%3DHello%26hl%3Den%26sl%3Den%26tl%3Den%26multires%3D1%26prev%3Dbtn%26ssel%3D0%26tsel%3D0%26sc%3D1&q=CGMSBHqsFhAY_L3FqQUiGQDxp4NLxnAO-gsMAyd56ktUpufqNjEC280
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1459)
at com.gtranslate.utils.WebUtils.source(WebUtils.java:24)
at com.gtranslate.parsing.ParseTextTranslate.parse(ParseTextTranslate.java:19)

Just found a simple 2-step solution. Please see Comment #4 at the following URL (requires only a minor modification of the sources):
https://code.google.com/p/java-google-translate-text-to-speech/issues/detail?id=8
STEP 1 in Comment #4 is straightforward. Let me cite it from the above webpage:
In class com.gtranslate.URLCONSTANT change public static final String GOOGLE_TRANSLATE_TEXT = "http://translate.google.com.br/translate_a/t?";
TO public static final String GOOGLE_TRANSLATE_TEXT1 = "http://translate.google.com.br/translate_a/single?";
...however in STEP 2 it is much simpler just to add a &dt=t URL parameter-value pair at the end of the generated URL in the com.gtranslate.parsing.ParseTextTranslate.appendURL() method.
...the original STEP 2 in Comment #4 above was the following, I cite (FYR):
STEP2) In the class, the appendURL function needs to be changed as shown com.gtranslate.parsing.ParseTextTranslate #Override public void appendURL() { Text input = textTranslate.getInput(); Text output = textTranslate.getOutput(); url = new StringBuilder(URLCONSTANTS.GOOGLE_TRANSLATE_TEXT); /* url.append("client=t&text=" + input.getText().replace(" ", "%20")); url.append("&hl=" + input.getLanguage()); url.append("&sl=" + input.getLanguage()); url.append("&tl=" + output.getLanguage()); url.append("&multires=1&prev=btn&ssel=0&tsel=0&sc=1"); */
url = new StringBuilder(URLCONSTANTS.GOOGLE_TRANSLATE_TEXT);
url.append("client=t&sl=auto&tl="+ output.getLanguage()
+"&hl=" + input.getLanguage()
+"&dt=bd&dt=ex&dt=ld&dt=md&dt=qc&dt=rw&dt=rm&dt=ss&dt=t&dt=at&ie=UTF-8&oe=UTF-8&otf=1&rom=1&ssel=0&tsel=3&kc=1&tk=620730|996163"
+ "&q=" + input.getText().replace(" ", "%20"));
}
.........end of citation...and sooooooooo, for example, just replace this line in the appendURL() method:
url.append("&multires=1&prev=btn&ssel=0&tsel=0&sc=1");
...to this:
url.append("&multires=1&prev=btn&ssel=0&tsel=0&sc=1&dt=t");
Additionally here are some values for the dt URL param, which practically specifies what to return in the reply:
t - translation of source text
at - alternate translations
rm - transcription / transliteration of source and translated texts
bd - dictionary, in case source text is one word (you get translations with articles, reverse translations, etc.)
md - definitions of source text, if it's one word
ss - synonyms of source text, if it's one word
ex - examples
...
P.S.: A similar HTTP 503 error happens with Google TTS (due to the background API change). You can find the solution to that problem here: Text to Speech 503 and Captcha Now

HTTP Response Code: 503 : Service Unavailable says : The server is currently unavailable may be because it is overloaded or down for maintenance.
The server might currently be unable to handle the request due to a temporary overloading or maintenance of the server.
Note: Some servers may simply refuse the connection and might result in 503 response

Handling non-english URL with Spring and App Engine Task Queue

I have this problem where I need to queue a page link with TaskQueue:
Queue queue = QueueFactory.getDefaultQueue();
for (String href : hrefs){
href = baseUrl + href;
pageLinks = pageLinks + "\n" + href;
queue.add(TaskOptions.Builder
.withUrl("/crawler")
.param("url", href));
l("Added to queue url=["+href+"]");
}
The problem here is that, I think the URL that gets passed into the queue contains ?'s for Arabic characters. As it keeps on rescheduling.
The String pageLinks however is outputed in the browser through Spring MVC, and I can properly see the Arabic character being displayed. So I'm pretty the links are ok.
If I copy one of the links output on the browser, and paste it to the browser URL it works fine. So I'm pretty sure that the reason that the queue keeps on recheduling because it gets the wrong URL.
What could I be missing here? Do I need to convert the String href before passing it into the queue?
The crawl service looks like this:
#RequestMapping(method = RequestMethod.GET, value = "/crawl",
produces = "application/json; charset=iso-8859-6")
public #ResponseBody String crawl(HttpServletRequest req, HttpServletResponse res,
#RequestParam(value="url", required = false) String url) {
l("Processs url:" + url);
}
Also do I need to convert the #QueryParam String url here to Arabic or not?

You must Url-encode the parameters. See this question: Java URL encoding of query string parameters