I'm making a call to the Google Translate API, one via Apache HTTP Client and one via Spring's RestTemplate, and getting different results back. Both are GETing exactly the same URL:
I want to translate "Professeur des écoles" from French to English.
The URL used is (split onto two lines for sake of readability):
private static String URL = "https://www.googleapis.com/language/translate/v2?
key=AIzaSyBNv1lOS...&source=fr&target=en&q=Professeur+des+%C3%A9coles";
Apache:
#Test
public void apache() throws IOException {
String response = Request.Get(URL).execute().returnContent().asString();
System.out.println(response);
}
Returns (correctely):
{
"data": {
"translations": [
{
"translatedText": "School teacher"
}
]
}
}
#Test
public void spring() {
RestTemplate template = new RestTemplate();
String response = template.getForObject(URL, String.class);
System.out.println(response);
}
Returns (incorrectely):
{
"data": {
"translations": [
{
"translatedText": "Professor + of +% C3% A9coles"
}
]
}
}
Am I missing something in RestTemplate HTTP header configuration?
RestTemplate methods that accept a String URL perform URL encoding.
For each HTTP method there are three variants: two accept a URI
template string and URI variables (array or map) while a third accepts
a URI. Note that for URI templates it is assumed encoding is
necessary, e.g. restTemplate.getForObject("http://example.com/hotel
list") becomes "http://example.com/hotel%20list". This also means if
the URI template or URI variables are already encoded, double encoding
will occur, e.g. http://example.com/hotel%20list becomes
http://example.com/hotel%2520list).
Presumably you've provided the following String as the first argument
https://www.googleapis.com/language/translate/v2?key=MY_KEY&source=fr&target=en&q=Professeur+des+%C3%A9coles
The character % must be encoded. Your q parameter's value therefore becomes
Professeur%2Bdes%2B%25C3%25A9coles
which, if you decode, is equivalent to
Professeur+des+%C3%A9coles
Google's translation services doesn't know what to do with %C3%A9coles.
As the documentation suggests
To avoid that use a URI method variant to provide (or re-use) a
previously encoded URI. To prepare such an URI with full control over
encoding, consider using UriComponentsBuilder.
instead of using overloads that accept a String URL, construct a URI yourself and use that.
Apache's HttpComponents Fluent API does not specify the behavior but it seems the String value is taken as is.
Related
We do have a service that generates pdf documents dynamically(each request - slightly different PDF document). So I have to create a contract test for that service.
Problem statement
STUB side should return a predefined pdf file as a byte array. - OK
SERVER side should not check the response in assertions, or at least check if it matches by some regexp. - NOT WORKING
Here is my contract.groovy
import org.springframework.cloud.contract.spec.Contract
[
Contract.make {
request {
method "GET"
urlPath("/pdfEndpoint")
}
response {
status 200
headers {
contentType(applicationPdf())
}
body(value(
consumer(fileAsBytes("staticFileToSentOnStubResponse.pdf")),
producer(regex(nonBlank())) /*the issue is with that line*/
))
}
}
]
GenereatedTestClass.java
#Test
public void contractPdf() throws Exception {
// given:
MockMvcRequestSpecification request = given();
// when:
ResponseOptions response = given().spec(request)
.get("/pdfEndpoint");
// then:
assertThat(response.statusCode()).isEqualTo(200);
assertThat(response.header("Content-Type")).matches("application/pdf.*");
// and:
String responseBody = response.getBody().asString();
assertThat(responseBody).isEqualTo("^\\s*\\S[\\S\\s]*"); // .isEqualTo() but needed matches()
}
Is there any way to update the groovy file to have in the generated class the folowing assertion
assertThat(responseBody).matches("^\\s*\\S[\\S\\s]*");
instead of
assertThat(responseBody).isEqualTo("^\\s*\\S[\\S\\s]*");
UPDATE:
Also, I have noticed that it generates assertThat(..).isEqualTo() in case i have contentType(applicationPdf()) in the response headers.
If I put 'application/json' instead - it will work as expected.
Resources:
org.springframework.cloud:spring-cloud-dependencies - Hoxton.SR8
java11
I'm trying to obtain data from Botify's REST API to use it inside a project, which is also a REST API. I'm using an instance of Spring's RestTemplate class to make the actual requests to Botify, specifically the .exchange method as I need to pass Botify's key as a header parameter.
My problem comes when I need to call to a method of the endpoint which takes a URL as a part of the request's URI (not a parameter). Documentation of this endpoint is in https://developers.botify.com/api/reference/#!/Analysis/getUrlDetail
Basically the structure of the requests is like this:
/analyses/{username}/{project_slug}/{analysis_slug}/urls/{url}
The last part of that URI is a URL address, which needs to be encoded in UTF-8 to make it possible to separate it from the actual request.
The problem is (I believe) that the .exchange method always encodes the request, so what I try to send like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com
...ends up like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%253A%252F%252Fwww.example.com'
Which obviously doesn't work. This is an excerpt from the method that makes the call to Botify:
public String callBotifyEndpoint(String reportType, String parameters) throws UnsupportedEncodingException {
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com"
HttpHeaders headers = new HttpHeaders();
headers.set("Authorization", "Token " + this.apiKey);
HttpEntity<String> entity = new HttpEntity<>(headers);
UriComponentsBuilder botifyQueryBuilder = UriComponentsBuilder.fromUriString(request);
String queryStringBuild = botifyQueryBuilder.build(true).toUriString();
String botifyResult = null;
try {
System.out.println("Calling Botify API: " + queryStringBuild);
ResponseEntity<String> response = botifyTemplate.exchange(queryStringBuild, HttpMethod.GET, entity, String.class);
if(response.hasBody()) {
botifyResult = response.getBody();
}
} catch(RestClientException ex) {
ex.printStackTrace();
}
try {
} catch (Exception e) {
// TODO: handle exception
}
return botifyResult;
}
In this line:
botifyQueryBuilder.build(true).toUriString();
The "true" parameter indicates whether the data is already encoded or not. I've tried to disable it but the result is the same.
I've removed actual request generation process (along with my user and project's name) to simplify things, but this should return a response from Botify with the existing data for that URL.
Instead, it returns a 400 bad request error (which makes sense, because the URL is not correct).
I'm feeling like this may be a bug in RestTemplate's .exchange method, but maybe I'm not using it properly. Any suggestions?
Don't encode prematurly as you do here:
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com";
Use parameter placeholders feature in RestTemplate instead of text concatenation.
Refer to:
Spring RestTemplate GET with parameters
I am trying to access the contents of an API and I need to send a URL using RestTemplate.
String url1 = "http://api.example.com/Search?key=52ddafbe3ee659bad97fcce7c53592916a6bfd73&term=&limit=100&sort={\"price\":\"desc\"}";
OutputPage page = restTemplate.getForObject(url1, OutputPage .class);
But, I am getting the following error.
Exception in thread "main" java.lang.IllegalArgumentException: Not enough variable values available to expand '"price"'
at org.springframework.web.util.UriComponents$VarArgsTemplateVariables.getValue(UriComponents.java:284)
at org.springframework.web.util.UriComponents.expandUriComponent(UriComponents.java:220)
at org.springframework.web.util.HierarchicalUriComponents.expandInternal(HierarchicalUriComponents.java:317)
at org.springframework.web.util.HierarchicalUriComponents.expandInternal(HierarchicalUriComponents.java:46)
at org.springframework.web.util.UriComponents.expand(UriComponents.java:162)
at org.springframework.web.util.UriTemplate.expand(UriTemplate.java:119)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:501)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:239)
at hello.Application.main(Application.java:26)
If I remove the sort criteria, it is working properly.
I need to parse the JSON using sort criteria.
Any help will be much appreciated.
Thanks
The root cause is that RestTemplate considers curly braces {...} in the given URL as a placeholder for URI variables and tries to replace them based on their name. For example
{pageSize}
would try to get a URI variable called pageSize. These URI variables are specified with some of the other overloaded getForObject methods. You haven't provided any, but your URL expects one, so the method throws an exception.
One solution is to make a String object containing the value
String sort = "{\"price\":\"desc\"}";
and provide a real URI variable in your URL
String url1 = "http://api.example.com/Search?key=52ddafbe3ee659bad97fcce7c53592916a6bfd73&term=&limit=100&sort={sort}";
You would call your getForObject() like so
OutputPage page = restTemplate.getForObject(url1, OutputPage.class, sort);
I strongly suggest you do not send any JSON in a request parameter of a GET request but rather send it in the body of a POST request.
If the solution suggested by sotirios-delimanolis is a little difficult to implement in a scenario, and if the URI string containing curly braces and other characters is guaranteed to be correct, it might be simpler to pass the encoded URI string to a method of RestTemplate that hits the ReST server.
The URI string can be built using UriComponentsBuilder.build(), encoded using UriComponents.encode(), and sent using RestTemplate.exchange() like this:
public ResponseEntity<Object> requestRestServer()
{
HttpEntity<?> entity = new HttpEntity<>(requestHeaders);
UriComponentsBuilder builder = UriComponentsBuilder.fromUriString(rawValidUrl)
.queryParams(
(LinkedMultiValueMap<String, String>) allRequestParams);
UriComponents uriComponents = builder.build().encode();
ResponseEntity<Object> responseEntity = restTemplate.exchange(uriComponents.toUri(), HttpMethod.GET,
entity, String.class);
return responseEntity;
}
Building, encoding, and extracting URI have been seperated out for clarity in the above code snippet.
You can URL encode the parameter values:
String url1 = "http://api.example.com/Search?key=52ddafbe3ee659bad97fcce7c53592916a6bfd73&term=&limit=100&sort=";
org.apache.commons.codec.net.URLCodec codec = new org.apache.commons.codec.net.URLCodec();
url1 = url1 + codec.encode("{\"price\":\"desc\"}");
OutputPage page = restTemplate.getForObject(url1, OutputPage.class);
You can set a specific UriTemplateHandler in your restTemplate. This handler would just ignore uriVariables :
UriTemplateHandler skipVariablePlaceHolderUriTemplateHandler = new UriTemplateHandler() {
#Override
public URI expand(String uriTemplate, Object... uriVariables) {
return retrieveURI(uriTemplate);
}
#Override
public URI expand(String uriTemplate, Map<String, ?> uriVariables) {
return retrieveURI(uriTemplate);
}
private URI retrieveURI(String uriTemplate) {
return UriComponentsBuilder.fromUriString(uriTemplate).build().toUri();
}
};
restTemplate.setUriTemplateHandler(skipVariablePlaceHolderUriTemplateHandler);
You can encode url before using RestTemplate
URLEncoder.encode(data, StandardCharsets.UTF_8.toString());
You can simply append a variable key to the URL and give the value using the restTemplate.getForObject() method.
Example:
String url = "http://example.com/api?key=12345&sort={data}";
String data="{\"price\":\"desc\"}";
OutputPage page = restTemplate.getForObject(url, OutputPage.class, data);
how to check protocol is present in URL , if not present need to append it.
is there any class to achieve this in java?
eg: String URL = www.google.com
need to get http://www.google.com
Just use String.startsWith("http://") to check this.
public String ensure_has_protocol(final String a_url)
{
if (!a_url.startsWith("http://"))
{
return "http://" + a_url;
}
return a_url;
}
EDIT:
An alternative would use a java.net.URL instance, whose constructor would throw an java.net.MalformedURLException if the URL did not contain a (legal) protocol (or was invalid for any other reason):
public URL make_url(final String a_url) throws MalformedURLException
{
try
{
return new URL(a_url);
}
catch (final MalformedURLException e)
{
}
return new URL("http://" + a_url);
}
You can use URL.toString() to obtain string representation of the URL. This is an improvement on the startsWith() approach as it guarantees that return URL is valid.
Let's say you have String url = www.google.com. String class methods would be enough for the goal of checking protocol identifiers. For example, url.startsWith("https://") would check whether a specific string is starting with the given protocol name.
However, are these controls enough for validation?
I think they aren't enough. First of all, you should define a list of valid protocol identifiers, e.g. a String array like {"http", "ftp", "https", ...}. Then you can parse your input String with regex ("://") and test your URL header whether it belongs to the list of valid protocol identifiers. And domain name validation methods are beyond this question, you can/should handle it with different techniques as well.
Just for completeness, I would do something like the following:
import com.google.common.base.Strings;
private static boolean isUrlHttps(String url){
if(Strings.isNullOrEmpty(url))
return false;
return url.toLowerCase().startsWith("https://");
}
I'm not writing a mail application, so I don't have access to all the headers and such. All I have is something like the block at the end of this question. I've tried using the JavaMail API to parse this, using something like
Session s = Session.getDefaultInstance(new Properties());
InputStream is = new ByteArrayInputStream(<< String to parse >>);
MimeMessage message = new MimeMessage(s, is);
Multipart multipart = (Multipart) message.getContent();
But, it just tells me that message.getContent is a String, not a Multipart or MimeMultipart. Plus, I don't really need all the overhead of the whole JavaMail API, I just need to parse the text into it's parts. Here's an example:
This is a multi-part message in MIME format.\n\n------=_NextPart_000_005D_01CC73D5.3BA43FB0\nContent-Type: text/plain;\n\tcharset="iso-8859-1"\nContent-Transfer-Encoding: quoted-printable\n\nStuff:\n\n Please read this stuff at the beginning of each week. =\nFeel free to discuss it throughout the week.\n\n\n--=20\n\nMrs. Suzy M. Smith\n555-555-5555\nsuzy#suzy.com\n------=_NextPart_000_005D_01CC73D5.3BA43FB0\nContent-Type: text/html;\n\tcharset="iso-8859-1"\nContent-Transfer-Encoding: quoted-printable\n\n\n\n\n\n\n\n\n\nStuff:\n =20\nPlease read this stuff at the beginning of each =\nweek. Feel=20\nfree to discuss it throughout the week.\n-- Mrs. Suzy M. Smith555-555-5555suzy#suzy.com\n\n------=_NextPart_000_005D_01CC73D5.3BA43FB0--\n\n
First I took your example message and replaced all occurrences of \n with newlines and \t with tabs.
Then I downloaded the JARs from the Mime4J project, a subproject of Apache James, and executed the GUI parsing example org.apache.james.mime4j.samples.tree.MessageTree with the transformed message above as input. And apparently Mime4J was able to parse the message and to extract the HTML message part.
There are a few things wrong with the text you posted.
It is not a valid multi-part mime. Check out wikipedia reference which, while non-normative, is still correct.
The mime boundary is not defined. From the wikipedia example: Content-Type: multipart/mixed; boundary="frontier" shows that the boundary is "frontier". In your example, "----=_NextPart_000_005D_01CC73D5.3BA43FB0" is the boundary, but that can only be determined by scanning the text (i.e. the mime is malformed). You need to instruct the goofball that is passing you the mime content that you also need to know the mime boundary value, which is not defined in a message header. If you get the entire body of the message you will have enough because the body of the message starts with MIME-Version: 1.0 followed by Content-Type: multipart/mixed; boundary="frontier" where frontier will be replaced with the value of the boundary for the encoded mime.
If the person who is sending the body is a goofball (changed from monkey because monkey is too judgemental - my bad DwB), and will not (more likely does not know how to) send the full body, you can derive the boundary by scanning the text for a line that starts and ends with "--" (i.e. --boundary--). Note that I mentioned a "line". The terminal boundary is actually "--boundary--\n".
Finally, the stuff you posted has 2 parts. The first part appears to define substitutions to take place in the second part. If this is true, the Content-Type: of the first part should probably be something other than "text/plain". Perhaps "companyname/substitution-definition" or something like that. This will allow for multiple (as in future enhancements) substitution formats.
Can create MimeMultipart from http request.
javax.mail.internet.MimeMultipart m = new MimeMultipart(new ServletMultipartDataSource(httpRequest));
public class ServletMultipartDataSource implements DataSource {
String contentType;
InputStream inputStream;
public ServletMultipartDataSource(ServletRequest request) throws IOException {
inputStream = new SequenceInputStream(new ByteArrayInputStream("\n".getBytes()), request.getInputStream());
contentType = request.getContentType();
}
public InputStream getInputStream() throws IOException {
return inputStream;
}
public OutputStream getOutputStream() throws IOException {
return null;
}
public String getContentType() {
return contentType;
}
public String getName() {
return "ServletMultipartDataSource";
}
}
For get submitted form parameter need parse BodyPart headers:
public String getStringParameter(String name) throws MessagingException, IOException {
for (int i = 0; i < getCount(); i++) {
BodyPart bodyPart = m.getBodyPart(i);
String[] nameHeader = bodyPart.getHeader("Content-Disposition");
if (nameHeader != null && content instanceof String) {
for (String bodyName : nameHeader) {
if (bodyName.contains("name=\"" + name + "\"")) return String.valueOf(bodyPart.getContent());
}
}
}
return null;
}
If you are using javax.servlet.http.HttpServlet to receive the message, you will have to use HttpServletRequests.getHeaders to obtain the value of the HTTP header content-type. You will then use org.apache.james.mime4j.stream.MimeConfig.setHeadlessParsing to set the MimeConfig with the information so that it can properly process the mime message.
It appears that you are using HttpServletRequest.getInputStream to read the contents of the request. The input stream returned only has the content of the message after the HTTP headers (terminated by a blank line). That is why you have to extract content-type from the HTTP headers and feed it to the parser using setHeadlessParsing.