I'm trying to obtain data from Botify's REST API to use it inside a project, which is also a REST API. I'm using an instance of Spring's RestTemplate class to make the actual requests to Botify, specifically the .exchange method as I need to pass Botify's key as a header parameter.
My problem comes when I need to call to a method of the endpoint which takes a URL as a part of the request's URI (not a parameter). Documentation of this endpoint is in https://developers.botify.com/api/reference/#!/Analysis/getUrlDetail
Basically the structure of the requests is like this:
/analyses/{username}/{project_slug}/{analysis_slug}/urls/{url}
The last part of that URI is a URL address, which needs to be encoded in UTF-8 to make it possible to separate it from the actual request.
The problem is (I believe) that the .exchange method always encodes the request, so what I try to send like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com
...ends up like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%253A%252F%252Fwww.example.com'
Which obviously doesn't work. This is an excerpt from the method that makes the call to Botify:
public String callBotifyEndpoint(String reportType, String parameters) throws UnsupportedEncodingException {
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com"
HttpHeaders headers = new HttpHeaders();
headers.set("Authorization", "Token " + this.apiKey);
HttpEntity<String> entity = new HttpEntity<>(headers);
UriComponentsBuilder botifyQueryBuilder = UriComponentsBuilder.fromUriString(request);
String queryStringBuild = botifyQueryBuilder.build(true).toUriString();
String botifyResult = null;
try {
System.out.println("Calling Botify API: " + queryStringBuild);
ResponseEntity<String> response = botifyTemplate.exchange(queryStringBuild, HttpMethod.GET, entity, String.class);
if(response.hasBody()) {
botifyResult = response.getBody();
}
} catch(RestClientException ex) {
ex.printStackTrace();
}
try {
} catch (Exception e) {
// TODO: handle exception
}
return botifyResult;
}
In this line:
botifyQueryBuilder.build(true).toUriString();
The "true" parameter indicates whether the data is already encoded or not. I've tried to disable it but the result is the same.
I've removed actual request generation process (along with my user and project's name) to simplify things, but this should return a response from Botify with the existing data for that URL.
Instead, it returns a 400 bad request error (which makes sense, because the URL is not correct).
I'm feeling like this may be a bug in RestTemplate's .exchange method, but maybe I'm not using it properly. Any suggestions?
Don't encode prematurly as you do here:
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com";
Use parameter placeholders feature in RestTemplate instead of text concatenation.
Refer to:
Spring RestTemplate GET with parameters
Related
I facing a SonarQube bug and am not able to figure out whats the issue. SonnarQube's issue is, change this code to not construct the URL from user-controlled data.
#Value("${...}")
String apiKey;
#Value("${...}")
String apiUrl;
public Response apiResponse(String location) {
HttpHeaders headers = new HttpHeaders();
headers.add("x-apikey", apiKey);
HttpEntity<Object> entity = new HttpEntity<>(headers);
String url = apiUrl + location; // SonarQube issue: tainted value is propagated
Response response = null;
try {
ResponseEntity<Response> responseEntity = restTemplate.exchange(url, HttpMethod.GET, entity, Response.class); // SonarQube issue: Tainted value is used to perform a security- sensitive operation.
response = responseEntity.getBody();
} catch(Exception){
// doesn't throw anything
}
return response;
}
#Cacheable(...)
Response cacheResponse(String location, String tokenKey) {
return apiResponse(location); // SonarQube issue: tainted value is propagated
}
This fixed the issue, but why is that so? and how can I apply this in the above code?
String url = apiUrl + location; // SonarQube issue: tainted
Instead, I just tried hardcoding the value of location and fixed the issue.
String url = apiUrl + "location";
So weird...
I added validation for the Location variable and this solved the issue
if(!location.matches(...)) {
throw error.....
}
String url = apiUrl + location;
What SonarQube is trying to tell you is that you are exposing your logic to input from the clients. A better solution would be to refactor your code to not depend on a specific header from the client to perform some action. Its hard to suggest sample code without seeing a little more of the codebase.
You are using input from the client/user (namly in the variable location) to construct an URL. So if the client/user supplies an malicious value to location he could form an invalid URL.
In the second example String url = apiUrl + "location"; you are not using user input, as "location" is a hard coded String.
I don't know what you try to achieve with the code. But maybe it's better to hold a list of possible URLs and the user supplies and enum value that maps to an URL.
String url = "https://someurl/%s";
url = String.format(url,location);
sendRequest(url);
Maybe this approach won't give error.
I am working on an HTTP Client SDK where I implemented the bridge pattern, the architecture looks like this:
The reasoning for this is that I can have multiple types of bodies and HTTP methods to implement for each type of Message, so with this I believed I would reduce the amount of classes I would create.
Something that I've noticed and has bothered me a lot is in the SMS request portion I am repeating a lot of the RestTemplate code to send the submits example:
#Override
public AdvancedSmsResponse postMessage() {
super.httpHeaders.set("Authorization", super.authorization.toBase64());
super.httpHeaders.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
super.httpHeaders.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<Object> entity = new HttpEntity<>(smsAdvanced, httpHeaders);
try{
ResponseEntity<AdvancedSmsResponse> response = super.requestRestTemplate.exchange(getUrlHost(), HttpMethod.POST, entity, AdvancedSmsResponse.class);
objectMapper.configure(SerializationFeature.INDENT_OUTPUT, true);
String smsResponseLog = objectMapper.writeValueAsString(response.getBody());
super.LOGGER.info( "\n" + response.getStatusCode() + "\n" + smsResponseLog);
return response.getBody();
}catch (HttpStatusCodeException | JsonProcessingException e){
return null;
}
}
#Override
public Object getDeliveryReport() {
super.httpHeaders.set("Authorization", super.authorization.toBase64());
super.httpHeaders.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
super.httpHeaders.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<Object> entity = new HttpEntity<>(null, httpHeaders);
try{
ResponseEntity<AdvancedSmsResponse> response = super.requestRestTemplate.exchange(getUrlHost() + "/report", HttpMethod.GET, entity, AdvancedSmsResponse.class);
objectMapper.configure(SerializationFeature.INDENT_OUTPUT, true);
String smsResponseLog = objectMapper.writeValueAsString(response.getBody());
super.LOGGER.info( "\n" + response.getStatusCode() + "\n" + smsResponseLog);
return response.getBody();
}catch (HttpStatusCodeException | JsonProcessingException e){
return null;
}
}
Is this implementation good? And also is there is a way to avoid so much repetition for the request execution part?
I would be tempted to extract a method that accepts the HttpEntity, the path and the HTTP method. However, I have not seen the full set of your requests. If you have a small number of things that varies, then this simple extraction might be enough to avoid the repeating code. If you have a lot of things that vary then a builder might be the way to go.
I am trying to access the contents of an API and I need to send a URL using RestTemplate.
String url1 = "http://api.example.com/Search?key=52ddafbe3ee659bad97fcce7c53592916a6bfd73&term=&limit=100&sort={\"price\":\"desc\"}";
OutputPage page = restTemplate.getForObject(url1, OutputPage .class);
But, I am getting the following error.
Exception in thread "main" java.lang.IllegalArgumentException: Not enough variable values available to expand '"price"'
at org.springframework.web.util.UriComponents$VarArgsTemplateVariables.getValue(UriComponents.java:284)
at org.springframework.web.util.UriComponents.expandUriComponent(UriComponents.java:220)
at org.springframework.web.util.HierarchicalUriComponents.expandInternal(HierarchicalUriComponents.java:317)
at org.springframework.web.util.HierarchicalUriComponents.expandInternal(HierarchicalUriComponents.java:46)
at org.springframework.web.util.UriComponents.expand(UriComponents.java:162)
at org.springframework.web.util.UriTemplate.expand(UriTemplate.java:119)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:501)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:239)
at hello.Application.main(Application.java:26)
If I remove the sort criteria, it is working properly.
I need to parse the JSON using sort criteria.
Any help will be much appreciated.
Thanks
The root cause is that RestTemplate considers curly braces {...} in the given URL as a placeholder for URI variables and tries to replace them based on their name. For example
{pageSize}
would try to get a URI variable called pageSize. These URI variables are specified with some of the other overloaded getForObject methods. You haven't provided any, but your URL expects one, so the method throws an exception.
One solution is to make a String object containing the value
String sort = "{\"price\":\"desc\"}";
and provide a real URI variable in your URL
String url1 = "http://api.example.com/Search?key=52ddafbe3ee659bad97fcce7c53592916a6bfd73&term=&limit=100&sort={sort}";
You would call your getForObject() like so
OutputPage page = restTemplate.getForObject(url1, OutputPage.class, sort);
I strongly suggest you do not send any JSON in a request parameter of a GET request but rather send it in the body of a POST request.
If the solution suggested by sotirios-delimanolis is a little difficult to implement in a scenario, and if the URI string containing curly braces and other characters is guaranteed to be correct, it might be simpler to pass the encoded URI string to a method of RestTemplate that hits the ReST server.
The URI string can be built using UriComponentsBuilder.build(), encoded using UriComponents.encode(), and sent using RestTemplate.exchange() like this:
public ResponseEntity<Object> requestRestServer()
{
HttpEntity<?> entity = new HttpEntity<>(requestHeaders);
UriComponentsBuilder builder = UriComponentsBuilder.fromUriString(rawValidUrl)
.queryParams(
(LinkedMultiValueMap<String, String>) allRequestParams);
UriComponents uriComponents = builder.build().encode();
ResponseEntity<Object> responseEntity = restTemplate.exchange(uriComponents.toUri(), HttpMethod.GET,
entity, String.class);
return responseEntity;
}
Building, encoding, and extracting URI have been seperated out for clarity in the above code snippet.
You can URL encode the parameter values:
String url1 = "http://api.example.com/Search?key=52ddafbe3ee659bad97fcce7c53592916a6bfd73&term=&limit=100&sort=";
org.apache.commons.codec.net.URLCodec codec = new org.apache.commons.codec.net.URLCodec();
url1 = url1 + codec.encode("{\"price\":\"desc\"}");
OutputPage page = restTemplate.getForObject(url1, OutputPage.class);
You can set a specific UriTemplateHandler in your restTemplate. This handler would just ignore uriVariables :
UriTemplateHandler skipVariablePlaceHolderUriTemplateHandler = new UriTemplateHandler() {
#Override
public URI expand(String uriTemplate, Object... uriVariables) {
return retrieveURI(uriTemplate);
}
#Override
public URI expand(String uriTemplate, Map<String, ?> uriVariables) {
return retrieveURI(uriTemplate);
}
private URI retrieveURI(String uriTemplate) {
return UriComponentsBuilder.fromUriString(uriTemplate).build().toUri();
}
};
restTemplate.setUriTemplateHandler(skipVariablePlaceHolderUriTemplateHandler);
You can encode url before using RestTemplate
URLEncoder.encode(data, StandardCharsets.UTF_8.toString());
You can simply append a variable key to the URL and give the value using the restTemplate.getForObject() method.
Example:
String url = "http://example.com/api?key=12345&sort={data}";
String data="{\"price\":\"desc\"}";
OutputPage page = restTemplate.getForObject(url, OutputPage.class, data);
I'm making a call to the Google Translate API, one via Apache HTTP Client and one via Spring's RestTemplate, and getting different results back. Both are GETing exactly the same URL:
I want to translate "Professeur des écoles" from French to English.
The URL used is (split onto two lines for sake of readability):
private static String URL = "https://www.googleapis.com/language/translate/v2?
key=AIzaSyBNv1lOS...&source=fr&target=en&q=Professeur+des+%C3%A9coles";
Apache:
#Test
public void apache() throws IOException {
String response = Request.Get(URL).execute().returnContent().asString();
System.out.println(response);
}
Returns (correctely):
{
"data": {
"translations": [
{
"translatedText": "School teacher"
}
]
}
}
#Test
public void spring() {
RestTemplate template = new RestTemplate();
String response = template.getForObject(URL, String.class);
System.out.println(response);
}
Returns (incorrectely):
{
"data": {
"translations": [
{
"translatedText": "Professor + of +% C3% A9coles"
}
]
}
}
Am I missing something in RestTemplate HTTP header configuration?
RestTemplate methods that accept a String URL perform URL encoding.
For each HTTP method there are three variants: two accept a URI
template string and URI variables (array or map) while a third accepts
a URI. Note that for URI templates it is assumed encoding is
necessary, e.g. restTemplate.getForObject("http://example.com/hotel
list") becomes "http://example.com/hotel%20list". This also means if
the URI template or URI variables are already encoded, double encoding
will occur, e.g. http://example.com/hotel%20list becomes
http://example.com/hotel%2520list).
Presumably you've provided the following String as the first argument
https://www.googleapis.com/language/translate/v2?key=MY_KEY&source=fr&target=en&q=Professeur+des+%C3%A9coles
The character % must be encoded. Your q parameter's value therefore becomes
Professeur%2Bdes%2B%25C3%25A9coles
which, if you decode, is equivalent to
Professeur+des+%C3%A9coles
Google's translation services doesn't know what to do with %C3%A9coles.
As the documentation suggests
To avoid that use a URI method variant to provide (or re-use) a
previously encoded URI. To prepare such an URI with full control over
encoding, consider using UriComponentsBuilder.
instead of using overloads that accept a String URL, construct a URI yourself and use that.
Apache's HttpComponents Fluent API does not specify the behavior but it seems the String value is taken as is.
When Twilio invokes a callback method to fetch the TwiML <Say> for Voice, I see that Twilio sets "x-twilio-signature" in the HTTP header.
I need to verify that the actual request came from Twilio.
I have a simple war file running on Tomcat and the app is built using Spring.
I did something like the following:
//Get the TwilioUtils object initialized
TwilioUtils twilioUtils = new TwilioUtils("******myAuthToken");
//Get the URL from HttpRequest
String url = httpRequest.getRequestURL().toString();
Map<String, String> allRequestParams = getAllRequestParams(httpRequest);
Map<String, String> headers = getAllRequestHeaders(httpRequest);
//Get the signature generated for the Url and request parameters
//allRequestParams is a map of all request values posted to my service by Twilio
String validSig = twilioUtils.getValidationSignature(url, allRequestParams);
//Get the x-twilio-signature value from the http header map
String xTwilioSignature = headers.get("x-twilio-signature”);
//This is different from what I get below
logger.info("validSig = " + validSig);
logger.info("xTwilioSignature = " + xTwilioSignature );
//This is always false
logger.info("Signature matched : " + twilioUtils.validateRequest(xTwilioSignature, url,
allRequestParams));
I would like to know what am I doing wrong. Is my approach to validate "x-twilio-signature" incorrect?
If it is incorrect, what's the right way to do it?
I am using the helper library class TwilioUtils provided by Twilio to validate it.
All the time the signature from Twilio is different from what I get from the TwilioUtils object.
Megan from Twilio here.
Are you following the steps suggested in the security documentation?
validateRequest expects three arguments. I believe you're missing the url there.
Consider this example:
public class TwilioUtilsExample {
public static void main(String[] args) {
// Account details
String accountSid = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
String authToken = "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY";
//This is the signature we expect
String expected_sig = "SSSSSSSSSSSSSSSSSSSSSSSSSSSS";
//This is the url that twilio requested
String url = "http://UUUUUUUUUUUUUUU";
//These are the post params twilio sent in its request
Map<String,String> params = new HashMap<String,String>();
// Be sure to see the signing notes at twilio.com/docs/security
TwilioUtils util = new TwilioUtils(authToken, accountSid);
boolean result = util.validateRequest(expected_sig, url, params);
if (result) {
System.out.print( "The signature is valid!\n" );
} else {
System.out.print( "The signature was NOT VALID. It might have been spoofed!\n" );
}
}
}
Hope this is helpful!