I am writing a Java lib and need to perform a request to a URL - currently using async-http-client from ning - and fetch its content. So I have a get method that returns a String
of the content of the fetched document. However, to be able to get it, I must perform a HTTP basic authentication and I'm not succeeding at this in my Java code:
public String get(String token) throws IOException {
String fetchURL = "https://www.eventick.com.br/api/v1/events/492";
try {
String encoded = URLEncoder.encode(token + ":", "UTF-8");
return this.asyncClient.prepareGet(fetchURL)
.addHeader("Authorization", "Basic " + encoded).execute().get().getResponseBody();
}
}
The code returns no error, it just doesn't fetch the URL because the authentication header is not being properly set, somehow.
With curl -u option I can easily get what I want:
curl https://www.eventick.com.br/api/v1/events/492 -u 'xxxxxxxxxxxxxxx:'
Returns:
{"events":[{"id":492,"title":"Festa da Bagaceira","venue":"Mangueirão de Paulista",
"slug":"bagaceira-fest", "start_at":"2012-07-29T16:00:00-03:00",
"links":{"tickets":[{"id":738,"name":"Normal"}]}}]}
How can this be done in Java? With the async-http-client lib? Or if you know how to do it using another way..
Any help is welcome!
You're close. You need to base 64 encode rather than URL encode. That is, you need
String encoded = Base64.getEncoder().encodeToString((user + ':' + password).getBytes(StandardCharsets.UTF_8));
rather than
String encoded = URLEncoder.encode(token + ":", "UTF-8");
(Note that for the benefit of others, since I'm answering 2 years later, in my answer I'm using the more standard "user:password" whereas your question has "token:". If "token:" is what you needed, then stick with that. But maybe that was part of the problem, too?)
Here is a short, self-contained, correct example
package so17380731;
import com.ning.http.client.AsyncHttpClient;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import javax.ws.rs.core.HttpHeaders;
public class BasicAuth {
public static void main(String... args) throws Exception {
try(AsyncHttpClient asyncClient = new AsyncHttpClient()) {
final String user = "StackOverflow";
final String password = "17380731";
final String fetchURL = "https://www.eventick.com.br/api/v1/events/492";
final String encoded = Base64.getEncoder().encodeToString((user + ':' + password).getBytes(StandardCharsets.UTF_8));
final String body = asyncClient
.prepareGet(fetchURL)
.addHeader(HttpHeaders.AUTHORIZATION, "Basic " + encoded)
.execute()
.get()
.getResponseBody(StandardCharsets.UTF_8.name());
System.out.println(body);
}
}
}
The documentation is very sketchy, but I think that you need to use a RequestBuilder following the pattern shown in the Request javadoc:
Request r = new RequestBuilder().setUrl("url")
.setRealm((new Realm.RealmBuilder()).setPrincipal(user)
.setPassword(admin)
.setRealmName("MyRealm")
.setScheme(Realm.AuthScheme.DIGEST).build());
r.execute();
(Obviously, this example is not Basic Auth, but there are clues as to how you would do it.)
FWIW, one problem with your current code is that a Basic Auth header uses base64 encoding not URL encoding; see the RFC2617 for details.
basically, do it like this:
BoundRequestBuilder request = asyncHttpClient
.preparePost(getUrl())
.setHeader("Accept", "application/json")
.setHeader("Content-Type", "application/json")
.setRealm(org.asynchttpclient.Dsl.basicAuthRealm(getUser(), getPassword()))
// ^^^^^^^^^^^-- this is the important part
.setBody(json);
Test can be found here:
https://github.com/AsyncHttpClient/async-http-client/blob/master/client/src/test/java/org/asynchttpclient/BasicAuthTest.java
This is also another way of adding Basic Authorization,
you can use any of two the classes for your use AsyncHttpClient,HttpClient,in this case i will use AsyncHttpClient
AsyncHttpClient client=new AsyncHttpClient();
Request request = client.prepareGet("https://www.eventick.com.br/api/v1/events/492").
setHeader("Content-Type","application/json")
.setHeader("Authorization","Basic b2pAbml1LXR2LmNvbTpnMGFRNzVDUnhzQ0ZleFQ=")
.setBody(jsonObjectRepresentation.toString()).build();
after adding header part
ListenableFuture<Response> r = null;
//ListenableFuture<Integer> f= null;
try{
r = client.executeRequest(request);
System.out.println(r.get().getResponseBody());
}catch(IOException e){
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
client.close();
it may be useful for you
Related
I facing a SonarQube bug and am not able to figure out whats the issue. SonnarQube's issue is, change this code to not construct the URL from user-controlled data.
#Value("${...}")
String apiKey;
#Value("${...}")
String apiUrl;
public Response apiResponse(String location) {
HttpHeaders headers = new HttpHeaders();
headers.add("x-apikey", apiKey);
HttpEntity<Object> entity = new HttpEntity<>(headers);
String url = apiUrl + location; // SonarQube issue: tainted value is propagated
Response response = null;
try {
ResponseEntity<Response> responseEntity = restTemplate.exchange(url, HttpMethod.GET, entity, Response.class); // SonarQube issue: Tainted value is used to perform a security- sensitive operation.
response = responseEntity.getBody();
} catch(Exception){
// doesn't throw anything
}
return response;
}
#Cacheable(...)
Response cacheResponse(String location, String tokenKey) {
return apiResponse(location); // SonarQube issue: tainted value is propagated
}
This fixed the issue, but why is that so? and how can I apply this in the above code?
String url = apiUrl + location; // SonarQube issue: tainted
Instead, I just tried hardcoding the value of location and fixed the issue.
String url = apiUrl + "location";
So weird...
I added validation for the Location variable and this solved the issue
if(!location.matches(...)) {
throw error.....
}
String url = apiUrl + location;
What SonarQube is trying to tell you is that you are exposing your logic to input from the clients. A better solution would be to refactor your code to not depend on a specific header from the client to perform some action. Its hard to suggest sample code without seeing a little more of the codebase.
You are using input from the client/user (namly in the variable location) to construct an URL. So if the client/user supplies an malicious value to location he could form an invalid URL.
In the second example String url = apiUrl + "location"; you are not using user input, as "location" is a hard coded String.
I don't know what you try to achieve with the code. But maybe it's better to hold a list of possible URLs and the user supplies and enum value that maps to an URL.
String url = "https://someurl/%s";
url = String.format(url,location);
sendRequest(url);
Maybe this approach won't give error.
I'm trying to obtain data from Botify's REST API to use it inside a project, which is also a REST API. I'm using an instance of Spring's RestTemplate class to make the actual requests to Botify, specifically the .exchange method as I need to pass Botify's key as a header parameter.
My problem comes when I need to call to a method of the endpoint which takes a URL as a part of the request's URI (not a parameter). Documentation of this endpoint is in https://developers.botify.com/api/reference/#!/Analysis/getUrlDetail
Basically the structure of the requests is like this:
/analyses/{username}/{project_slug}/{analysis_slug}/urls/{url}
The last part of that URI is a URL address, which needs to be encoded in UTF-8 to make it possible to separate it from the actual request.
The problem is (I believe) that the .exchange method always encodes the request, so what I try to send like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com
...ends up like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%253A%252F%252Fwww.example.com'
Which obviously doesn't work. This is an excerpt from the method that makes the call to Botify:
public String callBotifyEndpoint(String reportType, String parameters) throws UnsupportedEncodingException {
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com"
HttpHeaders headers = new HttpHeaders();
headers.set("Authorization", "Token " + this.apiKey);
HttpEntity<String> entity = new HttpEntity<>(headers);
UriComponentsBuilder botifyQueryBuilder = UriComponentsBuilder.fromUriString(request);
String queryStringBuild = botifyQueryBuilder.build(true).toUriString();
String botifyResult = null;
try {
System.out.println("Calling Botify API: " + queryStringBuild);
ResponseEntity<String> response = botifyTemplate.exchange(queryStringBuild, HttpMethod.GET, entity, String.class);
if(response.hasBody()) {
botifyResult = response.getBody();
}
} catch(RestClientException ex) {
ex.printStackTrace();
}
try {
} catch (Exception e) {
// TODO: handle exception
}
return botifyResult;
}
In this line:
botifyQueryBuilder.build(true).toUriString();
The "true" parameter indicates whether the data is already encoded or not. I've tried to disable it but the result is the same.
I've removed actual request generation process (along with my user and project's name) to simplify things, but this should return a response from Botify with the existing data for that URL.
Instead, it returns a 400 bad request error (which makes sense, because the URL is not correct).
I'm feeling like this may be a bug in RestTemplate's .exchange method, but maybe I'm not using it properly. Any suggestions?
Don't encode prematurly as you do here:
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com";
Use parameter placeholders feature in RestTemplate instead of text concatenation.
Refer to:
Spring RestTemplate GET with parameters
When Twilio invokes a callback method to fetch the TwiML <Say> for Voice, I see that Twilio sets "x-twilio-signature" in the HTTP header.
I need to verify that the actual request came from Twilio.
I have a simple war file running on Tomcat and the app is built using Spring.
I did something like the following:
//Get the TwilioUtils object initialized
TwilioUtils twilioUtils = new TwilioUtils("******myAuthToken");
//Get the URL from HttpRequest
String url = httpRequest.getRequestURL().toString();
Map<String, String> allRequestParams = getAllRequestParams(httpRequest);
Map<String, String> headers = getAllRequestHeaders(httpRequest);
//Get the signature generated for the Url and request parameters
//allRequestParams is a map of all request values posted to my service by Twilio
String validSig = twilioUtils.getValidationSignature(url, allRequestParams);
//Get the x-twilio-signature value from the http header map
String xTwilioSignature = headers.get("x-twilio-signature”);
//This is different from what I get below
logger.info("validSig = " + validSig);
logger.info("xTwilioSignature = " + xTwilioSignature );
//This is always false
logger.info("Signature matched : " + twilioUtils.validateRequest(xTwilioSignature, url,
allRequestParams));
I would like to know what am I doing wrong. Is my approach to validate "x-twilio-signature" incorrect?
If it is incorrect, what's the right way to do it?
I am using the helper library class TwilioUtils provided by Twilio to validate it.
All the time the signature from Twilio is different from what I get from the TwilioUtils object.
Megan from Twilio here.
Are you following the steps suggested in the security documentation?
validateRequest expects three arguments. I believe you're missing the url there.
Consider this example:
public class TwilioUtilsExample {
public static void main(String[] args) {
// Account details
String accountSid = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
String authToken = "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY";
//This is the signature we expect
String expected_sig = "SSSSSSSSSSSSSSSSSSSSSSSSSSSS";
//This is the url that twilio requested
String url = "http://UUUUUUUUUUUUUUU";
//These are the post params twilio sent in its request
Map<String,String> params = new HashMap<String,String>();
// Be sure to see the signing notes at twilio.com/docs/security
TwilioUtils util = new TwilioUtils(authToken, accountSid);
boolean result = util.validateRequest(expected_sig, url, params);
if (result) {
System.out.print( "The signature is valid!\n" );
} else {
System.out.print( "The signature was NOT VALID. It might have been spoofed!\n" );
}
}
}
Hope this is helpful!
I have spring application
which has URL of format
{URL}/locations/USA#CA#92121
I want to get 'USA#CA#92121' as one parameter in my REST controller .. How can I achieve that?
Currently its only giving "USA" when I use #PathVariable
Your URL will need to be url encoded. You can use URLEncoder in Java 8 for this.
See Java URL encoding of query string parameters
basically
String q = "random word £500 bank $";
String url = "http://example.com/query?q=" + URLEncoder.encode(q, "UTF-8");
You need to encode the URL so that it can handle all types of characters including spaces and special characters like #, $ etc..
The best way to do that would be to use an URLEncoder
and then to get back the value that you want, you will need to use the URLDecoder
For Example:
package com.test;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.net.URLDecoder;
public class EncodeURL {
public static void main(String args[]){
string url = "{URL}/locations/USA#CA#92121";
try {
string encodedString = URLEncoder.encode(url, "UTF-8");
System.out.println("Encoded String: " + encodedString);
System.out.println("Decoded String: " + URLDecoder.dencode(encodedString, "UTF-8"));
} catch (UnsupportedEncodingException ex) {
ex.printStackTrace();
}
}
}
So I was attempting to use this String in a URL :-
http://site-test.com/Meetings/IC/DownloadDocument?meetingId=c21c905c-8359-4bd6-b864-844709e05754&itemId=a4b724d1-282e-4b36-9d16-d619a807ba67&file=\\s604132shvw140\Test-Documents\c21c905c-8359-4bd6-b864-844709e05754_attachments\7e89c3cb-ce53-4a04-a9ee-1a584e157987\myDoc.pdf
In this code: -
String fileToDownloadLocation = //The above string
URL fileToDownload = new URL(fileToDownloadLocation);
HttpGet httpget = new HttpGet(fileToDownload.toURI());
But at this point I get the error: -
java.net.URISyntaxException: Illegal character in query at index 169:Blahblahblah
I realised with a bit of googling this was due to the characters in the URL (guessing the &), so I then added in some code so it now looks like so: -
String fileToDownloadLocation = //The above string
fileToDownloadLocation = URLEncoder.encode(fileToDownloadLocation, "UTF-8");
URL fileToDownload = new URL(fileToDownloadLocation);
HttpGet httpget = new HttpGet(fileToDownload.toURI());
However, when I try and run this I get an error when I try and create the URL, the error then reads: -
java.net.MalformedURLException: no protocol: http%3A%2F%2Fsite-test.testsite.com%2FMeetings%2FIC%2FDownloadDocument%3FmeetingId%3Dc21c905c-8359-4bd6-b864-844709e05754%26itemId%3Da4b724d1-282e-4b36-9d16-d619a807ba67%26file%3D%5C%5Cs604132shvw140%5CTest-Documents%5Cc21c905c-8359-4bd6-b864-844709e05754_attachments%5C7e89c3cb-ce53-4a04-a9ee-1a584e157987%myDoc.pdf
It looks like I can't do the encoding until after I've created the URL else it replaces slashes and things which it shouldn't, but I can't see how I can create the URL with the string and then format it so its suitable for use. I'm not particularly familiar with all this and was hoping someone might be able to point out to me what I'm missing to get string A into a suitably formatted URL to then use with the correct characters replaced?
Any suggestions greatly appreciated!
You need to encode your parameter's values before concatenating them to URL.
Backslash \ is special character which have to be escaped as %5C
Escaping example:
String paramValue = "param\\with\\backslash";
String yourURLStr = "http://host.com?param=" + java.net.URLEncoder.encode(paramValue, "UTF-8");
java.net.URL url = new java.net.URL(yourURLStr);
The result is http://host.com?param=param%5Cwith%5Cbackslash which is properly formatted url string.
I have the same problem, i read the url with an properties file:
String configFile = System.getenv("system.Environment");
if (configFile == null || "".equalsIgnoreCase(configFile.trim())) {
configFile = "dev.properties";
}
// Load properties
Properties properties = new Properties();
properties.load(getClass().getResourceAsStream("/" + configFile));
//read url from file
apiUrl = properties.getProperty("url").trim();
URL url = new URL(apiUrl);
//throw exception here
URLConnection conn = url.openConnection();
dev.properties
url = "https://myDevServer.com/dev/api/gate"
it should be
dev.properties
url = https://myDevServer.com/dev/api/gate
without "" and my problem is solved.
According to oracle documentation
Thrown to indicate that a malformed URL has occurred. Either no legal protocol could be found in a specification string or the string
could not be parsed.
So it means it is not parsed inside the string.
You want to use URI templates. Look carefully at the README of this project: URLEncoder.encode() does NOT work for URIs.
Let us take your original URL:
http://site-test.test.com/Meetings/IC/DownloadDocument?meetingId=c21c905c-8359-4bd6-b864-844709e05754&itemId=a4b724d1-282e-4b36-9d16-d619a807ba67&file=\s604132shvw140\Test-Documents\c21c905c-8359-4bd6-b864-844709e05754_attachments\7e89c3cb-ce53-4a04-a9ee-1a584e157987\myDoc.pdf
and convert it to a URI template with two variables (on multiple lines for clarity):
http://site-test.test.com/Meetings/IC/DownloadDocument
?meetingId={meetingID}&itemId={itemID}&file={file}
Now let us build a variable map with these three variables using the library mentioned in the link:
final VariableMap = VariableMap.newBuilder()
.addScalarValue("meetingID", "c21c905c-8359-4bd6-b864-844709e05754")
.addScalarValue("itemID", "a4b724d1-282e-4b36-9d16-d619a807ba67e")
.addScalarValue("file", "\\\\s604132shvw140\\Test-Documents"
+ "\\c21c905c-8359-4bd6-b864-844709e05754_attachments"
+ "\\7e89c3cb-ce53-4a04-a9ee-1a584e157987\\myDoc.pdf")
.build();
final URITemplate template
= new URITemplate("http://site-test.test.com/Meetings/IC/DownloadDocument"
+ "meetingId={meetingID}&itemId={itemID}&file={file}");
// Generate URL as a String
final String theURL = template.expand(vars);
This is GUARANTEED to return a fully functional URL!
Thanks to Erhun's answer I finally realised that my JSON mapper was returning the quotation marks around my data too! I needed to use "asText()" instead of "toString()"
It's not an uncommon issue - one's brain doesn't see anything wrong with the correct data, surrounded by quotes!
discoveryJson.path("some_endpoint").toString();
"https://what.the.com/heck"
discoveryJson.path("some_endpoint").asText();
https://what.the.com/heck
This code worked for me
public static void main(String[] args) {
try {
java.net.URL url = new java.net.URL("http://path");
System.out.println("Instantiated new URL: " + url);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
}
Instantiated new URL: http://path
Very simple fix
String encodedURL = UriUtils.encodePath(request.getUrl(), "UTF-8");
Works no extra functionality needed.