Develop Reference

Problem with RestTemplate when including URL addresses in REST API request

I'm trying to obtain data from Botify's REST API to use it inside a project, which is also a REST API. I'm using an instance of Spring's RestTemplate class to make the actual requests to Botify, specifically the .exchange method as I need to pass Botify's key as a header parameter.
My problem comes when I need to call to a method of the endpoint which takes a URL as a part of the request's URI (not a parameter). Documentation of this endpoint is in https://developers.botify.com/api/reference/#!/Analysis/getUrlDetail
Basically the structure of the requests is like this:
/analyses/{username}/{project_slug}/{analysis_slug}/urls/{url}
The last part of that URI is a URL address, which needs to be encoded in UTF-8 to make it possible to separate it from the actual request.
The problem is (I believe) that the .exchange method always encodes the request, so what I try to send like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com
...ends up like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%253A%252F%252Fwww.example.com'
Which obviously doesn't work. This is an excerpt from the method that makes the call to Botify:
public String callBotifyEndpoint(String reportType, String parameters) throws UnsupportedEncodingException {
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com"
HttpHeaders headers = new HttpHeaders();
headers.set("Authorization", "Token " + this.apiKey);
HttpEntity<String> entity = new HttpEntity<>(headers);
UriComponentsBuilder botifyQueryBuilder = UriComponentsBuilder.fromUriString(request);
String queryStringBuild = botifyQueryBuilder.build(true).toUriString();
String botifyResult = null;
try {
System.out.println("Calling Botify API: " + queryStringBuild);
ResponseEntity<String> response = botifyTemplate.exchange(queryStringBuild, HttpMethod.GET, entity, String.class);
if(response.hasBody()) {
botifyResult = response.getBody();
}
} catch(RestClientException ex) {
ex.printStackTrace();
}
try {
} catch (Exception e) {
// TODO: handle exception
}
return botifyResult;
}
In this line:
botifyQueryBuilder.build(true).toUriString();
The "true" parameter indicates whether the data is already encoded or not. I've tried to disable it but the result is the same.
I've removed actual request generation process (along with my user and project's name) to simplify things, but this should return a response from Botify with the existing data for that URL.
Instead, it returns a 400 bad request error (which makes sense, because the URL is not correct).
I'm feeling like this may be a bug in RestTemplate's .exchange method, but maybe I'm not using it properly. Any suggestions?

Don't encode prematurly as you do here:
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com";
Use parameter placeholders feature in RestTemplate instead of text concatenation.
Refer to:
Spring RestTemplate GET with parameters

Best practice to log out everything related to HttpRequest and HttpResponse

Im executing a http request like this:
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpRequest = new HttpGet("http://localhost:8080/test");
HttpResponse httpResponse = httpClient.execute(httpRequest);
What is the best way to log out everything about the request/response? I mean like what method was used for request and what url and all the headers and response code and response message body and so on.
Edit:
At the moment i worked out something like this:
private static void logHttpRequest(HttpMessage httpMessage) {
StringBuilder builder = new StringBuilder();
addHeaders(httpMessage, builder);
builder.append(httpMessage.toString() + "\n");
System.out.println("Request:\n" + builder.toString());
}
private static void logHttpResponse(HttpResponse httpResponse) throws IOException {
StringBuilder builder = new StringBuilder();
builder.append(httpResponse.getStatusLine() + "\n");
addHeaders(httpResponse, builder);
builder.append(EntityUtils.toString(httpResponse.getEntity()));
System.out.println("Response:\n" + builder.toString() + "\n");
}
private static void addHeaders(HttpMessage httpMessage, StringBuilder builder) {
Header[] headers = httpMessage.getAllHeaders();
if (headers != null) {
for (Header header : headers) {
builder.append(header.toString() + "\n");
}
}
}
Is there anything else that should be logged? Can anything be done better/more optimal?

I think this primarily depends on context and your requirements. Keep in mind though, that huge amounts of logging is also not smart. For example, you should log exceptions at a high log level, eg. SEVERE or ERROR (depends on the logging framework you're using). However, messages like, "trying to make a GET request to http://soAndSo/whatevz" could do with lower log level of, say DEBUG, so you could switch your code to log DEBUG level messages when something bad happens in production.
Something I have typically seen people do with the response is that they definitely log ERROR if there's no response at all but when there is a response, it's imperative to check the http status code of the response and if it's not what you expect it to be (typically 200/2xx), you log the response body, headers, etc at ERROR level.
When making POST or PUT requests with some body, try and see if it's useful to log the request body and headers if any at, say, INFO or DEBUG level. These things sometimes help in tracing out failure scenarios or lost requests/lost data.
At the consuming end too, your system should ideally have a monitor on the no.of/rate of ERROR level messages otherwise you'll come to know of something very bad only after it happens, which is not favourable.

Unable to add product details while creating new quote

I am developing an android app to create a new Quote in vTiger(ver 5.4) CRM server.
I was able to generate the new quote but the product_id and quantity that I sent for addition in quote details were not added in it. The other details are being shown in the new quote except the list of products, their quantities and pricing.
I have also studied the vTiger webservices tutorial but it was not helpful in this case.
I found an accepted answer of similar question but it is in php not in Android/JAVA.
This is how I am sending the details required to create a new quote in vTiger server.:-
try {
objectJson.put("subject", subject);
objectJson.put("account_id", accountId);
objectJson.put("bill_street", address);
objectJson.put("assigned_user_id", "19x1");
objectJson.put("conversion_rate", "1.000");
objectJson.put("currency_id", "21x1");
objectJson.put("hdnTaxType", "group");
objectJson.put("productid", productId);
objectJson.put("quantity", quantity);
}
catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String data = null;
try {
data = URLEncoder.encode("sessionName", "UTF-8")
+ "=" + URLEncoder.encode(sessionId, "UTF-8");
data += "&" + URLEncoder.encode("element", "UTF-8") + "="
+ URLEncoder.encode(objectJson.toString(), "ISO-8859-1");
data += "&" + URLEncoder.encode("elementType", "UTF-8") + "="
+ URLEncoder.encode(moduleName, "UTF-8"); //moduleName='Quotes'
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String text = "";
BufferedReader reader=null;
// Send data
try
{
// Defined URL where to send data
URL url = new URL("http://vtiger_url/webservice.php?operation=create");
// Send POST data request
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write( data );
wr.flush();
}
catch(Exception ex)
{
}
The above code helps me to generate the quote without product details.
After studying the above mentioned php answer, I changed the URL that I was using in my code to this:- http://vtiger_url/webservice.php?total=23000&operation=create. This helped me to add the total amount to the newly created quote but I wasn't successful to add rest of the details using this method.

The answer you have found seems to be suitable, even if it's a php solution. The problem you descirbed:
I have tried this too but unfortunately I am not able to access Inventoryproductrel table.
maybe already indicates a lack of proper authentication or missing privileges. So my suggestion is to
Find out why you can't access the table Inventoryproductrel. Try to access that table using your authentication privileges (sessionID), examine the response for any hints and try to solve the table isn't accessible issue.
Finally follow the suggestions from this answer.
Just another hint. If you can successfully access that table via you web browser, than I would sniff the request and have a look at the http parameters and their values. Based on those findings you can modify your request.
Always consider, the web browser can only do the same as you android application.

These lines were added in existing code:-
JSONArray pdoInformation = new JSONArray();
try{
// Added these lines in try block to send product details
for(int i=0; i<productIds.size(); i++) {
JSONObject obj = new JSONObject();
obj.put("productid", productIds.get(i) );
obj.put("qty", quantities.get(i));
obj.put("listprice", listprices.get(i));
pdoInformation.put(obj);
}
objectJson.put("pdoInformation", pdoInformation);
}
Here product details were needed to be sent in a JSONArray with name as "pdoInformation".
for loop is used to send multiple product details.
Here productIds, quantities and listprices are three mandatory product details stored as ArrayList.

Why don't you use the web service to create products as well? That should be supported as per the documentation.
Once you create the products and get their ids, you can create a quote object that includes these ids. The objects and their fields are not very well documented for the rest APIs, so you could use the query/describe APIs to get as much information as possible about what data needs to be supplied for creating the different objects.
From the description of the Quotes module, you would need to include item_details which will contain the product and quantity information. The exact field name and format can be obtained by the describe API as described in the web service documentation
To get a description of the vTiger objects
String modeleName = "Quotes"; //Use Products if you want a description of the Products module
String data = null;
try {
data = URLEncoder.encode("sessionName", "UTF-8")
+ "=" + URLEncoder.encode(sessionId, "UTF-8");
data += "&" + URLEncoder.encode("elementType", "UTF-8") + "="
+ URLEncoder.encode(moduleName, "UTF-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
String text = "";
BufferedReader reader=null;
System.out.println(data);
// Send data
try
{
// Defined URL where to send data
URL url = new URL("http://vtiger_url/webservice.php?operation=describeobject");
// Send GET data request
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write( data );
wr.flush();
} catch(Exception ex) {
}
Note that it's pretty much the same as what you are doing, only the url & parameters are different, and the request type is GET instead of POST
To create a product, you would follow the same procedure as you did for the quote, only the url and parameters would be different

#help as far i understand your question is how to send JsonObject with data to specified Server is that correct ? if it is the correct then i suggest you to use Volley library for networking and here are many examples that may useful to you .http://arnab.ch/blog/2013/08/asynchronous-http-requests-in-android-using-volley/ and http://www.androidhive.info/2014/05/android-working-with-volley-library-1/
Just go through it. It provides easiest way to perform networking operations and also cancellation of request is also possible with this library.
Sample Code:
final String URL = "SERVER_URL";
// Post params to be sent to the server
HashMap<String, String> params = new HashMap<String, String>();
params.put("token", "AbCdEfGh123456");
JsonObjectRequest req = new JsonObjectRequest(URL, new JSONObject(params),
new Response.Listener<JSONObject>() {
#Override
public void onResponse(JSONObject response) {
try {
VolleyLog.v("Response:%n %s", response.toString(4));
} catch (JSONException e) {
e.printStackTrace();
}
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
VolleyLog.e("Error: ", error.getMessage());
}
});
// add the request object to the queue to be executed
ApplicationController.getInstance().addToRequestQueue(req);
Using this code you can send your data to your server and at server side you can use php code to receive this jsondata and parse it.
Thanks.

Get URL content with Basic Authentication with Java and async-http-client

I am writing a Java lib and need to perform a request to a URL - currently using async-http-client from ning - and fetch its content. So I have a get method that returns a String
of the content of the fetched document. However, to be able to get it, I must perform a HTTP basic authentication and I'm not succeeding at this in my Java code:
public String get(String token) throws IOException {
String fetchURL = "https://www.eventick.com.br/api/v1/events/492";
try {
String encoded = URLEncoder.encode(token + ":", "UTF-8");
return this.asyncClient.prepareGet(fetchURL)
.addHeader("Authorization", "Basic " + encoded).execute().get().getResponseBody();
}
}
The code returns no error, it just doesn't fetch the URL because the authentication header is not being properly set, somehow.
With curl -u option I can easily get what I want:
curl https://www.eventick.com.br/api/v1/events/492 -u 'xxxxxxxxxxxxxxx:'
Returns:
{"events":[{"id":492,"title":"Festa da Bagaceira","venue":"Mangueirão de Paulista",
"slug":"bagaceira-fest", "start_at":"2012-07-29T16:00:00-03:00",
"links":{"tickets":[{"id":738,"name":"Normal"}]}}]}
How can this be done in Java? With the async-http-client lib? Or if you know how to do it using another way..
Any help is welcome!

You're close. You need to base 64 encode rather than URL encode. That is, you need
String encoded = Base64.getEncoder().encodeToString((user + ':' + password).getBytes(StandardCharsets.UTF_8));
rather than
String encoded = URLEncoder.encode(token + ":", "UTF-8");
(Note that for the benefit of others, since I'm answering 2 years later, in my answer I'm using the more standard "user:password" whereas your question has "token:". If "token:" is what you needed, then stick with that. But maybe that was part of the problem, too?)
Here is a short, self-contained, correct example
package so17380731;
import com.ning.http.client.AsyncHttpClient;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import javax.ws.rs.core.HttpHeaders;
public class BasicAuth {
public static void main(String... args) throws Exception {
try(AsyncHttpClient asyncClient = new AsyncHttpClient()) {
final String user = "StackOverflow";
final String password = "17380731";
final String fetchURL = "https://www.eventick.com.br/api/v1/events/492";
final String encoded = Base64.getEncoder().encodeToString((user + ':' + password).getBytes(StandardCharsets.UTF_8));
final String body = asyncClient
.prepareGet(fetchURL)
.addHeader(HttpHeaders.AUTHORIZATION, "Basic " + encoded)
.execute()
.get()
.getResponseBody(StandardCharsets.UTF_8.name());
System.out.println(body);
}
}
}

The documentation is very sketchy, but I think that you need to use a RequestBuilder following the pattern shown in the Request javadoc:
Request r = new RequestBuilder().setUrl("url")
.setRealm((new Realm.RealmBuilder()).setPrincipal(user)
.setPassword(admin)
.setRealmName("MyRealm")
.setScheme(Realm.AuthScheme.DIGEST).build());
r.execute();
(Obviously, this example is not Basic Auth, but there are clues as to how you would do it.)
FWIW, one problem with your current code is that a Basic Auth header uses base64 encoding not URL encoding; see the RFC2617 for details.

basically, do it like this:
BoundRequestBuilder request = asyncHttpClient
.preparePost(getUrl())
.setHeader("Accept", "application/json")
.setHeader("Content-Type", "application/json")
.setRealm(org.asynchttpclient.Dsl.basicAuthRealm(getUser(), getPassword()))
// ^^^^^^^^^^^-- this is the important part
.setBody(json);
Test can be found here:
https://github.com/AsyncHttpClient/async-http-client/blob/master/client/src/test/java/org/asynchttpclient/BasicAuthTest.java

This is also another way of adding Basic Authorization,
you can use any of two the classes for your use AsyncHttpClient,HttpClient,in this case i will use AsyncHttpClient
AsyncHttpClient client=new AsyncHttpClient();
Request request = client.prepareGet("https://www.eventick.com.br/api/v1/events/492").
setHeader("Content-Type","application/json")
.setHeader("Authorization","Basic b2pAbml1LXR2LmNvbTpnMGFRNzVDUnhzQ0ZleFQ=")
.setBody(jsonObjectRepresentation.toString()).build();
after adding header part
ListenableFuture<Response> r = null;
//ListenableFuture<Integer> f= null;
try{
r = client.executeRequest(request);
System.out.println(r.get().getResponseBody());
}catch(IOException e){
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
client.close();
it may be useful for you

Async urlfetch Http post on App engine using Future

My goal is to rapidly make posts to a server from appengine(java). I am attempting to do this using UrlFetchService.fetchAsync. I have been basing my code after this blog post. I have been able to make the request using the code below, however I get some strange behavior:
private void futureRequests() {
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
URL url = new URL("https://someserver.com");
FetchOptions fetchOptions = FetchOptions.Builder.withDefaults();
fetchOptions.doNotValidateCertificate();
fetchOptions.setDeadline(60D);
ArrayList<Future<HTTPResponse>> asyncResponses = new ArrayList<Future<HTTPResponse>>();
for (int i = 0; i < postDatas.size(); i++) {
HTTPRequest request = new HTTPRequest(url, HTTPMethod.POST, fetchOptions);
request.setPayload(postDatas.get(i).getBytes(UTF8));
HTTPHeader header = new HTTPHeader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
request.setHeader(header);
header = new HTTPHeader("Content-Length", Integer.toString(postDatas.get(i).getBytes().length));
request.setHeader(header);
header = new HTTPHeader("Authorization", "auth=" + authToken);
request.setHeader(header);
Future<HTTPResponse> responseFuture = fetcher.fetchAsync(request);
asyncResponses.add(responseFuture);
}
for (Future<HTTPResponse> future : asyncResponses) {
HTTPResponse response;
try {
response = future.get();
int responseCode = response.getResponseCode();
resp.getWriter().println("response: " + responseCode);
logger.warning("Response: " + responseCode);
} catch (Exception e) {
}
}
}
The strange behavior is that I get duplicate posts on the server, and according to my appstats page I use 10x-20x more urlFetches than what was added with the code above. Below is my appstats screen:
There are more urlFetch calls that could not fit on the screen. It appears that the requests are still completing in a synchronous fashion(circled items), but there are many urlFetches that appear to go on at the same time. My question is how am I getting all this calls to urlFetch when I only had 14 Future ?? Could the server be giving an error or 503 and urlFetch retrying until it goes through? And how can I be getting 2 posts for each request??
I understand that I could use the task queue to do asyc request, however I am dealing with a relatively low number of request(20-100) and the cold start time of ramping up another instance would probably make this not a good option for my situation. Can anyone explain this behavior or have experience with this?

This was simply a mistake in my code that was causing my app to make more request than I thought..