i'm creating java module to parse JSON file.
To receive file i need to send HTTP request. When I use curl my request looks like this:
curl -X GET "https://***" -H "accept: application/json" -H "apikey: ***"
How can I send the equivalent HTTP request from Java
Java has a lot of options to work with HTTP.
Option 1
Since Java 9, there is a built-in HTTP client. So You can use it to create a request without any third-party libraries.
A simple example is something like this:
HttpRequest request2 = HttpRequest.newBuilder()
.uri(new URI("some url"))
.header("someHeader", "value1")
.header("anotherHeader", "value2")
.GET()
.build();
For more examples see here
Option 2
Use third party libraries, there are many: OkHttpClient, More "old-school" Apache Http Client (HttpComponents
Option 3
If you're using spring, you might consider using Spring's WebClient. There are also wrappers in spring like RestTemplate that can come handy, but it really depends on what would you like to work with.
Many clients are coming with http connection pools that should be properly set up.
In addition, in your example, I see that you work with https - all these clients support it but it should be properly set up.
If you are using Spring then try WebClient - it is a bit harder to understand in the begging (at least harder than RestTemplate) but it pays of since RestTemplate will be discontinued.
You can find an example here
https://www.baeldung.com/spring-webclient-resttemplate
#GetMapping(value = "/tweets-non-blocking",
produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<Tweet> getTweetsNonBlocking() {
log.info("Starting NON-BLOCKING Controller!");
Flux<Tweet> tweetFlux = WebClient.create()
.get()
.uri(getSlowServiceUri())
.retrieve()
.bodyToFlux(Tweet.class);
tweetFlux.subscribe(tweet -> log.info(tweet.toString()));
log.info("Exiting NON-BLOCKING Controller!");
return tweetFlux;
}
Just be aware that this is non-blocking (e.g. asynchronous) solution so you won't get the response right away, but you subscribe to the request and then process the response when it is available. There are also blocking options in WebClient
Java has its own classes that allow you to send HTTP request. See class HttpURLConnection. However, I recommend using 3d party libraries that significantly simplify this task. Good libraries would be Apache Http client or OK Http client. I also can offer you to use another Open source library that has an HTTP client as well. It is called MgntUtils library and it is written by me. In this case your code would look something like this:
HttpClient workingClient = new HttpClient();
workingClient.setRequestProperty("accept", "application/json;charset=UTF-8");
workingClient.setRequestProperty("apikey", "***");
workingClient.setConnectionUrl("https://***");
ByteBuffer buffer =
workingClient.sendHttpRequestForBinaryResponse(HttpMethod.GET);
//or of your API returns contents of file as a string
String jsonStr = workingClient.sendHttpRequest(HttpMethod.GET);
After that, your ByteBuffer buffer or String jsonStr will hold the content of your JSON file. And now you can do whatever you need with it. Here is Javadoc for HttpClient class. The MgntUtils library can be obtained as maven artifacts here or on Github (including source code and Javadoc)
Related
I am downloading a large file from a third party URL using Spring RestTemplate . Im using the below method to download the file so as not to load the entirety into memory.
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor
This works fine, however we intermittently get a Premature end of Content-Length delimited message body
I read from another stack overflow post that this is a server side issue. How can I make sure it's a server side issue? How can I know if it's a memory issue on the third party side or incorrect content length in the response header and such? I need to make sure before I approach the third party for a resolution.
Depending how easy it is to trigger the error, you can use a tool like curl to call the URL (just for testing, outside your application). If you still see the failure, then it is definitely not because of the client, because those well-known third-party tools are well tested and do not have such bugs.
I want to use HttpURLConnection class in Groovy to send GET and POST (with Jsonbody) request to an api. But what can I tell you. With HttpURLConnection it is soo difficult. I do not know how to use it. There is not even a send method. It looks like when you call getResponseCode() this method sends a request. Then you have to use InputStream and for POST you even have to use OutputStream. Oh my god. For what ?? I am used to handy libraries like Jersey Client. But this HttpURLConnection is just a nightmare. I have to use it because I have jenkinsfile and in my pipeline I need to upload something. And that is possible with calling an REST Api. What do you think about HttpURLConnection ? Does someone know a good website with a GET Request and a POST Request with a body.
I think this is what you are trying to do https://www.baeldung.com/httpurlconnection-post
Making it Groovy should be pretty trivial.
If you are doing an HTTP request and you want to use vanilla Java or Groovy, then I would recommend using HttpClient and HttpRequest.Builder; it's somewhat fluid. And you supply a BodyHandler object, which can be used to get the content however you like it (String, JSON Object, whatever).
I have a few questions about a specific REST call I'm making in JAVA. I'm quite the novice, so I've cobbled this together from several sources. The call itself looks like this:
String src = AaRestCall.subTrackingNum(trackingNum);
The Rest call class looks like this:
public class AaRestCall {
public static String subTrackingNum (Sting trackingNum) throws IOException {
URL url = new URL("https://.../rest/" + trackingNum);
String query = "{'TRACKINGNUM': trackingNum}";
//make connection
URLConnection urlc = url.openConnection();
//use post mode
urlc.setDoOutput(true);
urlc.setAllowUserInteraction(false);
//send query
PrintStream ps = new PrintStream(urlc.getOutputStream());
ps.print(query);
ps.close();
//get result
BufferedReader br = new BufferedReader(new InputStreamReader(urlc
.getInputStream()));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line=br.readLine())!=null) {
sb.append(line);
}
br.close();
return sb.toString();
}
}
Now, I have a few questions on top of the what is wrong with this in general.
1) If this rest call is returning a JSON object, is that going to get screwed up by going to a String?
2) What's the best way to parse out the JSON that is returning?
3) I'm not really certain how to format the query field. I assume that's supposed to be documented in the REST API?
Thanks in advance.
REST is a pattern applied on top of HTTP. From your questions, it seems to me that you first need to understand how HTTP (and chatty socket protocols in general) works and what the Java API offers for deal with it.
You can use whatever Json library out there to parse the HTTP response body (provided it's a 200 OK, that you need to check for, and also watch out for HTTP redirects!), but it's not how things are usually built.
If the service exposes a real RESTful interface (opposed to a simpler HTTP+JSON) you'll need to use four HTTP verbs, and URLConnection doesn't let you do so. Plus, you'll likely want to add headers for authentication, or maybe cookies (which in fact are just HTTP headers, but are still worth to be considered separately). So my suggestion is building the client-side part of the service with the HttpClient from Apache commons, or maybe some JAX-RS library with client support (for example Apache CXF). In that way you'll have full control of the communication while also getting nicer abstractions to work with, instead of consuming the InputStream provided by your URLConnection and manually serializing/deserializing parameters/responses.
Regarding the bit about how to format the query field, again you first need to grasp the basics of HTTP. Anyway, the definite answer depends on the remote service implementation, but you'll face four options:
The query string in the service URL
A form-encoded body of your HTTP request
A multipart body of your HTTP request (similar to the former, but the different MIME type is enough to give some headache) - this is often used in HTTP+JSON services that also have a website, and the same URL can be used for uploading a form that contains a file input
A service-defined (for example application/json, or application/xml) encoding for your HTTP body (again, it's really the same as the previous two points, but the different MIME encoding means that you'll have to use a different API)
Oh my. There are a couple of areas where you can improve on this code. I'm not even going to point out the errors since I'd like you to replace the HTTP calls with a HTTP client library. I'm also unaware of the spec required by your API so getting you to use the POST or GET methods properly at this level of abstraction will take more work.
1) If this rest call is returning a JSON object, is that going to get
screwed up by going to a String?
No, but marshalling that json into an obect is your job. A library like google gson can help.
2) What's the best way to parse out the JSON that is returning?
I like to use gson like I mentioned above, but you can use another marshal/unmarhal library.
3) I'm not really certain how to format the query field. I assume
that's supposed to be documented in the REST API?
Yes. Take a look at the documentation and come up with java objects that mirror the json structure. You can then parse them with the following code.
gson.fromJson(json, MyStructure.class);
Http client
Please take a look at writing your HTTP client using a library like apache HTTP client which will make your job much easier.
Testing
Since you seem to be new to this, I'd also suggest you take a look at a tool like Postman which can help you test your API calls if you suspect that the code you've written is faulty.
I think that you should use a REST client library instead of writing your own, unless it is for educational purposes - then by all means go nuts!
The REST service will respond to your call with a HTTP response, the payload may and may not be formatted as a JSON string. If it is, I suggest that you use a JSON parsing library to convert that String into a Java representation.
And yes, you will have to resort to the particular REST API:s documentation for details.
P.S. The java URL class is broken, use URI instead.
I am somewhat new to HTTP REST operations on Android, and a server I am working with uses PUT commands to process updates. I am having a difficult time using Spring (for Android) with Jackson2. The server doesn't seem to work with application/json put requests (though it will reply with them), and only seem to work with application/x-www-form-urlencoded versions (tested with python and curl. On python, if I set the header type to application/json, it fails.
I am using the latest versions of Spring and Jackson2, and I know everything is setup properly because my get request on the same URL gets me all the correct information.
I am using Robospice, but I don't really think that is relevant. Here is my request code.
#Override
public GPIO loadDataFromNetwork() throws Exception {
String url = String.format("http://%s/api/control/gpio", ip);
RestTemplate rt = getRestTemplate();
DefaultHttpClient client = new DefaultHttpClient();
Credentials defaultcreds = new UsernamePasswordCredentials("admin",
password);
client.getCredentialsProvider().setCredentials(
new AuthScope(routerip, 80, AuthScope.ANY_REALM), defaultcreds);
// Makes authentication work.
rt.setRequestFactory(new HttpComponentsClientHttpRequestFactory(client));
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
HttpEntity request = new HttpEntity(data, requestHeaders);
List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>();
MappingJackson2HttpMessageConverter map = new MappingJackson2HttpMessageConverter();
messageConverters.add(map);
messageConverters.add(new FormHttpMessageConverter());
rt.setMessageConverters(messageConverters);
ResponseEntity<GPIO> r = rt.exchange(url, HttpMethod.PUT, request, GPIO.class);
return r.getBody();
}
I am getting the exception stating it cannot find a way to convert:
02-01 10:59:29.466: E//DefaultRequestRunner.java:138(30086):
10:59:29.474 Thread-11651 An exception occurred during request network
execution :Could not write request: no suitable HttpMessageConverter
found for request type [com.xxxxx.control.gpio.GPIO] and content
type [application/x-www-form-urlencoded]
GPIO is my POJO object. I want to 'put' that to the server, as in serialize and put it.
I have looked at the following question that seems fairly relevant:
Deserializing Nested objects using RestTemplate
However, I need the result of my put command, and that requires me to use exchange() because Spring's put() returns nothing.
I have tried several different items (such as removing GPIO references, setting specific headres...) and none seem to work. I have a feeling this is probably an easy solution that I don't know how to fix. If anyone can help me that would be great.
TLDR: I'm using Spring for Android with Jackson2. I want to serialize my object (in this example, GPIO) so I can do a PUT request with the content type application/x-www-form-urlencoded. However I cannot get jackson to convert to that type, only to application/json, which does not work for me. I am not sure how to fix this, and I have run out of ideas. If I can't find a solution I'll probably have to dumb robospice. (or jackson, not sure which yet.)
Solution
Spring for Android doesn't seem to simplify things, so I dumped it and used the apache client directly in my loadDataFromNetwork() method. Robospice handles it pretty well and I can get the responses I need. If you are new to HTTP like I was take the time and learn the apache client, it's far easier in my opinion. Tweaking the ObjectMapper (like making a JsonTree and parsing that) made it much easier to get the data I needed without having to do as much work with POJO objects.
If you can format the data you want to send into MultiValueMap<String, String>, then a possible way around this is to use FormHttpMessageConverter.
FormHttpMessageConverter: (you can see examples in the link)
An HttpMessageConverter implementation that can read and write form
data from the HTTP request and response. By default, this converter
reads and writes the media type application/x-www-form-urlencoded.
Form data is read from and written into a MultiValueMap<String,String>.
After re-reading, here's a shot at a real answer - you are explicitly using the x-www-form-urlencoded content type by using this RequestHeader:
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
You should be using APPLICATION_JSON - Here's the Javadoc.
You should also consider specifying the charset and datatype in the headers. Jaxson is very specific about this, and if you don't have access to the server code you don't know what headers they expect.
dude i am using Loopj's AsyncHttpClient for rest and json dataset.
Here is the link below Here
Very simple and easy to understand. U can try this thing.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
What is the best Java library to use for HTTP POST, GET etc. in terms of performance, stability, maturity etc.? Is there one particular library that is used more than others?
My requirements are submitting HTTPS POST requests to a remote server. I have used the java.net.* package in the past as well as org.apache.commons.httpclient.* package. Both have got the job done, but I would like some of your opinions/recommendations.
imho: Apache HTTP Client
usage example:
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.params.HttpMethodParams;
import java.io.*;
public class HttpClientTutorial {
private static String url = "http://www.apache.org/";
public static void main(String[] args) {
// Create an instance of HttpClient.
HttpClient client = new HttpClient();
// Create a method instance.
GetMethod method = new GetMethod(url);
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
// Read the response body.
byte[] responseBody = method.getResponseBody();
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
System.out.println(new String(responseBody));
} catch (HttpException e) {
System.err.println("Fatal protocol violation: " + e.getMessage());
e.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}
}
}
some highlight features:
Standards based, pure Java, implementation of HTTP versions 1.0
and 1.1
Full implementation of all HTTP methods (GET, POST, PUT, DELETE,
HEAD, OPTIONS, and TRACE) in an
extensible OO framework.
Supports encryption with HTTPS (HTTP over SSL) protocol.
Granular non-standards configuration and tracking.
Transparent connections through HTTP proxies.
Tunneled HTTPS connections through HTTP proxies, via the CONNECT
method.
Transparent connections through SOCKS proxies (version 4 & 5) using
native Java socket support.
Authentication using Basic, Digest and the encrypting NTLM (NT
Lan Manager) methods.
Plug-in mechanism for custom authentication methods.
Multi-Part form POST for uploading large files.
Pluggable secure sockets implementations, making it easier to
use third party solutions
Connection management support for use in multi-threaded
applications. Supports setting the
maximum total connections as well as
the maximum connections per host.
Detects and closes stale connections.
Automatic Cookie handling for reading Set-Cookie: headers from the
server and sending them back out in a
Cookie: header when appropriate.
Plug-in mechanism for custom cookie policies.
Request output streams to avoid buffering any content body by
streaming directly to the socket to
the server.
Response input streams to efficiently read the response body by
streaming directly from the socket to
the server.
Persistent connections using KeepAlive in HTTP/1.0 and persistance
in HTTP/1.1
Direct access to the response code and headers sent by the server.
The ability to set connection timeouts.
HttpMethods implement the Command Pattern to allow for parallel
requests and efficient re-use of
connections.
Source code is freely available under the Apache Software License.
I would recommend Apache HttpComponents HttpClient, a successor of Commons HttpClient
I would also recommend to take a look at HtmlUnit.
HtmlUnit is a "GUI-Less browser for Java programs".
http://htmlunit.sourceforge.net/
I'm somewhat partial to Jersey. We use 1.10 in all our projects and haven't run into an issue we couldn't solve with it.
Some reasons why I like it:
Providers - created soap 1.1/1.2 providers in Jersey and have eliminated the need to use the bulky AXIS for our JAX-WS calls
Filters - created database logging filters to log the entire request (including the request/response headers) while preventing logging of sensitive information.
JAXB - supports marshaling to/from objects straight from the request/response
API is easy to use
In truth, HTTPClient and Jersey are very similar in implementation and API. There is also an extension for Jersey that allows it to support HTTPClient.
Some code samples with Jersey 1.x:
https://blogs.oracle.com/enterprisetechtips/entry/consuming_restful_web_services_with
http://www.mkyong.com/webservices/jax-rs/restful-java-client-with-jersey-client/
HTTPClient with Jersey Client:
https://blogs.oracle.com/PavelBucek/entry/jersey_client_apache_http_client
I agree httpclient is something of a standard - but I guess you are looking for options so...
Restlet provides a http client specially designed for interactong with Restful web services.
Example code:
Client client = new Client(Protocol.HTTP);
Request r = new Request();
r.setResourceRef("http://127.0.0.1:8182/sample");
r.setMethod(Method.GET);
r.getClientInfo().getAcceptedMediaTypes().add(new Preference<MediaType>(MediaType.TEXT_XML));
client.handle(r).getEntity().write(System.out);
See https://restlet.talend.com/ for more details
I want to mention the Ning Async Http Client Library. I've never used it but my colleague raves about it as compared to the Apache Http Client, which I've always used in the past. I was particularly interested to learn it is based on Netty, the high-performance asynchronous i/o framework, with which I am more familiar and hold in high esteem.
May I recommend you corn-httpclient. It's simple,fast and enough for most cases.
HttpForm form = new HttpForm(new URI("http://localhost:8080/test/formtest.jsp"));
//Authentication form.setCredentials("user1", "password");
form.putFieldValue("input1", "your value");
HttpResponse response = form.doPost();
assertFalse(response.hasError());
assertNotNull(response.getData());
assertTrue(response.getData().contains("received " + val));
maven dependency
<dependency>
<groupId>net.sf.corn</groupId>
<artifactId>corn-httpclient</artifactId>
<version>1.0.0</version>
</dependency>
Google HTTP Java Client looks good to me because it can run on Android and App Engine as well.