I am downloading a large file from a third party URL using Spring RestTemplate . Im using the below method to download the file so as not to load the entirety into memory.
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor
This works fine, however we intermittently get a Premature end of Content-Length delimited message body
I read from another stack overflow post that this is a server side issue. How can I make sure it's a server side issue? How can I know if it's a memory issue on the third party side or incorrect content length in the response header and such? I need to make sure before I approach the third party for a resolution.
Depending how easy it is to trigger the error, you can use a tool like curl to call the URL (just for testing, outside your application). If you still see the failure, then it is definitely not because of the client, because those well-known third-party tools are well tested and do not have such bugs.
Related
i'm creating java module to parse JSON file.
To receive file i need to send HTTP request. When I use curl my request looks like this:
curl -X GET "https://***" -H "accept: application/json" -H "apikey: ***"
How can I send the equivalent HTTP request from Java
Java has a lot of options to work with HTTP.
Option 1
Since Java 9, there is a built-in HTTP client. So You can use it to create a request without any third-party libraries.
A simple example is something like this:
HttpRequest request2 = HttpRequest.newBuilder()
.uri(new URI("some url"))
.header("someHeader", "value1")
.header("anotherHeader", "value2")
.GET()
.build();
For more examples see here
Option 2
Use third party libraries, there are many: OkHttpClient, More "old-school" Apache Http Client (HttpComponents
Option 3
If you're using spring, you might consider using Spring's WebClient. There are also wrappers in spring like RestTemplate that can come handy, but it really depends on what would you like to work with.
Many clients are coming with http connection pools that should be properly set up.
In addition, in your example, I see that you work with https - all these clients support it but it should be properly set up.
If you are using Spring then try WebClient - it is a bit harder to understand in the begging (at least harder than RestTemplate) but it pays of since RestTemplate will be discontinued.
You can find an example here
https://www.baeldung.com/spring-webclient-resttemplate
#GetMapping(value = "/tweets-non-blocking",
produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<Tweet> getTweetsNonBlocking() {
log.info("Starting NON-BLOCKING Controller!");
Flux<Tweet> tweetFlux = WebClient.create()
.get()
.uri(getSlowServiceUri())
.retrieve()
.bodyToFlux(Tweet.class);
tweetFlux.subscribe(tweet -> log.info(tweet.toString()));
log.info("Exiting NON-BLOCKING Controller!");
return tweetFlux;
}
Just be aware that this is non-blocking (e.g. asynchronous) solution so you won't get the response right away, but you subscribe to the request and then process the response when it is available. There are also blocking options in WebClient
Java has its own classes that allow you to send HTTP request. See class HttpURLConnection. However, I recommend using 3d party libraries that significantly simplify this task. Good libraries would be Apache Http client or OK Http client. I also can offer you to use another Open source library that has an HTTP client as well. It is called MgntUtils library and it is written by me. In this case your code would look something like this:
HttpClient workingClient = new HttpClient();
workingClient.setRequestProperty("accept", "application/json;charset=UTF-8");
workingClient.setRequestProperty("apikey", "***");
workingClient.setConnectionUrl("https://***");
ByteBuffer buffer =
workingClient.sendHttpRequestForBinaryResponse(HttpMethod.GET);
//or of your API returns contents of file as a string
String jsonStr = workingClient.sendHttpRequest(HttpMethod.GET);
After that, your ByteBuffer buffer or String jsonStr will hold the content of your JSON file. And now you can do whatever you need with it. Here is Javadoc for HttpClient class. The MgntUtils library can be obtained as maven artifacts here or on Github (including source code and Javadoc)
I'm new to Java and found a confusing behaviour related with RestTemplate.
It happened with an API returning large body (~5MB) over a quite slow network condition. The code is like below
ResponseEntity<MyEntity[]> result = restTemplate.exchange(url, HttpMethod.GET, entity, MyEntity[].class);
And also a ClientHttpRequestInterceptor is set to log before and after the request.
The confusing thing is that the after request log is logged only a while after remote server giving the response, and the HTTP Status code can be print in the log.
But the above statement took much more time to finally receive the data. Look inside the thread stack, it was reading data from socket.
I also look inside the resttemplate class and found:
response = request.execute();
handleResponse(url, method, response);
if (responseExtractor != null) {
return responseExtractor.extractData(response);
}
It seems to extractData after the execute().
My doubt is:
How does the client side know the status code even before get all the data? It just extracts necessary fields from the top packets?
Since the server has already sent out the response, where the response data is stored during the process?
It stores the data that it receives from the underlying HTTP in memory.
Client side can know what's the status code because with HTTP you get the headers and status code first before the response body. But this doesn't matter with RestTemplate as it promises to give you an object of ResponseEntity in the end, which contains everything from the http response be it status codex headers or body.
RestTemplate is an abstraction over an HttpClient, most client's give you the option to implement callbacks for separate events like onHeadersReceived(), onStatusReceived() etc. But if you are using RestTemplate this means you don't require such fine grained control.
Simple restTemplate exchange used for a GET request.
Header information
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.ALL));
headers.setContentType(MediaType.APPLICATION_JSON_UTF8);
headers.add("Authorization", ************);
URL:
/api/odata/GetAvailableObjects?fromTime=datetime'2018-02-02T12:00:00'&$format=json&toTime=datetime'2018-02-10T12:00:00'
I tried building this URL with either MessageFormat.format or URIBuilder.
The single quotes are the problem. So far I tried double-ing them, escaping them
or leaving them as they are.
I enabled some extra logging with :
logging.level.org.springframework.web.client.RestTemplate=DEBUG
This displays the URL.. if I copy that URL and paste it in POSTMAN, I receive results, from spring I receive 400 Bad request.
I suppose it might be some additional encoding to blame.
Any ideas?
The JWT had way to many scopes, the server API decided to throw a 400 without any message and the server logs were not available..
A flag in application.properties that sets the maximum header size.
I'm trying to make a little utility that will synchronise data between two servers. Most of the calls there are REST calls with JSON, so I decided to use Apache HttpClient for this.
There is however a section where I need to upload a file. I'm trying to do this using the mutipart form data with the MutipartEntityBuilder but I encounter a Content too long problem. (I tried to gzip the contents of the file too, but I'm still going over the limit).
Here's my java code:
HttpPost request = new HttpPost(baseUrl+URL);
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
//create upload file params
builder.addTextBody("scanName", "Test upload");
builder.addBinaryBody("myfile", f);
HttpEntity params= builder.build();
request.setEntity(params);
request.addHeader("content-type","multipart/form-data");
HttpResponse response = httpClient.execute(request);
Are there better atlernatives that I should be using for the file upload part? I'm also going to download the files from one of the server. Will I hit a similar issue when try to handle those responses?
Is there something I'm doing wrong?
I try to use your code and send some file with size something about 33MB and it was successful. So, I think your problem one of the follows:
Created http client has limitations for request size - in this case you need to change properties of client or use another client;
In some peace of code you call HttpEntity.getContent() method. For multipart request for this method exists limitations - 25kB. For this case you need to use writeTo(OutputStream) instead of getContent()
In comments you told about swagger, but I don't understand what does it mean. If you use swagger generated api, that problems maybe occurred at their code and you need to fix generation logic (or something like this - I never used swagger)
I hope my answer will help you
I am trying to make a Http POST request using apache HTTP client. I am trying to copy contents of an HTTP POST request (received at my application) to another HTTP POST request (initiated from my application to another URL). Code is shown below:
httpPost = new HttpPost(inputURL);
// copy headers
for (Enumeration<String> e = request.getHeaderNames(); e.hasMoreElements();) {
String headerName = e.nextElement().toString();
httpPost.setHeader(headerName, request.getHeader(headerName));
}
BufferedInputStream clientToProxyBuf = new BufferedInputStream(request.getInputStream());
BasicHttpEntity basicHttpEntity = new BasicHttpEntity();
basicHttpEntity.setContent(clientToProxyBuf);
basicHttpEntity.setContentLength(clientToProxyBuf.available());
httpPost.setEntity(basicHttpEntity);
HttpResponse responseFromWeb = httpclient.execute(httpPost);
Basically, I am trying to implement a proxy application which will get a url as parameter, froward the request to the URL and then serve pages etc in custom look and feel.
Here request is HttpServletRequest. I am facing problem in setting content length. Through debugging I found out that clientToProxyBuf.available() is not giving me correct length of input stream and I am getting Http error 400 IE and Error 354 (net::ERR_CONTENT_LENGTH_MISMATCH): The server unexpectedly closed the connection in chrome.
Am I doing it wrong? Is there any other way to achieve it?
The available() function doesn't provide the actual length of the content of the stream, rather
Returns the number of bytes that can be read from this input stream without blocking. (From javadoc)
I would suggest you to first read the whole content from the stream, and then set that to the content, rather than passing the stream object. That way, you will also have the actual length of the content.
It was rather simple and very obvious. I just needed to get content length from header as:
basicHttpEntity.setContentLength(Integer.parseInt(request.getHeader("Content-Length")));