Resteasy docs does not explain who is responsible for closing streams passed to MultipartFormDataOutput. Let's consider the following example:
WebTarget target = ClientBuilder.newClient().target("url");
MultipartFormDataOutput formData = new MultipartFormDataOutput();
FileInputStream fis1 = new FileInputStream(new File("/path/to/image1"));
FileInputStream fis2 = new FileInputStream(new File("/path/to/image2"));
formData.addFormData("image", fis1, MediaType.APPLICATION_OCTET_STREAM_TYPE);
formData.addFormData("image", fis2, MediaType.APPLICATION_OCTET_STREAM_TYPE);
Entity<MultipartFormDataOutput> entity = Entity.entity(formData, MediaType.MULTIPART_FORM_DATA);
Response response = target.request().post(entity);
Will the fis1 and fis2 be closed by the resteasy or the user should take care of closing these streams?
I would suggest to use try-with-resource to be sure that they will be closed.
WebTarget target = ClientBuilder.newClient().target("url");
MultipartFormDataOutput formData = new MultipartFormDataOutput();
try(FileInputStream fis1 = new FileInputStream(new File("/path/to/image1")));
FileInputStream fis2 = new FileInputStream(new File("/path/to/image2")))
{
formData.addFormData("image", fis1, MediaType.APPLICATION_OCTET_STREAM_TYPE);
formData.addFormData("image", fis2, MediaType.APPLICATION_OCTET_STREAM_TYPE);
Entity<MultipartFormDataOutput> entity = Entity.entity(formData,
MediaType.MULTIPART_FORM_DATA);
Response response = target.request().post(entity);
}
So I can answer my question myself, hope someone will benefit from this.
Resteasy will close the passed stream. In my case, the InputStreamProvider will take care of closing the FileInputStream.
public void writeTo(InputStream inputStream, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException
{
LogMessages.LOGGER.debugf("Provider : %s, Method : writeTo", getClass().getName());
try
{
int c = inputStream.read();
if (c == -1)
{
httpHeaders.putSingle(HttpHeaderNames.CONTENT_LENGTH, Integer.toString(0));
entityStream.write(new byte[0]); // fix RESTEASY-204
return;
}
else
entityStream.write(c);
ProviderHelper.writeTo(inputStream, entityStream);
}
finally
{
inputStream.close();
}
}
Related
I want to send a big object (lets say 4G) over http.
We have a custom serializer that writes the object to an OutputStream. At the moment we write the object to disk and use that file for the inputstream which is used for the request.
Something like these lines:
private static Response sendObject(Object bigObject) throws IOException {
File tempFile = File.createTempFile("x", "y");
OutputStream out = new FileOutputStream(tempFile);
CustomSerializer.serialize(bigObject, out);
out.close();
WebTarget resource = service.path("data");
FormDataMultiPart multiPartEntity = new FormDataMultiPart();
InputStream inputStream = new FileInputStream(tempFile);
StreamDataBodyPart streamBodyPart = new StreamDataBodyPart(
"data",
inputStream,
"data",
MediaType.APPLICATION_OCTET_STREAM_TYPE);
MultiPart multiPart = multiPartEntity.bodyPart(streamBodyPart);
return resource.request(MediaType.APPLICATION_JSON_TYPE)
.post(Entity.entity(multiPart, multiPart.getMediaType()));
}
We save some memory because we don't serialize to a byte array in memory. Thats nice. But could I save the memory without writing to disk.
Could you write directly to the input stream without rewriting the CustomSerializer?
Could you write directly to the input stream while it reads into the request?
-
It is a little hard to explain, But I think I am after something like this pseudo code:
private static Response sendObject(Object bigObject) throws IOException {
WebTarget resource = service.path("data");
FormDataMultiPart multiPartEntity = new FormDataMultiPart();
// A type of stream I don't know if exist
OutputStream outIn = new OutputInputStream() {
public void openInputStream() {
CustomSerializer.serialize(bigObject, this);
}
};
StreamDataBodyPart streamBodyPart = new StreamDataBodyPart(
"data",
outIn.getInputStream(),
"data",
MediaType.APPLICATION_OCTET_STREAM_TYPE);
MultiPart multiPart = multiPartEntity.bodyPart(streamBodyPart);
return resource.request(MediaType.APPLICATION_JSON_TYPE)
.post(Entity.entity(multiPart, multiPart.getMediaType()));
}
You could use a StreamingOutput and use your CustomSerializer to write to the provided OutputStream
StreamingOutput entity = new StreamingOutput() {
#Override
public void write(OutputStream out)
throws IOException, WebApplicationException {
CustomSerializer.serialize(bigObject, out);
}
};
The write() method will be called by Jersey, giving you a chance to write directly to response entity stream.
Then just use a FormDataBodyPart
BodyPart bigPart = new FormDataBodyPart(
"data", entity, MediaType.APPLICATION_OCTET_STREAM_TYPE);
MultiPart multiPart = new FormDataMultiPart().bodyPart(bigPart);
I am trying to download csv file from REST endpoint. Here is what I am trying.
#ApiOperation(value = "export",
notes = "Export Cache details for a given criteria")
#ApiImplicitParams({
})
#ApiResponses(value = {
#ApiResponse(code = 400, message = "Bad Request"),
#ApiResponse(code = 404, message = "Not Found"),
#ApiResponse(code = 500, message = "Internal Server Error") })
#RequestMapping(method = RequestMethod.GET, value = "/export")
public ResponseEntity export( HttpServletRequest request )
{
CacheDataManager cacheResultHandler = new CacheDataManager();
InputStreamResource inputStreamResource = null;
HttpHeaders httpHeaders = new HttpHeaders();
long contentLengthOfStream;
try
{
inputStreamResource = cacheResultHandler.exportCacheResults( request );
httpHeaders.set( HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=" + "test.csv" );
contentLengthOfStream = inputStreamResource.contentLength();
httpHeaders.setContentLength( contentLengthOfStream );
}
catch ( IOException e )
{
e.printStackTrace();
}
return new ResponseEntity( inputStreamResource, httpHeaders, HttpStatus.OK );
}
My export function.
#Override
public InputStreamResource export( HttpServletRequest request )
{
StringBuilder sb = new StringBuilder();
StringBuilder fileName = new StringBuilder( VALIDATION_REPORT );
sb.append( "Column A" );
sb.append( "," );
sb.append( "Column B" );
sb.append( "\n" );
try
{
sb.append( "TEST A");
sb.append( ',' );
sb.append( "TEST B" );
sb.append( '\n' );
fileName.append( "_" ).append( sdf.format( new Date() ) ).append( ".csv" );
return CsvFileWriter.csvFileWrite( fileName.toString(), sb );
}
catch ( Exception e )
{
e.printStackTrace();
}
return null;
}
CsvFileWriter.java
package it.app.ext.dashboard.util;
import org.springframework.core.io.InputStreamResource;
import java.io.*;
public class CsvFileWriter
{
public static InputStreamResource csvFileWrite( String fileName, StringBuilder content ) throws FileNotFoundException
{
File file = null;
PrintWriter pw = null;
try
{
file = new File( fileName );
pw = new PrintWriter( file );
pw.write( content.toString() );
}
catch ( FileNotFoundException e )
{
e.printStackTrace();
}
finally
{
pw.flush();
pw.close();
}
InputStream inputStream = new FileInputStream( file );
return new InputStreamResource( inputStream );
}
}
File is generating with content inside the tomcat/bin folder but exception occurred.
java.lang.IllegalStateException: InputStream has already been read - do not use InputStreamResource if a stream needs to be read multiple times.
I want to download a .csv file once call this endpoint.
Any suggestions are appreciated.
Thanks You
Explainations:
You got inputStream first:
contentLengthOfStream =inputStreamResource.contentLength();
Then Spring's returnValueHandlers got inputStream again:
new ResponseEntity( inputStreamResource, httpHeaders, HttpStatus.OK ).
But the inputStream wrapped by inputStreamResource only can be used once:
/**
* This implementation throws IllegalStateException if attempting to
* read the underlying stream multiple times.
*/
public InputStream getInputStream() throws IOException, IllegalStateException {
if (this.read) {
throw new IllegalStateException("InputStream has already been read - " +
"do not use InputStreamResource if a stream needs to be read multiple times");
}
this.read = true;
return this.inputStream;
}
Solution: You can get bytes from inputStream and return the ResponseEntity with bytes.
#ApiOperation(value = "export",
notes = "Export Cache details for a given criteria")
#ApiImplicitParams({
})
#ApiResponses(value = {
#ApiResponse(code = 400, message = "Bad Request"),
#ApiResponse(code = 404, message = "Not Found"),
#ApiResponse(code = 500, message = "Internal Server Error") })
#RequestMapping(method = RequestMethod.GET, value = "/export")
public ResponseEntity export( HttpServletRequest request )
{
CacheDataManager cacheResultHandler = new CacheDataManager();
InputStreamResource inputStreamResource = null;
InputStream inputStream = null;
byte[] byteArray = new byte[0];
HttpHeaders httpHeaders = new HttpHeaders();
try
{
inputStreamResource = cacheResultHandler.exportCacheResults( request );
httpHeaders.set( HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=" + "test.csv" );
//convert inputStream to bytes
inputStream = inputStreamResource.getInputStream();
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[1024];
while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
byteArray = buffer.toByteArray();
httpHeaders.setContentLength(byteArray.length);
}
catch ( IOException e )
{
e.printStackTrace();
}
return new ResponseEntity( byteArray, httpHeaders, HttpStatus.OK );
}
Suggest: using Apache Commons IO to convert InputStream to bytes.Need to add a lib dependency,which can make your code brief
byte[] byteArray = IOUtils.toByteArray(inputStream);
don't use the same file twice, use the separate code for returning InputStream:
return new InputStreamResource( new FileInputStream( new File( fileName ) ) );
Is there a way to set the httpEntiy in the restTemplate.execute Method? I have to put the Authorization in the header, so thats why I can not exclude it. As a ResponseEntity I get a InputStreamResource.
This is working without HttpEntiy set:
File responseFile = restTemplate.execute(
uriComponents.toUri(),
HttpMethod.GET, null,
new ResponseExtractor<File>() {
#Override
public File extractData(ClientHttpResponse response) throws IOException {
File serverFile = fileProcessHelper.createFile(pathToFile);
BufferedOutputStream stream = new BufferedOutputStream(
new FileOutputStream(serverFile));
byte[] bytes = IOUtils.toByteArray(response.getBody());
stream.write(bytes);
stream.close();
return serverFile;
}
});
This is NOT working. Error is: java.io.IOException: stream is closed
ResponseEntity<InputStreamResource> responseEntity = restTemplate.exchange(
uriComponents.toUri(),
HttpMethod.GET, requestEntity,
InputStreamResource.class);
InputStreamResource stream = new InputStreamResource(responseEntity.getBody().getInputStream());
HttpHeaders respHeaders = new HttpHeaders();
respHeaders.setContentLength(stream.contentLength());
response.setHeader("Content-Disposition", "attachment; filename=" + stream.getFilename());
return ResponseEntity.ok().headers(respHeaders).body(stream);
Or is there a way to reopen the inputstreamresource?
Thanks in advance!
Ok. I found a solution:
in the RquestCallback you can set the headers:
RequestCallback requestCallback = new RequestCallback() {
#Override
public void doWithRequest(ClientHttpRequest request) throws IOException {
byte[] plainCredsBytes = plainCreds.getBytes();
byte[] base64CredsBytes = Base64.encodeBase64(plainCredsBytes);
String base64Creds = new String(base64CredsBytes);
request.getHeaders().set("Authorization", "Basic " + base64Creds);
}
};
Server Code :
#POST
#Path("reportDownload")
#Consumes(MediaType.APPLICATION_JSON)
public Response generateReport(QueryData queryData) {
File file = new File("report.xlsx") // large file
StreamingOutput stream = new FileStreamingOutput(file) ;
return Response.ok(stream, MediaType.APPLICATION_OCTET_STREAM)
.header("filename" , file.getName())
.build();
}
Client Code :
Using the following code I'm able to download files upto some limit. Getting out of memory heap error for large files.
final String uri = buildUri("/reportGenerate/reportDownload");
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setReadTimeout(read_timeout);
factory.setConnectTimeout(connection_timeout);
RestTemplate restTemplate = new RestTemplate(factory);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
List<MediaType> mediaTypeList = new ArrayList<>();
mediaTypeList.add(MediaType.APPLICATION_OCTET_STREAM);
headers.setAccept(mediaTypeList);
HttpEntity entity = new HttpEntity(queryData, headers);
ResponseEntity<byte[]> data = restTemplate.exchange(uri, HttpMethod.POST, entity, byte[].class);
HttpHeaders responseHeader = data.getHeaders();
String fileName = (String) responseHeader.get("filename").get(0);
String downloadFolder = ApplicationConfig.REPORT_DOWNLOAD_FOLDER.getValue();
if (data.getStatusCode() == HttpStatus.OK) {
FileOutputStream fos = null;
File toFile = null;
try {
toFile = new File(downloadFolder + File.separator + fileName);
fos = new FileOutputStream(toFile);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
IOUtils.write(data.getBody(), bos);
bos.writeTo(fos);
} catch (Exception e) {
convertReportException(e);
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException ex) {
convertReportException(ex);
}
}
return toFile;
}
}
How to use stream for download larger files.
Here is how I do it with a ResponseExtractor. Based on hints from this Spring Jira issue.
RestTemplate restTemplate // = ...;
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);
update
Here is what RestTemplate does behind the scenes on postForObject and friends (inline comments from me):
#Override
public <T> T postForObject(String url, Object request, Class<T> responseType, Map<String, ?> uriVariables)
throws RestClientException {
// From RequestCallback's javadoc:
// Callback interface for code that operates on a ClientHttpRequest.
// Allows to manipulate the request headers, and write to the request body.
//
// Used internally by the RestTemplate, but also useful for application code.
RequestCallback requestCallback = httpEntityCallback(request, responseType);
// HttpMessageConverterExtractor checks the response type header and requested
// responseType class to select the proper message converter to handle the response.
// It also implements ResponseExtractor.
HttpMessageConverterExtractor<T> responseExtractor =
new HttpMessageConverterExtractor<T>(responseType, getMessageConverters(), logger);
return execute(url, HttpMethod.POST, requestCallback, responseExtractor, uriVariables);
}
/**
* Returns a request callback implementation that writes the given object to the
* request stream.
*/
protected <T> RequestCallback httpEntityCallback(Object requestBody, Type responseType) {
return new HttpEntityRequestCallback(requestBody, responseType);
}
Note: This is essentially a duplicate of my answer at https://stackoverflow.com/a/38664475/1030527 but I can't mark the questions as duplicate since neither this one or that one have upvoted answers.
At the client that you mentioned
don't store the file in memory to download a large via the RestTemplate, it can cause the Java heap exception.
it should be stored on disk.
Here is some code sample to download a large file via the RestTemplate
#GetMapping("largeFile")
public ResponseEntity<InputStreamResource> downloadLargeFile(
#RequestParam("fileName") String fileName
) throws IOException {
RestTemplate restTemplate = new RestTemplate();
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<InputStreamResource> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("tmp/" + fileName);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
return new InputStreamResource(new FileInputStream(String.format("tmp/%s", fileName)));
};
InputStreamResource response = restTemplate.execute(
String.format("http://%s:%s/file/largeFileRestTemplate?fileName=%s", host, "9091", fileName),
HttpMethod.GET,
requestCallback,
responseExtractor
);
return ResponseEntity
.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s", fileName))
.body(response);
}
I am developing an Android applicaiton with AppEngine backend. I am creating the server part with Google Cloud Endpoints in Java. My problem is that I cannot send a Bitmap from the client to the server.
I used the answer from this question but even if the client part does not seem to have any problems at all, the server part does not receive the data at all. I also think this solution might be a bit complicated and that it might work a different, easier way, however this is my first time implementing a server and first time sending a picture to it so I accept any good tips on this. Thanks!
Here is my code:
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
String charset = "UTF-8";
HttpURLConnection connection = (HttpURLConnection) new URL("https://path_to_my_app/_ah/api/registration/v1/uploadImage").openConnection();
connection.setDoOutput(true);
connection.setReadTimeout(60000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
PrintWriter writer = null;
try {
OutputStream output = connection.getOutputStream();
writer = new PrintWriter(new OutputStreamWriter(output, charset), true); // true = autoFlush, important!
// Send text file.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + somename + "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
writer.append(CRLF).flush();
BufferedReader reader = null;
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
photo.compress(Bitmap.CompressFormat.PNG, 100, outputStream);
byteArray = stream.toByteArray();
try {
reader = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(byteArray), charset));
for (String line; (line = reader.readLine()) != null;) {
writer.append(line).append(CRLF);
}
} finally {
if (reader != null) try { reader.close(); } catch (IOException logOrIgnore) {}
}
writer.flush();
// End of multipart/form-data.
writer.append("--" + boundary + "--").append(CRLF);
}
finally
{
if (writer != null)
{
writer.close();
}
}
The server part:
#ApiMethod(name = "uploadImage", httpMethod = "POST")
public void uploadImage(HttpServletRequest request, HttpServletResponse response) throws IOException
{
ServletFileUpload fileUpload = new ServletFileUpload();
try
{
FileItemIterator iterator = fileUpload.getItemIterator(request);
while(iterator.hasNext()){
FileItemStream itemStream = iterator.next();
String fieldName = itemStream.getFieldName();
log.info("field name:"+fieldName);
InputStream stream = itemStream.openStream();
String result = getStringFromInputStream(stream);
log.info("result: "+result);
stream.close();
}
}
catch (FileUploadException e)
{
e.printStackTrace();
}
}
I am getting 204 no Content type now.
I did it!
I think this is not the best way of doing it but it´s working so I am fine until I get a better solution.
So I take the Bitmap image and convert it to String:
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, outputStream);
byte[] bitmapByte = outputStream.toByteArray();
String stringEncodedImage = Base64.encodeToString(bitmapByte, Base64.DEFAULT);
Then I create a httpPostRequest and set a JsonObject to it with the image converted to String in it.
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("https://my_app_path/_ah/api/registration/v1/uploadImage");
JSONObject jsonObject = new JSONObject();
jsonObject.put("image",stringEncodedImage);
StringEntity stringEntity = new StringEntity(jsonObject.toString());
httpPost.addHeader("Content-Type", "application/json");
httpPost.setEntity(stringEntity);
HttpResponse response = httpClient.execute(httpPost);
On the server side, in my Endpoint, I do this:
#ApiMethod(name = "uploadImage", httpMethod = "POST")
public JSONObject uploadImage(JSONObject request) throws IOException
{
String imageInString = (String) request.get("image");
Blob blob = new Blob(imageInString.getBytes());
....save blob and do whatever you want...
}
The same goes the other way. I pack Blob into JsonObject and send it over.