I have a url which download the large size zip file.It returns the response as stream.though file size is large first it returns 200(HTTPSTATUK.OK) and continues download.
I have to implement a new spring controller which call the above url through rest template.I have to read the response returned by rest template and pass to controller.initially I have implemented in below way
#GetMapping("/export/downloadFile")
public ResponseEntity<byte[]> downloadData(Model model,
#ModelAttribute(EXCEPTION_COLLECTOR) ExceptionCollector exceptionCollector,
#RequestParam("userName") String userName,
#RequestParam("startDate") Date startDate,
#RequestParam("endDate") Date endDate,
#RequestParam("reason") String reason) {
URI uri = /building url here/;
return restTemplate.exchange(uri, HttpMethod.GET, new HttpEntity<>(httpHeaders), byte[].class);
}
since I am using ResponseEntity<byte[]> , rest template waits till entire file loaded into memory.so very frequently I am getting socket timeout issue.
Do we have way to read the response as stream and return to controller.
I found few things about restTemplate.execute .
restTemplate.execute(uri,HttpMethod.GET,requestCallBack,clientHttpResponse -> {
File ret = File.createTempFile("download", ".zip",new File("/Users/bokkavijay/Desktop"));
StreamUtils.copy(clientHttpResponse.getBody(), new FileOutputStream(ret));
return ret;
});
above snippet can copy the file to our local with out time out but this is not what I need.
how can we pipe the stream in clientHttpResponse to controller ?
I found the working implementation
Controller
#GetMapping("/readResponseAsStream")
public ResponseEntity<StreamingResponseBody> downloadAsStream(Model model,HttpServletResponse response) {
HttpHeaders httpHeaders=new HttpHeaders();
httpHeaders.add("Transfer-Encoding","chunked");
httpHeaders.add("Content-Type","x-zip-compressed");
httpHeaders.add("Content-Disposition", "attachment;filename=sample.zip");
ServletOutputStream servletOutputStream=response.getOutputStream();
StreamingResponseBody downloadFile = out -> {
RequestCallback requestCallBack=request->{
request.getHeaders().add(//Add headers here);
};
ResponseExtractor<ServletOutputStream> responseExtractor = clientHttpResponse -> {
//code snippet if you want to write response stream to HttpServletResponse
byte[] buff = new byte[800000];
int bytesRead = 0;
while ((bytesRead = clientHttpResponse.getBody().read(buff)) != -1) {
servletOutputStream.write(buff, 0, bytesRead);
}
return servletOutputStream;
//Incase if you want to copy file to you local
File ret = File.createTempFile("download", ".zip",new File("Add Local system directory address here"));
StreamUtils.copy(clientHttpResponse.getBody(), new FileOutputStream(ret));
//You can copy the the clientHttpResponse.getBody() to ByteArrayInputStream and return
// Don't return clientHttpResponse.getBody() directly because rest template will close the inputStream(clientHttpResponse.getBody()) after .execute completed
//if you want to write restTemplate.execute in dao layer , pass servletOutputStream as a argument to method
};
restTemplate.execute(URL_ADDRESS,HttpMethod.GET,requestCallBack,responseExtractor);
};
return new ResponseEntity(downloadFile,httpHeaders,HttpStatus.OK);
}
If you write the response directly to HttpServletResponse , controller download the file when we access in browser
Greetings to the community! I am currently developing a RESTful web service in Java using the JAX-rs library. What I would like to do is make clients able to upload a file via the service. I successfully managed to achieve this using the following piece of code
#Consumes({"application/json"})
#Produces({"application/json"})
#Path("uploadfileservice")
public interface UploadFileService {
#Path("/fileupload")
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
Response uploadFile(#FormDataParam("file") InputStream uploadedInputStream)
}
Implementation class
#Service
public class UploadFileServiceImpl implements UploadFileService {
#Override
public Response uploadFile(InputStream uploadedInputStream){
String fileToWrite = "//path/file.txt" //assuming a upload a txt file
writeToFile(uploadedInputStream, fileToWrite); //write the file
}
private void writeToFile(InputStream uploadedInputStream,
String uploadedFileLocation) {
try {
OutputStream out = new FileOutputStream(new File(
uploadedFileLocation));
int read = 0;
byte[] bytes = new byte[1024];
out = new FileOutputStream(new File(uploadedFileLocation));
while ((read = uploadedInputStream.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I am using POSTMAN as a client to test my web service and I am facing the following problem: When I upload a .txt file, that files gets appended with some other details of the file
Example:
File sent
Postman Request
File stored on my filesystem
Any idea why this happens? Maybe I am missing something in the Headers section of my request? Or maybe any issue is caused because of the MediaType I am consuming in my web service endpoint?
Thanks in advance for any help :)
PS
If I upload a .pdf file it is not leaded to corruption and the .pdf file is stored normally on my filesystem
Your method signature should be
Response uploadFile(#FormDataParam("file") FormDataBodyPart uploadedFile)
and you can get content of the file as
InputStream uploadedInputStream = uploadedFile.getValueAs(InputStream.class)
Hope this helps.
Finally I found a workaround for my problem using the Attachment class of org.apache.cxf:
#Consumes({"application/json"})
#Produces({"application/json"})
#Path("uploadfileservice")
public interface UploadFileService {
#Path("/fileupload")
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
Response uploadFile(#Multipart("file") Attachment attr)
}
#Service
public class UploadFileServiceImpl implements UploadFileService {
#Override
public Response uploadFile(Attachment attr){
String pathToUpload= "//path//.txt"
try{
attr.transferTo(new File(pathToUpload)); //will copy the uploaded file in
//this destination
}
catch(Exception e){
}
}
I'm using spring mvc and want to create a method which must return binary file. I've found some tutorials but nothing works for me. This is my part of my code:
#RequestMapping(value = "/get-frame/{fileId}", method = RequestMethod.GET)
public void getVideoFrame(#PathVariable("fileId") String filename,
HttpServletResponse response) throws IOException {
...
...
...
byte[] image = ...
OutputStream outputStream = response.getOutputStream();
outputStream.write(image);
outputStream.close();
response.flushBuffer();
}
This method always returns error 406. What's wrong?
Thanks
You should set the header, content type and length of the response.
try (OutputStream out = response.getOutputStream()) {
byte[] image = ...
response.setContentLength(image.length);
response.setContentType("image/png"); //or something more generic...
response.setHeader("Accept-Ranges", "bytes");
response.setStatus(HttpServletResponse.SC_OK);
out.write(image);
}
I have 3 machines:
server where the file is located
server where REST service is running ( Jersey)
client(browser) with access to 2nd server but no access to 1st server
How can I directly (without saving the file on 2nd server) download the file from 1st server to client's machine?
From 2nd server I can get a ByteArrayOutputStream to get the file from 1st server, can I pass this stream further to the client using the REST service?
It will work this way?
So basically what I want to achieve is to allow the client to download a file from 1st server using the REST service on 2nd server (since there is no direct access from client to 1st server) using only data streams (so no data touching the file system of 2nd server).
What I try now with EasyStream library:
final FTDClient client = FTDClient.getInstance();
try {
final InputStreamFromOutputStream <String> isOs = new InputStreamFromOutputStream <String>() {
#Override
public String produce(final OutputStream dataSink) throws Exception {
return client.downloadFile2(location, Integer.valueOf(spaceId), URLDecoder.decode(filePath, "UTF-8"), dataSink);
}
};
try {
String fileName = filePath.substring(filePath.lastIndexOf("/") + 1);
StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream outputStream) throws IOException, WebApplicationException {
int length;
byte[] buffer = new byte[1024];
while ((length = isOs.read(buffer)) != -1) {
outputStream.write(buffer, 0, length);
}
outputStream.flush();
}
};
return Response.ok(output, MediaType.APPLICATION_OCTET_STREAM)
.header("Content-Disposition", "attachment; filename=\"" + fileName + "\"")
.build();
}
}
UPDATE2
So my code now with the custom MessageBodyWriter looks simple:
ByteArrayOutputStream baos = new ByteArrayOutputStream(2048) ;
client.downloadFile(location, spaceId, filePath, baos);
return Response.ok(baos).build();
But I get the same heap error when trying with large files.
UPDATE3
Finally managed to get it working !
StreamingOutput did the trick.
Thank you #peeskillet ! Many thanks !
"How can I directly (without saving the file on 2nd server) download the file from 1st server to client's machine?"
Just use the Client API and get the InputStream from the response
Client client = ClientBuilder.newClient();
String url = "...";
final InputStream responseStream = client.target(url).request().get(InputStream.class);
There are two flavors to get the InputStream. You can also use
Response response = client.target(url).request().get();
InputStream is = (InputStream)response.getEntity();
Which one is the more efficient? I'm not sure, but the returned InputStreams are different classes, so you may want to look into that if you care to.
From 2nd server I can get a ByteArrayOutputStream to get the file from 1st server, can I pass this stream further to the client using the REST service?
So most of the answers you'll see in the link provided by #GradyGCooper seem to favor the use of StreamingOutput. An example implementation might be something like
final InputStream responseStream = client.target(url).request().get(InputStream.class);
System.out.println(responseStream.getClass());
StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream out) throws IOException, WebApplicationException {
int length;
byte[] buffer = new byte[1024];
while((length = responseStream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
out.flush();
responseStream.close();
}
};
return Response.ok(output).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
But if we look at the source code for StreamingOutputProvider, you'll see in the writeTo, that it simply writes the data from one stream to another. So with our implementation above, we have to write twice.
How can we get only one write? Simple return the InputStream as the Response
final InputStream responseStream = client.target(url).request().get(InputStream.class);
return Response.ok(responseStream).header(
"Content-Disposition", "attachment, filename=\"...\"").build();
If we look at the source code for InputStreamProvider, it simply delegates to ReadWriter.writeTo(in, out), which simply does what we did above in the StreamingOutput implementation
public static void writeTo(InputStream in, OutputStream out) throws IOException {
int read;
final byte[] data = new byte[BUFFER_SIZE];
while ((read = in.read(data)) != -1) {
out.write(data, 0, read);
}
}
Asides:
Client objects are expensive resources. You may want to reuse the same Client for request. You can extract a WebTarget from the client for each request.
WebTarget target = client.target(url);
InputStream is = target.request().get(InputStream.class);
I think the WebTarget can even be shared. I can't find anything in the Jersey 2.x documentation (only because it is a larger document, and I'm too lazy to scan through it right now :-), but in the Jersey 1.x documentation, it says the Client and WebResource (which is equivalent to WebTarget in 2.x) can be shared between threads. So I'm guessing Jersey 2.x would be the same. but you may want to confirm for yourself.
You don't have to make use of the Client API. A download can be easily achieved with the java.net package APIs. But since you're already using Jersey, it doesn't hurt to use its APIs
The above is assuming Jersey 2.x. For Jersey 1.x, a simple Google search should get you a bunch of hits for working with the API (or the documentation I linked to above)
UPDATE
I'm such a dufus. While the OP and I are contemplating ways to turn a ByteArrayOutputStream to an InputStream, I missed the simplest solution, which is simply to write a MessageBodyWriter for the ByteArrayOutputStream
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
#Provider
public class OutputStreamWriter implements MessageBodyWriter<ByteArrayOutputStream> {
#Override
public boolean isWriteable(Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return ByteArrayOutputStream.class == type;
}
#Override
public long getSize(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType) {
return -1;
}
#Override
public void writeTo(ByteArrayOutputStream t, Class<?> type, Type genericType,
Annotation[] annotations, MediaType mediaType,
MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream)
throws IOException, WebApplicationException {
t.writeTo(entityStream);
}
}
Then we can simply return the ByteArrayOutputStream in the response
return Response.ok(baos).build();
D'OH!
UPDATE 2
Here are the tests I used (
Resource class
#Path("test")
public class TestResource {
final String path = "some_150_mb_file";
#GET
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response doTest() throws Exception {
InputStream is = new FileInputStream(path);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int len;
byte[] buffer = new byte[4096];
while ((len = is.read(buffer, 0, buffer.length)) != -1) {
baos.write(buffer, 0, len);
}
System.out.println("Server size: " + baos.size());
return Response.ok(baos).build();
}
}
Client test
public class Main {
public static void main(String[] args) throws Exception {
Client client = ClientBuilder.newClient();
String url = "http://localhost:8080/api/test";
Response response = client.target(url).request().get();
String location = "some_location";
FileOutputStream out = new FileOutputStream(location);
InputStream is = (InputStream)response.getEntity();
int len = 0;
byte[] buffer = new byte[4096];
while((len = is.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
out.flush();
out.close();
is.close();
}
}
UPDATE 3
So the final solution for this particular use case was for the OP to simply pass the OutputStream from the StreamingOutput's write method. Seems the third-party API, required a OutputStream as an argument.
StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream out) {
thirdPartyApi.downloadFile(.., .., .., out);
}
}
return Response.ok(output).build();
Not quite sure, but seems the reading/writing within the resource method, using ByteArrayOutputStream`, realized something into memory.
The point of the downloadFile method accepting an OutputStream is so that it can write the result directly to the OutputStream provided. For instance a FileOutputStream, if you wrote it to file, while the download is coming in, it would get directly streamed to the file.
It's not meant for us to keep a reference to the OutputStream, as you were trying to do with the baos, which is where the memory realization comes in.
So with the way that works, we are writing directly to the response stream provided for us. The method write doesn't actually get called until the writeTo method (in the MessageBodyWriter), where the OutputStream is passed to it.
You can get a better picture looking at the MessageBodyWriter I wrote. Basically in the writeTo method, replace the ByteArrayOutputStream with StreamingOutput, then inside the method, call streamingOutput.write(entityStream). You can see the link I provided in the earlier part of the answer, where I link to the StreamingOutputProvider. This is exactly what happens
See example here: Input and Output binary streams using JERSEY?
Pseudo code would be something like this (there are a few other similar options in above mentioned post):
#Path("file/")
#GET
#Produces({"application/pdf"})
public StreamingOutput getFileContent() throws Exception {
public void write(OutputStream output) throws IOException, WebApplicationException {
try {
//
// 1. Get Stream to file from first server
//
while(<read stream from first server>) {
output.write(<bytes read from first server>)
}
} catch (Exception e) {
throw new WebApplicationException(e);
} finally {
// close input stream
}
}
}
Refer this:
#RequestMapping(value="download", method=RequestMethod.GET)
public void getDownload(HttpServletResponse response) {
// Get your file stream from wherever.
InputStream myStream = someClass.returnFile();
// Set the content type and attachment header.
response.addHeader("Content-disposition", "attachment;filename=myfilename.txt");
response.setContentType("txt/plain");
// Copy the stream to the response's output stream.
IOUtils.copy(myStream, response.getOutputStream());
response.flushBuffer();
}
Details at: https://twilblog.github.io/java/spring/rest/file/stream/2015/08/14/return-a-file-stream-from-spring-rest.html
Good day!
I am trying to output a JPG file contained in the web application to the
user using the following code:
public class JpegOutput extends HttpServlet {
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
byte bufferArray[] = new byte[1024];
ServletContext ctxt = getServletContext();
response.setContentType("image/jpeg");
ServletOutputStream os = response.getOutputStream();
InputStream is = ctxt.getResource("/WEB-INF/image/image1.jpg").openStream();
int read = is.read(bufferArray);
while (read != 1) {
os.write(bufferArray);
read = is.read(bufferArray);
}
is.close();
os.close();
}
}
But an error appears:
HTTP Status 500 -
exception java.lang.NullPointerException
I am not sure if it can't read the source image or something. Anyway, I put the image inside this folder /WEB-INF/image/image1.jpg
What am I doing wrong? How can I resolve this issue?
EDIT: I solved the problem by renaming the filename... the file name is case sensitive, instead of image1.jpg, it should be image1.JPG
Thank you.
You might use getServletContext().getRealPath("/") to get the path to /WEB-INF/. E.g.
String path = getServletContext().getRealPath("/") + "WEB-INF/image/image1.jpg";
InputStream is = new FileInputStream(path);
Though its not sure that this is the reason for the NPE. Can you check the log file and post the stacktrace?
Not sure about the error, but I think it would be better to forward the request rather than serve the image manually:
public class JpegOutput extends HttpServlet {
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
request.getRequestDispatcher("/WEB-INF/image/image1.jpg")
.forward(request, response);
}
}
Also note that you content-serving loop is incorrect, the correct one looks like this:
while ((read = is.read(bufferArray)) != -1)
os.write(bufferArray, 0, read);