I have this block of code that copies a file to a remote server using an InputStream:
final InputStream fis = new FileInputStream(new File(CACHED_CLONE_FILE_NAME));
final RequestCallback requestCallback = new RequestCallback()
{
public void doWithRequest(final ClientHttpRequest request)
{
try {
request.getHeaders().add("Content-type", "application/octet-stream");
OutputStream procOutput = request.getBody();
BufferedOutputStream bos = new BufferedOutputStream(procOutput);
IOUtils.copy(fis, bos);
}
catch (IOException ex) {
}
}
};
final RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new StringHttpMessageConverter());
restTemplate.getMessageConverters().add(new MappingJacksonHttpMessageConverter());
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);
final HttpMessageConverterExtractor<String> responseExtractor = new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());
restTemplate.execute(urlToPost, HttpMethod.POST, requestCallback, responseExtractor);
It works, except that for a small file (4MB) it takes hours to copy it over. Why is this occurring, and what can I do about the performance?
This code is being used on an Android device, and there are no network problems.
Related
String uri = "URL";
RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new ByteArrayHttpMessageConverter());
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));
HttpEntity<String> entity = new HttpEntity<String>(headers);
ResponseEntity<byte[]> response = restTemplate.exchange(uri, HttpMethod.GET, entity, byte[].class, "1");
I get response like-> <200,[B#c505096,[Pragma:"no-cache", Content-Disposition:"inline; filename=935436242330664960_pratikpopo.txt", Expires:"0", Cache-Control:"no-cache, no-store, must-revalidate", Content-Type:"application/octet-stream", Content-Length:"18", Date:"Mon, 31 Jan 2022 04:16:56 GMT"]> I want to downlod (935436242330664960_pratikpopo.txt) this file. Is there any way to download this file
{
URL link = new URL("your full url http://........");
InputStream in = new BufferedInputStream(link.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1!=(n=in.read(buf)))
{
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
FileOutputStream fos = new FileOutputStream("C:/download-file/YourFileName.txt");
fos.write(response);
fos.close();
}
#RequestMapping(value = "/download3/{path}", method = RequestMethod.GET)
public String downloadFile4( #RequestParam("file_name") String path) throws IOException
{
String uri = "File Path-URL(location wehere your file is stored on server)";
RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new ByteArrayHttpMessageConverter());
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));
HttpEntity<String> entity = new HttpEntity<String>(headers);
ResponseEntity<byte[]> response = restTemplate.exchange(uri, HttpMethod.GET, entity, byte[].class, "1");
URL furl = new URL(uri);
ReadableByteChannel rbc = Channels.newChannel(furl.openStream());
FileOutputStream fos = new FileOutputStream("C:/download-file/"+ response.getHeaders().getContentDisposition().getFilename());
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
return "file downloaded";
}
I am trying to upload a file on my server to another server. While doing this
I am getting this error : Resolved [org.springframework.web.multipart.support.MissingServletRequestPartException: Required request part 'file' is not present]
I have put the 'file' key in my body. I am unable to figure out why this is hapenning.
Uploading Server:
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
MultiValueMap<String, Object> body
= new LinkedMultiValueMap<>();
body.add("file", Paths.get("pathToFile").toFile());
HttpEntity<MultiValueMap<String, Object>> requestEntity
= new HttpEntity<>(body, headers);
RestTemplate restTemplate = new RestTemplate();
ResponseEntity<String> response = restTemplate
.postForEntity(url, requestEntity, String.class);
Server that we are uploading to:
#RequestMapping(value = "/api", method = RequestMethod.POST, consumes = MediaType.ALL_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<ResponseDTO> solveUsingJar(#RequestParam("file") MultipartFile file) {
}
I share some code to upload a file one server to another server.
STEP-1 : Upload a file in Server-1 :
#PostMapping("/upload/profile")
public UploadFileResponse uploadProfileImg(#RequestPart("file") MultipartFile file)
{
return fileService.uploadProfileImg(file);
}
STEP-2 : Upload that file in Server-2 using RestTemplate.
public String upload(MultipartFile file) throws IOException
{
ResponseEntity<UploadFileResponse> response = null;
try
{
File convFile = new File(file.getOriginalFilename());
convFile.createNewFile();
FileOutputStream fos = new FileOutputStream(convFile);
fos.write(file.getBytes());
fos.close();
LinkedMultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
String uploadURI = KeyConstant.FILE_UPLOAD_API_URI+"/file"; // Server-2(file Directory) FILE_UPLOAD_API_URI : http://27.34.2.33:28181/upload/file
map.add("file", new FileSystemResource(convFile));
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<LinkedMultiValueMap<String, Object>> requestEntity = new HttpEntity<>(map, headers);
RestTemplate restTemplate = new RestTemplate();
response = restTemplate.exchange(uploadURI, HttpMethod.POST, requestEntity, UploadFileResponse.class);
}
catch(Exception e) {e.printStackTrace();}
return response.getBody();
}
STEP-3 : Server-2 File Upload Rest Controller :
#PostMapping("/upload/file")
public UploadFileResponse uploadFileByDirectory(#RequestPart(value = "file") MultipartFile file)
{
return this.amazonClient.uploadFilebyDirectory(KeyConstant.AMAZON_S3_PATH, file);
}
I am trying to upload a zip file and form data using RestTemplate.
Please find the code below.
#Controller
#RequestMapping("/test")
public class EdicomGatewayClient {
#RequestMapping(value ="/publisEdicomDocument", method = RequestMethod.POST)
public void publisEdicomDocument() {
List<ApiError> errors = null;
try {
RestTemplate restTemplate = new RestTemplate();
String restURL = "";
File file = new File("");
DiskFileItem fileItem = new DiskFileItem("file", "text/plain", false, file.getName(), (int) file.length(), file.getParentFile());
fileItem.getOutputStream();
MultipartFile multipartFile = new CommonsMultipartFile(fileItem);
List<MediaType> acceptableMediaTypes = new ArrayList<MediaType>();
acceptableMediaTypes.add(MediaType.MULTIPART_FORM_DATA);
HttpHeaders headers = new HttpHeaders();
headers.setAccept(acceptableMediaTypes);
MultiValueMap<String, Object> multipartRequest = new LinkedMultiValueMap<String, Object>();
multipartRequest.add("user","OPENTEXT_AR_TST");
multipartRequest.add("password","4t4qqdbddp");
multipartRequest.add("domain","OPENTEXT_AR_TST");
multipartRequest.add("group","ASPEDI41");
multipartRequest.add("publishType","2");
multipartRequest.add("process","MAPA_AFIP_WSMTXCA_EDICOM_OPENTEXT");
multipartRequest.add("saveOriginalImage",false);
multipartRequest.add("sendDocument",true);
multipartRequest.add("returnData",false);
multipartRequest.add("returnDataType","2");
multipartRequest.add("message", new ByteArrayResource(multipartFile.getBytes()));
multipartRequest.add("filename", multipartFile.getOriginalFilename());
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<MultiValueMap<String, Object>>(multipartRequest, headers);
ResponseEntity response = restTemplate.exchange(restURL,HttpMethod.POST, requestEntity, String.class);
System.out.println("Response ******************************************:"+response);
} catch (Exception e) {
}
}
public static void main(String[] args) {
EdicomGatewayClient edicom = new EdicomGatewayClient();
edicom.publisEdicomDocument();
}
}
This is giving me the following error:
org.springframework.web.client.HttpClientErrorException: 400 bad Request.
Is there a way to set the httpEntiy in the restTemplate.execute Method? I have to put the Authorization in the header, so thats why I can not exclude it. As a ResponseEntity I get a InputStreamResource.
This is working without HttpEntiy set:
File responseFile = restTemplate.execute(
uriComponents.toUri(),
HttpMethod.GET, null,
new ResponseExtractor<File>() {
#Override
public File extractData(ClientHttpResponse response) throws IOException {
File serverFile = fileProcessHelper.createFile(pathToFile);
BufferedOutputStream stream = new BufferedOutputStream(
new FileOutputStream(serverFile));
byte[] bytes = IOUtils.toByteArray(response.getBody());
stream.write(bytes);
stream.close();
return serverFile;
}
});
This is NOT working. Error is: java.io.IOException: stream is closed
ResponseEntity<InputStreamResource> responseEntity = restTemplate.exchange(
uriComponents.toUri(),
HttpMethod.GET, requestEntity,
InputStreamResource.class);
InputStreamResource stream = new InputStreamResource(responseEntity.getBody().getInputStream());
HttpHeaders respHeaders = new HttpHeaders();
respHeaders.setContentLength(stream.contentLength());
response.setHeader("Content-Disposition", "attachment; filename=" + stream.getFilename());
return ResponseEntity.ok().headers(respHeaders).body(stream);
Or is there a way to reopen the inputstreamresource?
Thanks in advance!
Ok. I found a solution:
in the RquestCallback you can set the headers:
RequestCallback requestCallback = new RequestCallback() {
#Override
public void doWithRequest(ClientHttpRequest request) throws IOException {
byte[] plainCredsBytes = plainCreds.getBytes();
byte[] base64CredsBytes = Base64.encodeBase64(plainCredsBytes);
String base64Creds = new String(base64CredsBytes);
request.getHeaders().set("Authorization", "Basic " + base64Creds);
}
};
Server Code :
#POST
#Path("reportDownload")
#Consumes(MediaType.APPLICATION_JSON)
public Response generateReport(QueryData queryData) {
File file = new File("report.xlsx") // large file
StreamingOutput stream = new FileStreamingOutput(file) ;
return Response.ok(stream, MediaType.APPLICATION_OCTET_STREAM)
.header("filename" , file.getName())
.build();
}
Client Code :
Using the following code I'm able to download files upto some limit. Getting out of memory heap error for large files.
final String uri = buildUri("/reportGenerate/reportDownload");
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setReadTimeout(read_timeout);
factory.setConnectTimeout(connection_timeout);
RestTemplate restTemplate = new RestTemplate(factory);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
List<MediaType> mediaTypeList = new ArrayList<>();
mediaTypeList.add(MediaType.APPLICATION_OCTET_STREAM);
headers.setAccept(mediaTypeList);
HttpEntity entity = new HttpEntity(queryData, headers);
ResponseEntity<byte[]> data = restTemplate.exchange(uri, HttpMethod.POST, entity, byte[].class);
HttpHeaders responseHeader = data.getHeaders();
String fileName = (String) responseHeader.get("filename").get(0);
String downloadFolder = ApplicationConfig.REPORT_DOWNLOAD_FOLDER.getValue();
if (data.getStatusCode() == HttpStatus.OK) {
FileOutputStream fos = null;
File toFile = null;
try {
toFile = new File(downloadFolder + File.separator + fileName);
fos = new FileOutputStream(toFile);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
IOUtils.write(data.getBody(), bos);
bos.writeTo(fos);
} catch (Exception e) {
convertReportException(e);
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException ex) {
convertReportException(ex);
}
}
return toFile;
}
}
How to use stream for download larger files.
Here is how I do it with a ResponseExtractor. Based on hints from this Spring Jira issue.
RestTemplate restTemplate // = ...;
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);
update
Here is what RestTemplate does behind the scenes on postForObject and friends (inline comments from me):
#Override
public <T> T postForObject(String url, Object request, Class<T> responseType, Map<String, ?> uriVariables)
throws RestClientException {
// From RequestCallback's javadoc:
// Callback interface for code that operates on a ClientHttpRequest.
// Allows to manipulate the request headers, and write to the request body.
//
// Used internally by the RestTemplate, but also useful for application code.
RequestCallback requestCallback = httpEntityCallback(request, responseType);
// HttpMessageConverterExtractor checks the response type header and requested
// responseType class to select the proper message converter to handle the response.
// It also implements ResponseExtractor.
HttpMessageConverterExtractor<T> responseExtractor =
new HttpMessageConverterExtractor<T>(responseType, getMessageConverters(), logger);
return execute(url, HttpMethod.POST, requestCallback, responseExtractor, uriVariables);
}
/**
* Returns a request callback implementation that writes the given object to the
* request stream.
*/
protected <T> RequestCallback httpEntityCallback(Object requestBody, Type responseType) {
return new HttpEntityRequestCallback(requestBody, responseType);
}
Note: This is essentially a duplicate of my answer at https://stackoverflow.com/a/38664475/1030527 but I can't mark the questions as duplicate since neither this one or that one have upvoted answers.
At the client that you mentioned
don't store the file in memory to download a large via the RestTemplate, it can cause the Java heap exception.
it should be stored on disk.
Here is some code sample to download a large file via the RestTemplate
#GetMapping("largeFile")
public ResponseEntity<InputStreamResource> downloadLargeFile(
#RequestParam("fileName") String fileName
) throws IOException {
RestTemplate restTemplate = new RestTemplate();
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<InputStreamResource> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("tmp/" + fileName);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
return new InputStreamResource(new FileInputStream(String.format("tmp/%s", fileName)));
};
InputStreamResource response = restTemplate.execute(
String.format("http://%s:%s/file/largeFileRestTemplate?fileName=%s", host, "9091", fileName),
HttpMethod.GET,
requestCallback,
responseExtractor
);
return ResponseEntity
.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s", fileName))
.body(response);
}