Develop Reference

Response returning status but not data

I have Jersey endpoint that is attempting to create a CSV file and return it via GET; however, all these attempts end up with the same result - the ok status (200) is returned but no file.
#GET
#Path("/research")
#Produces(MediaType.TEXT_PLAIN)
public Response dumpAllResearchAndSupportData() {
QuickDumpBuilderService dumpBuilderService = getQuickDumpService();
List<DevelopmentProposal> researchOtherProposals = dumpBuilderService.getAllResearchRecords();
String csvLocation = dumpBuilderService.buildCSV(researchOtherProposals);
File file = new File(csvLocation);
String filename = "reserach_" + UUID.randomUUID().toString() + ".csv";
return Response.ok(file, MediaType.TEXT_PLAIN).header("Content-Disposition", "attachment; filename=" + filename).build();
}
My CSV file is properly created, but it fails to be returned along with the response.
Notice above, I'm writing the CSV to a temporary location in my tomcat folder and then passing the path to that file back and then attempting to read that from the location here.
Another attempt with the same result where instead of writing the CSV to temp location, I'm just trying to write the ByteArrayOutputStream to the response object.
#GET
#Path("/research")
#Produces(MediaType.APPLICATION_JSON)
public Response dumpAllResearchAndSupportData() {
QuickDumpBuilderService dumpBuilderService = getQuickDumpService();
// Step 1. Retrieve all research and other proposals
List<DevelopmentProposal> researchOtherProposals = dumpBuilderService.getAllResearchRecords();
// Step 2. Create CSV File
ByteArrayOutputStream csvBaos = dumpBuilderService.buildCSV(researchOtherProposals);
// Step 3. Create spreadsheet
ByteArrayOutputStream excelBaos = dumpBuilderService.createExcelSpreadsheet(csvBaos, servlet);
// Step 4. Return spreadsheet
Response.ResponseBuilder response = Response.ok(excelBaos);
return response.build();
Another attempt where I added this "writer" into the response. This attempt generated an error that a "MessageBodyWriter for the ByteArrayStream was not found." That prompted the attempt below.
#GET
#Path("/research")
#Produces(MediaType.TEXT_PLAIN)
public Response dumpAllResearchAndSupportData() {
....
// Step 4. Return spreadsheet
return Response.ok(getOut(csvBaos.toByteArray())).build();
}
private StreamingOutput getOut(final byte[] csvBytes) {
return new StreamingOutput() {
#Override
public void write(OutputStream out) throws IOException, WebApplicationException {
out.write(csvBytes);
out.flush();
}
};
}
I've looked at the following similar answers, and I've attempted all of them with no success.
Not sure what I'm doing wrong - I suspect that it's something to do with how I setup my endpoint and defer to the Java REST API experts here.
File download/save from a Jersey rest api call using ANGULARJS . file is attached with response as "Content-Disposition"
Download CSV file via Rest
jersey - StreamingOutput as Response entity
Thanks for your help.

Spring Boot File Download Progress

I have a small problem regarding a download controller I want to implement. I want to make files accessible through an url, therefore I am using a spring boot controller. Because there could be several data types, I use the Apache Tika lib to determine the correct media type. I am currently bound to use the JAX-RS Requests. To save the byte array to a File, I use guava.
#GET
#Path("/getMedia")
public Response downloadMedia(#QueryParam("company") final String company,
#QueryParam("username") final String username,
#QueryParam("password") final String password,
#QueryParam("messageId") final int messageId) throws IOException {
ApplicationContext ctx = CyMega.getInstance().getContext(company);
if (ctx == null) {
return Response.status(Response.Status.UNAUTHORIZED).entity("Company not found").build();
}
ExternalAuthenticationService extService = ctx.getBean(ExternalAuthenticationService.class);
ExternalAuthObj authObj = extService.validateLogin(company, username, password);
if (authObj.getCode() != 0) {
return Response.status(Response.Status.UNAUTHORIZED).entity(authObj)build();
}
ChatService chatService = ctx.getBean(ChatService.class);
ChatGroupMessage message = chatService.getSingleChatGroupMessage(messageId);
if (message != null) {
Byte[] blobs = chatService.getBlob(messageId);
byte[] blob = ArrayUtils.toPrimitive(blobs);
File file = new File(message.getFilename());
Files.write(blob, file);
TikaConfig config = TikaConfig.getDefaultConfig();
Detector detector = config.getDetector();
Metadata metadata = new Metadata();
ByteArrayInputStream inputStream = new ByteArrayInputStream(blob);
TikaInputStream tikaInputStream = TikaInputStream.get(inputStream);
metadata.add(TikaCoreProperties.RESOURCE_NAME_KEY, message.getFilename());
org.apache.tika.mime.MediaType mediaType = detector.detect(tikaInputStream, metadata);
return Response.status(Response.Status.OK)
.header("Content-Length", String.valueOf(blob.length))
.header(HttpHeaders.CONTENT_TYPE, mediaType)
.entity(file).build();
} else {
return Response.status(Response.Status.NOT_FOUND).build();
}
}
By using this code, the file successfully will be send to the client. The only problem is that I also want to track the progress of the download. I tried defining the Content-Length, but this causes my request to load way longer than necessary. Is there any way to achieve that? Should I use a ByteArrayStream as response entity? It would be awesome if someone could provide an example on how to do that properly.
Thanks in advance!

ConnectionPoolTimeoutException: Timeout waiting for connection from pool putObject () s3Client Java

I'm uploading image files to s3 using the s3 aws client in my java application, but sometimes I've been getting the error
ERROR 9 --- [io-8080-exec-27] b.c.i.h.e.handler.HttpExceptionHandler : com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
but I haven't identified the reason for this error to occur and what solution I need to implement. I observed in the documentation the implementation of a ClientConfiguration setMaxConnections and passing it to the AmazonS3ClientBuilder object, but I believe that this would be increasing the problem and not actually correcting it, would I be correct?
I did not find detail why this problem with connection pooling occurs when using putObject (), if someone knows the reason or can explain through my implementation why this problem occurs. In our application there is also a configuration for SQS Config for queues
S3Config
public class S3Config {
#Bean
public AmazonS3 s3client() {
return AmazonS3ClientBuilder.standard()
.build();
}
}
Service Upload
public List<String> uploadImage(Long id, List<MultipartFile> files) throws Exception {
Random rand = new Random();
Product product = this.productService.findById(id);
List<String> imgPath = new ArrayList<>();
for (MultipartFile file : files) {
String name = (product.getName() + this.brandService.findBrandById(product.getBrand()).getName() + rand.nextInt(999999)).replaceAll(" ", "-");
String fullPath = this.s3Service.uploadImageFile(
file,'.' + Objects.requireNonNull(file.getOriginalFilename()).split("\\.")[1],
name,
awsBucketProperties.getName(),
awsBucketProperties.getEndpoint());
imgPath.add(this.utils.removeImageDomain(fullPath));
}
return imgPath;
}
Service S3
public String uploadImageFile(final MultipartFile file, final String ext, final String filename, final String bucketName, final String bucketEndpoint) throws IOException {
byte[] imageData = file.getBytes();
InputStream stream = new ByteArrayInputStream(imageData);
String s3FileName = filename + ext;
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(stream.available());
try {
s3client.putObject(new PutObjectRequest(bucketName, s3FileName, stream, metadata)
.withCannedAcl(CannedAccessControlList.PublicRead));
} catch (AmazonClientException ex) {
ex.printStackTrace();
}
return String.format("%s/%s", bucketEndpoint, s3FileName);
}

Url to download file from GCS bucket still accessible after deleting using Java

So I have problem when deleting file from GCS bucket, I create my file using java, the code is like:
public void upload(String projectId, String bucketName, String filePath, String fileName)
throws IOException, URISyntaxException {
File f = new File(gcsCredDirectory+gcsCredFileName);
if (!f.exists()) {
f.mkdirs();
}
try(InputStream is = new FileInputStream(f)) {
StorageOptions storageOptions = StorageOptions.newBuilder()
.setProjectId(projectId).setCredentials(fromStream(is)).build();
Storage storage = storageOptions.getService();
BlobId blobId = BlobId.of(bucketName, fileName);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
Blob result = storage.create(blobInfo, Files.readAllBytes(Paths.get(filePath)));
URL url = storage.signUrl(blobInfo, MAX_EXPIRED_DATE, TimeUnit.DAYS, SignUrlOption.withV4Signature());
} catch (Exception e) {
LOGGER.error("ERROR at GoogleCloudStorageServiceImpl.upload cause : ", e);
throw e;
}
}
The code to create went well, I get the Url to download the file I uploaded and actually can download the file, but after I deleting the file through this code:
public boolean delete(String projectId, String bucketName, String fileName) {
File f = new File(gcsCredDir+gcsCredFileName);
if (!f.exists()) {
f.mkdirs();
}
try(InputStream is = new FileInputStream(f)) {
StorageOptions storageOptions = StorageOptions.newBuilder()
.setProjectId(projectId)
.setCredentials(fromStream(is))
.build();
boolean result = storageOptions.getService().delete(bucketName, fileName);
LOGGER.info("Object " + fileName + " was deleted from " + bucketName);
return result;
} catch (Exception e) {
return false;
}
}
I was able to see the log Object + fileName + was deleted from + bucketName, but when I access the Url to download the file, I can still download it. I expect the download should failed because the file was deleted.
Any advice?
Thank you

Google has its own caches which will store what you upload for some time after you delete it. You need to override the settings using Headers on upload. Set Cache-Control: max-age=0, no-cache. You can also specify public or private.
public means intermediate servers may cache the result (for faster response times).
private means only the requesting client may cache the response, but not intermediate servers. This is usually set to enable a client to get a fresh copy each time the request is made.
To try and force the cache to drop the data, some servers accept PURGE requests. These can be issued via curl -vv -X PURGE http(s)://example.com/path/to/resource
Edit:
You can set the cache control headers using gsutil: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata

Multipart File Upload via REST API corrupts file and increases file size

Am uploading a file (any format doc, docx, pdf, text, etc) as multipart form/data to a REST API from Postman or application UI. The text file uploads fine. All other non-text formats get corrupted. I cant open those files.
The size of the uploaded file increases drastically. Check the following server log:
File size just before call to request.getRequestDispatcher().forward(): 27583
Controller, first line in method:39439
The size of the uploaded file is 27.3Kb
I am guessing the files gets corrupted because of the other data appended to the file.
Controller method is
#RequestMapping(value="/entity/{entity}/{entityId}/type/{type}/{derive}",method = RequestMethod.POST)
#ResponseBody
public String uploadFile(#RequestParam("file") MultipartFile multipartFile,#PathVariable("entity")String entity,#PathVariable("type")String type,#PathVariable("entityId")Long entityId,#PathVariable("derive") boolean derive) throws Exception
Since text file is saving correctly and other files also get written correctly, don't think the code to write the file is incorrect.
Code to get inputStream
public String storeFile(MultipartFile multipartFile, String entity, Long id, String uploadType, boolean isDerive,String customName)
throws Exception
{
try
{
System.out.println(multipartFile.getSize());
String fileName = "";
String contentType = "";
if (multipartFile != null)
{
fileName = multipartFile.getOriginalFilename();
contentType = multipartFile.getContentType();
if (contentType == null)
{
contentType = "application/msword";
}
}
InputStream is = multipartFile.getInputStream();
String filePath = getFileName(entity, uploadType, id, fileName, isDerive,customName);
Helper.storeFile(is, filePath);
precedingPath = precedingPath.length() > 0 ? precedingPath + "/":"";
return precedingPath + filePath;
}
catch (WebException e)
{
e.printStackTrace();
throw e;
}
catch (Exception e)
{
e.printStackTrace();
throw new WebException(e.getMessage(), IHttpConstants.INTERNAL_SERVER_ERROR, e);
}
}
Helper.storeFile
public static File storeFile(InputStream is, String filePath) throws IOException {
try {
String staticRepoPath = null;
if (MasterData.getInstance().getSettingsMap().containsKey(Settings.REPO_LOCATION.toString())) {
staticRepoPath = MasterData.getInstance().getSettingsMap().get(Settings.REPO_LOCATION.toString());
} else {
throw new WebException("Invalid Settings");
}
byte[] buffer = new byte[is.available()];
is.read(buffer);
File targetFile = new File(staticRepoPath + File.separator + filePath);
#SuppressWarnings("resource")
OutputStream outStream = new FileOutputStream(targetFile);
outStream.write(buffer);
return targetFile;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
My Ajax request is as follows
var fd = new FormData();
//Take the first selected file
fd.append("file", document.actualFile);
//Generic AJAX call
CandidateService.ajax_uploadDocumentWithDocType($scope.candidate.id, fd, document.docType, function (error, json)
Content type while uploading:
var config = {headers:{'X-Auth-Token':authToken, 'Content-Type': undefined}, transformRequest: angular.identity};
Would anyone know how I can fix this and upload the file successfully?
Q1) Why does the file size change between the request dispatcher and the controller that handles the file data.
Q2) Could this change of file size be the cause of file corruption? Libre Office cause General Input/Output Error.

I figured the problem with the file upload. I had a spring filter in between that was changing the request to a wrappedRequest. This was adding additional data to the multipart data and causing the file to be corrupted.

Well in my case I had this exact same problem when accesing the API through Amazon API Gateway. Turned out I forgot to allow multipart ContentType on API Gateway.
Kind of weird the requests still made it to my server and text files worked fine.