I am new to Dropwizard. I want to implement AWS S3 File upload service in my project.
I am not getting any tutorial to upload file on AWS S3 through dropwizard.
I have added following dependecies in pom.xml
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.9.28.1</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-multipart</artifactId>
<version>2.17</version>
</dependency>
I have registered MultiPartfeature.class in Application class's run() method as -
environment.jersey().register(MultiPartFeature.class);
Then in resource class defined method as -
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Path("/updateProfilePicture")
public String updateProfile(#FormDataParam("file") InputStream fileInputStream,
#FormDataParam("file") FormDataContentDisposition contentDispositionHeader) throws Exception {
String url = "";
AmazonS3 s3client = new AmazonS3Client(new BasicAWSCredentials("MY-ACCESS-KEY", "MY-SECRET_KEY"));
try {
File file = new File(contentDispositionHeader.getFileName());
PutObjectResult putObjectResult = s3client.putObject(new PutObjectRequest("BUCKET-NAME", s3SourceFactory.getSecretAccessKey(), fileInputStream, new ObjectMetadata()));
} catch (AmazonServiceException ase) {
ase.printStackTrace();
} catch (AmazonClientException ace) {
ace.printStackTrace();
}
return url;
}
But at run-time it shows the following log -
com.amazonaws.services.s3.AmazonS3Client: No content length specified for stream data. Stream contents will be buffered in memory and could result in out of memory errors.
How can I get the url of uploaded file? How to check file is uploaded through coding? Am I missing anything? Does anybody have any idea about this. If there any tutorial available with dropwizard, it will be helpful.
Thanks in advance
If the access key and secret key are correct. My guess is towards the S3 bucket permissions, once you go to your s3 bucket on aws console, to the right top you will find "properties" , once you open that you will have permissions make sure you gave an entry for your server there.
Related
I am trying to download documents/files store in google drive using google drive API version (V2). though I am able to download files successfully except few that are failing with Bad request.
I have tried rest API approach
https://developers.google.com/drive/api/v2/reference/files/get
as well as native SDK method
https://developers.google.com/drive/api/v2/manage-downloads#download_a_file_stored_on_google_drive.
Code snipped to download non google file:
public InputStream download() throws IOException {
HttpRequest httpRequest = service.getRequestFactory().buildGetRequest(new GenericUrl(file.getDownloadUrl()));
try {
return httpRequest.execute().getContent();
} catch (IllegalArgumentException e) {
log.warn("File '{}' could not be downloaded, illegal argument: {}", file.toString(), e.getMessage());
throw e
} catch (HttpResponseException e) {
throw e;
} catch (Exception e) {
throw e;
}
}
and Native SDK method
public InputStream download() throws IOException {
return driveService.files()
.get(file.getId())
.executeMediaAsInputStream();
}
But nothing working and there is nothing in error logs except below:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
Bad Request
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1067)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380)
at com.google.api.services.drive.Drive$Files$Get.executeMedia(Drive.java:5765)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAsInputStream(AbstractGoogleClientRequest.java:523)
at com.google.api.services.drive.Drive$Files$Get.executeMediaAsInputStream(Drive.java:5760)
I have also reported this issue as bug : https://github.com/googleapis/google-api-java-client/issues/1910
Its a little hard to help as you still have not included a complete example. However by looking at this method you are calling by passing the DownloadUrl link I suspect you have forgotten to check if it is null. Not all files have a download Url link in some cases it is null. If I recall with the v2 api binary type files will not have a download link. All files that are not Google drive mime type will be considered binary type files. Google sheet will have a download url, xlsx will not.
new GenericUrl(file.getDownloadUrl()
May I suggest that you use the supplied download method rather than relying on the downloadUrl
String fileId = "0BwwA4oUTeiV1UVNwOHItT0xfa2M";
OutputStream outputStream = new ByteArrayOutputStream();
driveService.files().get(fileId)
.executeMediaAndDownloadTo(outputStream);
File.get responds with the download stream of the file itself. You don't need to do an export links on an xlsx file its already a binary file. You export a google sheet to xlsx. Try using the example given for all but internal google mime types those you should be preforming an export on.
For more information please consult Manage downloads
I strongly suggest that you switch to the V3 of the api soon.
I have a java application in which I would like to process around 10GB records of file and zip them to a single folder and upload to S3. Since the overall size is around 10GB I cannot add all the files in memory and then upload to S3, and hence I would need to create a zip file in S3 and update the contents of the zip file by partitioning my files. Is there any means by which I can update an existing zip file in S3 without downloading to my local folder?
You can use aws java sdk for it
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.398</version>
</dependency>
Create a amazon s3 client using following
BasicAWSCredentials credentials = new BasicAWSCredentials("access_key", "secret_key");
AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials)).build();
Create a TransferManager and set the MultipartUploadThresholdoad.
Amazon S3 impose minimum part size of 5mb, So we are using 5mb here. You can increase the size as per your requirement.
TransferManager tm = TransferManagerBuilder.standard()
.withS3Client(amazonS3)
.withMultipartUploadThreshold((long) (5 * 1024 * 1024))
.build();
Set yours S3 bucket name where you want to upload and keyName will used to name the uploaded file. tm.upload will start the uploading process in background.
String bucketName = "my-bucket";
String keyName = "mydata.zip";
File file = new File("path_to_file/mydata.zip");
Upload upload = tm.upload(bucketName, keyName, file);
waitForCompletion is the blocking call and will return result once function completes its execution to upload file to s3.
try {
upload.waitForCompletion();
} catch (AmazonClientException e) {
// ...
}
I am trying to create RESUfull web services with swagger documentation. After i googled and found the below mentioned two dependencies are the latest and both works fine.
<dependency>
<groupId>com.wordnik</groupId>
<artifactId>swagger-jersey2-jaxrs_2.10</artifactId>
<version>1.3.12</version>
</dependency>
OR
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-jersey2-jaxrs</artifactId>
<version>1.5.13</version>
</dependency>
I have created API for GET and POST request. and I am able to get the responses for both GET and POST.
Now my new requirement is to create file upload POST API. So i fowlloed the stranded coding proacties,
1) added MultipartFeature in web.xml from glassfish api
org.glassfish.jersey.media.multipart.MultiPartFeature
2) my POST API
**import org.glassfish.jersey.media.multipart.FormDataContentDisposition;
import org.glassfish.jersey.media.multipart.FormDataParam;**
#POST
#Path("/pdfupload")
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Produces(MediaType.APPLICATION_XHTML_XML)
#ApiImplicitParams({#ApiImplicitParam(dataType="file", paramType="formData", name="file")})
public Response fileUpload(#ApiParam(value="PDF File Upload", required=true) #FormDataParam("file") InputStream pdfInputStream,
#FormDataParam("file") FormDataContentDisposition fileDetails) {
String uploadSuccessful = null;
if (null != pdfInputStream) {
uploadSuccessful = "API successfully called "+ fileDetails.getFileName();
}
return Response.ok(uploadSuccessful).build();
}
I am facing two issue with this API now,
First, I am not getting choose file button in swagger documentation and I tried different approach to get the button #ApiImplicitParams.
Secondly, I am getting "The request sent by the client was syntactically incorrect" error response(400), I am not sure what is missing here.
Could you please someone help me this.
Appreciate your help.
I am trying to download a PDF file from a response of Java REST call after custom authentication check.
I can see downloaded file but it is empty file.
Below is my code snippet.
//Custom HTTPClient
HTTPAuthClient client = new HTTPAuthClient(url,username,password)
Request request = new Request(downloadURL); //I'm downloading file content of an URL.
Response response = client.executeGet(request);
String response1 = response.getResponseBody();
InputStream is = new ByteArrayInputStream(response.getBytes());
response.setContentType("Content-type",application/pdf); //here response is //javax.servlet.HttpServletResponse
response.setHeader("Content-Disposition","attachment;filename="myfile.pdf");
IOUtils.copy(is,response.getOutPutStream());
response.flushBuffer();
With this code I could download the file but when I open the file and verified there is no data.
As part of response body also I can see some data.
Could you please help me out where I'm doing mistake I tried many options but did not find solution.
How can you use setContentType like this
response.setContentType("Content-type",application/pdf);
If only one avalible param in this method is String void setContentType(String type) so your method should be:
response.setContentType("application/pdf");
Java Doc to be sure.
I want to send a folder or list of files in the particular folder as response to the client via REST using java. My server side is EJB.
I am trying with following code. But got FileNotFoundException
#GET
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getFiles() {
File file = new File("C:\Documents and Settings\Administrator\Desktop\MyFolder");
return Response.ok(file, MediaType.APPLICATION_OCTET_STREAM).build();
}
Tried with the following code. But got NullPointerException
#GET
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getFiles() {
File file = new File("C:\Documents and Settings\Administrator\Desktop\MyFolder");
ResponseBuilder response = Response.ok(file.listFiles());
response.header("Content-Disposition", "inline; filename=units.zip");
return response.build();
}
Working in a windows machine.
Can anyone give me any advice, or examples of working code to conform to?
Either use forward slashes or double backslashes in the file name.
Also, don't expect the folder to be zipped automagically. You'll have to create the ZIP archive and pass it to the Response.