I'm uploading image files to s3 using the s3 aws client in my java application, but sometimes I've been getting the error
ERROR 9 --- [io-8080-exec-27] b.c.i.h.e.handler.HttpExceptionHandler : com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
but I haven't identified the reason for this error to occur and what solution I need to implement. I observed in the documentation the implementation of a ClientConfiguration setMaxConnections and passing it to the AmazonS3ClientBuilder object, but I believe that this would be increasing the problem and not actually correcting it, would I be correct?
I did not find detail why this problem with connection pooling occurs when using putObject (), if someone knows the reason or can explain through my implementation why this problem occurs. In our application there is also a configuration for SQS Config for queues
S3Config
public class S3Config {
#Bean
public AmazonS3 s3client() {
return AmazonS3ClientBuilder.standard()
.build();
}
}
Service Upload
public List<String> uploadImage(Long id, List<MultipartFile> files) throws Exception {
Random rand = new Random();
Product product = this.productService.findById(id);
List<String> imgPath = new ArrayList<>();
for (MultipartFile file : files) {
String name = (product.getName() + this.brandService.findBrandById(product.getBrand()).getName() + rand.nextInt(999999)).replaceAll(" ", "-");
String fullPath = this.s3Service.uploadImageFile(
file,'.' + Objects.requireNonNull(file.getOriginalFilename()).split("\\.")[1],
name,
awsBucketProperties.getName(),
awsBucketProperties.getEndpoint());
imgPath.add(this.utils.removeImageDomain(fullPath));
}
return imgPath;
}
Service S3
public String uploadImageFile(final MultipartFile file, final String ext, final String filename, final String bucketName, final String bucketEndpoint) throws IOException {
byte[] imageData = file.getBytes();
InputStream stream = new ByteArrayInputStream(imageData);
String s3FileName = filename + ext;
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(stream.available());
try {
s3client.putObject(new PutObjectRequest(bucketName, s3FileName, stream, metadata)
.withCannedAcl(CannedAccessControlList.PublicRead));
} catch (AmazonClientException ex) {
ex.printStackTrace();
}
return String.format("%s/%s", bucketEndpoint, s3FileName);
}
Related
I have a small problem regarding a download controller I want to implement. I want to make files accessible through an url, therefore I am using a spring boot controller. Because there could be several data types, I use the Apache Tika lib to determine the correct media type. I am currently bound to use the JAX-RS Requests. To save the byte array to a File, I use guava.
#GET
#Path("/getMedia")
public Response downloadMedia(#QueryParam("company") final String company,
#QueryParam("username") final String username,
#QueryParam("password") final String password,
#QueryParam("messageId") final int messageId) throws IOException {
ApplicationContext ctx = CyMega.getInstance().getContext(company);
if (ctx == null) {
return Response.status(Response.Status.UNAUTHORIZED).entity("Company not found").build();
}
ExternalAuthenticationService extService = ctx.getBean(ExternalAuthenticationService.class);
ExternalAuthObj authObj = extService.validateLogin(company, username, password);
if (authObj.getCode() != 0) {
return Response.status(Response.Status.UNAUTHORIZED).entity(authObj)build();
}
ChatService chatService = ctx.getBean(ChatService.class);
ChatGroupMessage message = chatService.getSingleChatGroupMessage(messageId);
if (message != null) {
Byte[] blobs = chatService.getBlob(messageId);
byte[] blob = ArrayUtils.toPrimitive(blobs);
File file = new File(message.getFilename());
Files.write(blob, file);
TikaConfig config = TikaConfig.getDefaultConfig();
Detector detector = config.getDetector();
Metadata metadata = new Metadata();
ByteArrayInputStream inputStream = new ByteArrayInputStream(blob);
TikaInputStream tikaInputStream = TikaInputStream.get(inputStream);
metadata.add(TikaCoreProperties.RESOURCE_NAME_KEY, message.getFilename());
org.apache.tika.mime.MediaType mediaType = detector.detect(tikaInputStream, metadata);
return Response.status(Response.Status.OK)
.header("Content-Length", String.valueOf(blob.length))
.header(HttpHeaders.CONTENT_TYPE, mediaType)
.entity(file).build();
} else {
return Response.status(Response.Status.NOT_FOUND).build();
}
}
By using this code, the file successfully will be send to the client. The only problem is that I also want to track the progress of the download. I tried defining the Content-Length, but this causes my request to load way longer than necessary. Is there any way to achieve that? Should I use a ByteArrayStream as response entity? It would be awesome if someone could provide an example on how to do that properly.
Thanks in advance!
i have set on my properties like this
gcp.storage.json-path=${GOOGLE_APPLICATION_CREDENTIALS}
and have code like this
public CloudStorageService(
#Value("${gcp.storage.bucket-name-file}") String fileBucketName,
#Value("${gcp.storage.bucket-name}") String bucketName,
#Value("${gcp.storage.json-path}") String jsonPath
) {
this.fileBucketName = fileBucketName;
this.bucketName = bucketName;
this.jsonPath = jsonPath;
log.info("fileBuckerName: {}", fileBucketName);
log.info("bucketName: {}", bucketName);
log.info("jsonPath: {}", jsonPath);
}
#PostConstruct
void init() {
try {
GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonPath))
.createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/devstorage.read_write"));
// storage = StorageOptions.getDefaultInstance().getService();
but when i deploy my code on kubernetes i cannot upload my file to GCS. my question is how to read data from my kubernetes workload identity using java ?
So I have problem when deleting file from GCS bucket, I create my file using java, the code is like:
public void upload(String projectId, String bucketName, String filePath, String fileName)
throws IOException, URISyntaxException {
File f = new File(gcsCredDirectory+gcsCredFileName);
if (!f.exists()) {
f.mkdirs();
}
try(InputStream is = new FileInputStream(f)) {
StorageOptions storageOptions = StorageOptions.newBuilder()
.setProjectId(projectId).setCredentials(fromStream(is)).build();
Storage storage = storageOptions.getService();
BlobId blobId = BlobId.of(bucketName, fileName);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
Blob result = storage.create(blobInfo, Files.readAllBytes(Paths.get(filePath)));
URL url = storage.signUrl(blobInfo, MAX_EXPIRED_DATE, TimeUnit.DAYS, SignUrlOption.withV4Signature());
} catch (Exception e) {
LOGGER.error("ERROR at GoogleCloudStorageServiceImpl.upload cause : ", e);
throw e;
}
}
The code to create went well, I get the Url to download the file I uploaded and actually can download the file, but after I deleting the file through this code:
public boolean delete(String projectId, String bucketName, String fileName) {
File f = new File(gcsCredDir+gcsCredFileName);
if (!f.exists()) {
f.mkdirs();
}
try(InputStream is = new FileInputStream(f)) {
StorageOptions storageOptions = StorageOptions.newBuilder()
.setProjectId(projectId)
.setCredentials(fromStream(is))
.build();
boolean result = storageOptions.getService().delete(bucketName, fileName);
LOGGER.info("Object " + fileName + " was deleted from " + bucketName);
return result;
} catch (Exception e) {
return false;
}
}
I was able to see the log Object + fileName + was deleted from + bucketName, but when I access the Url to download the file, I can still download it. I expect the download should failed because the file was deleted.
Any advice?
Thank you
Google has its own caches which will store what you upload for some time after you delete it. You need to override the settings using Headers on upload. Set Cache-Control: max-age=0, no-cache. You can also specify public or private.
public means intermediate servers may cache the result (for faster response times).
private means only the requesting client may cache the response, but not intermediate servers. This is usually set to enable a client to get a fresh copy each time the request is made.
To try and force the cache to drop the data, some servers accept PURGE requests. These can be issued via curl -vv -X PURGE http(s)://example.com/path/to/resource
Edit:
You can set the cache control headers using gsutil: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata
I am trying to read a file from aws s3 bucket and set it as resource inside my spring batch reader class. When I test the application on aws lambda function I got below error. any suggestion experts?
Caused by: java.lang.IllegalStateException: InputStream has already been read - do not use InputStreamResource if a stream needs to be read multiple times
at org.springframework.core.io.InputStreamResource.getInputStream(InputStreamResource.java:97) ~[task/:na]
at org.springframework.batch.item.file.DefaultBufferedReaderFactory.create(DefaultBufferedReaderFactory.java:34) ~[task/:na]
at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:266) ~[task/:na]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:146) ~[task/:na]
Class to read from s3 bucket
#Service
public class S3BucketProcessing {
private static final AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();
public InputStreamResource readFile() throws IOException{
String bucketName = "mybuckey";
String key = "File.txt";
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
return new InputStreamResource(object.getObjectContent());
}
Spring batch reader class
#Component
public class MyReader extends FlatFileItemReader<MyEntity> {
MyLineMapper mapper;
MyTokenizer tokenizer;
S3BucketProcessing s3BucketProcessing;
#Autowired
public MyReader(MyTokenizer tokenizer, MyLineMapper mapper, S3BucketProcessing s3BucketProcessing) throws Exception{
LOG.info("CardCustomerNotificationReader constructor");
this.mapper = mapper;
this.tokenizer = tokenizer;
this.s3BucketProcessing= s3BucketProcessing;
this.setResource(s3BucketProcessing.readFile());
mapper.setLineTokenizer(tokenizer);
this.setLineMapper(mapper);
}
}
The docs suggest using ByteArrayResource to cache the content in memory, rather than InputStreamResource.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/core/io/InputStreamResource.html
Just change the returns part like this:
//As suggested by berzerk
byte[] content = IOUtils.toByteArray(object.getObjectContent());
//Then
return new ByteArrayResource( content );
Instead of returning InputStreamResource , you shud return content of the stream may be byte[ ].
byte[] content = IOUtils.toByteArray(object.getObjectContent());
return content ;
So I had to send in JSON/XML object as Output Stream. I was using the InputStreamResource and was getting the same error as OP.
Here's the solution that worked for me.
#Override
public Resource dataExportForFieldExtractorModel() {
ObjectMapper xmlMapper = new XmlMapper().enable(SerializationFeature.INDENT_OUTPUT);
byte[] data;
Resource resource = null;
try {
data = xmlMapper.writerWithDefaultPrettyPrinter().writeValueAsBytes(new DataExportResponse());
resource = new ByteArrayResource(data);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return resource;
}
As the resource needs to be byte[], I changed resource = new InputStreamResource(new ByteArrayInputStream(data)); to
resource = new ByteArrayResource(data);.
I have some code that attempts to upload images to an s3 bucket. All of them are around 100-200kb.
However, after a few attempts uploads I always get the following stacktrace:
com.amazonaws.AmazonClientException: Unable to execute HTTP request:
Timeout waiting for connection from pool at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:713)
at
com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:453)
at
com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:415)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:364)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3964)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1538)
If I do GetObject on my code, I do not get that problem, only on uploads.
Code is the following:
public PutObjectResult uploadImage(String key, InputStream inputStream, ObjectMetadata metadata) {
Optional<String> bucketName = propertyResolver.instance().value("s3.bucket.url");
String resourcePath = BASE_PATH + key;
PutObjectRequest request = new PutObjectRequest(bucketName.get(), resourcePath, inputStream, metadata);
PutObjectResult result;
try {
result = s3Client.putObject(request);
} catch (AmazonClientException amazonClientException) {
amazonClientException.printStackTrace();
}
return result;
}
I've attempted to find a solution online but all I could find were issues regarding the GetObject and not consuming the response properly.