Develop Reference

Spring Boot Configuration Class isn't Properly passing values from application.properties

I'm creating a spring boot application with the AWS SDK. I am having some troubles with my S3 configuration class. I originally had all of the access keys and secrets keys stored in the code, and am transitioning to having an IAM user credentials in application.properties that has access to secrets manager, and getting all of the credentials for services through that. For some reason, the values are not properly populating. I believe it's possible that the constructor is executing before the value injection. Not sure on the best solution to fix this, thanks in advance for any help.
Here's the code for the S3 config class –
#Configuration
public class S3Config {
private String accessKeyId;
private String secretAccessKey;
#Bean
public AmazonS3 amazonS3Client(#Value("${aws.access-key-id}") String accessKeyId, #Value("${aws.secret-access-key") String secretAccessKey) {
this.accessKeyId = accessKeyId;
this.secretAccessKey = secretAccessKey;
BasicAWSCredentials awsCredentials = getBasicAWSCredentials();
return AmazonS3ClientBuilder
.standard()
.withRegion("xx")
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.build();
}
private BasicAWSCredentials getBasicAWSCredentials() {
String secretName = "xx";
String region = "xx";
AWSSecretsManager client = AWSSecretsManagerClientBuilder.standard()
.withRegion(region)
.withCredentials(
new AWSStaticCredentialsProvider(
new BasicAWSCredentials(
accessKeyId,
secretAccessKey
)
)
)
.build();
String secret = "";
String decodedBinarySecret;
GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
.withSecretId(secretName);
GetSecretValueResult getSecretValueResult = null;
try {
getSecretValueResult = client.getSecretValue(getSecretValueRequest);
} catch (DecryptionFailureException e) {
// Secrets Manager can't decrypt the protected secret text using the provided KMS key.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (InternalServiceErrorException e) {
// An error occurred on the server side.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (InvalidParameterException e) {
// You provided an invalid value for a parameter.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (InvalidRequestException e) {
// You provided a parameter value that is not valid for the current state of the resource.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
} catch (ResourceNotFoundException e) {
// We can't find the resource that you asked for.
// Deal with the exception here, and/or rethrow at your discretion.
throw e;
}
if (getSecretValueResult.getSecretString() != null) {
secret = getSecretValueResult.getSecretString();
} else {
decodedBinarySecret = new String(Base64.getDecoder().decode(getSecretValueResult.getSecretBinary()).array());
}
JsonObject secrets = new Gson().fromJson(secret, JsonObject.class);
return new BasicAWSCredentials(secrets.get("accessKey").getAsString(), secrets.get("secretKey").getAsString());
}
And here's an example of me using the S3 config in a service class. This was all working before I introduced the application.properties file and #Value annotations.
#Service
public class FileService {
private AmazonS3 amazonS3Client;
private String bucketName;
private FileRepository fileRepository;
public FileService(AmazonS3 amazonS3Client, FileRepository fileRepository) {
this.amazonS3Client = amazonS3Client;
bucketName = "xx";
this.fileRepository = fileRepository;
}

No primary or single public constructor found for class java.io.File

I'm trying to pass a file to my springboot backend. (which will then be uploaded to an s3 bucket), but I'm receiving this error that I can't figure out.
The file itself will contain an array, of an array of strings
Error -
java.lang.IllegalStateException: No primary or single public constructor found for class java.io.File - and no default constructor found either
Data Source -
// Data saved to S3 bucket / downloadableData function
if (this.lng !== "0.000000" && this.lng !== "") {
this.locationData.push([`Longitude: ${this.lng}, Latitude: ${this.lat}, Uncertainty Radius: ${this.uncertainty_radius} meters, Address: ${this.place_name}, Source: TEXT`])
this.locationData = JSON.parse(JSON.stringify(this.locationData))
}
Axios Post -
downloadableData() {
const blob = new Blob([this.locationData], {type: 'application/json'});
const data = new FormData();
data.append("document", blob);
axios.post("http://localhost:8080/api/v1/targetLocation/uploadStreamToS3Bucket", blob)
},
Springboot method -
public void uploadStreamToS3Bucket(File locations) {
try {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(String.valueOf(awsRegion))
.build();
String bucketName = "downloadable-cases";
String fileName = connectionRequestRepository.findStream() +".json";
s3Client.putObject(new PutObjectRequest(bucketName, fileName, locations));
} catch (AmazonServiceException ex) {
System.out.println("Error: " + ex.getMessage());
}
}
Data example

I see you want to upload a file that contain JSON data in it. This can be done in a Spring BOOT app via logic like this.
<p>Upload images to an S3 Bucket. Each image will be analyzed!</p>
<form method="POST" onsubmit="myFunction()" action="/upload" enctype="multipart/form-data">
<input type="file" name="file" /><br/><br/>
<input type="submit" value="Submit" />
</form>
To handle this upload in a Spring Controller, you can use this logic:
// Upload a file to place into an Amazon S3 bucket.
#RequestMapping(value = "/upload", method = RequestMethod.POST)
#ResponseBody
public ModelAndView singleFileUpload(#RequestParam("file") MultipartFile file) {
try {
byte[] bytes = file.getBytes();
String name = file.getOriginalFilename() ;
// Put the file into the bucket.
s3Client.putObject(bytes, bucketName, name);
} catch (IOException e) {
e.printStackTrace();
}
return new ModelAndView(new RedirectView("photo"));
}
Now you have the byte array and file name. You can place this into an Amazon S3 bucket by using the AWS SDK for Java V2.
private S3Client getClient() {
Region region = Region.US_WEST_2;
S3Client s3 = S3Client.builder()
.credentialsProvider(EnvironmentVariableCredentialsProvider.create())
.region(region)
.build();
return s3;
}
// Places an image into a S3 bucket.
public String putObject(byte[] data, String bucketName, String objectKey) {
s3 = getClient();
try {
PutObjectResponse response = s3.putObject(PutObjectRequest.builder()
.bucket(bucketName)
.key(objectKey)
.build(),
RequestBody.fromBytes(data));
return response.eTag();
} catch (S3Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
return "";
}
Here is a complete document that shows this use case. This use case actually uses the Amazon Rekognition service to analyze photos in an Amazon S3 bucket; however, it still demonstrates how to successfully upload a file from your desktop to an Amazon S3 bucket Also - it's implemented using the AWS SDK For Java V2 - which is the version that Amazon Recommends.
Creating a dynamic web application that analyzes photos using the AWS SDK for Java

ConnectionPoolTimeoutException: Timeout waiting for connection from pool putObject () s3Client Java

I'm uploading image files to s3 using the s3 aws client in my java application, but sometimes I've been getting the error
ERROR 9 --- [io-8080-exec-27] b.c.i.h.e.handler.HttpExceptionHandler : com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
but I haven't identified the reason for this error to occur and what solution I need to implement. I observed in the documentation the implementation of a ClientConfiguration setMaxConnections and passing it to the AmazonS3ClientBuilder object, but I believe that this would be increasing the problem and not actually correcting it, would I be correct?
I did not find detail why this problem with connection pooling occurs when using putObject (), if someone knows the reason or can explain through my implementation why this problem occurs. In our application there is also a configuration for SQS Config for queues
S3Config
public class S3Config {
#Bean
public AmazonS3 s3client() {
return AmazonS3ClientBuilder.standard()
.build();
}
}
Service Upload
public List<String> uploadImage(Long id, List<MultipartFile> files) throws Exception {
Random rand = new Random();
Product product = this.productService.findById(id);
List<String> imgPath = new ArrayList<>();
for (MultipartFile file : files) {
String name = (product.getName() + this.brandService.findBrandById(product.getBrand()).getName() + rand.nextInt(999999)).replaceAll(" ", "-");
String fullPath = this.s3Service.uploadImageFile(
file,'.' + Objects.requireNonNull(file.getOriginalFilename()).split("\\.")[1],
name,
awsBucketProperties.getName(),
awsBucketProperties.getEndpoint());
imgPath.add(this.utils.removeImageDomain(fullPath));
}
return imgPath;
}
Service S3
public String uploadImageFile(final MultipartFile file, final String ext, final String filename, final String bucketName, final String bucketEndpoint) throws IOException {
byte[] imageData = file.getBytes();
InputStream stream = new ByteArrayInputStream(imageData);
String s3FileName = filename + ext;
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(stream.available());
try {
s3client.putObject(new PutObjectRequest(bucketName, s3FileName, stream, metadata)
.withCannedAcl(CannedAccessControlList.PublicRead));
} catch (AmazonClientException ex) {
ex.printStackTrace();
}
return String.format("%s/%s", bucketEndpoint, s3FileName);
}

Url to download file from GCS bucket still accessible after deleting using Java

So I have problem when deleting file from GCS bucket, I create my file using java, the code is like:
public void upload(String projectId, String bucketName, String filePath, String fileName)
throws IOException, URISyntaxException {
File f = new File(gcsCredDirectory+gcsCredFileName);
if (!f.exists()) {
f.mkdirs();
}
try(InputStream is = new FileInputStream(f)) {
StorageOptions storageOptions = StorageOptions.newBuilder()
.setProjectId(projectId).setCredentials(fromStream(is)).build();
Storage storage = storageOptions.getService();
BlobId blobId = BlobId.of(bucketName, fileName);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
Blob result = storage.create(blobInfo, Files.readAllBytes(Paths.get(filePath)));
URL url = storage.signUrl(blobInfo, MAX_EXPIRED_DATE, TimeUnit.DAYS, SignUrlOption.withV4Signature());
} catch (Exception e) {
LOGGER.error("ERROR at GoogleCloudStorageServiceImpl.upload cause : ", e);
throw e;
}
}
The code to create went well, I get the Url to download the file I uploaded and actually can download the file, but after I deleting the file through this code:
public boolean delete(String projectId, String bucketName, String fileName) {
File f = new File(gcsCredDir+gcsCredFileName);
if (!f.exists()) {
f.mkdirs();
}
try(InputStream is = new FileInputStream(f)) {
StorageOptions storageOptions = StorageOptions.newBuilder()
.setProjectId(projectId)
.setCredentials(fromStream(is))
.build();
boolean result = storageOptions.getService().delete(bucketName, fileName);
LOGGER.info("Object " + fileName + " was deleted from " + bucketName);
return result;
} catch (Exception e) {
return false;
}
}
I was able to see the log Object + fileName + was deleted from + bucketName, but when I access the Url to download the file, I can still download it. I expect the download should failed because the file was deleted.
Any advice?
Thank you

Google has its own caches which will store what you upload for some time after you delete it. You need to override the settings using Headers on upload. Set Cache-Control: max-age=0, no-cache. You can also specify public or private.
public means intermediate servers may cache the result (for faster response times).
private means only the requesting client may cache the response, but not intermediate servers. This is usually set to enable a client to get a fresh copy each time the request is made.
To try and force the cache to drop the data, some servers accept PURGE requests. These can be issued via curl -vv -X PURGE http(s)://example.com/path/to/resource
Edit:
You can set the cache control headers using gsutil: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata

How To upload Images/Files to Firebase storage in Java?

I am working in a functionality where i need to upload an Image/File to firebase storage using java and expose it as an API. I have already achieved this functionality in angular 4 typescript. But now i need this method as an Java Rest API, so that my peer can also consume the same method instead of writing a new one. So is there any API or methods to write the image to firebase storage ?

Try this:
FirebaseOptions options = FirebaseOptions.builder()
.setCredentials(credential)
.setDatabaseUrl(projectUrl)
.setStorageBucket("YOUR BUCKET LINK")
.build();
FirebaseApp fireApp = FirebaseApp.initializeApp(options);
StorageClient storageClient = StorageClient.getInstance(fireApp);
InputStream testFile = new FileInputStream("YOUR FILE PATH");
String blobString = "NEW_FOLDER/" + "FILE_NAME.EXT";
storageClient.bucket().create(blobString, testFile , Bucket.BlobWriteOption.userProject("YOUR PROJECT ID"));

If the Java project is running in a trusted environment (such as their development machine, a server you control, or Cloud Functions), they can use the Firebase Admin SDK to access Cloud Storage.
See the Firebase Admin SDK documentation on how to get started, and then the Google Cloud Storage documentation for Java clients for more. Specifically have a look at the sample of uploading a file in Java:
BlobId blobId = BlobId.of("bucket", "blob_name");
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("text/plain").build();
Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));

If you are using Spring Boot, you can try this:
Create a class to expose it as a web service in your API:
import com.yourcompany.yourproject.services.FirebaseFileService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
#RestController
public class ResourceController {
#Autowired
private FirebaseFileService firebaseFileService;
#PostMapping("/api/v1/test")
public ResponseEntity create(#RequestParam(name = "file") MultipartFile file) {
try {
String fileName = firebaseFileService.saveTest(file);
// do whatever you want with that
} catch (Exception e) {
// throw internal error;
}
return ResponseEntity.ok().build();
}
}
Create a service to upload the image to firebase storage.
#Service
public class FirebaseFileService {
private Storage storage;
#EventListener
public void init(ApplicationReadyEvent event) {
try {
ClassPathResource serviceAccount = new ClassPathResource("firebase.json");
storage = StorageOptions.newBuilder().
setCredentials(GoogleCredentials.fromStream(serviceAccount.getInputStream())).
setProjectId("YOUR_PROJECT_ID").build().getService();
} catch (Exception ex) {
ex.printStackTrace();
}
}
public String saveTest(MultipartFile file) throws IOException{
String imageName = generateFileName(file.getOriginalFilename());
Map<String, String> map = new HashMap<>();
map.put("firebaseStorageDownloadTokens", imageName);
BlobId blobId = BlobId.of("YOUR_BUCKET_NAME", imageName);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId)
.setMetadata(map)
.setContentType(file.getContentType())
.build();
storage.create(blobInfo, file.getInputStream());
return imageName;
}
private String generateFileName(String originalFileName) {
return UUID.randomUUID().toString() + "." + getExtension(originalFileName);
}
private String getExtension(String originalFileName) {
return StringUtils.getFilenameExtension(originalFileName);
}
}
Note you need to download Firebase config file and store it as "firebase.json" under the src/main/resources folder.
https://support.google.com/firebase/answer/7015592?hl=en
Also you need to add the Maven dependency:
<dependency>
<groupId>com.google.firebase</groupId>
<artifactId>firebase-admin</artifactId>
<version>6.14.0</version>
</dependency>

String blobString = DIR + fileIdWithExtension;
Blob blob = storageClient.bucket().create(blobString, file.getInputStream(), Bucket.BlobWriteOption.userProject(PROJECT_ID));
But keep in mind that you should initialize firebase. For example
private FirebaseApp initFirebase() {
FileInputStream serviceAccount;
try {
serviceAccount = new FileInputStream(fileUploadPath);
} catch (FileNotFoundException e) {
throw new FileStorageException(ErrorMessage.FILE_NOT_FOUND + "firebaseConfig.json");
}
FirebaseOptions options;
try {
options = new FirebaseOptions.Builder()
.setCredentials(GoogleCredentials.fromStream(serviceAccount))
.setStorageBucket(BUCKET_NAME)
.build();
} catch (IOException e) {
e.printStackTrace();
throw new FailedToSetCredentialsException(ErrorMessage.COULD_NOT_SET_CREDENTIALS);
}
return FirebaseApp.initializeApp(options);
}