How can we use minio storage as same as S3. Is there any need to change the code of java spring boot?. Previous codes are aws related. I don't want to change the code, But i want to access the storage from another source. Is that possible with minio.
I assume, that you are using the aws-java-sdk s3client. Therefore you just need to set the endpoint-configuration (for example http://localhost:9000, if minio runs on port 9000 on your local machine)
For more infos about the endpoint-configuration, you can look here
Related
I am following: AWS Docs to setup the credentials. The problem is that it requires me to create a .aws folder in the machine. I want to get the keys and other secrets from a custom location. How can it be achieved?
P.S. If I follow the tutorial recommendation then all machines running the project would have to setup that .aws folder which would be a big hassle for everyone.
Where exactly would you suggest getting the credentials from? You could store them somewhere else, like a HashiCorp Vault server, and write a script or something to pull the values and set them as environment variables, but then you'll need to figure out how to give each computer secure credentials to access the Vault server.
If by "custom location" you simply mean a different local file system location, like a mapped drive or something, then you can specify that using the AWS_CREDENTIAL_PROFILES_FILE environment variable. Although it sounds like you want to do this on multiple people's workstations, and I would caution against sharing credentials files in that scenario. You really want to assign each person different AWS access keys so that you can track each person's AWS API actions, and revoke one person's access if they leave the company or something.
I recommend reading this page for understanding all the options to configure credentials for the AWS SDK.
Assuming you are using Amazon EC2 to host your application, then you can use IAM role to grant permissions, by attaching IAM role to your EC2 instances.
Furthermore, using IAM role avoid storing sensitive credential file in your instances.
Read this document, or watch this video to implement it.
My java service will run on my computers (let's say I'll have more than 1000 computers) and will send some data to S3. I use AWS Java SDK for it.
If I'm right, for doing it I need to use access key & secret key on my computers. (let's say it will be in .aws/credential file)
I read a lot of AWS documentation about the best practices for resources programmatic access, but still can't understand it.
Rotating access keys. After an access key is rotated, how can I change it in all applications that run my computers? Should my application be self-updated?
Temporary credentials. In this approach I still need to have access key & secret key on my computers. If yes, I have the same problem as in Q1.
Can somebody advise me what the best way and secure to programmatically access AWS resources in my situation? What do I need to do with access key & secret key?
Thank you.
UPDATES:
Computers are in different networks
Java app sends to S3 and also reads from S3
New computers can be added every time
The computers will need AWS credentials to talk with S3.
The simplest way is to store the credentials on each computer. However, as you say, it makes it hard to rotate the keys.
Another option is to store the credentials in a database that they can access, so they always get the latest credentials. However, they will need some sort of login to access the database.
Alternatively, you could setup identity federation, so that that the computers can authenticate against something like Active Directory, and then you can write a central service that will provide temporary credentials to each computer.
The process is basically:
The computers authenticate to AD
They call your service and prove that they are authenticated to AD
Your service then calls STS and generates temporary credentials valid for up to 36 hours
It provides those credentials to the computers
See: GetFederationToken - AWS Security Token Service
AFAIK you need to ensure that your application on computer has up-to-date access key. My recommendation is to store the access key on centralized place from which application will retrieve it. Thus, once you rotate the key and update the centralized storage, it will be reflected in all your application instances.
The AWS Java SDKs use a credential chain. The credential chain just means the SDK will look for credentials in 6 different places in this order:
Java system properties–aws.accessKeyId and aws.secretAccessKey. The AWS SDK for Java uses the SystemPropertyCredentialsProvider to load these credentials.
Environment variables–AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK for Java uses the EnvironmentVariableCredentialsProvider class to load these credentials.
The default credential profiles file– The specific location of this file can vary per platform, but is typically located at ~/.aws/credentials. This file is shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.
You can create a credentials file by using the aws configure command provided by the AWS CLI. You can also create it by editing the file with a text editor. For information about the credentials file format, see AWS Credentials File Format.
Amazon ECS container credentials– This is loaded from Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set. The AWS SDK for Java uses the ContainerCredentialsProvider to load these credentials.
Instance profile credentials– This is used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service. The AWS SDK for Java uses the InstanceProfileCredentialsProvider
to load these credentials.
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/credentials.html
I'm trying to practice using AWS more and I'm at a point where I can generate a S3 bucket URL. Now that I have that set up I'm trying to put a document (file) into that URL. Is there any useful documentation or things I should know when I try to do that? I can't seem to find anything on the web for Java users. (maybe I'm just bad at searching idk). Thanks
There is AWS SDK for Java provided by Amazon
AWS SDK For Java
Example work with S3
AWS example of loading file to s3:
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-objects.html#upload-object
and you would need a token to access you account resources from a java client more info here : https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/
The question is how can I set application secrets to make them available in application.yml?
On heroku I was doing it simply, by setting environment variable for dyno, and acces it as:
server:
port: ${PORT}
security:
user:
password: ${USERPASSWORD}
eureka:
client:
register-with-eureka: false
fetch-registry: false
instance:
hostname: localhost
securePortEnabled: true
password: ${EUREKAPASSWORD}
How to achieve that in Google App Engine? I was trying with datastore:
Unfornately I don't know how to inject those values into my *.yml file.
EDIT:
One more important thing to add. I am using maven appengine plugin to deploy my app via CI pipeline, so there is no possibility for me to push app.yaml file to App Engine
If you want to store secrets that are available to the app at runtime, keeping them in the datastore isn't a bad idea. I know of many apps that do that.
Here's an app used by the Khan Academy that's a good example of storing secret credentials in the datastore. It's in Python, but you can get the general idea. Note that on first admin login, it prompts for secrets to store.
Google has also a tutorial on how to store encrypted secrets.
https://cloud.google.com/kms/docs/store-secrets
TLDR: a separate bucket to store the encrypted secrets, instances download it when needed, decrypt using Google KMS (https://cloud.google.com/kms/) and remove afterwards.
The best and secure way is to use GCP KMS or some third party secrets manager product like vault.
GCP KMS
We need to use a service account with encrypt and decrypt permission(role) to encrypt the credentials(secrets) file.
Upload the encrypted credential file to GCS
Fetch the encrypted credential from GCS and decrypt and parse it(E.g. parse to plain java object) at runtime in your application code.
Datastore
Yes. We can store credentials/secrets environment variables into datastore and fetch them at runtime in application code.
Pros:
Simple
It can be used almost everywhere, GAE standard environment, GAE flexible environment, GCE, GCF, GKE, Cloud Run.
Cons:
Security is not as good as KMS.
GCE metadata
I used to use GCE metadata server to store my secret environment variables.
Pros:
It supports GAE, GCE, GKE.
Very simple. We just need to send HTTP requests to http://metadata.google.internal/computeMetadata/v1/ endpoint to fetch our custom metadatas(the secrets environment variables).
Cons:
Last year, GCE metadata doesn't support Cloud Function. (runtime: nodejs10).I can't fetch my custom secrets environment variables from GCE metadata within cloud function. But built-in metadatas can be fetched, like projectId.
security is not as good as KMS.
configmap and secrets(Only for GKE)
Simple base64 encryption is possible. Medium difficulty to use. Security is not as good as KMS.
Another hack way
I also create a post for this question here: How to pass system environment variables to app.yaml?
Yes, the Linux script way can do everything. But I don't like these hack way.
Is it possible to access localstorage object using Java directly? If yes how?
Update: I am aware that localstorage is client side and java is server side but I read on net that GWT have api that allow to read localstorage.
The local storage, as its name indicates, stores information locally, which means in the browser, at client-side. The servlet filter executes at server-side. There is no way to access the local storage at server-side.
If you need to access some ID stored in the local storage from a servlet filter, then retrieve this ID from the local storage in JavaSCript, and send a request containing this ID to the server.
Localstorage is better alternative thamn Cookie. So, I am waiting JAVA support without external library for accessing local web storage, too. Now I am using sellenium support. Please, look at https://gist.github.com/dariodiaz/5079695