I want to use certificate based authentication in AWS Lambda to generate oauth tokens. Currently I am storing the certificates and private keys locally and running it like a normal java application.
I am planning to use AWS Secrets manager to store these certificates and keys. However the issue is since we are using terraform to provision AWS resources, it seems like we will have to keep these certs and keys in our bitbucket repo which will have security risks. Is there any other way I can use these certificates in AWS lambda without actually storing them in bitbucket repo?
The Terraform aws_secretsmanager_secret_version resource takes a string value, but that doesn't mean you have to hard-code the string inside that resource. You need to think about how you can read that key value into Terraform and reference it inside the resource.
For example, that string could come from a local file, or an S3 object. Terraform could also generate the TLS key for you.
Related
My java service will run on my computers (let's say I'll have more than 1000 computers) and will send some data to S3. I use AWS Java SDK for it.
If I'm right, for doing it I need to use access key & secret key on my computers. (let's say it will be in .aws/credential file)
I read a lot of AWS documentation about the best practices for resources programmatic access, but still can't understand it.
Rotating access keys. After an access key is rotated, how can I change it in all applications that run my computers? Should my application be self-updated?
Temporary credentials. In this approach I still need to have access key & secret key on my computers. If yes, I have the same problem as in Q1.
Can somebody advise me what the best way and secure to programmatically access AWS resources in my situation? What do I need to do with access key & secret key?
Thank you.
UPDATES:
Computers are in different networks
Java app sends to S3 and also reads from S3
New computers can be added every time
The computers will need AWS credentials to talk with S3.
The simplest way is to store the credentials on each computer. However, as you say, it makes it hard to rotate the keys.
Another option is to store the credentials in a database that they can access, so they always get the latest credentials. However, they will need some sort of login to access the database.
Alternatively, you could setup identity federation, so that that the computers can authenticate against something like Active Directory, and then you can write a central service that will provide temporary credentials to each computer.
The process is basically:
The computers authenticate to AD
They call your service and prove that they are authenticated to AD
Your service then calls STS and generates temporary credentials valid for up to 36 hours
It provides those credentials to the computers
See: GetFederationToken - AWS Security Token Service
AFAIK you need to ensure that your application on computer has up-to-date access key. My recommendation is to store the access key on centralized place from which application will retrieve it. Thus, once you rotate the key and update the centralized storage, it will be reflected in all your application instances.
The AWS Java SDKs use a credential chain. The credential chain just means the SDK will look for credentials in 6 different places in this order:
Java system properties–aws.accessKeyId and aws.secretAccessKey. The AWS SDK for Java uses the SystemPropertyCredentialsProvider to load these credentials.
Environment variables–AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK for Java uses the EnvironmentVariableCredentialsProvider class to load these credentials.
The default credential profiles file– The specific location of this file can vary per platform, but is typically located at ~/.aws/credentials. This file is shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.
You can create a credentials file by using the aws configure command provided by the AWS CLI. You can also create it by editing the file with a text editor. For information about the credentials file format, see AWS Credentials File Format.
Amazon ECS container credentials– This is loaded from Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set. The AWS SDK for Java uses the ContainerCredentialsProvider to load these credentials.
Instance profile credentials– This is used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service. The AWS SDK for Java uses the InstanceProfileCredentialsProvider
to load these credentials.
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/credentials.html
I've read following tutorial:Vault Configuration
Ok we installed the Vault server and put 2 pairs of secret properties:
$ vault kv put secret/gs-vault-config example.username=demouser example.password=demopassword
$ vault kv put secret/gs-vault-config/cloud example.username=clouduser example.password=cloudpassword
Spring boot application has following properties(bootstrap.properties):
spring.application.name=gs-vault-config
spring.cloud.vault.token=00000000-0000-0000-0000-000000000000
spring.cloud.vault.scheme=http
spring.cloud.vault.kv.enabled=true
So based on spring.cloud.vault.token application able to read secure properties(name and password) but spring.cloud.vault.token is stored in the insecure place - bootstrap.properties which is stored in the code repository. Could you please explain why it is safe?
P.S.
As we found out it is insecure. How to make it secure ? I understand that there are might be several solutions to make it secure but single simplified example would be enough for me.
Could you please explain why it is safe?
The answer is that it is NOT safe ... if you do it that way. For example, the Spring Vault reference manual says:
"Consider carefully your security requirements. Static token authentication is fine if you want quickly get started with Vault, but a static token is not protected any further. Any disclosure to unintended parties allows Vault use with the associated token roles."
You should either protect your static tokens, or only grant them access to "secrets" in the vault which you are happy to be widely known.
Alternatively, have your application use an authenticated method to generate short-term dynamic tokens.
As I understand initial problem it is bad to store passwords in an application.properties file on Github.
And storing a static Vault token in an application.properties file on Github is equally as bad.
What is the difference ?
There is almost no difference1. This is simply the wrong way to use Vault.
1 - There is a small advantage in that you could invalidate the token if you discover it leaked by accident. But this doesn't mean that it is sensible to publish it deliberately.
So how do you do things securely?
First, you MUST secure the machines where the secrets are going to be used. Even if you are not going to store the actual secrets on disk, you will need to store a different secret (securely) on each of your machines so that they can authenticate themselves to the place where the real secrets are kept.
Here is an example using Chef.
Set up a secure Chef server that holds the configs for your machines; i.e. recipes for all of the things that need to be installed, node descriptions to say what recipes to apply, etc.
When you bootstrap a machine as a node, a keypair is generated for the machine and registered with the Chef server. The keypair is also held on the machine, and has to be held securely.
Then you use the Chef client to run the recipes that install and configure your server.
Note that this relies on having a properly secured system to run the Chef server. It also relies on each of the nodes being sufficiently secure to protect their own keys.
There are other ways to do this, but nothing will work if you cannot secure your host sufficiently.
Storing spring.cloud.vault.token in the application.properties that is checked into VCS (e.g. Git) might compromise all secrets stored in Vault.
The solution is not to store Vault token in application.properties as a plain text. There are several options.
Remove Vault token from application.properties
Simply remove spring.cloud.vault.token from application.properties and instead provide it via system property -Dspring.cloud.vault.token=00000000-0000-0000-0000-000000000000 (when starting the application) or environment variable SPRING_CLOUD_VAULT_TOKEN. Environment variable is especially convenient if you use containers (Docker or Kubernetes).
Store encrypted Vault token in application.properties
You can keep spring.cloud.vault.token property in application.properties if it is encrypted.
Spring Cloud Config supports decrypting properties with values starting with {cipher}:
spring.cloud.vault.token={cipher}encrypted_vault_token
To use properties encryption and decryption you will need the following dependencies (example for Gradle):
implementation 'org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE'
implementation 'org.bouncycastle:bcprov-jdk15on:1.64'
Symmetric encryption
The simplest way to encrypt the properties is to use symmetric encryption.
Come up with a symmetric key (for example s3cr3t).
To encrypt Vault token you can use Spring Boot CLI with Spring Boot Cloud CLI:
curl "https://repo.spring.io/release/org/springframework/boot/spring-boot-cli/2.2.2.RELEASE/spring-boot-cli-2.2.2.RELEASE-bin.tar.gz" -o spring-boot-cli-bin.tar.gz
tar -xf spring-boot-cli-bin.tar.gz
cd spring-2.2.2.RELEASE
bin/spring install org.springframework.cloud:spring-cloud-cli:2.2.1.RELEASE
bin/spring encrypt 00000000-0000-0000-0000-000000000000 --key s3cr3t
# 507cd1614682535ab8237b448ca73dc74058d3ae9145d63a7381ee67f3046eb1598da6960abdbf2dbf22c47206db5222e45fc74fd6122bc707b61c62f5051e0f
bin/spring decrypt 507cd1614682535ab8237b448ca73dc74058d3ae9145d63a7381ee67f3046eb1598da6960abdbf2dbf22c47206db5222e45fc74fd6122bc707b61c62f5051e0f --key s3cr3t
# 00000000-0000-0000-0000-000000000000
Pass the symmetric key to the application in ENCRYPT_KEY environment variable.
Symmetric encryption key must be never checked into a VCS.
Asymmetric encryption
Consider asymmetric encryption using public and private key pair as a more secure alternative to symmetric encryption.
Instead of a symmetric encryption key you need to generate a keystore (using the keytool utility that comes with the JDK or openssl).
In the bootstrap.properties specify the following properties:
encrypt.keyStore.location
encrypt.keyStore.password
encrypt.keyStore.alias
encrypt.keyStore.type
The keystore has to be mounted at the location specified in encrypt.keyStore.location and never checked into a VCS.
Also, password that unlocks the keystore better to pass in ENCRYPT_KEYSTORE_PASSWORD environment variable.
Read about key management in Spring Cloud Config.
Answering your questions
Could you please explain why it is safe?
It is not safe! Never put a plain secret in source control.
As we found out it is insecure. How to make it secure?
Some ways to increase security:
Use environment variables instead of properties file;
Restrict the access at the network level to the Vault server only to workload servers. This guarantees that nobody outside that network can exchange the token;
Vault works generating real but temporary credentials each time the token is used. Reduce the scope of the real credentials to read-only whenever be possible;
Rotate the token regularly.
Regarding the Spring specifics
The bootstrap properties should contain only non-critical properties. For critical properties, you can pass them to the application using environment variables.
spring.cloud.vault.token = ${SPRING_CLOUD_VAULT_TOKEN}
Summary
The question remains "Who guards the keys?". But the Vault Token serves actually to protect the real sensitive data. In case the vault token is leaked, you can invalidate just the token.
Improve the constraints from where applications can access the vault server and reduce the scope of the real credentials are additional ways to ensure that only the servers that run the application can exchange the tokens by the real credentials and the real credentials have the minimum privilege as possible.
There are lots of questions and articles on how to do this with .NET, but how/is it possible to easily authenticate for local development through Azure AD shared secret credential using Java (Spring Boot specifically)?
For .NET, it is as easy as specifying the RunAs=CurrentUser property in the connection string to connect to the Azure Key Vault (per this article: https://learn.microsoft.com/en-us/azure/key-vault/service-to-service-authentication), connecting automatically (assuming my account is listed in the access policy for the key vault I want to access). Ideally, I would not want to use a thousand Java dependencies to do this, I could manually obtain a token to authenticate, but it would be nice to save developers the hassle of having to manually obtain a token from Azure every time we wanted to test things for local development.
Thanks!
Here is an example os using MSICredentials Read Azure key vault secret through MSI in Java
Just try using the AzureCliCredentials instead https://azure.github.io/azure-sdk-for-java/com/microsoft/azure/credentials/AzureCliCredentials.html
I have some secret api keys for my server and i have to use in my project. But i am not feeling safe to keep in properties file or any physical location in server. Can any one tell me what are possible ways to keep it secret and use wherever it's require in application.
If you are really considered about the passwords you've got, the best way is using HSM (Hardware security module). This way you will be assured that keeping your private key in a secure place is not part of your consideration anymore.
If that is expensive, then you can think of storing your private key in JKS or PKCS12 and choose a strong password.
Generally you are looking for a vault. In PKI you can have a vault to store your secret in it. there are couple of ways I can think of. One of them is HSM as I described above and the other one is following:
For storing your password you need to salt it or store it in another server w/ salt and whenever you need the password, it's better to have a secure connection between your real server and password only stored server to request the password.
Also you can take a look at this project from Hashicorp
https://github.com/hashicorp/vault
There are many options here depending on your actual security requirements.
You've mentioned java and spring boot, so I'll mainly specify solutions related to this technology stack.
Since you're already using Spring Boot, consider spring-cloud-config server.
It has some interesting "encryption" features:
Step 1
Create a Key (public/private pair or shared depending on your requirements)
Specify it when starting the config server (via -- or env variable):
encrypt.key=ABC123ABC123ABC123
Step 2
Encode value of the password by calling a rest request of this server (you do it once)
curl localhost:8888/encrypt -d mysecretdbpassword
>>AZXCASDAZXC341234ZXCASDFedr453
Step 3
In application.properties specify that the value is "encrypted" with a special syntax
The file will be stored on a configuration server and not on the microservice. The
property will look like this
db.password={cipher}AZXCASDAZXC341234ZXCASDFedr453
The property will be kept encrypted in files but decrypted in memory. So the microservice will get a decrypted value.
The documentation is available here. This is by far the fastest way to implement the requirements. You can keep it in a dedicated microservice or embedded into the already-existing application.
In addition, it allows integration with JKS, so its possible to store passwords there.
Complementary/Alternatives
Spring cloud config is also integrated with Hashicorp Vault which can be a good solution for your situation on its own (even without spring boot cloud config).
Yet Another possible solution that comes from Spring Universe is called Spring Vault
Its intended for working with secrets.
I use onboardbase to hide secret i would recommend that they are few other out there to check like doppler
You can setup your application.properties as below
application.properties
secret.key1=${SECRET_KEY1}
secret.key2=${SECRET_KEY2}
While building the application we can pass it as an environment variable
mvn clean install -dSECRET_KEY1=123456 -dSECRET_KEY2=abcdef
We can pass the environment variables in docker as well by using -e SECRET_KEY1=123456
The question is how can I set application secrets to make them available in application.yml?
On heroku I was doing it simply, by setting environment variable for dyno, and acces it as:
server:
port: ${PORT}
security:
user:
password: ${USERPASSWORD}
eureka:
client:
register-with-eureka: false
fetch-registry: false
instance:
hostname: localhost
securePortEnabled: true
password: ${EUREKAPASSWORD}
How to achieve that in Google App Engine? I was trying with datastore:
Unfornately I don't know how to inject those values into my *.yml file.
EDIT:
One more important thing to add. I am using maven appengine plugin to deploy my app via CI pipeline, so there is no possibility for me to push app.yaml file to App Engine
If you want to store secrets that are available to the app at runtime, keeping them in the datastore isn't a bad idea. I know of many apps that do that.
Here's an app used by the Khan Academy that's a good example of storing secret credentials in the datastore. It's in Python, but you can get the general idea. Note that on first admin login, it prompts for secrets to store.
Google has also a tutorial on how to store encrypted secrets.
https://cloud.google.com/kms/docs/store-secrets
TLDR: a separate bucket to store the encrypted secrets, instances download it when needed, decrypt using Google KMS (https://cloud.google.com/kms/) and remove afterwards.
The best and secure way is to use GCP KMS or some third party secrets manager product like vault.
GCP KMS
We need to use a service account with encrypt and decrypt permission(role) to encrypt the credentials(secrets) file.
Upload the encrypted credential file to GCS
Fetch the encrypted credential from GCS and decrypt and parse it(E.g. parse to plain java object) at runtime in your application code.
Datastore
Yes. We can store credentials/secrets environment variables into datastore and fetch them at runtime in application code.
Pros:
Simple
It can be used almost everywhere, GAE standard environment, GAE flexible environment, GCE, GCF, GKE, Cloud Run.
Cons:
Security is not as good as KMS.
GCE metadata
I used to use GCE metadata server to store my secret environment variables.
Pros:
It supports GAE, GCE, GKE.
Very simple. We just need to send HTTP requests to http://metadata.google.internal/computeMetadata/v1/ endpoint to fetch our custom metadatas(the secrets environment variables).
Cons:
Last year, GCE metadata doesn't support Cloud Function. (runtime: nodejs10).I can't fetch my custom secrets environment variables from GCE metadata within cloud function. But built-in metadatas can be fetched, like projectId.
security is not as good as KMS.
configmap and secrets(Only for GKE)
Simple base64 encryption is possible. Medium difficulty to use. Security is not as good as KMS.
Another hack way
I also create a post for this question here: How to pass system environment variables to app.yaml?
Yes, the Linux script way can do everything. But I don't like these hack way.