I'm having a brainstorming issue on how to get user uploaded pictures viewed by only the friends of the users.
So what I've come up with so far is:
Create a DynamoDB table for each user, with a dynamic list of friends/new friends added.
Generate a Signed URL for every user-uploaded picture.
Allow access to the Signed URL to every friend listed in the DynamoDB table to view set picture/s.
Does this sound correct? Also, would I technically have just one bucket for ALL user uploaded pictures? Something about my design sounds off...
Can anyone give me a quick tutorial on how to accomplish this via Java?
There two basic approaches:
Permissions in Amazon S3, or
Application-controlled access to object in Amazon S3
Permissions in Amazon S3
You can provide credentials (either via IAM or Amazon Cognito) that allow users to access a particular path within an Amazon S3 bucket. For example, each user could have their own path within the bucket.
Your application would generate URLs that include signatures that identify them as that particular user and Amazon S3 would grant access to the objects.
One benefit of this approach is that you could provide the AWS credentials to the users and they could interact directly with AWS, such as using the AWS Command-Line Interface (CLI) to upload/download files without having to always go via your application.
Application-controlled access to object in Amazon S3
In this scenario, users have no permissions within Amazon S3. Instead, each time that your application wishes to generate a URL to an object in S3 (eg in an <img> tag), you created a pre-signed URL. This will grant access to the object for a limited time. It only takes a couple of lines of code and can be done within the application without communication with AWS to generate the URL.
There is no need to store pre-signed URLs. They are generated on-the-fly.
The benefit of this approach is that your application has full control over which objects they can access. Friends could share pictures with other users and the application would grant access, whereas the first method only grants access to objects within the user's specific path.
Related
I am following: AWS Docs to setup the credentials. The problem is that it requires me to create a .aws folder in the machine. I want to get the keys and other secrets from a custom location. How can it be achieved?
P.S. If I follow the tutorial recommendation then all machines running the project would have to setup that .aws folder which would be a big hassle for everyone.
Where exactly would you suggest getting the credentials from? You could store them somewhere else, like a HashiCorp Vault server, and write a script or something to pull the values and set them as environment variables, but then you'll need to figure out how to give each computer secure credentials to access the Vault server.
If by "custom location" you simply mean a different local file system location, like a mapped drive or something, then you can specify that using the AWS_CREDENTIAL_PROFILES_FILE environment variable. Although it sounds like you want to do this on multiple people's workstations, and I would caution against sharing credentials files in that scenario. You really want to assign each person different AWS access keys so that you can track each person's AWS API actions, and revoke one person's access if they leave the company or something.
I recommend reading this page for understanding all the options to configure credentials for the AWS SDK.
Assuming you are using Amazon EC2 to host your application, then you can use IAM role to grant permissions, by attaching IAM role to your EC2 instances.
Furthermore, using IAM role avoid storing sensitive credential file in your instances.
Read this document, or watch this video to implement it.
My java service will run on my computers (let's say I'll have more than 1000 computers) and will send some data to S3. I use AWS Java SDK for it.
If I'm right, for doing it I need to use access key & secret key on my computers. (let's say it will be in .aws/credential file)
I read a lot of AWS documentation about the best practices for resources programmatic access, but still can't understand it.
Rotating access keys. After an access key is rotated, how can I change it in all applications that run my computers? Should my application be self-updated?
Temporary credentials. In this approach I still need to have access key & secret key on my computers. If yes, I have the same problem as in Q1.
Can somebody advise me what the best way and secure to programmatically access AWS resources in my situation? What do I need to do with access key & secret key?
Thank you.
UPDATES:
Computers are in different networks
Java app sends to S3 and also reads from S3
New computers can be added every time
The computers will need AWS credentials to talk with S3.
The simplest way is to store the credentials on each computer. However, as you say, it makes it hard to rotate the keys.
Another option is to store the credentials in a database that they can access, so they always get the latest credentials. However, they will need some sort of login to access the database.
Alternatively, you could setup identity federation, so that that the computers can authenticate against something like Active Directory, and then you can write a central service that will provide temporary credentials to each computer.
The process is basically:
The computers authenticate to AD
They call your service and prove that they are authenticated to AD
Your service then calls STS and generates temporary credentials valid for up to 36 hours
It provides those credentials to the computers
See: GetFederationToken - AWS Security Token Service
AFAIK you need to ensure that your application on computer has up-to-date access key. My recommendation is to store the access key on centralized place from which application will retrieve it. Thus, once you rotate the key and update the centralized storage, it will be reflected in all your application instances.
The AWS Java SDKs use a credential chain. The credential chain just means the SDK will look for credentials in 6 different places in this order:
Java system properties–aws.accessKeyId and aws.secretAccessKey. The AWS SDK for Java uses the SystemPropertyCredentialsProvider to load these credentials.
Environment variables–AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK for Java uses the EnvironmentVariableCredentialsProvider class to load these credentials.
The default credential profiles file– The specific location of this file can vary per platform, but is typically located at ~/.aws/credentials. This file is shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.
You can create a credentials file by using the aws configure command provided by the AWS CLI. You can also create it by editing the file with a text editor. For information about the credentials file format, see AWS Credentials File Format.
Amazon ECS container credentials– This is loaded from Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set. The AWS SDK for Java uses the ContainerCredentialsProvider to load these credentials.
Instance profile credentials– This is used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service. The AWS SDK for Java uses the InstanceProfileCredentialsProvider
to load these credentials.
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/credentials.html
So we are planning to move static content to s3 for operational reasons. I just want to understand where to place s3 in the workflow of handling a request.
If website requires an image, should the request hit our service first which would fetch the image from s3 (reverse-proxy) or should client directly request the file.
How to hide file names ,pathnames and manage permissions in request for file?
Same questions applicable for uploading new content.
Handle s3 quota and parallel requests
I was going to comment, but this turned into a full answer instead...
Either. If your assets are public, the lowest-weight method is to just request them from a public S3 bucket. If they're not, though, it's probably easiest to use Cloudfront rather than rolling-your-own auth around S3 requests.
You can make it look like your asset A.jpeg in S3.yourBucket/A.jpeg is at yourWebsite.com/A.jpeg using Cloudfront. If you want to also obscure the filename A, you need to use e.g. API gateway to serve you the file without revealing anything about it to your front end. If it were me, I wouldn't bother.
Unless you absolutely have to, don't let users upload to the same bucket that other users download from. There are several approaches to uploads depending on the use-case. Pre-signed URL's are good for one-time use. You can also just provide the user with AWS credentials that are allowed to write-only to the upload bucket, by using Cognito.
There's no S3 quota. You get charged for reads and writes. For a simple site, these charges will be tiny. If you're worried, you can use Cloudfront to rate-limit your users. You can also use API Gateway to create limits for individual users. S3 is extremely parallelizable.
I want to create or update a image file with password protection. Scenario is, our infra team will upload an image file to AWS S3. Later we want to protect this image file with password from java. Password will be auto generated and will not be disclosed with anyone. If any one trying to download the image directly from AWS S3, it should not open. I have tried Server-Side encryption in AWS S3
CopyObjectRequest request = new CopyObjectRequest(bucket, key, bucket, key);
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setServerSideEncryption(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
request.setNewObjectMetadata(objectMetadata);
s3client.copyObject(request)
But still i'm able to open it. Is there any other way to do it.
Server-Side Encryption merely encrypts the data stored on disk. It is not a method for protecting access to data.
Rather, it appears that your requirement is:
Store some data (eg an image) on Amazon S3 and keep it private
Selectively allow people to download it if they have been authorized
The most suitable solution would be to use an Amazon S3 Pre-Signed URL.
By default, all objects in Amazon S3 are private. You can then add permissions so that people can access your objects. This can be done via:
Access Control List permissions on individual objects
A Bucket Policy (that grants wide-ranging access based on path, IP address, referrer, etc)
IAM Users and Groups (that grant permissions to Users with AWS credentials)
Pre-Signed URLs
A Pre-Signed URL can be used to grant access to S3 objects as a way of "overriding" access controls. A normally private object can be accessed via a URL by appending an expiry time and signature. This is a great way to serve private content without requiring a web server.
It would be the responsibility of your application to appropriately authenticate users to determine whether they are allowed access to objects in S3. If they are granted access, then your application should generate a pre-signed URL as an authenticated link to the objects. The URL will only be valid for a limited time duration.
This is best done by having a back-end app (probably running on Amazon EC2 or AWS Lambda) perform the authentication and then generate the URL. Your authenticated user can then use the pre-signed URL to download the object during the allocated time period (eg 5 minutes).
This method has several benefits over the use of a password:
It properly authenticates the user (through your code) rather than merely trusting anyone who knows the password
It allows you to log access, so you know who is accessing the object
Your back-end app could generate an HTML page willed with many pre-signed URLs and your users could simply click the links to access the objects, rather than having to provide a password for every object they wish to download
I have a website where users can upload resources (e.g. pdf-files) to their account. I am using AWS S3 to host all the uploaded files, and I am using the AWS Java SDK 1.8.9.1 for communications between my website and S3.
Now, I want to allow users to be able to download and view the files that they have uploaded, but I only want this to be possible through my website. That is, on my web site, users should have a download link for each of their files that they can click, after which the download starts. However, if they copy the URL of the download link and send it to their friend, that friend should not be able to download the file.
I know that it is possible to restrict access to S3 buckets to specified referring URLs. However, I have also been told that this can easily be forged and is not the way to go. I am thinking that there might exist a solution with signed requests.
How can I achieve this?
You could modify your application so that the download links are proxied through it. i.e. The application should do the reverse of the upload process.
So, you can provide a link to your java application, which will then go to S3 and retrieve the file and return it to the user. This way, if someone shares a link, you can protect the url and require users to login before they can download the file.