I'm developing an budgeting application that uses a set of tokens and secret keys to access an external financial service, where a token-secret pair map to a single account. In this system, a user can have multiple accounts and therefore multiple sets of token-secret pairs. The token and secret can be used to access the transactions for an account, which means that the token-secret pairs should be securely stored (and be guarded against nefarious access or tampering). Although these pairs should be securely stored, the external financial service API requires the token and secret in plaintext.
What is a secure technique for storing these credentials at rest but providing the external service API with the original, plaintext credentials?
My application is a REST-based Spring Boot application written in Java 9+. Although there have been other answers I've seen on this topic, many of them are specific to Android and use Android security techniques (as are thus not applicable to my application). This application also uses Spring Data MonogDB to store other non-sensitive information, but if another technology is required for satisfying the above security requirements, I am open to suggestions.
This is not a problem that can be solved inside Java.
The token and secret can be used to access the transactions for an account, which means that the token-secret pairs should be securely stored (and be guarded against nefarious access or tampering).
The fundamental issue is who you are securing this against:
If you are trying to secure it against the people who manage the platform, it is pretty much unsolvable1.
If you are trying to secure it against "ordinary" (i.e. non-privileged) users, then you can either rely on ordinary file system security (plus standard service "hardening"), or you can use something like a Spring Vault or a Hardware Security Module if local file system security is insufficient2.
If you are trying to secure against a hacker who might be able to acquire the privilege of a full administrator, this is probably unsolvable too. (Though a hacker may need to be sophisticated ... )
Note that you can do things like saving the secrets in a Java keystore, but in order to do that the JVM needs the secret key to the keystore in order to get the secret. And where do you store that secret?
Aside:
... many of them are specific to Android and use Android security techniques (as are thus not applicable to my application).
Those techniques typically assume that the platform itself is secure, and / or that it hasn't been "rooted" by the user.
1 - So, if your goal is to embed some secrets in an app that you give to a client to use ... give up. If that problem was solvable, priracy of software, music, videos, etcetera would have a simple and reliable technological solution. Clearly ... it hasn't.
2 - For a lot of purposes, it isn't sufficient. However, the decision should be based on an assessment the risks, and balancing risks versus the cost / severity of the consequences of security failure.
Related
Let's say we have a very simple Java application, that edits resources on remote servers, that it authenticates with using Access Tokens. Application always uses the same identity, so it is always using the same client id, secret and refresh token to obtain access token.
The whole authentication process is supposed to go through without user intervention and app should perform actions automatically triggered by the user from another application. The other app is sending HTTP requests, but the whole thing would only be accessed in internal network and there would be no "legal" way to access it outside of it.
Is there a way to keep this data (refresh token, client id, secret...) securely within my application?
I have seen similar questions, but they all talked about websites and cookies, but this is supposed to happen under the hood, without any frontend etc. so I don't think those apply to my issue.
Edit: the application will be deployed on an internal server so it's not a Desktop solution. Basically there is an internal app that will send HTTP request to mine, triggering edit on a remote server that is outside of the internal network.
It is not a good idea to store client secrets, access tokens, refresh tokens etc in persistence storage unless it is stored in a secret store (like Vault). But there are other options.
If you are using Spring then you can use Spring OAuth2RestTemplate or else you can write something similar by looking at the code.
It acquires or renews an access token transparently and caches to avoid round trips to Authorization server.
The simplest option is to use memory storage, but if that diesn't work because you need to deal with restarts etc, operating systems provide per-user secure storage. This is a model sometimes used by OAuth desktop or console clients:
Credential Manager on Windows
Keychain on macOS
Passwords and Keys on Linux
It would require some native interop to interact with these credential stores, via use of a library such as java-keytar.
DESKTOP EXAMPLE
For something to compare against see these resources of mine:
Node.js desktop keytar code
This blog post has some related screenshots towards the end
I am wondering if its safe enough to put the AWS Api Credentials (or credentials in general) in a .conf file? I dont think so and want to ask if there is any easy to use approach to encrypt and decrypt the credentials. I am using JAVA with Eclipse IDE. Does anyone have a hint for a newbie in this section?
What is the ".conf" file? Is it a configuration file for an application? If that's the case, then no, it's not safe enough.
Amazon has a document describing best practices for access keys. Their recommendation for applications is to attach a role to whatever is running the application (EC2 / ECS / Lambda).
The biggest benefit of using a role is that the credentials it provides are temporary: they have a maximum lifetime of 12 hours, and a default lifetime of 1 hour. So if someone manages to extract them from your server, they won't be able to do long-term damage (unlike the "permanent" access keys associated with users).
While that helps, you still need to restrict the scope of those credentials (ie, don't use wildcards in their permission policies). And you should be monitoring your deployment for invalid credential use (ie, an API call that is made from somewhere outside of your VPC).
If you're talking about credentials for development use, AWS already has a place to store those. However, you don't need to store credentials anywhere if you use AWS Single-SignOn, which gives you limited-lifetime credentials for CLI/SDK use.
And if you're talking about credentials for a mobile application, look into Cognito and AWS Amplify.
We have run into a requirement to encrypt certain sensitive data before storing it in a PostgreSQL database. Our application is a Spring app running on top of Tomcat. We need to store the keys so that our web app can encrypt data before inserting it and decrypt data after retrieving it.
We would like to avoid storing this type of thing in our config files (or files that are filtered by Maven) since those files are checked into source control and readable by anyone with access. We seem to have the same issue with database credentials but it looks like JNDI datasources might solve those.
Are there best practices for this when using Tomcat? Things like environment variables might work but would be cumbersome to maintain.
EDIT: To clarify, we're trying to prevent sensitive data loss due to unauthorized access to the actual database file. Our db lives on a different physical box from our app server so we feel comfortable keeping the keys on the server that the application server lives on.
As far as I have seen, all the best practices around credentials optimize ease-of-use (for the operations team) and code simplification (for the development team), rather than optimize for security. This is very disappointing.
I would state the requirement as: Do not ever store any credentials or encryption keys in an unencrypted form, whether embedded in properties/config files, source code, or files managed separately from the application.
Some effects of this requirement:
JNDI data sources do not work for DB credentials, because the configuration of the JNDI data source resides in a configuration file in the container.
Passwords/passphrases for encryption/decryption (e.g., for the data in a database) cannot reside in the code or a properties file.
You must implement a way for the operations team to pass the secrets to the application as it is instantiated.
How might you design a system to meet this requirement?
Store all of your secrets in an encrypted file in a centralized location (e.g., in Amazon S3 or a Hardware Security Module). The passphrase/credentials to read this encrypted data must never be stored digitally - it is provided (e.g., in the body of a POST) when the app spins up. In addition to the passphrase to the secrets file, you may also need to dynamically provide credentials to get access to the file itself.
When the application receives the passphrase to the secrets file, it cracks it open and initializes it's internals (e.g., create DB data sources, initialize the encryption services, etc.).
You don't need encryption. You need access control.
Best practice? Yes. Cheap? No:
Don't use a single web server user account to log in to the db. This allows privilege escalation and data theft via SQL Injection. Use a unique connection per user. This kills connection pooling and allows maybe 500 simultaneous users max per db box.
Use a database that supports Row Level Security and Column Level Security. Right now, this is just Oracle. PG9.5 will have it. You can mimic it in 9.4
Put the db in its own network zone. Only the web server can connect.
Use a DAM (database activity monitor) to look for bad queries in and too much data out
A guide: https://securosis.com/assets/library/reports/Securosis_Understanding_DBEncryption.V_.1_.pdf
I have read a lot of about the way to provide acces to a REST API and I still cannot come with a decision what to use.
In my case I am writing a REST API that will be used by the users of the mobile application(android&iOS), thus I do not provide or require access from third parties and this makes me think that I don't have to use OAuth.
However I have considerations about how to provide access of one user's account from multiple devices and how to provide offline access.
Another consideration I have is how should I restrict the API access, for example if using API Tokens what are the best practices for expiration and renewal of the tokens?
You have several topics in your question:
What are the benefits of OAuth2 for an internal API exposed on the Internet?
How should I manage tokens?
How can a user gain access via multiple devices?
How can a user have offline access?
I discuss these questions below.
Oauth2
OAuth2 offers a standardized protocol for several authentication schemes of varying complexity. One of the most complex use cases is the 'Authorization Code Grant' flow which allows a resource owner (user) to grant specific access to a client application via an intermediary, the Authorization server. This is what happens when you 'login using google'. The advantage of using OAuth2 over a homebrew solution is that the protocol is clear to all parties and less likely to contain fundamental flaws. A drawback can be that the protocol is not that flexible so some custom scenario's might be hard to support within the boundaries of OAuth2. If you don't have the immediate need for any of the typical OAuth2 scenario's (or a stakeholder demanding use of OAuth2) then I suggest not starting off with it, but to implement a simple token scheme yourself.
Managing tokens
The most common way to manage API access is by using tokens. A token is generated when the user logs in, typically with username and password over HTTPS. The token is persisted on the server and must be supplied by the app in each request. This is similar to the session ID used in web applications which is automatically generated and handled in-memory by the application container on the server and passed via a cookie or request parameter. An API token is typically handled by the security layer of the application itself, persisted in the database and passed via the 'Authorization' header.
A token should have an expiration date. One should decide on the best interval for this and whether token renewal is automatic (each time the user accesses the API) or explicit (force the user to re-enter credentials after expiration). This depends on the type of application and the level of security required. Tokens can also be revoked manually on the server.
Multiple devices
Each token can be associated with a specific user and device to allow access on multiple devices. This means each device must be uniquely identified, typically with the IMEI code. This makes it easy to revoke all tokens for a specific device or user at once.
Offline access
The typical way to offer offline access is to cache relevant data on the device. For example the Google Maps app allows you to make specific regions of the map available offline. To avoid (too) stale data you could keep track of the token's expiration date and invalidate the cached data after this date. An issue to be aware of is the handling of offline edits by the user. These edits have to be processed when the device comes online again. When simultaneous edits on the same data are encountered a strategy is needed to resolve the conflict, e.g.:
one edit overrides the other depending on the type of edit or the role of the user
the last edit is ignored or offered for resolution to the last editor
some types of edits might be 'merged' automatically
etc.
Another nice and simple strategy is to disallow all edits whilst offline.
There are 2 things you want to protect / authenticate
That the client app is authorized to use the service
That the user is authorized to access personal data
App authentication
A mobile application is an untrusted client. Even if you gave nobody access to the app source you must expect that any kind of authorization secret or mechanism is unsafe and can come from a hacked app or other malicious tool that emulates the behaviour of your apps.
For authenticating the app, all you can do is to have a client id, but not a client secret. E.g.
http://service.com/rest?client_id=android
Reply method(String client_id) {
if (!client_id in ["andoid", "ios"])
return Unauthorized();
}
You can change that schema to something a little harder to guess but anything you do boils down to the same security level.
User authentication
Protecting user data is crucial and luckily possible. The key difference is that the secret is not statically hardcoded into the app, it is only known to the user.
One "easy" way to authenticate users is to use other accounts they have. Schemas like http://openid.net/connect/faq/ allow you to do exactly that.
You basically delegate the authentication to some other service. and get a (per service) unique user id which you can use in your code as key to all user data. An attacker can not forge this since your server can authenticate that the token is valid by asking another service. Looks roughly like
http://service.com/rest?client_id=android&user_token=aasjkbn9nah9z23&user_auth_service=facebook
Reply method(String client_id, user_token, user_auth_service) {
if (!client_id in ["andoid", "ios"])
return Unauthorized();
authenticated_user_id = user_auth_service.getUserIdOrFail(user_token);
accessDatabase(authenticated_user_id);
}
An attacker can still use your service from some evil app but there is no way to access accounts he has no access to anyways.
And if you hardcode access tokens into the app, you better don't expire them or make sure to handle that case specifically in the app somehow. There are always users with outdated app versions.
I am currently in the early stages of creating a web application (HTML5, JS, etc.) that uses a REST API on the backend (Java, specifically Jersey v1.18). The nature of the data that will be stored is highly sensitive, so security is something that I’ve started looking at, even though the application is only in the early stages. The eventual goal will be to have native mobile applications as well, and to potentially provide access to the data for external clients via the same API.
In my research thus far, I have identified a variety of authentication methods, including HTTP Basic, token-based, session cookie, OAuth, HMAC, etc. The key component here is that the REST API will be primarily accessed by users, rather than other applications or backends. Thus, having a “login/logout” equivalent is important, and this boils down to user level authentication.
So far, HMAC authentication looks to be the most promising, as we have no plans to integrate with any OAuth provider at this moment.
I have already read through dozens of SO posts, as well as articles such as:
http://www.thebuzzmedia.com/designing-a-secure-rest-api-without-oauth-authentication/
http://www.errant.me.uk/blog/2013/04/authenticating-restful-web-applications/ (note: this is clearly bad, as salting with the username is not recommended)
Ideally, HMAC seems like the way to go, but I have yet to see a recommended approach to handling the shared secret. The idea of using a resource to validate credentials, which then provides a token/nonce to be used with the HMAC scheme, seems to be an option, but I’m questioning the advantages over just using this token/nonce strictly as a token.
I know that HMAC authentication for a REST API has been discussed at length, but when used in conjunction with the authentication details that users have come to expect (username, email, password, etc.), is there any recommended approach that doesn’t require a pre-shared secret key?
This is primarily an opinion-based question, but I'll offer my two cents: just go for a session cookie.
If your primary audience is humans, and you don't need to integrate with third parties, don't bother with OAuth. Just make sure your API is only available over HTTPS, and issue a session token that the server can revoke after login. Strictly speaking it doesn't need to be a cookie; I've seen APIs that stash the token in HTML5 session storage and provide it in the Authorization header or as a query param.
If you have SSL set up properly, your users will get the expected padlock in the browser, and you'll be safe from anyone in between you and the client. If the client is compromised, you're screwed anyways. And since the client can't keep a secret, there aren't a lot of advantages to more complex HMAC schemes.