Problem
We use java WAR files and keep config files in s3 buckets. Our environments: DEV,QA, Stage, and PROD each have their own config files and s3 buckets. If I add a new field, such as "Polling_RATE=5000", it must be manually added to each env because these config files also store passwords so they can not be tied to the application or kept inside Github. Not every engineer has access to each env so you must remember to inform the upper level engineers (DEVOPS) before the prod deployment date to add the new field for the application to work. Its a really messy process currently.
Question
Is there a utility or architectural design pattern meant to deal with this? How do you "version control" sensitive configuration fields that you can not store within github?
Recognizable problem.
Usually config fields with sensitive information like passwords change a lot less often than non-sensitive configuration fields. A possible solution is to split the config in two parts:
Config that's environment-specific but doesn't contain sensitive information. I would advise you to keep these files together with your source code and if possible, generate the files and automatically upload then to your configuration store (S3 in your case) at build time. They must be versioned and tied to the version of your application.
Config that contains sensitive information. Looking at the question, not all team members are allowed to read/write this information. You could store these in S3 with specific access rights so that only authorized members can access them. You would need a mechanism to join the files back together at deployment, or change the app to read from different config files.
However, this will only solve part of your problem. The ops guys will still need to perform changes when sensitive config keys change. Whether this is acceptable depends on how often sensitive config keys change.
An alternative to S3 could be to run a private Git repository (AWS's CodecCommit, for example). You'd have better version control and easier access for the devs to perform changes, since you're already using Git. You'll still have to fix the split access rights between dev and ops, or let that go (since DevOps is about trust and cooperation, that might be a good idea). You could apply a similar pattern here as I described above.
Another solution could be to move the configuration of sensitive values from property files to the system configuration. When you already use a provisioning system like Puppet or Chef, this will feel natural for the ops guys. Or set all sensitive values like passwords as environment variables and have the app read it as system properties.
Hope this helps!
We have been using dynamodb for keeping config values. The advantage with this approach is that the values are easily readable from console and validated.
Also another advantage is that we periodically check the values from dynamodb so if any value needs to be changed we just change it and the app automatically picks the new value instead of starting it again.
Sensitive values are stored encrypted using KMS keys and only the ec2 role that is running the application has right to decrypt using that Key.
We enhanced the Netflix archiaus project to fit our needs. May be you can check that out.
Related
So I read this question on Programmers SE and I got a little confused. In short, the solution is to keep all secret information in config files. Now this is where I'm confused. Couldn't a user just go searching for this file?
What methods are used to prevent users from finding the file? I'm using Java on Windows if that changes the answer at all.
I would think Encryption would come up but I'm not sure how that helps if the user can just decompile your source.
EDIT: To clarify further, my intention is to use API Keys (in this case 1, singular key) in an executable JAR file.
This depends what you are trying to achieve. I will assume here that the user has access to the enviroment in which your software is running. If you need to store some secret information, then you can use encryption, where the user has to supply a key. This is difficult to implement yourself correctly, but there are plenty of libraries and resources for this.
If your problem is exactly as described in the Programmers SE- you want to share code without sharing the 'secret' settings- extract the settings into a configuration file and don't share the configuration file. You can give a template, but with the secrets missing.
You can't really protect your informations if they are somehow available to the end-user - even if they are somehow crypted as long as your software is also available on the client. It only depends on the criminal energy the attacker will invest to get them.
One possibility could be to use ie. connection pooling for a database connect.
With this technique you get the connection object remote without the knowledge of the password - so you can use the connection within your app but the user cannot use informations to connect to the database with some sql tool.
See here for details: https://en.wikipedia.org/wiki/Connection_pool
Having read the link you provided I can think of the following below. Also please feel free to check the Security network (one interesting question here).
For a example in a database config file you could have an entry like
<%= ENV['USER_NAME'] %>
<%= ENV['PASSWORD'] %>
and those two variables are environment variables that you set.
Using encryption would be similar. You'd store an encrypted/hashed key in a config file and you use that data to see if there is a match when needed.
Encrypted data can be decrypted with a corresponding key
Cryptographically hashed data is very hard to crack
You'd need to design that appropriately whatever your requirements are.
And you can combine both methods too if you wish.
Example
I invite to have a look at my GitHub project's config folder. In particular the yaml files. I use the first method above mostly. It is a RoR project.
This question isn't in regards to how to get/read the properties files, but instead what is the standard or accepted location to keep them. Specifically, I've got two types of properties, one that is sort of a server config file and a couple that are written to and read from to contain information sent to/from a client. So where should these be placed?
What do you see as the options?
Config files should be on the classpath, unless they need to be trivially modifiable. If they do, then they should be in a location external to the app, and the location set by either default, a JNDI string, a system property, or whatever.
Files that are regularly updated as part of system operation must be external, and configured in the same way. Although I'd question whether or not it makes sense to do that kind of persistence with a file.
I am looking for a open-source solutions which allow hosting different properties for different applications and allow changes. On any change of properties it should notify or push the change to the application appropriately.
So, instead every application managing the properties in physical file and deploying physically; these properties can be pushed to a single system. User will have GUI to load and change the properties as per right. Should allow push as mentioned.
If you have already similar open source solutions in mind please advice.
Is this something that Puppet can manage for you?
I don't think what you've described (as nice as it would be) will be likely to exist in an app server. If a program is looking for a file, it's either going to load it with a FileReader (or similar), or it will use ClassLoader.getResourceAsStream(). It might be looking for data that is returned in properties, format, XML properties format, or even something completely different like RDF with which to configure itself. Also many programs read their config on start-up and then hold the values in memory, so you would still need to reboot them to get them to change.
To get something like this to work there would need to be a standard for configuration provisioning and live updates. Once that existed the webapp authors and server vendors would each need to add support for the standard.
If you are the one writing the programs to be managed however, then you can write your programs to request configuration from a service, and have a configuration push feature.... there may be packages out there that can speed up adding that to your code, but I get the impression you are looking to manage programs written by others.
Have you considered to use JMX? I think he could be a good starting point to implement your requirements.
MBeans's attributes can store your application's properties, the MBeanServer will allow you to make them available from remotting, JConsole offers you an GUI to update properties values.
You also can write within the MBean some code that notify the corrrespondig application when a user change any properties using the GUI.
My Java web application (myapp.war) ist deployed by placing it into the webapps direcotry on Tomcat on Ubuntu 10.04.
This application needs to save some data in files. But the user, which is running Tomcat (tomcat6) has no write access to the home directory /usr/share/tomcat6/ and no write access to the current working direcotry /var/lib/tomcat6/, since both belong to root.
So where should a web application store its data? I hope it is not the extracted archive in the webapps direcotry. This one could be deleted very easily by accident. And Tomcat can be configured, not to extract .war files. Then there would be no extracted direcotry.
Perhaps /var/lib/tomcat6/ should belong to user tomcat6 ant this is a bug in Ubuntu?
If the files need not persist longer than the life-cycle of the servlet context, the servlet container provides a private temporary directory for each servlet context, specified by javax.servlet.context.tempdir attribute.
See Servlet Specification 2.3, Chapter 3 Servlet Context
3.7.1 Temporary Working
Directories
The convenience of a temporary
storage directory is required for each
servlet context. Servlet containers
must provide a private temporary
directory per servlet context and
make it available via the
javax.servlet.context.tempdircontext
attribute. The object associated with
the attribute must be of
type java.io.File
Answering his own question, Witek stated /var/lib/tomcat6/webapps/ is writable -- at least on his installation of his version of Ubuntu. On my RHEL 5.2 system /var/lib/tomcat<X> doesn't even exist, so there is no webapps subdirectory writable or not, which leads to my answer.
Q: Where should a Java web application store its data?
A: Wherever you've configured it to store its data.
Make the location configurable, in web.xml as a <context-param> or in a myApplication.properties file.
I can put it where I want on my box, the SysAdmins can put it where they want on the production system.
You can change your mind later.
You don't need symbolic links (which have magically disappeared on me before, breaking the system.)
You can have several sets of test data, and just point the configuration at whichever one you want.
You can put it wherever there's disk space.
You are going to change your mind later.
I think it depends on what kind of data you are dealing with. Most of the time, data goes into the database simply because it is fast and easy to perform a CRUD. If you want to store localized user configuration and you don't care how portable it is, perhaps you can store under user.home, I did that for one of my projects and that works fine. All that being said, I really don't think there's any best practice on this and database seems to be the most obvious choice because you can do whole lot of different tasks against it, and most of them are free to begin with. :)
I found the solution on Launchpad. /var/lib/tomcat6/webapps/ is writable. This means, that the following works:
File myFile = new File("webapps/myfile.txt");
I haven't seen any specific guidance on where you should store that kind of data locally - probably because you'd normally store that kind of data in a database.
When I need to store quick-and-dirty data like that, I store it on a filesystem specifically for that data, to isolate it from other areas where it might get caught up in other activity. Haven't had any issues with the approach so far. Also, if it's important, make sure you store it somewhere where you're backing it up :)
Where do you store user-specific and machine-specific runtime configuration data for J2SE application?
(For example, C:\Users\USERNAME\AppData\Roaming</em> on Windows and /home/username on Unix)
How do you get these locations in the filesystem in platform-independent way?
First on the format:
Java property files are good for key/value pairs (also automatically handle the newline chars). A degree of structure is possible by using 'dot notation'. Drawback is that the structure does not allow you to easily enumerate top-level configuration entities and work in drill-down manner. Best used for a small set of often tweakable environment-specific settings
XML files - quite often used for more complex configuration of various Java frameworks (notably J2EE and Spring). I would advice that you at least learn about Spring - it contains many ideas worth knowing even if you decide not to use it. If you decide to roll your own XML configuration, I'd recommend using XStream with customized serialization options or if you just need to parse some XML, take a look at XOM. BTW Spring also allows you to plug your custom XML configuration language, but it's a relatively complex task. XML configuration is best used for more complex 'internal' configuration that is not seen or tweaked by the end user.
Serialized Java objects - a quick and easy way to persist the state of an object and restore it later. Useful if you write a configuration GUI and you don't care if the configuration is human readable. Beware of compatibility issues when you evolve classes.
Preferences - introduced in Java 1.4, allow you to store typed text, numbers, byte arrays and other primitives in platform-specific storage. On Windows, that is the registry (you can choose between /Software/JavaSoft/Prefs under HKLM or HKCU). Under Unix the same API creates files under the user home or /etc. Each prefs hive can be exported and imported as XML file. You can specify custom implementation of the PreferencesFactory interface by setting the "java.util.prefs.PreferencesFactory" JVM property to your implementation class name.
In general using the prefs API can be a good or a bad thing based on your app scenario.
If you plan to have multiple versions of the same code running on the same machine with different configuration, then using the Preferences API is a bad idea.
If you plan using the application in a restricted environment (Windows domain or tightly managed Unix box) you need to make sure that you have proper access to the necessary registry keys/directories. This has caught me by surprise more than once.
Beware from roaming profiles (replicated home dirs) they make up for some funny scenarios when more than one active machines are involved.
Preferences are not as obvious as a configuration file under the application's directory. most of the desktop support staff doesn't expect and doesn't like them.
Regarding the file layout of the prefs it again depends on your application. A generic suggestion is:
Package most of your XML files inside application's JAR either in the root or under /META-INF directory. These files will be read-only and are considered private for the application.
Put the user modifiable configuration under $APP_HOME/conf . It should consist mainly of properties files and occasionally a simple XML file (XStream serialization). These files are tweaked as part of the installation process and are usually not user serviceable.
Under the user-home, in a dot-directory (i.e. '~/.myapplication') store any user configuration. The user configuration may override the one in the application conf directory. Any changes made from within the application go here (see also next point).
You can also use an $APP_HOME/var directory to store any other mutable data which is specific to this application instance (as opposed to the user). Another advantage of this approach is that you can move and backup the whole application and it's configuration by simple copy of one directory.
This illustrates some standard techniques for managing configuration. You can implement them using different libraries and tools, starting from raw JRE, adding Spring/Guice or going for a full J2EE container (possibly with embedded Spring)
Other approaches for managing configuration are:
Using multiple base directories for running multiple instances of the application using different configurations.
Using lightweight registries for centralized configuration management
A centrally managed Configuration Management Database (CMDB) file, containing the host-specific values for each machine is rsync-ed every night to all production hosts. The application uses a templated configuration and selects from the CMDB during runtime based on the current hostname.
That depends on your kind of J2SE Application:
J2SE executable JAR file (very simple): use user.home System property to find home-dir. Then make a subdir accordingly (like e.g. PGP, SVN, ... do)
Java Web Start provides very nice included methods to safe properties. Always user-specific
Finally Eclipse RCP: There you have the notion of the workspace (also derived from user.home) for users and configuration (not totally sure how to access that tricky in Vista) for computer wide usage
All these approaches are, when used with care -- use correct separatorChar -- OS neutral.
Java has a library specifically for doing this in java.util.prefs.Preferences.
Preferences userPrefs = Preferences.getUserNodeForPackage(MyClass.class); // Gets user preferences node for MyClass
Preferences systemPrefs = Preferences.getSysteNodeForPackage(MyClass.class); // Gets system preferences node for MyClass
Preferences userPrefsRoot = Preferences.getUserRoot(); // Gets user preferences root node
Preferences systemPrefsRoot = Preferences.getSystemRoot(); // Gets system preferences root node
I use this
String pathFile = null;
if(OS.contains("win")){
pathFile = System.getenv("AppData");
}else{
pathFile = System.getProperty("user.home");
}
I save the settings of my application here
C:\Users\USERNAME\AppData\ on windows
user.home (/home/USERNAME) on other platfroms
For user specific config, you could write a config file to the folder pointed to by the "user.home" system property. Would only work on that machine of course.
You might want to look at Resource Bundles.