I am looking for a open-source solutions which allow hosting different properties for different applications and allow changes. On any change of properties it should notify or push the change to the application appropriately.
So, instead every application managing the properties in physical file and deploying physically; these properties can be pushed to a single system. User will have GUI to load and change the properties as per right. Should allow push as mentioned.
If you have already similar open source solutions in mind please advice.
Is this something that Puppet can manage for you?
I don't think what you've described (as nice as it would be) will be likely to exist in an app server. If a program is looking for a file, it's either going to load it with a FileReader (or similar), or it will use ClassLoader.getResourceAsStream(). It might be looking for data that is returned in properties, format, XML properties format, or even something completely different like RDF with which to configure itself. Also many programs read their config on start-up and then hold the values in memory, so you would still need to reboot them to get them to change.
To get something like this to work there would need to be a standard for configuration provisioning and live updates. Once that existed the webapp authors and server vendors would each need to add support for the standard.
If you are the one writing the programs to be managed however, then you can write your programs to request configuration from a service, and have a configuration push feature.... there may be packages out there that can speed up adding that to your code, but I get the impression you are looking to manage programs written by others.
Have you considered to use JMX? I think he could be a good starting point to implement your requirements.
MBeans's attributes can store your application's properties, the MBeanServer will allow you to make them available from remotting, JConsole offers you an GUI to update properties values.
You also can write within the MBean some code that notify the corrrespondig application when a user change any properties using the GUI.
Related
Problem
We use java WAR files and keep config files in s3 buckets. Our environments: DEV,QA, Stage, and PROD each have their own config files and s3 buckets. If I add a new field, such as "Polling_RATE=5000", it must be manually added to each env because these config files also store passwords so they can not be tied to the application or kept inside Github. Not every engineer has access to each env so you must remember to inform the upper level engineers (DEVOPS) before the prod deployment date to add the new field for the application to work. Its a really messy process currently.
Question
Is there a utility or architectural design pattern meant to deal with this? How do you "version control" sensitive configuration fields that you can not store within github?
Recognizable problem.
Usually config fields with sensitive information like passwords change a lot less often than non-sensitive configuration fields. A possible solution is to split the config in two parts:
Config that's environment-specific but doesn't contain sensitive information. I would advise you to keep these files together with your source code and if possible, generate the files and automatically upload then to your configuration store (S3 in your case) at build time. They must be versioned and tied to the version of your application.
Config that contains sensitive information. Looking at the question, not all team members are allowed to read/write this information. You could store these in S3 with specific access rights so that only authorized members can access them. You would need a mechanism to join the files back together at deployment, or change the app to read from different config files.
However, this will only solve part of your problem. The ops guys will still need to perform changes when sensitive config keys change. Whether this is acceptable depends on how often sensitive config keys change.
An alternative to S3 could be to run a private Git repository (AWS's CodecCommit, for example). You'd have better version control and easier access for the devs to perform changes, since you're already using Git. You'll still have to fix the split access rights between dev and ops, or let that go (since DevOps is about trust and cooperation, that might be a good idea). You could apply a similar pattern here as I described above.
Another solution could be to move the configuration of sensitive values from property files to the system configuration. When you already use a provisioning system like Puppet or Chef, this will feel natural for the ops guys. Or set all sensitive values like passwords as environment variables and have the app read it as system properties.
Hope this helps!
We have been using dynamodb for keeping config values. The advantage with this approach is that the values are easily readable from console and validated.
Also another advantage is that we periodically check the values from dynamodb so if any value needs to be changed we just change it and the app automatically picks the new value instead of starting it again.
Sensitive values are stored encrypted using KMS keys and only the ec2 role that is running the application has right to decrypt using that Key.
We enhanced the Netflix archiaus project to fit our needs. May be you can check that out.
A lot of the advice on the web on storing variables which may change depending on the env/other conditions is to put them in web.xml, but isn't the web.xml within the war file? even if you find the exploded war and change it, wouldn't it get overriden if you update the war file? Or does the webcontainer provide any method to configure the web.xml without tinkering with the war file?
The web.xml variables are of very limited use, in my experience - the only advantage is that it's a standard location to look for hard-coded "configuration".
There are several common solutions to get a more sensible way to configure web apps, none of which is standard:
Use system properties (which usually involves fiddling around with startup scripts, and it can be hard to get a good overview of your entire config)
Use environment variables (same drawbacks as system properties)
Read a config file from a predefined location; often from the classpath by using getResourceAsStream (IIRC that usually means putting the config files in Tomcat's lib directory)
You can also use JNDI, which has the disadvantage of being rather heavy-weight both to set up and read (if you're using vanilla Java, anyways - Spring for example has rather good support for reading from JNDI). However, JNDI is rather good because it's per-application, and not a process-global setting. If you need to run several instances of the same app on the same server, JNDI is pretty much the only option (although you can use it to just point out a config file somewhere, which makes things easier to work with).
This may be relevant to your interests: How can I store Java EE configuration parameters outside of an EAR or WAR?
Advantages of specifying Parameter Values in web.xml
Using your own settings file requires additional coding and management.
Hard-coding parameter values directly into your application code makes them more difficult to change in the future, and more difficult to use different settings for different deployments (eg: JDBC settings, mail server address).
Other developers using your code will be able to find any relevant parameters more easily, as this is a standard location for such parameters to be set.
See also:
Advantages of specifying Parameter Values in web.xml
Web.xml.EnvEntry
Referencing Environment Variables in web.xml
As far as I know web.xml does not provide ability to store custom variables. Typical way to configure your web application is to store configuration in database, separate properties/xml/json/other file, get configuration from separate web service or provide it through environment variables.
Often a mixture of all these is used. For example you can add system variable using -D switch when running your container. This variable will contain path to file or URL where your configuration can be found.
You can supply parameters using OS environment.
You choice should depend on how many parameters do you have, what kind of application are you developing and how can you configure application server or computer OS. For example if you a hosting application on server you cannot configure these ways are not for you, so DB or web service are your only ways.
The folks that work on the Tomcat container recognize the irony that you have identified and have implemented a way to work-around the issue.
The solution that they implemented for the issues that you have alluded to is to create another xml file... the context.xml file, which is read by the server.
It appears that you can edit this file and have the new values read by the Tomcat without a restart... as long as you keep the elements out of the server.xml.
I do not use Tomcat so I might be mis-interpreting the docs
The GlassFish web container supports a similar feature, but does it via a couple admin cli command (asadmin):
set-web-env-entry
set-web-context-param
There is probably web admin console support and you can set them up by editing the domain.xml. It seems like it isn't as flexible as the Tomcat implementation... but it does make it really easy to use.
You need to disable and then enable your application for the changed values to 'take'. Do not redeploy you app, since that will delete the value that you just set.
I have a lot of configuration files that modify how my application behaves. I want to be able to make a change and it gets reflected in the application right away when saving the file. Is there a Java library to help with this?
I could simply keep a list of files with their timestamps and continuously check in a background thread when a timestamp changes. Doesn't seem too difficult, but maybe there's a more efficient way to do this? Custom triggers when certain properties have changed would be nice.
I'm using Spring 3.1, is there a built-in mechanism or solution which works nicely with Spring?
UPDATE: Apparently JDK7 now includes this functionality through its Watch Service API: "Most file system implementations have native support for file change notification. The Watch Service API takes advantage of this support where available. However, when a file system does not support this mechanism, the Watch Service will poll the file system, waiting for events." So this'll be my motivation to migrate to JDK7.
Edited:
http://commons.apache.org/configuration/userguide/howto_filebased.html
I want a simple file format to store and retrieve data from disk in Java.
name=value
list=value1,value2,value3
this is mostly going to be used for initial config settings used at startup of the app. I could invision having a watcher on the file to notify the app if it changes so the new settings can be applied potentially but that would be a nice to have. The first part would be pretty easy to write. I just don't want to reinvent the wheel if something is already out there for this and I'd prefer to avoid something as heavy as spring.
Take a look at the java.util.Properties class.
Properties
You can use the Preferences class. It has a notification system, but alas it doesn't notice changes made outside the running JVM or directly to the underlying configuration store (e.g. the config file). It's a really nice class though.
Have a look at OWNER API.
It incorporates most of the feature of java.util.Properties and adds more.
Version 1.0.4 is under development and it will have:
support for arrays and collections (list, set, arrays). It is already implemented on master branch.
"hot reload", when you change the file the config object gets reloaded (it can be synchronous or asynchronous and it does support event notification for reload). Already implemented in master branch.
a lot of features (variable expansion, type conversion). Available since version 1.0.3 and available on maven central repository.
Also for 1.0.4 is planned a validation mechanism that will check the file to be compliant before discarding the old content of the config file during the reload. (not implemented yet)
If you need some particular feature, just ask on github issues or become a contributor.
I am interested to learn about Eclipse RCP, I have got some basic knowledge, but I wanted to know more what it is capable of. So I encouraged my self to create a set of requirements, analyze them, and come up with design decisions about how they can be met using Eclipse RCP as the base framework, and eventually implement them with Eclipse RCP. Now, maybe the requirements are too hard or I just do not understand about Eclipse RCP much yet, I am struggling to come up with proper solutions to meet the requirements!
The following is the summary of the requirements (please excuse the probable lack of details, this is really just some example to encourage myself):
I wanted to have an Eclipse RCP application to monitor some servers. These servers will initially be programs that the application knows about (meaning it knows exactly about their ins and outs). In the future, however, the application should be able to allow users to specify arbitrary programs with different charateristics for the application to monitor as well (so not just known servers, but also some other servers that it did not know about before). The application will also require an XML configuration file that contains all of the details of the servers that need to be monitored (e.g. host, port, username, and password). This XML configuration file will be encoded and decoded using JAXB.
So based on the above requirements, I have come up with the following details:
The XML will look something like this:
<configuration>
<components>
<serverA>
<host></host>
<port></port>
<username></username>
<password></password>
</serverA>
<serverB>
<host></host>
<port></port>
<username></username>
<password></password>
</serverB>
<!--- this will be the place for other components specified by user -->
</components>
</configuration>
Where and are servers that the application knows about.
In the source code, there is the following hierarchy of classes:
Component <--- Server <--- ServerA, ServerB
ServerA and ServerB descend from Server and map to and element respectively.
The point entry for the configuration is in the class called Configuration that contains a list of ServerA and a list of ServerB. Now, because the application should be able to monitor other programs that it did not know about, the XML configuration file should be extensible as well, so the Configuration class also contains a list of Object which maps to any other component specified by user in the configuration file.
Configuration.java
public class Configuration
{
#XmlElement
private List<ServerA> serveras;
#XmlElement
private List<ServerB> serverbs;
#XmlAnyElement
private List<Object> otherServers;
}
Now, is this something that you guys will do as well to approach the problems? I guess, I do not know, I am just confused about the requirement for the application to be able to monitor other programs specified by user. I know I set it up in the first place, but I did it having in mind saying that "this looks like something that can utilize Eclipse RCP's extension points", but now having jumped into the configuration file, I am not clear about how should the configuration file relate to the plugin.xml?
In the back of my mind, I wanted configuration file to specify the details (host, port, username, and password) of the programs that the application needs to monitor. The plugin.xml is used to specify the extension points and extensions for user-defined programs that the application also needs to monitor. So does this mean, that in the end, for the user-defined programs, users need to configure them as extensions in plugin.xml, and then specify their other details in configuration file?
There is several ways to approach this issue. But let me give a shot to it.
You have a bunch of different servers, with different monitoring characteristics. But for your eclipse application they all must look similar.
Let's say you have an Eclipse RCP application that contains some UI to monitor a server. For this application, it shouldn't matter what the servers actually are, but there should be a common interface to connect to them.
One possibility is that you have an interface that represents the server communication protocol and then, you define an extension point in your main plugin that allows contributing implementations of the protocol. So you would then be able to create a collection of instances of some interface (Lets call it IMonitoringProtocol). This interface would contain the methods you need to display the status on the UI.
Additionally you would have an XML configuration file that lists all of the servers. One of the elements on this configuration file is the protocol to use for monitoring.
So, when you launch your application, you would instantiate all the contributed protocols, and read the configuration file. You can then find the right protocol to communicate to a server by matching the configuration entries.
This allows you to add new protocols in the future, for servers that are not known yet.