We're developing a Java application that reads a Config file at runtime. My question is that which of the following scenario is efficient for reading a Config file.
Scenario #1: Retrieves a value from config by opening the file, get the value, then close the file.
So this means that file will be open and close every time retrieving a value.
Scenario #2: Open the file during initialization, then expose the object statically across the runtime.
File will be open once, then retrieving a value using the Config object.
Honestly, we currently using the scenario #2. A reason why we choose it because Config file will be open once. Opening file needs syncrhonization, which may lead to untimely retrieval of value. Also, scenario #1 may cause runtime error if the Config file is moved from it's absolute path (anything is possible). But scenario #1 is efficient when it comes in modifying the Config values during runtime.
So which is efficient?
Is your application a mulit-threaded application? if yes, you need to make sure that you have scenario #1 with necessary synchronisation and semaphores. The reason is that your config file is a general item in your application which will be used by all the threads. You don't want the config file reading operation to be interleaved. Even with single threaded operation, it is good to do #1 because you will have less chances of having IO Error. The RTE that you mentioned in your question can happen with anything if you decide to move the file (? Why would you if you need it at a certain place?).
If I should have to access the file more than once, probably I would create a class with a field for each configuration, I would read the file once (only to initialize the class fields) and then I would close the file. In this way you read the file once, during the init, and then you could simply access the configuration by reading the field values stored in the class.
public class Configuration
{
private static String confOne;
private static int confTwo;
private static boolean confThree;
public static init(File configFile) {
/* read the file and init fields */
}
public static String getConfigOne() {
return configOne;
}
public static int getConfigTwo() {
return configOne;
}
public static boolean getConfigThree() {
return configOne;
}
}
A little advice: tries to never hold open a file (or resources) unless absolutely necessary.
I think neither of the scenarios is the best one:
As you you have already stated in sencario #1 you may have a lot of unnecessary IO-Workload and in #2 you cannot change config parameters dynamically at runtime.
I suggest to use a mixture of both. You can load your configuration statically once and then check periodically, if changes have happend. If so, reload your config.
You can see how it may be done by inspecting Java's ResourceBundle-Class. In fact, if the config can be placed as properties-file in your classpath, you can (ab?)use this implementation for your purpose:
private static Control MY_CONTROL = new Control()
{
#Override
public long getTimeToLive(String baseName, Locale locale)
{
return MY_TTL; //make shure, that changes are checked periodically
}
};
public static String getParamter(String name)
{
ResourceBundle config = ResourceBundle.getBundle("config", MY_CONTROL);
return config.getString(name);
}
Related
I have essentially the same question as here but am hoping to get a less vague, more informative answer.
I'm looking for a way to configure DropWizard programmatically, or at the very least, to be able to tweak configs at runtime. Specifically I have a use case where I'd like to configure metrics in the YAML file to be published with a frequency of, say, 2 minutes. This would be the "normal" default. However, under certain circumstances, I may want to speed that up to, say, every 10 seconds, and then throttle it back to the normal/default.
How can I do this, and not just for the metrics.frequency property, but for any config that might be present inside the YAML config file?
Dropwizard reads the YAML config file and configures all the components only once on startup. Neither the YAML file nor the Configuration object is used ever again. That means there is no direct way to configure on run-time.
It also doesn't provide special interfaces/delegates where you can manipulate the components. However, you can access the objects of the components (usually; if not you can always send a pull request) and configure them manually as you see fit. You may need to read the source code a bit but it's usually easy to navigate.
In the case of metrics.frequency you can see that MetricsFactory class creates ScheduledReporterManager objects per metric type using the frequency setting and doesn't look like you can change them on runtime. But you can probably work around it somehow or even better, modify the code and send a Pull Request to dropwizard community.
Although this feature isn't supported out of the box by dropwizard, you're able to accomplish this fairly easy with the tools they give you. Note that the below solution definitely works on config values you've provided, but it may not work for built in configuration values.
Also note that this doesn't persist the updated config values to the config.yml. However, this would be easy enough to implement yourself simply by writing to the config file from the application. If anyone would like to write this implementation feel free to open a PR on the example project I've linked below.
Code
Start off with a minimal config:
config.yml
myConfigValue: "hello"
And it's corresponding configuration file:
ExampleConfiguration.java
public class ExampleConfiguration extends Configuration {
private String myConfigValue;
public String getMyConfigValue() {
return myConfigValue;
}
public void setMyConfigValue(String value) {
myConfigValue = value;
}
}
Then create a task which updates the config:
UpdateConfigTask.java
public class UpdateConfigTask extends Task {
ExampleConfiguration config;
public UpdateConfigTask(ExampleConfiguration config) {
super("updateconfig");
this.config = config;
}
#Override
public void execute(Map<String, List<String>> parameters, PrintWriter output) {
config.setMyConfigValue("goodbye");
}
}
Also for demonstration purposes, create a resource which allows you to get the config value:
ConfigResource.java
#Path("/config")
public class ConfigResource {
private final ExampleConfiguration config;
public ConfigResource(ExampleConfiguration config) {
this.config = config;
}
#GET
public Response handleGet() {
return Response.ok().entity(config.getMyConfigValue()).build();
}
}
Finally wire everything up in your application:
ExampleApplication.java (exerpt)
environment.jersey().register(new ConfigResource(configuration));
environment.admin().addTask(new UpdateConfigTask(configuration));
Usage
Start up the application then run:
$ curl 'http://localhost:8080/config'
hello
$ curl -X POST 'http://localhost:8081/tasks/updateconfig'
$ curl 'http://localhost:8080/config'
goodbye
How it works
This works simply by passing the same reference to the constructor of ConfigResource.java and UpdateConfigTask.java. If you aren't familiar with the concept see here:
Is Java "pass-by-reference" or "pass-by-value"?
The linked classes above are to a project I've created which demonstrates this as a complete solution. Here's a link to the project:
scottg489/dropwizard-runtime-config-example
Footnote: I haven't verified this works with the built in configuration. However, the dropwizard Configuration class which you need to extend for your own configuration does have various "setters" for internal configuration, but it may not be safe to update those outside of run().
Disclaimer: The project I've linked here was created by me.
I solved this with bytecode manipulation via Javassist
In my case, I wanted to change the "influx" reporter
and modifyInfluxDbReporterFactory should be ran BEFORE dropwizard starts
private static void modifyInfluxDbReporterFactory() throws Exception {
ClassPool cp = ClassPool.getDefault();
CtClass cc = cp.get("com.izettle.metrics.dw.InfluxDbReporterFactory"); // do NOT use InfluxDbReporterFactory.class.getName() as this will force the class into the classloader
CtMethod m = cc.getDeclaredMethod("setTags");
m.insertAfter(
"if (tags.get(\"cloud\") != null) tags.put(\"cloud_host\", tags.get(\"cloud\") + \"_\" + host);tags.put(\"app\", \"sam\");");
cc.toClass();
}
SCENARIOI've a util class that makes a zip file from folder, and returns the file. This works perfect.
public static class MakeZip {
// returns zip file, null otherwise
public static File doZip(File folderToZip)
}
PROBLEM came when I want to control the possible error scenarios, connection problems, empty folder, zip fail etc... I'm wondering best way to create the file and return the result, for this, I created an enum
public static enum ZIP_RESULT {
OK, KO, EMPTY_FOLDER;
}
IDEAS
a) Create a file class attribute
private File zipFile;
and return ZIP_FILE enum in doZip method.
public ZIP_RESULT doZip(File folderToZip)
b) Modify the File sent:
public ZIP_RESULT doZip(File folderToZip, File fileToZip)
c) ????
QUESTION
There is a pattern, convention or best practice for this kind of file modification expecting enum as result? I guess is a really typical scenario to have it...
Throw an Exception. You can extend Exception or implement Throwable.
The third bullet for when to use an exception describes your situation as described here:
Exceptions due to resource failures: Exceptions that get generated
when resources fail. For example: the system runs out of memory or a
network connection fails. The client's response to resource failures
is context-driven. The client can retry the operation after some time
or just log the resource failure and bring the application to a halt.
this is what I would do,
instead of this :
public static File doZip(File folderToZip)
and have something like this instead
public static Map.Entry<File,ZIP_RESULT> doZip(File folderToZip)
so you'll have both the data in Entry and you can use them as appropriate.
What I want to do is load key/value pairs from a file (excel file using Apache poi) into a static map that will be used as a lookup table. Once loaded the table will not change.
public final class LookupTable
{
private final static Map<String, String> map;
static {
map = new HashMap<String, String>();
// should do initialization here
// InputStream is = new FileInputStream(new File("pathToFile"));
// not sure how to pass pathToFile without hardcoding it?
}
private LookupTable() {
}
public static void loadTable(InputStream is) {
// read table from file
// load it into map
map.put("regex", "value");
}
public static String getValue(String key) {
return map.get(key);
}
}
Ideally I want to load the map within the static initialization block, but how would I pass the stream in without hard coding it? The problem I see using the loadTable static method is it might not be called before calling the other static methods.
// LookupTable.loadTable(stream);
LookupTable.getValue("regex"); // null since map was never populated.
Is there a better approach to this?
Anything you use will have to be accessible at startup. As far as I know, your options are:
Hard-code the path. This is bad for obvious reasons.
A static variable or static method. This presents a bit of a chicken-and-egg problem; ultimately it gets hard-coded, but at least you can do a search with a static method.
Use a variable, either Java or Environment. So, you'd use something System.getProperty("filename", "/default/filename"). Better because it's at least customizable using the environment or -D parameters at JVM startup.
Use the ClassLoader getResource* methods. This is probably The Right Answer. Specifically, you'll probably want to use the getResourceAsStream() method on the current thread's context ClassLoader Thread.currentThread().getContextClassLoader(). (So, Thread.currentThread().getContextClassLoader().getResourceAsStream("filename") in total.) The ClassLoader will then find your resource for you (as long as you put it somewhere sane in your CLASSPATH).
Yes, there is a better approach, use the Factory design pattern to initialise the object before you have to use it:
http://www.oodesign.com/factory-pattern.html
You cannot pass information into static initialization blocks - they are supposed to work in isolation. Since the stream that you are planning to pass needs to be known before the program begins execution, presumably your LookupTable should be able to find it too. For example, this could be some sort of configuration utility that provides the stream for you. Then you can write your initializer like this:
static {
InputStream exelStream = MyConfigUtil.getExcelStreamForLookup();
loadTable(exelStream);
}
Presumably, there is a class in the system that could get your Excel stream from a source that is known to it. The source does not need to be hard-coded: it could read the location from a configuration file, or receive the data from a predefined network location on your server. In all cases the process of getting the Excel stream has to "bottom out" somewhere, in the sense that something in your system needs to be able to find it without additional parameters.
This is not directly answering your question, but I don't see why map has to be static. You could change map to non-static and change the constructor to public LookupTable(File file) {...fill map...}. You could then even have many instances of that class if you have different excel files; it might not be the case now, but it would "future-proof" your code.
This is probably a case for using lazy loading of the map.
But you will need to set the inputFileName before calling getValue() the first time. This would be done in your initialization code for the applications. (Or you could have a static method to set it.)
This points out the advantage of lazy loading. You don't have to have the file name available until you call getValue() the first time. With a static initializer, you have to get the file name stored somewhere outside the class so it can be used to load the data when the class loads (but after the static fields have been initialized.
public static String inputFileName = null;
public static String getValue(String key) {
if (map == null) {
map = = new HashMap<String, String>();
// open the file using 'inputFileName'
loadTable(InputStream is);
}
return map.get(key);
}
If your code is multithreaded, let me know and I'll comment on the synchronization issues.
Alternate
You could also use Spring to inject the map and build it in some other class -- MapBuilder for example.
Try using System.getProperty() and pass parameter with -D in command line.
static String prop;
static {
prop = System.getProperty("java.home");
}
public static void main(String... args) {
System.out.println(prop);
}
I would suggest a singleton enum approach if it suits your case.
public enum LookupTable {
INSTANCE(FileManager.getFileName());
LookupTable(String fileName){
props = new HashMap<String,String>();
//Read from excel and fill the hashmap
}
private final Map<String, String> props;
public String getMapValue(String key){
return props.get(key);
}
}
Which can be called by
LookupTable.INSTANCE.getMapValue("mykey");
This will call these methods in order
Get filename from a filemanager class, which is parametrized on your needs
Call the constructor (it is private) and load properties from the excel file
getMapValue for the key and return
A subsequent call LookupTable.INSTANCE.getMapValue("mysecondkey") will only call getMapValue as the INSTANCE is initialized beforehand.
In java to pass the values between some classes we can use System.setProperty. But using System.getProperties() we can get all the system properties. So if i use any third party API's means they can also access my properties and also they can change. SO is System.setProperty safe ?
It depends what you mean by safe.
It is good practice1 treat the System Properties object as read only, but you can't rely on 3rd-party libraries to do that.
If you are worried about "trusted" 3rd-party code seeing or changing your application's properties, don't use System Properties to represent them. Create your own Properties object and put your properties there. This is probably the simplest approach overall.
If you use sandboxing, you can prevent untrusted code from access the System Properties ... provided that your code doesn't leak the System Properties object to the untrusted code. (The access checks are implemented in the System methods ...)
A Properties object is thread-safe ... if you are referring to that kind of safety.
1 - Occasionally it is necessary to modify system properties programmatically. However, you can end up with fragile applications by doing this. The system properties are typically used to configure JVM services during the initialization. If the order of class initialization changes for some reason, you could find that your application code is now setting the properties too late. If possible, it is better to set the properties via -D command line parameters.
If you need to worry about the behavior of libraries, you need to learn about and use a security policy and a SecurityManager. Amongst other things, this will allow you to restrict the use of System.setProperty.
As per the documentation
In general, be careful not to overwrite system properties.
The setProperties method changes the set of system properties for the
current running application. These changes are not persistent. That
is, changing the system properties within an application will not
affect future invocations of the Java interpreter for this or any
other application. The runtime system re-initializes the system
properties each time its starts up. If changes to system properties
are to be persistent, then the application must write the values to
some file before exiting and read them in again upon startup.
your concern is correct that some third party libraries might overwrite the properties that your app is using. Its always a good practice to use some naming convention to distinguish keys defined in your property file.
A very simple simulation of the problem
public class TestApp {
public static void main(String args[]) throws InterruptedException {
TestApp app = new TestApp();
app.new ThirdPartyLib("thirdParty").start();
while (true) {
Thread.currentThread().sleep(500);
System.setProperty("test", "orignalProperty");
System.out
.format("Thread Name '%s' setting the property with value '%s' \n ",
Thread.currentThread().getName(),
System.getProperty("test"));
}
}
class ThirdPartyLib extends Thread {
public ThirdPartyLib(String threadName) {
super(threadName);
}
#Override
public void run() {
super.run();
while (true) {
Thread.currentThread();
try {
Thread.sleep(400);
System.setProperty("test", "modifiedProperty");
System.out
.format("Thread Name '%s' setting the property with value '%s' \n ",
Thread.currentThread().getName(),
System.getProperty("test"));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
will result in output below - which might not be the intended one and I am sure difficult to debug also
Thread Name 'thirdParty' setting the property with value 'modifiedProperty'
Thread Name 'main' setting the property with value 'orignalProperty'
Thread Name 'thirdParty' setting the property with value 'modifiedProperty'
Thread Name 'main' setting the property with value 'orignalProperty'
Thread Name 'thirdParty' setting the property with value 'modifiedProperty'
Thread Name 'thirdParty' setting the property with value 'modifiedProperty'
Thread Name 'main' setting the property with value 'orignalProperty'
I wouldn't rely on using System properties to share info between threads. I tried creating a property in one thread, another thread couldn't find even after 10 seconds had elapsed. Behavior of modifying values of system properties already available to threads has been answered by others.
I am working on a project that has been through multiple hands with a sometimes rushed development. Over time the message.properties file has become out of sync with the jsps that use it. Now I don't know which properties are used and which aren't. Is there a tool (eclipse plugin perhaps) that can root out dead messages?
The problem is that messages may be accessed by JSP or Java, and resource names may be constructed rather than literal strings.
Simple grepping may be able to identify "obvious" resource access. The other solution, a resource lookup mechanism that tracks what's used, is only semi-reliable as well since code paths may determine which resources are used, and unless every path is traveled, you may miss some.
A combination of the two will catch most everything (over time).
Alternatively you can hide the functionality of ResourceBundle behind another façade ResourceBundle, which should generally pipe all calls to original one, but add logging and/or statistics collection on the top.
The example can be as following:
import java.util.Collection;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.NoSuchElementException;
import java.util.ResourceBundle;
public class WrapResourceBundle {
static class LoggingResourceBundle extends ResourceBundle {
private Collection<String> usedKeys = new HashSet<String>();
public LoggingResourceBundle(ResourceBundle parentResourceBundle) {
setParent(parentResourceBundle);
}
#Override
protected Object handleGetObject(String key) {
Object value = parent.getObject(key);
if (value != null) {
usedKeys.add(key);
return value;
}
return null;
}
#Override
public Enumeration<String> getKeys() {
return EMPTY_ENUMERATOR;
}
public Collection<String> getUsedKeys() {
return usedKeys;
}
private static EmptyEnumerator EMPTY_ENUMERATOR = new EmptyEnumerator();
private static class EmptyEnumerator implements Enumeration<String> {
EmptyEnumerator() {
}
public boolean hasMoreElements() {
return false;
}
public String nextElement() {
throw new NoSuchElementException("Empty Enumerator");
}
}
}
public static void main(String[] args) {
LoggingResourceBundle bundle = new LoggingResourceBundle(ResourceBundle.getBundle("test"));
bundle.getString("key1");
System.out.println("Used keys: " + bundle.getUsedKeys());
}
}
Considering that some of your keys are run-time generated, I don't think you'll ever be able to find a tool to validate which keys are in use and which ones are not.
Given the problem you posed, I would probably write an AOP aspect which wraps the MessageSource.getMessage() implementation and log all the requested codes that are being retrieved from the resource bundle. Given that MessageSource is an interface, you would need to know the implementation that you are using, but I suspect that you must know that already.
Given that you would be writing the aspect yourself, you can create a format that is easily correlated against your resource bundle and once you are confident that it contains all the keys required, it becomes a trivial task to compare the two files and eliminate any superfluous lines.
If you really want to be thorough about this, if you already have Spring configured for annotation scan, you could even package up your aspect as its own jar (or .class) and drop it in a production WEB-INF/lib (WEB-INF/classes) folder, restart the webapp and let it run for a while. The great thing about annotations is that it can all be self contained. Once you are sure that you have accumulated enough data you just delete the jar (.class) and you're good to go.
I know that at least two of the major java IDEs can offer this functionality.
IntelliJ IDEA has a (disabled, by default) Inspection that you can
use to do this:
go to Settings -> Inspections -> Properties files -> ... and enable
the 'Unused property'
..Only problem I had was that it didn't pick up some usages of the property from a custom tag library I had written, which I was using in a few JSPs.
Eclipse also has something like this ( http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2Ftasks%2Ftasks-202.htm ) but I haven't really exhausted the how well it works.