Develop Reference

Logback-rolling multiple services

I'm trying to write logs of multiple services to same file, but my rolling policy given is not working, tried with both time based and size based rollings. Thing is my services are running simultanously and writting there logs to same file in my local directory. When tried to write logs by single service it is working as expected.
Please help me to solve this issue tried with different rolling policies.
Appender to log to file​
${LOG_FILE}
Minimum logging level to be presented in the console logs
INFO
${LOG_PATH}/archived/log_%d{dd-MM-yyyy}_%i.log
10KB

I had an experience similar to yours with Log4j 1.x then I debugged an appender back then (~5-6 years ago) and came to the following conclusions:
I don't think you can write data from multiple services into the same file. In other words,
Logging framework usually assumes that only it can change the file. In some Operating Systems (windows) it will even stop writing into file if some other process will rename / change the current file.
Of course its just a code and you could create a more sophisticated appeneder that will probable make it work, but frankly I don't think it worth the effort.
So I suggest writing into different files, where file name can be generated in a way that it will contain a pid of the resource. The downside of this method is that if the process dies and then re-runs, on-one will take care of the old resources.
Another approach (somewhat similar) - is to create a folder with logs for each service so that they'll get different logs based on folder (even if files in these folders will be with the same name).

Change Log4J2 Output File During Runtime

I'm trying to integrate Log4J2 with a bit of custom logging functionality in a project of mine and running into some issues getting Log4J2 to behave similarly to my own implementation. I'd like to be able to change the log file being written to (make if doesn't exist, cease writing to old file, only write to new file) based on events that occur in execution.
I'll give an example that hopefully illustrates what I'm looking for. We start the application and write to some predetermined log file name but we're just recording preliminary logs because nothing of interest has really happened yet. Once environmental conditions are correct and we've received user input the software is engaged and we'd like to begin logging for both debugging information and data capture. To separate the preliminary data from the interesting data we'd like to change our log file to a new log file who's name contains more information about the state of the environment when the system was engaged, to make it easier to sort through which log files we want to analyze and post-process.
I've seen posts on how to accomplish similar things, but they seem to either require the new filename to be known before Log4J2 is initialized (i.e. setting a system property) or using a RoutingAppender which seems to be closer but appears to require knowing all possible values (to define the routes) that we might want to put in our file name. We'd have to define one route per each environment state (or worse yet, each state combination) we want to put in the file name.
Any ideas?

You use the RoutingAppender to dynamically create new files based on a ThreadContext key/value.
Here is an example of how to do this:
https://logging.apache.org/log4j/2.x/faq.html#separate_log_files
In the example, the value of ROUTINGKEY in the ThreadContext map determines the new file name.

Centralized Application properties for multiple system

I am looking for a open-source solutions which allow hosting different properties for different applications and allow changes. On any change of properties it should notify or push the change to the application appropriately.
So, instead every application managing the properties in physical file and deploying physically; these properties can be pushed to a single system. User will have GUI to load and change the properties as per right. Should allow push as mentioned.
If you have already similar open source solutions in mind please advice.

Is this something that Puppet can manage for you?

I don't think what you've described (as nice as it would be) will be likely to exist in an app server. If a program is looking for a file, it's either going to load it with a FileReader (or similar), or it will use ClassLoader.getResourceAsStream(). It might be looking for data that is returned in properties, format, XML properties format, or even something completely different like RDF with which to configure itself. Also many programs read their config on start-up and then hold the values in memory, so you would still need to reboot them to get them to change.
To get something like this to work there would need to be a standard for configuration provisioning and live updates. Once that existed the webapp authors and server vendors would each need to add support for the standard.
If you are the one writing the programs to be managed however, then you can write your programs to request configuration from a service, and have a configuration push feature.... there may be packages out there that can speed up adding that to your code, but I get the impression you are looking to manage programs written by others.

Have you considered to use JMX? I think he could be a good starting point to implement your requirements.
MBeans's attributes can store your application's properties, the MBeanServer will allow you to make them available from remotting, JConsole offers you an GUI to update properties values.
You also can write within the MBean some code that notify the corrrespondig application when a user change any properties using the GUI.

Running one file on multiple servers

Right now my team deals with about 4-5 different servers and about 2-3 different DB servers and we're using environmental variables to decide which server we're on and what server configuration to use.
Is there a better way to do this as my team continues to expand? I've considered compiler flags / args, but it doesn't seem as robust.

From my perspective, in java, you have basically 3 ways to crack this cookie:
Environment variables
-D JVM parameters (which are System Properties)
properties files
You've already discovered Environment Variables and that is pretty much "the unix way" to get the effect you are after; different configuration to common binary that customizes the running application for the environment it is executing on.
System Properties are really the Java "moral equivalent" of Environment Variables. They come in via -D parameters on your application's command line like...
java -Dlogback.configurationFile=/opt/dm/logback.xml -cp app.jar org.rekdev.App
Explicit Properties file processing http://docs.oracle.com/javase/tutorial/essential/environment/properties.html in Java is a third variant which you often see coupled with -D to get something like default behavior which can be overridden at runtime from the command line. That is what is basically going on with the logback.xml configuration above, the JAR file has a logback.xml inside it that will be used unless a System Property called "logback.configurationFile" exists, at which point the App will load it instead.
As you try to figure out how to keep this all in sync and working correctly in a multi-server environment, consider the use of chef http://wiki.opscode.com/display/chef/Home to do the deployments and put each specific environment's customizations under chefs control. Put the chef "recipes" in version control and, voila, full on configuration management.
SHIP IT!

I can see two scenarios
You embed all the different properties within your package (can be a war, ear, jar, or on the file system /yourapp/etc/)
You embed only one property file and this one is created during build (with ant or maven)
Say your app is named foo
Solution 1
It has the advantage that your app can be put as-is on any of the supported servers (all that have a property file in your app package).
Your properties will be named foo.dev.properties, foo.test.properties, foo.prod.properties, foo.damien.properties, foo.bob.properties.
One other advantage is that every developer working has its own dev file that he can safely push on svn/git/whatever and be sure that other developer won't destroy his config.
At runtime the application can check the -D parameter or even retrieve the hostname dinamycally, in order to load the correct property file.
Solution 2
It has the advantage that your package is not polluted by unnecessary properties files.
You must configure a lot of ant tasks/maven target in order to build for specific environment. You will have in your source directory the properties files for the environments also, but only one will be shipped with your app. This one foo.properties will only have placeholders for values, and values will be inferred within it using foo.ENV.properties with the right ant task/maven target.
At my actual job and the previous one also, we did use the solution 1, I think it brings flexibility.
Some parameter (like database user/password) were fetched directly from environment variables on the Unix servers though (so that only the server admins knew the credentials).
You can safely mix the solutions, in order to get where you feel there is the more flexibility for you and your team.

Is it correct for Java system properties to be used to set and get arbitrary program parameters?

It is possible using -Dproperty=value arguments to set arbitrary system properties (not some fixed set of system properties actually used by the JVM) and a program can get these properties later using System.getProperty("property"). Is it correct to do this?
I haven't found an authoritative answer on this, so that's why I'm asking here. It seems to me that program parameters should be set through command line arguments to the program, not to the JVM. However, perhaps this is an accepted practice that just isn't documented anywhere I've looked so far. I'd like to be sure. Thanks.

I think Java system properties are used to pass values from command line to libraries or plugins inside the execution. It is, that insider component has no direct way to receive the parameter from the main program that's executing it. So it reads it from a "context" that Java system properties are.
If we look at it as layers, command line arguments would be parameters for the inmediate lower layer, and system java properties are context for all the lower layers.
command line: arguments + properties
main program: uses arguments
some library/plugin: uses properties from context
If it's not this way the main program should have to carry all the parameters that user could set to the lower layers and it can be very cumbersome.
I don't like to depend on contextual properties so if I design a program I'd try to pass over all the properties in some non-global way. But it can be very handy some times (and using namespacing it's not probable they collide).

In my opinion this is not "incorrect" and there are programs and libraries that do use their own system properties for configuration.
However, it is probably more practical to put configuration parameters for your software in a configuration file (in whatever format you think is suitable - a .properties file, an XML file, or something else). It is cumbersome, especially if you have many configuration parameters, to have to put all those parameters on the command line with -Dname=value options.

You can use this method to set some of your application settings. I think of settings as a contrast to program arguments. For example. Let's think abount some file converter. It converts file A to B. So files A and B should be command line params. If you converter needs some temporary folder you can make id settable by -Dmy.package.tempfolder=\my\tmp\folder. But there should be a default setting for that and the program should work without this setting.
Think about it as an alternative to .properties file. .properties file will be more convinient of course.

You just have to know what you're doing. If your application is a standalone application, system properties provide an easy to use way to pass named arguments, in any order, to the program, and I don't see anything intrinsically bad in using them.
If your app is a webapp that must be deployed in a app server shared by multiple webapps, then it might not be a good idea, especially if you aren't allowed to change how the server is started, or if you have to deploy multiple versions of the same application.

From http://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html:
For example, the following invocation of getProperty looks up the
System property called subliminal.message. This is not a valid system
property, so instead of returning null, this method returns the
default value provided as a second argument: "Buy StayPuft
Marshmallows!"
System.getProperty("subliminal.message",
"Buy StayPuft Marshmallows!");
This implies that properties other than those used by the JVM are "invalid". However, later in the same document, it gives the example of loading a properties file containing the same property setting.
subliminal.message=Buy StayPuft Marshmallows!
It only advises "In general, be careful not to overwrite system properties."
So it seems that this is a supported use of System Properties. It looks like a case of misleading naming. When I hear "System Properties" I think "properties of the JVM" where the JVM is the system. While the properties are used for this, they can also be used for application settings. I think I'll make a mental note to think of this as "System and Application Properties".
Does anyone disagree with this?