Develop Reference

Java FileHandler and rolling log files

I understand how the Java FileHandler rolls to the next log file when a particular size is reached. What I want is a little different. I want to use the FileHandler to use the log file with the oldest last written time in the sequence when the program starts.
For example if I have specified to use 5 log files:
mylog.0.log, mylog.1.log...mylog.4.log
If the program last updated mylog.2.log then the next time I start the program I want it to start logging to mylog.3.log.
The problem I am trying to solve is when a user executes the program and something happens they typically restart the program and if mylog.0.log is available it will always use it and not go to mylog.1.log. I lose the information from the previous execution of the program.

Per the documentation for the java.util.logging.FileHandler:
Successively older files are named by adding "0", "1", "2", etc. into the base filename.
Seems to imply that the order will always be the opposite of what you want.
Looks like your only option is to implement a config class to generate the file name you want and pass it to the FileHandler.
You can manually rollover a log file but not in the order your want.

Is it possible to roll over a log fie, without creating a backup?

I would like to create a rolling metrics log file, where metrics data its written to it every few minutes. As a result, the file will get big very fast, but the actually relevant information in it, is just a small 1-hour time window. I want no backups of it.
Using log4j, is there a way to create a rolling appender, without creating backup files every hour?

From docs here you can set MaxBackupIndex is equal to zero
log4j.appender.R.MaxBackupIndex=0
rollOver
public void rollOver()
Implements the usual roll over behaviour.
If MaxBackupIndex is positive, then files {File.1, ..., File.MaxBackupIndex -1} are renamed to {File.2, ..., File.MaxBackupIndex}. Moreover, File is renamed File.1 and closed. A new File is created to receive further log output.
If MaxBackupIndex is equal to zero, then the File is truncated with no backup files created.

Java FileHandler Adding Unnecessary Digit to File Name

I'm working with JUL for my logging (no I can't change that). I've developed a simple wrapper that I pass in the parameters and it creates the FileHandler with the correct format every time so that I don't have to recreate the logging in every project.
My test app functions exactly as intended, but when I import the library into other projects I seem to be getting one (only one so far) unique error: Every single time, it adds a ".0" to the end of the log file.
It does this even when there is no conflicts and the Filehandler has been configured to append to the end an existing file (which it does fine). I've played with various file names, most recently I've been using the simple "mylog.log" and the log file still gets output as "mylog.log.0". I've checked and the fileHandler is being passed the correct file ("mylog.log"), but it isn't logging there.
This does not happen in my logging test, only in the project I actually want to use it in. Even when using the exact same parameters, I get different file names.
Is there some quirk about JUL that I'm missing? Code is very simple. Relevent code:
String logFilePath = directory+name; // directory and name are method arguments
Handler newFileHandler;
File dirFile = new File(directory);
if(!dirFile.exists())
{
dirFile.mkdirs();
}
newFileHandler = new FileHandler(logFilePath, true);
newFileHandler.setFormatter(myformatter);
//... etc

It does this even when there is no conflicts ....
Is that proven with evidence or an assumption? According to the FileHandler documentation:
If no "%g" field has been specified and the file count is greater than one, then the generation number will be added to the end of the generated filename, after a dot.
If there is a conflict and no "%u" field has been specified, it will be added at the end of the filename after a dot. (This will be after any automatically added generation number.)
Note that the use of unique ids to avoid conflicts is only guaranteed to work reliably when using a local disk file system.
A conflict can include opening more than one FileHandler to the same location. You need to verify each of these points. What helps is adding code to grab the RuntimeMXBean and then add a single log statement to record the calling ClassLoader, current thread, runtime name and the start time. The runtime name usually maps to a process id and a host name. Run the program and verify the contents of the file.
The code you have included helps but, you need to include details on how the application is launched and what is included in the logging.properties.

I eventually figured this out and forgot to post the cause.
Two things were at work:
Due to the environment I was in, the "rolling" logging was being activated by some back ground variables I wasn't aware of, hence why the ".0" was being added when it shouldn't have been, but I only saw it once I moved it out of testing and into the actual implementing project.
JUL is ridiculously inflexible in how it works. Really I can't speak poorly enough about it. Anyway, long story short, if rolling logging is enabled it will always append a file number such that the active log ends in ".0". Typically I've used APIs that only had the number on the secondary logs, while the current log would maintain the exact name you gave it - this caused me some trouble as JUL also has no "get current log file" method to get the name of active file, so I needed to create an ugly method that predicted what the name should be based on the parameters, and hope nothing went wrong. As an aside, you cannot change the format of the generation numbers (which also caused me some issue as it was preferred to have the files number 01, 02, ... 10, 11 rather than 0,1,2,...10).

How do I log from a mapper? (hadoop with commoncrawl)

I'm using the commoncrawl example code from their "Mapreduce for the Masses" tutorial. I'm trying to make modifications to the mapper and I'd like to be able to log strings to some output. I'm considering setting up some noSQL db and just pushing my output to it, but it doesn't feel like a good solution. What's the standard way to do this kind of logging from java?

While there is no special solution for the logs aside of usual logger (at least one I am aware about) I can see about some solutions.
a) if logs are of debug purpose - indeed write usual debug logs. In case of the failed tasks you can find them via UI and analyze.
b) if this logs are some kind of output you want to get alongside some other output from you job - assign them some specail key and write to the context. Then in the reducer you will need some special logic to put them to the output.
c) You can create directory on HDFS and make mapper to write to there. It is not classic way for MR because it is side effect - in some cases it can be fine. Especially taking to account that after each mapper will create its own file - you can use command hadoop fs -getmerge ... to get all logs as one file.
c) If you want to be able to monitor the progress of your job, number of error etc - you can use counters.

dynamically creating & destroying logging appenders

I have a legacy PSVM application which I'd like to redirect its logging output to unique files per execution. So, if I invoke it at 10:00, then have it redirect it's output to {thread-id}-10:00.log; and another thread of execution may begin an execution at 10:01, and its output would go to {thread-id}-10:01.log. I understand that this is not elegant.
My questions are:
is this possible?
does someone have an idea of how to approach?
is it possible to release/destroy an appender when it's no longer needed?
Thanks!

I would start with FileAppender and derive from that to create your own. Simply modify your version to get the current thread id and append a suitable thread-id/timestamp to the file prior to creation. You would maintain (say) a map of (buffered) FileWriters keyed on thread id.
Writing appenders is quite trivial - here's a Javaworld guide on how to do it.
In the above, is it at all likely that your program will start up twice in one minute ? Would you want to append a process id or similar to maintain uniqueness ?

It is not possible, at least not easy to do in log4j. However, if you look at SiftingAppender shipping with logback (log4j's successor), it is designed to handle the creation of appenders on runtime criteria as well as their removal when no longer needed.
If you application needs to create just one log file per application launch, you could simply name your log file based on a timestamp. Shout on the logback-user mailing list if you need further assistance.