Develop Reference

Any idea to prevent specific attribute to appear in the logs?

I'm looking for a way to prevent some sensitive data from being logged.
Ideally i would like to prevent / capture things like
String sensitive = "";
log.info ("This should be prevented or caught by something : {} ", sensitive);
this post is a bit of a longshot, I'm willing to investigate on any lead.
annotation, new types, Sonar Rules, logger hacking etc...
thx for your brainstorming :)
guillaume

Create custom type for it.
Make sure that toString doesn't return actual content.

I imagine there are multiple ways to do this, but one way is to use the Logback configuration file, to specify a message provider for the "arguments" and "message". In those providers, you define a "writeTo" method that looks for particular patterns in the output, and masks them.
This is the path to a solution, but I obviously don't provide many details here. I'm not aware of any "standard" solutions for this.
Another possibility would avail itself if your architecture has services running in transient containers, and the log output is sent to a centralized log aggregator, like Splunk. If you were ok with the initial logs written in the container having sensitive data, you could have the log aggregator look for patterns to mask for.

I would recommend two options, can you split your PII data into a separate log and then log that data securely?
If not, consider something like Cribl Logstream. Point your log shipper at it and let it strip away any PII you are concerned about. LogStream makes it very very easy to remove/mask/encrypt sensitive data. It has all sorts of other features as well.
At my last job we used LogStream as the router to make decisions about the data based on the content. PII data was detected and one copy was pushed to a secure PII certified logging platform and another copy was pushed to the operational logging platform but the PII data was masked so a wider audience could use the logging with no risk. It was a very useful workflow that solved a log of problems.

How do I log from a mapper? (hadoop with commoncrawl)

I'm using the commoncrawl example code from their "Mapreduce for the Masses" tutorial. I'm trying to make modifications to the mapper and I'd like to be able to log strings to some output. I'm considering setting up some noSQL db and just pushing my output to it, but it doesn't feel like a good solution. What's the standard way to do this kind of logging from java?

While there is no special solution for the logs aside of usual logger (at least one I am aware about) I can see about some solutions.
a) if logs are of debug purpose - indeed write usual debug logs. In case of the failed tasks you can find them via UI and analyze.
b) if this logs are some kind of output you want to get alongside some other output from you job - assign them some specail key and write to the context. Then in the reducer you will need some special logic to put them to the output.
c) You can create directory on HDFS and make mapper to write to there. It is not classic way for MR because it is side effect - in some cases it can be fine. Especially taking to account that after each mapper will create its own file - you can use command hadoop fs -getmerge ... to get all logs as one file.
c) If you want to be able to monitor the progress of your job, number of error etc - you can use counters.

How to write statistical data into rolling files in java

I would like to create rolling files that contains statistical data about my service.
for example, logging each request that contained parameter X with a certain result set.
I have to write these files to comply with other systems statistical data:
Roll the file every half an hour
Each file have to have column headers
I have to follow a strict file name convention such as tracking.display.1314116577.done
My service is written in Java.
Since I need to roll files, using loggers seems like a good direction so I have tried an approach where I would log the data using logback logger (my logger of choise), but the conventional rolling file appender cannot role the file every half an hour (or am i wrong?), cannot add column headers and has a strict naming convention of its own.
I have tried to write my own RollingPolicy, but can't find enough resources or examples of how it's done.
Can anyone show/refer me how to accomplish this?
If not, would you recommend a different approach?
Thank you!

Yes, you can do it with a logback appender.
Take a look at TimeBasedRollingPolicy of RollingFileAppender you can easily roll the file each half hour.
To write the header you can extend RollingFileAppender and add the header based on your needs.

A very dumb answer: getting logging right costs much time. So it is easier to implement your own logger from scratch. It seems you have very rigid requirements. For instance that ".done" - you probably first write it as ".part" and then rename it when done.

Cleaner debug level logging in Java?

Is there a cleaner way for me to write debug level log statements? In some ways one could say that the string literals are basically commenting the code and providing logging in one line and that it is already very clean. But after I add debug level log statements, I find the code much less easier to read up and down. Take this example (I may update to a real example if I get back to my home PC):
int i = 0;
logger.debug("Setting i to 0,"); //Just an example, would show something more complex
i++;
InputStream is = socket.getInputStream();
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
IOUtils.write(request, dos);
logger.debug("request written to output");
while (!is.read(buffer))
logger.debug("Reading into buffer");
logger.debug("Data read completely from socket");
CustomObject.doStuff(buffer);
logger.debug("Stuff has been done to buffer");

You could try using aspects, although these have the limitation that you can only put log statements "around" method calls, i.e. before entering and/or after leaving a specific method.
For more detailed logging, I am afraid there is no other way than hand-coded log messages.
I typically strive to remove the not-so-much-needed debug log statements from the code once I made sure that it works the way it should (for which unit tests are a must).

Ask yourself if I run this in a different machine/country/planet, and things go wrong and all I have is only a log file what information do I need to know what has gone wrong ?
Use debug logs in a for loop, or a while loop sparingly. For example, if you are reading 1000 records from a file, performing an op for each record. You could record before the for loop that "file exists and is readable and is going to read 1000 records" and print status after the process is done. If it is say 1000000 records then you could print something every say 100 or 1000 iterations
In your code except for the logger for setting i to 0 everything else sorta makes sense to me. Also care to use log.isDebugEnabled() if your string in the logger statmeent is hard to compute..
ex:
if(log.isDebugEnabled) {
logger.debug("Here " + obj.aMethodCallThatTakes5MinsToEvaluateToString());
}
UPDATE 1: SLF4J solves only half the problem.
if(slfLog.isDebugEnabled) {
slfLog.debug(obj.getObjectThatTakes5Mins());
}
Yes the toString is prevented but if you are logging an actual object which is result of some computation you are not prevented.

If you want very fine grained debug instructions I am not sure you can separate the actual code from the debug code.
If you want it at a higher level, maybe adding your logging using AOP could help make things easier to read, maybe use a proxy object?
But if you have debug instructions as fine grained as in the example you provided, IMHO you could gain more by replacing the loggers with unit tests. Don't write in a log that something happened, test that it did.

You will not be able to do much if you dislike the log statements. The information needs to be there somehow.
What you CAN do, is strongly considering what NEEDS to be there. You are basically writing for the log file reader who per definition does not know about how your program works, so the information needs to be concise and correct. Personally I very frequently add the method name to the log statement.
Also note that slf4j allows you to use the {}-syntax which helps somewhat
log.debug("main() date={}, args={}", new java.util.Date(), args);
Also note that having unit tests, allow you to move much stuff to there simply because you know that THAT works.

dynamically creating & destroying logging appenders

I have a legacy PSVM application which I'd like to redirect its logging output to unique files per execution. So, if I invoke it at 10:00, then have it redirect it's output to {thread-id}-10:00.log; and another thread of execution may begin an execution at 10:01, and its output would go to {thread-id}-10:01.log. I understand that this is not elegant.
My questions are:
is this possible?
does someone have an idea of how to approach?
is it possible to release/destroy an appender when it's no longer needed?
Thanks!

I would start with FileAppender and derive from that to create your own. Simply modify your version to get the current thread id and append a suitable thread-id/timestamp to the file prior to creation. You would maintain (say) a map of (buffered) FileWriters keyed on thread id.
Writing appenders is quite trivial - here's a Javaworld guide on how to do it.
In the above, is it at all likely that your program will start up twice in one minute ? Would you want to append a process id or similar to maintain uniqueness ?

It is not possible, at least not easy to do in log4j. However, if you look at SiftingAppender shipping with logback (log4j's successor), it is designed to handle the creation of appenders on runtime criteria as well as their removal when no longer needed.
If you application needs to create just one log file per application launch, you could simply name your log file based on a timestamp. Shout on the logback-user mailing list if you need further assistance.