Develop Reference

Log4j2 - Transaction logging in file and application logging on console

I am developing a logger where certain transaction messages need to be logged to a file and some application logs need to be logged to the console. Currently, the transaction messages are also being logged onto the console and I want to avoid that.
for eg:
if i write logger.info("application process started");
this needs to be logged on console.
and when i write logger.info(new ObjectArrayMessage("msg1","msg2","msg3"));
this need to be logged to a csv file.
This is my current output:
19:18:42.230 [main5] INFO New.CSVlog.App - application process started
19:18:42.233 [main5] INFO New.CSVlog.App - [msg1,msg2,msg3]
I want to log only the first log to the console and the second one to the CSV file.
I have the following properties file:
appenders = csvFile, console
appender.csvFile.type = RollingFile
appender.csvFile.name = CSVFILE
appender.csvFile.fileName = csvLog.csv
appender.csvFile.filePattern= csvLog-%d{MM-dd-yyyy}-%i.csv
appender.csvFile.layout.type = CsvParameterLayout
appender.csvFile.layout.delimiter = ,
appender.csvFile.layout.header = column1,column2,column3\n
appender.csvFile.policies.type = Policies
appender.csvFile.policies.time.type = TimeBasedTriggeringPolicy
appender.csvFile.policies.time.interval = 1
appender.csvFile.policies.time.modulate = true
appender.csvFile.policies.size.type = SizeBasedTriggeringPolicy
appender.csvFile.policies.size.size=10MB
appender.csvFile.strategy.type = DefaultRolloverStrategy
appender.csvFile.strategy.max = 200
appender.console.type = Console
appender.console.name = consoleLog
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%t5] %-5level %logger{36} - %msg%n
rootLogger.level =debug
rootLogger.appenderRefs = csvFile, consoleLog
rootLogger.appenderRef.csvFile.ref = CSVFILE
rootLogger.appenderRef.console.ref = consoleLog
Can anyone please help me to use the same logger for both transaction and application logging?
and what changes do I need to make in my properties file? Thanks!

You can use Filters to seperate ObjectArrayMessage messages from the rest (cf. documentation). Although there is no built-in filter that checks the class of the Message object, you can easily write one:
#Plugin(name = "ObjectArrayMessageFilter", category = Node.CATEGORY, elementType = Filter.ELEMENT_TYPE)
public class ObjectArrayMessageFilter extends AbstractFilter {
public static class Builder extends AbstractFilterBuilder<Builder>
implements org.apache.logging.log4j.core.util.Builder<ObjectArrayMessageFilter> {
#Override
public ObjectArrayMessageFilter build() {
return new ObjectArrayMessageFilter(getOnMatch(), getOnMismatch());
}
}
#PluginBuilderFactory
public static Builder newBuilder() {
return new Builder();
}
private ObjectArrayMessageFilter(Result onMatch, Result onMismatch) {
super(onMatch, onMismatch);
}
private static Result filter(Object msg) {
return msg instanceof ObjectArrayMessage ? onMatch : onMismatch;
}
#Override
public Result filter(final LogEvent event) {
return filter(event.getMessage());
}
#Override
public Result filter(final Logger logger, final Level level, final Marker marker, final Message msg,
final Throwable t) {
return filter(msg);
}
#Override
public Result filter(final Logger logger, final Level level, final Marker marker, final Object msg,
final Throwable t) {
return filter(msg);
}
}
You can then attach the filter to the appenders, so that it is always applied to a given appender:
appender.csvFile.type = RollingFile
...
appender.csvFile.filter.omaf.type = ObjectArrayMessageFilter
appender.csvFile.filter.omaf.onMatch = ACCEPT
appender.csvFile.filter.omaf.onMismatch = DENY
appender.console.type = Console
...
appender.console.filter.omaf.type = ObjectArrayMessageFilter
appender.console.filter.omaf.onMatch = DENY
appender.console.filter.omaf.onMismatch = ACCEPT
Remark: When developing custom plugins, some standard rules apply: cf. documentation. Basically the plugin's code must be compiled with the annotation processor in log4j-core.

Logging exception stacktrace with custom rollingFileAppender and custom JsonLayout

I have written a custom logger with its own rolling file appender. I am logging an exception using one of the log messages (e.g., info(), debug(), etc.), I am excepting that it would print the entire stacktrace of the exception, but it is printing only the message.
public class Logger {
private static ConcurrentHashMap<String, Logger> loggerMap = new ConcurrentHashMap<String, Logger>();
private static ConcurrentHashMap<String, FileAppender> appenderMap = new ConcurrentHashMap<String, FileAppender>();
private static LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
public static org.slf4j.Logger getLoggerWithRollingFileAppender(String appenderName,
String loggerName,
Level logLevel) {
RollingFileAppender rollingFileAppender;
// Check if the appender is already created.
if (! appenderMap.containsKey(appenderName)) {
LayoutWrappingEncoder jsonEncoder = new LayoutWrappingEncoder();
JsonFormatter jsonFormatter = new JacksonJsonFormatter();
((JacksonJsonFormatter) jsonFormatter).setPrettyPrint(false);
JsonLayout jsonLayout = new JsonLayout();
jsonLayout.setJsonFormatter(jsonFormatter);
jsonLayout.setTimestampFormat("yyyy-MM-dd HH:mm:ss.SSS");
jsonLayout.setContext(loggerContext);
jsonLayout.setAppendLineSeparator(true);
jsonLayout.setIncludeException(true);
jsonLayout.setIncludeMDC(true);
jsonEncoder.setLayout(jsonLayout);
jsonEncoder.setContext(loggerContext);
jsonEncoder.start();
rollingFileAppender = new RollingFileAppender();
rollingFileAppender.setContext(loggerContext);
rollingFileAppender.setName(appenderName);
rollingFileAppender.setEncoder(jsonEncoder);
rollingFileAppender.setAppend(true);
SizeAndTimeBasedRollingPolicy sizeAndTimeBasedRollingPolicy =
new SizeAndTimeBasedRollingPolicy();
sizeAndTimeBasedRollingPolicy.setContext(loggerContext);
sizeAndTimeBasedRollingPolicy.setParent(rollingFileAppender);
sizeAndTimeBasedRollingPolicy.setFileNamePattern(logBaseDir +
File.separator +
appenderName + "-" +
applnName + "-%d{yyyy-MM-dd}.%i.log");
sizeAndTimeBasedRollingPolicy.setMaxHistory(maxLogHistory);
sizeAndTimeBasedRollingPolicy.setMaxFileSize(FileSize.valueOf(maxLogFileSize));
sizeAndTimeBasedRollingPolicy.setTotalSizeCap(FileSize.valueOf(totalLogSizeCap));
sizeAndTimeBasedRollingPolicy.start();
rollingFileAppender.setRollingPolicy(sizeAndTimeBasedRollingPolicy);
rollingFileAppender.start();
appenderMap.put(appenderName, rollingFileAppender);
} else {
rollingFileAppender = (RollingFileAppender) appenderMap.get(appenderName);
}
// Return the logger
org.slf4j.Logger slf4jLogger = getLogger(loggerName);
Logger logger = (ch.qos.logback.classic.Logger) slf4jLogger;
logger.setAdditive(false);
logger.setLevel(logLevel);
logger.addAppender(rollingFileAppender);
return slf4jLogger;
}
}
#Test
void testExceptionWithLoggerWithRollingFileAppender() {
org.slf4j.Logger slf4jLogger = Logger.getLoggerWithRollingFileAppender
("testAppender",
"com.test7",
Level.DEBUG);
try {
FileInputStream fis=new FileInputStream("dummyfile.txt");
} catch (FileNotFoundException e) {
slf4jLogger.info("Exception is", e);
}
Logger logger = (ch.qos.logback.classic.Logger) slf4jLogger;
logger.debug("This is a log message");
try {
FileInputStream fis=new FileInputStream("dummyfile.txt");
} catch (FileNotFoundException e) {
logger.info("Exception is", e);
}
}
But the log file do not contain the complete stack trace of the exception. This is not useful from debugging point of view. What could be missing here?
{"timestamp":"2021-03-09 18:03:21.443","level":"INFO","thread":"main","logger":"com.test7","message":"Exception is","context":"default","exception":"java.io.FileNotFoundException: linessssss.txt (No such file or directory)\n"}
{"timestamp":"2021-03-09 18:03:21.476","level":"DEBUG","thread":"main","logger":"com.test7","message":"This is a log message","context":"default"}
{"timestamp":"2021-03-09 18:03:21.477","level":"INFO","thread":"main","logger":"com.test7","message":"Exception is","context":"default","exception":"java.io.FileNotFoundException: linessssss.txt (No such file or directory)\n"}

Looking at the source for JsonLayout it uses a ThrowableProxyConverter to control what ends up in the JSON - so even if you call setIncludeException (as recommended by responses to other similar questions) what it does will depend on which converter is selected (didn't spend long enough to see how). You probably want something like the RootCauseFirstThrowableProxyConverter or something like it being involved.

I found the fix for this. Added a CustomThrowableProxyConverter and handled the stack trace processing through it.
public class CustomThrowableProxyConverter extends ThrowableHandlingConverter {
public CustomThrowableProxyConverter() {
}
#Override
public String convert(ILoggingEvent event) {
StringBuilder sb = new StringBuilder();
IThrowableProxy itp = event.getThrowableProxy();
if (itp instanceof ThrowableProxy) {
ThrowableProxy tp = (ThrowableProxy)itp;
sb.append(tp.getClassName() + ": " + tp.getMessage());
for (StackTraceElementProxy element : tp.getStackTraceElementProxyArray()) {
sb.append("\t\n");
sb.append(element.getSTEAsString());
}
return sb.toString();
}
return "";
}
}
Then set this into jsonLayout instance.
JsonLayout jsonLayout = new JsonLayout();
jsonLayout.setThrowableProxyConverter(new CustomThrowableProxyConverter());
jsonEncoder.setLayout(jsonLayout);

log4j - Intercept logging created within a class/method

I have an App which has many ejb Timer methods which are fired at various intervals, in an attempt to log the performance of these scheduled tasks I have written two methods which log the start and end of every task and records some info such as name, running duration and log info each time the timer fires
The intention with log info was to intercept all logging statements raised from within these methods, output to a string and save...I am doing this by creating a new WriterAppender, appending to the Logger for all logging levels, then when the timer completes capture the string output and remove the appender
This works well but it seems to have 2 unwanted effects:
1) It is stopping any logging from the timer class appearing in our normal file appender logs, eg. the log messages are being captured but they now don't appear in our normal logs (which are configured to show everything > INFO
2) Logging statements from different timers which overlap are appearing in each other logs, I guess this is the result of the Logger working on a single thread, not sure if theres a way around this?
I am no expert in log4j so I am hoping someone can point out a simple fix for 1, 2 or both
Code below shows the 2 methods which are called at start and end of every job
private Writer w = new StringWriter();
private WriterAppender wa;
private Logger logOutput = Logger.getLogger(ScheduledActions.class);
private TimerEvent logStartTimerEvent(ScheduledTask task){
RedisActions redisActions = (RedisActions)Component.getInstance("redisActions");
TimerEvent event = new TimerEvent(task);
//Initialise a new log appender to intercept all log4j entries
w = new StringWriter();
Layout l = new PatternLayout("%m%n");
wa = new WriterAppender(l, w);
wa.setEncoding("UTF-8");
wa.setThreshold(Level.ALL);
wa.activateOptions();
// Add it to logger
logOutput = Logger.getLogger(ScheduledActions.class);
logOutput.addAppender(wa);
//Push this timer event to redis
redisActions.addToFixedLengthList(RedisItem.TIMER_HISTORY.getKey()+"_"+task.getName(), gson.toJson(event), 100L);
return event;
}
private void logEndTimerEvent(TimerEvent event){
try{
RedisActions redisActions = (RedisActions)Component.getInstance("redisActions");
event.setLogInfo(w.toString());
event.setEndTime(new Date());
redisActions.editLastMemberInList(RedisItem.TIMER_HISTORY.getKey()+"_"+event.getTask().getName(), gson.toJson(event));
w.flush();
} catch (IOException e) {
e.printStackTrace();
}finally{
//clean up the temp log appender
logOutput.removeAppender(wa);
wa.close();
}
}

I found this old question while trying to do something similar. For now, my solution is creating an independent logger/appender, with a unique loggerId, like:
var loggerId = operation.getId(); // This is a unique ID
var logger = (org.apache.logging.log4j.core.Logger) LogManager.getLogger(loggerId);
var appender = createAppender();
logger.addAppender(appender);
logger.info("using logger");
appender.stop();
logger.removeAppender(appender);
The problem that I have is that I need to pass this logger variable around. I can't use the classic "static logger".

Another solution that I am considering, is adding an AbstractAppender to the rootLogger, and from there do whatever with the logs.
I will update this answer when I develop it a bit more.
var layout = PatternLayout.newBuilder()
.withPattern("%d{ISO8601_OFFSET_DATE_TIME_HHCMM} [%t] %-5level %c{1.} %X %msg%n%throwable")
.build();
var rootLogger = (org.apache.logging.log4j.core.Logger) LogManager.getRootLogger();
var appender = new AbstractAppender("intercept-appender", null, layout, false, null) {
#Override
public void append(LogEvent event) {
System.out.println("INTERCEPTED: " + getLayout().toSerializable(event));
}
};
appender.start();
rootLogger.addAppender(appender);
log.info("Message after adding intercept appender");

My final solution is adding an Appender with ThreadContextMapFilter.
// With %X you will see the thread context values
var layout = PatternLayout.newBuilder()
.withPattern("%d{ISO8601_OFFSET_DATE_TIME_HHCMM} [%t] %-5level %c{1.} %X %msg%n%throwable")
.build();
var appender = FileAppender.newBuilder()
.setName(fileId)
.withFileName("/tmp/my-file.log")
.setLayout(layout)
// --- IMPORTANT PART ---
.setFilter(new ThreadContextMapFilter(
Map.of("my-key", List.of("my-value")), true, null, null))
.build();
// Add the appender
// https://logging.apache.org/log4j/2.x/manual/customconfig.html
appender.start();
final LoggerContext context = LoggerContext.getContext(false);
final Configuration config = context.getConfiguration();
config.addAppender(appender);
config.getRootLogger().addAppender(appender, null, null);
context.updateLoggers();
// --- IMPORTANT PART ---
try (var ignore = CloseableThreadContext.put("my-key", "my-value")) {
// the logs here will be captured by the appender
log.info("This log WILL be captured by the appender");
// Note: In case you start threads from here,
// you should pass the ThreadContext to them.
}
log.info("This log will NOT be captured by the appender");
// Wait for the logs to be flushed. Any improvement ideas?
// https://stackoverflow.com/a/71081005/1121497
try {
Thread.sleep(500);
} catch (InterruptedException ignore) {
}
// Remove the appender
final LoggerContext context = LoggerContext.getContext(false);
final Configuration config = context.getConfiguration();
config.getRootLogger().removeAppender(appender.getName());
context.updateLoggers();
appender.stop();

How to rotate a log4j log manually

I have log4j configured to rotate the log every day.
In special situations I would like to trigger an additional log rotation manually.
Is this possible - and if so: How?
Solved like this:
void rolloverLogs() {
for(final Enumeration<?> loggers = LogManager.getCurrentLoggers(); loggers.hasMoreElements(); ) {
final Logger logger = (Logger) loggers.nextElement();
for (final Enumeration<?> appenders = logger.getAllAppenders(); appenders.hasMoreElements(); ) {
final Appender a = (Appender) appenders.nextElement();
if(!RollingFileAppender.class.isInstance(a))
continue;
((RollingFileAppender)a).rollOver();
}
}
}

The technique shown by Lahiru will only locate Appenders configured for a specific Logger which is based on the class the code resides in. The exact Appender(s) located may may vary if the configuration changes.
If you know the Appender name the best way to do this is:
org.apache.logging.log4j.core.LoggerContext context = LogManager.getContext(false);
Appender appender = context.getConfiguration().getAppender(appenderName);
if (appender instanceof RollingFileAppender) {
((RollingFileAppender) appender).getManager().rollover();
}
If you want to roll over all RollingFileAppenders you could do:
org.apache.logging.log4j.core.LoggerContext context = (org.apache.logging.log4j.core.LoggerContext) LogManager.getContext(false);
for (Appender appender : context.getConfiguration().getAppenders().values()) {
if (appender instanceof RollingFileAppender) {
((RollingFileAppender) appender).getManager().rollover();
}
}

For log4j2, you can use following.
org.apache.logging.log4j.Logger logManagerLogger = LogManager.getLogger();
Map<String, org.apache.logging.log4j.core.Appender> appenders = ((org.apache.logging.log4j.core.Logger) logManagerLogger).getAppenders();
appenders.forEach((appenderName, appender) -> {
if (appender instanceof RollingFileAppender) {
logger.info("Rolling over appender " + appenderName);
((RollingFileAppender) appender).getManager().rollover();
}
});

If you're keeping track of your appenders, you could call
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#rollOver()
That should do it. I guess it's also possible to iterate through all appenders you can find starting from root level - just make sure you keep track of which appenders you already rolled.

log4j 1.x : detect if no config file given and programmatically set it up

(If your using log4j version 2 this is not applicable)
Know how to set log4j to print debug and how to pro grammatically construct a default Properties object.
Want help with code to detect if log4j did not find a config file in classpath.
This for the times when don't have a log4j.xml to at least see all logs in console.
Code I have:
private static void log4Default() {
boolean noLog = true;
//how to set this to false if log4j did find a config file
// File log4f = new File("log4j.xml");
// File log4f2 = new File("log4j.properties");
//
// if(log4f.exists() || log4f2.exists()) {
// noLog = false;
// }else {
// log4f = new File("target/classes/log4j.xml");
// log4f2 = new File("target/classes/log4j.properties");
// if(log4f.exists() || log4f2.exists()) {
// noLog = false;
// }
// }
// if(noLog) {
// log4f = new File("target/test-classes/log4j.xml");
// log4f2 = new File("target/test-classes/log4j.properties");
// if(log4f.exists() || log4f2.exists()) {
// noLog = false;
// }
// }
if(noLog) {
System.out.println("no log4j config, using default");
Layout layout = new PatternLayout(" %-5p %t %d [%t][%F:%L] : %m%n");
Appender ap = new ConsoleAppender(layout , ConsoleAppender.SYSTEM_OUT);
Logger.getRootLogger().setLevel(Level.ALL);
//Logger.getRootLogger().addAppender(new ConsoleAppender(layout, ConsoleAppender.SYSTEM_ERR));
Logger.getRootLogger().addAppender(ap);
}
}
Commented out the file exists as there could be an over ride and above is not foul proof.

You can use this to check whether there is any log4j.xml/.properties file present in the classpath:
public void checkForLog4jConfigFile() {
org.apache.log4j.Logger rootLogger = org.apache.log4j.Logger.getRootLogger();
Enumeration appenders = rootLogger.getAllAppenders();
if (!appenders.hasMoreElements()) {
System.out.println("LOG4J config file is missing");
} else {
System.out.println("appender found "
+ ((Appender) appenders.nextElement()).getName());
}
}