I want to create a custom log4j2 rolling file appender. I need to create this custom appender because I want to wrap the file name with current thread name. We are trying to migrate log4j 1.x to recent log4j2 version and previously we had used DailyRollingFileAppender to log all activities of our application.
please find the below code.
Here we are trying to append the log to a file on daily basis with help of DailyRollingFileAppender based on threadName.
Since DailyRollingFileAppender is deprecated in recent version -so, how to create custom rolling file appender with incorporating our thread based logic.?
Find the below log4j.properties file
log4j.logger.***=INFO, FileLogger
# log4j.appender.FileLogger=org.apache.log4j.DailyRollingFileAppender
# Custom Appendar which will redirect the logs based on thread names configured using
# log4j.appender.FileLogger.threadNameMapping property below
log4j.appender.FileLogger=********.framework.log4j.appender.ThreadNamePatternAppender
log4j.appender.FileLogger.DatePattern='.'yyyy-MM-dd
log4j.appender.FileLogger.file=/logs/fileName.log
log4j.appender.FileLogger.layout=org.apache.log4j.PatternLayout
log4j.appender.FileLogger.layout.ConversionPattern=%d [%-5p] [%t] [%c{1}] [%M] - %m%n
# Custom property to hold mapping between thread names and log file for plug-in
# Beware - ThreadNamePatternAppender class inherits DailyRollingFileAppender hence it will not work for any other type of appender
# This can be distuingished using - ThreadName1>ThreadName1.log|ThreadName2>ThreadName2.log|.....|ThreadNameN>ThreadNameN.log
# Note - If there is no mapping for a particular thread then logs will be written to default log file
log4j.appender.FileLogger.threadNameMapping=********/logs/fileName-fileName.log
Thanks!
import java.util.HashMap;
import java.util.Map;
import org.apache.logging.log4j.core.appender.RollingFileAppender;
import org.apache.log4j.helpers.LogLog;
import org.apache.log4j.spi.LoggingEvent;
public class ThreadNamePatternAppender extends DailyRollingFileAppender {
private Map<String, DailyRollingFileAppender> threadBasedSubAppenders = new HashMap<String, DailyRollingFileAppender>();
private String threadNameMapping;
public String getThreadNameMapping() {
return threadNameMapping;
}
public void setThreadNameMapping(String threadNameMapping) {
this.threadNameMapping = threadNameMapping;
}
#Override
public void activateOptions() {
super.activateOptions();
if (threadNameMapping != null && threadNameMapping.trim().length() > 0) {
DailyRollingFileAppender tempAppender;
String[] threadNames = threadNameMapping.split("\\|");
for (String threadName : threadNames) {
if (threadName != null && threadName.length() > 0) {
try {
LogLog.debug(String.format("Creating new appender for thread %s", threadName));
tempAppender = new DailyRollingFileAppender(getLayout(), threadName.split(">")[1],
getDatePattern());
threadBasedSubAppenders.put(threadName.split(">")[0], tempAppender);
} catch (Exception ex) {
LogLog.error("Failed to create appender", ex);
}
}
}
}
}
#Override
public void append(LoggingEvent event) {
String threadName = event.getThreadName().split(" ")[0];
if (threadBasedSubAppenders.containsKey(threadName)) {
threadBasedSubAppenders.get(threadName).append(event);
} else {
super.append(event);
}
}
#Override
public synchronized void close() {
LogLog.debug("Calling Close on ThreadNamePatternAppender" + getName());
for (DailyRollingFileAppender appender : threadBasedSubAppenders.values()) {
appender.close();
}
this.closed = true;
}
}
The RollingFileAppender in Log4j 2.x is final, so you can not extend it. However you can obtain the functionality of your custom Log4j 1.x appender using:
A RoutingAppender, which can create appenders on demand,
Multiple RollingFileAppender that will write and rotate your files,
The EventLookup to retrieve the current thread name.
For a simple logfile-per-thread appender you can use:
<Routing name="Routing">
<Routes pattern="$${event:ThreadName}">
<Route>
<RollingFile name="Rolling-${event:ThreadName}"
fileName="logs/thread-${event:ThreadName}.log"
filePattern="logs/thread-${event:ThreadName}.log.%d{yyyy-MM-dd}">
<PatternLayout pattern="%d [%-5p] [%t] [%c{1}] [%M] - %m%n" />
<TimeBasedTriggeringPolicy />
</RollingFile>
</Route>
</Routes>
</Routing>
For a more complex configuration both the <Routing> appender and the <Routes> can contain a <Script> (cf. documentation):
the script in the <Routing> appender can initialize the staticVariables map and return a default route,
the script in the <Routes> component chooses the appropriate route based on staticVariables and the logging event.
Related
Following is the code, I am working on :
Purpose is to configure SmtpAppender programmatically. Along with the SmtpAppender, I also need to add RollingFileAppender as well as Console appender programmatically.
package vish;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.core.appender.SmtpAppender;
import org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilder;
import org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilderFactory;
import org.apache.logging.log4j.core.config.builder.api.LayoutComponentBuilder;
import org.apache.logging.log4j.core.config.builder.api.RootLoggerComponentBuilder;
import org.apache.logging.log4j.core.config.builder.impl.BuiltConfiguration;
public class SmtpAppenderBuilder {
public static void main(String[] args) {
String pattern = "%d{MM-dd#HH\\:mm\\:ss}%-4r %-5p [%t] %37c %3x - %m%n";
ConfigurationBuilder<BuiltConfiguration> builder = ConfigurationBuilderFactory.newConfigurationBuilder();
LayoutComponentBuilder layoutBuilder = builder.newLayout("PatternLayout").addAttribute("pattern", pattern);
RootLoggerComponentBuilder rootLogger = builder.newRootLogger(Level.DEBUG);
builder.setStatusLevel(Level.DEBUG);
org.apache.logging.log4j.core.appender.SmtpAppender.Builder smtpBuilder = SmtpAppender.newBuilder();
smtpBuilder.setName("emailAppender");
smtpBuilder.setSmtpUsername("test1#gmail.com");
smtpBuilder.setSmtpPassword("###YpSv1925");
smtpBuilder.setSmtpProtocol("https");
smtpBuilder.setSmtpHost("smtp.gmail.com");
smtpBuilder.setBufferSize(512);
smtpBuilder.setTo("test2#gmail.com");
smtpBuilder.setSubject("testing");
}
}
How should I add the smtpAppender to the configutation or the rootLogger ?
You are mixing up two APIs:
the ConfigurationBuilder API, which is the closest code equivalent to the configuration files. It only creates definitions of the actual logging components, the real ones are created when Configuration#initialize() is called on the configuration object. You can create the definition of an SMTPAppender like this:
private Configuration createConfig() {
ConfigurationBuilder<BuiltConfiguration> builder = ConfigurationBuilderFactory.newConfigurationBuilder()//
.setStatusLevel(Level.DEBUG);
LayoutComponentBuilder layoutBuilder = builder.newLayout("PatternLayout")//
.addAttribute("pattern", "%d{MM-dd#HH\\:mm\\:ss}%-4r %-5p [%t] %37c %3x - %m%n");
AppenderComponentBuilder appenderBuilder = builder.newAppender("emailAppender", "SMTP")//
.addAttribute("smtpUsername", "test1#gmail.com")
.addAttribute("smtpPassword", "###YpSv1925")
.addAttribute("smtpProtocol", "smtps")
.addAttribute("smtpHost", "smtp.gmail.com")
.addAttribute("to", "test2#gmail.com")
.addAttribute("subject", "testing")
.add(layoutBuilder);
AppenderRefComponentBuilder appenderRefBuilder = builder.newAppenderRef("emailAppender");
RootLoggerComponentBuilder rootLoggerBuilder = builder.newRootLogger(Level.DEBUG)//
.add(appenderRefBuilder);
return builder.add(appenderBuilder)//
.add(rootLoggerBuilder)
.build();
}
the actual builders of Log4j 2.x components, which are called by reflection by Configuration#initialize using the definitions above. You can also use them directly:
private static Configuration createConfig2() {
return new AbstractConfiguration(null, ConfigurationSource.NULL_SOURCE) {
#Override
protected void doConfigure() {
Layout<String> layout = PatternLayout.newBuilder()//
.withPattern("%d{MM-dd#HH\\:mm\\:ss}%-4r %-5p [%t] %37c %3x - %m%n")
.withConfiguration(this)
.build();
Appender appender = SmtpAppender.newBuilder()//
.setName("emailAppender")
.setSmtpUsername("test1#gmail.com")
.setSmtpPassword("###YpSv1925")
.setSmtpProtocol("smtps")
.setTo("test2#gmail.com")
.setSubject("testing")
.setLayout(layout)
.setConfiguration(this)
.build();
LoggerConfig rootLogger = getRootLogger();
rootLogger.setLevel(Level.DEBUG);
rootLogger.addAppender(appender, null, null);
}
};
}
Both Configurations are equivalent and you can apply them on the current context with:
Configurator.reconfigure(config);
However they will be lost upon Configurator.reconfigure(), unless you define your own ConfigurationFactory.
Log4j2 does not recreate log files if they were deleted in runtime. For example, careless admin have removed log-files where app currently write own logs.
Actual result: logs doesn't write to file.
Wanted result: log4j2 recreate file after first attempt to write into it and continue to work with this file.
Manual recreating by cron or somehow else is not working because log4j2 "remembers" file descriptor of file and continiue to work with it even after old file was deleted and new was created.
On the StackOverflow I found only one workaround (https://stackoverflow.com/a/51593404/5747662) and it looks like this:
package org.apache.log4j;
import java.io.File;
import org.apache.log4j.spi.LoggingEvent;
public class ModifiedRollingFileAppender extends RollingFileAppender {
#Override
public void append(LoggingEvent event) {
checkLogFileExist();
super.append(event);
}
private void checkLogFileExist(){
File logFile = new File(super.fileName);
if (!logFile.exists()) {
this.activateOptions();
}
}
}
I don't like it beŃause:
1) It "little bit" slow
Each time when we will write event we will also execute checkLogFileExist() and check file in filesystem.
2) It doesn't works for Log4j2
There is no method activateOptions() in Log4j2 infractucture.
So does anybody faced with same problem? How do you solved it?
UPDATE
I have tried to initialyze Triggering Policy to manually "rollover" deleted file, but it's not working for me.
My code:
final LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
// loggerName is name of logger which should work with the file has been deleted.
LoggerConfig loggerConfig = ctx.getConfiguration().getLoggerConfig(loggerName);
// I also know what appender (appenderName) should work with this file.
RollingFileAppender appender = (RollingFileAppender) loggerConfig.getAppenders().get(appenderName);
appender.getTriggeringPolicy().initialize(appender.getManager());
Also my config:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR">
<Appenders>
<RollingFile name="FILE_LOG">
<FileName>../log/temp/server.log</FileName>
<FilePattern>../log/server/SERVER_%d{yyyy-MM-dd-hh-mm}.log</FilePattern>
<PatternLayout pattern="%d{dd.MM.yyyy HH:mm:ss} [%t] %-5level %msg%n"/>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
</RollingFile>
<RollingFile name="OUTPUT_LOG">
<FileName>../log/temp/output.log</FileName>
<FilePattern>../log/output/OUTPUT_%d{yyyy-MM-dd-hh-mm}.log</FilePattern>
<PatternLayout>
<Pattern>%d{dd.MM.yyyy HH:mm:ss} %msg</Pattern>
</PatternLayout>
<Policies>
<CronTriggeringPolicy schedule="0 0 * * * ?"/>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="50 MB" />
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="OUTPUT" level="debug" additivity="false">
<AppenderRef ref="OUTPUT_LOG" />
</Logger>
<Root level="debug">
<AppenderRef ref="FILE_LOG" />
</Root>
</Loggers>
</Configuration>
I'm finally find the solution. Thanx #Alexander in comments for tip.
Short: We can manually initialize rollover process when detect file deletition.
Longer:
I implement it this way:
1) Create FileWatchService which will (1) subscribe for the log-file deletiiton events in your log folder and (2) notify you when these events will occur. It can be done by java.nio.file.WatchService (https://docs.oracle.com/javase/tutorial/essential/io/notification.html). I will provide my code below.
2) Create some other class which will initialize rollover when FileWatchService will notify about file deletition. I'm also will provide my full code below, but main magic will be occur this way:
final LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
// You should know only appender name.
RollingFileAppender appender = (RollingFileAppender) ctx.getConfiguration().getAppenders().get(appenderName);
if (appender != null) {
// Manually start rollover logic.
appender.getManager().rollover();
}
My code looks like this (not ideal but it's working for me):
FileWatchService:
public class FileWatchService implements Runnable {
private final org.apache.logging.log4j.Logger logger = LogManager.getLogger(FileWatchService.class);
private WatchService watchService = null;
private Map<WatchKey,Path> keys = null;
private String tempPath;
public FileWatchService(String tempPath) {
try {
this.watchService = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<WatchKey,Path>();
this.tempPath = tempPath;
Path path = Paths.get(tempPath);
register(path);
logger.info("Watch service has been initiated.");
}
catch (Exception e) {
logger.error("The error occurred in process of registering watch service", e);
}
}
// Method which register folder to watch service.
private void register(Path tempPath) throws IOException {
logger.debug("Registering folder {} for watching.", tempPath.getFileName());
// Registering only for delete events.
WatchKey key = tempPath.register(watchService, ENTRY_DELETE);
keys.put(key, tempPath);
}
#Override
public void run() {
try {
Thread.currentThread().setName("FileWatchService");
this.processEvents();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void processEvents() throws InterruptedException {
WatchKey key;
// Waiting until event occur.
while ((key = watchService.take()) != null) {
// Poll all events when event occur.
for (WatchEvent<?> event : key.pollEvents()) {
// Getting type of event - delete, modify or create.
WatchEvent.Kind kind = event.kind();
// We are interested only for delete events.
if (kind == ENTRY_DELETE) {
// Sending "notification" to appender watcher service.
logger.debug("Received event about file deletion. File: {}", event.context());
AppenderWatcher.hadleLogFileDeletionEvent(this.tempPath + event.context());
}
}
key.reset();
}
}
}
Another class for initilize rollover (I have called it AppenderWatcher):
public class AppenderWatcher {
private static final org.apache.logging.log4j.Logger logger = LogManager.getLogger(AppenderWatcher.class);
public static void hadleLogFileDeletionEvent(String logFile) {
File file = new File(logFile);
if (!checkFileExist(file)) {
logger.info("File {} is not exist. Starting manual rollover...", file.toString());
// Getting possible appender name by log-file.
String appenderName = getAppenderNameByFileName(logFile);
// Getting appender from list of all appender
RollingFileAppender appender = (RollingFileAppender) getAppender(appenderName);
if (appender != null) {
// Manually start rollover logic.
appender.getManager().rollover();
logger.info("Rollover finished");
}
else {
logger.error("Can't get appender {}. Please, check lo4j2 config.", appenderName);
}
} else {
logger.warn("Received notification what file {} was deleted, but it exist.", file.getAbsolutePath());
}
}
// Method which checks is file exist. It need to prevent conflicts with Log4J rolling file logic.
// When Log4j rotate file it deletes it first and create after.
private static boolean checkFileExist(File logFile) {
return logFile.exists();
}
// Method which gets appender by name from list of all configured appenders.
private static Appender getAppender(String appenderName) {
final LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
return ctx.getConfiguration().getAppenders().get(appenderName);
}
// Method which returns name of appender by log file name.
// ===Here I'm explaining some customer specific moments of log4j config.
private static String getAppenderNameByFileName(String fileName) {
return getLoggerNameByFileName(fileName) + "_LOG";
}
// This method fully customer specific.
private static String getLoggerNameByFileName(String fileName) {
// File name looks like "../log/temp/uber.log" (example).
String[] parts = fileName.split("/");
// Last part should look like "uber.log"
String lastPart = parts[parts.length - 1];
// We need only "uber" part.
String componentName = lastPart.substring(0, lastPart.indexOf("."));
return componentName.toUpperCase();
}
}
Is there a way to print a different logging.pattern given a specific logging.level?
The yml file I'm using in my Spring Boot project looks like this
logging:
level:
com.netflix: warn
org.springframework: warn
org.apache: warn
org.mongodb: warn
com.twocatholler: debug
pattern:
console: "%d{dd-MM-yyyy HH:mm:ss.SSS} %white([%thread]) %highlight(%-5level) %white(%logger) - %white(%msg) %n"
If having multiple appenders is acceptable, a straightforward way to have different patterns for different levels would be using two different appenders with ThresholdFilters.
There can be issues when more appenders are writing to the same file. Check this issue for example: https://jira.qos.ch/browse/LOGBACK-114
However, you can have your own appender which formats logging events differently, based on their level.
Check this LevelPatternLayout demo: https://github.com/riskop/logback_LevelPatternLayout
example logback.xml:
<configuration scan="true" debug="true">
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="pack.LevelPatternLayout">
<defaultPattern>DEFAULT PATTERN %c - %m\n</defaultPattern>
<debugPattern>DEBUG PATTERN %c - %m\n</debugPattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
test code:
public void test() throws Exception{
log.debug("some debug level log");
log.info("some info level log");
log.error("some error level log");
}
output:
DEBUG PATTERN pack.TestStart - some debug level log
DEFAULT PATTERN pack.TestStart - some info level log
DEFAULT PATTERN pack.TestStart - some error level log
LevelPatternLayout code:
package pack;
import java.util.HashMap;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.PatternLayout;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.LayoutBase;
/**
* A logback layout which can be configured with different pattern for different levels.
*
* E.g. you can configure a pattern for DEBUG messages and an *other* pattern
* for INFO messages and so on.
*
*
*/
public class LevelPatternLayout extends LayoutBase<ILoggingEvent> {
private boolean started;
// store for patternLayouts per level. The layout stored with 'null' key is the fallback
private HashMap<Level, PatternLayout> layouts = new HashMap<>();
public LevelPatternLayout() {
}
public void setDefaultPattern(String pattern) {
setLevelPattern(null, pattern);
}
public void setTracePattern(String pattern) {
setLevelPattern(Level.TRACE, pattern);
}
public void setDebugPattern(String pattern) {
setLevelPattern(Level.DEBUG, pattern);
}
public void setInfoPattern(String pattern) {
setLevelPattern(Level.INFO, pattern);
}
public void setWarnPattern(String pattern) {
setLevelPattern(Level.WARN, pattern);
}
public void setErrorPattern(String pattern) {
setLevelPattern(Level.ERROR, pattern);
}
private void setLevelPattern(Level level, String pattern) {
PatternLayout layout = new PatternLayout();
layout.setContext(context);
layout.setPattern(pattern);
layouts.put(level, layout);
}
#Override
public String doLayout(ILoggingEvent event) {
PatternLayout matchingLayout = layouts.get(event.getLevel());
if(matchingLayout != null) {
return matchingLayout.doLayout(event);
}
return layouts.get(null).doLayout(event);
}
#Override
public void start() {
if(layouts.get(null) == null) {
throw new RuntimeException("default layout is not initialized, probably no 'defaultPattern' is configured!");
}
layouts.values().forEach(layout -> layout.start());
this.started = true;
}
#Override
public void stop() {
layouts.values().forEach(layout -> layout.stop());
this.started = false;
}
#Override
public boolean isStarted() {
return started;
}
}
I have "special" requirement on loggin - I need every logger in separate file.
Java
Logger log1 = LoggerFactory.getLogger("dynamic.log1");
Logger log2 = LoggerFactory.getLogger("dynamic.log2");
//...
Then I want logback any output from log1 to be written to file log1.log and so on. Is it possible to "dynamicaly" create new appender's like that with logback?
Can some other logging framework be used to solve this use-case?
I could configure appenders manualy but this is what I want to avoid. Like whenever I add dynamic logger, new appender/file is accordingly created.
EDIT:
I implemented custom discriminator:
public class LoggerBasedDiscriminator extends AbstractDiscriminator<ILoggingEvent> {
private static final String LOGGER_NAME = "loggerName";
#Override
public String getDiscriminatingValue(ILoggingEvent e) {
return e.getLoggerName();
}
#Override
public String getKey() {
return LOGGER_NAME;
}
}
And then my appender config looks like this:
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="cz.svobol.logging.LoggerBasedDiscriminator">
<key>loggerName</key>
<defaultValue>root</defaultValue>
</discriminator>
<sift>
<appender name="FILE-${loggerName}" class="ch.qos.logback.core.FileAppender">
<file>${loggerName}.log</file>
<encoder>
<pattern>%msg%n</pattern>
</encoder>
</appender>
</sift>
</appender>
You can use a SiftingAppender.
This way you have one appender that can divide the log into different files dynamically.
I have a program, where I would like to be able to separate each single log-message into its own log file.
So if the class generates 10 ERROR logs and 10 DEBUG logs, within a single program execution, then that should create 20 log files , and they're names can ideally be something like :
logoutput1
logoutput2
logoutput3
..etc
And each log file just has a single line .
I'm working on an project where I'd like to implement some autonomic ability - the idea is that we can have a third , externally running program which can read in those log files(and then react based on them)
Is this possible with Log4j ? how can this be done ?
thanks !
Yes, you can use the RoutingAppender. See this question for details: Log4j2: Dynamic creation of log files for multiple logs
Write your own Log File Appender and create a new file every time when it attempts to write some log. The following piece of code might help you.
public class SingleLogMsgFileAppender extends FileAppender {
private String file = null;
private static long fileNo;
public SingleLogMsgFileAppender() {
super();
fileNo = 1;
}
#Override
protected void subAppend(LoggingEvent event) {
createNewFile(true);
synchronized (this) {
super.subAppend(event);
}
}
#Override
public void setFile(String file) {
this.file = file;
createNewFile(false);
}
public void createNewFile(boolean incrementFileNo) {
try {
String fileName = file + "testlogfile." + fileNo + ".log";
super.setFile(fileName);
super.activateOptions();
} catch (Exception e) {
e.printStackTrace();
}
if (incrementFileNo) {
fileNo++;
}
}
}
and here is the log4j configuration file
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="CustomAppender" class="loga.randd.threads.log4j.SingleLogMsgFileAppender">
<param name="File" value="log/" />
<param name="Append" value="true" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{MM_dd_yyyy HH_mm_ss}%m%n" />
</layout>
</appender>
<root>
<priority value="debug" />
<appender-ref ref="CustomAppender" />
</root>
</log4j:configuration>