Expectation is to have the current log with the same name (i.e test.log)
and the archival files name should be test_(CurrentDate).log.1.
/logs/projectlogs/test.log
/logs/projectlogs/ test_20160430.log.1
/logs/projectlogs/ test_20160430.log.2
/logs/projectlogs/ test_20160430.log.3
Using below properties file, current date is getting appended with all the files.
log4j.properties::
log4j.rootLogger= ALL, A1, file, rollingAppender
#log4j.date=contains current date
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=C:/logs/projectlogs/test_${log4j.date}.log
log4j.appender.file.MaxFileSize=100KB
log4j.appender.file.MaxBackupIndex=5
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss:SSS zzz} %-5p[%t] %m%n
The log4j 2 manual has many example configurations, and the section on RollingFileAppender has an example that matches your requirements:
http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppender
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
<Appenders>
<RollingFile name="RollingFile" fileName="logs/projectlogs/test.log"
filePattern="logs/projectlogs/$${date:yyyy-MM}/test-%d{yyyy-MM-dd-HH}-%i.log.gz">
<PatternLayout>
<Pattern>%d %-5p[%t] %c{1.} %m%n</Pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval="5" modulate="true"/>
<SizeBasedTriggeringPolicy size="100 MB"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
What may also be of interest to you is that Log4j-2.5 introduced a Delete action that gives users more control over what files are deleted at rollover time. See http://logging.apache.org/log4j/2.x/manual/appenders.html#CustomDeleteOnRollover
What you want to achieve is called "Size And Time Based File Naming And Triggering Policy" (SizeAndTimeBasedFNATP) and can be found in the latest release of logback package, which is at the moment 1.1.7, however, it can be found in 1.1.2 as well, which I use since it produces more predictable results without too much of asynchronous processes (even though 1.1.2 is slower).
The logback package works in similar way as log4j, but uses XML based configuration file (logback.xml) which in your case may look like one below (note test_%d{yyyyMMdd}.log.%i -- this is your pattern).
Please note, that unfortunately, it is not (yet) possible to limit indexes within dates. So you can't have only 5 chunks of logs for each date -- each date will be logged entirely with log chunks indexes independent for each date. You can, however, limit total number of dates and (in 1.1.7) total size of log folder (using ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy).
Also I suggest you use chunk size of 10Mb at least (100Kb is tiny), as bigger chunks, generally speaking, easier to maintain by log lib (less CPU consumption).
<?xml version="1.0" ?>
<configuration>
<property name="log.folder" value="C:/logs/projectlogs"/>
<!-- UNCOMMENT BELOW SECTION IF CONSOLE IS REQUIRED -->
<!--
<appender class="ch.qos.logback.core.ConsoleAppender" name="CONSOLE">
<encoder>
<pattern>[%p] [%thread] %logger - %msg%n</pattern>
</encoder>
</appender>
-->
<appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="FILE">
<File>${log.folder}/test.log</File>
<Append>true</Append>
<encoder>
<pattern>%d{ISO8601} [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.folder}/test_%d{yyyyMMdd}.log.%i</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>100KB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- keep 30 days' worth of history -->
<maxHistory>30</maxHistory>
<!-- up to 10 GB max -->
<totalSizeCap>10GB</totalSizeCap>
<MaxFileSize>100KB</MaxFileSize>
</rollingPolicy>
</appender>
<root>
<level value="INFO"/>
<appender-ref ref="FILE"/>
</root>
<logger name="Main">
<level value="DEBUG" />
</logger>
<logger name="ch.qos">
<level value="WARN"/>
</logger>
</configuration>
Test Java App
package logtester;
import org.apache.log4j.Logger;
public class LogTester {
public static void main(String[] args) {
Logger logger = Logger.getLogger("Main");
for(int i = 1; i<=20000; i++)
logger.info("Log message "+i);
}
}
Folder structure after run:
13,230 test.log
102,929 test_20160430.log.0
103,168 test_20160430.log.1
102,816 test_20160430.log.10
102,816 test_20160430.log.11
103,168 test_20160430.log.2
103,168 test_20160430.log.3
103,168 test_20160430.log.4
103,168 test_20160430.log.5
102,815 test_20160430.log.6
102,816 test_20160430.log.7
102,816 test_20160430.log.8
102,816 test_20160430.log.9
Hope it helps.
Related
I have configured logback xml for a spring boot project.
I want to configure another appender based on the property configured. We want to create an appender either for JSON logs or for text log, this will be decided either by property file or by environment variable.
So I am thinking about the best approach to do this.
Using filters to print logs to 1 of the file (either to JSON or to Txt). But this will create both of the appenders. I want to create only 1 appender.
Use "If else" blocks in logback XML file. To put if else around appenders, loggers seems untidy and error prone. So will try to avoid as much as possible.
So now exploring options where I can add appender at runtime.
So I want to know if it is possible to add appender at runtime. And will it be added before spring boots up or it could be done anytime in the project.
What could be the best approach to include this scenario.
As you're already using Spring, I suggest using Spring Profiles, lot cleaner than trying to do the same programmatically. This approach is also outlined in Spring Boot docs.
You can set an active profile from either property file:
spring.profiles.active=jsonlogs
or from environment value:
spring_profiles_active=jsonlogs
of from startup parameter:
-Dspring.profiles.active=jsonlogs
Then have separate configurations per profile:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout-classic" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{dd-MM-yyyy HH:mm:ss.SSS} %magenta([%thread]) %highlight(%-5level) %logger{36}.%M - %msg%n</pattern>
</encoder>
</appender>
<appender name="stdout-json" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
</encoder>
</appender>
<!-- begin profile-specific stuff -->
<springProfile name="jsonlogs">
<root level="info">
<appender-ref ref="stdout-json" />
</root>
</springProfile>
<springProfile name="classiclogs">
<root level="info">
<appender-ref ref="stdout-classic" />
</root>
</springProfile>
</configuration>
As the previous answer states, you can set different appenders based on Spring Profiles.
However, if you do not want to rely on that feature, you can use environments variables as described in the Logback manual. I.e.:
<appender name="json" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
<appendLineSeparator>true</appendLineSeparator>
</layout>
</encoder>
</appender>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>
%cyan(%d{HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}) - %msg%n
</pattern>
</encoder>
</appender>
<root level="info">
<!--
! Use the content of the LOGBACK_APPENDER environment variable falling back
! to 'json' if it is not defined
-->
<appender-ref ref="${LOGBACK_APPENDER:-json}"/>
</root>
I have the following logback.xml file for my spring boot application where I intend to create daily log files:
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true">
<appender name="rollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>./logs/my-log-file.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>15</maxHistory>
<totalSizeCap>15MB</totalSizeCap>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d [%-5thread] [%-1p] [%logger{35}] - %msg%n</Pattern>
</layout>
</appender>
<root level="WARN">
<appender-ref ref="rollingFile"/>
</root>
<root level="INFO">
<appender-ref ref="rollingFile"/>
</root>
</configuration>
The files are created like this:
my-log-file.2018-11-10.log
my-log-file.2018-11-11.log
my-log-file.2018-11-12.log
I am looking to have the current active log file without the date so it is generic name like my-log-file.log and only when the date rolls to a new day, the file is renamed with date.
What change is required on the logback.xml to enable this configuration?
Add a <file> to the <appender>, like so:
<appender name="rollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>./logs/my-log-file.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>./logs/my-log-file.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>15</maxHistory>
<totalSizeCap>15MB</totalSizeCap>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d [%-5thread] [%-1p] [%logger{35}] - %msg%n</Pattern>
</layout>
</appender>
From the Logback documentation:
Note that the file property in RollingFileAppender (the parent of
TimeBasedRollingPolicy) can be either set or omitted. By setting the
file property of the containing FileAppender, you can decouple the
location of the active log file and the location of the archived log
files. The current logs will be always targeted at the file specified
by the file property. It follows that the name of the currently active
log file will not change over time. However, if you choose to omit the
file property, then the active file will be computed anew for each
period based on the value of fileNamePattern.
I am Configured Logger For my application And Give the Logfilename as Current Time Stamp So It Expected to create one log File with the name as current Time stamp BuT INSTEDE IT CREATE ONE LOGFILE WITH CURRENT TIMESTAMP AND ANOTHER FILE WHICH IS BLANK CANT FIGURE OUT WHY IT CREATING EXTRA FILE??
I am using Logback logger in my application and Here is my logback.xml looks like My application is simple core java application. where i user logger to log the statements
<?xml version="1.0" encoding="UTF-8"?>
<configuration>`enter code here`
<timestamp key="byDay" datePattern="yyyy'-'MM'-'dd'''AT'''HH'-'mm'-'ss"/>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file> Myapplication${byDay}.txt </file>
<append>FALSE</append>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="FILE" />
</root>
</configuration>
My team is developing a telecom realtime application with high calls per second rate. We are using logback to filter log based on a key-value match (traffic live values, like Calling Party, and so on). The filtered log file is correctly created, once verified the match from live values and db values, but we would get rid of default file which is filled with logs when there is no match. It might happen that a traffic node needs to be monitored for a while before a key-value match takes place, so in the meantime the default could indefinitely increase in size and cause problems to performance and stability of node itself. What should I do in my logback.xml to avoid generation of default log file? Is it possible? Any other option to achieve same result?
logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<property scope="context" name="LOG_LEVEL" value="INFO" />
<appender name="SIFT_LOGGER" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="com.ericsson.jee.ngin.services.log.ServiceKeyDiscriminator">
</discriminator>
<sift>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<prudent>true</prudent>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>/var/log/tit/logback_${serviceKey}_%d{yyyy-MM-dd}_%i.log</fileNamePattern>
<maxFileSize>1MB</maxFileSize>
<maxHistory>10</maxHistory>
<totalSizeCap>2GB</totalSizeCap>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>${LOG_LEVEL}</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<!-- encoders are by default assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder -->
<encoder>
<pattern> %d{yyyy-MM-dd HH:mm:ss.SSSZ} [%thread] %-5level %logger{36} %msg%n</pattern>
</encoder>
</appender>
</sift>
</appender>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are by default assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder -->
<encoder>
<pattern> %d{yyyy-MM-dd HH:mm:ss.SSSZ} [%thread] %-5level %logger{36} %msg%n</pattern>
</encoder>
</appender>
<turboFilter class="ch.qos.logback.classic.turbo.DynamicThresholdFilter">
<key>serviceKey</key>
<defaultThreshold>DEBUG</defaultThreshold>
<onHigherOrEqual>ACCEPT</onHigherOrEqual>
<onLower>ACCEPT</onLower>
</turboFilter>
<root level="DEBUG">
<appender-ref ref="SIFT_LOGGER" />
<appender-ref ref="STDOUT" />
</root>
ATTACHMENTS: FILTERED LOGBACK CLASS
The provided FL class is only working for a SK which has a java Discriminator in FL module.
You must move the filter to the general sift appender.
<appender name="SIFT-TRACE"
class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator
class="ch.qos.logback.classic.sift.MDCBasedDiscriminator">
<Key>loggerFileName</Key>
<DefaultValue>unknown</DefaultValue>
</discriminator>
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator
class="ch.qos.logback.classic.boolex.JaninoEventEvaluator">
<expression>
mdc.get("loggerFileName")!=null
</expression>
</evaluator>
<OnMismatch>DENY</OnMismatch>
<OnMatch>NEUTRAL</OnMatch>
</filter>
<sift>
<appender name="TRACE-${loggerFileName}"
class="ch.qos.logback.core.FileAppender">
<File>D:/U/${loggerFileName}.log</File>
<Append>true</Append>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d [%thread] %level %mdc %logger - %msg%n</Pattern>
</layout>
</appender>
</sift>
</appender>
<logger name="org.springframework" level="DEBUG" />
<root level="DEBUG">
<appender-ref ref="SIFT-TRACE" />
</root>
Also to make it work correctly, you MUST after each Thread/file/marker/etc. to put those statements:
public void handle()
{
MDC.put("loggerFileName","some value");
...
MDC.remove("loggerFileName");
}
You have defined this root logger:
<root level="DEBUG">
<appender-ref ref="SIFT_LOGGER" />
<appender-ref ref="STDOUT" />
</root>
This means that all log events with Level >= DEBUG will be directed to two appenders:
SIFT_LOGGER
STDOUT
If I understand your question correctly then you do want logs to be written via your SIFT_APPENDER but you don't want any other log output. If so, then just remove this entry:
<appender-ref ref="STDOUT" />
The STDOUT appender is a console appender so it doesn't actually write to a log file instead it writes to System.out. I suspect the reason you are seeing these log events in some file is that whatever is running your application is redirecting System.out to a file. As long as you only have your SIFT_APPENDER in the root logger definition then you can be confident that this will be the only appender in play. Note: once you remove the appender from the root logger you can probbaly remove it from logback.xml since it is unused.
Update 1: Based on your last comment I now understand that you want to discard the logs which arrive at the SiftingAppender but do not match a given condition. I suspect what's happeneing here is that some log events arrive at the sifting appender with an 'unknown' value for serviceKey and these events are then written to /var/log/tit/logback_[unknownValue]_%d{yyyy-MM-dd}_%i.log. Is this the crux of the issue? If so, then you can add a filter into the nested appender. Here are some examples:
Using Groovy to express the 'contains unknown serviceKey condition':
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<!-- GEventEvaluator requires Groovy -->
<evaluator
class="ch.qos.logback.classic.boolex.GEventEvaluator">
<expression>
serviceKey == null
</expression>
</evaluator>
<OnMismatch>NEUTRAL</OnMismatch>
<OnMatch>DENY</OnMatch>
</filter>
Using Janino to express the 'contains unknown serviceKey condition':
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<!-- JaninoEventEvaluator requires Janino -->
<evaluator
class="ch.qos.logback.classic.boolex.JaninoEventEvaluator">
<expression>
serviceKey == null
</expression>
</evaluator>
<OnMismatch>NEUTRAL</OnMismatch>
<OnMatch>DENY</OnMatch>
</filter>
With either of these filters in place any log events which arrive at the sifting appender and have the 'unknown' serviceKey will be ignored. Note: I have written the 'contains unknown serviceKey condition' as serviceKey == null your logic might differ but the above examples show should you how to tell Logback to apply this filter for you.
Just to notify to #glitch (and all others interested) the happy conclusion of this issue: I have managed to make the tag expression working was this:
<expression>mdc.get("servicekey") == null</expression>
Thanks to this expression, I have got the wanted behavior: the default file "IS_UNDEFINED is not generated when the key doesn't match with the runtime traffic values.
The reason is because the type of Event in JaninoEventEvaluator is LoggingEvent that has a reserve object "mdc" (the type is Map).
Regards,
Pierluigi
We have WAR deployments running on Tomcat containers which are using log4j 2.5 for logging events. We have now amended the deployments' log4j2.xml configuration to have the log files roll over every 24 hours but, with this new configuration, the rollover of files are not taking place as we would expect.
Sample configuration:
<RollingFile name="file"
fileName="${sys:catalina.base}/logs/${web:contextPath}.log"
filePattern="${sys:catalina.base}/logs/${web:contextPath}-%d{dd-MMM-yyyy}.log"
append="true">
<PatternLayout pattern="%d{dd-MMM-yyyy HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" >
<header>LOG START DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</header>
<footer>LOG END DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</footer>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
Any ideas why the rollover is not taking place?
NOTE: The same configuration but with a <CronTriggeringPolicy schedule="0 0 0 * * ?" /> instead of TimeBasedTriggeringPolicy does rollover but, in this case, the rolled over files get created with today's date in the filename and NOT yesterday's date.
NOTE2: We have other deployments with similar configuration that do rollover every 24 hours but those configurations have the filename hardcoded instead of using ${web:contextPath}. Could this lookup have something to do with why RollingFile might not work?
--- EDIT ---
UPDATE: We are able to get TimeBasedTriggeringPolicy to rollover files using above configuration when the Tomcat instance is running on Windows but NOT when the Tomcat instance is running on Linux.
There is nothing wrong with your configuration snippet as I get the desired behaviour of time based rolling.
To test, I changed the dd-MMM-yyyy to dd-MMM-yyyy-HH-mm and my log file rolls every minute.
It must be something else that is preventing you from achieving the desired behaviour.
My setup #1:
Log4j2 v2.8.2
Apache Tomcat 8.5.13
Windows 7 Enterprise SP1
My setup #2:
Log4j2 v2.5
Apache Tomcat 7.0.77
CentOS 7 (1611)
I have the following 3 JARs in WEB-INF/lib:
log4j-api-2.5.jar
log4j-core-2.5.jar
log4j-web-2.5.jar
Here is my complete log4j2.xml for your reference:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
<Appenders>
<RollingFile name="RollingFileAppender"
fileName="${sys:catalina.base}/logs/${web:contextPath}.log"
filePattern="${sys:catalina.base}/logs/${web:contextPath}-%d{dd-MMM-yyyy-HH-mm}.log"
append="true">
<PatternLayout pattern="%d{dd-MMM-yyyy HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" >
<header>LOG START DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</header>
<footer>LOG END DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</footer>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="root" level="debug" additivity="false">
<appender-ref ref="RollingFileAppender" level="debug"/>
</Logger>
<Root level="debug" additivity="false">
<AppenderRef ref="RollingFileAppender"/>
</Root>
</Loggers>
</Configuration>