I support a solution that is using logback to generate logfiles. The issue is that the generated logfiles are created so fast that they are filling up the hard drive and causing the system to crash.
When analyzing the created logfiles we can see that they are 5MB in size and when they roll over the next logfile is the same data from the previous one with only 15-20 trace lines added, then it will create the next one in the same format and continue at a fast rate. Has anyone ever seen this behavior using logback? The issue is with the esuiteStore* logs:
${esStoreLogFilePath}/esuiteStore-%d{yyyy-MM-dd}.%i.log
This is not seen in our other customers and we have not been able to recreate it in our test environment (using the same SW and logback.xml). On my system the logfiles are created correctly (the same logging is not copied into the next rollover file). My understanding is that it should be creating 5MB rollover logfiles, and save four days worth of logging.
I am not exactly sure what version of logback I am using but I found the following in the c:\java directory.
Manifest.mf in logback-classic-0.9.26.jar file Build-Jdk: 1.6.0_16
Java(TM) 6 Update 13 6.0.130 is installed.
Logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<contextName>SSCStoreServer</contextName>
<jmxConfigurator/>
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<level value="INFO" />
<file>${esStoreLogFilePath}/esuiteStore.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>${esStoreLogFilePath}/esuiteStore-%d{yyyy-MM-dd}.%i.log</FileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>5MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<MaxHistory>4</MaxHistory>
<MinIndex>1</MinIndex>
<MaxIndex>4</MaxIndex>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%date %level [%thread] %logger{10}:%L %msg%n
</Pattern>
</layout>
<filter class="com.ncr.ssc.cf.common.util.LoggerNameFilter">
<LoggerName>org.hibernate</LoggerName>
<OnMismatch>NEUTRAL</OnMismatch>
<OnMatch>DENY</OnMatch>
</filter>
</appender>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<level value="INFO" />
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%date %level [%thread] %logger{10}:%L %msg%n
</Pattern>
</layout>
<filter class="com.ncr.ssc.cf.common.util.LoggerNameFilter">
<LoggerName>org.hibernate</LoggerName>
<OnMismatch>NEUTRAL</OnMismatch>
<OnMatch>DENY</OnMatch>
</filter>
</appender>
<appender name="METRICS"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<level value="INFO" />
<file>${esStoreLogFilePath}/personalizationMetrics.csv</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<FileNamePattern>personalizationMetrics%i.csv</FileNamePattern>
<MinIndex>1</MinIndex>
<MaxIndex>2</MaxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>5MB</MaxFileSize>
</triggeringPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%msg%n
</Pattern>
</layout>
<filter class="com.ncr.ssc.ss.personalizationcontroller.LoggerMetricsFilter">
<LoggerName>Metrics</LoggerName>
<OnMismatch>DENY</OnMismatch>
<OnMatch>ACCEPT</OnMatch>
</filter>
</appender>
<root>
<level value="INFO" />
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
<appender-ref ref="METRICS" />
</root>
</configuration>
The issue was that there were multiple processes running trying to access the logfile so once it tried to rollover it started creating multiple logfiles with the same data. The process list showed multiple instances of Java running.
Related
I have my current logback file, which to my understanding is a classic logback implementation:
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>Hello %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="debug">
<appender-ref ref="STDOUT" />
</root>
</configuration>
I have a specific log:
log.info(String.format("ProxyICA %s has an integrity violation. %s.",
processorId.getProxyIca(),
fkViolationMsg)
);
that I am trying to get in a text file. The only configuration I need in my logback is to put every instance of this log into a text file. Every other log can default log to the console (which it was doing without the logback.xml).
Thank you in advanced!
Here is an example of setting it up log to a specific file "Custom-Errors.txt", and to have the logs be deleted if they reach a certain size 20MB or 182 days elapse
<appender name="Custom_Logger
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>Custom-Errors.txt</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>Custom-Errors-%d{yyyy-MM-dd}.%i.txt</fileNamePattern>
<!-- each file should be at most 20MB, keep 182 days worth of history,but
at most 500MB -->
<maxFileSize>20MB</maxFileSize>
<maxHistory>182</maxHistory>
<totalSizeCap>500MB</totalSizeCap>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO, DEBUG, TRACE</level>
</filter>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%date %level [%thread] [%file:%line] %msg%n%throwable
</Pattern>
</layout>
</appender>
<logger name="CUSTOM_LOGGER" additivity="false">
<appender-ref ref="Custom_Logger" />
</logger>
Then in your java class make a private variable to use the logger like this
private static final Logger CUSTOM_LOGGER =
LoggerFactory.getLogger("CUSTOM_LOGGER");
logging a dependency, or use to log anything you want based on package
here is an example of logging a library
<appender name="ConsoleOutForSpring" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%black(%d{ISO8601}) %highlight(%-5level) [%blue(%t)] %yellow(%C{1.}): %msg%n%throwable
</Pattern>
</layout>
</appender>
<logger name="org.springframework" additivity="false">
<appender-ref ref="ConsoleOutForSpring" />
</logger>
I do admit there are a lot of tags and settings to this logger that are kind of complicated, but reading docs and looking at others examples has helped me.
Here were some docs I noted in my logback.xml
<!-- http://logback.qos.ch/manual/configuration.html this helped me with this weird looking stuff i.e. ===> %date %level [%thread] [%file:%line] %msg%n%throwable -->
<!-- http://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy
helped me set this up:
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>Errors-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<maxFileSize>20MB</maxFileSize>
<maxHistory>182</maxHistory>
<totalSizeCap>500MB</totalSizeCap>
</rollingPolicy>
I have a spring boot app(spring-boot-starter-parent --> 2.3.0.RELEASE) that I am implementing log.
Basically what I am trying to achieve is that I have 2 log files one bridgeError.log which contains only error messages and bridge.log which will contain all log level except Error level since it is being logged in brideError.log file.
I am able to create 2 logs file brideError.log which contains only error messages but the issue is that
bridge.log also contains error messages.
Please find my logback-spring.xml configuration below:
<configuration>
<property name="HOME_LOG" value="logs/bridge.log"/>
<property name="HOME_LOG_ERROR" value="logs/bridgeError.log"/>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE-ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${HOME_LOG}</file>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>DEBUG</level>
</filter>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>logs/archived/bridge.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<totalSizeCap>20GB</totalSizeCap>
<!-- 60 days to keep -->
<maxHistory>60</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
</encoder>
</appender>
<appender name="FILE-ROLLING-ERROR" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${HOME_LOG_ERROR}</file>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>logs/archived/bridgeError.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<totalSizeCap>20GB</totalSizeCap>
<!-- 60 days to keep -->
<maxHistory>60</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
</encoder>
</appender>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n
</Pattern>
</layout>
</appender>
<logger name="com.dummy.package" level="debug" additivity="false">
<!-- <appender-ref ref="FILE-ROLLING"/>-->
<appender-ref ref="CONSOLE"/>
</logger>
<!-- <root level="error">
<appender-ref ref="FILE-ROLLING"/>
<!– <appender-ref ref="CONSOLE"/>–>
</root>-->
<logger name="com.dummy.package" level="DEBUG">
<appender-ref ref="FILE-ROLLING"/>
<appender-ref ref="FILE-ROLLING-ERROR"/>
</logger>
</configuration>
Is it possible to exclude error messages in bridge.log file?
Thanks in advance
You can use a level filter for that purpose
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>DENY</onMatch>
<onMismatch>ACCEPT</onMismatch>
</filter>
Docs: http://logback.qos.ch/manual/filters.html
I am using below logback.xml in my Micronaut project, it is not generating new log file as per rollingPolicy provided in xml configuration. I tried with SizeAndTimeBasedRollingPolicy and TimeBasedRollingPolicy but it did't worked.
Micronaut version: 1.2.2
logback-classic : 1.2.3
File: logback.xml
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>logs/my-app.log</file>
<encoder>
<pattern>%cyan(%d{yyyy-MM-dd HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}):%line- %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.RollingFileAppender">
<fileNamePattern>
logs/my-app.log-%d{yyyy-MM-dd}-%i.log.gz
</fileNamePattern>
<maxHistory>30</maxHistory>
<totalSizeCap>1MB</totalSizeCap>
</rollingPolicy>
</appender>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%cyan(%d{yyyy-MM-dd HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}):%line- %msg%n</pattern>
</encoder>
</appender>
<root level="ALL">
<appender-ref ref="FILE" />
<appender-ref ref="STDOUT" />
</root>
</configuration>
If im not wrong, your maxHistory policy is preventing you from building more than one file. Instead it deletes the old one and just creates a new one for the new day or when the 1mb in size is reached.
maxHistory in this case has to be combined with the TimeBasedRollingPolicy as you stated and maxHistory is the parameter for days then. Without the TimeBasedRollingPolicy it might just be the number of files.
Based on my own experience with logback, you should do something like:
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
...
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>logs/my-app.log-%d{yyyy-MM-dd}-%i.log.gz</fileNamePattern>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>1MB</MaxFileSize>
</triggeringPolicy>
...
</appender>
Currently, you are using "RollingFileAppender" for policy, so it can't work.
I have a java project
I am using Logback as a logging framework instead of the log4j project.
Logback's architecture is sufficiently generic so as to apply under different circumstances.
I have this logback.xml file expecting no more than 10 log files at the same time, but Its not the case
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property name="DEV_HOME" value="" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>
%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n
</Pattern>
</layout>
</appender>
<appender name="FILE-AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>Calzada.log</file>
<encoder>
<pattern>%d{"yyyy-MM-dd HH:mm"} [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>ATrackAT1Handler.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
<maxHistory>10</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10KB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-ERROR" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>CalzadaError.log</file>
<encoder>
<pattern>%d{"yyyy-MM-dd HH:mm"} [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>ATrackAT1HandlerError.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>10KB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<logger name="com.calzada" level="debug"
additivity="false">
<appender-ref ref="FILE-AUDIT" />
<appender-ref ref="STDOUT" />
</logger>
<root level="error">
<appender-ref ref="FILE-ERROR" />
</root>
</configuration>
I think your configuration is mixing up Logback policies.
In your configuration you use: TimeBasedRollingPolicy and then attempt to set minIndex and maxIndex but those configuration properties are not available on a TimeBasedRollingPolicy, instead they are properties of a FixedWindowRollingPolicy.
See the docs for TimeBasedRollingPolicy and FixedWindowRollingPolicy.
Here are some examples:
The following appender definition will cause Calzada.log to be rolled over to ATrackAT1Handler.%i.log everytime Calzada.log reaches 10KB and Logback will retain a maximum of 10 historic files i.e. ATrackAT1Handler.1.log to ATrackAT1Handler.10.log
<appender name="FILE-AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>Calzada.log</file>
<encoder>
<pattern>%d{"yyyy-MM-dd HH:mm"} [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>ATrackAT1Handler.%i.log</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10KB</maxFileSize>
</triggeringPolicy>
</appender>
The following appender definition will cause Calzada.log to be rolled over to ATrackAT1Handler.<yyy-MM-dd>.%i.log everytime Calzada.log reaches 10KB and Logback will retain a maximum of 10 days of historic log files but (and this is an important distinction) there is no cap on the number of files Logback will rollover within each day. So, you could have ATrackAT1Handler.2017-09-06.0.log to ATrackAT1Handler.2017-09-06.20.log and then the next day you could have ATrackAT1Handler.2017-09-07.0.log to ATrackAT1Handler.2017-09-07.12.log etc.
<appender name="FILE-AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>Calzada.log</file>
<encoder>
<pattern>%d{"yyyy-MM-dd HH:mm"} [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>ATrackAT1Handler.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<maxFileSize>10KB</maxFileSize>
<maxHistory>10</maxHistory>
</rollingPolicy>
</appender>
So, FixedWindowRollingPolicy allows you to control the number and size of historic files whereas TimeBasedRollingPolicy allows you to control the number of days of historic files and the size of each file. There seems to be a piece missing with TimeBasedRollingPolicy, namely, how can we limit the total size of all files across all days? To fill this gap the TimeBasedRollingPolicy provides the configuration property: totalSizeCap. From the docs:
totalSizeCap int
The optional totalSizeCap property controls the total size of all archive files. Oldest archives are deleted asynchronously when the total size cap is exceeded. The totalSizeCap property requires maxHistory property to be set as well. Moreover, the "max history" restriction is always applied first and the "total size cap" restriction applied second.
So, you can either go with FixedWindowRollingPolicy and you'll limit the number of files but you won't have daily rollovers or use TimeBasedRollingPolicy to get daily rollovers and retain some control over historic log file footprint by ising totalSizeCap.
This is my logback configuration file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<contextName>xpofacebook</contextName>
<appender name="ERROR"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>log/xpofacebook-error.log</File>
<Append>true</Append>
<BufferedIO>false</BufferedIO>
<ImmediateFlush>true</ImmediateFlush>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%date %level [%thread] %logger{10} [%file:%line] %msg%n</Pattern>
</layout>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<FileNamePattern>log/error.log.%i.zip</FileNamePattern>
<MinIndex>1</MinIndex>
<MaxIndex>3</MaxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>5MB</MaxFileSize>
</triggeringPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>WARN</level>
</filter>
</appender>
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>log/xpofacebook-application.log</File>
<Append>true</Append>
<BufferedIO>false</BufferedIO>
<ImmediateFlush>true</ImmediateFlush>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%date{ISO8601} [%-5level] %logger{35} - %msg%n</Pattern>
</layout>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<FileNamePattern>log/xpofacebook-application.log.%i.zip</FileNamePattern>
<MinIndex>1</MinIndex>
<MaxIndex>3</MaxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>5MB</MaxFileSize>
</triggeringPolicy>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="ERROR" />
<appender-ref ref="FILE" />
</root>
</configuration>
so far it works great
i need to add the following code to each class that i want to log messages in:
private static Logger log = Red5LoggerFactory.getLogger(ClassName.class, "xpofacebook");
i want the main log file xpofacebook-application.log to log messages outside of the classes that i work with.
If a flex client is connected to the red5 server and is trying to invoke a command that does not exist, the following error message will be show:
[NioProcessor-1] org.red5.server.service.ServiceInvoker - Method
getLiveChallenges with parameters [] not found in
com.xpogames.xpofacebook.Application#55e03a61
How can i make sure that these type of error messages will be included in my log file as well?
This will take a change in the core logging functionality of Red5. Right now the stuff outside your application is controlled by a different logger because they fall within a classloader above yours. Without getting into the fun of Java classloaders here, I will simply say that a fix or workaround will eventually come.