I have a spring boot application and using log4j2 to generate console and persists logs in a centos linux.
I wanted to maintain only 5mb of log files in archive.
But the problem is, my archived log files are 5mb in total. but my main console log which is saving in the main log file i.e wc-notification.out is going beyond 1mb.
so my disk gets full and it causes an issue.
The brute force method solution is:
whenever restarting(hard stop and start) my spring boot application, the log is cleared in from wc-notification.out.
my log4j2 configuration xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" monitorInterval="30">
<Properties>
<Property name="LOG_PATTERN">
[ %d{yyyy-MMM-dd HH:mm:ss a} ] - [%t] %-5level %logger{36} - %msg%n
</Property>
</Properties>
<Appenders>
<Console name="ConsoleAppender" target="SYSTEM_OUT" follow="true">
<PatternLayout pattern="${LOG_PATTERN}"/>
</Console>
<RollingFile name="FileAppender" fileName="/home/ec2-user/apps/wc-notification-service/wc-notification.out"
filePattern="/home/ec2-user/apps/wc-notification-service/archives_test/wc-notification.out-%d{yyyy-MM-dd}-%i">
<PatternLayout>
<Pattern>${LOG_PATTERN}</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="logs" maxDepth="1">
<IfFileName glob="wc-notification.out-*.log" />
<IfLastModified age="1m" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>
<Loggers>
<Root level="info">
<!--<AppenderRef ref="ConsoleAppender" /> -->
<AppenderRef ref="FileAppender" />
</Root>
</Loggers>
</Configuration>
somehow, the files are in range of 1mb, and the roll strategy is working, it is deleting the file
but, my disk space is still occupied with the space. what can be the reason?
As you are providing your own log4j2.xml configuration file, you are overwriting the Spring Boot default logging configuration, and it is safe to assume that it will be configuration used by Log4j2.
Your configuration is almost correct. If you want to achieve the desired behavior, I would suggest you the following changes:
Be aware that you are pointing your Delete action basePath to the wrong location, it should point to the directory in which your logs are stored.
The IfFileName glob pattern is wrong as well, it should match your logs filenames.
Finally, your are using the IfLastModified condition. As its name implies, this condition has to do with the last modification date of the logs files, not with their size. Please, consider to use instead IfAccumulatedFileSize (or maybe IfAccumulatedFileCount). See the relevant documentation.
For these reasons, your logs are not being deleted correctly and the disk space occupied is greater than the desired amount. Without deletion, your log files are being rotated every 1 MB as configured by your SizeBasedTriggeringPolicy, and will keep until the value of the max attribute of DefaultRolloverStrategy, 7 by default, is reached, and always, plus the amount of your current log file.
In summary, please, try a configuration like this:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" monitorInterval="30">
<Properties>
<Property name="LOG_PATTERN">
[ %d{yyyy-MMM-dd HH:mm:ss a} ] - [%t] %-5level %logger{36} - %msg%n
</Property>
</Properties>
<Appenders>
<Console name="ConsoleAppender" target="SYSTEM_OUT" follow="true">
<PatternLayout pattern="${LOG_PATTERN}"/>
</Console>
<RollingFile name="FileAppender" fileName="/home/ec2-user/apps/wc-notification-service/wc-notification.out"
filePattern="/home/ec2-user/apps/wc-notification-service/wc-notification.out-%d{yyyy-MM-dd}-%i">
<PatternLayout>
<Pattern>${LOG_PATTERN}</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="/home/ec2-user/apps/wc-notification-service">
<IfFileName glob="wc-notification.out-*" />
<IfAccumulatedFileSize exceeds="5 MB" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>
<Loggers>
<Root level="info">
<!--<AppenderRef ref="ConsoleAppender" /> -->
<AppenderRef ref="FileAppender" />
</Root>
</Loggers>
</Configuration>
The solution relies in storing all your logs in the same location. It will affect your RollingFile filePattern attribute.
Please, be careful with the delete action, it can delete not only your log files, but all that matches your glob pattern.
In addition, although maybe irrelevant fo your use case, be aware that if the filePattern of your archive files ends with ".gz", ".zip", ".bz2", among others, the resulting archive will be compressed using the compression scheme that matches the suffix, which can allow you to store more archives for the same space if required.
For your comments, it seems you encountered a problem when using large file sizes: please, see this bug, I think that clearly describes your problem.
My best advice will be to reduce the size of the logs files to one that it is properly working, or try a more recent version of the library.
I am aware that you are using Spring Boot to manage your dependencies:
please, verify your maven tree and see the actual version library you are using and change it if necessary.
You need to leverage the DeleteAction of the RollingFileAppender. I would recommend taking a look at the documentation as there are a lot of good examples.
The reason for your behavior is as follows: look at your parameters
appender.gateway.policies.size.size=1MB
appender.gateway.strategy.type = DefaultRolloverStrategy
appender.gateway.strategy.max = 5
What you are asking here is log file should be allowed to grow up to 1M. Once it reaches 1M size it is copied to file logfile.log-1 and a new file logfile.log is created. As the files get produced you required that you want to keep only 5 last files. The older files get deleted automatically. So it looks like the behavior is exactly as you configured. What you can do is to configure to keep more than 5 files back and possibly in a different folder where you have sufficient space.
Just for information to people who are facing similar issue, i just changing my logging mechanism to logback. it works like an charm and have no issue.
the reference link which i used: spring boot with logback
Related
java 1.8
maven 3.5
In pom.xml
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</dependency>
In my Maven java project I want to generate new log file every new day (midnight).
So I use RollingFile and cron 0 0 0 * * ?. Here my log4j2.xml
<Properties>
<Property name="baseDir">logs</Property>
<Property name="patterLayout">%d{[dd.MM.yyyy HH:mm:ss.SSS]} %l %p:%n %m%n</Property>
</Properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${patterLayout}"/>-
</Console>
<RollingFile name="RollingFile" fileName="${baseDir}/app.log"
filePattern="${baseDir}/$${date:yyyy-MM}/app-%d{yyyy-MM-dd}.log.gz">
<PatternLayout pattern="${patterLayout}"/>-
<CronTriggeringPolicy schedule="0 0 0 * * ?"/>
<DefaultRolloverStrategy>
<Delete basePath="${baseDir}" maxDepth="2">
<IfFileName glob="*/app-*.log.gz"/>
<IfLastModified age="60d"/>
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="Console"/>
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
But all logs print in ONE file app.log.
Here result:
[27.11.2021 23:59:05.235] com.myproject.xls2metro2.App.main(App.java:18) TRACE:
JDK: 1.8.0_202
[27.11.2021 23:59:05.238] com.myproject.xls2metro2.App.main(App.java:33) WARN:
File path is mandatory
[28.11.2021 00:01:13.347] com.myproject.xls2metro2.App.main(App.java:18) TRACE:
JDK: 1.8.0_202
[28.11.2021 00:01:13.351] com.myproject.xls2metro2.App.main(App.java:33) WARN:
File path is mandatory
At first, verify that your original configuration doesn't have this hyphen character (probably you have added it accidentally while writing a question)
<PatternLayout pattern="${patterLayout}"/>-
Also, make sure that your code is wrapped inside the Configuration tag (I assume that you have omitted it for the sake of brevity):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
... your code here
</Configuration>
At second, your config looks exactly the same as in the official documentation examples, so it should be fine.
But in the documentation stated that:
The CronTriggeringPolicy triggers rollover based on a cron expression.
This policy is controlled by a timer and is asynchronous to processing
log events, so it is possible that log events from the previous or
next time period may appear at the beginning or end of the log file.
The filePattern attribute of the Appender should contain a timestamp
otherwise the target file will be overwritten on each rollover.
So probably, this is your case. Just wait a little longer to verify, that the next logs will be written to a separate file.
I am new to log4j2. Previously I am using log4j. The reason I am migrating into part 2 is for Asynchronous logging. After searching Internet I am able to write a configuration file that actually creates two log files "Errors.log" and "Messages.log". Now the problem is : I would be communicating with Servers that are kept far away from me. I wrote a client that communicates with the server and sends a request and in back the Server sends me a response. In any situation it takes at least 10 milli-seconds for the request to reach the Server and get back the response from it. But in my log files it is showing that the request sent to the Server and receive from the Server is at same time (Same milli-second). I am using the Asynchronous logging. Is this causing the wrong timestamp? or else the policies which I have used are creating these issues?
Below is my Log4j2 XML CONFIG file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
<Appenders>
<File name="my_file_appender" fileName="LOG4j_LOGS/Errors.log" immediateFlush="false" append="false">
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</Pattern>
</PatternLayout>
</File>
<Async name="async_appender">
<AppenderRef ref="my_file_appender" />
</Async>
<!-- file appender -->
<RollingFile name="Error-log" fileName="LOG4j_LOGS/Messages.log"
filePattern="LOG4j_LOGS/Messages-%d{yyyy-MM-dd}.log">
<!-- log pattern -->
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</Pattern>
</PatternLayout>
<!-- set file size policy -->
<Policies>
<TimeBasedTriggeringPolicy />
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
<DefaultRolloverStrategy max="25"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="Error-log" level="info" additivity="false">
<appender-ref ref="Error-log" level="debug"/>
</Logger>
<Root level="info" includeLocation="false">
<AppenderRef ref="async_appender"/>
</Root>
</Loggers>
</Configuration>
Can anyone please check my CONFIG file. All I want is to create two separate log files, one for storing info messages and other for storing errors. And this should create a new file every time I run my application and it should not delete the previous logs. The size of the logs can be anything. If the size has exceeded it should create a new file and write the data into it. No matter how many days I run the application the daily logs needs to be stored and the entire process has to be done in Asynchronously.
I am also using the below VM options for logging asynchronously :
-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
When logging is done asynchronously you log messages got onto a separate queue. They are timestamped at the moment of processing by background thread that writes logs to disk (since timestamp is part of you Appender pattern). So only the order is preserved. Timestamps may be slightly different.
See https://logging.apache.org/log4j/2.x/manual/async.html for more info.
I am looking for a checklist of some kind, that would help me in solving the below problem:
I have a webapp running on Tomcat 8. I use Log4J2 v2.1 and Apache Commons Logging v1.2. My applications logs fine for a few days, then abruptly the logging stops. There aren't any exceptions being logged. Nothing!
I am unable to understand why this happens and what could be root cause. Also I don't know how to work towards finding the root cause.
BTW, the system has enough free disk space ~500GB and the tomcat logs, like localhost_access logs and others continue to log uninterrupted.
EDIT: As requested, posting the log4j2 configuration -
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%p [%t] %c{1}.%M(%L) | %m%n"/>
</Console>
<RollingFile name="RollingFile" fileName="../logs/app.log"
filePattern="../app/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout>
<Pattern>%d{ISO8601} [REQ_ID|%X{REQUEST_UUID}] %p [%t] %c{1}.%M(%L) | %m%n</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy/>
<TimeBasedTriggeringPolicy interval="6" modulate="true"/>
<SizeBasedTriggeringPolicy size="5 MB"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger ... />
...
<Root level="debug">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
The usual culprit here is that the file is renamed by a log rotation tool. If it happens after a few days, then the log rotation tool is configured to move the file away when it has reached a certain size. The tools then often compress the log file.
What happens is:
Java process opens log file. OS allocates a file descriptor.
After some time, the log rotation tool renames the file. Since Java writes to the file descriptor, it doesn't notice or care about the rename (the file descriptor stays the same).
Then the tool compresses the file. For this, a new file (the compressed one) is created.
When the log is compressed, the tool deletes the original log file.
Java continues to write to the file descriptor. Since the descriptor is no longer connected to a directory entry, you can't see it.
Solution: Configure log rotation in your Java logging tool or configure it to work together with the external log rotation tool.
1. I am trying to add log message at beginning of file so that I can see latest message first on log file.
2. I want to add month and year after log file but I am not getting that in current active file. eg - test_2015-04-22.log
property file is given below -
log4j.appender.APP=org.apache.log4j.DailyRollingFileAppender
log4j.appender.APP.File=${catalina.base}/logs/test.log
log4j.appender.APP.Append=true
log4j.appender.APP.Encoding=UTF-8
log4j.appender.APP.DatePattern='.'yyyy-MM
log4j.appender.APP.layout = org.apache.log4j.PatternLayout
log4j.appender.APP.layout.ConversionPattern =%d{yyyy-MM-dd HH:mm} - %m%n
log4j.appender.APP.filePattern =Test_%d{yyyy-MM-dd}.log
Your question shows a log4j-1.2 configuration, but since the question also has a log4j2 tag, I feel free to answer this by showing you how to accomplish this with log4j2.
In log4j2, you can declare a property that formats the date, then use this property to configure your appender file name.
Also, you can use the header attribute of the pattern layout to set a header that is output at the beginning of a file. For RollingFileAppender this header will be output on every rollover.
Use one of the lookups built-in to log4j2 to dynamically change the output of your header at rollover time.
Example:
<Configuration status="WARN"><!-- use TRACE to troubleshoot your config if needed-->
<Properties>
<property name="yyyyMMdd">${date:yyyyMMdd}</property>
</Properties>
<Appenders>
<RollingFile name="Application"
fileName="${sys:catalina.base}/logs/test${sys:yyyyMMdd}.log"
filePattern="logs/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout header="File: ${main:--file}">
<Pattern>%d{yyyy-MM-dd HH:mm} - %m%n</Pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy />
</Policies>
</RollingFile>
<Appenders>
<Loggers>
<root level="trace">
<AppenderRef ref="Application" />
</root>
</Loggers>
</Configuration>
it's not possible , think of the file as a stack which respects LAST In FIRST Out which makes perfect sense when debugging to see the last message at the end.
The log for today will have the name test.log but tomorrow it will be renamed to Test_2015-04-22.log
I'm using Log4J 2.0 to create logs for a project that I'm doing. The logs are small and I have a requirement to maintain them for 3 months. I'd like to have the current month's log with 3 archives (each containing a month's worth of logs).
The problem that I need help with is configuring log4j to rotate the logs at the beginning of the month (or the end of the month).
Pretty much every thing that I've found researching this problem is for log4j 1.x and talks about a datePattern parameter that doesn't appear to exist in 2.0.
Here's my log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="warn" name="NKMS" packages="">
<appenders>
<FastRollingFile name="LogFile" fileName="logs/Tier2HttpServer.log" filePattern="logs/app-%d{yyyy-MM-dd}.log.gz">
<ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d %p %c{1.} [%t] %m%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
<DefaultRolloverStrategy max="4"/>
</FastRollingFile>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>
</Console>
</appenders>
<loggers>
<logger name="mil.navy.nrl.itd.xml_filter" level="trace"/>
<root level="trace">
<appender-ref ref="STDOUT"/>
<appender-ref ref="LogFile"/>
</root>
</loggers>
</configuration>
I'm writing INFO and above to the log file and debug to the console (for now). The files are written to just fine, but they appear to rollover daily (which appears to be the default).
I've tried changing the FastRollingFile:filePattern to "'.'yyyy-MM" but that causes weird things to happen (only a single entry is written to file and an archive is immediately created).
I downloaded the source for log4j-2.0-beta8 and the PatternProcessor parses a RolloverFrequency that contains the enum RolloverFrequency.MONTHLY, but there again, I can't figure out how to implement / use it.
As always, any assistance or advice that you can provide would be GREATLY APPRECIATED!
-Ace
You may have found a bug. I would expect the filePattern of "logs/app-%d{yyyy-MM}.log.gz" to give you what you're looking for.
To clarify my understanding of the problem: The initial log event immediately triggers a rollover (creating an archive file). Instead, it should collect log events into the log file and not roll over until the end/beginning of the month. Is that description correct? Is there any other issue in addition to this initial unnecessary rollover?
Could I ask you to raise a JIRA ticket for this? https://issues.apache.org/jira/browse/LOG4J2