I have a spring boot application and using log4j2 to generate console and persists logs in a centos linux.
I wanted to maintain only 5mb of log files in archive.
But the problem is, my archived log files are 5mb in total. but my main console log which is saving in the main log file i.e wc-notification.out is going beyond 1mb.
so my disk gets full and it causes an issue.
The brute force method solution is:
whenever restarting(hard stop and start) my spring boot application, the log is cleared in from wc-notification.out.
my log4j2 configuration xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" monitorInterval="30">
<Properties>
<Property name="LOG_PATTERN">
[ %d{yyyy-MMM-dd HH:mm:ss a} ] - [%t] %-5level %logger{36} - %msg%n
</Property>
</Properties>
<Appenders>
<Console name="ConsoleAppender" target="SYSTEM_OUT" follow="true">
<PatternLayout pattern="${LOG_PATTERN}"/>
</Console>
<RollingFile name="FileAppender" fileName="/home/ec2-user/apps/wc-notification-service/wc-notification.out"
filePattern="/home/ec2-user/apps/wc-notification-service/archives_test/wc-notification.out-%d{yyyy-MM-dd}-%i">
<PatternLayout>
<Pattern>${LOG_PATTERN}</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="logs" maxDepth="1">
<IfFileName glob="wc-notification.out-*.log" />
<IfLastModified age="1m" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>
<Loggers>
<Root level="info">
<!--<AppenderRef ref="ConsoleAppender" /> -->
<AppenderRef ref="FileAppender" />
</Root>
</Loggers>
</Configuration>
somehow, the files are in range of 1mb, and the roll strategy is working, it is deleting the file
but, my disk space is still occupied with the space. what can be the reason?
As you are providing your own log4j2.xml configuration file, you are overwriting the Spring Boot default logging configuration, and it is safe to assume that it will be configuration used by Log4j2.
Your configuration is almost correct. If you want to achieve the desired behavior, I would suggest you the following changes:
Be aware that you are pointing your Delete action basePath to the wrong location, it should point to the directory in which your logs are stored.
The IfFileName glob pattern is wrong as well, it should match your logs filenames.
Finally, your are using the IfLastModified condition. As its name implies, this condition has to do with the last modification date of the logs files, not with their size. Please, consider to use instead IfAccumulatedFileSize (or maybe IfAccumulatedFileCount). See the relevant documentation.
For these reasons, your logs are not being deleted correctly and the disk space occupied is greater than the desired amount. Without deletion, your log files are being rotated every 1 MB as configured by your SizeBasedTriggeringPolicy, and will keep until the value of the max attribute of DefaultRolloverStrategy, 7 by default, is reached, and always, plus the amount of your current log file.
In summary, please, try a configuration like this:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" monitorInterval="30">
<Properties>
<Property name="LOG_PATTERN">
[ %d{yyyy-MMM-dd HH:mm:ss a} ] - [%t] %-5level %logger{36} - %msg%n
</Property>
</Properties>
<Appenders>
<Console name="ConsoleAppender" target="SYSTEM_OUT" follow="true">
<PatternLayout pattern="${LOG_PATTERN}"/>
</Console>
<RollingFile name="FileAppender" fileName="/home/ec2-user/apps/wc-notification-service/wc-notification.out"
filePattern="/home/ec2-user/apps/wc-notification-service/wc-notification.out-%d{yyyy-MM-dd}-%i">
<PatternLayout>
<Pattern>${LOG_PATTERN}</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="/home/ec2-user/apps/wc-notification-service">
<IfFileName glob="wc-notification.out-*" />
<IfAccumulatedFileSize exceeds="5 MB" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
</Appenders>
<Loggers>
<Root level="info">
<!--<AppenderRef ref="ConsoleAppender" /> -->
<AppenderRef ref="FileAppender" />
</Root>
</Loggers>
</Configuration>
The solution relies in storing all your logs in the same location. It will affect your RollingFile filePattern attribute.
Please, be careful with the delete action, it can delete not only your log files, but all that matches your glob pattern.
In addition, although maybe irrelevant fo your use case, be aware that if the filePattern of your archive files ends with ".gz", ".zip", ".bz2", among others, the resulting archive will be compressed using the compression scheme that matches the suffix, which can allow you to store more archives for the same space if required.
For your comments, it seems you encountered a problem when using large file sizes: please, see this bug, I think that clearly describes your problem.
My best advice will be to reduce the size of the logs files to one that it is properly working, or try a more recent version of the library.
I am aware that you are using Spring Boot to manage your dependencies:
please, verify your maven tree and see the actual version library you are using and change it if necessary.
You need to leverage the DeleteAction of the RollingFileAppender. I would recommend taking a look at the documentation as there are a lot of good examples.
The reason for your behavior is as follows: look at your parameters
appender.gateway.policies.size.size=1MB
appender.gateway.strategy.type = DefaultRolloverStrategy
appender.gateway.strategy.max = 5
What you are asking here is log file should be allowed to grow up to 1M. Once it reaches 1M size it is copied to file logfile.log-1 and a new file logfile.log is created. As the files get produced you required that you want to keep only 5 last files. The older files get deleted automatically. So it looks like the behavior is exactly as you configured. What you can do is to configure to keep more than 5 files back and possibly in a different folder where you have sufficient space.
Just for information to people who are facing similar issue, i just changing my logging mechanism to logback. it works like an charm and have no issue.
the reference link which i used: spring boot with logback
I am new to log4j2. Previously I am using log4j. The reason I am migrating into part 2 is for Asynchronous logging. After searching Internet I am able to write a configuration file that actually creates two log files "Errors.log" and "Messages.log". Now the problem is : I would be communicating with Servers that are kept far away from me. I wrote a client that communicates with the server and sends a request and in back the Server sends me a response. In any situation it takes at least 10 milli-seconds for the request to reach the Server and get back the response from it. But in my log files it is showing that the request sent to the Server and receive from the Server is at same time (Same milli-second). I am using the Asynchronous logging. Is this causing the wrong timestamp? or else the policies which I have used are creating these issues?
Below is my Log4j2 XML CONFIG file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
<Appenders>
<File name="my_file_appender" fileName="LOG4j_LOGS/Errors.log" immediateFlush="false" append="false">
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</Pattern>
</PatternLayout>
</File>
<Async name="async_appender">
<AppenderRef ref="my_file_appender" />
</Async>
<!-- file appender -->
<RollingFile name="Error-log" fileName="LOG4j_LOGS/Messages.log"
filePattern="LOG4j_LOGS/Messages-%d{yyyy-MM-dd}.log">
<!-- log pattern -->
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</Pattern>
</PatternLayout>
<!-- set file size policy -->
<Policies>
<TimeBasedTriggeringPolicy />
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
<DefaultRolloverStrategy max="25"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="Error-log" level="info" additivity="false">
<appender-ref ref="Error-log" level="debug"/>
</Logger>
<Root level="info" includeLocation="false">
<AppenderRef ref="async_appender"/>
</Root>
</Loggers>
</Configuration>
Can anyone please check my CONFIG file. All I want is to create two separate log files, one for storing info messages and other for storing errors. And this should create a new file every time I run my application and it should not delete the previous logs. The size of the logs can be anything. If the size has exceeded it should create a new file and write the data into it. No matter how many days I run the application the daily logs needs to be stored and the entire process has to be done in Asynchronously.
I am also using the below VM options for logging asynchronously :
-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
When logging is done asynchronously you log messages got onto a separate queue. They are timestamped at the moment of processing by background thread that writes logs to disk (since timestamp is part of you Appender pattern). So only the order is preserved. Timestamps may be slightly different.
See https://logging.apache.org/log4j/2.x/manual/async.html for more info.
First time trying to use log4j version log4j-1.2.17.jar.
On an existing application the client has log4j in place and there is a log4j.properties file which specifies a light log output. What I want to do is depending on the log level (ERROR & WARN) output a more refined entry.
On the log4j site I came across this but I think it is to be in some .xml file. I need some assistance in understanding how I can put in place the formatting option to alter based on log level.
You don't need to declare separate loggers to achieve this. You can set the logging level on the AppenderRef element.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<File name="file" fileName="app.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m %ex%n</Pattern>
</PatternLayout>
</File>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="file" level="DEBUG"/>
<AppenderRef ref="STDOUT" level="INFO"/>
</Root>
</Loggers>
</Configuration>
Would I put this xml content into the web.xml file or another file?
a) If another file what file name and where would it go?
How do I get log4j to realize that I need it to use the xml file?
Will the use of the xml ignore the log4j.properties file?
I know it is a lot of questions but there is only me on the project and the client has a production crisis that needs to be figured out today so I don't have time to go off to read and do tutorials with the client calling me every hour. I figured it may help to get this logging more useful. As the logs are right now I have a date and message output to the log but no idea where the entries are created from without doing extensive searches through the code.
You could do this by defining multiple non-additive Loggers so that the first one only logs errors, the next one warnings and the final one infos and debug.
I have a JAX-RS 2.0 application running on a Tomcat 7 server, and I'm using log4j2 along with SLF4J to record the server logs to a file.
I can't seem to get any logs to show up properly in my log file when running the server in production, although when I run my integration tests, logs are output correctly.
In production, the logs are merely redirected to the console instead.
My log4j2.xml file is located in the WEB-INF/classes folder, and I've included all the necessary dependencies as well.
My configuration file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="trace">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
<RollingFile name="file" fileName="log/trace.log" append="true" filePattern="log/trace.%i.log">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %X %logger{36} - %msg%n"/>
<Policies>
<OnStartupTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="my.package" level="TRACE" additivity="false">
<AppenderRef ref="file"/>
</Logger>
<Root level="WARN">
<AppenderRef ref="file"/>
</Root>
</Loggers>
</Configuration>
The web.xml needs no configuration (I'm following the documentation found on the log4j2 website).
EDIT
I've tried setting the Root level to TRACE but everything still gets redirected to console. The file log/trace.log itself is created, it's just never written to. I also tried setting immediateFlush=true but that didn't have any impact either.
I noticed you have status logging (log4j internal logging) set to TRACE. This will dump log4j internal initialization messages to the console. It this what you mean?
Otherwise, the config you provide shows there is no logger that has an appender-ref pointing to the ConsoleAppender.
So, if you are also seeing your application logs being output to the console (in addition to log4j's status log messages), I suspect there is another log4j2.xml (or log4j2-test.xml) config file in the classpath somewhere.
Fortunately log4j's status logs should also show the location of the log4j config file, so you can confirm which config file is actually being loaded.
You can switch off status logging by setting <Configuration status="WARN"> after confirming all works correctly.
I figured it out!
Turns out I was using the gretty plug-in with gradle, which contains it's own logging package (the logback library).
It used it's own internal logback.xml which was redirecting to console. I fixed this by overwriting the internal logback.xml with my own (using logback's configuration) and now everything works as expected!
I'm using Log4J 2.0 to create logs for a project that I'm doing. The logs are small and I have a requirement to maintain them for 3 months. I'd like to have the current month's log with 3 archives (each containing a month's worth of logs).
The problem that I need help with is configuring log4j to rotate the logs at the beginning of the month (or the end of the month).
Pretty much every thing that I've found researching this problem is for log4j 1.x and talks about a datePattern parameter that doesn't appear to exist in 2.0.
Here's my log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="warn" name="NKMS" packages="">
<appenders>
<FastRollingFile name="LogFile" fileName="logs/Tier2HttpServer.log" filePattern="logs/app-%d{yyyy-MM-dd}.log.gz">
<ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d %p %c{1.} [%t] %m%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
<DefaultRolloverStrategy max="4"/>
</FastRollingFile>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>
</Console>
</appenders>
<loggers>
<logger name="mil.navy.nrl.itd.xml_filter" level="trace"/>
<root level="trace">
<appender-ref ref="STDOUT"/>
<appender-ref ref="LogFile"/>
</root>
</loggers>
</configuration>
I'm writing INFO and above to the log file and debug to the console (for now). The files are written to just fine, but they appear to rollover daily (which appears to be the default).
I've tried changing the FastRollingFile:filePattern to "'.'yyyy-MM" but that causes weird things to happen (only a single entry is written to file and an archive is immediately created).
I downloaded the source for log4j-2.0-beta8 and the PatternProcessor parses a RolloverFrequency that contains the enum RolloverFrequency.MONTHLY, but there again, I can't figure out how to implement / use it.
As always, any assistance or advice that you can provide would be GREATLY APPRECIATED!
-Ace
You may have found a bug. I would expect the filePattern of "logs/app-%d{yyyy-MM}.log.gz" to give you what you're looking for.
To clarify my understanding of the problem: The initial log event immediately triggers a rollover (creating an archive file). Instead, it should collect log events into the log file and not roll over until the end/beginning of the month. Is that description correct? Is there any other issue in addition to this initial unnecessary rollover?
Could I ask you to raise a JIRA ticket for this? https://issues.apache.org/jira/browse/LOG4J2