Develop Reference

How to configure log4j2 to rollover with time stamp on startup?

I have a Java application that runs for a few minutes each hour via a cron job (java -jar...). It runs in its own JVM, not in a continuously running environment like Tomcat. I am using Log4j 2.11 to do logging. Within the application, I have a particular logger with specific rollover requirements:
Log to "rolling.log"
At the end of each day (or on the first logging event of a new day), rolling.log should be rolled over to rolling-yyyy-MM-dd.log.gz and new logging events added to a fresh rolling.log.
I have not been able to get the rollover to work. All log messages are to "rolling.log" only and no rolling-yyyy-MM-dd.log.gz is ever created. To test this in the most simple way possible, I created a simple Java console application with the following two files. I would expect the log file to roll over on every execution as long as the system clock is showing a different minute, but this does not happen.
LoggingTest.java:
package log4jtest;
import java.time.Instant;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class LoggingTest {
private static final Logger logger = LogManager.getLogger();
public static void main(String[] args) {
System.out.println(Instant.now() + " - BEGIN: Logging to log4j");
logger.error("Test log message");
System.out.println(Instant.now() + " - DONE");
}
}
log4j2.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configuration>
<Configuration status="WARN">
<Appenders>
<RollingFile name="RollingLogFile"
fileName="logs/rolling.log"
filePattern="logs/rolling-%d{yyyy-MM-dd-HH-mm}.log.gz"
createOnDemand="true">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} %-5level - %msg%n" />
<Policies>
<TimeBasedTriggeringPolicy modulate="true"
interval="1" />
<OnStartupTriggeringPolicy />
</Policies>
<DefaultRolloverStrategy />
</RollingFile>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="RollingLogFile" />
</Root>
</Loggers>
</Configuration>
My guess is that since the JVM hosting the application and the Log4j instance is not running at the time the rollover would happen, then the rollover does not get triggered.
Ultimately, I abandoned the use of the RollingFileAppender and went with a straight-up FileAppender:
<File name="RollingLogFile"
fileName=logs/rolling-${date:yyyy-MM-dd}.log"
createOnDemand="true">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} %-5level - %msg%n" />
</File>
This works, but it has a few disadvantages:
I cannot monitor the simply named "rolling.log" (since it doesn't exist), but have to use the date-specific version of the filename.
I cannot make use of log4j's compression on rollover features.
I cannot make use of log4j's delete on rollover retention policies.
So, the question, restated, is: Using log4j, is it possible for an application executed in brief intervals to use a time-based log file rollover strategy in the same way that continuously running applications can?

Please, try the following configuration:
<RollingFile name="LogSpecial"
fileName="${sys:special.directory}/special.csv"
filePattern="${sys:special.directory}/special-%d{yyyy-MM-dd}.csv.gz"
createOnDemand="true">
<CsvParameterLayout />
<Policies>
<TimeBasedTriggeringPolicy modulate="true" interval="1" />
<OnStartupTriggeringPolicy />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="{sys:special.directory}">
<IfFileName glob="special-*.csv.gz" />
<IfLastModified age="60d" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
You can also switch on log4j2.debug system property to debug this configuration and investigate what's wrong.

It turns out that log4j2 is capable of doing exactly what I was aiming for but part of my configuration, for unknown reasons, prevented it from working properly. The solution was to remove the createOnDemand="true" attribute from the RollingFile element.
After excluding that attribute, the following configuration works exactly as I would expect (rolling over at startup, at most every minute):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configuration>
<Configuration status="WARN">
<Appenders>
<RollingFile name="RollingLogFile"
fileName="logs/rolling.log"
filePattern="logs/rolling-%d{yyyy-MM-dd-HH-mm}.log.gz">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} %-5level - %msg%n" />
<TimeBasedTriggeringPolicy />
</RollingFile>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="RollingLogFile" />
</Root>
</Loggers>
</Configuration>

log4j 2 RollingFile not rolling over

We have WAR deployments running on Tomcat containers which are using log4j 2.5 for logging events. We have now amended the deployments' log4j2.xml configuration to have the log files roll over every 24 hours but, with this new configuration, the rollover of files are not taking place as we would expect.
Sample configuration:
<RollingFile name="file"
fileName="${sys:catalina.base}/logs/${web:contextPath}.log"
filePattern="${sys:catalina.base}/logs/${web:contextPath}-%d{dd-MMM-yyyy}.log"
append="true">
<PatternLayout pattern="%d{dd-MMM-yyyy HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" >
<header>LOG START DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</header>
<footer>LOG END DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</footer>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
Any ideas why the rollover is not taking place?
NOTE: The same configuration but with a <CronTriggeringPolicy schedule="0 0 0 * * ?" /> instead of TimeBasedTriggeringPolicy does rollover but, in this case, the rolled over files get created with today's date in the filename and NOT yesterday's date.
NOTE2: We have other deployments with similar configuration that do rollover every 24 hours but those configurations have the filename hardcoded instead of using ${web:contextPath}. Could this lookup have something to do with why RollingFile might not work?
--- EDIT ---
UPDATE: We are able to get TimeBasedTriggeringPolicy to rollover files using above configuration when the Tomcat instance is running on Windows but NOT when the Tomcat instance is running on Linux.

There is nothing wrong with your configuration snippet as I get the desired behaviour of time based rolling.
To test, I changed the dd-MMM-yyyy to dd-MMM-yyyy-HH-mm and my log file rolls every minute.
It must be something else that is preventing you from achieving the desired behaviour.
My setup #1:
Log4j2 v2.8.2
Apache Tomcat 8.5.13
Windows 7 Enterprise SP1
My setup #2:
Log4j2 v2.5
Apache Tomcat 7.0.77
CentOS 7 (1611)
I have the following 3 JARs in WEB-INF/lib:
log4j-api-2.5.jar
log4j-core-2.5.jar
log4j-web-2.5.jar
Here is my complete log4j2.xml for your reference:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
<Appenders>
<RollingFile name="RollingFileAppender"
fileName="${sys:catalina.base}/logs/${web:contextPath}.log"
filePattern="${sys:catalina.base}/logs/${web:contextPath}-%d{dd-MMM-yyyy-HH-mm}.log"
append="true">
<PatternLayout pattern="%d{dd-MMM-yyyy HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" >
<header>LOG START DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</header>
<footer>LOG END DATE=${date:dd-MMM-yyyy HH:mm:ss.SSS} APP=${web:contextPath} TOMCAT=${env:HOSTNAME}:${env:CONNECTOR_PORT}${sys:line.separator}</footer>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="root" level="debug" additivity="false">
<appender-ref ref="RollingFileAppender" level="debug"/>
</Logger>
<Root level="debug" additivity="false">
<AppenderRef ref="RollingFileAppender"/>
</Root>
</Loggers>
</Configuration>

How to identify the cause of logs not getting logged?

I am looking for a checklist of some kind, that would help me in solving the below problem:
I have a webapp running on Tomcat 8. I use Log4J2 v2.1 and Apache Commons Logging v1.2. My applications logs fine for a few days, then abruptly the logging stops. There aren't any exceptions being logged. Nothing!
I am unable to understand why this happens and what could be root cause. Also I don't know how to work towards finding the root cause.
BTW, the system has enough free disk space ~500GB and the tomcat logs, like localhost_access logs and others continue to log uninterrupted.
EDIT: As requested, posting the log4j2 configuration -
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%p [%t] %c{1}.%M(%L) | %m%n"/>
</Console>
<RollingFile name="RollingFile" fileName="../logs/app.log"
filePattern="../app/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout>
<Pattern>%d{ISO8601} [REQ_ID|%X{REQUEST_UUID}] %p [%t] %c{1}.%M(%L) | %m%n</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy/>
<TimeBasedTriggeringPolicy interval="6" modulate="true"/>
<SizeBasedTriggeringPolicy size="5 MB"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger ... />
...
<Root level="debug">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>

The usual culprit here is that the file is renamed by a log rotation tool. If it happens after a few days, then the log rotation tool is configured to move the file away when it has reached a certain size. The tools then often compress the log file.
What happens is:
Java process opens log file. OS allocates a file descriptor.
After some time, the log rotation tool renames the file. Since Java writes to the file descriptor, it doesn't notice or care about the rename (the file descriptor stays the same).
Then the tool compresses the file. For this, a new file (the compressed one) is created.
When the log is compressed, the tool deletes the original log file.
Java continues to write to the file descriptor. Since the descriptor is no longer connected to a directory entry, you can't see it.
Solution: Configure log rotation in your Java logging tool or configure it to work together with the external log rotation tool.

How can I specify a specific format for each Log4j log level

First time trying to use log4j version log4j-1.2.17.jar.
On an existing application the client has log4j in place and there is a log4j.properties file which specifies a light log output. What I want to do is depending on the log level (ERROR & WARN) output a more refined entry.
On the log4j site I came across this but I think it is to be in some .xml file. I need some assistance in understanding how I can put in place the formatting option to alter based on log level.
You don't need to declare separate loggers to achieve this. You can set the logging level on the AppenderRef element.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<File name="file" fileName="app.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m %ex%n</Pattern>
</PatternLayout>
</File>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="file" level="DEBUG"/>
<AppenderRef ref="STDOUT" level="INFO"/>
</Root>
</Loggers>
</Configuration>
Would I put this xml content into the web.xml file or another file?
a) If another file what file name and where would it go?
How do I get log4j to realize that I need it to use the xml file?
Will the use of the xml ignore the log4j.properties file?
I know it is a lot of questions but there is only me on the project and the client has a production crisis that needs to be figured out today so I don't have time to go off to read and do tutorials with the client calling me every hour. I figured it may help to get this logging more useful. As the logs are right now I have a date and message output to the log but no idea where the entries are created from without doing extensive searches through the code.

You could do this by defining multiple non-additive Loggers so that the first one only logs errors, the next one warnings and the final one infos and debug.

How to configure log4j to roll by time,size and on start?

I need to configure logger to roll the logs by time (hourly or daily), by size and on start. I searched a lot and red log4j docs, now i am confused.
Looks like it can be done by following ways:
using Simon library
creating custom classes
using log4j 2 beta
I can't add new 3rd party libs to my project (log4j 2 beta is already added) so i consider the last two options.
What exactly classes should i create if i go with the 2nd option? Should it be appender, rollingPolicy or triggeringPolicy?
Will log4j2 really support that?
Thanks for help, Yuri

I don't think you need to create any classes, you should be able to achieve what you want with configuration only. The Log4J2 documentation provides a number of examples.
Here is a good place to start: http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppender
If this is not sufficient, don't hesitate to ask a question on the log4j-2 user mailing list.
Config example:
<?xml version="1.0" encoding="UTF-8"?><configuration name="install" status="info">
<appenders>
<!-- ################# InstallAppender ############################### -->
<RollingFile name="InstallAppender"
fileName="${sys:installation.path}/installation/logs/post_install.log"
filePattern="${sys:installation.path}/installation/logs/post_install.log">
<PatternLayout>
<pattern>%d{dd/MM/yyyy HH:mm:ss} %-5p [%t] [%c{1}] %m%n</pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy/>
</Policies>
<DefaultRolloverStrategy max="1"/>
</RollingFile>
</appenders>
<loggers>
<root level="info">
<appender-ref ref="InstallAppender"/>
</root>
</loggers>