Develop Reference

How to configure log4j2 to rollover with time stamp on startup?

I have a Java application that runs for a few minutes each hour via a cron job (java -jar...). It runs in its own JVM, not in a continuously running environment like Tomcat. I am using Log4j 2.11 to do logging. Within the application, I have a particular logger with specific rollover requirements:
Log to "rolling.log"
At the end of each day (or on the first logging event of a new day), rolling.log should be rolled over to rolling-yyyy-MM-dd.log.gz and new logging events added to a fresh rolling.log.
I have not been able to get the rollover to work. All log messages are to "rolling.log" only and no rolling-yyyy-MM-dd.log.gz is ever created. To test this in the most simple way possible, I created a simple Java console application with the following two files. I would expect the log file to roll over on every execution as long as the system clock is showing a different minute, but this does not happen.
LoggingTest.java:
package log4jtest;
import java.time.Instant;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class LoggingTest {
private static final Logger logger = LogManager.getLogger();
public static void main(String[] args) {
System.out.println(Instant.now() + " - BEGIN: Logging to log4j");
logger.error("Test log message");
System.out.println(Instant.now() + " - DONE");
}
}
log4j2.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configuration>
<Configuration status="WARN">
<Appenders>
<RollingFile name="RollingLogFile"
fileName="logs/rolling.log"
filePattern="logs/rolling-%d{yyyy-MM-dd-HH-mm}.log.gz"
createOnDemand="true">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} %-5level - %msg%n" />
<Policies>
<TimeBasedTriggeringPolicy modulate="true"
interval="1" />
<OnStartupTriggeringPolicy />
</Policies>
<DefaultRolloverStrategy />
</RollingFile>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="RollingLogFile" />
</Root>
</Loggers>
</Configuration>
My guess is that since the JVM hosting the application and the Log4j instance is not running at the time the rollover would happen, then the rollover does not get triggered.
Ultimately, I abandoned the use of the RollingFileAppender and went with a straight-up FileAppender:
<File name="RollingLogFile"
fileName=logs/rolling-${date:yyyy-MM-dd}.log"
createOnDemand="true">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} %-5level - %msg%n" />
</File>
This works, but it has a few disadvantages:
I cannot monitor the simply named "rolling.log" (since it doesn't exist), but have to use the date-specific version of the filename.
I cannot make use of log4j's compression on rollover features.
I cannot make use of log4j's delete on rollover retention policies.
So, the question, restated, is: Using log4j, is it possible for an application executed in brief intervals to use a time-based log file rollover strategy in the same way that continuously running applications can?

Please, try the following configuration:
<RollingFile name="LogSpecial"
fileName="${sys:special.directory}/special.csv"
filePattern="${sys:special.directory}/special-%d{yyyy-MM-dd}.csv.gz"
createOnDemand="true">
<CsvParameterLayout />
<Policies>
<TimeBasedTriggeringPolicy modulate="true" interval="1" />
<OnStartupTriggeringPolicy />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="{sys:special.directory}">
<IfFileName glob="special-*.csv.gz" />
<IfLastModified age="60d" />
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
You can also switch on log4j2.debug system property to debug this configuration and investigate what's wrong.

It turns out that log4j2 is capable of doing exactly what I was aiming for but part of my configuration, for unknown reasons, prevented it from working properly. The solution was to remove the createOnDemand="true" attribute from the RollingFile element.
After excluding that attribute, the following configuration works exactly as I would expect (rolling over at startup, at most every minute):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configuration>
<Configuration status="WARN">
<Appenders>
<RollingFile name="RollingLogFile"
fileName="logs/rolling.log"
filePattern="logs/rolling-%d{yyyy-MM-dd-HH-mm}.log.gz">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} %-5level - %msg%n" />
<TimeBasedTriggeringPolicy />
</RollingFile>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="RollingLogFile" />
</Root>
</Loggers>
</Configuration>

Can I have Below Log files structure using log4j?

Expectation is to have the current log with the same name (i.e test.log)
and the archival files name should be test_(CurrentDate).log.1.
/logs/projectlogs/test.log
/logs/projectlogs/ test_20160430.log.1
/logs/projectlogs/ test_20160430.log.2
/logs/projectlogs/ test_20160430.log.3
Using below properties file, current date is getting appended with all the files.
log4j.properties::
log4j.rootLogger= ALL, A1, file, rollingAppender
#log4j.date=contains current date
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=C:/logs/projectlogs/test_${log4j.date}.log
log4j.appender.file.MaxFileSize=100KB
log4j.appender.file.MaxBackupIndex=5
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss:SSS zzz} %-5p[%t] %m%n

The log4j 2 manual has many example configurations, and the section on RollingFileAppender has an example that matches your requirements:
http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppender
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
<Appenders>
<RollingFile name="RollingFile" fileName="logs/projectlogs/test.log"
filePattern="logs/projectlogs/$${date:yyyy-MM}/test-%d{yyyy-MM-dd-HH}-%i.log.gz">
<PatternLayout>
<Pattern>%d %-5p[%t] %c{1.} %m%n</Pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval="5" modulate="true"/>
<SizeBasedTriggeringPolicy size="100 MB"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
What may also be of interest to you is that Log4j-2.5 introduced a Delete action that gives users more control over what files are deleted at rollover time. See http://logging.apache.org/log4j/2.x/manual/appenders.html#CustomDeleteOnRollover

What you want to achieve is called "Size And Time Based File Naming And Triggering Policy" (SizeAndTimeBasedFNATP) and can be found in the latest release of logback package, which is at the moment 1.1.7, however, it can be found in 1.1.2 as well, which I use since it produces more predictable results without too much of asynchronous processes (even though 1.1.2 is slower).
The logback package works in similar way as log4j, but uses XML based configuration file (logback.xml) which in your case may look like one below (note test_%d{yyyyMMdd}.log.%i -- this is your pattern).
Please note, that unfortunately, it is not (yet) possible to limit indexes within dates. So you can't have only 5 chunks of logs for each date -- each date will be logged entirely with log chunks indexes independent for each date. You can, however, limit total number of dates and (in 1.1.7) total size of log folder (using ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy).
Also I suggest you use chunk size of 10Mb at least (100Kb is tiny), as bigger chunks, generally speaking, easier to maintain by log lib (less CPU consumption).
<?xml version="1.0" ?>
<configuration>
<property name="log.folder" value="C:/logs/projectlogs"/>
<!-- UNCOMMENT BELOW SECTION IF CONSOLE IS REQUIRED -->
<!--
<appender class="ch.qos.logback.core.ConsoleAppender" name="CONSOLE">
<encoder>
<pattern>[%p] [%thread] %logger - %msg%n</pattern>
</encoder>
</appender>
-->
<appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="FILE">
<File>${log.folder}/test.log</File>
<Append>true</Append>
<encoder>
<pattern>%d{ISO8601} [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.folder}/test_%d{yyyyMMdd}.log.%i</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>100KB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- keep 30 days' worth of history -->
<maxHistory>30</maxHistory>
<!-- up to 10 GB max -->
<totalSizeCap>10GB</totalSizeCap>
<MaxFileSize>100KB</MaxFileSize>
</rollingPolicy>
</appender>
<root>
<level value="INFO"/>
<appender-ref ref="FILE"/>
</root>
<logger name="Main">
<level value="DEBUG" />
</logger>
<logger name="ch.qos">
<level value="WARN"/>
</logger>
</configuration>
Test Java App
package logtester;
import org.apache.log4j.Logger;
public class LogTester {
public static void main(String[] args) {
Logger logger = Logger.getLogger("Main");
for(int i = 1; i<=20000; i++)
logger.info("Log message "+i);
}
}
Folder structure after run:
13,230 test.log
102,929 test_20160430.log.0
103,168 test_20160430.log.1
102,816 test_20160430.log.10
102,816 test_20160430.log.11
103,168 test_20160430.log.2
103,168 test_20160430.log.3
103,168 test_20160430.log.4
103,168 test_20160430.log.5
102,815 test_20160430.log.6
102,816 test_20160430.log.7
102,816 test_20160430.log.8
102,816 test_20160430.log.9
Hope it helps.

Log4j2: file logging not working with jar

I have a java application with log4j2 configured in order to log to console and to file.
When i start my application from ecplise, all the log are ok (console and file).
When i start the jar of the application from CMD or Powershell (as admin), only the console log works.
If i start the jar by only double click, the file log works (but no console is displayed).
This my log4j2.xml configuration:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<File name="file_all" fileName="logs/ALL.log">
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss} [%t] %-5level %logger{36} - %msg%n</Pattern>
</PatternLayout>
</File>
<File name="file_error" fileName="logs/ERROR.log">
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss} [%t] %-5level %logger{36} - %msg%n</Pattern>
</PatternLayout>
</File>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="file_all" level="INFO"/>
<AppenderRef ref="file_error" level="ERROR"/>
<AppenderRef ref="STDOUT" level="INFO"/>
</Root>
</Loggers>
</Configuration>

Your log4j2.xml config file starts with <configuration status="off" ... (or perhaps WARN or ERROR, you haven't shown the full config).
To investigate, switch on log4j's internal status logging by changing this to <configuration status="trace" ...
This will show output on the console. (You can redirect this to a file with java -jar myjar.jar > out.log.)
I hope this will give us more info on what is going on. Please paste that output into your question.
EDIT: the status output looks good; no errors or anything. Could it be that double-clicking a jar file is associated with javaw (not java), so there is no console? You mention that when starting the application from a CMD prompt or Powershell, you cannot find the log file. The line "Starting FileManager logs/ALL.log" in the status log tells me that log4j2 successfully created an output stream. However, this is a relative path, and the status log does not mention the absolute path. I think it is relative to the current directory, so what is the current directory of the CMD prompt or Powershell?

log4j2 to chainsaw not working - nothing shows up in the 'Zeroconf' tab of Chainsaw

I am trying to to use Chainsaw to view my application's logger events but there is nothing showing up under the 'Zeroconf' tab in chansaw.
I've followed Scott's guide in log4j2 to chainsaw hello world not working… what am I doing wrong? - but no luck. I was going to comment on that question asking how teryet got it working in the end, but as my reputation is below 50, the site didn't allow me.
Environment
OS: OSX Mavericks
IDE: Netbeans 8.0 (Build 201403101706)
Java: 1.7.0_45; Java HotSpot(TM) 64-Bit Server VM 24.45-b08
log4j: 2.0rc1
Chainsaw: downloaded the latest DMG from http://people.apache.org/~sdeboy/
Things I've made sure
- included jmdns.jar in the classpath of my application
- Used PatternLayout in my config log4j.xml
- Ensure advertiser URL has three slashes
My log4j.xml config file
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" advertiser="multicastdns">
<appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%date{ABSOLUTE} [%thread] %logger{3}.%style{%method}{Blue}%style{(line%line)}{Red}%X %highlight{%-5level} - %msg%n%xEx"/>
</Console>
<RollingFile name="RollingFile" fileName="../logs/POS.log" filePattern="../logs/$${date:yyyy-MM}/POS-%d{yyyyMMdd-HHmmss}.log">
<PatternLayout pattern="%date{ABSOLUTE} [%thread] %logger{3}.%style{%method}{Blue}%style{(line%line)}{Red}%X %highlight{%-5level} - %msg%n%xEx"/>
<Policies>
<OnStartupTriggeringPolicy/>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
<File name="testFile" fileName="../logs/POS2.log" bufferedIO="false" advertiseURI="file:///localhost/Users/arthurhsieh/Documents/NetBeansProjects/AES/logs/POS2.log" advertise="true">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %m%n"/>
</File>
</appenders>
<loggers>
<root level="all"> <!-- <root level="trace"> -->
<appender-ref ref="Console"/>
<appender-ref ref="RollingFile"/>
<appender-ref ref="testFile" />
</root>
</loggers>
</configuration>
I can see logger events in the POS2.log file though.
Thanks in advance for any help/guidances. Cheers.

My issue went away after I restarted my system, i.e., Chainsaw is working and i can view my logs by connecting through the Zeroconf tab.
My guess is this is an Apple OS issue rather than Chainsaw itself.

log4j2 monthly log rollover

I'm using Log4J 2.0 to create logs for a project that I'm doing. The logs are small and I have a requirement to maintain them for 3 months. I'd like to have the current month's log with 3 archives (each containing a month's worth of logs).
The problem that I need help with is configuring log4j to rotate the logs at the beginning of the month (or the end of the month).
Pretty much every thing that I've found researching this problem is for log4j 1.x and talks about a datePattern parameter that doesn't appear to exist in 2.0.
Here's my log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="warn" name="NKMS" packages="">
<appenders>
<FastRollingFile name="LogFile" fileName="logs/Tier2HttpServer.log" filePattern="logs/app-%d{yyyy-MM-dd}.log.gz">
<ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d %p %c{1.} [%t] %m%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
<DefaultRolloverStrategy max="4"/>
</FastRollingFile>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>
</Console>
</appenders>
<loggers>
<logger name="mil.navy.nrl.itd.xml_filter" level="trace"/>
<root level="trace">
<appender-ref ref="STDOUT"/>
<appender-ref ref="LogFile"/>
</root>
</loggers>
</configuration>
I'm writing INFO and above to the log file and debug to the console (for now). The files are written to just fine, but they appear to rollover daily (which appears to be the default).
I've tried changing the FastRollingFile:filePattern to "'.'yyyy-MM" but that causes weird things to happen (only a single entry is written to file and an archive is immediately created).
I downloaded the source for log4j-2.0-beta8 and the PatternProcessor parses a RolloverFrequency that contains the enum RolloverFrequency.MONTHLY, but there again, I can't figure out how to implement / use it.
As always, any assistance or advice that you can provide would be GREATLY APPRECIATED!
-Ace

You may have found a bug. I would expect the filePattern of "logs/app-%d{yyyy-MM}.log.gz" to give you what you're looking for.
To clarify my understanding of the problem: The initial log event immediately triggers a rollover (creating an archive file). Instead, it should collect log events into the log file and not roll over until the end/beginning of the month. Is that description correct? Is there any other issue in addition to this initial unnecessary rollover?
Could I ask you to raise a JIRA ticket for this? https://issues.apache.org/jira/browse/LOG4J2