Develop Reference

log4j gets stuck at a maximum number of 13 files even though the configured limit is 50?

This is the log4j properties file we are using, could you please explain why the limit for number of files is stuck at 13 instead of 50 as defined in "maxIndex" Property?
# Root logger option
log4j.rootLogger=INFO,file,stdout
log4j.logger.com.cisco.assurance=DEBUG, file
# Redirect log messages to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.Threshold=DEBUG
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SZ} %-5p %m%n
# Redirect log messages to a log file
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.rollingPolicy.FileNamePattern=/var/log/smeu/AssuranceMS-%i.log.gz
log4j.appender.file.rollingPolicy.ActiveFileName=/var/log/smeu/AssuranceMS.log
log4j.appender.file.triggeringPolicy.MaxFileSize=204800000
log4j.appender.file.rollingPolicy.maxIndex=50
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d [%t] %-5p (%F:%L) - %m%n
log4j.appender.file.Threshold=DEBUG

From the API of org.apache.log4j.rolling.FixedWindowRollingPolicy, it is reduced by design to 12 backups plus the current one, so 13 maximum.
You can look at it : https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/FixedWindowRollingPolicy.html

This is happening because large window size is discouraged and the maxIndex is automatically reduced to 12. It is well documented here.
Given that this rollover algorithm requires as many file renaming operations as the window size, large window sizes are discouraged. The current implementation will automatically reduce the window size to 12 when larger values are specified by the user.

Encode pattern layout in apache log4j

Trying to use the following pattern layout to print encoded log messages, so that \n will not be printed as a new line in the logs.
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{MM-dd-yyyy HH:mm:ss,SSS} %5p [%t] (%F:%L) - %enc{%m}%n"/>
</layout>
But it literally prints %enc{ before every message and prints } at the end of the messages, instead of encoding them. \n is also not being replaced.
Example: For this code logger.info("my \n message") following is the output log
08-01-2018 01:26:37,642 INFO [localhost-startStop-1] (ClassName1.java:53) - %enc{my
message}
instead it should be like,
08-01-2018 01:26:37,642 INFO [localhost-startStop-1] (ClassName1.java:53) - my message
Followed this document https://logging.apache.org/log4j/log4j-2.1/manual/layouts.html But I'm not sure what is wrong with the pattern.

You should determine the encoding format in curly brackets. For example %enc{%m}{HTML} encodes the message in HTML format. You could also use CRLF: enc{%m}{CRLF}

log4j grok with pipes fail

i need to format a grok input for the next log format
[#|2017-10-12T07:40:16.232-0700|INFO|sun-appserver2.1|RugDaemonCMLog|_ThreadID=17;_ThreadName=Thread-25;|Wake Up!!#]
[#|2017-10-12T07:40:16.241-0700|INFO|sun-appserver2.1|RugDaemonCMLog|_ThreadID=17;_ThreadName=Thread-25;|--Don't have a work in batch|#]
[#|2017-10-12T07:40:16.241-0700|INFO|sun-appserver2.1|RugDaemonCMLog|_ThreadID=17;_ThreadName=Thread-25;|go to sleep!|#]
[#|2017-10-12T07:40:16.567-0700|INFO|sun-appserver2.1|RugProject|_ThreadID=16;_ThreadName=Thread-23;| sleep---|#]
[#|2017-10-12T07:40:16.568-0700|INFO|sun-appserver2.1|RugProject|_ThreadID=16;_ThreadName=Thread-23;|Sending Mail|#]
[#|2017-10-12T07:40:16.569-0700|INFO|sun-appserver2.1|RugProject|_ThreadID=16;_ThreadName=Thread-23;|--lookup--|#]
i have the following sintax in my log4j file
log4j.appender.file.datePattern='.'yyyy-MM-dd_HH_mm
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=100
log4j.appender.file.encoding=UTF-8
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{dd-MM-yyyy HH:mm:ss} %-5p %c{1}:%L - %m%n
in the debugger online i'm trying this format
%{TIMESTAMP_ISO8601:logdate}\|%{LOGLEVEL:loglevel}|%{WORD:caller}\|%{NONNEGINT:line} - %{GREEDYDATA:message}$
that format parse the date and the log level info, but not the caller and the thread information

Use this:
%{TIMESTAMP_ISO8601:logdate}\|%{LOGLEVEL:loglevel}\|%{DATA:caller}\|%{DATA:line}\|_ThreadID=%{DATA:threadid};_ThreadName=%{DATA:threadname}\|%{GREEDYDATA:message}\|%{GREEDYDATA:fin}$

How do I show all whitespace log4j supplied messages?

I supply a formatted java string with significant whitespace
String.format("%1$12s %2$15s", "someString", "anotherOne");
to be logged using the following conversion pattern in log4j
<param name="ConversionPattern" value="%d{ISO8601} %-5p [%t] %c{1} %m%n" />
In the logs, I receive the variable with just a single preceding whitespace.
... PatternStuff ... someString anotherOne
slf4j version 1.7.3
log4j version 1.2.14
Is it possible to show the whitespace added by String.format? Some other conversion pattern variable other than %m?

My workplace has an internal web-based logviewer that quickly gives developers a way to access the contents of log files. We have the ability to print logs out in a generated webpage or to download the log. When I view them in the browser, the spaces are collapsed (because of standard HTML output). However, when I download the log files, they appear with the given formatting! Sorry for the confusion.

Log4j to write json array to disk

Im creating a log file system. the log file will be in json format so a server can read it after but i dont think thats too important. What i need to know is can log4j be configured to write into to a file but without any tags like info,debug, timestamp etc in the file. I have looked here
but this polutes the file with with other things. I want ONLY the data i write to show up in the file. I'd also like to set some kind of file rotation on the file if it gets too big after a max size is reached.

This is relatively easy, using a log4j.properties configuration file (place it at the top of your classpath, and Log4j will 'just find it'):
# This is the default logger, simply logs to console
log4j.logger.com.foo.bar=DEBUG,A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
# Note the Pattern here, emits a lot of stuff - btw, don't use this in production
# %C is expensive - see the Javadoc for ConversionPattern for the meaning of all
# the % modifiers:
log4j.appender.A1.layout.ConversionPattern=%d{MMM dd, HH:mm:ss} [%C{2}] %-5p - %m%n
# Logging to file can be enabled by using this one
log4j.logger.com.example=DEBUG, R
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=/var/log/generic.log
log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1
# This is the most minimalist layout you can have: just the 'm'essage is emitted
# (and a \n newline):
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%m%n
All the classes in the com.foo.bar package (and subpackages) will log to console, those in com.example (and below) will log to /var/log/generic.log.
To emit JSON, just use Jackson (com.fasterxml) convert your data to a JSON object and write it out as a string.

What you want is the PatternLayout with %m%n only, and combine with the answer to previously asked question here

You should be able to write a custom log appender, I have a vague recollection of there being an example to do this with MongoDB.
MongoDD stores its data as JSON, so it have converted the log to json format before inserting it.
I would start by searching for a MondgoDB appender, and looking at the file appender that is shipped, that should give you a starting point for an appender if one doesnt already exist.
http://log4mongo.org/display/PUB/Home