We have a weblogic batch application which processes multiple requests from consumers at the same time. We use log4j for logging puposes. Right now we log into a single log file for multiple requests. It becomes tedious to debug an issue for a given request as for all requests the logs are in a single file.
So plan is to have one log file per request. The consumer sends a request ID for which processing has to be performed. Now, in reality there could be multiple consumers sending the request IDs to our application. So question is how to seggregate the log files based on the request.
We cannot start & stop the production server every time so the point in using an overridden file appender with date time stamp or request ID is ruled out. This is what is explained in the article below:
http://veerasundar.com/blog/2009/08/how-to-create-a-new-log-file-for-each-time-the-application-runs/
I also tried playing around with these alternatives:
http://cognitivecache.blogspot.com/2008/08/log4j-writing-to-dynamic-log-file-for.html
http://www.mail-archive.com/log4j-user#logging.apache.org/msg05099.html
This approach gives the desired results but it does not work properly if multiple request are send at the same time. Due to some concurrency issues logs go here and there.
I anticipate some help from you folks. Thanks in advance....
Here's my question on the same topic:
dynamically creating & destroying logging appenders
I follow this up on a thread where I discuss doing something exactly like this, on the Log4J mailing list:
http://www.qos.ch/pipermail/logback-user/2009-August/001220.html
Ceci Gulcu (inventor of log4j) didn't think it was a good idea...suggested using Logback instead.
We went ahead and did this anyway, using a custom file appender. See my discussions above for more details.
Look at SiftingAppender shipping with logback (log4j's successor), it is designed to handle the creation of appenders on runtime criteria.
If you application needs to create just one log file per session, simply create a discriminator based on the session id. Writing a discriminator involves 3 or 4 lines of code and thus should be fairly easy. Shout on the logback-user mailing list if you need help.
This problem is handled very well by Logback. I suggest to opt for it if you have the freedom.
Assuming you can, what you will need to use is is SiftingAppender. It allows you to separate log files according to some runtime value. Which means that you have a wide array of options of how to split log files.
To split your files on requestId, you could do something like this:
logback.xml
<configuration>
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<key>requestId</key>
<defaultValue>unknown</defaultValue>
</discriminator>
<sift>
<appender name="FILE-${requestId}" class="ch.qos.logback.core.FileAppender">
<file>${requestId}.log</file>
<append>false</append>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d [%thread] %level %mdc %logger{35} - %msg%n</pattern>
</layout>
</appender>
</sift>
</appender>
<root level="DEBUG">
<appender-ref ref="SIFT" />
</root>
</configuration>
As you can see (inside discriminator element), you are going to discriminate the files used for writing logs on requestId. That means that each request will go to a file that has a matching requestId. Hence, if you had two requests where requestId=1 and one request where requestId=2, you would have 2 log files: 1.log (2 entries) and 2.log (1 entry).
At this point you might wonder how to set the key. This is done by putting key-value pairs in MDC (note that key matches the one defined in logback.xml file):
RequestProcessor.java
public class RequestProcessor {
private static final Logger log = LoggerFactory.getLogger(RequestProcessor.java);
public void process(Request request) {
MDC.put("requestId", request.getId());
log.debug("Request received: {}", request);
}
}
And that's basically it for a simple use case. Now each time a request with a different (not yet encountered) id comes in, a new file will be created for it.
using filePattern
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Properties>
<property name="filePattern">${date:yyyy-MM-dd-HH_mm_ss}</property>
</Properties>
<Appenders>
<File name="File" fileName="export/logs/app_${filePattern}.log" append="false">
<PatternLayout
pattern="%d{yyyy-MMM-dd HH:mm:ss a} [%t] %-5level %logger{36} - %msg%n" />
</File>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="Console" />
<AppenderRef ref="File" />
</Root>
</Loggers>
</Configuration>
Related
I am new to log4j2. Previously I am using log4j. The reason I am migrating into part 2 is for Asynchronous logging. After searching Internet I am able to write a configuration file that actually creates two log files "Errors.log" and "Messages.log". Now the problem is : I would be communicating with Servers that are kept far away from me. I wrote a client that communicates with the server and sends a request and in back the Server sends me a response. In any situation it takes at least 10 milli-seconds for the request to reach the Server and get back the response from it. But in my log files it is showing that the request sent to the Server and receive from the Server is at same time (Same milli-second). I am using the Asynchronous logging. Is this causing the wrong timestamp? or else the policies which I have used are creating these issues?
Below is my Log4j2 XML CONFIG file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
<Appenders>
<File name="my_file_appender" fileName="LOG4j_LOGS/Errors.log" immediateFlush="false" append="false">
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</Pattern>
</PatternLayout>
</File>
<Async name="async_appender">
<AppenderRef ref="my_file_appender" />
</Async>
<!-- file appender -->
<RollingFile name="Error-log" fileName="LOG4j_LOGS/Messages.log"
filePattern="LOG4j_LOGS/Messages-%d{yyyy-MM-dd}.log">
<!-- log pattern -->
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</Pattern>
</PatternLayout>
<!-- set file size policy -->
<Policies>
<TimeBasedTriggeringPolicy />
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
<DefaultRolloverStrategy max="25"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="Error-log" level="info" additivity="false">
<appender-ref ref="Error-log" level="debug"/>
</Logger>
<Root level="info" includeLocation="false">
<AppenderRef ref="async_appender"/>
</Root>
</Loggers>
</Configuration>
Can anyone please check my CONFIG file. All I want is to create two separate log files, one for storing info messages and other for storing errors. And this should create a new file every time I run my application and it should not delete the previous logs. The size of the logs can be anything. If the size has exceeded it should create a new file and write the data into it. No matter how many days I run the application the daily logs needs to be stored and the entire process has to be done in Asynchronously.
I am also using the below VM options for logging asynchronously :
-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
When logging is done asynchronously you log messages got onto a separate queue. They are timestamped at the moment of processing by background thread that writes logs to disk (since timestamp is part of you Appender pattern). So only the order is preserved. Timestamps may be slightly different.
See https://logging.apache.org/log4j/2.x/manual/async.html for more info.
Am trying to understand if there are any best practices/utilities/out of the box functionality for logging messages that are quite long (typically json file responses/xmls/CSVs etc).
While logging as part of my application I do something like
log.info("Incompatible order found {}", order.asJson())
The problem being that the asJson() representation could be pretty long. In the log files the actual json is only relevant say 1% of the time. So it is important enough to retain but big enough to make me lose focus when I am trawling through logs.
Is there anyway I can could do something like
log.info("Incompatible order found, file dumped at {}", SomeUtility.dumpString(order.asJson()));
where the utility dumps the file into a location consistent with other log files and then in my log file I can see the following
Incompatible order found, file dumped at /abc/your/log/location/tmpfiles/xy23nmg
Key things to note being
It would be preferable to simply use the existing logging api to somehow configure this so that the location of these temp files are the same as log itself this way it goes through the cleanup cycle after N days etc just like the actual log files.
I can obviously write something but I am keen on existing utilities or features if already available withing log4j
I am aware that when logs such as these are imported into analysis systems like Splunk then only the filenames will be present without the actual files and this is okay.
(Suggestion was given based on logback instead of log4j2. However I believe similar facilities exists in log4j2)
In Logback, there is a facility called SiftingAppender, which allow separate appender (e.g. file appender) to be created according to some discriminator (e.g. in MDC).
So, by configuring an appender (e.g. orderDetailAppender) which separates file based on a discriminator (e.g. by putting order ID in MDC), and make use of a separate logger to connect to the appender, this should give you the result you want:
pseudo code:
logback config:
<appender name="ORDER_DETAIL_APPENDER" class="ch.qos.logback.classic.sift.SiftingAppender">
<!-- use MDC disciminator by default, you may choose/develop other appropriate discrimator -->
<sift>
<appender name="ORDER-DETAIL-${ORDER_ID}" class="ch.qos.logback.core.FileAppender">
....
</appender>
</sift>
</appender>
<logger name="ORDER_DETAIL_LOGGER">
<appender-ref name="ORDER_DETAIL_APPENDER"/>
</logger>
and your code looks like:
Logger logger = LoggerFactory.getLogger(YourClass.class); // logger you use normally
Logger orderDetailLogger = LoggerFactory.getLogger("ORDER_DETAIL_LOGGER");
.....
MDC.put("ORDER_ID", order.getId());
logger.warn("something wrong in order {}. Find the file in separate file", order.getId());
orderDetailLogger.warn("Order detail:\n{}", order.getJson());
// consider making order.getJson() an lambda, or wrap the line with logger
// level checking, to avoid json construction even not required to log
MDC.remove("ORDER_ID");
The simplest approach is to use a separate logger for the JSON dump, and configure that logger to put its output in a different file.
As an example:
private static final Logger log = LogManager.getLogger();
private static final Logger jsonLog = LogManager.getLogger("jsondump");
...
log.info("Incompatible order found, id = {}", order.getId());
jsonLog.info("Incompatible order found, id = {}, JSON = {}",
order.getId(), order.asJson());
Then, configure the logger named jsondump to go to a separate file, possibly with a separate rotation schedule. Make sure to set additivity to false for the JSON logger, so that messages sent to that logger won't be sent to the root logger as well. See Additivity
in the Log4J docs for details.
Example (XML configuration, Log4J 2):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<!-- By setting only filePattern, not fileName, all log files
will use the pattern for naming. Files will be rotated once
an hour, since that's the most specific unit in the
pattern. -->
<RollingFile name="LogFile" filePattern="app-%d{yyyy-MM-dd HH}00.log">
<PatternLayout pattern="%d %p %c [%t] %m %ex%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
<!-- See comment for appender named "LogFile" above. -->
<RollingFile name="JsonFile" filePattern="json-%d{yyyy-MM-dd HH}00.log">
<!-- Omitting logger name for this appender, as it's only used by
one logger. -->
<PatternLayout pattern="%d %p [%t] %m %ex%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<!-- Note that additivity="false" to prevent JSON dumps from being
sent to the root logger as well. -->
<Logger name="jsondump" level="debug" additivity="false">
<appender-ref ref="JsonFile"/>
</Logger>
<Root level="debug">
<appender-ref ref="LogFile"/>
</Root>
</Loggers>
</Configuration>
First time trying to use log4j version log4j-1.2.17.jar.
On an existing application the client has log4j in place and there is a log4j.properties file which specifies a light log output. What I want to do is depending on the log level (ERROR & WARN) output a more refined entry.
On the log4j site I came across this but I think it is to be in some .xml file. I need some assistance in understanding how I can put in place the formatting option to alter based on log level.
You don't need to declare separate loggers to achieve this. You can set the logging level on the AppenderRef element.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<File name="file" fileName="app.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m %ex%n</Pattern>
</PatternLayout>
</File>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="file" level="DEBUG"/>
<AppenderRef ref="STDOUT" level="INFO"/>
</Root>
</Loggers>
</Configuration>
Would I put this xml content into the web.xml file or another file?
a) If another file what file name and where would it go?
How do I get log4j to realize that I need it to use the xml file?
Will the use of the xml ignore the log4j.properties file?
I know it is a lot of questions but there is only me on the project and the client has a production crisis that needs to be figured out today so I don't have time to go off to read and do tutorials with the client calling me every hour. I figured it may help to get this logging more useful. As the logs are right now I have a date and message output to the log but no idea where the entries are created from without doing extensive searches through the code.
You could do this by defining multiple non-additive Loggers so that the first one only logs errors, the next one warnings and the final one infos and debug.
I use log4j2 in my project something like this:
logger.log(Level.ERROR, this.logData);
My configuration file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR" DLog4jContextSelector="org.apache.logging.log4j.core.async.AsyncLoggerContextSelector">
<Appenders>
<!-- Async Loggers will auto-flush in batches, so switch off immediateFlush. -->
<RandomAccessFile name="RandomAccessFile" fileName="C:\\logs\\log1.log" immediateFlush="false" append="false">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m %ex%n</Pattern>
</PatternLayout>
</RandomAccessFile>
</Appenders>
<Loggers>
<Root level="error" includeLocation="false">
<AppenderRef ref="RandomAccessFile"/>
</Root>
</Loggers>
It creates my file, I log something to it, but it's still empty. When I trying to delete this file, OS told me that it in use (if app currently working), but even if I stop application, file still empty.
So which settings should I change to make it work correctly?
I suspect that asynchronous logging is not switched on correctly.
As of beta-9 it is not possible to switch on Async Loggers in the XML configuration, you must set the system property Log4jContextSelector to "org.apache.logging.log4j.core.async.AsyncLoggerContextSelector".
The reason you are not seeing anything in the log is that your log messages are still in the buffer and have not been flushed to disk yet. If you switch on Async Loggers the log messages will be flushed to disk automatically.
I share a cleaner and easier solution.
https://stackoverflow.com/a/33467370/3397345
Add a file named log4j2.component.properties to your classpath. This can be done in most maven or gradle projects by saving it in src/main/resources.
Set the value for the context selector by adding the following line to the file.
Log4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
Log4j will attempt to read the system property first. If the system property is null, then it will fall back to the values stored in this file by default.
Is there a way to configure log4j so that it outputs different levels of logging to different appenders?
I'm trying to set up multiple log files. The main log file would catch all INFO and above messages for all classes. (In development, it would catch all DEBUG and above messages, and TRACE for specific classes.)
Then, I would like to have a separate log file. That log file would catch all DEBUG messages for a specific subset of classes, and ignore all messages for any other class.
Is there a way to get what I'm after?
This should get you started:
log4j.rootLogger=QuietAppender, LoudAppender, TRACE
# setup A1
log4j.appender.QuietAppender=org.apache.log4j.RollingFileAppender
log4j.appender.QuietAppender.Threshold=INFO
log4j.appender.QuietAppender.File=quiet.log
...
# setup A2
log4j.appender.LoudAppender=org.apache.log4j.RollingFileAppender
log4j.appender.LoudAppender.Threshold=DEBUG
log4j.appender.LoudAppender.File=loud.log
...
log4j.logger.com.yourpackage.yourclazz=TRACE
Perhaps something like this?
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<!-- general application log -->
<appender name="MainLogFile" class="org.apache.log4j.FileAppender">
<param name="File" value="server.log" />
<param name="Threshold" value="INFO" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %t [%-40.40c] %x - %m%n"/>
</layout>
</appender>
<!-- additional fooSystem logging -->
<appender name="FooLogFile" class="org.apache.log4j.FileAppender">
<param name="File" value="foo.log" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %t [%-40.40c] %x - %m%n"/>
</layout>
</appender>
<!-- foo logging -->
<logger name="com.example.foo">
<level value="DEBUG"/>
<appender-ref ref="FooLogFile"/>
</logger>
<!-- default logging -->
<root>
<level value="INFO"/>
<appender-ref ref="MainLogFile"/>
</root>
</log4j:configuration>
Thus, all info messages are written to server.log; by contrast, foo.log contains only com.example.foo messages, including debug-level messages.
I had this question, but with a twist - I was trying to log different content to different files. I had information for a LowLevel debug log, and a HighLevel user log. I wanted the LowLevel to go to only one file, and the HighLevel to go to both a file, and a syslogd.
My solution was to configure the 3 appenders, and then setup the logging like this:
log4j.threshold=ALL
log4j.rootLogger=,LowLogger
log4j.logger.HighLevel=ALL,Syslog,HighLogger
log4j.additivity.HighLevel=false
The part that was difficult for me to figure out was that the 'log4j.logger' could have multiple appenders listed. I was trying to do it one line at a time.
Hope this helps someone at some point!
For the main logfile/appender, set up a .Threshold = INFO to limit what is actually logged in the appender to INFO and above, regardless of whether or not the loggers have DEBUG, TRACE, etc, enabled.
As for catching DEBUG and nothing above that... you'd probably have to write a custom appender.
However I'd recommend not doing this, as it sounds like it would make troubleshooting and analysis pretty hard:
If your goal is to have a single file where you can look to troubleshoot something, then spanning your log data across different files will be annoying - unless you have a very regimented logging policy, you'll likely need content from both DEBUG and INFO to be able to trace execution of the problematic code effectively.
By still logging all of your debug messages, you are losing any performance gains you usually get in a production system by turning the logging (way) down.
Demo link: https://github.com/RazvanSebastian/spring_multiple_log_files_demo.git
My solution is based on XML configuration using spring-boot-starter-log4j. The example is a basic example using spring-boot-starter and the two Loggers writes into different log files.