i am quite new ot apache Kafka and log4j. i am trying to send my log messages into Kafka. Here is my log4j properties file
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% %m%n
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.BrokerList=localhost:9092
log4j.appender.KAFKA.Topic=kfkLogs
log4j.appender.KAFKA.SerializerClass=kafka.producer.DefaultStringEncoder
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n
log4j.logger.logGen=DEBUG, KAFKA
however, i am not able to receive any messages in my consumer. I have tested the consumer with some other producer code and it works fine.
also, i get this warning
log4j:WARN No such property [serializerClass] in kafka.producer.KafkaLog4jAppender.
Edit
and here is the code that generates my log messages
package logGen;
import org.apache.log4j.Logger;
public class TestLog4j {
static Logger log = Logger.getLogger(TestLog4j.class.getName());
public static void main(String[] args) {
log.debug("Debug message");
log.info("Info message");
log.error("Error Message");
log.fatal("Fatal Message");
log.warn("Warn Message");
log.trace("Trace Message");
}
}
Also, if i write the log messages to a file by using something like
log4j.appender.KAFKA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.KAFKA.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.KAFKA.File=logs/server.log
i can see the log messages in the server.log file
thanks for the suggestions everyone. i think the weird behavior that i see might be related to my kafka set up. here are the contents of my server.properties file which i use to start up my kafka server. can you see anything odd about it ?
broker.id=0
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/Users/xyz/kafka/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
I have took a look at the source code of KafkaLog4jAppender.scala and here are the valid and exhaustive properties for Kafka log4j appender as of version 0.8.2.1 : topic, brokerList, compressionType, requiredNumAcks, syncSend.
The log4j.properties that worked for me is below :
log4j.rootLogger=ERROR, stdout
log4j.logger.logGen=DEBUG, KAFKA
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% %m%n
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.topic=LOG
log4j.appender.KAFKA.brokerList=localhost:9092
log4j.appender.KAFKA.compressionType=none
log4j.appender.KAFKA.requiredNumAcks=0
log4j.appender.KAFKA.syncSend=true
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n
You need to add KAFKA to your log4j.rootLogger like this:
log4j.rootLogger=INFO, stdout, KAFKA
This will add the KAFKA appender to your rootLogger.
i had to specify
log4j.appender.KAFKA.producer.type=async
log4j.logger.logGen.TestLog4j=TRACE, KAFKA
and that seems to work. however i experience a delay of 10-30 seconds. Specifically, if i publish right now and can see the messages in teh consumer, then the next time i publish has to be about 30 secs later else i dont see anything in my consumer. any ideas on why this might be happening ? maybe its an eclispe issue ?
log4j.appender.KAFKA.SerializerClass=kafka.producer.DefaultStringEncoder
could you try
log4j.appender.KAFKA.Serializer=kafka.producer.DefaultStringEncoder
Instead?
I believe the async mode of sending msgs does so by batching, hence the delay, have you you tried sending using sync?
Related
I'm trying to make a POC using log4j JMSappender and ActiveMQ. I successfully follow this example -> http://activemq.apache.org/how-do-i-use-log4j-jms-appender-with-activemq.html
But this is for INFO level. I made some modification to use it in DEBUG level and my program and activeMQ can't exchange their wire format. Here are my log4j (client side)
log4j.rootLogger=DEBUG, stdout, jms
log4j.logger.org.apache.activemq=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %-5p %c - %m%n
log4j.appender.stdout.Threshold=INFO
log4j.appender.jms=org.apache.log4j.net.JMSAppender
log4j.appender.jms.InitialContextFactoryName=org.apache.activemq.jndi.ActiveMQInitialContextFactory
log4j.appender.jms.ProviderURL=tcp://localhost:61616
log4j.appender.jms.TopicBindingName=logTopic
log4j.appender.jms.TopicConnectionFactoryBindingName=ConnectionFactory
log4j.appender.jms.Threshold=DEBUG
When running my client, I have the wire format error.
Producer program:
import org.apache.log4j.Logger;
public class ProducerDebug {
public static void main( String[] args ) throws InterruptedException
{
new ProducerDebug();
}
private ProducerDebug() throws InterruptedException {
Logger logger = Logger.getLogger(ProducerInfo.class);
while (true)
{
logger.debug("push DEBUG");
logger.info("push INFO2");
Thread.sleep(10000);
}
}
}
And the error message:
log4j:ERROR Error while activating options for appender named [jms].
javax.jms.JMSException: Wire format negotiation timeout: peer did not
send his wire format. at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1412)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1501)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:323)
at
org.apache.activemq.ActiveMQConnection.createTopicSession(ActiveMQConnection.java:1134)
at
org.apache.log4j.net.JMSAppender.activateOptions(JMSAppender.java:218)
at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127) at
org.apache.log4j.Logger.getLogger(Logger.java:117) at
awg.adb.producer.ProducerDebug.(ProducerDebug.java:13) at
awg.adb.producer.ProducerDebug.main(ProducerDebug.java:9) Caused by:
java.io.IOException: Wire format negotiation timeout: peer did not
send his wire format. at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:98)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1387)
... 17 more
I tried many things and I cannot understand why I have this error.
For information: I'm using
- ActiveMQ 5.12.1
- Log4j 1.2.17
yes it is due to AMQ logs if level is debug, one solution is to do the opposite like this :
log4j.rootLogger=INFO, stdout
## Be sure that ActiveMQ messages are not logged to 'jms' appender
log4j.logger.org.apache=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %-5p %c - %m%n
## Configure 'jms' appender. You'll also need jndi.properties file in order to make it work
log4j.appender.jms=org.apache.log4j.net.JMSAppender
log4j.appender.jms.InitialContextFactoryName=org.apache.activemq.jndi.ActiveMQInitialContextFactory
log4j.appender.jms.ProviderURL=tcp://localhost:61616
log4j.appender.jms.TopicBindingName=logTopic
log4j.appender.jms.TopicConnectionFactoryBindingName=ConnectionFactory
log4j.appender.jms.Threshold=DEBUG
log4j.logger.your.package=jms
Can you explain me, how to configure log4j.properties - so to save only levels: "trace","debug","info" in the file. This code saved in the file only levels from "INFO" to "FATAL"
log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=log4j-HigherWarnSaveInFile.log
log4j.appender.file.MaxFileSize=1MB
log4j.appender.file.MaxBackupIndex=1
log4j.appender.file.Append=true
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{2}:%L - %m%n
Java file
LOGGER.trace("Trace Message!");
LOGGER.debug("Debug Message!");
LOGGER.info("Info Message!");
LOGGER.warn("Warn Message!");
LOGGER.error("Error Message!");
LOGGER.fatal("Fatal Message!");
Log4j works with a threshold mechanism, so usually you define a threshold and all messages logged on or above that level will be logged.
So if you set the threshold level to INFO, you'll log INFO, WARN, ERROR and FATAL. TRACE and DEBUG will be filtered out. This is not what you need.
But you can set the root logger's threshold to TRACE or ALL and then use a LevelRangeFilter to filter out the higher level messages.
log4.rootLogger=ALL, file
[...]
log4j.appender.file.filter=org.apache.log4j.varia.LevelRangeFilter
log4j.appender.file.filter.LevelMin=TRACE
log4j.appender.file.filter.LevelMax=INFO
I want to create 2 types of log 1 Debug log which will create all log another I want to create activity log I mean each method how much time took to execute or any specific info,
I am using below log4j property file-
please correct me, its logging all messages in only one file,in java I have instantiated both log object, kindly don't send any pointers or just Google because I have tried since last 2 days all options as described in Google,
Thanks in advance for your kind support,
log4j.rootLogger=debugLog,reportsLog
log4j.appender.debugLog=org.apache.log4j.FileAppender
log4j.appender.debugLog.File=logs/debug.log
log4j.appender.debugLog.layout=org.apache.log4j.PatternLayout
log4j.appender.debugLog.layout.ConversionPattern=%d [%24F:%t:%L] - %m%n
log4j.appender.reportsLog=org.apache.log4j.FileAppender
log4j.appender.reportsLog.File=logs/reports.log
log4j.appender.reportsLog.layout=org.apache.log4j.PatternLayout
log4j.appender.reportsLog.layout.ConversionPattern=%d [%24F:%t:%L] - %m%n
log4j.category.debugLogger=INFO, debugLog
log4j.additivity.debugLogger=false
log4j.category.reportsLogger=DEBUG, reportsLog
log4j.additivity.reportsLogger=false
Below log4j.properties file will configure the logger to log the messages with debug level to the logs/debug.log file. Messages with level INFO...FATAL are logged to logs/reports.log.
log4j.rootLogger=DEBUG, debugLog, reportsLog
log4j.appender.debugLog=org.apache.log4j.FileAppender
log4j.appender.debugLog.File=logs/debug.log
log4j.appender.debugLog.layout=org.apache.log4j.PatternLayout
log4j.appender.debugLog.layout.ConversionPattern=%d [%24F:%t:%L] - %m%n
log4j.appender.debugLog.filter.f1=org.apache.log4j.varia.LevelRangeFilter
log4j.appender.debugLog.filter.f1.LevelMax=DEBUG
log4j.appender.debugLog.filter.f1.LevelMin=DEBUG
log4j.appender.reportsLog=org.apache.log4j.FileAppender
log4j.appender.reportsLog.File=logs/reports.log
log4j.appender.reportsLog.layout=org.apache.log4j.PatternLayout
log4j.appender.reportsLog.layout.ConversionPattern=%d [%24F:%t:%L] - %m%n
log4j.appender.reportsLog.filter.f1=org.apache.log4j.varia.LevelRangeFilter
log4j.appender.reportsLog.filter.f1.LevelMax=FATAL
log4j.appender.reportsLog.filter.f1.LevelMin=INFO
log4j.category.debugLogger=DEBUG, debugLog
log4j.additivity.debugLogger=false
log4j.category.reportsLogger=INFO, reportsLog
log4j.additivity.reportsLogger=false
I am trying to use KafkaLog4jAppender to write to kafka and also a file appender to write to a file. The conversion pattern works for the file, but for the kafka appender it is not working
The output of the file appender is 2014-09-19T22:30:14.781Z INFO com.test.poc.StartProgram Message1
But the Kafka appender has output as Message1
Find below my log4j.properties file
log4j.rootCategory=INFO
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.file.File=logs/test.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}{UTC} %p %C %m%n
log4j.logger.com=INFO,file,KAFKA
#Kafka Appender
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}{UTC} %p %C %m%n
log4j.appender.KAFKA.ProducerType=async
log4j.appender.KAFKA.BrokerList=localhost:9092
log4j.appender.KAFKA.Topic=test`enter code here`
log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
This is a known Kafka bug; the fix is in Kafka 0.8.2:
https://issues.apache.org/jira/browse/KAFKA-847
I'm trying to log certain info messages onto a file but as soon as I run the application both warn and info messages are logged. Now, from what I've read from this site, you cannot log one without logging the other. Has anyone tried this before? If so, how did your properties file look like?
My properties file looks like this:
***** Set root logger level to INFO and its two appenders to stdout and R.
log4j.rootLogger=INFO, stdout, R
# ***** stdout is set to be a ConsoleAppender.
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
# ***** stdout uses PatternLayout.
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# ***** Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%M has started] (%F:%L) - %m%n
/
# ***** R is set to be a RollingFileAppender.
log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.R.File="folder where log will be saved"
log4j.appender.R.layout.ConversionPattern=%5p [%m has started] %c{2}.[%x] (%F:%L) %d{yyyy-MM-dd HH:mm:ss} - %m%n
# ***** R uses PatternLayout.
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%5p [%m has started] %c{2}.[%x] (%F:%L) %d{yyyy-MM-dd HH:mm:ss} - %m%n
AFAIK there's no standard way to suppress higher log levels than those you are interested in.
However, you might be able to use a custom appender to do that.
It might look similar to this:
public class MyAppender extends AppenderSkeleton {
protected void append(LoggingEvent event) {
if( event.getLevel() == Level.INFO ) {
//append here, maybe call a nested appender
}
}
}
The log level WARN is higher than INFO, and the logging configuration defines the minimum threshold level to be logged by the appender. So, all messages higher than that level will also be logged.
Hence, the WARN messages are expected. And I don't think you can configure it to the way you want.
If the WARN messages that should not be printed come from a different package than the INFO messages, then you can define different log levels for these packages. The first can specify level ERROR and the second INFO
it should look something like this:
log4j.logger.com.test.something=ERROR
log4j.logger.com.other.package=INFO
cheers
Why do you want to filter the WARN level? As peshkira said, you could use two different loggers for splitting/filtering your log output, or you could use tools like grep for filtering (offline) or you could simply remove the WARN logs from your code if your don't need them anyway.
As far as I understand, advanced filters like LevelMatchFilter and LevelRangeFilter can do the trick for you.
Thing worth keeping in mind though is that using these may require xml config instead of properties: Can't set LevelRangeFilter for log4j