I understand that I can send log messages from applications which use log4j to Apache Kafka by using the log4j kafka appender. For example,
log4j.appender.KAFKA_HIVE_AUDIT=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
log4j.appender.KAFKA_HIVE_AUDIT.Topic=hive_audit_log
log4j.appender.KAFKA_HIVE_AUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n
This is described here: how to use Kafka 0.8 Log4j appender
I'm running Kafka 0.9 which has kerberos enabled. Is there a way to have the log4j appender perform authentication to Kafka? Some kind of service account?
Is there a way to for the machine which the java program is running on to authenticate to the Kafka cluster via kerberos before running the log4j appender?
If that doesn't work, is there a way to grant write privileges to unauthenticated producers on kerberized Kafka by machine? (And still require kerberos authentication for consumers)?
If you look at the Kafka 0.9 source, you will see that Kerberos authentication is not supported by the KafkaLog4jAppender, despite the fact that it was added for Kafka producers and consumers generally.
Kerberos support was only added to the KafkaLog4jAppender in version 0.10.
From a quick glance at the code, it looks like it would be straightforward to create a custom appender that extends the KafkaLog4jAppender and adds the necessary bits for Kerberos support.
Kafka is Open Source, why don't you just check the code and work out which properties are related to Kerberos and how they are used?
A quick look at "trunk" i.e. https://github.com/apache/kafka/blob/trunk/log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java hints that you must attach a JAAS config file, and specify which "context" to use in that file, so that the Kafka client can use the standard Java Security libraries. And optionally provide a custom Kerberos config file if it is not in the standard location (i.e. /etc/krb5.conf).
So in the end it's a matter of understanding JAAS -- and finding out which Kafka and/or Log4J properties to set.
And with Google, a tutorial is always a few clicks away -- e.g. that kerberized_kafka post.
Related
I am looking for a way to get more detail, like debug or verbose level logging, of a JMS message send over amqps to AzureServiceBus.
I am using qpid client 0.60.1 and I have no access to the calling code. I am working with a web application running in Weblogic. The application provides a servlet that has generic JMS functions, and I can use configuration that maps those to a specific providers' JMS connection factory libraries. To make qpid available to use, I add the qpid client jars to the CLASSPATH for when I start weblogic, and I provide a jndi.properties file that currently contains only two entries:
con
connectionfactory.ServiceBusConnectionFactory=amqps://?jms.username=&jms.password=
queue.inbound-general-q-QueueLookup=
Currently, this is the only message that I see in the weblogic log:
Connection ID:6147a0e7-1870-4a1a-8dd5-bd7102fc1aa4:106 connected to server: amqps://
I have been told that we don't have enough information to open a case with Microsoft.
I am looking for a way to get more detail, like debug or verbose level logging, of a message send. Ideally, want to see as much as possible: headers, properties, payload, etc.
The things I have access to change:
Weblogic environment, including classpath and any other java runtime flags
The jnd.properties file
I am reviewed the qpid.apache.org documentation on logging, but it has not been helpful to me as it is too vague.
The main application running in weblogic has these parameters in its runtime:
-Djava.util.logging.config.file=properties/logging.properties
-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger
I have tried adding some things to logging.properties, but it has never changed the output of the resulting log file to include anything from amqp.
I am using a Java micro-service architecture in my application and generating separate log files for each micro-service.
I am using ELK stack approach to visualize the logs in Kibana, but the problem is whatever the fields that I'm getting from Elastic Search that are related to server logs fields. some example fields are #timestamp,#version,#path,#version.keyword,#host.
i want to customize this fields by adding some fields like customerId,txn-Id,mobile no so that we can analyze the data easily.
I'm using org.apache.logging.log4j2 to write the logs. Can I set above fields (customerId,txn-Id,mobile) to log files? And then Elastic will store these fields with the above default fields and then these custom fields should available in a Kibana dashboard. Is this possible?
It's definitely possible to do that. I've not done it with the log4j2 stack (I have with slf4j/logback), but the basic approach is:
set those fields in the Mapped Diagnostic Context (I'm fairly sure log4j2 supports that)
use a log appender which logs to logstash-structured JSON
configure filebeat to ship the JSON logs
if filebeat is shipping to logstash, you'll need to configure logstash to pass those preformatted JSON logs directly to elasticsearch
It is definitely possible. I am doing that now with my applications. However, the output looks a bit different from yours. The basic guide for doing this can be found at Logging in the Cloud on the Log4j2 web site.
The "normal" log view looks very similar to what you would see when logging to a file.
However, if you select a message you can see the individual fieds.
The Log4j2 configuration uses a TCP Socket appender that is configured to write to a cluster of Logstash servers that use a single DNS entry and to use the Gelf layout.
You can also use MapMessages to capture individual data elements and log them. While this currently works it is slightly cumbersome so I have recently committed improvements that will be available in Log4j 2.15.0.
It is important to note that the Logging in the Cloud page briefly mentions storing your logging configuration in Spring Cloud Config. If you want to have a common base configuration while allowing apps to do some customization this works very, very well. However, The Gelf, Json Template Layout and TCP Appender are all independent from that and can be used without Spring Boot.
I'm having a hard time trying to find useful information and examples for using a log4j2 JMS appender to log XML message to an IBM MQ Series queue. It's been a while since I've used log4j and that was back before version 2.x came out.
In my log4j2.xml file, I have the following definition for the JMS appender:
<JMS name="jmsQueue"
destinationBindingName="BLTCAuditDest"
factoryName="com.sun.jndi.fscontext.RefFSContextFactory"
providerURL="file:///C:\\JNDI-Directory"
factoryBindingName="com.ibm.msg.client.jms.JmsQueueConnectionFactory">
<PatternLayout pattern="%m"/>
</JMS>
I created a .bindings file, that's in my C:\JNDI-Directory folder, using IBM MQ Explorer to create an Initial Context under the JMS Administered Objects folder. The "BLTCAuditDest" is the name of the Destination I created that's in the C:\JNDI-Directory.bindings file. And I'm guessing the IBM JmsQueueConnectionFactory is the proper class to use for the factoryBindingName. However, with these settings I get the following error:
ERROR An exception occurred processing Appender jmsQueue org.apache.logging.log4j.core.appender.AppenderLoggingException: Error sending to JMS Manager 'jmsQueue': JMS message producer not available
So I'm not sure if my destinationBindingName and factoryBindingName are correct, but I'm not sure what values I should be using here.
Does anyone have a simple sample or have a link to a simple sample that would show how to use a JMS Appender with IBM MQ?
Thank you!
Thanks to Maarten I was able to get basic ActiveMQ JMS topics and connection factories working in WAS. He has a nice write up in his reply to this topic: ActiveMQ 5.11 with WebSphere Application Server 8.5
But I cannot find a way to define any ActiveMQ JMS Activation Specs in the WAS admin console. And of course I need those in order to trigger my MDBs. ActiveMQ simply doesn't show up as a JMS provider when creating a new AS.
How do I configure Activation Specs in WAS using ActiveMQ as the provider? Am I missing a jar file?
activemq-client-5.11.0.jar
hawtbuf-1.11.jar
slf4j-api-1.7.10.jar
If you want to use Activation specification, you need to install ActiveMQ as JCA 1.5 compliant resource adapter. As far as I know, ActiveMQ provides resource adapter as separate install.
See also:
Deploying the ActiveMQ Resource Adapter into IBM WebSphere
Managing messaging with a third-party JCA 1.5 or 1.6-compliant messaging provider
ActiveMQ resource adapter
Listener ports are stabilized, and should only be used if provider doesn't support JCA.
Really straight forward once you understand (of course).
From the IBM Redbook mentioned above, sg247770.pdf, we need to configure ActiveMQ as a Generic JMS provider in WAS. And since we want to use Activation Specs, again from the Redbook, we need to use the ActiveMQ Resource Adapter, or rar file. There is a link on the ActiveMQ page to the latest rar, I don't need to provide it here. Once the rar is installed, using the WAS Console/Resources/Resource Adapters menu, you can configure J2C CFs, ASs, and administered objects including Queues and Topics from the rar configuration page. These will all have custom properties where you will enter your destinations, etc.
Am using Standard apache logging (org.apache.log4j.logging )
Currently, taking the data to be logged manually, and publishing in to Apache Active MQ.
Is it possible to configure the logging output to publish directly in to Active MQ??
This might sound stupid, but since both are from Apache, I have a doubt that whether, it has any implicit support, which I could not grab it.
log4j provides JMSAppender out of the box. It allows publishing logging events to JMS Topic.
For configuration specific to ActiveMQ please check the documentation - How do I use log4j JMS appender with ActiveMQ
Not sure if you were looking for log4j-1.x or log4j-2.0, but here are the links for log4j-2.0:
http://logging.apache.org/log4j/2.x/manual/appenders.html#JMSQueueAppender
http://logging.apache.org/log4j/2.x/manual/appenders.html#JMSTopicAppender