Develop Reference

log output from qpid library running in weblogic

I am looking for a way to get more detail, like debug or verbose level logging, of a JMS message send over amqps to AzureServiceBus.
I am using qpid client 0.60.1 and I have no access to the calling code. I am working with a web application running in Weblogic. The application provides a servlet that has generic JMS functions, and I can use configuration that maps those to a specific providers' JMS connection factory libraries. To make qpid available to use, I add the qpid client jars to the CLASSPATH for when I start weblogic, and I provide a jndi.properties file that currently contains only two entries:
con
connectionfactory.ServiceBusConnectionFactory=amqps://?jms.username=&jms.password=
queue.inbound-general-q-QueueLookup=
Currently, this is the only message that I see in the weblogic log:
Connection ID:6147a0e7-1870-4a1a-8dd5-bd7102fc1aa4:106 connected to server: amqps://
I have been told that we don't have enough information to open a case with Microsoft.
I am looking for a way to get more detail, like debug or verbose level logging, of a message send. Ideally, want to see as much as possible: headers, properties, payload, etc.
The things I have access to change:
Weblogic environment, including classpath and any other java runtime flags
The jnd.properties file
I am reviewed the qpid.apache.org documentation on logging, but it has not been helpful to me as it is too vague.
The main application running in weblogic has these parameters in its runtime:
-Djava.util.logging.config.file=properties/logging.properties
-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger
I have tried adding some things to logging.properties, but it has never changed the output of the resulting log file to include anything from amqp.

Is it possible to create custom fields in a Kibana dashboard?

I am using a Java micro-service architecture in my application and generating separate log files for each micro-service.
I am using ELK stack approach to visualize the logs in Kibana, but the problem is whatever the fields that I'm getting from Elastic Search that are related to server logs fields. some example fields are #timestamp,#version,#path,#version.keyword,#host.
i want to customize this fields by adding some fields like customerId,txn-Id,mobile no so that we can analyze the data easily.
I'm using org.apache.logging.log4j2 to write the logs. Can I set above fields (customerId,txn-Id,mobile) to log files? And then Elastic will store these fields with the above default fields and then these custom fields should available in a Kibana dashboard. Is this possible?

It's definitely possible to do that. I've not done it with the log4j2 stack (I have with slf4j/logback), but the basic approach is:
set those fields in the Mapped Diagnostic Context (I'm fairly sure log4j2 supports that)
use a log appender which logs to logstash-structured JSON
configure filebeat to ship the JSON logs
if filebeat is shipping to logstash, you'll need to configure logstash to pass those preformatted JSON logs directly to elasticsearch

It is definitely possible. I am doing that now with my applications. However, the output looks a bit different from yours. The basic guide for doing this can be found at Logging in the Cloud on the Log4j2 web site.
The "normal" log view looks very similar to what you would see when logging to a file.
However, if you select a message you can see the individual fieds.
The Log4j2 configuration uses a TCP Socket appender that is configured to write to a cluster of Logstash servers that use a single DNS entry and to use the Gelf layout.
You can also use MapMessages to capture individual data elements and log them. While this currently works it is slightly cumbersome so I have recently committed improvements that will be available in Log4j 2.15.0.
It is important to note that the Logging in the Cloud page briefly mentions storing your logging configuration in Spring Cloud Config. If you want to have a common base configuration while allowing apps to do some customization this works very, very well. However, The Gelf, Json Template Layout and TCP Appender are all independent from that and can be used without Spring Boot.

How to prevent Vertx from writing logs automatically?

When starting my TCP server using Vertx, I have the following output :
[2018-06-04 12:15:45] [FINEST ] Net server listening on 0.0.0.0:/0:0:0:0:0:0:0:0:8600
[2018-06-04 12:15:45] [INFO ] Server is now listening on port : 8600
I was expected the second line, since I am telling Vertx to write it :
server.listen(res -> {
if (res.succeeded()) {
logger.info("Server is now listening on port : {0, number, #}", server.actualPort());
}
else {
logger.error("Server failed to bind");
}
});
The first line though, is written by Vertx itself. I am bit surprised, since I could not see anywhere in Vertx documentation that this would happen nor how to prevent it from doing so.
How can I make Vertx stop logging automatically?
Thanks in advance.

Well, the manual states that vert.x by default uses java.util.logging, often referred to by its nickname JUL. It's configurable so depending on your use case you should be able to tune the log output. Alternatively vert.x can be instructed to use an external logging framework, they each have their own advantages and disadvantages.
The documentation for JUL isn't really the most helpful prose ever written, fortunately there are plenty of third party sites covering that topic, like http://tutorials.jenkov.com/java-logging/index.html but a quick Google will point you to many others too.
Resuming:
you will need to write a logging.properties file that reflects the output you want to obtain, and where (in logfiles and/or on the console)
you will have to pass that file to your vert.x application via the system property java.util.logging.config.file
Limiting the info produced by certain application parts can be done by using the filtering capabilities present in JUL. So, for example, in your logging.properties you could put
java.util.logging.FileHandler.level=INFO
which will restrict logging that goes to the logfile to INFO or higher. That like for example would already do away with the vert.x log you see in your example. You can also restrict logging per package, group of packages or even individual classes. A nice writeup of these possibilities can be found here: java.util.logging: how to set level by logger package (or prefix)? . I think vert.x uses the prefix io.vertx

Log4j appender authentication

I understand that I can send log messages from applications which use log4j to Apache Kafka by using the log4j kafka appender. For example,
log4j.appender.KAFKA_HIVE_AUDIT=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
log4j.appender.KAFKA_HIVE_AUDIT.Topic=hive_audit_log
log4j.appender.KAFKA_HIVE_AUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n
This is described here: how to use Kafka 0.8 Log4j appender
I'm running Kafka 0.9 which has kerberos enabled. Is there a way to have the log4j appender perform authentication to Kafka? Some kind of service account?
Is there a way to for the machine which the java program is running on to authenticate to the Kafka cluster via kerberos before running the log4j appender?
If that doesn't work, is there a way to grant write privileges to unauthenticated producers on kerberized Kafka by machine? (And still require kerberos authentication for consumers)?

If you look at the Kafka 0.9 source, you will see that Kerberos authentication is not supported by the KafkaLog4jAppender, despite the fact that it was added for Kafka producers and consumers generally.
Kerberos support was only added to the KafkaLog4jAppender in version 0.10.
From a quick glance at the code, it looks like it would be straightforward to create a custom appender that extends the KafkaLog4jAppender and adds the necessary bits for Kerberos support.

Kafka is Open Source, why don't you just check the code and work out which properties are related to Kerberos and how they are used?
A quick look at "trunk" i.e. https://github.com/apache/kafka/blob/trunk/log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java hints that you must attach a JAAS config file, and specify which "context" to use in that file, so that the Kafka client can use the standard Java Security libraries. And optionally provide a custom Kerberos config file if it is not in the standard location (i.e. /etc/krb5.conf).
So in the end it's a matter of understanding JAAS -- and finding out which Kafka and/or Log4J properties to set.
And with Google, a tutorial is always a few clicks away -- e.g. that kerberized_kafka post.

JAX-WS logging & timestamp

I am using JAX-WS in WebLogic and I was able to enable the message logging by using following parameters (as specified at https://metro.java.net/guide/ch02.html#logging):
com.sun.xml.ws.transport.http.HttpAdapter.dump=true
com.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true
The log entries however do not contain any timestamps so I have almost no information about when each message was received/sent.
Is there any way to add timestamps to the resulting log entries?

I guess you need to configure log messages format:
java.util.logging.SimpleFormatter.format="%4$s: %5$s [%1$tc]%n
More about configuration options could be found here:
http://docs.oracle.com/javase/7/docs/api/java/util/logging/SimpleFormatter.html

I have finally found a solution for this.
The only thing that must be done is to redirect stdout logging to WebLogic logging system by checking “Redirect stdout logging enabled” in Advanced logging section of your managed server’s configuration (e.g. Logging -> General -> Advanced).
This will add timestamps to the SOAP message logs.