I am using a Java micro-service architecture in my application and generating separate log files for each micro-service.
I am using ELK stack approach to visualize the logs in Kibana, but the problem is whatever the fields that I'm getting from Elastic Search that are related to server logs fields. some example fields are #timestamp,#version,#path,#version.keyword,#host.
i want to customize this fields by adding some fields like customerId,txn-Id,mobile no so that we can analyze the data easily.
I'm using org.apache.logging.log4j2 to write the logs. Can I set above fields (customerId,txn-Id,mobile) to log files? And then Elastic will store these fields with the above default fields and then these custom fields should available in a Kibana dashboard. Is this possible?
It's definitely possible to do that. I've not done it with the log4j2 stack (I have with slf4j/logback), but the basic approach is:
set those fields in the Mapped Diagnostic Context (I'm fairly sure log4j2 supports that)
use a log appender which logs to logstash-structured JSON
configure filebeat to ship the JSON logs
if filebeat is shipping to logstash, you'll need to configure logstash to pass those preformatted JSON logs directly to elasticsearch
It is definitely possible. I am doing that now with my applications. However, the output looks a bit different from yours. The basic guide for doing this can be found at Logging in the Cloud on the Log4j2 web site.
The "normal" log view looks very similar to what you would see when logging to a file.
However, if you select a message you can see the individual fieds.
The Log4j2 configuration uses a TCP Socket appender that is configured to write to a cluster of Logstash servers that use a single DNS entry and to use the Gelf layout.
You can also use MapMessages to capture individual data elements and log them. While this currently works it is slightly cumbersome so I have recently committed improvements that will be available in Log4j 2.15.0.
It is important to note that the Logging in the Cloud page briefly mentions storing your logging configuration in Spring Cloud Config. If you want to have a common base configuration while allowing apps to do some customization this works very, very well. However, The Gelf, Json Template Layout and TCP Appender are all independent from that and can be used without Spring Boot.
Related
I have a spring boot application and I want to configure log tracing to all the application, I added the setup for **datadog Agent ** as in in the documentation: https://docs.datadoghq.com/tracing/setup_overview/setup/java/?tabs=containers and I can see trace ids and spanIds associated with each request, there are other options such as Spring cloud Sleuth to add tracing and span ids as well:
https://spring.io/projects/spring-cloud-sleuth
and also in the datadog documentation, there is a way for connecting Java logs and tracing in https://docs.datadoghq.com/tracing/connect_logs_and_traces/java/?tabs=log4j2
I have a confusion with this three approaches and which one of them I can use for the tracing purpose, with spring sleuth I know that is easier to generate span a trace ids, but not sure if them are sent to datadog or I need to configure something additional.
Also the connecting Java logs and tracing vs datadog agent its not clear for me.
I am new with this topic and it is not clear for me how can I implement tracing for all the process included in a request.
I have a SpringBoot 2 app that uses using Spring Data Couchbase.
I have this message on the logs every minute
2019-11-12 13:48:48,924 WARN : gid: trace= span= [cb-orphan-1] c.c.c.c.t.DefaultOrphanResponseReporter Orphan responses observed: [{"top":[{"r":"10.120.93.220:8092","s":"view","c":"5BE128F6F96A4D28/FFFFFFFFDA2C8C52","l":"10.125.216.233:49893"}],"service":"view","count":1}]
That is from the new Response Time Observability feature underlying the Java SDK.
It would seem to indicate that you have view requests which are timing out, but eventually received later, but I have no views defined in Couchbase DB
I would like to know if it is possible to disable OrphanResponseLogReporter via YML file config in a SpringBoot app. , setting the logIntervalNanos to 0
No, unfortunately, you cannot do it. Only a subset of Couchbase's configuration properties is supported in the application.yml, namely the ones present in the CouchbaseProperties.java class.
You could although use an environment variable: com.couchbase.orphanResponseReportingEnabled=false. It is independent of Spring, it's read directly by Couchbase SDK.
Edit:
As a workaround, you can set logging level in the application.yml:
logging.level.com.couchbase.client.core.tracing.DefaultOrphanResponseReporter: ERROR
I'm trying to essentially configure logging in three places, independently. Ideally, each component that is logging is fully unaware of the others. I'd like to use logback for this, as it seems to have some decent features and performance.
Here are the places from which I would like to log from and to:
Tomcat (7) should log to ${catalina_home}/logs/catalina.out, and should only log Tomcat events (deployments, server startup, etc)
A web application hosted in Tomcat should log to ${catalina_home}/logs/application.log, and should only log application things, like results of request validations or errors
A library that is included in the web application should log to ${catalina_home}/logs/library.log, and should only log things specific to that library, like time it takes to interact with some other web-service or library-specific errors
I know this is probably not the way it would work, but I would think I need to have a logback.xml file for each concern. Actually, I have created them and added to the classpath such that I get a "logback.xml occurs multiple times on the classpath" error.
I can see why I would need to consolidat my application and my library logback configuration to a single logback.xml file, but how would I keep the container logging config separate from the application+library logging config? Adding a logback config file to my application, as well as logback enabling Tomcat as described here, still yields a "multiple logback.xml" error. And, Chapter 9 of the logback user manual, which talks about separation of logging, doesn't really show how to separate the container and applications (just multiple applications), unless I am missing something there.
We need to log all incoming SOAP requests, preferably by persisting to a DB as we have identifying properties that we'd like to associate with it. Is there any way of getting the raw XML data in Spring?
I suggest you take a look at the source for SoapEnvelopeLoggingInterceptor and/or PayloadLoggingInterceptor. You can probably modify this to include what you want.
Another solution could be to put a servlet Filter in front of everything that puts the identifying properties into the MDC (assuming you are using SLF4J and/or Log4j/Logback) that way you could configure a jdbc backed Appender which logs to the database.
Am using Standard apache logging (org.apache.log4j.logging )
Currently, taking the data to be logged manually, and publishing in to Apache Active MQ.
Is it possible to configure the logging output to publish directly in to Active MQ??
This might sound stupid, but since both are from Apache, I have a doubt that whether, it has any implicit support, which I could not grab it.
log4j provides JMSAppender out of the box. It allows publishing logging events to JMS Topic.
For configuration specific to ActiveMQ please check the documentation - How do I use log4j JMS appender with ActiveMQ
Not sure if you were looking for log4j-1.x or log4j-2.0, but here are the links for log4j-2.0:
http://logging.apache.org/log4j/2.x/manual/appenders.html#JMSQueueAppender
http://logging.apache.org/log4j/2.x/manual/appenders.html#JMSTopicAppender