Develop Reference

How to efficiently index logs to elastic search

I'm developing a web application in which i'll upload a log file, the file will be read and classified based on logger levels (INFO,ERROR,WARNING etc..). I need to index those logs to elasticsearch using java high level rest client api.
Currently i'm creating one index for a classname(Logs will contain classname) and store those class logs in that particular index. I feel this approach is wont be nice in some cases if a log file contains logs from 100 different classes, i'll be creating 100 indices for it and storing those logs.
Is there any efficient way of indexing logs to elasticsearch?How to determine indices in my case?
Sample log:
02-Jul-2021|10:03:10.040|INFO|[main]|org.apache.catalina.startup.VersionLoggerListener.log|Server
built: Jun 11 2021 13:32:01 UTC

Here is the practice that we use in some of our java based stack and it has many privileges for the usage of Apache Kafka as middle data pipe line and logstash as data ingestion pipeline.
First you need to remove default providers for logs in your spring boot application inside your pom.xml file, Which are Logback and perhaps Log-classic then you need to add log4j2 as new log provider and adding Kafka appender. After adding dependencies you need xml configuration file where you can add your Kafka appender configurations. By default you need to locate your configuration file in resource path of your project and name it as "log4j2.xml".
You can find many others Log4j2 appenders like Cassandra or Failover appenders and add them beside your Kafka appender inside your configuration file. You can find an applicable and correct example in below.
<!--excluding logback -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--adding log4j2 and kafka appender-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
Kafka appender configuration
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" name="kafka-appender" packages="Citydi.ElasticDemo">
<Appenders>
<Kafka name="kafkaLogAppender" topic="Second-Topic">
<JSONLayout />
<Property name="bootstrap.servers">localhost:9092</Property>
<MarkerFilter marker="Recorder" onMatch="DENY" onMismatch="ACCEPT"/>
</Kafka>
<Console name="stdout" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} stdout %highlight{%-5p} [%-7t] %F:%L - %m%n"/>
<MarkerFilter marker="Recorder" onMatch="DENY" onMismatch="ACCEPT"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="kafkaLogAppender"/>
<AppenderRef ref="stdout"/>
</Root>
<Logger name="org.apache.kafka" level="warn" />
</Loggers>
</Configuration>
Activating Zookeeper broker
./zookeeper-server-start.sh ../config/zookeeper.properties
Activating Kafka broker
./kafka-server-start.sh ../config/server.properties
Create Topic
./kafka-topics.sh --create --topic test-topic -zookeeper localhost:2181 --replication-factor 1 --partitions 4
Active consumer of the created topic
./kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic
Then add the log appender for created topic for consuming logs(This one is up to you) and after that create a Logstash pipeline such as below configuration as ingest your logs into your desired index in elastic .
input {
kafka{
group_id => "35834"
topics => ["yourtopicname"]
bootstrap_servers => "localhost:9092"
codec => json
}
}
filter {
}
output {
file {
path => "C:\somedirectory"
}
elasticsearch {
hosts => ["localhost:9200"]
document_type => "_doc"
index => "yourindexname"
}
stdout { codec => rubydebug
}
}

Disable Azure service bus printing logs in cloud watch

I was getting following error message in aws lambda logs :
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
so i added maven depdenecies as :
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.21</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.21</version>
</dependency>
But on adding slfj4j dependencies unwated logs from azure service bus are also getting printed now eg.
[ReactorThread1288184d-400a-4928-b174-b819c8bd9ee1] INFO com.microsoft.azure.servicebus.primitives.MessagingFactory - starting reactor instance.
my log4j.xml looks like this :
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.amazonaws.services.lambda.runtime.log4j2">
<Appenders>
<Lambda name="Lambda">
<PatternLayout>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n</pattern>
</PatternLayout>
</Lambda>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="Lambda" />
</Root>
</Loggers>
</Configuration>
How can i disable these logs getting printed ?

I think the problem is related with the logging dependencies of your project.
On one hand, you have AWS Lambda over Log4j2, which uses Log4j2 for logging. On the other hand you have Azure Service Bus, which uses the SLF4J API facade for logging. You need to configure your system to support both logging approaches.
First of all, you need the following dependencies in your pom.xml file:
<dependencies>
...
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-log4j2</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.13.2</version>
</dependency>
....
</dependencies>
The first dependency allows you to emit log traces to AWS Lambda.
The last one is the Log4j2 SLF4J java bridge, necessary for Azure Service Bus logging.
Please, also remove the dependency slf4j-simple associated with the group org.slf4j, it will be no necessary.
With this dependencies in place, please, include the following line:
<Logger name="com.microsoft.azure.servicebus" level="OFF"/>
In your XML configuration file (by convention, it is better to name your Log4j2 configuration file like log4j2.xml instead of log4j.xml, more appropriate for the old Log4j library version).
Your log4j2.xml should looks something like the following:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.amazonaws.services.lambda.runtime.log4j2">
<Appenders>
<Lambda name="Lambda">
<PatternLayout>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n</pattern>
</PatternLayout>
</Lambda>
</Appenders>
<Loggers>
<Logger name="com.microsoft.azure.servicebus" level="OFF"/>
<Root level="info">
<AppenderRef ref="Lambda" />
</Root>
</Loggers>
</Configuration>

Firstly, this is not an error. The message means that there is no logger implementation present in the classpath and hence it is defaulting to NOP (No-operation) log. And as SLF4J is an abstraction of different logging frameworks, you also need to include specific logging framework in your classpath apart from only having SLF4J. Since you are trying to use Lambda appender, you need to add aws-lambda-java-log4j2 to your classpath. This will bring in required SLF4J dependencies also.
Latest version as of May 05, 2020:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-log4j2</artifactId>
<version>1.2.0</version>
</dependency>

How to make java.util.logging send logs to Logback?

I'm working on an app that logs using the slf4j api:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
...
private static final Logger LOG = LoggerFactory.getLogger(FreemarkerEmailPreviewGenerator.class);
...
LOG.error("Error generating email preview", e);
(Code above posted to show classes and packages in use, but pretty standard stuff.)
We use logback configured as follows:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>[%d{HH:mm:ss.SSS}] [%thread] [%-5level %logger{26} - %msg]%n
</pattern>
</encoder>
</appender>
<root>
<level value="debug" />
<appender-ref ref="STDOUT" />
</root>
</configuration>
Some of our code makes use of 3rd party libraries that logs with java.util.logging - specifically freemarker. As you can see from the following console log entries, both logback and j.u.l are logging to the console, but they are not using the same config (the logback entries use our pattern, the j.u.l ones don't)
[12:24:38.842] [pool-2-thread-19] [INFO u.o.n.r.l.s.e.t.TemplateLoaderFromService - Finding template workflow/mail/templates/common/workflow-macros.ftl]
[12:24:38.859] [pool-2-thread-19] [INFO u.o.n.r.l.s.e.t.TemplateLoaderFromService - Loaded template workflow/mail/templates/common/workflow-macros.ftl as /workflow/mail/templates/common/workflow-macros.ftl from RegistryMailTemplateService.]
11-Jan-2017 12:24:38 freemarker.log.JDK14LoggerFactory$JDK14Logger error
SEVERE:
Expression domainContact is undefined on line 9, column 74 in workflow/mail/templates/common/workflow-macros.ftl.
The problematic instruction:
----------
==> ${domainContact.name} [on line 9, column 72 in workflow/mail/templates/common/workflow-macros.ftl]
Is it possible to make j.u.l logging use the logback config so that we have a single consistent logging config for the whole app?

Your application needs to have the following jars:
Application -> Freemarker -> java.util.logging -> SLF4J Api: jul-to-slf4j.jar
Application -> SLF4J API: slf4j-api.jar
SLF4J API -> logback: logback-classic.jar and logback-core.jar
Since your application already contains slf4j-api.jar and logback-classic.jar, you probably only have to add the jul-to-slf4j.jar
If you use maven:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
<version>1.7.22</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.1.8</version>
</dependency>
logback classic will transitively add logback-core and slf4j-api

Log4j2 configuration - No log4j2 configuration file found

Lately I decided to learn how to use the log4j2 logger. I downloaded required jar files, created library, xml comfiguration file and tried to use it. Unfortunately i get this statement in console (Eclipse) :
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
This is my testing class code:
package log4j.test;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class Log4jTest
{
static final Logger logger = LogManager.getLogger(Logger.class.getName());
public static void main(String[] args) {
logger.trace("trace");
logger.debug("debug");
logger.info("info");
logger.warn("warn");
logger.error("error");
logger.fatal("fatal");
}
}
And my xml config file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration package="log4j.test"
status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Logger name="log4j.test.Log4jTest" level="trace">
<AppenderRef ref="Console"/>
</Logger>
<Root level="trace">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
I tried also with xml without <Logger> tags, and package specification and in various folders/packages directories, but it didn't help. Now my log4j2.xml file is located directly in project folder in eclipse.

Is this a simple eclipse java project without maven etc? In that case you will need to put the log4j2.xml file under src folder in order to be able to find it on the classpath.
If you use maven put it under src/main/resources or src/test/resources

You need to choose one of the following solutions:
Put the log4j2.xml file in resource directory in your project so the log4j will locate files under class path.
Use system property -Dlog4j.configurationFile=file:/path/to/file/log4j2.xml

Was following the documentations - Apache Log4j2 Configuratoin and Apache Log4j2 Maven in configuring log4j2 with yaml. As per the documentation, the following maven dependencies are required:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.8.1</version>
</dependency>
and
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.8.6</version>
</dependency>
Just adding these didn't pick the configuration and always gave error. The way of debugging configuration by adding -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE helped in seeing the logs. Later had to download the source using Maven and debugging helped in understanding the depended classes of log4j2. They are listed in org.apache.logging.log4j.core.config.yaml.YamlConfigurationFactory:
com.fasterxml.jackson.databind.ObjectMapper
com.fasterxml.jackson.databind.JsonNode
com.fasterxml.jackson.core.JsonParser
com.fasterxml.jackson.dataformat.yaml.YAMLFactory
Adding dependency mapping for jackson-dataformat-yaml will not have the first two classes. Hence, add the jackson-databind dependency to get yaml configuration working:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.8.6</version>
</dependency>
You may add the version by referring to the Test Dependencies section of log4j-api version item from MVN Repository. E.g. for 2.8.1 version of log4j-api, refer this link and locate the jackson-databind version.
Moreover, you can use the below Java code to check if the classes are available in the classpath:
System.out.println(ClassLoader.getSystemResource("log4j2.yml")); //Check if file is available in CP
ClassLoader cl = Thread.currentThread().getContextClassLoader(); //Code as in log4j2 API. Version: 2.8.1
String [] classes = {"com.fasterxml.jackson.databind.ObjectMapper",
"com.fasterxml.jackson.databind.JsonNode",
"com.fasterxml.jackson.core.JsonParser",
"com.fasterxml.jackson.dataformat.yaml.YAMLFactory"};
for(String className : classes) {
cl.loadClass(className);
}

Eclipse will never see a file until you force a refresh of the IDE. Its a feature! So you can put the file all over the project and Eclipse will ignore it completely and throw these errors. Hit refresh in Eclipse project view and then it works.

Additionally to what Tomasz W wrote, by starting your application you could use settings:
-Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE
to get most of configuration problems.
For details see Log4j2 FAQ: How do I debug my configuration?

In my case I had to put it in the bin folder of my project even the fact that my classpath is set to the src folder. I have no idea why, but it's worth a try.

log4j-test.xml => name of the configuration file if placed in /test/resources
#TestPropertySource(properties = {
"log4j.configurationFile","log4j-test.xml"
})
#CucumberContextConfiguration
#ContextConfiguration(classes = TestConfiguration.class)
public class Steps{
private Logger log = LogManager.getLogger(Steps.class);
//implementation
}

SLF4J writes to file, but not to eclipse console

I am using slf4j for logging and Glassfish as app server.
My logback.xml
<configuration debug="true" scan="true">
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>C:\glassfish4\glassfish\logs\log.log</file>
<encoder>
<Pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{52} - %msg%n</Pattern>
</encoder>
</appender>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<Pattern>%msg%n</Pattern>
</encoder>
</appender>
<logger level="DEBUG" name="ru.vmakarenko"/>
<root>
<level value="INFO"/>
<appender-ref ref="FILE"/>
<appender-ref ref="STDOUT"/>
</root>
</configuration>
My logging.properties
javax.enterprise.system.tools.admin.level=INFO
handlers=org.slf4j.bridge.SLF4JBridgeHandler
java.util.logging.ConsoleHandler.formatter=com.sun.enterprise.server.logging.UniformLogFormatter
javax.enterprise.system.ssl.security.level=INFO
org.apache.jasper.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.flushFrequency=1
org.eclipse.persistence.session.level=INFO
javax.enterprise.system.tools.backup.level=INFO
javax.enterprise.resource.corba.level=INFO
javax.enterprise.resource.webcontainer.jsf.resource.level=INFO
javax.enterprise.system.core.classloading.level=INFO
javax.enterprise.resource.jta.level=INFO
java.util.logging.ConsoleHandler.level=FINEST
com.sun.enterprise.server.logging.GFFileHandler.file=${com.sun.aas.instanceRoot}/logs/server.log
javax.enterprise.system.webservices.saaj.level=INFO
javax.enterprise.system.tools.deployment.level=INFO
javax.enterprise.system.container.ejb.level=INFO
org.glassfish.naming.level=INFO
javax.enterprise.system.core.transaction.level=INFO
org.apache.catalina.level=INFO
javax.enterprise.resource.webcontainer.jsf.lifecycle.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.rotationTimelimitInMinutes=0
javax.enterprise.resource.webcontainer.jsf.config.level=INFO
javax.enterprise.system.container.ejb.mdb.level=INFO
javax.enterprise.resource.webcontainer.jsf.timing.level=INFO
javax.enterprise.system.core.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.rotationOnDateChange=false
com.sun.enterprise.server.logging.GFFileHandler.excludeFields=
org.apache.coyote.level=INFO
ShoalLogger.level=CONFIG
javax.level=INFO
javax.enterprise.resource.webcontainer.jsf.taglib.level=INFO
java.util.logging.FileHandler.limit=50000
javax.enterprise.system.webservices.rpc.level=INFO
javax.enterprise.resource.javamail.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.logtoConsole=true
javax.enterprise.system.container.web.level=INFO
javax.enterprise.resource.webcontainer.jsf.facelets.level=INFO
javax.enterprise.system.util.level=INFO
javax.enterprise.resource.resourceadapter.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.level=ALL
javax.org.glassfish.persistence.level=INFO
javax.enterprise.resource.webcontainer.jsf.context.level=INFO
javax.enterprise.resource.webcontainer.jsf.application.level=INFO
javax.enterprise.resource.jms.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.multiLineMode=true
com.sun.enterprise.server.logging.GFFileHandler.rotationLimitInBytes=2000000
javax.enterprise.system.core.config.level=INFO
org.jvnet.hk2.osgiadapter.level=INFO
javax.enterprise.system.level=INFO
javax.enterprise.system.core.security.level=INFO
javax.enterprise.system.container.cmp.level=INFO
java.util.logging.FileHandler.pattern=%h/java%u.log
com.sun.enterprise.server.logging.SyslogHandler.useSystemLogging=false
javax.enterprise.resource.sqltrace.level=FINE
javax.enterprise.resource.webcontainer.jsf.renderkit.level=INFO
handlerServices=com.sun.enterprise.server.logging.GFFileHandler
javax.enterprise.system.webservices.registry.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.alarms=false
javax.enterprise.system.core.selfmanagement.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.formatter=com.sun.enterprise.server.logging.UniformLogFormatter
.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.maxHistoryFiles=0
log4j.logger.org.hibernate.validator.util.Version=warn
java.util.logging.FileHandler.count=1
javax.enterprise.resource.webcontainer.jsf.managedbean.level=INFO
org.glassfish.admingui.level=INFO
javax.enterprise.resource.jdo.level=INFO
com.sun.enterprise.server.logging.GFFileHandler.retainErrorsStasticsForHours=0
And domain.xml (jvm-options)
<jvm-options>-Djava.util.logging.config.file=file:///${com.sun.aas.instanceRoot}/config/logging.properties</jvm-options>
<jvm-options>-Dlogback.configurationFile=file:///${com.sun.aas.instanceRoot}/config/logback.xml</jvm-options>
So i get the C:\glassfish4\glassfish\logs\log.log file with all logged stuff I need.
But I get nothing at Eclipse Console. I have Glassfish Tools installed, and server is managed from eclipse. What is my mistake, how can i redirect output both to file and console?
Also, when I run maven, I get
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Maybe it's part of the problem?

There seems to be a bug in the console processing of the Eclipse GlassFish Tools plugin, check this thread and this related bug report.
I have a similar problem - I can't get the JUL logging to work properly. If I set the logging level of my application class to anything lower than CONFIG, nothing is logged to the Eclipse console, but it is also still logged to server.log. Also, if I start the GlassFish server outside of Eclipse, I do get all my FINE level messages printed on the terminal.
UPDATE:
I could finally make log entries with a level below CONFIG work by using a thin wrapper which for example calls Logger.logp(Level.FINE, null, null, msg). GlassFish Tools apparently can't handle the format produced by com.sun.enterprise.server.logging.ODLLogFormatter for levels FINE and lower, which includes additional CLASSNAME and METHODNAME fields, so setting the source explicitly to null did the trick.

If you are using log4j, then declares slf4j log4j binding too in your pom.xml file :
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.6</version>
</dependency>
or if you are not using log4j then make sure you have all the below dependency into your pom.xml file :
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.6.6</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>1.0.7</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.0.7</version>
</dependency>