Develop Reference

Log4j2 Syslog appender is not writing the 1st message to syslog after the syslog service is restarted

We have configured our application to write some specific log messages to System's Syslog file using the Syslog appender of Log4j2. No issue in writing the Syslog to the file. But when the syslog service is restarted, the first log message is not written to the syslog. The subsequent messages are written.
Enabled debug logs of Log4j, no exception is seen while writing 1st message to syslog after the restart. But for the subsequent request, the following messages were captured in the Log4j2 log.
2022-01-27 18:07:40,120 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Reconnecting localhost/127.0.0.1:514
2022-01-27 18:07:40,121 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Creating socket localhost/127.0.0.1:514
2022-01-27 18:07:40,122 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Closing SocketOutputStream java.net.SocketOutputStream#1a769d7
2022-01-27 18:07:40,122 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Connection to localhost:514 reestablished: Socket[addr=localhost/127.0.0.1,port=514,localport=57852]
I took threaddump and checked whether the Reconnector thread is running but no such exists in the threaddump. I am clueless here, any help on finding the reason for missing the message would be helpful.
Environment details:
CentOS 7.9 + RSyslog Service,
Application deployed in Tomcat and running on Java 11,
Log4j2 version is 2.17.1

This is due to the way plain text TCP syslog works. Check out this post for further information.
This "bug" exists, since version 8.1901 and newer.
The only way you can fix this - as far as i know - is to send the messages over the RELP protocol. See omrelp module.

Stacktrace does not print in Glassfish 4.1 Cluster log

Doing our first cluster setup on Glassfish (4.1). Application(EAR) level logs (ex printing a stacktrace) don't seem to reach the server.log in (GF-dir)/domains//logs/server.log or (GF-dir)/nodes/(node-name)/(instance-name)/server.log
(There is no cluster.log as stated in documentation)
We didn't change any of the default logging options in logging.properties.
The current logs only show cluster and instance related information.

I had similar probrem.
server.log is not output after
MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is EMBEDDED, connection mode is TCP
If the similar message is output in
(GF-dir)/nodes/(node-name)/(instance-name)/imq/instances/(clustername instancename)/log/log.txt
Invalid broker address for this broker to run in cluster: Loopback IP address is not allowed in broker address mq://127.0.1.1:27676/?instName=c1i1&brokerSessionUID=2263708429127912192&ha=false for cluster
Fixed the node server's /etc/hosts file to not to use the loopback to the node server.

It seems that the node stops to output to the log file when JMS broker is configured as embedded (the default). I tried to change it to LOCAL (in the JMS configuration, plus turned on JMS availability service with defaults), and the log file is now output as expected.
This is not a solution per se, as I started to get other errors due to the change in JMS configuration. But I'm able to see error messages from my application which I was not able to see before.

Logstash syslog input from docker container

I have a java application running in a docker container on a docker host. The application uses log4j for logging and logs to syslog. I want to be able to send my syslog logs to logstash.
I changed the configurations in rsyslog config file to :
*.* ##<logstash host ip>:514
and I have in my logstash config file for syslog:
input {
syslog {
type => syslog
port => 514
}
}
and in logstash logs I got errors saying syslog tcp listener died and
exception=>#<Errno::EACCES: Permission denied - bind(2)
I thought I should probably specify where the host is in logstash configs and added the ip address of my dockerhost + port to the config file but I still get the same errors.
How can I tell logstash to look at the docker container on dockerhost for logs? am I missing a component here?
Thanks.

You need to run the process as root. normal users (ie non root) cant bind to ports less than 1024 without some setuid trickery

How to design log structure for a Java application

Am new to java, I know Java has Log4J, logback etc for logging purposes. My question is more around how many log files should we have in a application. Should it be per thread, per group of threads, process, exception etc. In our application there is a possibility of having a large number of threads and am thinking about cons of having log file per thread. Are there best practices for logging in applications having huge number of threads .
Thanks in advance!

1 log for messages - Call it SystemOut.log
1 log for stack traces - Call it SystemErr.log
1 log for traces - Call it Trace.log
1 log for native stdout - Call it nativeStdOut.log
1 log for native stderr - Call it nativeStdErr.log
Have a config panel that sets:
maxSize
maxCount
When a log hits max size, starting rolling them upto maxCount and append a timestamp to the rolled filename.

I think that good solution would be to name your threads and write logs together with the name of thread in which log occurred. Thanks to that you will be able to both analysis logs separately for each thread or analysis all logs together.

Typically there is one log file per application (process) -- rarely for Thread and never be Exception. Sometimes this log file is split into various different log levels: debug messages in one bucket, information in another, warnings/errors in a third. This makes it easy to watch for errors by only looking at the warning-and-more-critical file.
log4j has a configuration file in which you can route certain messages to certain files using different criteria. Here's a sample of the log4j properties file:
# default is WARN and above going to the appender "logfile" and STDOUT
log4j.rootLogger=WARN, logfile, stdout
# write to ui.log for a day and then move to .yyyy-MM-dd suffix
log4j.appender.logfile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logfile.File=ui.log
log4j.appender.logfile.Append=true
log4j.appender.logfile.DatePattern='.'yyyy-MM-dd
# but we log information message from classes in the package com.mprew.be
log4j.logger.com.mprew.be=INFO
log4j, and custom loggers, decorate each time with a Class name, priority level, date/time, etc.. For example:
# date time priority Class-name Log message
2012-03-26 00:55:39,545 [INFO] CmsClientTask Content is up-to-date
Typically exceptions are written out as multiple lines so you can get the entire stack-trace.
2012-03-26 01:55:35,777 [INFO] ExceptionInterceptor Reporting problem to customer
org.springframework.NoSuchRequestException: No request handling method
at com.ui.base.BaseController.invokeNamedHandler(BaseController.java:240)
at com.ui.base.BaseController.handleRequestInternal(BaseController.java:100)
at com.ui.base.CoreServices.handleRequest(CoreServicesController.java:147)
...
In our distributed system, we route all logs from all of the system to 2 servers which write a debug, info, and warning logs. Along with the date/time, class name, priority, and message, the log messages also have the hostname and a specific log token so we can easily identify classes of problems. The following is on one line:
2012-03-26 00:00:00.045 UTC INFO FE8 TrayController TRAY_CLIENT_LOOKUP
message=created unknown client with partner STW9
Then we can easily grep for specific issues.
Hope this helps.

Java: Display request of an HttpURLConnection before sending

I want to make some API calls to a server using HttpURLConnection. But the requests are not successful, returning:
<error>
<http_status>400 Bad Request</http_status>
<message>Unexpected request Content-Type header ''. Expecting 'application/x-www-form-urlencoded'.</message>
</error>
So I want to check what the "real" content is that is being sent to the server. By real content I mean the exact HTTP request.
Any ideas how I can see this?
Edit:
Based on the first answers here I should clarify my problem: I want to avoid using an external program like HTTP sniffer or anything and I was hoping that there is a function or a property or whatever that holds the information I am looking for.
If that is not the case, does someone know if this information can be manually rebuilt (for example by calling several functions like getRequestMethod(), etc.)
I am facing this problem kinda often so that it's worth the effort to build such functionality myself. Just need to know how :)

You can put the HttpURLConnection in debug mode by enabling java.logging with
-Djava.util.logging.config.file=logging.properties
and put in logging.properties (by default in JRE_HOME\lib) the following property
sun.net.www.protocol.http.HttpURLConnection.level = ALL

tcpdump will work, but it can be hard to make it do what you want. NetCat is more user-friendly (here's the project page: http://netcat.sourceforge.net/ - most Unix platforms already include it).
nc -l 9999
This will listen on TCP port 9999, and when an HTTP client connects, it'll print out the full text of the request.

The accepted solution did not work for me. But what did was
static {
ConsoleHandler handler = new ConsoleHandler();
handler.setLevel(Level.ALL);
Logger log = LogManager.getLogManager().getLogger("");
log.addHandler(handler);
log.setLevel(Level.ALL);
System.setProperty("javax.net.debug","all");
}

Use something like tcpdump, which can dump the actual network packets that are emitted or received by your computer.

On JDK 11 I was able to log all the http connections, setting java.util.logging.ConsoleHandler.level to FINEST and adding the following lines in the file logging.properties which is by default in %JAVA_HOME%/conf:
sun.net.www.protocol.http.HttpURLConnection.level = FINEST
sun.net.www.protocol.http.HttpURLConnection.handlers = java.util.logging.ConsoleHandler