We got a couple spring boot applications in k8s that write both application log and tomcat access log to stdout.
When the log throughput is really high (either caused by amount of requests or amount of applciation logs) then it sometimes happens that log lines get interrupted.
In our case this looks like this:
[04/Aug/2021:13:39:27 +0200] - "GET /some/api/path?listWithIds=22838de1,e38e2021-08-04 13:39:26.774 ERROR 8 --- [ SomeThread-1] a.b.c.foo.bar.FooBarClass : Oh no, some error occured
e7fb,cd089756,1b6248ee HTTP/1.1" 200 (1 ms)
desired state:
[04/Aug/2021:13:39:27 +0200] - "GET /some/api/path?listWithIds=22838de1,e38ee7fb,cd089756,1b6248ee HTTP/1.1" 200 (1 ms)
2021-08-04 13:39:26.774 ERROR 8 --- [ SomeThread-1] a.b.c.foo.bar.FooBarClass : Oh no, some error occured
is there some way to prevent this?
maybe a tomcat, java or spring-boot setting?
or a setting on a container level to make sure that each line is buffered correctly
System.out had better be thread-safe, but that doesn't mean it won't interleave text when multiple threads write to it. Writing both application logs and HTTP server logs to the same stream seems like a mistake to me for at least this reason, but others as well.
If you want to aggregate logs together, using a character stream is not the way to do it. Instead, you need to use a logging framework that understands separate log-events which it can write coherently to that aggregate destination.
You may need to write your own AccessLogValvesubclass which uses your logging framework instead of writing directly to a stream.
Related
I'm using Cassandra on a CentOS machine. After it failed some time ago, I restarted it via
sudo service cassandra restart
and started getting Connection refused error all over the place - I couldn't even run nodetool status without running into that issue.
After some digging and subsequent restarts, I noticed in the debug.log that the startup sequence gets stuck at the following:
INFO [main] 2018-04-03 09:40:15,156 ColumnFamilyStore.java:389 - Initializing system.IndexInfo
INFO [SSTableBatchOpen:1] 2018-04-03 09:40:15,851 BufferPool.java:226 - Global buffer pool is enabled, when pool is exahusted (max is 512 mb) it will allocate on heap
DEBUG [SSTableBatchOpen:1] 2018-04-03 09:40:15,873 SSTableReader.java:479 - Opening <path>/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-300-big (58 bytes)
DEBUG [SSTableBatchOpen:2] 2018-04-03 09:40:15,873 SSTableReader.java:479 - Opening <path>3/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-301-big (59 bytes)
DEBUG [SSTableBatchOpen:3] 2018-04-03 09:40:15,873 SSTableReader.java:479 - Opening <path>/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-299-big (302 bytes)
Opening these files should take about a second - it's stuck on it for ages (as in, it never moved beyond this point). I suspect that some of the files involved must be corrupted (although I'm surprised the Java doesn't catch some sort of an exception here).
What should I do? If I delete these folders, would that result in me losing data? What other diagnostics can I run to establish the source of the problem? For the record, any sort of nodetool command exits with a "Connection Refused" error.
Version numbers:
Cassandra: 3.0.9
Java: 1.8.0_162
CentOS: 6.9
Thanks for help!
It turns out that the issue was in the files involved becoming corrupted - running touch on all the files in the data folder (Data, CompressionInfo, Index, etc.) and erasing the post-crash commitlogs allowed the Cassandra to get up. A few hundred datarows were lost (probably due to me deleting the commitlogs), but at least the database is back up!
Jmeter I keep getting: 'java.net.ConnectException: Connection timed out: connect' ?
I have created a load test which tests a specific url at 200 users
when running the load test for x1 iteration i keep seem to be getting: Connection timed outs?
I have made the following changes listed here: https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
But the issue is still there:
[1
You most probably don't have access to the target host from where you test.
Did you configure proxy as your web browser is probably configured
http://jmeter.apache.org/usermanual/get-started.html#proxy_server
But if failure is partial, then you server might be overloaded and rejecting some requests.
My expectation is that "problematic" requests are simply not able to finish in 20 seconds (most probably you have modified Connect timeout and set this value in HTTP Request or HTTP Request Defaults)
20 seconds looks like a long response time to me so your finding indicates application under test performance problem.
Going forward if you would like to see more "human readable" message in the results file switch to Duration Assertion instead of setting timeouts on protocol level
See How to Use JMeter Assertions in Three Easy Steps article for more information on conditionally failing JMeter requests.
Please check client configuration from where you are running your tests. It might be like your client system is not able to handle 200 threads. Do the test iteration wise means try with 10, 50 , 70 and so on. Check from which iteration onwards you are getting the error. It is also advisable not to include the listeners during load testing.
Please check the best practices for load testing using jmeter.
http://jmeter.apache.org/usermanual/best-practices.html
My Problem is:
Upload 10000 Record from Excel To my Database. The Excel Sheet has 10000 Rows and 70 to 100 Column . We Store the Value in 6 Mapping Table using Hibernate cascade.
I call the method using ajax method. Due to inserting large amount of data. it return timeout error(502 (Proxy Error) OR 504 (Gateway Error)).
I am using AWS Services. Is any configuration mistake.. please help
Thanks in Advance
How You Might See a 502 Bad Gateway Error
Based on your web server, you might see a different 502 error. These all mean the same thing, it is only their naming conventions that differ. Here are a few examples of what you might see:
“502 Bad Gateway”
“HTTP Error 502 – Bad Gateway”
“502 Service Temporarily Overloaded”
“Error 502”
“502 Proxy Error”
“HTTP 502”
“502 Bad Gateway NGINX”
You can see in greater detail what the error specifically entails by going to your web server’s error log file. All error / diagnostic information is stored in this file making it a valuable resource to check when you need more details about a particular error. You can locate this file in Apache by going to /var/log/apache2/error.log and in Nginx by going to /var/log/nginx/error.log.
How to solve 502 errors?
Check if your FQDN is resolving correctly by using our DNS test tool.
Verify if your server is reachable by using a ping test or trace-route.
Check your firewall logs if you are seeing unusual drops.
Open a support in the KeyCDN dashboard if you can not solve the 502 problem.
My GAE app suddenly stopped working last friday. I tried several times to start an instance in background, (both using cron and manually) but after a minute since the instance starts I get this error:
/cron/spreadsheet 500 4660ms 0kb instance=0 AppEngine-Google;
(+http://code.google.com/appengine)
0.1.0.1 - - [17/Feb/2013:12:40:05 -0800] "GET /cron/spreadsheet HTTP/1.1" 500 0 -
"AppEngine-Google; (+http://code.google.com/appengine)"
"backend.myclimagest.appspot.com" ms=4661 cpu_ms=692 queue_name=__cron
task_name=5ed789ec52683cb170ed3c2cc4ec48c9 pending_ms=3651 exit_code=121
instance=0
W 2013-02-17 12:40:05.268
A problem was encountered with the process that handled this request,
causing it to exit. This is likely to cause a new process to be used
for the next request to your application. (Error code 121)
I don't know what to do since this error code is undocumented. I also tried to redeploy but no luck. Are you experiencing the same problem and have you maybe found a solution?
I don't think it matters but the app is developed in Java using Eclipse.
The app is in production so I really need to solve this issue as quickly as I can.
Thank you
It seems your App Engine scheduler got problem. Try this, Removing the automatic settings from Performance settings which seeems to help. Google Groups
Check this Official Google page too Managing Your App's Resource Usage
Am new to java, I know Java has Log4J, logback etc for logging purposes. My question is more around how many log files should we have in a application. Should it be per thread, per group of threads, process, exception etc. In our application there is a possibility of having a large number of threads and am thinking about cons of having log file per thread. Are there best practices for logging in applications having huge number of threads .
Thanks in advance!
1 log for messages - Call it SystemOut.log
1 log for stack traces - Call it SystemErr.log
1 log for traces - Call it Trace.log
1 log for native stdout - Call it nativeStdOut.log
1 log for native stderr - Call it nativeStdErr.log
Have a config panel that sets:
maxSize
maxCount
When a log hits max size, starting rolling them upto maxCount and append a timestamp to the rolled filename.
I think that good solution would be to name your threads and write logs together with the name of thread in which log occurred. Thanks to that you will be able to both analysis logs separately for each thread or analysis all logs together.
Typically there is one log file per application (process) -- rarely for Thread and never be Exception. Sometimes this log file is split into various different log levels: debug messages in one bucket, information in another, warnings/errors in a third. This makes it easy to watch for errors by only looking at the warning-and-more-critical file.
log4j has a configuration file in which you can route certain messages to certain files using different criteria. Here's a sample of the log4j properties file:
# default is WARN and above going to the appender "logfile" and STDOUT
log4j.rootLogger=WARN, logfile, stdout
# write to ui.log for a day and then move to .yyyy-MM-dd suffix
log4j.appender.logfile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logfile.File=ui.log
log4j.appender.logfile.Append=true
log4j.appender.logfile.DatePattern='.'yyyy-MM-dd
# but we log information message from classes in the package com.mprew.be
log4j.logger.com.mprew.be=INFO
log4j, and custom loggers, decorate each time with a Class name, priority level, date/time, etc.. For example:
# date time priority Class-name Log message
2012-03-26 00:55:39,545 [INFO] CmsClientTask Content is up-to-date
Typically exceptions are written out as multiple lines so you can get the entire stack-trace.
2012-03-26 01:55:35,777 [INFO] ExceptionInterceptor Reporting problem to customer
org.springframework.NoSuchRequestException: No request handling method
at com.ui.base.BaseController.invokeNamedHandler(BaseController.java:240)
at com.ui.base.BaseController.handleRequestInternal(BaseController.java:100)
at com.ui.base.CoreServices.handleRequest(CoreServicesController.java:147)
...
In our distributed system, we route all logs from all of the system to 2 servers which write a debug, info, and warning logs. Along with the date/time, class name, priority, and message, the log messages also have the hostname and a specific log token so we can easily identify classes of problems. The following is on one line:
2012-03-26 00:00:00.045 UTC INFO FE8 TrayController TRAY_CLIENT_LOOKUP
message=created unknown client with partner STW9
Then we can easily grep for specific issues.
Hope this helps.