I have a service that reads from DynamoDB and I am seeing high latency for a particular request. On checking the debug logs, could find the following lines and the difference in their timestamps is over 1000 ms. It would be a great help if someone could please help to understand what the following lines mean, how it increases the latency and how the same can be prevented.
10 Mar 2022 18:17:46,036 com.amazonaws.retry.ClockSkewAdjuster: Reported server date (from 'Date' header): Thu, 10 Mar 2022 18:17:46 GMT
10 Mar 2022 18:17:47,254 javax.xml.bind: Checking system property javax.xml.bind.context.factory
Related
We face intermittent High Thread Counts followed by JVM's slowdown and during the same timeframe, we could find the below mentioned exceptions. Please suggest if these are SYMPTOMS or CAUSES and help understand the remediation steps.
Mar 23, 2022 2:44:13 PM org.apache.catalina.connector.CoyoteAdapter checkRecycled
INFO: Encountered a non-recycled response and recycled it forcedly.
org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException
Mar 23, 2022 2:57:56 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.NullPointerException: Inflater has been closed
We are running Kubernetes (1.18) with Docker 1.19 & systemd on an on-prem deployment with 3 masters and 3 workers. OS is RedHat 7.8.
Container is a Java 13 based spring boot app (using base image as openjdk:13-alpine) and below are the memory settings.
Pod:
memory - min 448M and max 2500M
cpu - min 0.1
Container:
Xms: 256M, Xmx: 512M
When traffic is send for a longer time, the container suddenly restarts; and in Prometheus I can see the Pod memory is below the max level (only around 1300MB).
In the pod events I can see warnings for liveness and readiness probes; and the pod getting restarted.
State: Running
Started: Sun, 23 Aug 2020 15:39:13 +0530
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Sun, 23 Aug 2020 15:23:03 +0530
Finished: Sun, 23 Aug 2020 15:39:12 +0530
Ready: True
Restart Count: 14
What are logs that I can refer to figure out why a restart was triggered? Application log is not helping at all; after the last log of the running app; I can see the starting line of the log as the next line.
What are the recommended approaches to troubleshoot this?
Thanks
137 means 128 + 9 (so it was killed with kill -9)
https://tldp.org/LDP/abs/html/exitcodes.html
Have a look at the pod and application logs.
Maybe the container needs more resources to start?
I am working on an android project which uses data from Google data store. For this I have JSP & Servlet based admin module which will be used to upload data. To read data from Android client application I have cloud entry points.
Thus, I have created a module comprising of JSP, servlets, and cloud entry points. This is working fine with local dev server. Able to access data through android app.
Now, trying to deploy the module to Google App Engine. But getting following error:
SEVERE: Endpoints configuration not updated. The app returned an
error when the Google Cloud Endpoints server attempted to communicate
with it.
What could be missing here?
Let me know if anymore information needed regarding this issue.
Please help me in this regard.
Full trace:
/usr/lib/jvm/java-8-oracle/bin/java -Dfile.encoding=UTF-8 -classpath /root/.gradle/appengine-sdk/appengine-java-sdk-1.9.18/lib/appengine-tools-api.jar com.google.appengine.tools.admin.AppCfg --application=rotary-bidar-01 --version=1 --oauth2 --oauth2_client_secret=***** --oauth2_client_id=***** --oauth2_refresh_token=***** update /mnt/4CF2AADEF2AACC10/Projects/Rotary_Bidar/AndroidApplication/RotaryBidar/server/build/exploded-app
Reading application configuration data...
Jan 16, 2016 11:28:59 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
INFO: Successfully processed /mnt/4CF2AADEF2AACC10/Projects/Rotary_Bidar/AndroidApplication/RotaryBidar/server/build/exploded-app/WEB-INF/appengine-web.xml
Jan 16, 2016 11:28:59 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed /mnt/4CF2AADEF2AACC10/Projects/Rotary_Bidar/AndroidApplication/RotaryBidar/server/build/exploded-app/WEB-INF/web.xml
Beginning interaction for module default...
0% Created staging directory at: '/tmp/appcfg5838290850638628913.tmp'
5% Scanning for jsp files.
8% Compiling jsp files.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /tmp/1452967141068-0/org/apache/jsp/admin_jsp.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Jan 16, 2016 11:29:10 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed /tmp/appcfg5838290850638628913.tmp/WEB-INF/web.xml
20% Scanning files on local disk.
25% Initiating update.
28% Cloning 13 static files.
31% Cloning 50 application files.
40% Uploading 2 files.
52% Uploaded 1 files.
61% Uploaded 2 files.
68% Initializing precompilation...
73% Sending batch containing 2 file(s) totaling 1KB.
90% Deploying new version.
95% Will check again in 1 seconds.
98% Will check again in 2 seconds.
99% Will check again in 4 seconds.
99% Will check again in 8 seconds.
99% Will check again in 16 seconds.
99% Will check again in 32 seconds.
99% Endpoints configuration not updated. The app returned an error when the Google Cloud Endpoints server attempted to communicate with it.
Jan 16, 2016 11:30:32 PM com.google.appengine.tools.admin.AppVersionUpload checkEndpointsServingStatusResult
99% See the deployment troubleshooting documentation for more information: https://developers.google.com/appengine/docs/java/endpoints/test_deploy#troubleshooting_a_deployment_failure
SEVERE: Endpoints configuration not updated. The app returned an error when the Google Cloud Endpoints server attempted to communicate with it.
99% Ignoring Endpoints failure and proceeding with update.
99% Closing update: new version is ready to start serving.
99% Uploading index definitions.
Update for module default completed successfully.
Success.
Cleaning up temporary files for module default...
My customer web application is currently on an AWS-EC2 Linux instance, running WildFly 8.2.0.Final "Tweek".
I'm not using Apache or other web servers, just accessing the application server on port 8080 (the system is forwarding to 8080 all traffic on port 80).
Application works fine everywhere, but yesterday I found a very weird issue on one of my customer's browsers. However, it's not related to the browser, because I already tried installing Firefox and Chrome, in addition to IE, and the problem is still there. The client OS is Win8.1.
The CSS and JS files are not loaded by the browser: Chrome prints this message in the console:
ERR_INVALID_CHUNKED_ENCODING
However, if I click on the file and 'Open link in new tab', I can actually see the file's content!
My static files are JSF resources, for example:
/javax.faces.resource/primefaces.css.htm?ln=primefaces&v=5.1
Response headers are, for example:
Content-Type:application/javascript
Date:Fri, 27 Mar 2015 16:32:51 GMT
ETag:W/"312861-1427473269000"
Expires:Fri, 03 Apr 2015 16:32:51 GMT
Last-Modified:Fri, 27 Mar 2015 16:21:09 GMT
I saw the content-length is also zero, which I suppose is the cause for the problem.
How to find out more info on that? What's the solution for that type of error? Thanks
I'm trying to stress my Application into Weblogic11g and I'm sending it many JMS Message on the queues.
But, the FileStore crashes with an OOE around 20K messages and a max size of 647,169 ko
Exception in thread "Thread-13" java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at weblogic.store.io.file.StoreFile.expand(StoreFile.java:324)
at weblogic.store.io.file.Heap.reserveSpace(Heap.java:305)
at weblogic.store.io.file.Heap.multiWrite(Heap.java:438)
at weblogic.store.io.file.FileStoreIO.flush(FileStoreIO.java:497)
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:638)
at weblogic.store.internal.PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
And a few line of the log file
Feb 25, 2014 7:53:19 PM CET Warning JTA BEA-110484 The JTA health state has changed from HEALTH_OK to HEALTH_WARN with reason codes: Resource WLStore_MyFS_stores-Node1-file-jms declared unhealthy.
Feb 25, 2014 7:53:19 PM CET Warning JTA BEA-110030 XA resource [WLStore_MyFS_stores-Node1-file-jms] has not responded in the last 120 second(s).
Feb 25, 2014 7:53:19 PM CET Warning JTA BEA-110405 Resource WLStore_MyFS_stores-Node1-file-jms was not assigned to any of these servers: Node1
Feb 25, 2014 7:54:19 PM CET Warning JTA BEA-110486 Transaction BEA1-5DA4B1F8A57C83AEDB1B cannot complete commit processing because resource [WLStore_MyFS_stores-Node1-file-jms] is unavailable. The transaction will be abandoned after 3,420 seconds unless all resources acknowledge the commit decision.
Is it possible to increase the size of this FileStore ?
When sending messages to WLS it keeps the message + header in memory until the message is consumed.
If your rate of production of messages is faster then rate of consumption of messages then you will eventually hit an OOM.
There's couple of things you can do to avoid getting OOM
1) Ensure you have enough consumers for the messages and they are able to consume messages quickly.
2) By default there's a JMS Paging feature which gets triggered when the memory consumption of JMS messages is about 1/3 of the overall heap. You can tune your server to trigger Paging earlier if you want. What paging does is, it leaves the header part of each message in memory and the body moves to the paging file, thereby releasing some memory. For simplistic calculation JMS headers only would consume about 1k of memory.
- Note Pending messages will have both header and body in memory.
3) Ofcourse increasing the JVM size for the managed server hosting your JMS server would directly enable you to keep more messages in memory.
Try increasing the heap allocation for the managed server by providing a larger value for the -Xmx parameter in the Server Start parameters or start script.
See How to increase memory in weblogic for more details.