I'm trying to stress my Application into Weblogic11g and I'm sending it many JMS Message on the queues.
But, the FileStore crashes with an OOE around 20K messages and a max size of 647,169 ko
Exception in thread "Thread-13" java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at weblogic.store.io.file.StoreFile.expand(StoreFile.java:324)
at weblogic.store.io.file.Heap.reserveSpace(Heap.java:305)
at weblogic.store.io.file.Heap.multiWrite(Heap.java:438)
at weblogic.store.io.file.FileStoreIO.flush(FileStoreIO.java:497)
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:638)
at weblogic.store.internal.PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
And a few line of the log file
Feb 25, 2014 7:53:19 PM CET Warning JTA BEA-110484 The JTA health state has changed from HEALTH_OK to HEALTH_WARN with reason codes: Resource WLStore_MyFS_stores-Node1-file-jms declared unhealthy.
Feb 25, 2014 7:53:19 PM CET Warning JTA BEA-110030 XA resource [WLStore_MyFS_stores-Node1-file-jms] has not responded in the last 120 second(s).
Feb 25, 2014 7:53:19 PM CET Warning JTA BEA-110405 Resource WLStore_MyFS_stores-Node1-file-jms was not assigned to any of these servers: Node1
Feb 25, 2014 7:54:19 PM CET Warning JTA BEA-110486 Transaction BEA1-5DA4B1F8A57C83AEDB1B cannot complete commit processing because resource [WLStore_MyFS_stores-Node1-file-jms] is unavailable. The transaction will be abandoned after 3,420 seconds unless all resources acknowledge the commit decision.
Is it possible to increase the size of this FileStore ?
When sending messages to WLS it keeps the message + header in memory until the message is consumed.
If your rate of production of messages is faster then rate of consumption of messages then you will eventually hit an OOM.
There's couple of things you can do to avoid getting OOM
1) Ensure you have enough consumers for the messages and they are able to consume messages quickly.
2) By default there's a JMS Paging feature which gets triggered when the memory consumption of JMS messages is about 1/3 of the overall heap. You can tune your server to trigger Paging earlier if you want. What paging does is, it leaves the header part of each message in memory and the body moves to the paging file, thereby releasing some memory. For simplistic calculation JMS headers only would consume about 1k of memory.
- Note Pending messages will have both header and body in memory.
3) Ofcourse increasing the JVM size for the managed server hosting your JMS server would directly enable you to keep more messages in memory.
Try increasing the heap allocation for the managed server by providing a larger value for the -Xmx parameter in the Server Start parameters or start script.
See How to increase memory in weblogic for more details.
Related
We are running Kubernetes (1.18) with Docker 1.19 & systemd on an on-prem deployment with 3 masters and 3 workers. OS is RedHat 7.8.
Container is a Java 13 based spring boot app (using base image as openjdk:13-alpine) and below are the memory settings.
Pod:
memory - min 448M and max 2500M
cpu - min 0.1
Container:
Xms: 256M, Xmx: 512M
When traffic is send for a longer time, the container suddenly restarts; and in Prometheus I can see the Pod memory is below the max level (only around 1300MB).
In the pod events I can see warnings for liveness and readiness probes; and the pod getting restarted.
State: Running
Started: Sun, 23 Aug 2020 15:39:13 +0530
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Sun, 23 Aug 2020 15:23:03 +0530
Finished: Sun, 23 Aug 2020 15:39:12 +0530
Ready: True
Restart Count: 14
What are logs that I can refer to figure out why a restart was triggered? Application log is not helping at all; after the last log of the running app; I can see the starting line of the log as the next line.
What are the recommended approaches to troubleshoot this?
Thanks
137 means 128 + 9 (so it was killed with kill -9)
https://tldp.org/LDP/abs/html/exitcodes.html
Have a look at the pod and application logs.
Maybe the container needs more resources to start?
I have a queue on Solace, where i am able to write data to queue.
But getting error while reading data from queue.
Below is the error i am getting.
Any idea what could be the issue.
Oct 26, 2017 3:13:54 PM com.solacesystems.jcsmp.protocol.impl.TcpClientChannel close
INFO: Channel Closed (smfclient 6)
Oct 26, 2017 3:13:54 PM com.solacesystems.jcsmp.impl.flow.BindRequestTask execute
INFO: Error Response (403) - Permission Not Allowed
javax.jms.JMSSecurityException: Error creating consumer - access
denied (Permission Not Allowed)
The issue is likely that the client is not the owner of the queue and the queue is configured with "Read Only" or "No Access" for its permission level.
The queue's permission level defines the level of access given to consuming clients that are not defined as the owner of the queue.
To resolve this issue, you can edit the queue's permission level to "Consume", "Modify-Topic", or "Delete". Note that you will need to disable the queue before making these changes.
I am working on an android project which uses data from Google data store. For this I have JSP & Servlet based admin module which will be used to upload data. To read data from Android client application I have cloud entry points.
Thus, I have created a module comprising of JSP, servlets, and cloud entry points. This is working fine with local dev server. Able to access data through android app.
Now, trying to deploy the module to Google App Engine. But getting following error:
SEVERE: Endpoints configuration not updated. The app returned an
error when the Google Cloud Endpoints server attempted to communicate
with it.
What could be missing here?
Let me know if anymore information needed regarding this issue.
Please help me in this regard.
Full trace:
/usr/lib/jvm/java-8-oracle/bin/java -Dfile.encoding=UTF-8 -classpath /root/.gradle/appengine-sdk/appengine-java-sdk-1.9.18/lib/appengine-tools-api.jar com.google.appengine.tools.admin.AppCfg --application=rotary-bidar-01 --version=1 --oauth2 --oauth2_client_secret=***** --oauth2_client_id=***** --oauth2_refresh_token=***** update /mnt/4CF2AADEF2AACC10/Projects/Rotary_Bidar/AndroidApplication/RotaryBidar/server/build/exploded-app
Reading application configuration data...
Jan 16, 2016 11:28:59 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
INFO: Successfully processed /mnt/4CF2AADEF2AACC10/Projects/Rotary_Bidar/AndroidApplication/RotaryBidar/server/build/exploded-app/WEB-INF/appengine-web.xml
Jan 16, 2016 11:28:59 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed /mnt/4CF2AADEF2AACC10/Projects/Rotary_Bidar/AndroidApplication/RotaryBidar/server/build/exploded-app/WEB-INF/web.xml
Beginning interaction for module default...
0% Created staging directory at: '/tmp/appcfg5838290850638628913.tmp'
5% Scanning for jsp files.
8% Compiling jsp files.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /tmp/1452967141068-0/org/apache/jsp/admin_jsp.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Jan 16, 2016 11:29:10 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed /tmp/appcfg5838290850638628913.tmp/WEB-INF/web.xml
20% Scanning files on local disk.
25% Initiating update.
28% Cloning 13 static files.
31% Cloning 50 application files.
40% Uploading 2 files.
52% Uploaded 1 files.
61% Uploaded 2 files.
68% Initializing precompilation...
73% Sending batch containing 2 file(s) totaling 1KB.
90% Deploying new version.
95% Will check again in 1 seconds.
98% Will check again in 2 seconds.
99% Will check again in 4 seconds.
99% Will check again in 8 seconds.
99% Will check again in 16 seconds.
99% Will check again in 32 seconds.
99% Endpoints configuration not updated. The app returned an error when the Google Cloud Endpoints server attempted to communicate with it.
Jan 16, 2016 11:30:32 PM com.google.appengine.tools.admin.AppVersionUpload checkEndpointsServingStatusResult
99% See the deployment troubleshooting documentation for more information: https://developers.google.com/appengine/docs/java/endpoints/test_deploy#troubleshooting_a_deployment_failure
SEVERE: Endpoints configuration not updated. The app returned an error when the Google Cloud Endpoints server attempted to communicate with it.
99% Ignoring Endpoints failure and proceeding with update.
99% Closing update: new version is ready to start serving.
99% Uploading index definitions.
Update for module default completed successfully.
Success.
Cleaning up temporary files for module default...
When my database request takes too long tomcat crashes / closes and I find this in my log. I am totally clueless right now :)
Any thoughts ?
SEVERE: The web application [/ir] is still processing a request that
has yet to finish. This is very likely to create a memory leak. You
can control the time allowed for requests to finish by using the
unloadDelay attribute of the standard Context implementation.
Mar 14, 2015 2:14:38 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/ir] appears to have started a thread
named [MySQL Statement Cancellation Timer] but has failed to stop it.
This is very likely to create a memory leak.
What should I be doing ?
I'm using Eclipse to develop a java web app and My IT department called and said I had over 75 open oracle connections to the development Oracle Server originating from my PC. The deployed app has been using the same singleton connection bean for a year now and I haven't had any trouble with exceeding connections there. IT even connfirmed I only had 4 open connections on the deployment server. Something is going on when I develop locally in Eclipse.
Looking at the console debug output, everything looks good on initial startup
Sep 27, 2010 9:39:08 AM org.apache.catalina.startup.Catalina start
== no ConnCache Bean instance found
== Initializing Connection Cache Data Src
== About to load the properties
== Initializing Datasource for using jdbc:oracle:thin:#hostname:port:oracle-sid
== Reading property file
== Setting InitialLimit to 3
== Properties {TimeToLiveTimeout=4, MinLimit=4, ConnectionWaitTimeout=44, MaxLimit=5, InitialLimit=3, AbandonedConnectionTimeout=281, InactivityTimeout=269}
But then I noticed the following SEVERE message in the console When Eclipse reloads the context
Sep 27, 2010 9:40:38 AM org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Sep 27, 2010 9:40:38 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [] registered the JBDC driver [oracle.jdbc.driver.OracleDriver] but failed to unregister
it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Sep 27, 2010 9:40:38 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [Thread-9] but has failed to stop it.
This is very likely to create a memory leak.
I'm guessing that Eclipse is recreating a connection cache everytime the it reloads the context and the old connections and cache are not getting destroyed. How can I get Eclipse to terminate the cache or close all previous connections when the context is reloaded. I can post my connectionBean class if necessary.
It appears that you are leaking memory via the unclosed connections of the previous version of the application. This is quite possible if you've chosen to perform automatic publishing of your web application from Eclipse to Tomcat.
Tomcat will discard the current classloader of the web application, when it attempts to perform a reload. All objects from this GC root (the classloader), that will still be held, will not be garbage collected. Due to the presence of the singleton (I'm assuming that the singleton stores a DataSource object or is maintaining a pool of Connection objects), these Connection objects will never be treated as unreachable objects, and hence, will not be eligible for GC.
In the short run, you can do the following:
If you can help it, restart Tomcat whenever you need to publish a new version of the application.
Implement a ServletContextListener, and attempt to discard the singleton reference when the servlet (application) context is destroyed. This makes the singleton object unreachable, and hence eligible for GC. This is recommended, for the same is bound to occur in production, unless Tomcat is restarted.
By the way, this behavior, where in singletons cause memory and hence connection leakage will occur in any application server, as long as the application is redeployed time and again without restarting the server. The only way to prevent this, is to ensure that the singleton is destroyed when the application context is destroyed.