Develop Reference

How to profile spring-boot application memory consumption?

I have a spring-boot app that I suspect might have a memory leak. Over time the memory consumption seems to increase, taking like 500M of memory until I restart the application. After a fresh restart it takes something like 150M. The spring-boot app should be a pretty stateless rest app, and there shouldn't be any objects left around after request is completed. I would wish the garbage collector would take care of this.
Currently on production the spring-boot app seems to use 343M of memory (RSS). I got the heapdump of the application and analysed it. According to the analysis the heapdump is only 31M of size. So where does the missing 300M lie in? How is the heapdump correlated with the actual memory the application is using? And how could I profile the memory consumption past the heapdump? If the memory used is not in the heap, then where is it? How to discover what is consuming the memory of the spring-boot application?

So where does the missing 300M lie in?
A lot of research has gone into this, especially in trying to tune the parameters that control the non-heap. One result of this research is the memory calculator (binary).
You see, in Docker environments with a hard limit on the available amount of memory, the JVM will crash when it tries to allocate more memory than is available. Even with all the research, the memory calculator still has a slack-option called "head-room" - usually set to 5 to 10% of total available memory in case the JVM decides to grab some more memory anyway (e.g. during intensive garbage collection).
Apart from "head-room", the memory calculator needs 4 additional input-parameters to calculate the Java options that control memory usage.
total-memory - a minimum of 384 MB for Spring Boot application, start with 512 MB.
loaded-class-count - for latest Spring Boot application about 19 000. This seems to grow with each Spring version. Note that this is a maximum: setting a too low value will result in all kinds of weird behavior (sometimes an "OutOfMemory: non-heap" exception is thrown, but not always).
thread-count - 40 for a "normal usage" Spring Boot web-application.
jvm-options - see the two parameters below.
The "Algorithm" section mentions additional parameters that can be tuned, of which I found two worth the effort to investigate per application and specify:
-Xss set to 256kb. Unless your application has really deep stacks (recursion), going from 1 MB to 256kb per thread saves a lot of memory.
-XX:ReservedCodeCacheSize set to 64MB. Peak "CodeCache" usage is often during application startup, going from 192 MB to 64 MB saves a lot of memory which can be used as heap. Applications that have a lot of active code during runtime (e.g. a web-application with a lot of endpoints) may need more "CodeCache". If "CodeCache" is too low, your application will use a lot of CPU without doing much (this can also manifest during startup: if "CodeCache" is too low, your application can take a very long time to startup). "CodeCache" is reported by the JVM as a non-heap memory region, it should not be hard to measure.
The output of the memory calculator is a bunch of Java options that all have an effect on what memory the JVM uses. If you really want to know where "the missing 300M" is, study and research each of these options in addition to the "Java Buildpack Memory Calculator v3" rationale.
# Memory calculator 4.2.0
$ ./java-buildpack-memory-calculator --total-memory 512M --loaded-class-count 19000 --thread-count 40 --head-room 5 --jvm-options "-Xss256k -XX:ReservedCodeCacheSize=64M"
-XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=121289K -Xmx290768K
# Combined JVM options to keep your total application memory usage under 512 MB:
-Xss256k -XX:ReservedCodeCacheSize=64M -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=121289K -Xmx290768K

Besides heap, you have thread stacks, meta space, JIT code cache, native shared libraries and the off-heap store (direct allocations).
I would start with thread stacks: how many threads does your application spawn at peak? Each thread is likely to allocate 1MB for its stack by default, depending on Java version, platform, etc. With (say) 300 active threads (idle or not), you'll allocate 300MB of stack memory.
Consider making all your thread pools fixed-size (or at least provide reasonable upper bounds). Even if this proves not to be root cause for what you observed, it makes the app behaviour more deterministic and will help you better isolate the problem.

We can view how much of memory consumption in spring boot app, in this way.
Create spring boot app as .jar file and execute it using java -jar springboot-example.jar
Now open the CMD and type jconsole and hit enter.
Note :- before opening the jconsole you need to run .jar file
Now you can see a window like below and it will appear application that previously ran in Local Process section.
Select springboot-example.jar and click below connect button.
After it will show the below prompt and give Insecure connection option.
Finally you can see Below OverView (Heap Memory, Threads...).

You can use "JProfiler" https://www.ej-technologies.com/products/jprofiler/overview.html
remotely or locally to monitor running java app memory usage.

You can using "yourkit" with IntelliJ if you are using that as your IDE to troubleshoot memory related issues for your spring boot app. I have used this before and it provides better insight to applications.
https://www.yourkit.com/docs/java/help/idea.jsp

Interesting article about memory profiling: https://www.baeldung.com/java-profilers

Why does RES memory keep slowly growing for Java processes even for out of the box Spring Boot admin?

I have 23 Java processes running on one machine with 32GB. No process specifies JVM memory params such as Xmx. java -XX:+PrintFlagsFinal -version | grep MaxHeapSize reports that max default heap size is 8GB as expected.
Every process runs embedded Tomcat (Spring Boot apps (most at v 2.3.4)) except one is a standalone tomcat 9 instance running three WARs. These apps have low usage (usually one user and 10 minutes use a day). They are not memory or CPU intensive. One of them is Spring Boot admin and another is Spring Cloud's Eureka service registry. For these two, I have only a main method that simply bootstraps the Spring Boot application.
Yet, RES memory as shown in top for every process keeps gradually increasing. For example, Spring Boot service registry has increased from 1.1GB to 1.5GB in the last 12 hours. All processes show a similar small increase in RES but the total increase has reduced free memory by 2 GB in that same 12 hour period. This was the same in the previous 12 hours (and so on)until current free memory is now only 4.7GB.
My concern is that I continue to see this trend (even without app usage). Memory is never freed from the apps so total free memory continues to decrease. Is this normal since perhaps each JVM sees that memory is still available in the OS and that 8GB heap space is available to it? Will the JVMs stop taking memory at some point say once an OS free memory threshold is reached? Or will it continue until all free memory is used?
Update
The heap used for most apps is under 200MB but the heap size is 1.5 - 2.8GB. Heap max is 8GB.

Resident memory as reported by the OS doesn't tell you what component is consuming it. You'll have to gather additional data to figure out which part of the process is growing
You'll have to track
java heap and metaspace use - you can monitor this with JMC, gc logging and many other java monitoring tools
jvm off-heap use - NMT
direct byte buffer use - MX beans, also available via JMC
use by mapped files - pmap -x <pid>
use by native libraries e.g. used via JNI - difficult to monitor

I also faced this situation and after a long time of research I found the solution here. Basically, for my case, it was just a matter of setting xms and xmx parameters on the jar invocation, forcing the GC to act constantly.

Managing Java Memory with Xms and Xmx

I had a java application that runs as a daemon process. Initially, I discovered that after about a week, the applicaiton would throw an out of memory error. I researching, I discovered that I could manage the memory and avoid the problem by setting the -Xms and -Xmx flags at startup to specify how much memory the application would have. This ultimately resolved my memory problem.
However, later on, I discovered that the application was causing performance issues on some machines that only had 1GB of memory, 1GB of memory was how much I had allocated for -Xmx. So the java application took all of available memory.
So using jconsole, I was about to determine that the application ran at about 150mb. So using that information, I dropped the -Xmx value to 300mb. Thinking that it would be plenty. But then I discovered that the application was only using about 15mb. Why? Why does the application use 150mb when its allocated 1GB, but only 15mb when its allocated 300mb?
And how do I go about figuring out what I should be using?

Monitoring Java internal objects & memory usage

I have a Java web server running as a Windows service.
I use Tomcat 8 with Java 1.8.*
For a few months now, I've detected that the memory usage is increasing quite rapidly. I cannot make up for sure if it's heap or stack.
The process starts with ~200MB and after a week or so, it can reach up to 2GB.
Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
This has repeated multiple times on multiple environments.
I would like to know if there's a way to monitor the process and view it's internal memory usage, even to the level of viewing which objects are using the most amount of memory.
Can 'Java Native Memory Tracking' be used for this?
This will help me to detect any memory leaks that might cause this.
Thanks in advance.

To monitor the memory usage of a Java process, I'd use a JMX client such as JVisualVM, which is bundled with the Oracle JDK:
https://visualvm.java.net/jmx_connections.html
To identify the cause of a memory leak, I'd instruct the JVM to take a heap dump when it runs out of memory (on the Oracle JVM, this can be accomplished by specifying -XX:-HeapDumpOnOutOfMemoryError when starting your Java program), and then analyze that heap dump using a tool such as Eclipse MAT.

quoting:
the process starts with ~200MB and after a week or so, it can reach up to 2GB. Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
The problem might not be as simple as seeing what java objects you have got in JVisualVM (e.g millions of strings)
What you need to do is identify the code that leaks.
One way you could do that is to force the execution of particular code and then monitor the memory.
The easiest way to force the execution of code inside classes/objects is to use a tool like https://github.com/lorenzoongithub/nudge4j (particularly since you are on java 8)
alternatively you could just wire up nashorn to a command line or run your progam via jjs https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/shell.html

Running multiple jvms for different applications in same machine

We are getting frequent out of memory errors in our dev. machines We are running webshpere, eclipse, soap UI and maven in it. Our server gets down due to this "out of memory errors" when we restart our applications in websphere 2/3 times, We already increased the virtual memory setting in wesphere to 1GB.
So what i did was copied the jre we use in eclipse and maven folders so that each of these uses individual jvms. But the performance of websphere is same. 2/3 restarts and out of memory errors.
Is there any may of making eclipse and maven use different jvms other than websphere's?

In response to the question:
If you start java multiple times, multiple copies of java will be running with each their own memory. Eclipse and websphere are probably started separately so use independent memory. Your trouble should not be there.
In response to your problem
Out of Memory
Both Eclipse and Websphere can gobble up memory like there's not tomorrow. Look al the -X flags, the flag for perm gen space should be added to the flag for heap space to get the memory consumption. Also allow some overhead for the OS, windowing environment, e-mail client, browser (500 MB - 1 GB or so, depending on the OS and what you're running). So it can be that the computer is out of memory.
More frequently the amount of memory assigned to the jvm is just not enough. Java has not been started with enough memory for the app assigned to it. It's up to you to deduce if it is Heap Space which ran out, or PermGen space. Both can be adjusted, have a look at this website. The flags are -Xmx and -XX:MaxPermSize. Look at the start scripts for Websphere, as that's the one complaining.
Recommendation
Check which kind of memory is out, and search for that on stack overflow; either PermGen or Heap Space should do.

You should set the JVM Xmx parameter less than or equal to 256MB in all the three process. It will never cross the 256 MB limit (considering that the program does not have memory leaks)

Copying the folder won't be enough.
In Eclipse, go to Preferences, under Java-> Installed JREs make sure the JRE used is on a different path than the one used by WebSphere.