I have a regular 64-bit hotspot JVM which has 1 MB stack size. Now I was trying to serialize an object that has hierarchy of 3022 parents and this is giving me SO(irony) exception.
Here's some code:
while(epc.getParent()!=null){
epc=epc.getParent();
count++;
}
print(count);//3022
Above code is just to tell the hierarchy but the actual problem occurs when I try to serialize the epc object onto ObjectOutputStream.
Question, what does 1 MB stack size states in JVM as I have no idea what size a stack frame of? I'm sure it's not 1KB per stack frame because I ran about code successfully at -Xss3000k.
One more question, does every thread has 3000k stack size if I put a JVM option of -Xss3000k ?
Question, what does 1 MB stack size states in JVM as I have no idea what size a stack frame of?
A 1 MB default thread stack size means that every thread has 1MB (1048576 bytes) of stack space ... by default. The exception is if your code creates a thread using one of the Thread constructors where you can provide a stack size argument.
The size of a stack frame depends on the method being called. It needs to hold the method's parameters and local variables, so the frame size depends on their size. Each frame also needs (I think) two extra words to hold a saved frame pointer and a saved return address.
Note that in a recursive algorithm, you can have more than one stack frame for one "level of recursion". For writeObject (in Java 8), the algorithm used is recursive, and there are typically 4 frames per level of the data structure being serialized:
writeObject0
writeOrdinaryObject
writeSerialData
defaultWriteFields
writeObject0
etcetera
The actual frame sizes will be platform dependent due to differences in compilers, and changes in the implementation of ObjectInputStream / ObjectOutputStream. You would better off trying to (roughly) measure the required stackspace rather than trying to predict frame sizes from first principles.
One more question, does every thread has 3000k stack size if I put a JVM option of -Xss3000k ?
Yes ... with the exception that I have described above.
One possible solution to your dilemma is create a special thread with an huge stack that you use for serialization. A similar thread with a huge stack will be required for deserialization. For the rest of the threads, the default stack size should be fine.
Other possible solutions:
Implement writeReplace and readResolve methods to flatten the parent structure of your epc objects into an array so that you don't get deep recursion. (Obviously, the flattening / unflattening needs to be done non-recursively.)
Do the same flattening before you call writeObject, etcetera.
Use a different serialization mechanism, or possible a custom one.
Does every thread has 3000k stack size if I put a JVM option of -Xss3000k ?
There is a constructor to specify the stack size for the new Thread. All threads that don't use that special constructor will get the default stack size specified in the JVM option (and that will cover all threads unless you create them yourself explicitly).
If your application does not need a great number of threads, the raised limit is probably not a problem and the easiest solution.
If not, you may want to construct such a big-stack thread especially for running your deeply recursive serialization code. You can wrap it up into an Executor and have application code call into it.
what does 1 MB stack size states in JVM as I have no idea what size a stack frame of?
That is indeed a bit of a moving target. You should make this configurable and experimentation will tell you what a good setting is.
Even the Javadoc says
Due to the platform-dependent nature of the behavior of this constructor, extreme care should be exercised in its use. The thread stack size necessary to perform a given computation will likely vary from one JRE implementation to another. In light of this variation, careful tuning of the stack size parameter may be required, and the tuning may need to be repeated for each JRE implementation on which an application is to run.
Related
Today morning I answered a question which is related to StackoverflowException . The person has asked when Stackoverflow exception occurs
See this link Simplest ways to cause stack overflow in C#, C++ and Java
So my question is that is there any method by which we can compute the method call stacks size dynamically in our program and then applying a check before calling a method which checks whether method call stack has space to accommodate it or not to prevent StackOverflowException.
As I am a java person I am looking for java but also looking for explanation related to the concept without boundation of any programming language.
The total memory available to a JVM is about 2-4GB for a 32bit JVM and the square of this for a 64bit JVM (about 4-16EB). The JVM splits it's memory into:
Heap Memory (allocation controlled via JVM options -Xms and -Xmx)
constructed object and array instances
static classes and array data (including contained object/array instances)
thread instances (object instances, runtime data & metadata including thread object monitor lock references)
Non-Heap Memory
aggregate stack memory
per-thread stack memory (per-thread allocation controlled via JVM option -Xss): method call frames, arguments, return values, locally declared primitives & references to objects
static constants (primitives)
String instance pool
java code: loaded classes and metadata
JVM internal-use memory (JVM code and data structures)
See http://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryMXBean.html and http://www.yourkit.com/docs/kb/sizes.jsp
Is there any method by which we can compute the method call stacks size dynamically in our program
There's no standard method included in Java SE/Java EE to obtain the per-thread stack actual memory usage.
There are standard methods to obtain the aggregate non-heap memory: MemoryMxBean.getNonHeapMemoryUsage(). Referring to this doesn't allow you to make dynamic in-code decisions to avoid StackOverflow exception
There are standard methods to obtain the call stack without it's memory usage: Thread.getStackTrace() ThreadMxBean.getThreadInfo() & ThreadInfo.getStackTrace()
I recommend that you don't do what you suggest in the question because:
You can't do it without some complex JVM-specific API that instruments/introspects on dynamic thread stack memory usage - where will you find such an API??
The per-thread stack normally consumes a tiny amount of memory relative to the entire JVM, so it is usually easy to assign enough to suit your algorithm (e.g. default of 128KB stack size for Windows 64bit JVM whilst 2GB of memory might have been budgeted for the entire JVM)
It would be very limited in power: if your logic actually needed to call a method, but you couldn't due to insufficient memory, then your program would be broken at that point. A StackOverflow exception would actually be the best response.
What you are trying to do could be an anti-design anti-pattern.
A "correct" approach would be to specify program requirements, specify required runtime environment (including minimum/needed memory!), and design your program accordingly for optimal performance and memory usage.
An anti-pattern is to not think about these things appropriately during design and development and just imagine some runtime introspection magic could cover for this. There may exist some (rare!) high-performance-demanding apps which need to drastically rearrange the algorithm at runtime to exactly match the discovered resources - but this is complex, ugly & expensive.
And even then, it would probably be better drive dynamic algorithm changes at a macro-level from the "-Xss" parameter, rather than at a micro-level from the exact stack memory consumption at a location in code.
I hope I am guessing what you are really asking. At first I thought you were asking how many calls deep your call was about to be. In other words, I thought you wanted to know how likely you were to trigger this exception, based on your current method circumstances. Then I decided you really wanted to find out how much stack depth you have to play with. In that case, there is another stack-overflow question that seems to address this, here. What is the maximum depth of the java call stack?
This tells you how to set that as a java command line parameter (to java, not your program).
Either way, I'd like to point out that stack overflow has mainly happened to me when I had an endless recursion. I had written methods (by mistake, of course) that called themselves, and were meant to stop when the problem got solved, but somehow the termination condition was never reached. This puts the method invocation onto the stack over and over until the max is exceeded. Not what I had in mind.
I hope that helps.
As far as I am aware, the stack limit in Java is quite abstract and not intended for measuring. In fact, I suspect that the stack size would vary from machine to machine, based on several factors such as memory.
I've never gotten a program to throw a stack overflow exception except for infinite loops / recursion. I'm scratching my head trying to figure out how it would even be possible to throw a stack overflow exception without an infinite loop. If your program is calling that many methods, then it is likely creating objects simultaneously, and you are much more likely to receive an OutOfMemory error than a stack overflow exception without infinite loop.
In fact, what the heck would be the point of a stack limit that could limit your ability to function properly? Java has memory limits to take care of you going overboard with resources. The purpose of stack overflow is to catch loops/recursion that have run amok and need to be caught.
The point I'm trying to make is: if stack overflow exceptions plague your unit testing, you ought to check those loops/recursive functions for some out of control behavior. The call stack is very, very long and I doubt you've reached it naturally.
Well, you can use something like it exists in C with Microsoft C++ compiler :
a specific function (i don't remember the name) which is called automatically on each start and end function.
Also, you count the number of calls and subcalls by increment and decrement the global counter after the start function and before the end function.
For example, with Microsoft .NET , you can insert some function call to increment and decrement your global counter on each call. It's JIT designed.
You can also use a nosql database in order to store your calls.
Also, there is an another thing : use a log system that automatically trace your calls.
Also, when your call stack is full, sometimes it is caused by a recursive function. With a few lines of code and an object, you can store some propagation on each function on each call.
That solution can be also used for detect in any function a special thing : "who is calling me ?"
Also, since Java is a byte-code generated, you can detect the byte-code of a function call and insert before one another function call and after one another function call in order to add your custom stack.
I am trying to display the consumed stack space during a recursive program execution, like inorder traversal of a tree. Is there a way to print the available stack space in Java, I know that available heap memory could be displayed using Runtime.getRuntime().freeMemory().
Java doesn't give you access to this kind of information, for various reasons:
keeping track of that information might actually make some kinds of optimizations impossible
giving an exact number at a certain point might slow down the execution (due to a potentially necessary de-optimization).
the exact amount of available space left might vary over time (for example if stack space were dynamically allocated from a shared space and not fixed).
Any value you might get is very likely to only be of limited value (as you can't know for sure how many bytes a given method invocation needs in various scenarios).
The closest you can get is to get a current stack trace (for example using Thread.getStackTrace()) and check the returned arrays size to know how many stack frames are in use (i.e. how deep the stack already is).
But even that operation is likely to be somewhat costly.
As I know in Java thread stack size depends on JVM and OS Architecture and by default (unless -Xss is set) varies between 256k and 1m. Is there a way or tool allowing to see total stack size consumed by all currently running threads in runtime like I can see Heap Size or Metaspace Size using JVisualVM from JDK package? I understand that this value can be calculated as thread stack size * number of currently running threads however it would be great to monitor this value in runtime.
You can try JProfiler.
If you want to see it live in action before trying it yourself, check this
Also, your idea that thread stack size varies between 256k and 1m is absolutely correct.
In JDK 8, HotSpot installation comes with a feature named Native Memory Tracking (default: disabled). To enable it, use:
-XX:NativeMemoryTracking=[off|detail|summary]
After enabling NMT, you can examine the memory footprint taken by either Thread or Thread Stack using:
jcmd <pid> VM.native_memory [summary | detail | baseline | summary.diff | detail.diff | shutdown] [scale= KB | MB | GB]
First of all, the default stack size is platform specific, but that does not mean that the platform specific default "varies". My recollection is that the defaults have not changed (for any given "architecture") for a long time. (Probably since Java 5, if not earlier. But don't quote me!)
You also can't use the default to accurately determine how much stack memory has actually been allocated, since:
The platform default can be overridden via the -xss command line option. As you noted.
A non-default stack size can be specified when each Thread object is created.
The stack is only actually allocated when the Thread is started. (And it is deallocated when the Thread terminates.)
So if you wanted to measure the actual allocated stack memory you would need to iterate all of the threads and find their actual stack sizes. The first should be relatively straight forwards: traverse the ThreadGroup tree. (I don't think you can guarantee that you will see all threads in a traversal, but that shouldn't matter.) The second is more difficult since there is no getter for the private Thread.stackSize field ... and that field only records the parameter that the application supplied to the Thread constructor.
However, since typical applications just use the default stack size, counting the threads and multiplying by the default size will typically give a good estimate for the total thread stack usage.
It may also be possible to infer a JVM's allocated stack memory by examining the processes memory segments using the methodology of Andrei Pangin's stackmem script. (Noting that this is Linux specific, and that it relies on the JVM requesting individual memory segments from the OS for thread stacks.)
On the other hand, if you wanted to know the amount of stack space currently used (not just allocated), that would be difficult to get from within the application. And if you wanted to get it via an agent, I suspect that you would need to freeze the JVM first. That wouldn't be acceptable for regular monitoring.
But the bottom line is that getting the information will be (at least!) non-trivial and (IMO) probably not worth the effort. There is not a lot that you can do1 with an accurate measure of allocated stack space that you can't already do by looking at thread counts and multiplying ...
... it would be great to monitor this value in runtime.
Not convinced :-)
1 - There are two possible reasons for wanting to know how much stack memory is used: you need to optimize or curiosity. In the former case knowing how much stack memory is used doesn't tell you directly how much ought to be used. To determine the latter, you actually need to determine whether you have too many threads, or if those threads' stacks need to be as big as they currently are. Reducing stack memory usage "on principle" or because some says it is "best practice" could get you into trouble.
Today morning I answered a question which is related to StackoverflowException . The person has asked when Stackoverflow exception occurs
See this link Simplest ways to cause stack overflow in C#, C++ and Java
So my question is that is there any method by which we can compute the method call stacks size dynamically in our program and then applying a check before calling a method which checks whether method call stack has space to accommodate it or not to prevent StackOverflowException.
As I am a java person I am looking for java but also looking for explanation related to the concept without boundation of any programming language.
The total memory available to a JVM is about 2-4GB for a 32bit JVM and the square of this for a 64bit JVM (about 4-16EB). The JVM splits it's memory into:
Heap Memory (allocation controlled via JVM options -Xms and -Xmx)
constructed object and array instances
static classes and array data (including contained object/array instances)
thread instances (object instances, runtime data & metadata including thread object monitor lock references)
Non-Heap Memory
aggregate stack memory
per-thread stack memory (per-thread allocation controlled via JVM option -Xss): method call frames, arguments, return values, locally declared primitives & references to objects
static constants (primitives)
String instance pool
java code: loaded classes and metadata
JVM internal-use memory (JVM code and data structures)
See http://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryMXBean.html and http://www.yourkit.com/docs/kb/sizes.jsp
Is there any method by which we can compute the method call stacks size dynamically in our program
There's no standard method included in Java SE/Java EE to obtain the per-thread stack actual memory usage.
There are standard methods to obtain the aggregate non-heap memory: MemoryMxBean.getNonHeapMemoryUsage(). Referring to this doesn't allow you to make dynamic in-code decisions to avoid StackOverflow exception
There are standard methods to obtain the call stack without it's memory usage: Thread.getStackTrace() ThreadMxBean.getThreadInfo() & ThreadInfo.getStackTrace()
I recommend that you don't do what you suggest in the question because:
You can't do it without some complex JVM-specific API that instruments/introspects on dynamic thread stack memory usage - where will you find such an API??
The per-thread stack normally consumes a tiny amount of memory relative to the entire JVM, so it is usually easy to assign enough to suit your algorithm (e.g. default of 128KB stack size for Windows 64bit JVM whilst 2GB of memory might have been budgeted for the entire JVM)
It would be very limited in power: if your logic actually needed to call a method, but you couldn't due to insufficient memory, then your program would be broken at that point. A StackOverflow exception would actually be the best response.
What you are trying to do could be an anti-design anti-pattern.
A "correct" approach would be to specify program requirements, specify required runtime environment (including minimum/needed memory!), and design your program accordingly for optimal performance and memory usage.
An anti-pattern is to not think about these things appropriately during design and development and just imagine some runtime introspection magic could cover for this. There may exist some (rare!) high-performance-demanding apps which need to drastically rearrange the algorithm at runtime to exactly match the discovered resources - but this is complex, ugly & expensive.
And even then, it would probably be better drive dynamic algorithm changes at a macro-level from the "-Xss" parameter, rather than at a micro-level from the exact stack memory consumption at a location in code.
I hope I am guessing what you are really asking. At first I thought you were asking how many calls deep your call was about to be. In other words, I thought you wanted to know how likely you were to trigger this exception, based on your current method circumstances. Then I decided you really wanted to find out how much stack depth you have to play with. In that case, there is another stack-overflow question that seems to address this, here. What is the maximum depth of the java call stack?
This tells you how to set that as a java command line parameter (to java, not your program).
Either way, I'd like to point out that stack overflow has mainly happened to me when I had an endless recursion. I had written methods (by mistake, of course) that called themselves, and were meant to stop when the problem got solved, but somehow the termination condition was never reached. This puts the method invocation onto the stack over and over until the max is exceeded. Not what I had in mind.
I hope that helps.
As far as I am aware, the stack limit in Java is quite abstract and not intended for measuring. In fact, I suspect that the stack size would vary from machine to machine, based on several factors such as memory.
I've never gotten a program to throw a stack overflow exception except for infinite loops / recursion. I'm scratching my head trying to figure out how it would even be possible to throw a stack overflow exception without an infinite loop. If your program is calling that many methods, then it is likely creating objects simultaneously, and you are much more likely to receive an OutOfMemory error than a stack overflow exception without infinite loop.
In fact, what the heck would be the point of a stack limit that could limit your ability to function properly? Java has memory limits to take care of you going overboard with resources. The purpose of stack overflow is to catch loops/recursion that have run amok and need to be caught.
The point I'm trying to make is: if stack overflow exceptions plague your unit testing, you ought to check those loops/recursive functions for some out of control behavior. The call stack is very, very long and I doubt you've reached it naturally.
Well, you can use something like it exists in C with Microsoft C++ compiler :
a specific function (i don't remember the name) which is called automatically on each start and end function.
Also, you count the number of calls and subcalls by increment and decrement the global counter after the start function and before the end function.
For example, with Microsoft .NET , you can insert some function call to increment and decrement your global counter on each call. It's JIT designed.
You can also use a nosql database in order to store your calls.
Also, there is an another thing : use a log system that automatically trace your calls.
Also, when your call stack is full, sometimes it is caused by a recursive function. With a few lines of code and an object, you can store some propagation on each function on each call.
That solution can be also used for detect in any function a special thing : "who is calling me ?"
Also, since Java is a byte-code generated, you can detect the byte-code of a function call and insert before one another function call and after one another function call in order to add your custom stack.
I've been looking recently on a deep stack with 2000 recursive function calls, and was wondering how is the limit defined on the number of function calls in the stack in Java?
The limit is based on the maximum amount of stack memory that is allocated to each thread. You can configure this limit using the -Xss JVM argument. See these links for more details:
Hotspot FAQ
Using Xss To Adjust Java Default Thread Stack Size
As the others say, it depends on the number of memory available to the JVM.
Theoretical you could compute from the memory the number of function calls, if (and this is a big if) you know the functions and their memory consumption. As a (unoptimized) recursive function has to place all locals and parameters on the stack this is very dependent on the the function. That means when you got X functions with no locals and one int parameter on the stack you get only X/4 when you got one with 2 int parameters and 2 int local variables (neglecting other stack frame overhead, as exception, return addresses, ...).