I would like to test how many bytes an object reference use in the Java VM that I'm using. Do you guys know how to test this?
Thanks!
Taking the question literally, on most JVMs, all references on 32-bit JVMs take 4 bytes, one 64-bit JVMs, a reference takes 8 bytes unless -XX:+UseCompressedOops has been used, in which case it takes 4-bytes.
I assume you are asking how to tell how much space an Object occupies. You can use Instrumentation (not a simple matter) but this will only give you a shallow depth. Java tends you break into many objects something which is C++ might be a single structure so it is not as useful.
However, ifyou have a memory issue, I suggest you a memory profiler. This will give you the shallow and deep space objects use and give you a picture across the whole system. This is often more useful as you can start with the biggest consumers and optimise those as even if you have been developing Java for ten years+ you will only be guessing where is the best place to optimise unless you have hard data.
Another way to get the object size if you don't want to use a profiler is to allocate a large array and see how much memory is consumed, You have to do this many times to get a good idea what the average size is. I would set the young space very high to avoid GCs confusing your results e.g. -XX:NewSize=1g
It can differ from JVM to JVM but "Sizeof for Java" says
You might recollect "Java Tip 130: Do You Know Your Data Size?" that described a technique based on creating a large number of identical class instances and carefully measuring the resulting increase in the JVM used heap size. When applicable, this idea works very well, and I will in fact use it to bootstrap the alternate approach in this article.
If you need to be fairly accurate, check out the Instrumentation framework.
This one is the one I use. Got to love those 16-byte references !
alphaworks.ibm.heapanalyzer
Related
I'm trying to find a simple and accurate reference for the cost in bytes of Java 64 bit Objects. I've not been able to find this. The primitives are clearly specified but there are all these edge cases and exceptions that I am trying to figure out like padding for an Object and cost vrs. space they actually take up on the heap, etc. From the gist of what I'm reading here: http://btoddb-java-sizing.blogspot.com/ that can actually be different?? :-/
If you turn off the TLAB, you will get accurate accounting and you can see exactly how much memory each object allocation uses.
The best way to see where your memory is being used, is via a memory profiler. Worry about bytes here and there is most likely a waste of time. When you have hundreds of MB, then it makes a difference and the best way to see that is in a profiler.
BTW Most systems use 32-bit references, even in 64-bit JVMs. There is no such thing as a 64-bit Object. Apart from the header, the object will uses the same space whether it is a 32-bit JVM or using 32-bit references in a 64-bit JVM.
You are essentially asking for a simple way to get an accurate prediction of object sizes in Java.
Unfortunately ... there isn't one!
The blog posting you found mentions a number of complicating factors. Another one is that the object sizing calculation can potentially change from one Java release to the next, or between different Java implementation vendors.
In practice, you options are:
Estimate the sizes based on what you know, and accept that your estimates may be wrong. (If you take account of enough factors, you should be able to get reasonable ballpark estimates, at least for a particular platform. But accurate predictions are inherently hard work.)
Write micro benchmarks using the TLAB technique to measure the size of the objects.
The other point is that in most cases it doesn't matter if your object size predictions are not entirely accurate. The recommended approach is to implement, measure and then optimize. This does not require accurate size information until you get to the optimization stage, and at that point you can measure the sizes ... if you need the information.
I have a HashMap that contains 12 million entries. It maps String values to Long values. Each string is about ten characters long. Is it possible to calculate how much memory this map will need in RAM?
You can guess, and you can make an educated guess by looking at the size of each item that goes into the map, but your best educated guess will be wrong.
JVMs have other structure used to track references and hold type information for the classes. That will add a fixed, yet unknown amount of memory to an accurate (if you can come up with an accurate input estimate) estimate.
As only some of the memory is memory directly holding the data, and some of the memory is memory used as overhead to hold the data, you need to profile your memory consumption and based your estimates on projections of the memory "growth" when using smaller maps.
Note that profiling a JVM is a tricky task, as it is optimizing memory usage in a manner that will present varying results depending on how long the JVM is running, the activity of the Map, etc. You need to do statistical sampling of the input in a variety of conditions; but, odds are good you will eventually be able to put your finger on a reasonable number. More importantly, you will also be able to say "Well it might peak up at around this number temporarily, but should settle down to this on average". Temporal changes to memory are often overlooked in static analysis.
The JVM would know because it has to allocate and manage memory but it doesn’t tell you. So, short of going native, no, there is no way to know how much memory is actually used by your objects.
A profiler will tell you how much memory is being used by the program, and which objects are using what. You might be able to find your objects' memory usage.
VisualVM is included in Java6, and will give you this information.
It's worth mentioning, though, this is not necessarily going to give you a memory 'requirement', just a view of how much memory it is using at that point in time.
This question already has answers here:
How to determine the size of an object in Java
(28 answers)
Closed 7 years ago.
I am working on calculaitng the size [memory used] of a java object [hashmap] . It contains elements of different data types [at runtime] so [ no-of-elem * size-of-element] is not that good an approach. The code right now does it by series of
if (x)
do something
else if (primitives)
lookup size and calculate
However this process is a CPU hog and in-efficient.
I am thinking of following 2 approaches instead:
Serialize the object to a buffer and get the size.
Look into java.lang.instrument to get the size
I am looking for anyones experience with these approaches for performance , efficiency, scaling etc OR if you know any better way.
P.S:
This is a background utility that I am building so the size need no be super accurate though it should be about correct. So I am willing to trade accuracy for performance
I am not interested in the deep-size [the size of objects that are refered by this object will not be computed.]
I am looking for a performance comparisons and understanding how getObjectSize() works internally ..so that I do not messup something else to improve the performance
Thanks
Use getObjectSize() method of the Instrumentation package.
Look here for implementation details:
The serialized size is definitely not the way to go, for two reasons:
In the standard java serialization there can be quite a lot of overhead which would add to the size.
It would not be any quicker than using the getObjectSize() method which we can presume will iterate over all the references, and use some kind of lookup to determine the size of the primitive values/references of an object.
If you need better performance then that really will depend on the distribution of your objects. One possiblility would be to do some random sampling of the values in your map, determine an average and calculate an estimate from this value.
For advice on how to look up a random value in a hashmap, see this question.
You may be interested in an article I wrote a while ago on how to calculate the memory usage of a Java object. It is admittedly aimed primarily at 32-bit Hotspot, although much of it applies in essence to other environments.
You can also download a simple agent for measuring Java object size from the same site which will take some of the hard work out of it for you and should work in 64-bit environments.
Note as others have I think mentioned that the serialised form of an object isn't the same as its form in memory, so using serialisation isn't suitable if you want to measure the memory footprint accurately.
right now, i need to load huge data from database into a vector, but when i loaded 38000 rows of data, the program throw out OutOfMemoryError exception.
What can i do to handle this ?
I think there may be some memory leak in my program, good methods to detect it ?thanks
Provide more memory to your JVM (usually using -Xmx/-Xms) or don't load all the data into memory.
For many operations on huge amounts of data there are algorithms which don't need access to all of it at once. One class of such algorithms are divide and conquer algorithms.
If you must have all the data in memory, try caching commonly appearing objects. For example, if you are looking at employee records and they all have a job title, use a HashMap when loading the data and reuse the job titles already found. This can dramatically lower the amount of memory you're using.
Also, before you do anything, use a profiler to see where memory is being wasted, and to check if things that can be garbage collected have no references floating around. Again, String is a common example, since if for example you're using the first 10 chars of a 2000 char string, and you have used substring instead of allocating a new String, what you actually have is a reference to a char[2000] array, with two indices pointing at 0 and 10. Again, a huge memory waster.
You can try increasing the heap size:
java -Xms<initial heap size> -Xmx<maximum heap size>
Default is
java -Xms32m -Xmx128m
Do you really need to have such a large object stored in memory?
Depending of what you have to do with that data you might want to split it in lesser chunks.
Load the data section by section. This will not let you work on all data at the same time, but you won't have to change the memory provided to the JVM.
You could run your code using a profiler to understand how and why the memory is being eaten up. Debug your way through the loop and watch what is being instantiated. There are any number of them; JProfiler, Java Memory Profiler, see the list of profilers here, and so forth.
Maybe optimize your data classes? I've seen a case someone has been using Strings in place of native datatypes such as int or double for every class member that gave an OutOfMemoryError when storing a relatively small amount of data objects in memory. Take a look that you aren't duplicating your objects. And, of course, increase the heap size:
java -Xmx512M (or whatever you deem necessary)
Let your program use more memory or much better rethink the strategy. Do you really need so much data in the memory?
I know you are trying to read the data into vector - otherwise, if you where trying to display them, I would have suggested you use NatTable. It is designed for reading huge amount of data into a table.
I believe it might come in handy for another reader here.
Use a memory mapped file. Memory mapped files can basically grow as big as you want, without hitting the heap. It does require that you encode your data in a decoding-friendly way. (Like, it would make sense to reserve a fixed size for every row in your data, in order to quickly skip a number of rows.)
Preon allows you deal with that easily. It's a framework that aims to do to binary encoded data what Hibernate has done for relational databases, and JAXB/XStream/XmlBeans to XML.
Hmmm. Is there a primer anywhere on memory usage in Java? I would have thought Sun or IBM would have had a good article on the subject but I can't find anything that looks really solid. I'm interested in knowing two things:
at runtime, figuring out how much memory the classes in my package are using at a given time
at design time, estimating general memory overhead requirements for various things like:
how much memory overhead is required for an empty object (in addition to the space required by its fields)
how much memory overhead is required when creating closures
how much memory overhead is required for collections like ArrayList
I may have hundreds of thousands of objects created and I want to be a "good neighbor" to not be overly wasteful of RAM. I mean I don't really care whether I'm using 10% more memory than the "optimal case" (whatever that is), but if I'm implementing something that uses 5x as much memory as I could if I made a simple change, I'd want to use less memory (or be able to create more objects for a fixed amount of memory available).
I found a few articles (Java Specialists' Newsletter and something from Javaworld) and one of the builtin classes java.lang.instrument.getObjectSize() which claims to measure an "approximation" (??) of memory use, but these all seem kind of vague...
(and yes I realize that a JVM running on two different OS's may be likely to use different amounts of memory for different objects)
I used JProfiler a number of years ago and it did a good job, and you could break down memory usage to a fairly granular level.
As of Java 5, on Hotspot and other VMs that support it, you can use the Instrumentation interface to ask the VM the memory usage of a given object. It's fiddly but you can do it.
In case you want to try this method, I've added a page to my web site on querying the memory size of a Java object using the Instrumentation framework.
As a rough guide in Hotspot on 32 bit machines:
objects use 8 bytes for
"housekeeping"
fields use what you'd expect them to
use given their bit length (though booleans tend to be allocated an entire byte)
object references use 4 bytes
overall obejct size has a
granularity of 8 bytes (i.e. if you
have an object with 1 boolean field
it will use 16 bytes; if you have an
object with 8 booleans it will also
use 16 bytes)
There's nothing special about collections in terms of how the VM treats them. Their memory usage is the total of their internal fields plus -- if you're counting this -- the usage of each object they contain. You need to factor in things like the default array size of an ArrayList, and the fact that that size increases by 1.5 whenever the list gets full. But either asking the VM or using the above metrics, looking at the source code to the collections and "working it through" will essentially get you to the answer.
If by "closure" you mean something like a Runnable or Callable, well again it's just a boring old object like any other. (N.B. They aren't really closures!!)
You can use JMP, but it's only caught up to Java 1.5.
I've used the profiler that comes with newer versions of Netbeans a couple of times and it works very well, supplying you with a ton of information about memory usage and runtime of your programs. Definitely a good place to start.
If you are using a pre 1.5 VM - You can get the approx size of objects by using serialization. Be warned though.. this can require double the amount of memory for that object.
See if PerfAnal will give you what you are looking for.
This might be not the exact answer you are looking for, but the bosts of the following link will give you very good pointers. Other Question about Memory
I believe the profiler included in Netbeans can moniter memory usage also, you can try that