Is there any Java api available which would help in simulating a fixed amount of memory being used ??
I am building a dummy application that contains no implementations in its methods. All i would like to do within this methods is simulate a certain amount of memory being used up - is this at all possible?
The simplest way to consume a fixed amount of memory is to create a byte array of that size and retain it.
byte[] bytes = new byte[1000*1000]; // use 1 MB of memory.
Could get tricky with the way Java handles memory, considering applications are run through the runtime environment, don't know if it's going to the heap, etc.
One simple way might be loading text files into memory of the specific sizes you want, then somehow making sure they don't get garbage collected once the method returns.
Related
I am creating two arrays in c++ which will be read in java side:
env->NewDirectByteBuffer
env->NewByteArray
Do these functions copy the buffer I send it?
Do I need to create the buffer on the heap in the c++ side or is it ok to create it on the stack because the jvm will copy it?
for example will this code run ok:
std::string stam = "12345";
const char *buff = stam.c_str();
jobject directBuff = env->NewDirectByteBuffer((void*)buff, (jlong) stam.length() );
Another example:
std::string md5 "12345";
jbyteArray md5ByteArray = env->NewByteArray((jsize) (md5.length()));
env->SetByteArrayRegion(md5ByteArray, 0, (jsize) (md5.length()), (jbyte*)
md5.c_str());
string is created on the stack. Will this code always work or do I need to create those strings on the heap and be responsible to delete it after java finishes using it
Your use of DirectByteBuffer will almost certainly fail in spectacular, core-dumping, and unpredictable ways. And its behavior may vary between JVM implementations and operating systems. The problem is that your direct memory must remain valid for the lifetime of the DirectByteBuffer. Since your string is on the stack, it will go out of scope rather quickly. Meanwhile the Java code may or may not continue to use the DirectByteBuffer, depending on what it is. Are you writing the Java code too? Can you guarantee that its use of the DirectByteBuffer will be complete before the string goes out of scope?
Even if you can guarantee that, realize that Java's GC is non-deterministic. It is all too easy to think that your DirectByteBuffer isn't being used any more, but meanwhile it is wandering around in unreclaimed objects, which eventually get hoovered up by the GC, which may call some finalize() method that accidentally touches the DirectByteBuffer, and -- kablooey! In practice, it is very difficult to make these guarantees except for blocks of "shared memory" that never go away for the life of your application.
NewDirectByteBuffer is also not that fast (at least not in Windows), despite the intuitive assumption that performance is what it is all about. I've found experimentally that it is faster to copy 1000 bytes than it is to create a single DirectByteBuffer. It is usually much faster to have your Java pass a byte[] into the C++ and have the C++ copy bytes into it (ahem, assuming they fit). Overall, I make these recommendations:
Call NewByteArray() and SetByteArrayRegion(), return the resulting
jBytearray to Java and have no worries.
If performance is a
requirement, pass the byte[] from Java to C++ and have C++ fill it
in. You might need two C++ calls, one to get the size and the next
to get the data.
If the data is huge, use NewDirectBtyeBuffer and
make sure that the C++ data stays around "forever", or until you are
darn certain that the DirectByteBuffer has been disposed.
I've also read that both C++ and Java can memory-map the same file, and that this works very well for large data.
NewDirectByteBuffer: "Allocates and returns a direct java.nio.ByteBuffer referring to the block of memory starting at the memory address address and extending capacity bytes.
"Native code that calls this function and returns the resulting byte-buffer object to Java-level code should ensure that the buffer refers to a valid region of memory that is accessible for reading and, if appropriate, writing. An attempt to access an invalid memory location from Java code will either return an arbitrary value, have no visible effect, or cause an unspecified exception to be thrown.".
No copying there.
New<Primitive>Array: only arguments are JNIEnv * and length, so there is nothing to copy.
Set<Primitive>Array: "A family of functions that copies back a region of a primitive array from a buffer."
I am developing an in memory data structure, and would like to add persistency.
I am looking for a way to do this fast. I thought about dumpping a heap-dump once in a while.
Is there a way to load this java heap dump as is, into my memory? or is it impossible?
Otherwise, other suggestions for fast write and fast read of the entire information?
(Serialization might take a lot of time)
-----------------edited explination:--------
Since my memory might be full of small pieces of information, referencing each other - and so serialization may require me to in efficeintly scan all my memory. reloading is also possibly problematic.
On the other hand, I can define a gigantic array, and each object I create, I shall put it in the array. Links will be a long number, reperesnting the place in the array. Now, I can just dump this array as is - and also reload it as is.
There are even some jvms like JRockit that utilize the disk space, and so maybe it is possible maybe to dump as is very quickly and to re-load very quicky.
To prove my point, java dump contains all the information of the jvm, and it is produced quickly.
Sorry, but serialization of 4GB isn't even close to being in the seconds dump is.
Also, memory is memory and there are operating systems that allow you a ram memory dump quicky.
https://superuser.com/questions/164960/how-do-i-dump-physical-memory-in-linux
When you think about it... this is quite a good strategy for persistant data structures. There is quite a hype about in-memory data bases in the last decade. But why settel for that? What if I want a fibonacci heap - to be "almost persistant". That is, every 5 minutes I will dump the inforamtion (quickly) and in case of a electrical outage, I have a backup from 5 minutes ago.
-----------------end of edited explination:--------
Thank you.
In general, there is no way to do this on HotSpot.
Objects in the heap have 2 words of header, the second of which points into permgen for the class metadata (known as a klassOop). You would have to dump all of permgen as well, which includes all the pointers to compiled code - so basically the entire process.
There would be no sane way to recover the heap state correctly.
It may be better to explain precisely what you want to build & why already-existing products don't do what you need.
Use Serialization. Implement java.io.Serializable, add serialVersionUID to all of your classes, and you can persist them to any OutputStream (file, network, whatever). Just create a starting object from where all your object are reachable (even indirectly).
I don't think that Serialization would take long time, it's optimized code in the JVM.
You can use jhat or jvisualvm to load your dump to analyze it. I don't know whether the dump file can be loaded and restarted again.
I have been working on a Java program that generates fractal orbits for quite some time now. Much like photographs, the larger the image, the better it will be when scaled down. The program uses a 2D object (Point) array, which is written to when a point's value is calculated. That is to say the Point is stored in it's corresponding value, I.e.:
Point p = new Point(25,30);
histogram[25][30] = p;
Of course, this is edited for simplicity. I could just write the point values to a CSV, and apply them to the raster later, but using similar methods has yielded undesirable results. I tried for quite some time because I enjoyed being able to make larger images with the space freed by not having this array. It just won't work. For clarity I'd like to add that the Point object also stores color data.
The next problem is the WriteableRaster, which will have the same dimensions as the array. Combined the two take up a great deal of memory. I have come to accept this, after trying to change the way it is done several times, each with lower quality results.
After trying to optimize for memory and time, I've come to the conclusion that I'm really limited by RAM. This is what I would like to change. I am aware of the -Xmx switch (set to 10GB). Is there any way to use Windows' virtual memory to store the raster and/or the array? I am well aware of the significant performance hit this will cause, but in lieu of lowering quality, there really doesn't seem to be much choice.
The OS is already making hard drive space into RAM for you and every process of course -- no magic needed. This will be more of a performance disaster than you think; it will be so slow as to effectively not work.
Are you looking for memory-mapped files?
http://docs.oracle.com/javase/6/docs/api/java/nio/MappedByteBuffer.html
If this is really to be done in memory, I would bet that you could dramatically lower your memory usage with some optimization. For example, your Point object is mostly overhead and not data. Count up the bytes needed for the reference, then for the Object overhead, compared to two ints.
You could reduce the overhead to nothing with two big parallel int arrays for your x and y coordinates. Of course you'd have to encapsulate this for access in your code. But it could halve your memory usage for this data structure. Millions fewer objects also speeds up GC runs.
Instead of putting a WritableRaster in memory, consider writing out the image file in some simple image format directly, yourself. BMP can be very simple. Then perhaps using an external tool to efficiently convert it.
Try -XX:+UseCompressedOops to reduce object overhead too. Also try -XX:NewRatio=20 or higher to make the JVM reserve almost all its heap for long-lived objects. This can actually let you use more heap.
It is not recommended to configure your JVM memory parameters (Xmx) in order to make the operating system to allocate from it's swap memory. apparently the garbage collection mechanism needs to have random access to heap memory and if doesn't, the program will thrash for a long time and possibly lock up. please check the answer given already to my question (last paragraph):
does large value for -Xmx postpone Garbage Collection
I have very large data structures that I define as static fields in a class. I think they get pushed into the heap because my code fails with that error message (heap memory exceeded). Now, I think I recall there to be a memory segment besides heap and stack that is much larger, called data. Is it possible for me to push the variables in that segment? If so, how is this accomplished? I can't afford to increase the heap size because my program will be used by others.
The only thing you could really possibly mean is the disk -- actually writing things to files.
I have a loop that spawns a lot of threads. These threads contains, among other things, 2 (massive) StringBuilder objects. These threads then run and do their thing.
However, I noticed that after a certain amount of threads, I get strange crashes. I know this is because of these StringBuilder, because when I reduce their initial capacity, I can start a lot more threads. Now for these StringBuilders, they are create like this in the constructor of the thread object:
StringBuilder a = new StringBuilder(30000);
StringBuilder b = new StringBuilder(30000);
The point where it generally crashes is around 550 threads, which results in a little bit more than 62MB. Combined with the rest of the program the memory in use is most likely 64MB, which I read online somewhere was the defaulf size of the JVM memory allocation pool. I don't know whether this is true or not.
Now, is there something I am doing wrong, that somehow because of the design, I am allocating memory the wrong way? Or is this the only way and should I tell the JVM to increase its memory pool? Or something else entirely?
Also, please do not tell me to set a lower capacity, I know these StringBuilders automatically increase their capacity when needed, but I would like to have a solution to this problem.
Use the -Xmx JVM option to increase the Java maximum heap size.
Before J2SE 5.0, the default maximum heap size was 64MB. You can override this default using the -Xmx command-line option.
If you are storing a lot of information in a StringBuilder, are you going to reference back to it at some point? If not then just write it to another medium (DB, File etc). The program has a finite amount of resources, it can't hold all of the state of an entire system at once. -Xmx will give you more space for storage in the memory, however it won't make your storage ability infinite.
Consider using a ThreadPoolExecutor, and set the pool size to the number of CPUs on your machine. Creating more threads than CPUs is just adding overhead.
ExecutorService service = Executors.newFixedThreadPool(cpuCount))
Also, you can reduce memory usage by writing your strings to files instead of keeping them in-memory with StringBuilders.
Assume 1MB per thread. That's the RAM cost of creating each one, over and above the memory allocated by its process.
As Gregory said, give the jvm, some options like -Xmx.
Also consider using a ThreadPool or Executor to ensure that only a given amount of threads are running simultaneously. That way the amount of memory can be kept limited without slowdown (as your processor is not capable of running 550 threads at the same time anyway).
And when you're using an Executor don't create the StringBuilders in the constructor, but in the run method.
You can use a FileWriter to output text to a file, then pull it back in with a FileReader. That way you'll only need to store the filename in memory, rather then the entire contents of the string.
To cut down on threads you can use an ExecutorService, or simply use a few threads that read out of a queue.
My guess is that with a little tinkering you can probably get your program down to not needing much memory at all.