This is a memory data security question.
Does java garbage collection securely wipe out garbage data?
Apparently after a chunk of data is garbage-collected, I cannot retrieve it anymore, but can a hacker still memory-dump to retrieve the data?
As other users already mentioned here, JVMs don't clean memory securely after garbage collection because it would affect performance badly. That's why many programs (especially security libraries) use mutable structures instead of immutable (char arrays instead of strings etc) and clean data themselves when they are no more needed.
Unfortunately, even such approach doesn't always work. Let's look at this scenario:
You create a char array with a password.
JVM performs a garbage collection and moves your char array to another place in the memory, leaving the memory previously occupied by it intact, just marking it as a free block. So, we now have a 'dirty copy' of your password.
You are done working with your password and explicitly zero all the characters in your char array thinking that everything is safe now.
An attacker makes a dump of your memory and finds your password in the memory where it was placed the first time, before step 2.
I can think of only one possible solution for this problem:
Use G1 garbage collector.
Make your sensitive data a single block (array of primitive values) that is large enough to occupy more than half of the region size, used by G1 (by default, this size depends on the maximum heap size, but you can also specify it manually). That would force the collector to treat your data as so called 'humongous object'. Such objects are not moved in memory by G1 GC.
In such case when you erase some data in your block manually, you can be sure that no other 'dirty copies' of the same data exist somewhere in the heap.
Another solution would be to use off-heap data that you can handle manually as you like, but that wouldn't be pure Java.
This depends on the JVM implementation and possibly options within it but I would assume that it won't clear the data. Garbage collection needs only track which areas are available. Setting all of that data to 0 or something else is a lot of unecessary writes. It's for this reason you will often see APIs use a char array for passwords instead of Strings.
Specifically Oracle JVM won't clear the space, it only copies data between Eden and Survivor spaces, objects that are no longer used just stay there as a garbage that will be overwritten eventually. Similar thing happens in the OldGen, some places are marked as used, and when object becomes eligible for garbage collection, the place it occupied is marked as not used. It will also be overwritten eventaully, given enough application time.
Related
I have some doubts about the garbage collector and how I can clear memory in Java.
I have a program that writes a binary search tree to a file and I made a function that inserts an element and another that removes an element, but in the method that removes I put the elements that I remove in a space in the file that I call "empty blocks" (which is a stack). In the C language there is a method that freed the memory that was free(), in Java there is the garbage collector that is at the discretion of Java. How can I free the memory of these blocks in the file (elements excluded).
Is there a way to free the memory of an element on file in Java (the element is of type int)?
I put the elements that I remove in a space in the file that I call “empty blocks ”(Which is a stack)
Whatever data structure you use to track your data will be in an object of some class.
When that object no longer has any references pointing to it, that object becomes a candidate for garbage collection. No need for you to do anything except not hang on to any reference longer than needed.
The garbage collector may clear the unneeded object immediately, or may clear it later. Either way, we as Java programmers do not care. Eventually the memory will be freed up.
If the reference variable pointing to an object is a local variable, that reference is dropped when the local variable goes out of scope.
If the reference variable is a member field on another object, the
object in question will be released when the other object becomes
garbage.
If the reference variable is static, you should assign null explicitly to let the referenced object become garbage. In Java, static variables stay in memory throughout the execution run of your app.
In the first two cases, you can release the object sooner by setting the reference variable to null. Generally this is not needed, but doing so may be wise if a large amount of memory is at stake. Ditto if other precious resources are being needlessly held.
Is there a way to free the memory of an element on file in Java (the element is of type int)?
Your question is really hard to understand, but I think you are asking about freeing up disk blocks in a data structure stored in a file1.
There is no Java support for this. If you write a data structure to a file, the problem of reclaiming space in the file is yours, not Java's. Indeed, I don't think that a typical OS will allow you to (literally) free disk blocks in the middle of a file2.
There may be 3rd-party libraries that support this kind of thing, but I don't have the background knowledge to make a recommendation.
If I have correctly understood what you are asking, your discussion of C's malloc / free versus Java's garbage collection is only peripherally relevant. Both of these schemes are for managing memory, not space in a random access file. Now you could conceivably implement similar schemes for managing space in a file, but you would need to take account of the different characteristics of memory and disk I/O. (Even if you are mapping the file into memory.)
1 - If you are actually talking about managing objects in heap memory in Java, your best bet is to just let the garbage collector deal with it; see Basil's answer. There are also 3rd-party libraries for storing objects in off-heap memory, but it is unclear if they would help you. I understand that such libraries typically leave it to the programmer to decide when to free an object. (They are not garbage collected.)
2 - It would be a bad idea. If the disk blocks thus freed were then used in a different file, you would get a lot of file fragmentation. That would be bad for file I/O performance.
I have a situation where there are 2 files A and B, and data is being written continuously in both of them (like a stream).
Now I know that both files A and B are going to be competing for memory and the garbage collector is going to decide what page for what file will be replaced.
I want to control garbage collection by making the garbage collector favor file A (i.e. garbage collector should always choose eviction of pages of file B compared to A). Other possibility is to force writing of file B to disk instead of caching in memory.
Can these things happen in java?
I suspect you are confusing memory management with garbage collection. Yes, garbage collection is a form of memory management, but it's not what you are talking about when discussing "which pages of memory will be swapped out to disk when memory space is low" That's not garbage collection because there are still active references to the A and B files. The Garbage Collector won't do anything until there are no references to an object.
You want to control memory page swapping not garbage collection. I'm sure I'll be corrected in comments if I'm wrong about this, but I don't think you can control in Java which pages of memory get swapped to disk when available memory is low.
You cannot forcefully ask Java to do garbage collection.
But you can call System.gc() to request the JVM to do a garbage collection.
To make sure an object is ready for garbage collection you can assign it to null. That way you can make sure that when the garbage collector runs it gets this object and is removed from the heap.
Java has automatic garbage collection and identifies which objects are in use and which are not, and deleting the unused objects.
A good source about garbage collection within Java is here
The description of your problem lacks certain details, specifically, are the writes to your files sequential or is there random access involved?
As geneSummons correctly points out, you have memory management in the JVM confused with that of the Operating System. Even sun.misc.Unsafe will not allow you control over paging activity at the OS level from a Java application.
What you may want to look at is using memory mapped files, but that does depend on whether you are using random access for your writes. If all you're doing is writing sequentially this is most likely no use. Although this does not give you control over the paging of the files at the OS level it may provide you with a more efficient way of solving your problem.
There is a useful article on this subject, https://howtodoinjava.com/java-7/nio/java-nio-2-0-memory-mapped-files-mappedbytebuffer-tutorial/
I'm using Guava cache for my application and was wondering the what the default behavior would be if the maximumSize was not set. I understand the behavior when the maximumSize is set as it's explained in https://github.com/google/guava/wiki/CachesExplained#size-based-eviction.
But what happens when the maximumSize is not set and JVM run out of heap space? I assume that garbage collector will run and will free up space which means that entries will be dropped from the cache?
Under the covers a Cache is just a fancy Map, so it has similar space limitations. Like Map it can't contain more than than Integer.MAX_VALUE entries (since that's the return type of size()), so your theoretical upper-bound cache size is ~2 billion elements. You might also be interested in Guava's awesome element-cost analysis, which details the exact number of bytes used by different data structures.
Of course in practice the real concern isn't usually the number of elements in the cache (its size), but the amount of memory consumed by the objects being cached. This is independent from the cache's size - a single cached object could be large enough consume all your heap.
By default Cache doesn't do anything special in this case, and the JVM crashes. Most of the time this is what you want - dropping elements from the cache silently could break your program's assumptions.
If you really do want to drop entries as you approach out-of-memory conditions you can use soft references with CacheBuilder.softValues(). The JVM will attempt to garbage collect soft references when it's at risk of running out of free heap space. I would encourage you to only use this option as a last resort - the JVM has to do extra work to handle soft references, and needing to use them is often a hint that you could be doing something else a different way.
I have a bunch of Objects in an ArrayList, if I call ArrayList.remove(object) Do I need to do anything else to remove the object from memory? I am adding and removing objects from this list at a fairly very quick pace, so if it doesn't get removed from memory I it will start taking up space and start to slow down the Game.
- When you call ArrayList.remove(object), you just remove the objects from the List Not from the Memory.
- It will depend on the Garbage Collector to decide when its gonna remove the object from the heap, under normal circumstances its an object is ready for garbage collection as it has No reference to it anymore.
- There is a classic example of why String which is an object in Java should not be used for storing password instead char[] should be used.
See this link...
Why is char[] preferred over String for passwords?
Java does automatic garbage collection. So once an object is no longer referred to, it can be deleted, that doesn't mean it will be deleted. Garbage collection is automatic, you can ask that it be done by calling System.gc() however this is just a sugestion to run it.
http://docs.oracle.com/javase/1.4.2/docs/api/java/lang/System.html#gc%28%29
No, you don't have to do anything else, as long as that's the only place that's referencing the object. Welcome to the joys of a garbage-collected language! Java will clean up old, unreferenced objects when it decides that it needs to reclaim some memory.
if you chew through the heap quick enough, you can nudge the gc along with some jvm args ... we have an app that handles billions of operations a day and have tuned it heavily with the ergonomic gc settings I'd encourage you to play with the adaptive size policy, and the max pause setting primarily. Run the program with a profiler exercising the arraylist as you normally would for a while (several minutes) and see how the ambient state of the various heap generations looks. You may end up having to also tweak the memory allocations to generations.
ArrayList.remove removes the object from the array, and then if the object is not referenced to by any other objects, the GC will delete that object.
I have memory leak in the web application (servlet) that I am working on. I am suspicious of 1 cause and wanted to hear your ideas about it.
I use hashmaps,hashsets etc. as DB (about 20MB data loaded). These maps,sets get reloaded once in every 10 min. There are huge amount of simultaneous requests. I read that, GC passes objects, that are not collected for a time period/cycle, to a generation (old and permanent generations) which is less checked or garbage collected. I think that my usage for static maps,sets is causing me leak problem. What do you think ?
As Romain noted, the static map is a suspect. If for some reason you can't regularly clean it up explicitly, you may consider using a WeakHashMap instead, which is
A hashtable-based Map implementation with weak keys. An entry in a WeakHashMap will automatically be removed when its key is no longer in ordinary use. More precisely, the presence of a mapping for a given key will not prevent the key from being discarded by the garbage collector, that is, made finalizable, finalized, and then reclaimed. When a key has been discarded its entry is effectively removed from the map, so this class behaves somewhat differently than other Map implementations.
Unfortunately, as of Java6 there seems to be no WeakHashSet in the standard library, but several implementations can be found on the net.
It is not a leak if you have removed all references to it. If you're clearing out your map entirely, then it's not the source of a leak. You should consider the fact that the JVM chooses not to GC tenured generation very often as irrelevant to you - all that matters is that you don't have a reference to it, so the JVM could GC it if it wants to.
There are different strategies that JVMs can use to manage GC, so I'm speaking in generalities instead of specifics here, but GCing tenured spaces tends to be very expensive and has a high impact on the application, so the JVM chooses not to do it often in general.
If you're looking at the amount of heap space used, you'll see a sawtooth pattern as items are added and eventually collected. Don't be concerned about where the top of the sawtooth is, be concerned about where the bottom is (and how close to the maximum amount of heap space available that is).
One way to test if it really is a leak is to load test your app for a long period of time. If you have a leak, the base amount of memory that your app is using will go up over time (the bottom of the sawtooth). If you don't, it will remain constant. If you do have a leak, you can use a profiler to help you find it.
Static Maps are a known source of leaks. The reason being that people put stuff in and do not remove them. If every ten minutes you simply clear the cache and then reload then you should be fine.
I would bet that you are not clearing it properly. The GC part is working properly, I would not worry that it is the issue.
You may also want to consider using WeakReference if you have some way of falling back to the real data if part of your cache is GC-ed but then subsequently required.
I suggest that you check the heap contents using a heap dump and a heap analyzer (such as JVisualVM). This will help you find leakage suspects. The fact that the old generation is collected less frequently does not mean that more memory is leaking; remember that while it may seem full, only a portion of it represents live objects, and the other portion is clearned by the next major GC. Like others said, the problem may be because of incomplete cleaning of the static collections.
The permanent generation never receives promoted objects. It is an out-of-heap area reserved for other purposes, such as reflective information of loaded classes, and interned strings.