Develop Reference

Can a java object be accessed from native code and vice versa?

If an object X exists in java heap, and if I knew the address of the object X on java heap, is it possible for the native code to access this object directly from memory without involving JNI? And vice versa, if java code does know the address of object Y on native heap, can java access it without involving JNI?
To be more precise, "Does the java objects gets stored in memory the same way as the native object or is it any different?". If not, wont byteArray objects in java and native gets stored in the same way?
Please provide your suggestions and references.
EDIT: Might be this one is the right question, why do the objects need to be transferred from java heap to native heap through JNI? Why cant the java heap object is accessible to native heap directly?

Can Java code access native objects? No. Java code is managed by the JVM. (More precisely, it's bytecode, not Java code.) The specification of the JVM does not allow bytecode to access arbitrary memory. Bytecode can't even access arbitrary addresses on the JVM heap. For example, private fields can only be accessed by bytecode in the same class.
Can native code access JVM heap objects directly (without JNI)? Yes. Native code is running in the same process and address space as the JVM. As far as I know, on most operating systems and hardware platforms this means that native code can do whatever it wants in that address space.
Should native code access JVM heap objects directly? Definitely not.
First of all, the JVM specification does not specify the layout of objects on the JVM heap, not even of byte arrays. For example, the JVM may split the array into chunks and transparently translate addresses when bytecode uses the array. If you tried to write native code that accesses the array, you would have to re-implement that translation. Such code may work in one JVM implementation, but probably not in another, or maybe not even in a newer version of the same JVM, or in the same JVM when it runs with a different configuration. That's one reason why you have to use JNI: it gives native code a well-defined "view" of objects on the JVM heap.
Secondly, the JVM garbage collector can move around objects on the heap anytime. Native code should access JVM heap objects through handles. The garbage collector knows about it and updates the handles if necessary. Native code that tries to bypass a handle can never be sure if the object is still there.
A third problem is native code that directly modifies pointers between objects on the JVM heap. Depending on the garbage collector algorithm, this may cause all kinds of problems.
In a nutshell: You probably could access JVM heap objects from native code directly, but you almost certainly shouldn't.

Short answer: No.
Other than being a Java/C++ issue, that contradicts with basic OS concepts. Since each process has its own address space, one process cannot reach any object of others.
This limitation can be mitigated only if the process (that tries to reach other's memory) runs in kernel space and the underlying OS allows operations, or some utility like "shared memory" is involved. Even if this were the case, you will face with virtual address space problem. The same physical portions of memory is addressed with different values in different processes. That's why, if you think that you know the address of an object, this address is virtual and useless in other processes.
EDIT: If they are not in different processes, then the answer is definitely yes. Theoretically, you can implement your own JNI :).

a possible answer is using the APR (Apache Portable Runtime) yeah i know it's JNI based but it have concept of Shared memory. so it's possible to bind a shared memory space created by another program (and vice-versa)
https://apr.apache.org/docs/apr/1.5/group__apr__shm.html
ouside of the JNI part, this not seems possible.

ChronicleMap (and more general off-heap data structures) implementation?

ChronicleMap on OpenHFT's repository on Github states in their documentation:
Chronicle Map implements the java.util.concurrent.ConcurrentMap, that stores
its data off the java heap.
I've built a compiler and contributed to a few off-shoot languages' compiler implementation. The one's I've worked with allocate everything on the stack (that's what's available during code generation). I've never worked on the JVM and the java compiler, but I do know that typically only the heap and stack are available to allocate instances of classes, local variables, function parameters, etc.
Could someone please explain how we're able to write code, where we can tell the compiler to instantiate data-structures such as the ChronicalMap, have them available to garbage collection by the JVM (and be kept track-of with JVM's general memory management features), but live off the heap?
I've read up on the simple construction documentation and the associate example. I see the how but the reasoning underlying what exactly is going on in-conjunction with the JVM is unclear.

An important thing to remember is that the javac compiler doesn't do much in the way of optimisation, nor does it give you any means of specifying where data is stored or how code should be optimised. (With a few obscure exceptions in Java 8 like #Contended)
Java derives much of it's extensibility from libraries which generally operate at runtime. (There is often a build time option as well) A key thing to realise is that a Java program can generate and alter code while it is running, so in fact much of the smarts are happening at runtime.
In the case of off-heap usage, you need a library which supports this functionality and this will directly, or indirectly use sun.misc.Unsafe (On most popular JVMs) This class allows you to do many things the language doesn't support, but is still really useful to have if you are a low level library builder.
While off heap memory is not directly managed by the GC, you can have proxy objects e.g. ByteBuffer which have a Cleaner so that when these objects are GC-ed the off heap memory associated with it is also cleaned up.
Disclaimer, I wrote most of ChronicleMap.

The term off heap refers to the ability to use "raw" memory buffers in java. these maybe regular memory buffers from the process address space, or memory mapped files.
These buffers are "raw" - you manage their content yourself - they are not managed by the garbage collector.

JVM memory usage out of control

I have a Tomcat webapp which does some pretty memory and CPU-intensive tasks on the behalf of clients. This is normal and is the desired functionality. However, when I run Tomcat, memory usage skyrockets over time to upwards of 4.0GB at which time I usually kill the process as it's messing with everything else running on my development machine:
I thought I had inadvertently introduced a memory leak with my code, but after checking into it with VisualVM, I'm seeing a different story:
VisualVM is showing the heap as taking up approximately a GB of RAM, which is what I set it to do with CATALINA_OPTS="-Xms256m -Xmx1024".
Why is my system seeing this process as taking up a ton of memory when according to VisualVM, it's taking up hardly any at all?
After a bit of further sniffing around, I'm noticing that if multiple jobs are running simultaneously in the applications, memory does not get freed. However, if I wait for each job to complete before submitting another to my BlockingQueue serviced by an ExecutorService, then memory is recycled effectively. How can I debug this? Why would garbage collection/memory reuse differ?

You can't control what you want to control, -Xmx only controls the Java Heap, it doesn't control consumption of native memory by the JVM, which is consumed completely differently based on implementation. VisualVM is only showing you what the Heap is comsuming, it doesn't show what the entire JVM is consuming as native memory as an OS process. You will have to use OS level tools to see that, and they will report radically different numbers, usually much much larger than anything VisualVM reports, because the JVM uses up native memory in an entirely different way.
From the following article Thanks for the Memory ( Understanding How the JVM uses Native Memory on Windows and Linux )
Maintaining the heap and garbage collector use native memory you can't control.
More native memory is required to maintain the state of the
memory-management system maintaining the Java heap. Data structures
must be allocated to track free storage and record progress when
collecting garbage. The exact size and nature of these data structures
varies with implementation, but many are proportional to the size of
the heap.
and the JIT compiler uses native memory just like javac would
Bytecode compilation uses native memory (in the same way that a static
compiler such as gcc requires memory to run), but both the input (the
bytecode) and the output (the executable code) from the JIT must also
be stored in native memory. Java applications that contain many
JIT-compiled methods use more native memory than smaller applications.
and then you have the classloader(s) which use native memory
Java applications are composed of classes that define object structure
and method logic. They also use classes from the Java runtime class
libraries (such as java.lang.String) and may use third-party
libraries. These classes need to be stored in memory for as long as
they are being used. How classes are stored varies by implementation.
I won't even start quoting the section on Threads, I think you get the idea that
-Xmx doesn't control what you think it controls, it controls the JVM heap, not everything
goes in the JVM heap, and the heap takes up way more native memory that what you specify for
management and book keeping.
Plain and simple the JVM uses more memory than what is supplied in -Xms and -Xmx and the other command line parameters.
Here is a very detailed article on how the JVM allocates and manages memory, it isn't as simple as what you are expected based on your assumptions in your question, it is well worth a comprehensive read.
ThreadStack size in many implementations have minimum limits that vary by Operating System and sometimes JVM version; the threadstack setting is ignored if you set the limit below the native OS limit for the JVM or the OS ( ulimit on *nix has to be set instead sometimes ). Other command line options work the same way, silently defaulting to higher values when too small values are supplied. Don't assume that all the values passed in represent what are actually used.
The Classloaders, and Tomcat has more than one, eat up lots of memory that isn't documented easily. The JIT eats up a lot of memory, trading space for time, which is a good trade off most of the time.

You should also check for CPU usage and garbage collector.
It is possible that garbage collection pauses and the CPU gc consumes further slow down your machine.

Multiple threads and garbage collection in Java

Can someone give me some advice on this? I am reading in an old text and some notes from my teacher that when using multiple threads with Java it's necessary to write a special program for garbage collection.
Does this still apply in Java SE6 and above? If it does could someone provide the standard way to do this.

Using a garbage collector makes writing multi-threaded code easier. This is because manual freeing of resources in a multi-threaded context is hard to get right. With GC its something you don't need to worry about most of the time.
I am reading that when using multiple threads it's necessary to write a special program for garbage collection.
I don't believe this was ever the case.
Does this still apply in SE6 and above and if so is there a standard way to do this.
The standard way to do this is to not reference objects you don't need. e.g. if you have a local variable you don't need, let it drop out of scope.
It doesn't have to be complicated.

As far as I know, as long if nothing is pointing to an object, that object get's freed by the garbage collector.
Java's garbage collector is very robust in terms of circular referencing, I don't see why It won't work with multiple threads running at the same time.
So it is safe for you to assume that you don't need to write a special program for garbage collection, because java will do it for you very effectively.
If you want to free objects in java, just make sure that no variables are referencing your object. (Including structures (lists, arrays, etc) from java collections or other libraries)

This article from JavaWorld in 2003, J2SE 1.4.1 boosts garbage collection, has this to say about the Java garbage collection prior to J2SE 1.4.1:
Mark and sweep is a "stop-the-world" garbage collection technique;
that is, all application threads stop until garbage collection
completes, or until a higher-priority thread interrupts the garbage
collector. If the garbage collector is interrupted, it must restart,
which can lead to application thrashing with little apparent result.
The other problem with mark and sweep is that many types of
applications can't tolerate its stop-the-world nature. That is
especially true of applications that require near real-time behavior
or those that service large numbers of transaction-oriented clients.
An article in Dr. Dobbs from 2009, G1: Java's Garbage First Garbage Collector, has this to say about Java garbage collector before SE 6.
Until recently, Java SE came with two main collectors: the parallel
collector, and the concurrent-mark-sweep (CMS) collector -- see the
sidebar Parallelism and Concurrency. As of the latest Java SE 6 update
release, the G1 collector is another option. The plan is for G1 to
eventually replace CMS as a low-pause, soft real-time collector. Let's
take a look at how it works.
So it may be that prior to SE 6 some additional precautions to assist with Java garbage collection may have helped, especially with multi-threaded applications with a fair amount of temporary variables generating garbage that needed collecting. However this should entail at most an explicit call to the garbage collector during slow times. Writing something special would seem very unusual.
However things are much more improved than they were. Plus garbage collection can vary between different versions of Java Virtual Machines.
So what may have been true years ago is almost definitely not true now with current technology.
This posting, How to monitor Java memory usage?, discusses monitoring Java memory usage as well as some of the pros and cons of calling the garbage collector explicitly.
Oracle has a Java Garbage Collection Basics tutorial that covers Java SE 7 Hotspot JVM.

Use following code to call garbage collector explicitly
Runtime runtime = Runtime.getRuntime();
runtime.gc();
But it is not needed, jvm will automatically handle correct timely running of GC.

Almost certainly your instructor's notes are stating (correctly) that since Java is a multithreaded environment, more care is needed when implementing the garbage collector inside the Java run time environment than would be necessary if only a single thread were involved. This is true of any multithreaded environment.
As others have said, you the programmer don't see any of this complexity. That's the gift of automatic memory management that gc provides.

What does JVM flag CMSClassUnloadingEnabled actually do?

I cannot for the life of me find a definition of what the Java VM flag CMSClassUnloadingEnabled actually does, other than some very fuzzy high-level definitions such as "gets rid of your PermGen problems" (which it doesn't, btw).
I have looked on Sun's/Oracle's site, and even the options list doesn't actually say what it does.
Based upon the name of the flag, I'm guessing that the CMS Garbage Collector doesn't by default unload classes, and this flag turns it on - but I can't be sure.

Update This answer is relevant for Java 5-7, Java 8 has this fixed: https://blogs.oracle.com/poonam/about-g1-garbage-collector,-permanent-generation-and-metaspace Kudos go to mt.uulu
For Java 5-7:
The standard Oracle/Sun VM look on the world is: Classes are forever. So once loaded, they stay in memory even if no one cares anymore. This usually is no problem since you don't have that many purely "setup" classes (= used once for setup and then never again). So even if they take up 1MB, who cares.
But lately, we have languages like Groovy, that define classes at runtime. Every time you run a script, one (or more) new classes are created and they stay in PermGen forever. If you're running a server, that means you have a memory leak.
If you enable CMSClassUnloadingEnabled the GC will sweep PermGen, too, and remove classes which are no longer used.
[EDIT] You will also have to enable UseConcMarkSweepGC (thanks to Sam Hasler). See this answer: https://stackoverflow.com/a/3720052/2541

According to the blog post The most complete list of -XX options for Java JVM, it determines if class unloading is enabled under the CMS garbage collector. The default is false. There is another option called ClassUnloading that is true by default which (presumably) affects the other garbage collectors.
The idea is that if the GC detects that a previously loaded class is no longer used anywhere in the JVM, it can reclaim the memory used to hold the classes bytecode and/or native code.
Setting CMSClassUnloadingEnabled might help with your permgen problem if you are currently using the CMS collector. But the chances are that you are not using the CMS, or that you have a genuine classloader related memory leak. In the latter case, your class will never appear to the GC to be unused ... and will therefore never be unloaded.
Aaron Digulla says "classes are for ever". This is not strictly true, even in the purely Java world. In fact, the lifetime of a class is tied to its classloader. So if you can arrange that a classloader is garbage collected (and that is not always an easy thing to do) the classes that it loaded will also be garbage collected.
In fact, this is what happens when you do a hot redeploy of a webapp. (Or at least, that's what should happen, if you can avoid the problems that lead to a permgen storage leak.)

An example where this is useful:
Setting -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled on our Weblogic 10.3 JVM helped resolving a problem where the JAX-WS implementation created a new proxy class for every web service call, eventually leading to out of memory errors.
It wasn't trivial to trace. The following code always returned the same proxy class for port
final MyPortType port =
Service.create(
getClass().getResource("/path/to.wsdl"),
new QName("http://www.example.com", "MyService"))
.getPort(
new QName("http://www.example.com", "MyPortType"),
MyPortType.class);
Internally, this proxy delegated to an instance of weblogic.wsee.jaxws.spi.ClientInstance, which again delegated to a new $Proxy[nnnn] class where n was incremented at every call. When adding the flags, n was still incremented, but at least those temporary classes were removed from memory.
On a more general note, this can be very useful when making heavy use of Java reflection and proxies through java.lang.reflect.Proxy