Develop Reference

java.lang.Error: Invalid memory access on 32 bit Windows

I have JNA wrapper for a C DLL. It works fine, except when used on a Windows 32-bit system. Here is a simplified example:
int SetData(const wchar_t* data);
int SetId(const wchar_t* id, uint32_t flags);
I created JNA bindings as follows:
public static native int SetData(WString data);
public static native int SetId(WString id, int flags);
The first function SetData() works fine on both 32-bit as well as 64-bit Windows, but the second function crashes on Windows 7 32-bit.
I tried using NativeLong as suggested in other related posts, but it didn't help.
Here is the link to the repository:
https://github.com/cryptlex/lexactivator-java/blob/master/src/main/java/com/cryptlex/lexactivator/LexActivatorNative.java

Your mappings are correct: WString is the correct mapping for const wchar_t*, and int (always 32 bits in Java) is the correct mapping for uint32_t (always 32 bits), with a caveat about signededness that shouldn't matter when used as a flags bitmask.
I'm not sure where you read that NativeLong would be appropriate here. This is primarily intended for *nix native code in which sizeof(long) differs based on the OS bitness. Practically, it doesn't actually matter on Windows since LONG is always 32-bit, but it involves unnecessary object creation vs. a primitive.
The "Invalid Memory Access" error thrown by JNA is a "graceful" handling of native memory errors caught by Structured Exception Handling. All you really need to know is that either You are attempting to access native memory that you do not own or Your native memory allocation failed.
Debugging these errors always involves carefully tracking memory allocations. When is memory allocated? When is it released? Who (Java-side or native side) is responsible for this allocation? Your program is likely crashing at the point you attempt to copy data from the user-provided ID string to some native memory that your native DLL is accessing. So that points to two memory pointers you need to investigate.
Look at the memory where the ID string is being written. Find out when the memory for it is allocated. Ensure it is large enough for the string (should be 2x string length + 2 bytes for a null terminator) and properly zeroed (or a null byte explicitly appended). Verify all WinAPI calls use the correct (W vs. A) unicode version.
I tried adding LA_IN_MEMORY to the flags bitmask and got an error message "Either trial has not started or has been tampered! Trial days left: 30". This is apparently produced by the next line (IsLicenseGenuine()), meaning that the setProductId() call was successful.
Identifying what your native code does differently when the LA_IN_MEMORY flag is not set will probably be very helpful. It's possible the invalid memory access is associated with identifying the directory or file to be used.
There is a recent changelog entry for 3.14.9 involving this flag. Looking at that commit might give a hint to the problem.
There's another recent change in 3.15.0 involving auto-detection of a file on Windows which also may be suspicious given that LA_IN_MEMORY makes the problem go away.
When given an invalid key, the error message "43: The product id is incorrect." is returned, so the point in native code where unowned memory is being accessed is after this error check.
Trace what is happening with the ID string defined on the Java side. Given the constant definition of the string, the actual memory allocation is likely not a problem, but keep track of the native pointer to this string to be sure it's not inadvertently overwritten.
As you've noted in the comments, reducing the native memory allocation solves this issue, indicating you are hitting a limit. It turns out the default 32-bit Java native memory allocation for stack size (-Xss) is 320 KB. From Oracle docs:
On Windows, the default thread stack size is read from the binary
(java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and
1024k in the 64-bit VM.
You can reduce your stack size by running with the -Xss option. For
example:
java -server -Xss64k
Note that on some versions of Windows, the OS may round up thread
stack sizes using very coarse granularity. If the requested size is
less than the default size by 1K or more, the stack size is rounded up
to the default; otherwise, the stack size is rounded up to a multiple
of 1 MB.
You could increase this limit to solve the problem or, as you've indicated in the remarks, lower your native allocation. You might wish to be more conservative than 300K as that only leaves a small amount for other use of the stack. You might also start smaller, check the return value for ERR_MORE_DATA and try again with a larger value. 300KB seems a rather huge amount to devote to registry values.
Note also that 32-bit Java has a total process memory size limit of either 2GB or 4GB, depending on the OS. If your Java heap allocation grows close to that limit, it reduces the native amount available to you. You can control how big the heap gets with the -Xmx switch and you can also ensure sufficient native memory allocation with the -Xss switch. Use these switches in a combination to avoid hitting the process size limit.

How does one find the amount of memory remaining to allocate objects?

How do you tell how much memory the JVM has left to use?
I know Runtime.getRuntime().freeMemory() would be the way to go, but it does not seem to return a reasonable value. For example when I run my main method, I print the free memory and get 78904576 bytes, which is only 75 megabytes. Since I have maxMemory set at 2 gigabytes that seems unreasonable.
Honestly, I just want to know how close I am to running out of memory, so an answer that provides an alternate mechanism for that would also be acceptable.

This is the formula
long availableMemory = Runtime.getRuntime().freeMemory() + (Runtime.getRuntime().maxMemory() - Runtime.getRuntime().totalMemory())
freeMemory() shows free memory only from currently allocated memory, to get all available memory we must also include memory which has not been allocated yet.
Another solution
MemoryUsage mu = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage();
long availableMemory = mu.getMax() - mu.getUsed();

If you really want to monitor the JVM memory you are probably better off using a profiling tool like YourKit or JProfilier.

How to calculate the max object size if max heap size is known?

I ran this test with -Xmx256M to determine the max object size that I can create on heap
for (int m = 128;; m++) {
try {
byte[] a = new byte[m * 1024 * 1024];
} catch (OutOfMemoryError e) {
System.out.println(m + "M");
break;
}
}
and got 171M. Is there a way to calculate this size?

Is there a way to calculate this size?
No. The maximum allocatable object size depends on the amount of contiguous free space available in the heap. AFAIK, there's no practical way of finding out what that might be ... apart from doing what you are currently doing.

This is interesting, I run the same thing with :
java -Xmx256M -XX:+AggressiveHeap HeapTest //I hoped this would help
But I constantly get less then with AggressiveHeap.

That solution is a bad idea. Any other processes running in the JVM may fail to allocate memory (because this code grabbed it all) and that is likely to cause problems.
System performance may be impaired because processes start disk swapping. Depending on how the JVM works it may do a lot of work to initialise memory that you never access.
You might try:
Runtime.getRuntime().freeMemory()
Runtime.getRuntime().maxMemory()
Runtime.getRuntime().totalMemory()
Refer to the documentation for the limitations of what they return.
Also remember that while you can obtain these values in bytes the size of Java objects is implementation defined and remains obscure.
If you're planning to use this you might also need Instrumentation.getObjectSize() to understand the approximate size of objects. Accessing instrumentation is frankly more trouble than it should be. Here's an answer about how to: In Java, what is the best way to determine the size of an object?
All of these values are guidelines only. They may make sense in a caching strategy in which you want to tune how much space is allocated to a discardable cache but not as exact values for any purpose.

Setting a smaller JVM heap size within a JNI client application

I'm attempting to debug a problem with pl/java, a procedural language for PostgreSQL. I'm running this stack on a Linux server.
Essentially, each Postgres backend (connection process) must start its own JVM, and does so using the JNI. This is generally a major limitation of pl/java, but it has one particularly nasty manifestation.
If native memory runs out (I realise that this may not actually be due to malloc() returning NULL, but the effect is about the same), this failure is handled rather poorly. It results in an OutOfMemoryError due to "native memory exhaustion". This results in a segfault of the Postgres backend, originating from within libjvm.so, and a javacore file that says something like:
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
1TIDATETIME Date: 2012/09/13 at 16:36:01
1TIFILENAME Javacore filename: /var/lib/PostgreSQL/9.1/data/javacore.20120913.104611.24742.0002.txt
***SNIP***
Now, there are reasonably well-defined ways of ameliorating these types of problems with Java, described here:
http://www.ibm.com/developerworks/java/library/j-nativememory-linux/
I think that it would be particularly effective if I could set the maximum heap size to a value that is far lower than the default. Ordinarily, it is possible to do something along these lines:
The heap's size is controlled from the Java command line using the -Xmx and -Xms options (mx is the maximum size of the heap, ms is the initial size). Although the logical heap (the area of memory that is actively used) can grow and shrink according to the number of objects on the heap and the amount of time spent in GC, the amount of native memory used remains constant and is dictated by the -Xmx value: the maximum heap size. Most GC algorithms rely on the heap being allocated as a contiguous slab of memory, so it's impossible to allocate more native memory when the heap needs to expand. All heap memory must be reserved up front.
However, it is not apparent how I can follow these steps such that pl/java's JNI initialisation initialises a JVM with a smaller heap; I can't very well pass these command line arguments to Postgres. So, my question is, how can I set the maximum heap size or otherwise control these problems in this context specifically? This appears to be a general problem with pl/java, so I expect to be able to share whatever solution I eventually arrive at with the Postgres community.
Please note that I am not experienced with JVM internals, and am not generally familiar with Java.
Thanks

According to slide 19 in this presentation postgresql.conf can have the parameter pljava.vmoptions where you can pass arguments to the JVM.

Error while initializing Array:OutOfMemoryError

I have to allocate space to an array int input[] depending on the configuration parameters height and width.
int input[]=new int[height * width]; //this is line no 538
One of the configurations has parameters height=8192 and width=8192. So the size of the array becomes 67108864. But when i do this i get OutOfMemoryError.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at Test.main(Test.java:538)
I have ran this program on eclipse as well as on cygwin but i am facing the same problem. I think this is not an error and not exception. How can i rectify this?

Since 8192 * 8192 * 4 = 256 M (integers are 4 bytes each), your matrix is using 256 MB of heap space by itself.
You can tell the JVM how much heap space should be available to your application. From running man java and looking through the nonstandard options:
-Xmxn
Specify the maximum size, in bytes, of the memory allocation
pool. This value must a multiple of 1024 greater than 2MB.
Append the letter k or K to indicate kilobytes, or m or M to
indicate megabytes. The default value is chosen at runtime
based on system configuration. For more information, see
HotSpot Ergonomics
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m
On Solaris 7 and Solaris 8 SPARC platforms, the upper limit for
this value is approximately 4000m minus overhead amounts. On
Solaris 2.6 and x86 platforms, the upper limit is approximately
2000m minus overhead amounts. On Linux platforms, the upper limit
is approximately 2000m minus overhead amounts.
To use this option, you would start your application with a command like
java -Xmxn1024m -jar foo.jar
In Eclipse, you can add command-line options as well. This page on eclipse.org describes how to add command-line arguments to a Java program. You should add the -Xmxn1024m (or some other sufficiently large heap specification) to the "VM arguments" section of the dialog shown on that site.

You probably have too little heap space to hold an array of the size you are targeting. You can increase the size of your heap with command line switches. For example, to set it to 256MB, include this switch:
-Xmx256m
If you multiply height * width * 4 (4 is the storage in bytes for an int) you can get a rough gauge of the amount of heap you will need, assuming the rest of the program does not need a significant amount. You will certainly need some more heap than that quick calculation suggests. Add maybe 20% extra, and try that out.
To get a better number than a rule-of-thumb calculation, you can look into heap profilers. There are several open source options:
http://java-source.net/open-source/profilers
See http://javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.html for a good discussion of the heap in Java.

memory is not enough for your program, may be memory leak there.
you may try below,if not solve try to increase jmx value.
java -xmx1g -xms512m

Depends on how much heap the JVM has. If you run it on the command line try adding -Xmx512m. If you work in an IDE add it to the "Run" properties.

An int is 32 bits (i.e. 4 bytes). So your array requires 8192*8192*4 bytes. This comes out at 256MB.
Java called with default arguments has only 64MB of heap space.
To get a larger heap, call Java using the -Xmx argument (Maximum memory size).
e.g. java -Xmx300M

Increase your memory arguments for your Java process by adding this flag to increase the heap. You might need to play around to get the optimal size for the heap. This will set the "max" heap size. The default is probably really small. 64M is a common max size for many Java EE containers.
*Note I'm not saying this is exactly the size you'll need. Your unique case will dictate the size you'll need which you may need to experiment with.
-Xmx256M