I have JNA wrapper for a C DLL. It works fine, except when used on a Windows 32-bit system. Here is a simplified example:
int SetData(const wchar_t* data);
int SetId(const wchar_t* id, uint32_t flags);
I created JNA bindings as follows:
public static native int SetData(WString data);
public static native int SetId(WString id, int flags);
The first function SetData() works fine on both 32-bit as well as 64-bit Windows, but the second function crashes on Windows 7 32-bit.
I tried using NativeLong as suggested in other related posts, but it didn't help.
Here is the link to the repository:
https://github.com/cryptlex/lexactivator-java/blob/master/src/main/java/com/cryptlex/lexactivator/LexActivatorNative.java
Your mappings are correct: WString is the correct mapping for const wchar_t*, and int (always 32 bits in Java) is the correct mapping for uint32_t (always 32 bits), with a caveat about signededness that shouldn't matter when used as a flags bitmask.
I'm not sure where you read that NativeLong would be appropriate here. This is primarily intended for *nix native code in which sizeof(long) differs based on the OS bitness. Practically, it doesn't actually matter on Windows since LONG is always 32-bit, but it involves unnecessary object creation vs. a primitive.
The "Invalid Memory Access" error thrown by JNA is a "graceful" handling of native memory errors caught by Structured Exception Handling. All you really need to know is that either You are attempting to access native memory that you do not own or Your native memory allocation failed.
Debugging these errors always involves carefully tracking memory allocations. When is memory allocated? When is it released? Who (Java-side or native side) is responsible for this allocation? Your program is likely crashing at the point you attempt to copy data from the user-provided ID string to some native memory that your native DLL is accessing. So that points to two memory pointers you need to investigate.
Look at the memory where the ID string is being written. Find out when the memory for it is allocated. Ensure it is large enough for the string (should be 2x string length + 2 bytes for a null terminator) and properly zeroed (or a null byte explicitly appended). Verify all WinAPI calls use the correct (W vs. A) unicode version.
I tried adding LA_IN_MEMORY to the flags bitmask and got an error message "Either trial has not started or has been tampered! Trial days left: 30". This is apparently produced by the next line (IsLicenseGenuine()), meaning that the setProductId() call was successful.
Identifying what your native code does differently when the LA_IN_MEMORY flag is not set will probably be very helpful. It's possible the invalid memory access is associated with identifying the directory or file to be used.
There is a recent changelog entry for 3.14.9 involving this flag. Looking at that commit might give a hint to the problem.
There's another recent change in 3.15.0 involving auto-detection of a file on Windows which also may be suspicious given that LA_IN_MEMORY makes the problem go away.
When given an invalid key, the error message "43: The product id is incorrect." is returned, so the point in native code where unowned memory is being accessed is after this error check.
Trace what is happening with the ID string defined on the Java side. Given the constant definition of the string, the actual memory allocation is likely not a problem, but keep track of the native pointer to this string to be sure it's not inadvertently overwritten.
As you've noted in the comments, reducing the native memory allocation solves this issue, indicating you are hitting a limit. It turns out the default 32-bit Java native memory allocation for stack size (-Xss) is 320 KB. From Oracle docs:
On Windows, the default thread stack size is read from the binary
(java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and
1024k in the 64-bit VM.
You can reduce your stack size by running with the -Xss option. For
example:
java -server -Xss64k
Note that on some versions of Windows, the OS may round up thread
stack sizes using very coarse granularity. If the requested size is
less than the default size by 1K or more, the stack size is rounded up
to the default; otherwise, the stack size is rounded up to a multiple
of 1 MB.
You could increase this limit to solve the problem or, as you've indicated in the remarks, lower your native allocation. You might wish to be more conservative than 300K as that only leaves a small amount for other use of the stack. You might also start smaller, check the return value for ERR_MORE_DATA and try again with a larger value. 300KB seems a rather huge amount to devote to registry values.
Note also that 32-bit Java has a total process memory size limit of either 2GB or 4GB, depending on the OS. If your Java heap allocation grows close to that limit, it reduces the native amount available to you. You can control how big the heap gets with the -Xmx switch and you can also ensure sufficient native memory allocation with the -Xss switch. Use these switches in a combination to avoid hitting the process size limit.
Related
The Maximum allowed int data type array range is 2147483647. But I am getting
Runtime Error: Exception in thread "main" java.lang.OutOfMemoryError:
Java heap space at Test.main(Test.java:3)
Can someone please explain me the memory representation for this size and why JVM raising a runtime error for this?
Code:
class Test{
public static void main(String[] args){
int [] x = new int[1000000000];
}
}
You're hitting a limit of the particular JVM you're using, or your system memory.
You're trying to allocate an array which will take up ~4GB of contiguous memory. I'd expect a 64-bit JVM on a machine with enough memory to handle that without any problems, but different JVMs may have different limitations. For example, I'd expect any 32-bit JVM to struggle with that.
The good news is that allocating an array that large is almost never required in regular programming. If you do need it, you'll need to make sure you work in an environment that supports it.
Even on my machine that can handle your example, if I increase the size further I get one of two errors, either the one you got:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
or for 2147483646 or 2147483647:
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
(The first happens somewhere between 1064000000 and 1065000000 on my machine.)
By default, the int data type is a 32-bit signed two's complement integer. You declare an array of one billion of those.
You can read more about primitive datatypes at https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html
You could look into increasing the available memory (see Increase heap size in Java) , or look into using the Collections API, which might suit your needs better than a primitive array.
You mentioned the biggest number that can be stored in an int i.e. 2147483647.
But you are creating an array of a billion ints. These are two different things.
Basically you are taking up 1000000000 * 4 bytes, because size of one int is 4 bytes. That makes about 4GB of memory!
Usually, this error is thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector.
"int[1000000000]" is consuming a lot of Java heap space and when all of the available memory in the heap region is filled and Garbage Collection is not able to clean it, the java.lang.OutOfMemoryError:Java heap space is thrown.
4GB is a lot of space
I think this is and old question.. well I will explain this:
Your Java code runs on a Java Virtual Machine (JVM) this machine has some limitations by default that is why you get this:
int[] array : 4 byte
0 l=1048576 s=4mb
1 l=2097152 s=8mb
java.lang.OutOfMemoryError: Java heap space l=4194304 s=16mb
How can you "fix" it on your environment?
Well easy just run your app adding a VM flag -Xmx12g.
Note:
Remember that, in practice, a JVM array size is limited by its internal representation. In the GC code, JVM passes around the size of an array in heap words as an int then converts back from heap words to jint this may cause an overflow. So in order to avoid crashes and unexpected behavior the maximum array length is limited by this max size - header size. Where header size depends on which C/C++ compiler was used to build the JVM you are running (i.e. gcc for linux, clang for macos), and runtime settings (i.e. UseCompressedClassPointers). For example, on my Linux environment the limits are:
Java HotSpot(TM) 64-Bit Server VM 1.6.0_45 limit Integer.MAX_VALUE
Java HotSpot(TM) 64-Bit Server VM 1.7.0_72 limit Integer.MAX_VALUE-1
Java HotSpot(TM) 64-Bit Server VM 1.8.0_40 limit Integer.MAX_VALUE-2
In my case, savePngImage() in java side is a static function that calls native nativeSavePngImage() in JNI that uses libpng library.
FYI, the function is accessed by multiple threads in the program and the application is saving more than hundred thousands of images progressively and at one point in which the image count reaches hundred thousand, png_create_write_struct() return NULL as a result of failure to allocate memory. The memory settings in java is -Xms24G -Xmx24G -Xmn20G and at the time when the error happens, physical memory usage is only at 30% of 128GB.
BTW, memory leaks in native code does not seems like a problem as the memory shown in task manager is not actually showing an increasing trend although i might be very wrong about it.
// create the png structures
png_structp pWriteStruct = png_create_write_struct(PNG_LIBPNG_VER_STRING,
(png_voidp)NULL,
pngErrorFunction,
pngErrorFunction);
if (pWriteStruct == NULL)
{
fclose(pFile);
sptAssert(false); //getting assertion here
return;
}
Thus, my question is how can we know whether the native memory in JNI is hitting its limit thus png_create_write_struct() is unable to allocate the memory.
I am really sorry if the current above information wont help in determining the causes and would update the post if more debugging information can be gotten from your guys suggestion.
malloc() returns memory out of the C heap, and whether that's the entire process's C heap or the DLL's own heap depends on whether the DLL and the process are sharing the same dynamic C library or not.
JNI and the Java heap have exactly nothing to do with it in either case.
The rest of your question is back to front. You know that png_create_write_struct() has run out of memory when it returns NULL.
I sometimes write Python programs which are very difficult to determine how much memory it will use before execution. As such, I sometimes invoke a Python program that tries to allocate massive amounts of RAM causing the kernel to heavily swap and degrade the performance of other running processes.
Because of this, I wish to restrict how much memory a Python heap can grow. When the limit is reached, the program can simply crash. What's the best way to do this?
If it matters, much code is written in Cython, so it should take into account memory allocated there. I am not married to a pure Python solution (it does not need to be portable), so anything that works on Linux is fine.
Check out resource.setrlimit(). It only works on Unix systems but it seems like it might be what you're looking for, as you can choose a maximum heap size for your process and your process's children with the resource.RLIMIT_DATA parameter.
EDIT: Adding an example:
import resource
rsrc = resource.RLIMIT_DATA
soft, hard = resource.getrlimit(rsrc)
print 'Soft limit starts as :', soft
resource.setrlimit(rsrc, (1024, hard)) #limit to one kilobyte
soft, hard = resource.getrlimit(rsrc)
print 'Soft limit changed to :', soft
I'm not sure what your use case is exactly but it's possible you need to place a limit on the size of the stack instead with resouce.RLIMIT_STACK. Going past this limit will send a SIGSEGV signal to your process, and to handle it you will need to employ an alternate signal stack as described in the setrlimit Linux manpage. I'm not sure if sigaltstack is implemented in python, though, so that could prove difficult if you want to recover from going over this boundary.
Have a look at ulimit. It allows resource quotas to be set. May need appropriate kernel settings as well.
Following code allocates memory to specified maximum resident set size
import resource
def set_memory_limit(memory_kilobytes):
# ru_maxrss: peak memory usage (bytes on OS X, kilobytes on Linux)
usage_kilobytes = lambda: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
rlimit_increment = 1024 * 1024
resource.setrlimit(resource.RLIMIT_DATA, (rlimit_increment, resource.RLIM_INFINITY))
memory_hog = []
while usage_kilobytes() < memory_kilobytes:
try:
for x in range(100):
memory_hog.append('x' * 400)
except MemoryError as err:
rlimit = resource.getrlimit(resource.RLIMIT_DATA)[0] + rlimit_increment
resource.setrlimit(resource.RLIMIT_DATA, (rlimit, resource.RLIM_INFINITY))
set_memory_limit(50 * 1024) # 50 mb
Tested on linux machine.
I'm attempting to debug a problem with pl/java, a procedural language for PostgreSQL. I'm running this stack on a Linux server.
Essentially, each Postgres backend (connection process) must start its own JVM, and does so using the JNI. This is generally a major limitation of pl/java, but it has one particularly nasty manifestation.
If native memory runs out (I realise that this may not actually be due to malloc() returning NULL, but the effect is about the same), this failure is handled rather poorly. It results in an OutOfMemoryError due to "native memory exhaustion". This results in a segfault of the Postgres backend, originating from within libjvm.so, and a javacore file that says something like:
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
1TIDATETIME Date: 2012/09/13 at 16:36:01
1TIFILENAME Javacore filename: /var/lib/PostgreSQL/9.1/data/javacore.20120913.104611.24742.0002.txt
***SNIP***
Now, there are reasonably well-defined ways of ameliorating these types of problems with Java, described here:
http://www.ibm.com/developerworks/java/library/j-nativememory-linux/
I think that it would be particularly effective if I could set the maximum heap size to a value that is far lower than the default. Ordinarily, it is possible to do something along these lines:
The heap's size is controlled from the Java command line using the -Xmx and -Xms options (mx is the maximum size of the heap, ms is the initial size). Although the logical heap (the area of memory that is actively used) can grow and shrink according to the number of objects on the heap and the amount of time spent in GC, the amount of native memory used remains constant and is dictated by the -Xmx value: the maximum heap size. Most GC algorithms rely on the heap being allocated as a contiguous slab of memory, so it's impossible to allocate more native memory when the heap needs to expand. All heap memory must be reserved up front.
However, it is not apparent how I can follow these steps such that pl/java's JNI initialisation initialises a JVM with a smaller heap; I can't very well pass these command line arguments to Postgres. So, my question is, how can I set the maximum heap size or otherwise control these problems in this context specifically? This appears to be a general problem with pl/java, so I expect to be able to share whatever solution I eventually arrive at with the Postgres community.
Please note that I am not experienced with JVM internals, and am not generally familiar with Java.
Thanks
According to slide 19 in this presentation postgresql.conf can have the parameter pljava.vmoptions where you can pass arguments to the JVM.
I have to allocate space to an array int input[] depending on the configuration parameters height and width.
int input[]=new int[height * width]; //this is line no 538
One of the configurations has parameters height=8192 and width=8192. So the size of the array becomes 67108864. But when i do this i get OutOfMemoryError.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at Test.main(Test.java:538)
I have ran this program on eclipse as well as on cygwin but i am facing the same problem. I think this is not an error and not exception. How can i rectify this?
Since 8192 * 8192 * 4 = 256 M (integers are 4 bytes each), your matrix is using 256 MB of heap space by itself.
You can tell the JVM how much heap space should be available to your application. From running man java and looking through the nonstandard options:
-Xmxn
Specify the maximum size, in bytes, of the memory allocation
pool. This value must a multiple of 1024 greater than 2MB.
Append the letter k or K to indicate kilobytes, or m or M to
indicate megabytes. The default value is chosen at runtime
based on system configuration. For more information, see
HotSpot Ergonomics
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m
On Solaris 7 and Solaris 8 SPARC platforms, the upper limit for
this value is approximately 4000m minus overhead amounts. On
Solaris 2.6 and x86 platforms, the upper limit is approximately
2000m minus overhead amounts. On Linux platforms, the upper limit
is approximately 2000m minus overhead amounts.
To use this option, you would start your application with a command like
java -Xmxn1024m -jar foo.jar
In Eclipse, you can add command-line options as well. This page on eclipse.org describes how to add command-line arguments to a Java program. You should add the -Xmxn1024m (or some other sufficiently large heap specification) to the "VM arguments" section of the dialog shown on that site.
You probably have too little heap space to hold an array of the size you are targeting. You can increase the size of your heap with command line switches. For example, to set it to 256MB, include this switch:
-Xmx256m
If you multiply height * width * 4 (4 is the storage in bytes for an int) you can get a rough gauge of the amount of heap you will need, assuming the rest of the program does not need a significant amount. You will certainly need some more heap than that quick calculation suggests. Add maybe 20% extra, and try that out.
To get a better number than a rule-of-thumb calculation, you can look into heap profilers. There are several open source options:
http://java-source.net/open-source/profilers
See http://javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.html for a good discussion of the heap in Java.
memory is not enough for your program, may be memory leak there.
you may try below,if not solve try to increase jmx value.
java -xmx1g -xms512m
Depends on how much heap the JVM has. If you run it on the command line try adding -Xmx512m. If you work in an IDE add it to the "Run" properties.
An int is 32 bits (i.e. 4 bytes). So your array requires 8192*8192*4 bytes. This comes out at 256MB.
Java called with default arguments has only 64MB of heap space.
To get a larger heap, call Java using the -Xmx argument (Maximum memory size).
e.g. java -Xmx300M
Increase your memory arguments for your Java process by adding this flag to increase the heap. You might need to play around to get the optimal size for the heap. This will set the "max" heap size. The default is probably really small. 64M is a common max size for many Java EE containers.
*Note I'm not saying this is exactly the size you'll need. Your unique case will dictate the size you'll need which you may need to experiment with.
-Xmx256M