I am writing a library for clojure which involves native code. How can I bundle the shared library (aka native dependencies) when I deploy the clojure libraries to public repositories (like clojars)?
Further Info:
My project structure looks roughly like:
src/
native/ - C code , C Object files and compiled shared libs
java/ - Java stuff
clojure/ - Clojure stuff
I am currently using leineingen. I have tried doing:
:jvm-opts [~(str "-Djava.library.path=src/native/:"
(System/getenv "$LD_LIBRARY_PATH"))]
It works if I am in the project. However, if I include this project as a dependency, I will get a UnsatisfiedLink error.
The answer depends on your exact use-case. In the simplest situation, you need to:
bundle the native lib inside the library jar, for example by including the native/ folder in :resource-paths in your project.clj.
when you use your library, you can specify a :native-prefix option to indicate a path inside your library jar from which native libraries should be extracted. For example, if your library contains a resource "mylib.so" in the root folder, you can specify it like this: [com.foo/bar "1.0.1" :native-prefix ""]
you should also specify where the extracted libs should go, using the :native-path option in your project.clj.
finally, you should add the :native-path you have specified to java.library.path, using :jvm-opts like you said.
These options are documented in the sample leiningen project.clj.
Now the reason I said it depends on your use-case is that if you want to create an uberjar that contains native libs, things start getting more messy. The main reason is that you can't directly link to a lib that is zipped inside your jar. If you're lucky, you'll be able to use the loadLibraryFromJar method in the NativeUtils class. However, I remember having ClassLoader-related issues that prevented me from using System/load. Instead I had to make sure the library was present in one of the paths that the JVM looks for, so that System/loadLibrary finds it correctly. Here is what I ended up doing:
manually extract the native lib from the uberjar at runtime to a temporary folder, with (-> my-lib io/resource io/input-stream (io/copy my-temp-file))
update the java.library.path system property at runtime to add the temporary folder to it, using System/setProperty
finally, use this reflection trick to force the library path to be updated
This is painful to setup, but after that it works pretty well.
Related
I have created a JNI library with visual studio to use in java application and internally this dll calls another DLL (openssl). Now when i load this library with eclipse I get an error saying "Can't find dependent libraries" unless I put the path of the other DLL on the PATH variable.
in thread "main" java.lang.UnsatisfiedLinkError: C:\*****\jniDLL.dll: Can't find dependent libraries
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1939)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1864)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
at java.lang.Runtime.loadLibrary0(Runtime.java:845)
at java.lang.System.loadLibrary(System.java:1084)
at iwinAlgTest.test.<clinit>(test.java:7)
And I've already tried adding it to -Djava.library.path and to the classpath, and i also included all of my dll and openssl dll and lib files like libeay32.lib but neither worked. Is there a way to do this?
I tried to use Dependency Walker to check if there are any files required by my jniDLL.dll
but i get interesting thing:
what i really do to create my jniDLL:
(jniDLL --> calls myDLL.dll) ==> (myDLL.dll --> calls openssl library)
but Dependency Walker says that
(jniDLL --> depends on myDLL.exe) not myDLL.dll
Note : at first it was a win32 application( which output myDLL.exe) but i changed the project setting to output myDLL.dll then i create another project to create a JNI dll from myDLL.dll which i wrote in c++
One way to bypass this is to load your openssl.dll prior to loading myDLL, so myDLL will find the symbols loaded when it tries to initiate .
Other ways to go around this :
add the openssl.dll to PATH (you already mentioned this)
copy openssl in system32 at install time
put openssl in the same folder as your executable path
This however will only work on Windows.
In linux, you will have to change the rpath inside the .so (but I believe this is out of scope).
I am using one third party jar in my code. In the jar file , in one of the classes, when I opened the class using de-compiler, the code below is written:
java.net.URL fileURL = ClassLoader.getSystemResource("SOAPConfig.xml");
Now I am using this in my webapplication, where should I place this SOAPConfig.xml so that it will find the fileURL.
Note: I have tried putting this XML in WEB-INF/classes folder. But it is not working. Your help will be appreciated.
In Addition: In the explaination you have given, It is telling me not to use this code snippet inside the third party jar in this way...What is the exact usage of this statement
ClassLoader.getSystemResource will load the resource from the system classloader, which uses the classpath of the application as started from the command line. Any classloaders created by the application at runtime (i.e. the one that looks in WEB-INF/classes) are not on the system classpath.
You need to
Look through the script that starts your server, find out which directories are on the classpath there, and put your SOAPConfig.xml in one of those. If necessary, change the classpath in the script to look in a separate directory that's just used for your config file.
Track down the person who used ClassLoader.getSystemResource in the library, kick them squarely in the nuts, and tell them never to do that again.
I have a dll file and I am trying to call functions of it through a Java program through JNA
But the problem is It is not able to locate my dll file and throwing the following exception:
java.lang.UnsatisfiedLinkError: Unable to load library 'UsbDll': The specified module could not be found.
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:163)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:236)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:199)
at com.sun.jna.Native.register(Native.java:1018)
at com.MainClass.<clinit>(MainClass.java:15)
Exception in thread "main"
Below is my program:
package com;
import com.sun.jna.Native
public class MainClass {
static {
Native.register("UsbDll");
}
public native int method();
public static void main(String[] args) {
}
}
The name of my dll file is UsbDll.dll and my operating system is Windows.
============================ EDITED ================================
The location of my dll file is "c:\UsbDll.dll"
When I placed another dll file at the same location, JNA has located it so I think that the problem is with my "UsbDll.dll" file only.
When I tried to load both the dll files (UsbDll.dll and the another dll) with the following command
System.load("c:\\UsbDll.dll");
System.load("c:\\another.dll");
It loaded the "another.dll" successfully but for "UsbDll.dll", it throws the following exception:
java.lang.UnsatisfiedLinkError: C:\UsbDll.dll: Can't find dependent libraries
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at com.MainClass.<clinit>(MainClass.java:16)
Exception in thread "main"
Q1. It looks like it is not finding some dependent libraries. But as I am totally new to these dlls, Is there anything I am missing or I need to ask the vendor to provide me the dependent libraries.
OR
Is it depends on some standard libraries which I can download from internet? If it is, then how to find the libraries name on which it depends?
============================ EDITED #2 ================================
I ran the Dependency Walker on my UsbDll.dll file to find the missing dependencies and found the following missing dependencies.
WER.DLL (referred by library WINNM.DLL found in my c:\windows\system32)
IESHIMS.DLL (referred by library WINNM.DLL found in my c:\windows\system32)
WDAPI920.DLL (referred directly by my UsbDll.dll)
Q1. As WINNM.DLL was found in my system32 folder, it seems as the standard dll. And if this standard dll is referring to 2 missing dlls (WER.DLL & IESHIMS.DLL), then I suspect how other applications are working who are using this WINNM.DLL file?
Q2. I googled for WDAPI920.dll (that was referred my UsbDll.dll directly) and many search results appeared with the same dll name. So it also looks like some standard library. So how to fix these dependencies? From where to download them? After downloading, Can I place them in the same directory in which my main dll (UsbDll.dll) is or I need to do something extra to load my dll (UsbDll.dll) sucessfully?
From your edited code it is quite evident that the UsbDll.dll depends on some standard modules which are not there on your system (for example if it uses ATL and if you have don't have proper runtime then it is guaranteed to fail). To resolve this you will need proper runtime environment.
Also it is possible that the dll in concern depends on some vendor specific module. For you the best option is (and fastest option would be) to contact the vendor. Otherwise, try to install proper runtime from the microsoft site (but its more of hit-and-trial)
Update
Use the below links for finding more about DLL dependency:
How do I determine the dependencies of a .NET application?
http://msdn.microsoft.com/en-us/library/ms235265.aspx
Command line tool to find Dll dependencies
Update 2
See the below mentioned link for the missing dll details (but it is specific to the windows version)
Dependency Walker reports IESHIMS.DLL and WER.DLL missing?
http://social.msdn.microsoft.com/Forums/en/vsx/thread/6bb7dcaf-6385-4d24-b2c3-ce7e3547e68b
From few simple google queries, WDAPIXXX.dll appears to be some win driver related thing (although i am not too sure). Check this link, they have something to say about WDAPI http://www.jungo.com/st/support/tech_docs/td131.html.
The DLL must by in the Path specified by LD_LIBRARY_PATH (see your env), or, in the case of JNA, in the current directory.
I have a jar embedded in a bundle that needs to fetch a resource packaged with it like so:
MyBundle
-\ src
-\lib
-\MyEmbeddedJar
-\src
-\SomeClass
-\someResource.xml
I am trying to access 'someResource.xml' from 'SomeClass' like so:
SomeClass.class.getResource( "someResource.xml" );
But I've had no luck. I've tried several variations with the CWD appended (eg: './someResource.xml') but I just can't get this resource to load.
I know that the "right" way is to use Activator to get hooks back to the proper classloader, but the embedded jar can be used in other projects, so I'd hate to have to add OSGi specific code to it just to get it to play nice with OSGi.
Is there any other way to load resources in OSGi agnostically of OSGi?
I Assume that SomeClass is inside the embedded jar (say, somejar.jar), and someResource.xml is in the outer jar, in a lib directory.
In this case, there is no way to get to that in a non-OSGi context. Let's look at both situations in isolation.
In OSGi
Your someResource.xml should very well be reachable using the regular (non-OSGi specific) resource loading mechanisms, provided that it is reachable from the Bundle-ClassPath. For instance, if you have the following manifest header,
Bundle-ClassPath: ., somejar.jar
you will be able to get to your resource using "lib/someResource.xml".
Notice the dot on the classpath: this means you can reach classes and resources from the root of the jar. If you forget that, you will only be able to get to classes and resources inside somejar.jar.
Not using OSGi
If you're not using OSGi, there is no (reasonably simple) way to get to classes and resources inside of the inner jar that I know of.
Your options
Depending on what you want your bundle to look like, you have two options now.
Is it really necessary that SomeClass is in an embedded jar? If so, you're at a loss, and you jar will only work using OSGi.
If you have the option to 'unpack' somejar.jar into your jar, you subvert the problem, and your jar can work in both situations.
Personally, I'd pick option 2.: unless you have resources that might overwrite each other when you 'merge' the jars, it is no problem at all to have a slight mess of resources inside your bundle.
My assumptions are that:
That I'm not quite sure I get how your description of the problem matches the diagram. Where is the *.jar file?
The bundle that is attempting to
access the embedded jar is the same
bundle that contains the embedded
jar.
Per the OSGi agnosticism, I am
assuming that the embedded jar is
not explicitly exposed as part of
the current bundle's classpath and
that it is not loaded as another
OSGi bundle.
If the jar in question is itself a resource of the current classloader, then you would first need to get the jar as a resource or as an InputStream, such as with MyBundleClass.class.getResourceAsStream("/pathToJar.jar"); then wrap it with a java.util.jar.JarInputStream. Then, continue to call getNextJarEntry() until you find the JarEntry object where "someResource.xml".equals(jarEntry.getName()).
I'm going to accept #Angelo's solution as it gave me the idea on how to work around this, though, I'd like to add more information in - thus my answer.
My work around was to add another constructor to SomeClass that takes in a java.net.URL instance. I also copied someResource.xml into the bundle's root.
I then updated the instantiation of SomeClass in the bundle like so:
new SomeClass( FileLocator.find( Activator.getDefault().getBundle(), new Path( "./someResource.xml" ), new HashMap< String, String >() ) );
This seems like a pretty big hack to me. What if I could not edit the contents of SomeClass ? I guess I would have to unpack it or I'd be forced to wrap it into it's own bundle?
I'm going to make the following assumption:
embedded.jar
src/SomeClass.class
someResource.xml
and the bundle contains embedded.jar and embedded.jar is on the Bundle-Classpath.
In this situation SomeClass.class.getResource("someResource.xml") will look for a resource on the classpath called src/someResource.xml because SomeClass is in the package src. In order to get someResource.xml in the root of the jar you need to do SomeClass.class.getResource("/someResource.xml") instead.
This isn't OSGi specific though, this is just how resource loading works in Java.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to bundle a native library and a JNI library inside a JAR?
I need to include native lib (jnotify but I think that it does't matter) to my jar. I want to do it with NetBeans.
I added Bundle-NativeCode: /lib/jnotify.dll; osname=win32 to my manifest.mf file and added jnotify.dll to projektHome\src\lib\ folder. But unfortunately NetBeans is overidning manifest.mf file.
How can I fixed? Can I do this using only NetBeans? Is it line 'Bundle-NativeCode: /lib/jnotify.dll; osname=win32 correct? I also heard I should put dlls hash in manifest.mf and sign my jar. Is that true?
I don't think the Java executable supports Bundle-NativeCode. I'm pretty sure that is an OSGi attribute. The list of supported attributes is defined in the JAR File Specification.
Outside frameworks that provide it, there is no built-in support for bundling native libraries inside JAR files. If I recall correctly, it is possible to extract the file to a temporary location and load it manually.
Sometimes i found the problem is not the Java way of loading native libs, but the 3rd party library that needs that native code.
The problem is that the 3rd party libs will do at some point (normally very early in initialization)
System.loadLibrary("native.dll");
And if native.dll is not at the appropiated place it throws an Error.
If you have access to the java source of the 3rd party library it might be easy to patch that code and you could easily extract your dll from the JAR and run System.load before using the 3rd party lib.
Update
I had a look into JNotify sources. It is exactly what i said:
public class JNotify_win32
{
static
{
System.loadLibrary("jnotify"); /* *** */
int res = nativeInit();
if (res != 0)
{
throw new RuntimeException("Error initialiing native library. (#" + res + ")");
}
}
Take line *** out or surround with try-catch, load with System.load() and you are done.
I ran into this problem when trying to hook a Windows shutdown event when the program runs on that OS. The solution I ended up using was essentially McDowell's - adding the DLL to the jar file and extracting it to a temporary location when the program starts. If it fits your program, you can leave the DLL in a more permanent place and then reference it on subsequent program startups. My application was used in an environment where the users might intentionally delete files they shouldn't, so I had to extract the DLL on every run. However, it hasn't resulted in any performance hit of significance.