How to trigger a jar working on Hadoop from a simple jar, so that it uses HDFS, Actully, I am manually running this command bin/hadoop jar ~/wordcount_classes/word.jar org.myorg.WordCount ~/hadoop-0.20.203.0/input1 ~/hadoop-0.20.203/output2 in which I have provided Input and Output directory in HDFS and I am using word.jar here, I want to make it such that it automatically gets triggered from Java Project.
In best of my understanding all you asking for is done by the Main of your jar. It read parameters, creates job configuration, sets input and output formats and finally runs the job.
I'm working on the same problem. I have a program (let's call it Driver) that must implement the following method:
public void runJar(File jar, String mainClass, File inputDir, File outputDir);
To do this, I was calling org.apache.hadoop.util.RunJar.main(String[]) which is what your command-line is calling. This works great only if you're running Driver from the command line.
If Driver is running inside a container (like Tomcat or Jetty), you're going to have a problem. You'll get errors like
java.lang.ClassNotFoundException: org.apache.hadoop.fs.Path
This is because of how RunJar messes with classloaders. You need to manually create a classloader like so:
final ClassLoader original = Thread.currentThread().getContextClassLoader();
try {
URL[] urls = new URL[] { jar.toURI().toURL() };
ClassLoader loader = new URLClassLoader(urls, originalLoader);
Thread.currentThread().setContextClassLoader(loader);
Class<?> mainClass = Class.forName(driverClass, true, loader);
Class[] argTypes = new Class[]{ Array.newInstance(String.class, 0).getClass()};
Method main = mainClass.getMethod("main", argTypes);
main.invoke(null, new Object[] { args });
} finally {
Thread.currentThread().setContextClassLoader(original);
}
Related
I am writing an eclipse-plugin witch run program.exe. I have added program.exe to plugin jar file. How can a execute this program?
public class Handler extends AbstractHandler {
public Object execute(ExecutionEvent event) throws ExecutionException {
Runtime.getRuntime().exec(/*What should I write here*/);
return null;
}
}
You can't run the program.exe from inside the plugin jar, so it needs to be extracted. In your plugin use:
Bundle bundle = Platform.getBundle("plugin id");
URL url = FileLocator.find(bundle, new Path("relative path to program"), null);
url = FileLocator.toFileURL(url);
This will find the program in the plugin jar and extract it to a temporary location (done by FileLocator.toFileURL).
You should just execute the program like you would in cmd, but now specify the whole path of the programs location.
Runtime.getRuntime().exec("C:\\your\\path\\program.exe");
In the Oracle documentation of the Runtime class you can see the acceptable inputs in exec().
I'm currently using the Alloy Analyzer API to build a program, and getting some peculiar behavior. Specifically, if I open a file and parse it (using CompUtil.parseEverything), then make a new Command and call TranslateAlloyToKodkod.execute_command on the parsed file and newly created command using MiniSat with UNSAT core, it runs fine. However, later in execution, my program parses a second input file (also using CompUtil.parseEverything), gets another world, makes a new command, and then I try to call TranslateAlloyToKodkod.execute_command again, it throws the following error:
ERROR: class edu.mit.csail.sdg.alloy4.ErrorFatal: The required JNI library cannot be found:
java.lang.UnsatisfiedLinkError: no minisatproverx5 in java.library.path
edu.mit.csail.sdg.alloy4compiler.translator.TranslateAlloyToKodkod.execute_command(TranslateAlloyToKodkod.java:390)
Does anyone have any idea why this is thrown the second time, but not the first?
To summarize, I have something similar to the following:
Module someWorld = CompUtil.parseEverything_fromFile(rep, null, "someFile.als");
//For the following, "sig" is a sig in someWorld.getAllReachableSigs();
Command command = sig.not();
A4Options options = new A4Options();
options.solver = A4Options.SatSolver.MiniSatProverJNI;
A4Solution ans =
TranslateAlloyToKodkod.execute_command(rep, someWorld, command, options);
//No thrown error
Module someOtherWorld = CompUtil.parseEverything_fromFile(rep, null, "someOtherFile.als");
//For the following, "sig" is a sig in someOtherWorld.getAllReachableSigs();
Command commandTwo = sig.not();
A4Solution ansTwo =
TranslateAlloyToKodkod.execute_command(rep, someOtherWorld, commandTwo, options);
//Thrown error above. Why?
I tried to reproduce this behavior, but I couldn't. If I don't add MiniSat binaries to the LD_LIBRARY_PATH environment variable, I get the exception you mentioned the very first time I invoke execute_command. After configuring LD_LIBRARY_PATH, the exception doesn't happen.
To configure LD_LIBRARY_PATH:
(1) if using Eclipse, you can right-click on one of your source folders, choose Build Path -> Configure Build Path, then on the "Source" tab make sure that "Native library location" points to a folder in which MiniSat binaries reside.
(2) if running from the shell, just add the path to a folder with MiniSat binaries to LD_LIBRARY_PATH, e.g., something like export LD_LIBRARY_PATH=alloy/extra/x86-linux:$LD_LIBRARY_PATH.
Here is the exact code that I was running, and everything worked
public static void main(String[] args) throws Exception {
A4Reporter rep = new A4Reporter();
A4Options options = new A4Options();
options.solver = A4Options.SatSolver.MiniSatProverJNI;
Module someWorld = CompUtil.parseEverything_fromFile(rep, null, "someFile.als");
Command command = someWorld.getAllCommands().get(0);
A4Solution ans = TranslateAlloyToKodkod.execute_command(rep, someWorld.getAllReachableSigs(), command, options);
System.out.println(ans);
Module someOtherWorld = CompUtil.parseEverything_fromFile(rep, null, "someOtherFile.als");
Command commandTwo = someOtherWorld.getAllCommands().get(0);
A4Solution ansTwo = TranslateAlloyToKodkod.execute_command(rep, someOtherWorld.getAllReachableSigs(), commandTwo, options);
System.out.println(ansTwo);
}
with "someFile.als" being
sig A {}
run { some A } for 4
and "someOtherFile.als"
sig A {}
run { no A } for 4
I use alloy4.2.jar as a library in my eclipse plugin project.
A4Reporter rep = new A4Reporter();
Module world = CompUtil.parseEverything_fromFile(rep, null, "civi.als");
A4Options options = new A4Options();
options.solver = A4Options.SatSolver.SAT4J;
options.skolemDepth = 1;
When I use SAT4J, the default solver, the problem mentioned here will not show up. But another exception comes out. The reason is that my civi.als file need Integer model, which located in alloy4.2.jar under the folder /models/util/. But when I run the application, it tries to find the file util/Integer.als directly. That causes the exception. Is it possible to fix that problem?
Besides, I also tried to put the alloy4.2.jar in eclipse plugin project and run my application as an eclipse application (running my application as a plugin). With the default solver, the application has no problem at all. But when I switch to MiniSatProverJNI, the problem mentioned here comes out (I have set the alloy4.2.jar as classpath).
I'm having problems using ProcessBuilder to run a class in my project.
My code:
public class Main {
public static void main(String[] args) {
try {
String pathToJar = Main.class.getProtectionDomain().getCodeSource()
.getLocation().toURI().getPath();
ArrayList<String> params = new ArrayList<String>();
params.add("javaw");
params.add("-classpath");
params.add(pathToJar);
params.add("Program");
ProcessBuilder pb = new ProcessBuilder(params);
Process process = pb.start();
} catch (Exception e) {
e.printStackTrace();
}
}
}
class Program is in same project (same bin folder) and works fine if ran directly but this way I get the error "Could not find the main class: Program". Where is the error in my code?
Thanks in advance.
[EDIT]
I came to the conclution that is some code on my Program class giving error. Basicly only runs with "clean" main. At eclipse, Program class is importing some libraries that are inside a jar file. Don't I need to reference it in ProcessBuilder? If so, how?
In response to your edit:
You can add the current path by switching params.add(pathToJar); with params.add(System.getProperty("java.class.path").concat(";").concat(pathToJar));.
Where is the error in my code?
(You are launching the javaw executable, so that is not the problem. It is also not that your entry point method's signature is incorrect, because that would have given a different diagnostic.)
The problem is either that the class name is incorrect (e.g. if should be "come.pkg.Program"), or the pathname for the JAR file is incorrect.
Assuming that you have eliminated the possibility that the class name is incorrect, my guess is that you are trying to use a relative pathname for the JAR file, but there is some confusion over what the current directory is; i.e. the directory in which the pathname needs to be resolved. Try using an absolute pathname in the classpath parameter.
I'm making a web application that, at a certain point, starts a new thread and this thread executes a jar file from command line.
The command line jar works fine when called from outside the application, but when i call it from the thread the relative path becomes C:\eclipse\ (i'm running the application from Eclipse) instead of the directory it's stored in, which messes up with its configuration since it looks for files in the wrong place.
The jar creates a log file, whenever i try to call it i have this line written in the log: "10/04/2012 17:09:03 - java.io.FileNotFoundException: C:\eclipse\descriptors\analysis_engine\AggregateAE.xml"
The jar is not inside C:\eclipse. When i call it from prompt i have no problems, but when it's called from a new spawned thread i have this error. I've tried it on a production environment and i have the same problem (this time the base path is the server's one)
Considering that i can't modify all the paths, what could be a solution to this problem?
EDIT: this is the thread class that calls the jar
public class UimaThread extends Thread {
private int mode=0;
private String path;
public UimaThread(int mode, String path){
this.mode=mode;
this.path=path;
}
public void run() {
Runtime run = Runtime.getRuntime();
try {
Properties config = ConfigLoader.getConfig();
String uimaPath=config.getProperty("uimaPath")+ControlPanelUtils.getDelimiter(config.getProperty("uimaPath"));
//uimaPath is the absolute path to the jar file, mode and path are just arguments passed to the jar
run.exec("java -jar "+uimaPath+"uimachainfull.jar "+mode+" "+path);
}
}
The code running this is:
public void startUima() throws IOException, ServletException {
Properties config = ConfigLoader.getConfig();
UimaThread uimaThread = new UimaThread(2, config.getProperty("docPath"));
uimaThread.start();
}
I need this to be executed asyncronously and outside the server, i've asked how to do that here in stackoverflow and i've been told to do so: Calling an application from a web server asynchronously
I've discovered how to do it, instead of using exec(String command) i had to use exec(String command, String[] envp, File dir) and the last argument (dir) is the working directory that can be passed as new argument
I need to add plugin functionality to an existing application for certain parts of the application. I want to be able to add a jar at runtime and the application should be able to load a class from the jar without restarting the app. So far so good. I found some samples online using URLClassLoader and it works fine.
I also wanted the ability to reload the same class when an updated version of the jar is available. I again found some samples and the key to achieving this as I understand is that I need to use a new classloader instance for each new load.
I wrote some sample code but hit a NullPointerException. First let me show you guys the code:
package test.misc;
import java.io.File;
import java.net.URL;
import java.net.URLClassLoader;
import plugin.misc.IPlugin;
public class TestJarLoading {
public static void main(String[] args) {
IPlugin plugin = null;
while(true) {
try {
File file = new File("C:\\plugins\\test.jar");
String classToLoad = "jartest.TestPlugin";
URL jarUrl = new URL("jar", "","file:" + file.getAbsolutePath()+"!/");
URLClassLoader cl = new URLClassLoader(new URL[] {jarUrl}, TestJarLoading.class.getClassLoader());
Class loadedClass = cl.loadClass(classToLoad);
plugin = (IPlugin) loadedClass.newInstance();
plugin.doProc();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
IPlugin is a simple interface with just one method doProc:
public interface IPlugin {
void doProc();
}
and jartest.TestPlugin is an implementation of this interface where doProc just prints out some statements.
Now, I package the jartest.TestPlugin class into a jar called test.jar and place it under C:\plugins and run this code. The first iteration runs smoothly and the class loads without issues.
When the program is executing the sleep statement, I replace C:\plugins\test.jar with a new jar containing an updated version of the same class and wait for the next iteration of while. Now here's what I don't understand. Sometimes the updated class gets reloaded without issues i.e. the next iteration runs fine. But sometimes, I see an exception thrown:
java.lang.NullPointerException
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream.close(JarURLConnection.java:90)
at sun.misc.Resource.getBytes(Resource.java:137)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at test.misc.TestJarLoading.main(TestJarLoading.java:22)
I have searched on the net and scratched my head but can't really arrive at any conclusion as to why this exception is thrown and that too - only sometimes, not always.
I need your experience and expertise to understand this. What's wrong with this code? Please help!!
Let me know if you need any more info. Thanks for looking!
For everyone's benefit, let me summarize the real problem and the solution that worked for me.
As Ryan pointed out, there is a bug in JVM, which affects Windows Platform. URLClassLoader does not close the open jar files after it opens them for loading classes, effectively locking the jar files. The jar files can't be deleted or replaced.
The solution is simple: close the open jar files after they've been read. However, to get a handle to the open jar files, we need to use reflection since the properties we need to traverse down are not public. So we traverse down this path
URLClassLoader -> URLClassPath ucp -> ArrayList<Loader> loaders
JarLoader -> JarFile jar -> jar.close()
The code to close the open jar files can be added to a close() method in a class extending URLClassLoader:
public class MyURLClassLoader extends URLClassLoader {
public PluginClassLoader(URL[] urls, ClassLoader parent) {
super(urls, parent);
}
/**
* Closes all open jar files
*/
public void close() {
try {
Class clazz = java.net.URLClassLoader.class;
Field ucp = clazz.getDeclaredField("ucp");
ucp.setAccessible(true);
Object sunMiscURLClassPath = ucp.get(this);
Field loaders = sunMiscURLClassPath.getClass().getDeclaredField("loaders");
loaders.setAccessible(true);
Object collection = loaders.get(sunMiscURLClassPath);
for (Object sunMiscURLClassPathJarLoader : ((Collection) collection).toArray()) {
try {
Field loader = sunMiscURLClassPathJarLoader.getClass().getDeclaredField("jar");
loader.setAccessible(true);
Object jarFile = loader.get(sunMiscURLClassPathJarLoader);
((JarFile) jarFile).close();
} catch (Throwable t) {
// if we got this far, this is probably not a JAR loader so skip it
}
}
} catch (Throwable t) {
// probably not a SUN VM
}
return;
}
}
(This code was taken from the second link that Ryan posted. This code is also posted on the bug report page.)
However, there's a catch: For this code to work and be able to get a handle to the open jar files to close them, the loader used to load the classes from the file by URLClassLoader implementation has to be a JarLoader. Looking at the source code of URLClassPath (method getLoader(URL url)), I noticed that it uses a JARLoader only if the file string used to create the URL does not end in "/". So, the URL must be defined like this:
URL jarUrl = new URL("file:" + file.getAbsolutePath());
The overall class loading code should look something like this:
void loadAndInstantiate() {
MyURLClassLoader cl = null;
try {
File file = new File("C:\\jars\\sample.jar");
String classToLoad = "com.abc.ClassToLoad";
URL jarUrl = new URL("file:" + file.getAbsolutePath());
cl = new MyURLClassLoader(new URL[] {jarUrl}, getClass().getClassLoader());
Class loadedClass = cl.loadClass(classToLoad);
Object o = loadedClass.getConstructor().newInstance();
} finally {
if(cl != null)
cl.close();
}
}
Update: JRE 7 has introduced a close() method in the class URLClassLoader which may have solved this issue. I haven't verified it.
This behaviour is related to a bug in the jvm
2 workarounds are documented here
Starting from Java 7, you indeed have a close() method in URLClassLoader but it is not enough to release completely the jar files if you call directly or indirectly methods of type ClassLoader#getResource(String), ClassLoader#getResourceAsStream(String) or ClassLoader#getResources(String). Indeed by default, the JarFile instances are automatically stored into the cache of JarFileFactory in case we call directly or indirectly one of the previous methods and those instances are not released even if we call java.net.URLClassLoader#close().
So a hack is still needed in this particular case even with Java 1.8.0_74, here is my hack https://github.com/essobedo/application-manager/blob/master/src/main/java/com/github/essobedo/appma/core/util/Classpath.java#L83 that I use here https://github.com/essobedo/application-manager/blob/master/src/main/java/com/github/essobedo/appma/core/DefaultApplicationManager.java#L388. Even with this hack, I still had to call the GC explicitly to fully release the jar files as you can see here https://github.com/essobedo/application-manager/blob/master/src/main/java/com/github/essobedo/appma/core/DefaultApplicationManager.java#L419
This is an update tested on java 7 with success. Now the URLClassLoader works fine for me
MyReloader
class MyReloaderMain {
...
//assuming ___BASE_DIRECTORY__/lib for jar and ___BASE_DIRECTORY__/conf for configuration
String dirBase = ___BASE_DIRECTORY__;
File file = new File(dirBase, "lib");
String[] jars = file.list();
URL[] jarUrls = new URL[jars.length + 1];
int i = 0;
for (String jar : jars) {
File fileJar = new File(file, jar);
jarUrls[i++] = fileJar.toURI().toURL();
System.out.println(fileJar);
}
jarUrls[i] = new File(dirBase, "conf").toURI().toURL();
URLClassLoader classLoader = new URLClassLoader(jarUrls, MyReloaderMain.class.getClassLoader());
// this is required to load file (such as spring/context.xml) into the jar
Thread.currentThread().setContextClassLoader(classLoader);
Class classToLoad = Class.forName("my.app.Main", true, classLoader);
instance = classToLoad.newInstance();
Method method = classToLoad.getDeclaredMethod("start", args.getClass());
Object result = method.invoke(instance, args);
...
}
Close and Restart the ClassReloader
then update your jar and call
classLoader.close();
then you can restart the app with the new version.
Do not include your jar into your base class loader
Do not include your jar into your base class loader "MyReloaderMain.class.getClassLoader()" of the "MyReloaderMain", in other words develop 2 project with 2 jars one for "MyReloaderMain" and the other one for your real application without dependency between the two, or you will not able to understand who i loading what.
The error is still present in jdk1.8.0_25 on Windows. Although #Nicolas' answer helps, I hit a ClassNotFound for sun.net.www.protocol.jar.JarFileFactory when running it on WildFly, and several vm crashes while debugging some box tests...
Therefore I ended up extracting the part of the code which deals with loading and unloading, to an external jar. From the main code I just call this with java -jar.... all looks fine for now.
NOTE: Windows does release the locks on the loaded jar files when the jvm exits, that is why this works.
In principle, a class that has already been loaded cannot be reloaded with the same classloader.
For a new load, it is necessary to create a new classloader and thus load the class.
Using URLClassLoader has one problem and that is that the jar file remains open.
If you have multiple classes loaded from one jar file by different instances of URLClassLoader and you change the jar file at runtime, you will usually get this error: java.util.zip.ZipException: ZipFile invalid LOC header (bad signature). The error may be different.
In order for the above errors not to occur, it is necessary to use the close method on all URLClassLoaders using the given jar file. But this is a solution that actually leads to a restart of the entire application.
A better solution is to modify the URLClassLoader so that the contents of the jar file are loaded into the RAM cache. This no longer affects other URLClassloaders that read data from the same jar file. The jar file can then be freely changed while the application is running. For example, you can use this modification of URLClassLoader for this purpose: in-memory URLClassLoader