I am trying to run this command in Python:
java JSHOP2.InternalDomain logistics
It works well when I run it in cmd.
I wrote this in Python:
args = ['java',
r"-classpath",
r".;./JSHOP2.jar;./antlr.jar",
r"JSHOP2.InternalDomain",
thisDir+"/logistics"
]
proc = subprocess.Popen(args, stdout=subprocess.PIPE)
proc.communicate()
I have the jar files in the current directory.
but I got this error:
Error: Could not find or load main class JSHOP2.InternalDomain
Does anyone know what the problem is? can't it find the jar files?
You can't count on the current working directory always being the same when running your Python code. Explicitly set a working directory using the cwd argument:
proc = subprocess.Popen(args, stdout=subprocess.PIPE,
cwd='/directory/containing/jarfiles')
Alternatively, use absolute paths in your -classpath commandline argument. If that path is thisDir, then use that:
proc = subprocess.Popen(args, stdout=subprocess.PIPE,
cwd=thisDir)
Related
I am trying to call python Scripts(in resources) from my Java class.
This is my code spinnet
String res = "/Scripts/Brokers/__init__.py";
URL pathUrl = this.getClass().getResource(res);
String path = "";
if(pathUrl != null)
path = pathUrl.toString();
ProcessBuilder pb = new ProcessBuilder("/usr/bin/python3.6", path);
ProcessBuilder is giving error No such file or directory.
P.S.
value of path = "file:/home/path-to-project/project-name/out/production/resources/Scripts/Brokers/\__init__.py"
Also how to include python scripts in jar file to run the project on linux server.
I am stuck with this issue since last 3 days. Please suggest.
From java doc:
https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
Starting a new process which uses the default working directory and
environment is easy:
Process p = new ProcessBuilder("myCommand", "myArg").start();
So basically, you going to fork a sub process that will
start a python interpreter
with the python script that you have provided as argument.
The path to your script should be a normal path that your OS can understand, therefore you should not give a URI? like path (protocol:address/path).
if (path.contains(":"))
path = (path.split(":"))[1];
Also the backslash \ before __init__.py looks suspicious.
You should be able to run ls /home/path-to-project/project-name/out/production/resources/Scripts/Brokers/__init__.py and see the file, the same goes for ls /usr/bin/python3.6.
I am trying to run a yarn job from a java wrapper program. The mapreduce jar takes two inputs:
A header file: I dont know the name of the file but the location and file extension and there's only one file at that location
A Input files directory
Apart from these I have an Output directory.
the processbuilder code looks like:
HEADER_PATH = INPUT_DIRECTORY+"/HEADER/*.tsv";
INPUT_FILES = INPUT_DIRECTORY+"/DATA/";
OUTPUT_DIRECTORY = OUTPUT_DIRECTORY+"/";
ProcessBuilder mapRProcessBuilder = new ProcessBuilder("yarn","jar",JAR_LOCATION,"-Dmapred.job.queue.name=name","-Dmapred.reduce.tasks=500",HEADER_PATH,INPUT_DIRECTORY,OUTPUT_DIRECTORY);
System.out.println(mapRProcessBuilder.command().toString());
Process mapRProcess = mapRProcessBuilder.start();
On run, I get the following error:
Exception in thread "main" java.io.FileNotFoundException: Requested
file /input/path/dir1/HEADER/*.tsv does not exist.
But when I run the same command as :
yarn jar jarfile.jar -Dmapred.job.queue.name=name -Dmapred.reduce.tasks=500 /input/path/dir1/HEADER/*.tsv /input/Dir /output/Dir/
It works all fine.
what can be the issue when running the command from java is causing this issue?
The * is being treated as part of the literal string in this case rather than a wildcard. Therefore globbing isn't expanding to your desired path name.
If there is only one file in the directory, why don't you find what its path is and pass that as the argument instead
eg.
File dir = new File(INPUT_DIRECTORY+"/HEADER);
if (dir.list().length > 0)
String HEADER_PATH = dir.list()[0].getAbsolutePath();
i am using jenkins in ubuntu and i need to call a java class from python script. The code:
import os
import shutil
import sys
from subprocess import call, STDOUT
param1=os.getenv(‘PARAM1’)
param2=os.getenv(‘PARAM2’)
param3=os.getenv(‘PARAM3’)
cmd1 =”cp /…/Class.class $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER/Class.class ”
cmd2=”java $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER/Class ” +””+param1+””+param2””+param3
print>>> sys.stder, “Launching command: “ + cmd2
call(cmd1,shell=True)
call(cmd2,shell=True)
But the console output shows “Error: Could not find or load main class”
I have checked an the file was copied, and Jenkis have installed the Java SE Development Kit 8u31 version.
I have try build the process in two step, first copy the java file and later set up the variables and do the second call but appears the same error.
Thanks,
i have changed the code to:
classpath=os.path.join(os.getenv('JENKINS_HOME'),"jobs",os.getenv(JOB_NAME'),"builds",os.getenv('BUILD_NUMBER'))
cmd2=[“java”,”-classpath”,classpath,”Class”,param1,param2,param3]
call(cmd2)
The code Works!!!
When i build with parameters the console output shows "Usage_ java [- options] class [args...]..."
Java doesn't support "run this file as a class" directly. Instead, you need to add the class to the classpath and then call it using the Java Fully Qualified name:
java -classpath $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER com.foo.Class ...
would run the Java code in .../builds/$BUILD_NUMBER/com/foo/Class.class
Note: Avoid call() with a string. Instead build a list of command plus arguments. That way, you can replace variables correctly and spaces in file names won't cause unexpected/hard to find problems:
classpath = os.path.join(os.genenv("JENKINS_HOME), "jobs", ...)
cmd = [
"java",
"-classpath",
classpath,
"Class",
...
]
call(cmd)
I am trying to execute hadoop fs -put <source> <destination> from Java code. When I execute this command directly from the terminal, it works fine but when I try to execute this command from within the Java code using
String[] str = {"/usr/bin/hadoop","fs -put", source, dest};
Runtime.getRuntime().exec(str);
I get error as Error: Could not find or load main class fs. I tried to execute some non-hadoop commands like ls,mkdir commands from Java and they worked fine but the hadoop commands are not getting executed even though they work fine from the terminal.
What could be the possible reason for this and how can I solve it?
JAVA API TRY: I tried to use java api to perform the copy operation but I get error. The Java code is :
String source = "/home/tmpe/file1.csv";
String dest = "/user/tmpe/file1.csv";
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://node1:8020");
FileSystem fs = FileSystem.get(conf);
Path targetPath = new Path(dest);
Path sourcePath = new Path(source);
fs.copyFromLocalFile(false,true,sourcePath,targetPath);
The error which I get is:
Exception in thread "main" java.io.IOException: Mkdirs failed to create /user/tmpe
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:378)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:564)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:545)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:452)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1230)
I have already created /user/tmpe folder and it has full read-write permissions but still this error comes. I am unable to get the issue resolved
I guess you probably do not have a HADOOP_HOME environment variable set.
But since you're in Java, why on earth would you want to do a haddop fs -put in an external process when the Java API is even more friendly than the shell ?
Came across old post but if you haven't tried already, execute it with hadoop jar app_name.jar instead of java -jar. This way if classpath of your jar does not have all hadoop jars it will fetch the jars predefined in $HADOOP_CLASSPATH.
I'm trying to execute a perl script from java with the following code:
ProcessBuilder script =
new ProcessBuilder("/opt/alert-ssdb.pl");
Process tmp = script.start();
But when I execute it it returns
java.io.IOException: Cannot run program "/opt/alert-ssdb.pl": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
at scripttest.main(scripttest.java:11)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
... 1 more
about the file
ls -l alert-ssdb.pl
-rwxr-xr-x. 1 root root alert-ssdb.pl
I tried running /usr/bin/perl/ with the script as an argument and it also failed with the same exception.
/bin/ls and other simple commands run without a problem though.
Also the first line of the script is #!/usr/bin/perl
and when run on command line it works
what am I missing?
//Update:
The big picture is that I'm trying to call the script via a storm bolt and it fails at that point.
I managed to make it work by defining a python script as a bolt
using
super(python,myscript.py)
(myscript imports the storm library) and from myscript I call the perl script.
I haven't tried yet but I suppose that If I modify the perl script to be a storm bolt it will run nicely.
Try changing
new ProcessBuilder("/opt/alert-ssdb.pl");
to:
new ProcessBuilder("/usr/bin/perl", "/opt/alert-ssdb.pl");
I've had past experiences where not all my environment variables from the shell exist when using ProcessBuilder.
Edited to reflect #dcsohl's comment.