I am trying to find the way to run the explain command over the entire pig script in java.
I was using PigServer but it offers only to do explain over the single query (alias) not the entire script.
Is there a way to do something like:
$ pig -x local -e 'explain -script Temp1/TPC_test.pig -out explain-out9.txt'
but from my Java code?
You may use PigRunner for this purpose.
E.g:
import org.apache.pig.PigRunner;
import org.apache.pig.tools.pigstats.PigStats;
public class PigTest {
public static void main(String[] args) throws Exception {
args = new String [] {
"-x", "local",
"-e", "explain -script Temp1/TPC_test.pig -out explain-out9.txt"
};
PigStats stats = PigRunner.run(args, null);
//print plan:
//stats.getJobGraph().explain(System.out, "text", true);
}
}
I found that the following runtime dependencies are needed to avoid NoClassDefFoundError:
jackson-mapper-asl
antlr-runtime
guava
You can use org.apache.pig.PigServer to run pig scripts from Java programs:
PigServer pigServer = new PigServer(ExecType.MAPREDUCE);
pigServer.registerScript("scripts/test.pig");
Requires 'pig.properties' on classpath.
fs.default.name=hdfs://<namenode-hostname>:<port>
mapred.job.tracker=<jobtracker-hostname>:<port>
Or pass an instance of java.util.Properties to PigServer constructor.
Properties props = new Properties();
props.setProperty("fs.default.name", "hdfs://<namenode-hostname>:<port>");
props.setProperty("mapred.job.tracker", "<jobtracker-hostname>:<port>");
PigServer pigServer = new PigServer(ExecType.MAPREDUCE, props);
Hope this helps
Of course you can also use the grunt shell! (I always forget about this.)
On our site, we're using a launcher script which prepares a pig invocation command like follows:
$ pig -p param1=foo -p param2=bar script.pig
You can use explain -script in the grunt shell:
invoke pig
wrap the script call with explain
It looks like:
$ pig
grunt> explain -param param1=foo -param param2=bar script.pig
Related
I have a script file which executes the java program like:
java -Dlog.file=/path -Dappname=name -DAsyncLogger.RingBufferSize=200000 -jar /path/main.jar -f /path/testfile -day 20211108 -p /path/test.properties
-D I understand puts the key-value pair into system properties,
I am unsure what these other -f, -day, -p do.
Code:
public class Mainprogram extends PPS{
#Parameter(names={"-f", "-fileToRep1ay'}, description = "sample", required=true)
protected static String fileToReplay;
public MainProgram(Logger logger) { super(logger);}
public static void main(string[] args) throws Exception {
new MainProgram(LoggerFactory.getLogger(MainProgram.class))
.withArgs(args)
.start(new DispatcherModuIe() ,
new SessionManagerModuIe() ,
new FixHarnessModUIe(fiLeToReplay,
currentBusinessDay)) ;
When you call java, all options after the class name or -jar <jarfile> are passed as-is to the public static void main(String[] args) method. In this case, the main method of this JAR will receive the array {"-f", "/path/testfile", "-day", "20211108", "-p", "/path/test.properties"}.
What they mean and how they are interpreted is application specific, so your question cannot be answered in general.
That said, based on the context, I guess -f specifies a file (possibly input, possibly output), -day specifies a date in ISO-8601 format without dashes, and -p specifies a properties file (e.g. configuration or something like that).
And as mentioned in the comments by g00se, most professionally developed applications will provide an explanation of the options by running it with -h and/or --help.
I agree with Schokokuchen Bäcker, the options could mean anything. If it helps though, from your description, I would say:
-f - path to test file
-day - date (ISO-8859-1 format sans dashes)
-p - path to properties file
Would like to execute selenium script/batch scripts using java. Based on input parameters to call script/batch scripts.
To understand, how to run script/batch using java code.
Please help me out here.
to run a bash script contained in a file in a java project, use the ProcessBuilder class like this:
ProcessBuilder procBuildScript = new ProcessBuilder ([your_script_path],arg1,arg2,...);
procBuildScript.start();
So you can pass arguments after your script path
as "script.sh",arg1,arg2
For example :
public void runMyScript(String aFirstArg, String aSecondArg){
ProcessBuilder procBuildScript = new ProcessBuilder("./your-script.sh",aFirstArg,aSecondArg);
procBuildScript.start();
}
In your script you can call these arguments using the expressions $ 1, $ 2 ... $ {10}, $ {11} corresponding to the index where the desired parameter is located :
#!/bin/bash
# your-script.sh
echo "First argument is : $1"
echo "Third argument is : $3"
So I just received a task for creating a Java Shell App, without using any 3rd party libraries, and without using Runtime.exec() or ProcessBuilder APIs.
I don't want the solution (obviously I want to do this myself) but I do need a hint how to do this? I want the app to open a shell prompt which will accept various commands with usage of JDK 8 (Nashorn?).
Thanks!
Not really clear what you want to achieve. If you want to run a Nashhorn shell you can achieve it like this (Java 8)
import jdk.nashorn.tools.Shell;
public class NashornShell {
public static void main(String[] args) {
Shell.main(new String[]{ "-scripting"});
}
}
When you see the Nashorn prompt jjs> you can execute Linux commands...
jjs> $EXEC("ls");
which will list the current directory (using the Linux ls command).
... or execute Java commands ...
jjs> java.lang.System.out.println("foo");
... or execute JavaScript commands ...
jjs> print("foo");
For more information have a look in the nashorn guide.
edit If you want to pass only yourCommand as parameter.
NashornScriptEngineFactory factory = new NashornScriptEngineFactory();
ScriptEngine engine = factory.getScriptEngine(new String[]{"-scripting"});
String yourCommand = "ls";
Object eval = engine.eval("$EXEC(\"" + yourCommand + "\")");
System.out.println(eval);
edit do you think that instead of Nashorn I could just use raw streams directed to the OS from JVM
Following is possible
Commands.java
class Commands {
public static void main(String[] args) {
System.out.println("ls");
System.out.println("whoami");
}
}
run.sh
#!/bin/sh
java Commands | while read command; do
echo
echo "command: $command"
$command
done
But obviously this is not to recommend when you want to execute the output of Commands:
your Java application has no control about the return state of the executed single commands
if one command wait for user input your Java application don't know it
your Java application has no access to the output produced by the commands
all commands are blindly exected
and some more downsides
Regardless of what you're trying to do, using nashorn internal class like jdk.nashorn.tools.Shell is not a good idea. With java 9, this package is not an exported package of jdk.scripting.nashorn module. So, with java 9, you'll get package access failure (even in the absence of security manager - as module read/export access check is more like member access check for classes).
My Groovy app has the following code:
def doSomething() {
def props = [
"fizz",
"buzz"
]
// Do something with 'props'...
}
Please note that props could grow quite long (10+ elements) over time.
I now want to read props from a switch at the command line (so, as System properties). So I'd like my code to be something like this:
def doSomething() {
def props = System.getProperty("props")
// Do something with 'props'...
}
So I could then run the app via:
java -Dprops=??? -jar myapp.jar
But now sure how to specify an array of Strings from the command line. Any ideas?
There doesn't seem to be built-in support for this, so I ended up going with:
System.getProperty("props").split(";")
Then from the command-line:
java -Dprops="fizz;buzz;etc" -jar myapp.jar
As you said, there is no built in support but you can try this:
ArrayList props = System.getProperty("props").replaceAll("\\[","").replaceAll("]","").split(",")
and call it with:
java -Dprops="['fizz','buzz']" -jar myapp.jar
Have you thought about using a property file and using ConfigSlurper
e.g. something like
File configFile = new File(configDir, 'config.groovy')
ConfigObject config = new ConfigSlurper().parse(configFile.toURI().toURL())
def props = config.props
where config.groovy looks like
props = ["fizz","buzz","etc"]
Or using CliBuilder to pass command line args
e.g. something like
CliBuilder cli = new CliBuilder(usage: "blah:")
cli.p(argName: 'prop', longOpt: 'prop', args: Option.UNLIMITED_VALUES, required: true, valueSeparator: ',', '')
def options = cli.parse(args)
def props = options.ps // ps gives you all the -p args
then
java -jar myapp.jar -p fizz,buzz,etc
or alternate allowed syntax
java -jar myapp.jar -p fizz -p buzz -p etc
I use ubuntu 10.04 with eclipse. I created a shell script, exam.sh:
#!/bin/bash
echo "Hello World"
with chmod 755 exam.sh
On the command line, I can execute ./exam.sh // ok command showing me Hello World
I want to call this exam.sh with java code, this is my java code:
public static void main(String[] args) {
Runtime r = Runtime.getRuntime();
Process p = null;
String cmd[] = {"/bin/bash","cd","/home/erdi/Desktop", ".","/","exam.sh"};
try {
p = r.exec(cmd);
System.out.println("testing...");//ok
} catch (Exception e) {
e.printStackTrace();
}
}
This doesn't function, where did I make a mistake?
Yes I know i can search by google but I didn't find an answer to my problem. It gives howTos and tutorials about this feature but I didn't find an answer.
Try this instead:
cmd[] = {"/bin/bash", "/home/ercan/Desktop/exam.sh"};
You can just invoke bash on the shell script directly. To run a command string (like cd) you would need to use the -c switch.
If you need the working directory of the script to be your Desktop, you can use another overload of Runtime.exec:
Process proc = Runtime.getRuntime().exec(cmd, new String[0], new File("/home/ercan/Desktop"));
Alternatively, the ProcessBuilder class makes executing processes a bit nicer.