I have a java code that generates pig script. I am wondering if there is option to execute that script directly within the java code. I found there is a option to embed pig script execution inside java code using PigServer class.
The problem is that I'm using AvroStorage to store the results and the class contains method Store() that apparently uses file storage.
Is there any way how to execute my pig script using AvroStorage inside JAVA using PigServer class?
Its generic code from their DOC, they use pigServer.store("D", "myoutput"); but instead of the file i need to call AvroStorage.
public class WordCount {
public static void main(String[] args) {
PigServer pigServer = new PigServer();
try {
pigServer.registerJar("/mylocation/tokenize.jar");
runMyQuery(pigServer, "myinput.txt";
}
catch (IOException e) {
e.printStackTrace();
}
}
public static void runMyQuery(PigServer pigServer, String inputFile) throws IOException {
pigServer.registerQuery("A = load '" + inputFile + "' using TextLoader();");
pigServer.registerQuery("B = foreach A generate flatten(tokenize($0));");
pigServer.registerQuery("C = group B by $1;");
pigServer.registerQuery("D = foreach C generate flatten(group), COUNT(B.$0);");
pigServer.store("D", "myoutput");
}
}
Related
I have a spring boot project and out of which i have created a jar, and i am calling this jar from another project by passing arguments.
Not able to get the output and it is getting stuck.
The below is the project from which i am getting a jar.
public class Demo1Application {
public static void main(String[] args) {
System.out.println("jar called");
for(String arg : args) {
System.out.println("next argument is"+ arg );
}
SpringApplication.run(Demo1Application.class, args);
}
}
Its a simple spring boot main class.
The below is the class of another project from which i want to invoke this jar by passing arguments.
public class AAAAAAAAAAAAAAAAA {
public static void main(String[] args) throws IOException, InterruptedException {
File jarFile = new File("D:\\NewConfigWorkSpace\\Demo1\\target\\Demo1-0.0.1-SNAPSHOT.jar");
Process p = Runtime.getRuntime().exec("java -jar D:\\NewConfigWorkSpace\\Demo1\\target\\Demo1-0.0.1-SNAPSHOT.jar bisnu mohan");
p.waitFor();
System.out.println("finished");
}
}
How to see the console when i am calling the jar, how to track how much execution has been happened.
What you need is the input stream of created process. This is what is normally returned to a console when you run your application.
Process p = Runtime.getRuntime().exec("java -jar D:\\NewConfigWorkSpace\\Demo1\\target\\Demo1-0.0.1-SNAPSHOT.jar bisnu mohan");
InputStream inputStream = p.getInputStream();
You can then read contents of it and print to a console of a running process like this:
StringBuilder outputLines = new StringBuilder();
String output;
try (BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, UTF_8))) {
String line;
while ((line = reader.readLine()) != null) {
logger.info("Command execution output: " + line);
outputLines.append(line).append("\n");
}
} finally {
output = outputLines.toString().trim();
}
It would also be a good idea to handle error stream the same way, because you can then see if your subprocess returns some errors.
InputStream errorStream = p.getErrorStream();
Use this streams handling in a separate threads so they don't block each other.
From my point of view, you don't need to execute JAR via command line.
If you include JAR into your project you can just import SpringApplication from JAR and run it directly like this:
public class AAAAAAAAAAAAAAAAA {
public static void main(String[] args) throws IOException, InterruptedException {
SpringApplication.run(Demo1Application.class, args);
}
}
I am new to Java programming. Using eclipse for coding. I want to execute JarFile.jar. Following code works fine when placing JarFile.jar inside the same directory where the project is created.
public class CmdTest2
{
public static void main(String[] args)
{
final String dosCommand = "java -jar \"D:\\Java_codes\\CmdTest2\JarFile.jar\"
try
{
final Process process = Runtime.getRuntime().exec(dosCommand);
final InputStream in = process.getInputStream();
int ch;
while((ch = in.read()) != -1)
{
System.out.print((char)ch);
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
But I want to execute the jar file from any other directory (say E:\Test\JarFile.jar). How can I do that? (i.e. changing directory through as well as triggering jar file).
In continuation to above pl. let me know how to run MS-DOS commands through Java.
I have a java project, which complied into an executable jar file v-agent-exe.jar. This jar is a log server, log rows is sent to it for processing.
I can execute it by using this command:
`java -jar v-agent-exe.jar -a watch -f config.ini`.
After executed, this jar file will create a ServerSocket at port 1235 and listen for incoming data from clients. After data received, the program will process the data and send the result back to the client. When I execute the jar from CMD windows, the processing is working perfect.
Now I am trying to wrap the Jar file as a Windows service (I am using Windows 10). I created a "Windows service project"
in Visual studio like below:
- Caller class have call() method to execute the jar file using process.
- AgentService is the service, which execute Caller->call() in another thread.
- Program is the main entry to load AgentService.
Caller.cs
public class Caller
{
static Process proc;
public Process GetProcess(){
return proc;
}
public void call() {
try
{
String dir = AppDomain.CurrentDomain.BaseDirectory;
proc = new Process
{
StartInfo = new ProcessStartInfo
{
WorkingDirectory = dir,
FileName = "java.exe",
Arguments = #"-jar v-agent-exe.jar -a watch -f config.ini",
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = true,
CreateNoWindow = true
}
};
proc.Start();
while (!proc.StandardError.EndOfStream)
{
string line = proc.StandardError.ReadLine();
}
}
catch (Exception ex) {
VAgentService.writeLog("Error when call process: " + ex.Message);
}
}
}
AgentService
public partial class AgentService : ServiceBase
{
private string jarPath;
private string iniPath;
static Process proc;
Caller caller;
public AgentService()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
writeLog("On start");
try
{
caller = new Caller();
writeLog("Prepare to launch thread");
Thread t = new Thread(new ThreadStart(caller.call));
t.Start();
}
catch (Exception ex)
{
EventLog.WriteEntry("Demo error: " + ex.Message);
}
}
protected override void OnStop()
{
proc = caller.GetProcess();
if (proc != null && !proc.HasExited)
{
proc.Kill();
}
else
{
...
}
}
}
Program.cs
static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main(String[] args)
{
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new AgentService()
};
ServiceBase.Run(ServicesToRun);
}
}
After build the the service project, I have AgentService.exe.
I install it to my system using:
sc create VAgentLogging binpath= %CD%\AgentService.exe depend= lmhosts start= auto
After start the service in service.msc, I can telnet to port "1235" which the java process is listening (I am sure about
only the jar running in this port). According to the
log of java program, it still can received some part of data but seem like it cannot send back to client or something,
which cause the followed process cannot be done.
I think my problem is: the jar file can executed as standalone but somehow it sucks when wrapped under my service project.
I haven't posted the jar's code yet because I think the error is related to the Windows service project. If you need the java code, please tell me and I will update it here.
Any help would be appreciated.
I have a Javascript program that call a Java program via Applet with the following command inside the Applet:
script.code = "InJava.class";
It was necessary to make a change in the Java program, and now it creates two classes: InJava.class and InJava$1.class
Now, when I run the program in Javascript, the program does not answer anything.
If I change the statement to:
script.code = "InJava$1.class"
It returns the following error:
runtimeexceptionjava.lang.reflect.invocationtargetexception
How must I stated in the Javascript program?
Must I change something in my Java program?
The Java program is below:
Java Program
public void sayHello() throws IOException {
java.io.FileInputStream fis = AccessController.doPrivileged(new PrivilegedAction<FileInputStream>() {
public FileInputStream run() {
try {
return new FileInputStream(parametro);
} catch (IOException e) {
String retorno_exc = e.toString();
}
return null;
}
});
The following code works when I execute the Pig script locally while specifying a local GeoIPASNum.dat file. However, it does not work when run in MapReduce distributed mode. What am I missing?
Pig job
DEFINE AsnResolver AsnResolver('/hdfs/location/of/GeoIPASNum.dat');
loaded = LOAD 'log_file' Using PigStorage() AS (ip:chararray);
columned = FOREACH loaded GENERATE AsnResolver(ip);
STORE columned INTO 'output/' USING PigStorage();
AsnResolver.java
public class AsnResolver extends EvalFunc<String> {
String ipAsnFile = null;
#Override
public String exec(Tuple input) throws IOException {
try {
LookupService lus = new LookupService(ipAsnFile,
LookupService.GEOIP_MEMORY_CACHE);
return lus.getOrg((String) input.get(0));
} catch (IOException e) {
}
return null;
}
public AsnResolver(String file) {
ipAsnFile = file;
}
...
}
The problem is that you are using a string reference to an HDFS path and the LookupService constructor can't resolve the file. It probably works when you run it locally since the LookupService has no problem with a file in your local FS.
Override the getCacheFiles method:
#Override
public List<String> getCacheFiles() {
List<String> list = new ArrayList<String>(1);
list.add(ipAsnFile + "#GeoIPASNum.dat");
return list;
}
Then change your LookupService constructor to use the Distributed Cache reference to "GeoIPASNum.dat" :
LookupService lus = new LookupService("GeoIPASNum.dat", LookupService.GEOIP_MEMORY_CACHE);
Search for "Distributed Cache" in this page of the Pig docs: http://pig.apache.org/docs/r0.11.0/udf.html
The example it shows using the getCacheFiles() method should ensure that the file is accessible to all the nodes in the cluster.