Spark compiles with Eclipse on Windows but not javac on CentOS VM - java

I am trying to compile the spark example from
https://spark.apache.org/docs/latest/quick-start.html
/* SimpleApp.java */
import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
public class Sparktest {
public static void main(String[] args) {
String logFile = args[0] + "/README.md"; // Should be some file on your system
SparkConf conf = new SparkConf().setAppName("Simple Application");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> logData = sc.textFile(logFile).cache();
long numAs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("a"); }
}).count();
long numBs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("b"); }
}).count();
System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
}
}
if I copy the code into eclipse and add the
spark-assembly-1.3.1.2.3.0.0-2130-hadoop2.7.1.2.3.0.0-2130.jar
file to the build path through eclipse everything resolves and eclipse places my class files in the projects bin directory.
When I move the code over to my CentOS VM and execute the following line
javac -cp ./spark-assembly-1.3.1.2.3.0.0-2130-hadoop2.7.1.2.3.0.0-2130.jar:. ./Sparktest.java
in a directory with the jar file it complains about
package org.apache.spark.api does not exist
package org.apache.spark does not exist
package org.apache.spark.api.java.function does not exist
I have opened the jar file with 7zip on Windows, and know that that directory structure exists and all the class are in the jar. I'm still new to command line compiling java, but am pretty sure that there is some kind of directory structure thing going on here.
What combination of a javac command and directory structure do I need to get this to compile?

Related

How to provide java program with external files when executing the run command in console?

So it might seem like a trivial question, but I cannot find any information out there that answers my question. Nonetheless, it is a very general coding question.
Suppose you have a java program that reads a file and creates a data structure based on the information provided by the file. So you do:
javac javaprogram.java
java javaprogram
Easy enough, but what I want to do here is to provide the program with a file specified in the command line, like this:
javac javaprogram.java
java javaprogram -file
What code do I have to write to conclude this very concern?
Thanks.
One of the best command-line utility libraries for Java out there is JCommander.
A trivial implementation based on your thread description would be:
public class javaprogram {
#Parameter(names={"-file"})
String filePath;
public static void main(String[] args) {
// instantiate your main class
javaprogram program = new javaprogram();
// intialize JCommander and parse input arguments
JCommander.newBuilder().addObject(program).build().parse(args);
// use your file path which is now accessible through the 'filePath' field
}
}
You should make sure that the library jar is available under your classpath when compiling the javaprogram.java class file.
Otherwise, in case you don't need an utility around you program argument, you may keep the program entry simple enough reading the file path as a raw program argument:
public class javaprogram {
private static final String FILE_SWITCH = "-file";
public static void main(String[] args) {
if ((args.length == 2) && (FILE_SWITCH.equals(args[0]))) {
final String filePath = args[1];
// use your file path which is now accessible through the 'filePath' local variable
}
}
}
The easiest way to do it is using -D, so if you have some file, you could call
java -Dmy.file=file.txt javaprogram
And inside you program you could read it with System.getProperty("my.file").
public class Main {
public static void main(String[] args) {
String filename = System.getProperty("my.file");
if (filename == null) {
System.exit(-1); // Or wharever you want
}
// Read and process your file
}
}
Or you could use third a party tool like picocli
import java.io.File;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
#Command(name = "Sample", header = "%n#|green Sample demo|#")
public class Sample implements Runnable {
#Option(names = {"-f", "--file"}, required = true, description = "Filename")
private File file;
#Override
public void run() {
System.out.printf("Loading %s%n", file.getAbsolutePath());
}
public static void main(String[] args) {
CommandLine.run(new Sample(), System.err, args);
}
}
You can pass file path as argument in two ways:
1)
public class Main {
public static void main(String[] args) {
if (args.length == 0) {
System.out.println("File path plz");
return;
}
System.out.println("File path: " + args[0]);
}
}
2) Use JCommander
Let's go step by step. First you need to pass the file path to your program.
Lets say you execute your program like this:
java javaprogram /foo/bar/file.txt
Strings that come after "javaprogram" will be passed as arguments to your program. This is the reason behind the syntax of the main method:
public static void main(String[] args) {
//args is the array that would store all the values passed when executing your program
String filePath = args[0]; //filePath will contain /foo/bar/file.txt
}
Now that you were able to get a the file path and name from the command-line, you need to open and read your file.
Take a look at File class and FileInputStream class.
https://www.mkyong.com/java/how-to-read-file-in-java-fileinputstream/
That should get you started.
Good luck!

ImportError: No Module named demiso in Jython

I'm new to the Jython and I'm trying to run a python class using Jython in Java. But I'm running into some issues.
The Java class that I've defined:
public class DemistoCalls {
PythonInterpreter interpreter = null;
public DemistoCalls()
{
PythonInterpreter.initialize(System.getProperties(),
System.getProperties(), new String[0]);
this.interpreter = new PythonInterpreter();
}
void execfile( final String fileName )
{
this.interpreter.execfile(fileName);
}
PyInstance createClass( final String className, final String opts )
{
return (PyInstance) this.interpreter.eval(className + "(" + opts + ")");
}
public static void main(String[] args) {
DemistoCalls demistoCalls = new DemistoCalls();
demistoCalls.execfile("C:\\Users\\AlokNath\\Desktop\\Demisto_Project\\demisto-py-master\\demisto\\SimpleConnect.py");
}
}
The SampleConnect.py file that I'm trying to run:
import sys
sys.path.append("C:\Users\AlokNath\Desktop\Demisto_Project\demisto-py-
master\demisto")
import demisto
While Running the java file, I'm getting this error:
File "C:\Users\AlokNath\Desktop\Demisto_Project\demisto-py-master\demisto\SimpleConnect.py", line 3, in <module>
import demisto
ImportError: No module named demisto
Although I've defined the "demisto" module in the system path and checked that the system path in python contains the appropriate path to Jython 2.7.lb2 Library. I'm not sure where am I going wrong. Any help is appreciated.
Regards,
Alok
I found a solution to the import error problem. We need to copy the missing module to the “site-packages” folder under "modeler-installation/lib/jython/Lib". This will resolve the dependency problem.

groovy win cmd line class and script

I'm trying to run a groovy(2.4.3) script on windows that calls a goovy class xxxxx.groovy. I've tried a number of variations using classpath and various scripts, some examples below, always getting MultipleCompliationErrorsException.... unable to resolve class
classfile is firstclass.groovy
import org.apache.commons.io.FilenameUtils
class firstclassstart {
def wluid, wlpwd, wlserver, port
private wlconnection, connectString, jmxConnector, Filpath, Filpass, Filname, OSRPDpath, Passphrase
// object constructor
firstclassstart(wluid, wlpwd, wlserver, port) {
this.wluid = wluid
this.wlpwd = wlpwd
this.wlserver = wlserver
this.port = port
}
def isFile(Filpath) {
// Create a File object representing the folder 'A/B'
def folder = new File(Filpath)
if (!org.apache.commons.io.FilenameUtils.isExtension(Filpath, "txt")) {
println "bad extension"
return false
} else if (!folder.exists()) {
// Create all folders up-to and including B
println " path is wrong"
return false
} else
println "file found"
return true
}
}
cmd line script test.groovy
import firstclass
def sample = new firstclass.firstclassstart("weblogic", "Admin123", "x.com", "7002")
//def sample = new firstclassstart("weblogic", "Admin123", "x.com", "7002")
sample.isFile("./firstclass.groovy")
..\groovy -cp "firstclass.groovy;commons-io-1.3.2.jar" testfc.groovy
script test.groovy
GroovyShell shell = new GroovyShell()
def script = shell.parse(new File('mylib/firstclass.groovy'))
firstclass sample = new script.firstclass("uid", "pwd", "url", "port")
sample.getstatus()
c:>groovy test.groovy
script test.groovy v2 put firstclass.groovy in directory test below script
import test.firstclass
firstclass sample = new script.firstclass("uid", "pwd", "url", "port")
sample.getstatus()
c:>groovy test.groovy
just looking for a bullet proof, portable way to oranize my java classes, .groovy classess, etc. and scripts.
Thanks
I think that you can do using for example your first approach:
groovy -cp mylib/firstclass.groovy mylib/test.groovy
However I see some problems in your code which are probably causing MultipleCompliationErrorsException.
Since you're including firstclass.groovy in your classpath, you've to add the import firstclass in the test.groovy.
Why are you using script.firstclass in test.groovy? you're class is called simply firstclass.
In your firstclass.groovy you're using import org.apache.commons.io.FilenameUtils and probably other, however you're not including it in the classpath.
So finally I think that, you've to change your test.groovy for something like:
import firstclass
firstclass sample = new firstclass("uid", "pwd", "url", "port")
sample.getstatus()
And in your command add the remaining includes for apache Commons IO to the classpath.
groovy -cp "mylib/firstclass.groovy;commons-io-2.4.jar;" mylib/testexe.groovy
Hope this helps,
UPDATE BASED ON OP CHANGES:
After the changes you've some things wrong, I try to enumerate it:
If your file is called firstclass.groovy your class must be class firstclass not class firstclassstart.
In your test.groovy use new firstclass not new firstclass.firstclassstart.
So the thing is, your code must be:
class file firstclass.groovy:
import org.apache.commons.io.FilenameUtils
class firstclass {
def wluid, wlpwd, wlserver, port
private wlconnection, connectString, jmxConnector, Filpath, Filpass, Filname, OSRPDpath, Passphrase
// object constructor
firstclass(wluid, wlpwd, wlserver, port) {
this.wluid = wluid
this.wlpwd = wlpwd
this.wlserver = wlserver
this.port = port
}
def isFile(Filpath) {
// Create a File object representing the folder 'A/B'
def folder = new File(Filpath)
if (!org.apache.commons.io.FilenameUtils.isExtension(Filpath, "txt")) {
println "bad extension"
return false
} else if (!folder.exists()) {
// Create all folders up-to and including B
println " path is wrong"
return false
} else
println "file found"
return true
}
}
script test.groovy:
import firstclass
def sample = new firstclass("weblogic", "Admin123", "x.com", "7002")
sample.isFile("./firstclass.groovy")
Finally the command to execute it:
groovy -cp "firstclass.groovy;commons-io-1.3.2.jar" test.groovy
With this changes your code must works, I try it and works as expected.

Netbeans imported JAR not loading on run

i see this question has been posted many times but it has been solved with adding
-Djava.library.path="./path" to the VM runtime options.
I have to build an app in JAVA which uses the JNotify classes.
this is the sample code:
package test;
import net.contentobjects.jnotify.JNotify;
import net.contentobjects.jnotify.JNotifyListener;
/**
*
* #author
*/
public class Test {
public void jnotifydemo() throws Exception {
// path to watch
String path = System.getProperty("user.home");
// watch mask, specify events you care about,
// or JNotify.FILE_ANY for all events.
int mask = JNotify.FILE_CREATED
| JNotify.FILE_DELETED
| JNotify.FILE_MODIFIED
| JNotify.FILE_RENAMED;
// watch subtree?
boolean watchSubtree = true;
// add actual watch
int watchID = JNotify.addWatch(path, mask, watchSubtree, new Listener());
// sleep a little, the application will exit if you
// don't (watching is asynchronous), depending on your
// application, this may not be required
Thread.sleep(1000000);
// to remove watch the watch
boolean res = JNotify.removeWatch(watchID);
if (!res) {
// invalid watch ID specified.
}
}
class Listener implements JNotifyListener {
public void fileRenamed(int wd, String rootPath, String oldName,
String newName) {
print("renamed " + rootPath + " : " + oldName + " -> " + newName);
}
public void fileModified(int wd, String rootPath, String name) {
print("modified " + rootPath + " : " + name);
}
public void fileDeleted(int wd, String rootPath, String name) {
print("deleted " + rootPath + " : " + name);
}
public void fileCreated(int wd, String rootPath, String name) {
print("created " + rootPath + " : " + name);
}
void print(String msg) {
System.err.println(msg);
}
}
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws Exception {
System.out.println("Hello World");
new Test().jnotifydemo();
}
}
When i run this i get:
Error loading library, java.library.path=C:\Program Files\Java\jdk1.6.0_26\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;(continues)
Exception in thread "main" java.lang.UnsatisfiedLinkError: no jnotify in java.library.path
I have setup a Netbeans project and added the JAR file to the project so that the JAR is correctly in the lib/ folder of my project and everything is set in NETBEANS.
This correctly works if is setup the -Djava.library.path="./path" argument of the java VM, but if i imported my lib in NETBEANS that should be included in the path automatically.
I am doing something wrong or it is necessary to put every .jar in the classpath system variable? I would like to release this app so it can run on other systems that does not have JNotify in their libs.
Thanks
I am using Netbeans 7.2 on Win 7 32Bit
You are messing java jar files as library which has to be added only in netbeans classpath:
Simply in NetBeans on project properties click and adjust Library having your JAR file.
For the native libraries (so,dll,...) you need to have set: -Djava.library.path. As you did in your question.
So you have 2 steps:
1. from http://sourceforge.net/projects/jnotify/files/jnotify/jnotify-0.94/jnotify-lib-0.94.zip/download add jnotify-0.94.jar to your libraries as in picture above (this will update your classpath automatically)
2. jnotify.dll, or jnotify_64bit.dll for 64-bit windows place is some directory and ad this to your -Djava.library.path - add this to VM option of the projects property

Running java program from another java program

I'm trying to run a Java program from another Java application. Here is my code:
public class Main {
public static int Exec() throws IOException {
Process p = Runtime.getRuntime().exec("javac -d C:/Users/Dinara/Desktop/D/bin "
+ "C:/Users/Dinara/Desktop/D/src/test.java");
Process p1 = Runtime.getRuntime().exec("java -classpath C:/Users/Dinara/Desktop/D/bin test");
return 0;
}
public static void main(String[] args) throws IOException {
Exec();
}
}
javac works fine and creates test.class file in bin directory. However java -classpath C:/Users/Dinara/Desktop/D/bin test does not run the test.class file.
the content of the test.java:
import java.io.*;
class test {
public static void main(String args[]) {
try {
FileWriter fstream = new FileWriter("out.txt");
BufferedWriter out = new BufferedWriter(fstream);
out.write("Hello Java");
out.close();
} catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
}
}
I suppose that something wrong with recognizing Java command. Could you please give me a sample code for fixing this problem or share idea? I'm using Netbeans to run Main class and the location of the application folder is C:\Users\Dinara\Main
Use
System.getProperty("java.home") + "/bin/java -classpath C:/Users/Dinara/Desktop/D/bin test"
instead of
"java -classpath C:/Users/Dinara/Desktop/D/bin test"
You need to supply the full path to the javac, exec won't use the ath to find it for you

Categories

Resources