Develop Reference

Application on spark using java langage

I'm new on a spark , and I want to run an application on this framework using java language. I tried the following code :
public class Alert_Arret {
private static final SparkSession sparkSession = SparkSession.builder().master("local[*]").appName("Stateful Streaming Example").config("spark.sql.warehouse.dir", "file:////C:/Users/sgulati/spark-warehouse").getOrCreate();
public static Properties Connection_db () {
Properties connectionProperties = new Properties();
connectionProperties.put("user", "xxxxx");
connectionProperties.put("password", "xxxxx");
connectionProperties.put("driver","com.mysql.jdbc.Driver");
connectionProperties.put("url","xxxxxxxxxxxxxxxxx");
return connectionProperties;
}
public static void GetData() {
boolean checked = true;
String dtable = "alerte_prog";
String dtable2 = "last_tracking_tdays";
Dataset<Row> df_a_prog = sparkSession.read().jdbc("jdbc:mysql://host:port/database", dtable, Connection_db());
// df_a_prog.printSchema();
Dataset<Row> track = sparkSession.read().jdbc("jdbc:mysql://host:port/database", dtable2, Connection_db());
if (df_a_prog.select("heureDebut") != null && df_a_prog.select("heureFin") != null ) {
track.withColumn("tracking_hour/minute", from_unixtime(unix_timestamp(col("tracking_time")), "HH:mm")).show() }
}
public static void main(String[] args) {
Connection_db();
GetData();
}
}
when I run this code nothing is displayed and I get this:
0/05/11 14:00:31 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
# A fatal error has been detected by the Java Runtime Environment:
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006c40022a,pid=3376, tid=0x0000000000002e84
I use IntelliJ IDEA and version of spark: 3.0.0.

How to provide java program with external files when executing the run command in console?

So it might seem like a trivial question, but I cannot find any information out there that answers my question. Nonetheless, it is a very general coding question.
Suppose you have a java program that reads a file and creates a data structure based on the information provided by the file. So you do:
javac javaprogram.java
java javaprogram
Easy enough, but what I want to do here is to provide the program with a file specified in the command line, like this:
javac javaprogram.java
java javaprogram -file
What code do I have to write to conclude this very concern?
Thanks.

One of the best command-line utility libraries for Java out there is JCommander.
A trivial implementation based on your thread description would be:
public class javaprogram {
#Parameter(names={"-file"})
String filePath;
public static void main(String[] args) {
// instantiate your main class
javaprogram program = new javaprogram();
// intialize JCommander and parse input arguments
JCommander.newBuilder().addObject(program).build().parse(args);
// use your file path which is now accessible through the 'filePath' field
}
}
You should make sure that the library jar is available under your classpath when compiling the javaprogram.java class file.
Otherwise, in case you don't need an utility around you program argument, you may keep the program entry simple enough reading the file path as a raw program argument:
public class javaprogram {
private static final String FILE_SWITCH = "-file";
public static void main(String[] args) {
if ((args.length == 2) && (FILE_SWITCH.equals(args[0]))) {
final String filePath = args[1];
// use your file path which is now accessible through the 'filePath' local variable
}
}
}

The easiest way to do it is using -D, so if you have some file, you could call
java -Dmy.file=file.txt javaprogram
And inside you program you could read it with System.getProperty("my.file").
public class Main {
public static void main(String[] args) {
String filename = System.getProperty("my.file");
if (filename == null) {
System.exit(-1); // Or wharever you want
}
// Read and process your file
}
}
Or you could use third a party tool like picocli
import java.io.File;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
#Command(name = "Sample", header = "%n#|green Sample demo|#")
public class Sample implements Runnable {
#Option(names = {"-f", "--file"}, required = true, description = "Filename")
private File file;
#Override
public void run() {
System.out.printf("Loading %s%n", file.getAbsolutePath());
}
public static void main(String[] args) {
CommandLine.run(new Sample(), System.err, args);
}
}

You can pass file path as argument in two ways:
1)
public class Main {
public static void main(String[] args) {
if (args.length == 0) {
System.out.println("File path plz");
return;
}
System.out.println("File path: " + args[0]);
}
}
2) Use JCommander

Let's go step by step. First you need to pass the file path to your program.
Lets say you execute your program like this:
java javaprogram /foo/bar/file.txt
Strings that come after "javaprogram" will be passed as arguments to your program. This is the reason behind the syntax of the main method:
public static void main(String[] args) {
//args is the array that would store all the values passed when executing your program
String filePath = args[0]; //filePath will contain /foo/bar/file.txt
}
Now that you were able to get a the file path and name from the command-line, you need to open and read your file.
Take a look at File class and FileInputStream class.
https://www.mkyong.com/java/how-to-read-file-in-java-fileinputstream/
That should get you started.
Good luck!

RJB Hello World Example

I am trying to call a function from a java class in my Ruby on Rails project using RJB (Ruby Java Bridge).
The Java class is
public class HelloWorld {
int fifty(){
return 50 ;
}
public static void main(String[] args) {
// Prints "Hello, World" in the terminal window.
System.out.println("Hello, World");
}
}
and in the controller I have
require "rjb"
def home
myclass = Rjb::load(classpath ='\\home\\mennatallah\\TopicalClusteringofTweets\\lib\\java_libs\\helloworld.class', jvmargs=[])
myclass_instance = myclass.new
#output = myclass_instance.fifty
end
It gives " undefined method `new' for nil:NilClass "
How can I do this ?

you can try the following. it might help:
Rjb::add_jar( Dir.glob("#{Rails.root}/lib/java_libs/*.jar").join(':'))
Rjb::load(Dir.glob("#{Rails.root}/lib/java_libs/*.jar").join(':'))
test = Rjb.import('HelloWorld')
instance_class = test.new

Error looking up function in a Borland DLL through JNA

Here is a running example of my DLLService :
import com.sun.jna.Native;
import com.sun.jna.win32.StdCallLibrary;
public class DLLService {
public DLLService() throws Exception {
System.out.println("version = " + version() + "\n");
testEvaluate();
}
public void testEvaluate() {
System.out.println(this.loadFile("file", 1));
int[] barre = new int[] {0, 1, 2, 3, 4};
int[] result = new int[]{};
evaluateParty(barre.length, barre, result, 1);
System.out.println(result);
}
public int version() {
return GDLL.INSTANCE.LireNoVersion();
}
public int loadFile(String tslName, int pdwNoSess) {
return GDLL.INSTANCE.ChargerFichier();
}
public interface GDLL extends StdCallLibrary {
GDLL INSTANCE = (GDLL) Native.loadLibrary("tsr32_mini", GDLL.class);
int LireNoVersion();
int ChargerFichier();
}
}
There is no problem with the function version() but not with the loadFile(...), I have the exception :
Exception in thread "main" java.lang.UnsatisfiedLinkError: Error looking up function 'EvaluerPartie':
at com.sun.jna.Function.(Function.java:212) at
com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:541) at
com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:518) at
com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:504) at
com.sun.jna.Library$Handler.invoke(Library.java:220) at
com.sun.proxy.$Proxy0.EvaluerPartie(Unknown Source) at
DLLService.evaluateParty(DLLService.java:29) at
DLLService.testEvaluate(DLLService.java:16) at
DLLService.(DLLService.java:9) at
ProblemsIsolator.main(ProblemsIsolator.java:53)
I already search but found nothing that work (changing types, arguments, names, ...). Here is the prototype of the function l: DWORD ChargerFichier (LPSTR chNom, DWORD *pdwNoSess).
EDIT :
Same error, even after using a name mapper; it works for the LireVersion only :
public void testFunctionMapper() throws Exception {
FunctionMapper mapper = StdCallLibrary.FUNCTION_MAPPER;
NativeLibrary lib = NativeLibrary.getInstance("tsr32_mini");
Method[] methods = {
GDLL.class.getMethod("LireNoVersion",
new Class[] { String.class, int.class }),
GDLL.class.getMethod("ChargerFichier",
new Class[] { String.class, int.class }),
};
for (int i=0;i < methods.length;i++) {
String name = mapper.getFunctionName(lib, methods[i]);
lib.getFunction(name, StdCallLibrary.STDCALL_CONVENTION).getName();
}
}
Also, a dependency worker showed me well of the methods.

As it seems to works in that link: How do I get a java JNA call to a DLL to get data returned in parameters?, I can see that your definition of ChargerFichier is missing parameters.
Maybe the definition should be something like:
int ChargerFichier(PointerByReference chNom, IntByReference pdwNoSess);
Maybe this link can help you too: http://www.viaboxx.de/code/java-interoperation-with-a-native-dll-using-jna/

This is how you'd use the StdCallFunctionMapper:
Map options = new HashMap();
options.put(Library.OPTION_FUNCTION_MAPPER, new StdCallFunctionMapper());
MyLib lib = (MyLib)Native.loadLibrary("tsr32_mini", MyLib.class, options);

Calling python module from Java

I want to call a function from a python module from Java using "PythonInterpreter" and here's my Java code
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("import sys\nsys.path.append('C:\\Python27\\Lib\\site-packages')\nimport helloworld");
PyObject someFunc = interpreter.get("helloworld.getName");
PyObject result = someFunc.__call__();
String realResult = (String) result.__tojava__(String.class);
System.out.println(realResult);
and the Python code (helloworld.py) is below:
from faker import Factory
fake = Factory.create()
def getName():
name = fake.name()
return name
The problem I'm facing is while I'm calling interpreter.get when it returns a null PyObject.
Any idea what's going wrong? The python code runs fine from IDLE
EDIT
I just did a bit of change in the code as below
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("import sys\nsys.path.append('C:\\Python27\\Lib\\site-packages')\nimport helloworld");
PyInstance wrapper = (PyInstance)interpreter.eval("helloworld" + "(" + ")");
PyObject result = wrapper.invoke("getName()");
String realResult = (String) result.__tojava__(String.class);
System.out.println(realResult);
and I introduced a class in my python module
from faker import Factory
class helloworld:
def init(self):
fake = Factory.create()
def getName():
name = fake.name()
return name
Now am getting the error below
Exception in thread "main" Traceback (innermost last):
File "<string>", line 1, in ?
TypeError: call of non-function (java package 'helloworld')

You cannot use python attribute access in PythonInterpreter.get. Just use attribute name to get the module, and retrieve attribute from it.
(EDIT PART) Python code is totally broken. Please see the proper example below. And, of course, www.python.org.
(EDIT PART) First parameter of PyInstance.invoke should be method name.
Below you can find working code sample for both cases
Main.java
import org.python.core.*;
import org.python.util.PythonInterpreter;
public class Main {
public static void main(String[] args) {
test1();
test2();
}
private static void test1() {
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("import hello_world1");
PyObject func = interpreter.get("hello_world1").__getattr__("get_name");
System.out.println(func.__call__().__tojava__(String.class));
}
private static void test2() {
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("from hello_world2 import HelloWorld");
PyInstance obj = (PyInstance)interpreter.eval("HelloWorld()");
System.out.println(obj.invoke("get_name").__tojava__(String.class));
}
}
hello_world1.py
def get_name():
return "world"
hello_world2.py
class HelloWorld:
def __init__(self):
self.world = "world"
def get_name(self):
return self.world