My application requires a series of values to be calculated using spark and i'm trying to make it metadata driven.
[
{
key : "myKeyName",
logic : "scala script"
}
...
]
I have a json that resembles above, which will be submitted with the "app.jar" to Spark. In the main() of spark job, I'm looking to load this json and execute the "logic" script in spark and get the value for the key. I think SparkContext.submitJob() is what I wan't but I'm not sure. Still looking for solutions in the web. Any help is deeply appreciated, thanks in advance.
The bundled jar is submitted to spark via SparkLauncher:
final SparkLauncher launcher = new SparkLauncher()
.setAppResource("path/to/app.jar")
.setMainClass("the.main.class")
.setMaster("spark.master")
.setConf(SparkLauncher.DRIVER_MEMORY, "3g");
//add the other dependent jar files
launcher.startApplication();
PS: the Spark application is implemented as a service in Docker.
Figured it out by myself.
//...
import scala.tools.nsc.interpreter.IMain;
import javax.script.ScriptContext;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
//...
private void scalaJob(SparkSession sparkSession, Dataset<Row> someData){
ScriptEngine e = new ScriptEngineManager().getEngineByName("scala");
//tell scala to use the classpath same as java
((IMain)e).settings().classpath().append(System.getProperty("java.class.path"));
//passing on some foo and bar
e.getContext().setAttribute("sparkSession",sparkSession, ScriptContext.ENGINE_SCOPE);
e.getContext().setAttribute("someData",someData, ScriptContext.ENGINE_SCOPE);
try {
//hello world
String script = "object HelloWorld {\n";
script += "def main(args: Array[String]): Unit = {\n";
script += "println(\"Hello, world!\")\n";
script += "}\n";
script += "}\n";
script += "HelloWorld.main(Array())";
e.eval(script);
//some serious work
script = "import org.apache.spark.sql.SparkSession\n";
script += "import org.apache.spark.sql.Dataset\n"
script += "import org.apache.spark.sql.Row\n";
script += "val processedData = someData.asInstanceOf[Dataset[Row]]\n";
script += "processedData.show(false)\n"
script += "processedData\n";
//getting back the result of serious work
Dataset<Row> ds = (Dataset<Row>) e.eval(script);
ds.show();
} catch (ScriptException ex) {
ex.printStackTrace();
System.exit(1);
}
}
The script is loaded from the json metadata.
PS: This is just an example, not the production code.
Related
I need to figure out a way to load content from a file containing list of ids in the preprocessing step in Jmeter. This needs to happen only once and not every time for each request. So it should be like -
Load all the list of static ids from the file once.
For every request pick one id randomly from this list.
POST the request
I am trying to explore JSR223 preprocessor but not much luck so far. Also I am not sure whether the preprocessor executes for every request which I do not want.
My current JSR Preprocessor looks something like the following -
import java.util.*;
import java.io.*;
try {
Random generator = new Random();
List<String> uuids = new ArrayList<String>();
int n = 1000;
try(BufferedReader br = new BufferedReader(new FileReader("/uuids.txt"))) {
String line = br.readLine();
while (line != null) {
uuids.add(line);
line = br.readLine();
}
}
int rn = uuids.get(generator.nextInt(n));
vars.put("some_file", "/files/" + uuids.get(rn) + ".json.gz");
} catch (Throwable ex) {
log.error("Something went wrong", ex);
throw ex;
}```
Your approach is a little bit wrong because:
JSR223 PreProcessor is executed before each request in its scope
JSR223 PreProcessor is executed by each thread (virtual user)
So I would recommend the following enhancement:
Add setUp Thread Group to your test plan
Add JSR223 Sampler to it with the following code:
SampleResult.setIgnore()
props.put('uuids', new File('uuids.txt').readLines())
this will let you read the file only once and only by one thread.
Whenever you want to access a random uuid you can use the following __groovy() function:
${__groovy(props.get('uuids').get(org.apache.commons.lang3.RandomUtils.nextInt(0\,props.get('uuids').size())),)}
More information on Groovy scripting in JMeter: Apache Groovy - Why and How You Should Use It
You can use instead JMeter's plugin bzm - Random CSV Data Set Config
Just input the CSV filename and it will generate random uuid every time
After exporting a matlab function to java package. I have tried to run the following code unix environment which contains MATLAB runtime.
import com.mathworks.toolbox.javabuilder.MWArray;
import com.mathworks.toolbox.javabuilder.MWCharArray;
import calibrationAPINew.Calibration;
public class CHIndexCalculation {
public static void main(String[] args) {
MWCharArray number1 = null;
Object[] result = null;
Calibration calibration = null;
String data = "{\r\n" +
"\"tsId\" : \"121213\",\r\n" +
"\"datapoints\" : [0.1,0.2,0.3,0.4]\r\n" +
"}";
try
{
System.out.println("Input data: " + data);
number1 = new MWCharArray(data); // Array of Input to be sent to MATLAB Runtime
System.out.println("MWCharArray: " + number1);
calibration = new Calibration();
result = calibration.calibrationAPI(1,number1);
System.out.println("Output : " + result[0]);
}
catch (Exception e)
{
System.out.println("Exception Occurred : " + e);
e.printStackTrace();
}
finally
{
//MWArray.disposeArray(number1);
//MWArray.disposeArray(result);
//calibration.dispose();
}
}
}
I have installed the matlabruntime on unix box and set the environment variables using export statement
export LD_LIBRARY_PATH=matlabrt/v92/runtime/glnxa64:matlabrt/v92/bin/glnxa64:matlabrt/v92/sys/os/glnxa64:matlabrt/v92/sys/opengl/lib/glnxa64
Now when I try to run the command ( calibrationAPINew.jar is the jar file exported from MATLAB)
java -cp javabuilder.jar:calibrationAPINew.jar:. CHIndexCalculation
I get the following exception
com.mathworks.toolbox.javabuilder.MWException: Undefined function 'pdist2mex' for input arguments of type 'double'.
at com.mathworks.toolbox.javabuilder.internal.MWMCR.mclFeval(Native Method)
at com.mathworks.toolbox.javabuilder.internal.MWMCR.access$600(MWMCR.java:31)
at com.mathworks.toolbox.javabuilder.internal.MWMCR$6.mclFeval(MWMCR.java:882)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.mathworks.toolbox.javabuilder.internal.MWMCR$5.invoke(MWMCR.java:769)
at com.sun.proxy.$Proxy0.mclFeval(Unknown Source)
at com.mathworks.toolbox.javabuilder.internal.MWMCR.invoke(MWMCR.java:443)
at calibrationAPINew.Calibration.calibrationAPI(Calibration.java:223)
at CHIndexCalculation.main(CHIndexCalculation.java:24)
I know this has something to do with MATLAB runtime but I am unable to figure out what ? The same set up works fine on windows environment.
I think the matlab program uses some mexfile which is already compiled for windows, so that it is working perfect on windows. Since the mexfile is already available it will work perfect on windows. But when your script failed to find the corresponding mexfile for linux it fails or tries to compile the source files for linux using the function 'pdist2mex' which is not included in the jar package.In windows there is no need of this function so it works fine. If this is the case then you can either add the function 'pdist2mex' to your jar from matlab or you should compile the sources to linux compatible mex-file manually using matlab and then try to repackage it then the matlab automatically wrap the corresponding mex file to the jar
Iv been looking into embedding jython into my java program to allow users to script in python. However i want to print the output of their python scripts into a java text box in my program. But i cannot find a way to embed the output of the jython engine:
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class Main {
public static void main(String[] args) throws ScriptException {
ScriptEngine pyEngine = new ScriptEngineManager().getEngineByName("python");
Object Pyoutput = pyEngine.eval("2*3");
System.out.println(Pyoutput.toString());
}
}
I tried this to get the output of eval.
This outputs 6
Which is correct however when i try the same from a print statement:
Object Pyoutput = pyEngine.eval("print('Hello World')");
System.out.println(Pyoutput.toString());
the output is null when it should be Hello World. Is there a way to print the entire output/terminal content of a script that has been eval/exec by jython?
You can set a Writer for the scripts to use through the engines ScriptContext. For example:
ScriptEngine pyEngine = new ScriptEngineManager().getEngineByName("python");
StringWriter sw = new StringWriter();
pyEngine.getContext().setWriter(sw);
pyEngine.eval("print('Hello World')");
System.out.println(sw.toString());
Prints
Hello World
I'm trying to run a Perl script file from java code but it's not working with me. I modified the Perl script and put the arguments in it instead of passing them via java code. The script works fine when running it from the command line but it's not working inside java code, always prints "wrong"!!. I wrote another Perl script (test.pl) and it's working but the desired script doesn't?? I'm working in netbeans7.3.1 (ubuntu).
Here is my code:
package program;
import java.io.*;
//import java.lang.ProcessBuilder;
/**
*
* #author seed
*/
public class Program {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws IOException,Exception {
File input = new File("//home//seed//Downloads//MADA-3.2//sample");
FileOutputStream out = new FileOutputStream(input);
PrintWriter p = new PrintWriter(out);
String s = "قصدنا في هذا القول ذكر";
p.println(s);
p.close();
Process pro = Runtime.getRuntime().exec("perl /home/seed/Downloads/MADA+TOKAN.pl");
pro.waitFor();
if(pro.exitValue() == 0)
{
System.out.println("Command Successful");
}
else{
System.out.print("wrong");}
// TODO code application logic here
}
}
My guess is that some kind of string/path conversion issue.
I see utf8 strings in your code, maybe the path is converted to something.
The filename (MADA+TOKAN.pl) contain special char, it would be better MADAplusTOKAN.pl.
Also your string in script and in question are not the same: (MADA 3.2 != MADA-3.2)
perl MADA+TOKAN.pl config=/home/seed/Downloads/mada/MADA-3.2/config files/template.madaconfig file=/home/seed/Downloads/mada/MADA 3.2/inputfile
vs
perl MADA+TOKAN.pl config=/home/seed/Downloads/MADA-3.2/config-files/template.madaconfig file=/home/seed/Downloads/MADA-3.2/sample
It sounds like it is finding your perl script and executing it, since test.perl and MADA.perl run OK.
It does sound like the arguments being passed in to the perl script are not what was expected. Can you modify the perl script to echo all its input parameters to a file?
As the title says, I'm wondering if it is possible for a program written in Java (and only java) to relaunch himself (preferably a .jar) with administrator privileges, showing in the way the native Windows UAC (in order to make it more trustable for the user), i did my homework and found out that it is possible to accomplish this using bridges between c++ and java, but i would really like to do this as a pure java project.
P.S: In the remote case that this result to be impossible, can someone show me the "easy" way to do this using another language (i mean, I've found tutorials, but they are to complicated for something I think it should not be that complicated).
P.S2: In case it is possible to accomplish this, would it work, on other platforms (OS X, Linux)
It cannot be done in pure java.
Best bet would be to write this to a file:
#echo Set objShell = CreateObject("Shell.Application") > %temp%\sudo.tmp.vbs
#echo args = Right("%*", (Len("%*") - Len("%1"))) >> %temp%\sudo.tmp.vbs
#echo objShell.ShellExecute "%1", args, "", "runas" >> %temp%\sudo.tmp.vbs
#cscript %temp%\sudo.tmp.vbs
and save it as something.bat in Windows temp directory (as we have access to this).
You would then execute this from your application using Runtime or ProcessBuilder and exit your application (System.exit(0);).
You should add an immediate start up check to your application that checks if the program has elevation, if it has proceed if not re-run the batch and exit.
Here is an example I made (this must be run when compiled as a Jar or it wont work):
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import javax.swing.JOptionPane;
/**
*
* #author David
*/
public class UacTest {
public static String jarName = "UacTest.jar", batName = "elevate.bat";
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
if (checkForUac()) {//uac is on
JOptionPane.showMessageDialog(null, "I am not elevated");
//attempt elevation
new UacTest().elevate();
System.exit(0);
} else {//uac is not on
//if we get here we are elevated
JOptionPane.showMessageDialog(null, "I am elevated");
}
}
private static boolean checkForUac() {
File dummyFile = new File("c:/aaa.txt");
dummyFile.deleteOnExit();
try {
//attempt to craete file in c:/
try (FileWriter fw = new FileWriter(dummyFile, true)) {
}
} catch (IOException ex) {//we cannot UAC muts be on
//ex.printStackTrace();
return true;
}
return false;
}
private void elevate() {
//create batch file in temporary directory as we have access to it regardless of UAC on or off
File file = new File(System.getProperty("java.io.tmpdir") + "/" + batName);
file.deleteOnExit();
createBatchFile(file);
runBatchFile();
}
private String getJarLocation() {
return getClass().getProtectionDomain().getCodeSource().getLocation().getPath().substring(1);
}
private void runBatchFile() {
//JOptionPane.showMessageDialog(null, getJarLocation());
Runtime runtime = Runtime.getRuntime();
String[] cmd = new String[]{"cmd.exe", "/C",
System.getProperty("java.io.tmpdir") + "/" + batName + " java -jar " + getJarLocation()};
try {
Process proc = runtime.exec(cmd);
//proc.waitFor();
} catch (Exception ex) {
ex.printStackTrace();
}
}
private void createBatchFile(File file) {
try {
try (FileWriter fw = new FileWriter(file, true)) {
fw.write(
"#echo Set objShell = CreateObject(\"Shell.Application\") > %temp%\\sudo.tmp.vbs\r\n"
+ "#echo args = Right(\"%*\", (Len(\"%*\") - Len(\"%1\"))) >> %temp%\\sudo.tmp.vbs\r\n"
+ "#echo objShell.ShellExecute \"%1\", args, \"\", \"runas\" >> %temp%\\sudo.tmp.vbs\r\n"
+ "#cscript %temp%\\sudo.tmp.vbs\r\n"
+ "del /f %temp%\\sudo.tmp.vbs\r\n");
}
} catch (IOException ex) {
//ex.printStackTrace();
}
}
}
Use a batch file and the runas command.
I doubt "only Java". At best you would have to have a JNI wrapper around the MSFT module. Unless just invoking the exe using ProcessBuilder counts as "only Java" -- your code to bring up the user console would be only Java but not what it invokes. IOW, Win does not come with a Java API
To relaunch your application elevated, you have to call ShellExecute or ShellExecuteEx function from Windows API and use runas verb.
You can use these API in pure Java with JNA library.
To relaunch yourself, you would have to know the full path to java.exe or javaw.exe, the command-line parameters (class path, if any, and the path to your jar). Obviously you can get this information by using Windows API.
What do you mean by remote case?
You cannot start remote elevated process this way.
You can re-launch your application elevated from a network share. Yet it won't work with mapped drives: after elevation there's no access to user's mapped drives.
No, this can't work on other platforms. UAC is a Windows feature. It's similar to sudo in Linux in some ways, so for Linux you can use sudo $pathtojava/java.exe <yourparameters>. However this won't work nicely if your application is not started from a console. Window Managers usually have wrappers which prompt for password in a GUI dialog.
Just do this with Hackaprofaw (v29). Also it was released in 2002 and started development in 1997 soooooo ye. in 2021 its on version 29.10.7 but-
if raw ram = 0
disable "featureII" program = "JAVA(math = any)"
run on "Hackaprofaw (math = v29(x))
when "featureII" disabled
end