Develop Reference

Pentaho SDK, how to define a text file input

I'm trying to define a Pentaho Kettle (ktr) transformation via code. I would like to add to the transformation a Text File Input Step: http://wiki.pentaho.com/display/EAI/Text+File+Input.
I don't know how to do this (note that I want to achieve the result in a custom Java application, not using the standard Spoon GUI). I think I should use the TextFileInputMeta class, but when I try to define the filename the trasformation doesn't work anymore (it seems empty in Spoon).
This is the code I'm using. I think the third line has something wrong:
PluginRegistry registry = PluginRegistry.getInstance();
TextFileInputMeta fileInMeta = new TextFileInputMeta();
fileInMeta.setFileName(new String[] {myFileName});
String fileInPluginId = registry.getPluginId(StepPluginType.class, fileInMeta);
StepMeta fileInStepMeta = new StepMeta(fileInPluginId, myStepName, fileInMeta);
fileInStepMeta.setDraw(true);
fileInStepMeta.setLocation(100, 200);
transAWMMeta.addStep(fileInStepMeta);

To run a transformation programmatically, you should do the following:
Initialise Kettle
Prepare a TransMeta object
Prepare your steps
Don't forget about Meta and Data objects!
Add them to TransMeta
Create Trans and run it
By default, each transformation germinates a thread per step, so use trans.waitUntilFinished() to force your thread to wait until execution completes
Pick execution's results if necessary
Use this test as example: https://github.com/pentaho/pentaho-kettle/blob/master/test/org/pentaho/di/trans/steps/textfileinput/TextFileInputTests.java
Also, I would recommend you create the transformation manually and to load it from file, if it is acceptable for your circumstances. This will help to avoid lots of boilerplate code. It is quite easy to run transformations in this case, see an example here: https://github.com/pentaho/pentaho-kettle/blob/master/test/org/pentaho/di/TestUtilities.java#L346

Java RCaller reuse a function

I'm using RCaller in order to execute a R file in Java.
Basically, it looks like this:
RCaller caller = new RCaller();
RCode code = new RCode();
caller.setRscriptExecutable(PATH_TO_EXECUTABLE);
caller.cleanRCode();
code.R_source(PATH_TO_FILE);
code.addRCode("result<-test('" + param + "')");
caller.setRCode(code);
caller.runAndReturnResult("result");
... my problem is that I'm using Rcpp in this R file (PATH_TO_FILE) and it takes several seconds to compile this external C++ file in R. Therefore I want to reuse the function and not to compile this file all the time when I execute .runAndReturnResult(...).
The R file looks like this:
library(Rcpp)
sourceCpp("/cppTest.cpp")
myfunc<- test(param)
{
t<-cppTest(4)
return(t)
}
Does anyone have an idea how to achieve this using RCaller or any other suggestions in case of other libraries which are capable of this?
Help is much appreciated!
Thanks in advance,
Chris

I don't see this as a Rcpp problem. It works as designed -- your deployment within Java may hamper you.
Reuse of R code is commonly achieved by creating a package which reliably allows you to store and reload object code. That is (among other things) what a package is for.
You essentially ask for a reinvention of sourceCpp() to also give you the benefits of a package. That won't happen. Please use a package.
And with that this then has nothing to do with the embedding inside a Java app.

Running Talend Job from within Java application

I am developing a web app using Spring MVC. Simply put, a user uploads a file which can be of different types (.csv, .xls, .txt, .xml) and the application parses this file and extracts data for further processing. The problem is that I format of the file can change frequently. So there must be some way for quick and easy customization. Being a bit familiar with Talend, I decided to give it a shot and use it as ETL tool for my app. This short tutorial shows how to run Talend job from within Java app - http://www.talendforge.org/forum/viewtopic.php?id=2901
However, jobs created using Talend can read from/write to physical files, directories or databases. Is it possible to modify Talend job so that it can be given some Java object as a parameter and then return Java object just as usual Java methods?
For example something like:
String[] param = new String[]{"John Doe"};
String talendJobOutput = teaPot.myjob_0_1.myJob.main(param);
where teaPot.myjob_0_1.myJob is the talend job integrated into my app

I did something similar I guess. I created a mapping in tallend using tMap and exported this as talend job (java se programm). If you include the libraries of that job, you can run the talend job as described by others.
To pass arbitrary java objects you can use the following methods which are present in every talend job:
public Object getValueObject() {
return this.valueObject;
}
public void setValueObject(Object valueObject) {
this.valueObject = valueObject;
}
In your job you have to cast this object. e.g. you can put in a List of HashMaps and use Java reflection to populate rows. Use tJavaFlex or a custom component for that.
Using this method I can adjust the mapping of my data visually in Talend, but still use the generated code as library in my java application.

Now I better understand your willing, I think this is NOT possible because Talend's architecture is made like a standalone app, with a "main" entry point merely as does the Java main() method :
public String[][] runJob(String[] args) {
int exitCode = runJobInTOS(args);
String[][] bufferValue = new String[][] { { Integer.toString(exitCode) } };
return bufferValue;
}
That is to say : the Talend execution entry point only accepts a String array as input and doesn't returns anything as output (except as a system return code).
So, you won't be able link to Talend (generated) code as a library but as an isolated tool that you can only parameterize (using context vars, see my other response) before launching.
You can see that in Talend help center or forum the only integration described is as an "external" job execution ... :
Talend knowledge base "Calling a Talend Job from an external Java application" article
Talend Community Forum "Java Object to Talend" topic
May be you have to rethink the architecture of your application if you want to use Talend as the ETL tool for your purpose.

Now from Talend ETL point of view : if you want to parameter the execution environment of your Jobs (for exemple the physical directory of the uploaded files), you should use context variables that can be loaded at execution time from a configuration file as mentioned here :
https://help.talend.com/display/TalendOpenStudioforDataIntegrationUserGuide53EN/2.6.6+Context+settings

It's possible to unit test modifications in windows environment variable?

My process simply add some content to the system variable PATH. Actually I'm doing this with a Process that use the setx.exe:
public void changePath(String newPath ) {
String path = System.getenv("PATH") + ";";
String[] cmd = new String[]{"C:\\Windows\\System32\\setx.exe", "PATH",
path+newPath, "-m"};
ProcessBuilder builder = new ProcessBuilder(cmd);
...
}
So I tried to write a test case to it.
Class UpdatePathTest {
#Test
public void testUpdatePath() {
//call the method that update the path
changePath("C:\\somebin");
assertTrue(System.getenv("PATH").contains("C:\\somebin")); //fails
// ProcessBuilder with command String[]{"cmd", "/C", "echo", "%PATH%"}; will fail too.
//and the above in a new Thread will fail too.
}
}
So, is there any way to get the new PATH value? Writing the new path is the only option, because I'm developing a jar that will install a desktop application.

I'm not sure changing the path is a good idea in a unit test. What if the test fails? You will have to make sure you do all the relevant tidy up.
Consider inverting your dependencies and use dependency injection.
This article explains it quite well I think.
So instead of having a method that does:
public void method() {
String path = System.getenv("PATH") + ";";
//do stuff on path
}
consider doing:
public void method(String path) {
//do stuff on path
}
which allows you to stub the path. If you cannot change the signature of the method then consider using the factory pattern and using a test factory to get the path.
EDIT: after update to question
What you have to think about here is what you are testing. When you call C:\Windows\System32\setx.exe you have read the API docs and are calling it with the correct parameters. This is much like calling another method on a java API. For example if you are manipulating a list you "know" it is zero based. You do not need to test this you just read the API and the community backs you up on this. For testing changePath I think you probably what to test what is going into ProcessBuilder. Again you have read the API docs and you have to assume that you are passing in the correct variables. (See //1 at bottom) And again you have to assume that ProcessBuilder works properly and that the Oracle (or most likely Sun) guys have implemented it to the API documents.
So what you want to do is check that you are passing variables to ProcessBuilder that match the specification as you understand it. For this you can mock ProcessBuilder and then verify that you are passing the correct parameters and calling the correct method on this class.
In general it is a hard one to test because you don't want to test the windows functions but want to test java's interaction with them.
//1 The main problem I have had with calling this external commands is understanding the API documents correctly or setting up the command. Usually you have to get the command line out and check that you are using methods correctly (esp cmd functions). This can mean that you work out how to use the cmd function, code it into ProcessBuilder and then write a test (or vice versa on the ProcessBuilder/test) Not the ideal way but sometimes documents are hard to understand.

Can android execute string as code?

So basically I would like my app to read info from a database, this info would be a string that is valid java code. Once read, I would like my app to execute this string as if it were code. Is there any functionality packaged with the android sdk that supports this?
Essentially I want the phone to populate some data with information queried from a database. One of these pieces of information would be a statement like:
"return data.ZombieKillTotal >= 100000;"
Which would be used inside a statement like:
registerAchievement(new Achievement("Zombie Killer", new AchievementValidator() {
public boolean isSatisfied(Data data) { ExecStringAsCode(query result) }
});
I just don't know what to use for 'ExecStringAsCode' :(

I ended up using the JRE library and imported javax.script.*;
http://java.sun.com/developer/technicalArticles/J2SE/Desktop/scripting/

The Java programming language doesn't really provide a feature for the execution of code in text strings.
You could embed an interpreter for a more script-friendly language (perhaps Python) and execute code that way.

if you want to execute code loaded dynamicly you will have to use somthing like COdeDOM to create code and compile it then use reflection to load and excute it.