Develop Reference

Flink - Create BroadcastStream from database properties

I have to create a BroadcastStream to be able to change the properties on the database and see the application of these properties in real time on the application.
The problems I have are 2:
1) When I read the database I need to have all the lines at the same time, via resultSet, HashMap or anything that can contain a structure of the type key-value, as some properties depend on other properties, so I cannot process them individually.
The structure of my MapStateDescriptor will be:
//String = topic name
//TopicProperties = object containing all the topic properties
MapStateDescriptor<String, TopicProperties> propertiesStateDescriptor = new MapStateDescriptor<String, TopicProperties>("properties",
BasicTypeInfo.STRING_TYPE_INFO,
BasicTypeInfo.of(new TypeHint<TopicProperties>() {}));
BroadcastStream<Row> propertiesBroadcastStream = env.createInput(JDBCInputFormat)
.map(new TopicPropertiesDbMapper()
.broadcast(propertiesStateDescriptor);
TopicPropertiesDbMapper converts what JDBCInputFormat returns to the String structure, TopicProperties.
The problem is that it is processed one row at a time, but I need to process them all together, as mentioned above.
2) Repeat the reading of the properties and update the BroadcastStream once an hour.
I specify that I have already made a version of the one above, but with the reading of the properties from file, through:
readFile (FileInputFormat, path file, FileProcessingMode, milliseconds of interval for re-reading)
it is working and I solved the two problems listed above for the database case with:
1) Set "unsplittable" flag of the FileInputFormat class to "true";
2) FileProcessingMode.PROCESS_CONTINUOUSLY.

Save a spark RDD using mapPartition with iterator

I have some intermediate data that I need to be stored in HDFS and local as well. I'm using Spark 1.6. In HDFS as intermediate form I'm getting data in /output/testDummy/part-00000 and /output/testDummy/part-00001. I want to save these partitions in local using Java/Scala so that I could save them as /users/home/indexes/index.nt(by merging both in local) or /users/home/indexes/index-0000.nt and /home/indexes/index-0001.nt separately.
Here is my code:
Note: testDummy is same as test, output is with two partitions. I want to store them separately or combined but local with index.nt file. I prefer to store separately in two data-nodes. I'm using cluster and submit spark job on YARN. I also added some comments, how many times and what data I'm getting. How could I do? Any help is appreciated.
val testDummy = outputFlatMapTuples.coalesce(Constants.INITIAL_PARTITIONS).saveAsTextFile(outputFilePathForHDFS+"/testDummy")
println("testDummy done") //1 time print
def savesData(iterator: Iterator[(String)]): Iterator[(String)] = {
println("Inside savesData") // now 4 times when coalesce(Constants.INITIAL_PARTITIONS)=2
println("iter size"+iterator.size) // 2 735 2 735 values
val filenamesWithExtension = outputPath + "/index.nt"
println("filenamesWithExtension "+filenamesWithExtension.length) //4 times
var list = List[(String)]()
val fileWritter = new FileWriter(filenamesWithExtension,true)
val bufferWritter = new BufferedWriter(fileWritter)
while (iterator.hasNext){ //iterator.hasNext is false
println("inside iterator") //0 times
val dat = iterator.next()
println("datadata "+iterator.next())
bufferWritter.write(dat + "\n")
bufferWritter.flush()
println("index files written")
val dataElements = dat.split(" ")
println("dataElements") //0
list = list.::(dataElements(0))
list = list.::(dataElements(1))
list = list.::(dataElements(2))
}
bufferWritter.close() //closing
println("savesData method end") //4 times when coal=2
list.iterator
}
println("before saving data into local") //1
val test = outputFlatMapTuples.coalesce(Constants.INITIAL_PARTITIONS).mapPartitions(savesData)
println("testRDD partitions "+test.getNumPartitions) //2
println("testRDD size "+test.collect().length) //0
println("after saving data into local") //1
PS: I followed, this and this but not exactly same what I'm looking for, I did somehow but not getting anything in index.nt

A couple of things:
Never call Iterator.size if you plan to use data later. Iterators are TraversableOnce. The only way to compute Iterator size is to traverse all its element and after that there is no more data to be read.
Don't use transformations like mapPartitions for side effects. If you want to perform some type of IO use actions like foreach / foreachPartition. It is a bad practice and doesn't guarantee that given piece of code will be executed only once.
Local path inside action or transformations is a local path of particular worker. If you want to write directly on the client machine you should fetch data first with collect or toLocalIterator. It could be better though to write to distributed storage and fetch data later.

Java 7 provides means to watch directories.
https://docs.oracle.com/javase/tutorial/essential/io/notification.html
The idea is to create a watch service, register it with the directory of interest (mention the events of your interest, like file creation, deletion, etc.,), do watch, you will be notified of any events like creation, deletion, etc., you can take whatever action you want then.
You will have to depend on Java hdfs api heavily wherever applicable.
Run the program in background since it waits for events forever. (You can write logic to quit after you do whatever you want)
On the other hand, shell scripting will also help.
Be aware of coherency model of hdfs file system while reading files.
Hope this helps with some idea.

Java how to read parameters from both file and cli?

I'm writing a tool in java and I need to provide some parameters that user can set.
I thought it is good to have ability to save all parameters in a file (and just run the .jar) and to alter saved parameters through command line.
So, I need to somehow handle parameters from two sources (priority, validity, etc.). Currently I use Apache.commons.cli to read cli-provided parameters and java.util.Properties for file-provided properties. And then I combine these properties together (and add some defaults if needed). But I don't like the result, it seems over-complicated to me.
So the code is something like this:
Properties fromFile = new Properties();
fromFile.load(new FileInputStream("settings.properties"));
cli.Options cliOptions = new cli.Options();
cliOptions.addOption(longName, shortName, hasArg, description);
//add more options
Parser parser = new DefaultParser();
CommandLine fromCli = parser.parse(cliOptions, args);
//at this point I have two different objects with properties I need,
//and I need to get every property from fromCli, check it's not empty,
// if it is, get it from fromFile, etc
So the question is: is there any library to handle properties from different sources (cli, file, defaults)? I tried googling, but did not succeed. Sorry if my googling skills are just not enough.
I'd like the code to be something like this:
import org.supertools.allPropsLib;
allPropsLib.PropsHandler handler = new allPropsLib.PropsHandler();
handler.addOptions(name, shortName, hasArg, description, defaultsTo);
handler.addSource(allPropsLib.Sources.CLI);
handler.addSource(allPropsLib.Sources.FILE);
handler.addSource(allPropsLib.Sources.DEFAULTS);
handler.setFileSource("filename");
allPropsLib.PropsContainer properties = handler.readAllProps();
// and at this point container should contain properties combined
// maybe there should be some handler function to tell the priorities,
// but I don't need to decide from where each properties should be taken

After you define the properties, load them into a java.util.Properties container regardless of the source. Then call the logic and pass it the container as a parameter.

Pentaho SDK, how to define a text file input

I'm trying to define a Pentaho Kettle (ktr) transformation via code. I would like to add to the transformation a Text File Input Step: http://wiki.pentaho.com/display/EAI/Text+File+Input.
I don't know how to do this (note that I want to achieve the result in a custom Java application, not using the standard Spoon GUI). I think I should use the TextFileInputMeta class, but when I try to define the filename the trasformation doesn't work anymore (it seems empty in Spoon).
This is the code I'm using. I think the third line has something wrong:
PluginRegistry registry = PluginRegistry.getInstance();
TextFileInputMeta fileInMeta = new TextFileInputMeta();
fileInMeta.setFileName(new String[] {myFileName});
String fileInPluginId = registry.getPluginId(StepPluginType.class, fileInMeta);
StepMeta fileInStepMeta = new StepMeta(fileInPluginId, myStepName, fileInMeta);
fileInStepMeta.setDraw(true);
fileInStepMeta.setLocation(100, 200);
transAWMMeta.addStep(fileInStepMeta);

To run a transformation programmatically, you should do the following:
Initialise Kettle
Prepare a TransMeta object
Prepare your steps
Don't forget about Meta and Data objects!
Add them to TransMeta
Create Trans and run it
By default, each transformation germinates a thread per step, so use trans.waitUntilFinished() to force your thread to wait until execution completes
Pick execution's results if necessary
Use this test as example: https://github.com/pentaho/pentaho-kettle/blob/master/test/org/pentaho/di/trans/steps/textfileinput/TextFileInputTests.java
Also, I would recommend you create the transformation manually and to load it from file, if it is acceptable for your circumstances. This will help to avoid lots of boilerplate code. It is quite easy to run transformations in this case, see an example here: https://github.com/pentaho/pentaho-kettle/blob/master/test/org/pentaho/di/TestUtilities.java#L346

create GraphHopper API for Desktop

I would like to use GraphHopper to create my own API for routing. I have taken a look into the GraphHopper.java to create my own class. I put a OSM file into the API I get a directory with edges, nodes etc. This seems to work well.
My question is, how can I load this data, so that I can call the route-Method? I try to understand the GraphHopper.java class, but my example does not work. I try to load the graph with
GHDirectory l_dir = new GHDirectory( m_graphlocation.getAbsolutePath(), DAType.RAM);
m_graph = new LevelGraphStorage( l_dir, m_EncodingManager );
Do I need the OSM file again for routing or can I use the directory with edges and nodes only?
IMHO I need a call
OSMReader l_reader = new OSMReader( l_graph, CConfiguration.getInstance().get().ExpectedCapacity).setWorkerThreads(-1).setEncodingManager(m_EncodingManager).setWayPointMaxDistance(CConfiguration.getInstance().get().WaypointMaxDistance).setEnableInstructions(false);
l_reader.doOSM2Graph(p_osm);
l_graph.optimize();
to create my graph, so is it correct to create my GraphHopperAPI class, overload the methods and on load the data with the code above and can call route?
Thanks a lot
Phil

You only need the OSM import once. In the GraphHopper class a new storage is instantiated and then loadExisting is called. If that fails you know that you need to import the OSM file and create new graphhopper files. E.g. out of my head it should be similar to this:
g = new LevelGraphStorage(dir, encodingManager);
if(!g.loadExisting()) {
reader = new OSMReader(g).setLotsOfThings
reader.doOSM2Graph..
}
The problem is that if you disable CH you can just use the GraphHopperStorage. Then you need the LocationIndex properly loaded or created at the correct place etc. Have a look into the existing unit tests where I also just use the raw stuff instead of the GraphHopper wrapper class.
But why not just create a subclass of GraphHopper and use the existing hooks (postProcessing, createWeighting, ...) to customize it to your needs?

I use this code:
GHDirectory l_dir = new GHDirectory( l_graphlocation.getAbsolutePath(), DAType.RAM_STORE);
m_graph = new LevelGraphStorage( l_dir, m_EncodingManager );
m_graph.setSegmentSize( CConfiguration.getInstance().get().SegmentSize );
if (!m_graph.loadExisting())
{
File l_osm = this.downloadOSMData();
OSMReader l_reader = new OSMReader( m_graph, CConfiguration.getInstance().get().ExpectedCapacity).setWorkerThreads(-1).setEncodingManager(m_EncodingManager).setWayPointMaxDistance(CConfiguration.getInstance().get().WaypointMaxDistance).setEnableInstructions(false);
l_reader.doOSM2Graph(l_osm);
m_graph.optimize();
m_graph.flush();
// do I need this?
PrepareRoutingSubnetworks l_preparation = new PrepareRoutingSubnetworks(m_graph, m_EncodingManager);
l_preparation.setMinNetworkSize( CConfiguration.getInstance().get().MinNetworkSize );
l_preparation.doWork();
}
// is this correct?
m_index = new LocationIndexTree(m_graph, l_dir);
m_index.setResolution( CConfiguration.getInstance().get().IndexResolution );
m_index.setSearchRegion(true);
if (!m_index.loadExisting())
throw new IOException("unable to load graph index file");
// does not work without the graph
m_graphhopper = new GraphHopper().setEncodingManager(m_EncodingManager).forDesktop();
I cannot create a working structure. I have taken a look into the test examples eg https://github.com/graphhopper/graphhopper/blob/master/core/src/test/java/com/graphhopper/GraphHopperAPITest.java but the loadGraph method cannot be called outside the package.
I would like to create a correct graph database from the OSM file, this seems to be working. Than I would like to finde the closest edge to a geo location with:
m_index.findClosest( p_position.getLatitude(), p_position.getLongitude(), EdgeFilter.ALL_EDGES );
but this returns a null pointer exception, so imho my index should be wrong. How can I create a correct working index?
After this I would like to create a fast / shortest route through the graph with
GHRequest l_request = new GHRequest( p_start.getLatitude(), p_start.getLongitude(), p_end.getLatitude(), p_end.getLongitude() );
l_request.setAlgorithm( CConfiguration.getInstance().get().RoutingAlgorithm );
return m_graphhopper.route(l_request);
but I cannot create a working graphhopper instance for call the route-method.