Develop Reference

Stackoverflowerror while using distinct in apache spark

I use Spark 2.0.1.
I am trying to find distinct values in a JavaRDD as below
JavaRDD<String> distinct_installedApp_Ids = filteredInstalledApp_Ids.distinct();
I see that this line is throwing the below exception
Exception in thread "main" java.lang.StackOverflowError
at org.apache.spark.rdd.RDD.checkpointRDD(RDD.scala:226)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:84)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:84)
..........
The same stacktrace is repeated again and again.
The input filteredInstalledApp_Ids has large input with millions of records.Will thh issue be the number of records or is there a efficient way to find distinct values in JavaRDD. Any help would be much appreciated. Thanks in advance. Cheers.
Edit 1:
Adding the filter method
JavaRDD<String> filteredInstalledApp_Ids = installedApp_Ids
.filter(new Function<String, Boolean>() {
#Override
public Boolean call(String v1) throws Exception {
return v1 != null;
}
}).cache();
Edit 2:
Added the method used to generate installedApp_Ids
public JavaRDD<String> getIdsWithInstalledApps(String inputPath, JavaSparkContext sc,
JavaRDD<String> installedApp_Ids) {
JavaRDD<String> appIdsRDD = sc.textFile(inputPath);
try {
JavaRDD<String> appIdsRDD1 = appIdsRDD.map(new Function<String, String>() {
#Override
public String call(String t) throws Exception {
String delimiter = "\t";
String[] id_Type = t.split(delimiter);
StringBuilder temp = new StringBuilder(id_Type[1]);
if ((temp.indexOf("\"")) != -1) {
String escaped = temp.toString().replace("\\", "");
escaped = escaped.replace("\"{", "{");
escaped = escaped.replace("}\"", "}");
temp = new StringBuilder(escaped);
}
// To remove empty character in the beginning of a
// string
JSONObject wholeventObj = new JSONObject(temp.toString());
JSONObject eventJsonObj = wholeventObj.getJSONObject("eventData");
int appType = eventJsonObj.getInt("appType");
if (appType == 1) {
try {
return (String.valueOf(appType));
} catch (JSONException e) {
return null;
}
}
return null;
}
}).cache();
if (installedApp_Ids != null)
return sc.union(installedApp_Ids, appIdsRDD1);
else
return appIdsRDD1;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}

I assume the main dataset is in inputPath. It appears that it's a comma-separated file with JSON-encoded values.
I think you could make your code a bit simpler by combination of Spark SQL's DataFrames and from_json function. I'm using Scala and leave converting the code to Java as a home exercise :)
The lines where you load a inputPath text file and the line parsing itself can be as simple as the following:
import org.apache.spark.sql.SparkSession
val spark: SparkSession = ...
val dataset = spark.read.csv(inputPath)
You can display the content using show operator.
dataset.show(truncate = false)
You should see the JSON-encoded lines.
It appears that the JSON lines contain eventData and appType fields.
val jsons = dataset.withColumn("asJson", from_json(...))
See functions object for reference.
With JSON lines, you can select the fields of your interest:
val apptypes = jsons.select("eventData.appType")
And then union it with installedApp_Ids.
I'm sure the code gets easier to read (and hopefully to write too). The migration will give you extra optimizations that you may or may not be able to write yourself using assembler-like RDD API.
And the best is that filtering out nulls is as simple as using na operator that gives DataFrameNaFunctions like drop. I'm sure you'll like them.
It does not necessarily answer your initial question, but this java.lang.StackOverflowError might get away just by doing the code migration and the code gets easier to maintain, too.

No such element in map, why?

I am working on a code in which if I try arrays everything works fine but when i try to solve that example with list it displays
Exception in thread "main" java.util.NoSuchElementException
at java.util.AbstractList$Itr.next(Unknown Source)
at com.delete.files.DeleatingFiles.check(DeleatingFiles.java:27)
at com.delete.files.DeleatingFiles.main(DeleatingFiles.java:60)
and the code is :
Map<String, String> map = new HashMap<String, String>();
File folder = new File("F://fileIO/");
if (folder.isDirectory()) {
List<File> filesName = Arrays.asList(folder.listFiles());
Iterator<File> itList = filesName.listIterator();
while (itList.hasNext()) {
map.put(itList.next().getName(), itList.next().toString());
}
System.out.println(map);
}
} else {
System.err.println("something is wrong");
}
}
EDIT 1: All I am trying is to save file name with absolute path as key value pair.
EDIT 2: can't use as Stringnext=itrList.next() as Iterator is of File type.
Now , can anyone tell me the cause of problem ??
Please tell me if there is something wrong.
Thanks.

Your code calls next twice, so if itList has an odd number of elements, the last call would result in NoSuchElementException.
Here is how you can fix your code:
while (itList.hasNext()) {
// Call "next()" once
File next = itList.next();
// Use "next" as many times as you need
map.put(next.getName(), next.toString());
}
is there any other way i can do that without using array?
You can simplify iteration considerably by switching to "for-each" loop:
if (folder.isDirectory()) {
for (File file : folder.listFiles()) {
map.put(file.getName(), file.toString());
}
System.out.println(map);
}
Now your code does not create unnecessary copies of lists, and is free of the NoSuchElementException bug.

In your code map.put(itList.next().getName(), itList.next().toString()); call next() twice even you check itList.hasNext() once.
while (itList.hasNext()) { // Check once
map.put(itList.next().getName(), itList.next().toString()); // next() Call twice here
}
Your code may have to be corrected as like follows
Map<String, String> map = new HashMap<String, String>();
File folder = new File("F://fileIO/");
if (folder.isDirectory()) {
List<File> filesName = Arrays.asList(folder.listFiles());
Iterator<File> itList = filesName.listIterator();
File file;
while (itList.hasNext()) {
file = itList.next();
map.put(file.getName(), file.toString());
}
System.out.println(map);
}
else {
System.err.println("something is wrong");
}

as said, you were calling next twice in each loop. It can also be resolved by using java-8 streams.
Map<String, String> map = new HashMap<String, String>();
File folder = new File("F://fileIO/");
if (folder.isDirectory()) {
map = Arrays.asList(folder.listFiles())
.stream()
.collect(Collectors.toMap(File::getName,
Object::toString));
System.out.println(map);
} else {
System.err.println("Something is wrong!");
}
Map<String, String> map = folder.isDirectory ?
map = Arrays.asList(folder.listFiles())
.stream()
.collect(Collectors.toMap(File::getName,
Object::toString)) :
new HashMap<String, String>();

Is there a way to get java.util.logging.LogManager to report all loggers for which properties are specified or to access the properties?

In LogManager, the method getLoggerNames appears to only return loggers that have been actually instantiated already. However, logging properties can be held "in reserve" until a logger with a given name is instantiated.
Is there a way to get the full list of loggers for which we have settings, or to at least get the current properties set/map, without reading the original file from my own code?

JDK-8033661: readConfiguration does not cleanly reinitialize the logging system was fixed in Java version 9 which added the LogManager.updateConfiguration(Function<String,BiFunction<String,String,String>>) method. Per the documentation this method will read configuration keys and returns a function whose returned value will be applied to the resulting configuration. By supplying an identity function you can iterate the existing configuration keys instead of the actual created loggers by doing something like the following:
Function<String, BiFunction<String,String,String>> consume
= new Function<String, BiFunction<String,String,String>>() {
#Override
public BiFunction<String, String, String> apply(final String k) {
return new BiFunction<String, String, String>() {
#Override
public String apply(String o, String n) {
System.out.println(k +"="+ o);
return o;
}
};
}
};
LogManager.getLogManager().updateConfiguration(consume);
For JDK 8 and older, you have to do one of the following:
Read the logging.properties file yourself.
Override the LogManager.readConfiguration(InputStream) to capture the bytes from the stream and create your own Properties object from the stream. The no arg readConfiguration will call this method so the given stream is the properties file as bytes.
Resort to reflection (yuck!).
The easy way to read the properties file is by using the java.util.Properties class.
final Properties props = new Properties();
try {
String v = System.getProperty("java.util.logging.config.file");
if (v == null) {
v = System.getProperty("java.home") + "/lib/logging.properties";
}
final File f = new File(v).getCanonicalFile();
final InputStream in = new FileInputStream(f);
try {
props.load(in);
} finally {
in.close();
}
} catch (final RuntimeException permissionsOrMalformed) {
} catch (final Exception ioe) {
}

Freemarker: read some template parameters programmatically

I am trying to read some freemarker template values programmatically from Java as follows:
public Map<String, Object> getRootAssignments() {
Enumeration en = template.getRootTreeNode().children();
Map<String, Object> map = new HashMap<>();
while (en.hasMoreElements()) {
Object next = en.nextElement();
// instanceof private class does not work
if (next.getClass().getSimpleName().equals("Assignment")) {
map.put(getInternalState(next, "variableName").toString(),
getInternalState(next, "value").toString());
}
}
return map;
}
private Object getInternalState(Object o, String fieldName) {
Field field = null;
boolean wasAccessible = false;
try {
field = o.getClass().getDeclaredField(fieldName);
wasAccessible = field.isAccessible();
field.setAccessible(true);
return field.get(o);
} catch (Exception e) {
return "";
} finally {
if (field != null) {
field.setAccessible(wasAccessible);
}
}
}
With this I can easily read <#assign variable = "value"/>. But once I need to get some variable which needs to be evaluated (for instance <#assign variable = "first" + "second"/>), this obviously does not work.
In fact I don't really like this approach, I would prefer to read the variable states AFTER the template rendering, so nothing would have to be evaluated again. Is it feasible?

You can read back variable values if you keep the freemarker.core.Environment alive, like this:
Environment env = myTemplate.createProcessingEnvironment(root, out);
env.process(); // process the template
TemplateModel x = env.getVariable("x");
As of your initial approach, note that you are using API-s that were marked is internal, means no long term backward compatibility guarantees. But what's really brittle is reading private fields with reflection. Then calling TemplateObject.getParameterCount/getParameterRole/getParameterValue is more stable, as that's at least an API, even if internal.

Writing and reading ListMultimap<Object, Object> in file

I tried writing ListMultimap to file using Properties, but it seems impossible, refer to question Writing and reading ListMultimap to file using Properties.
Going ahead, if using Properties to store ListMultimap is not correct way, how can we store ListMultimap into a file? And how can we read back from file?
e.g. lets say I have:
ListMultimap<Object, Object> index = ArrayListMultimap.create();
How can I write methods to write this ListMultimap to file and read back from file:
writeToFile(ListMultimap multiMap, String filePath){
//??
}
ListMultimap readFromFile(String filePath){
ListMultimap multiMap;
//multiMap = read from file
return multiMap;
}

You need to decide how you will represent each object in the file. For example, if your ListMultimap contained Strings you could simply write the string value but if you're dealing with complex objects you need to produce a representation of those object as a byte[], which if you want to use Properties should then be Base64 encoded.
The basic read method should be something like:
public ListMultimap<Object, Object> read(InputStream in) throws IOException
{
ListMultimap<Object, Object> index = ArrayListMultimap.create();
Properties properties = new Properties();
properties.load(in);
for (Object serializedKey : properties.keySet())
{
String deserializedKey = deserialize(serializedKey);
String values = properties.get(serializedKey);
for (String value : values.split(","))
{
index.put(deserializedKey, deserialize(value));
}
}
return index;
}
And the write method this:
public void write(ListMultimap<Object, Object> index, OutputStream out) throws IOException
{
Properties properties = new Properties();
for (Object key : index.keySet())
{
StringBuilder values = new StringBuilder();
for (Object value = index.get(key))
{
values.append(serailize(value)).append(",");
}
properties.setProperty(serailize(key), values.subString(0, values.length - 1));
}
properties.store(out, "saving");
}
This example makes use of serialize and deserialize methods that you'll need to define according to your requirements but the signatures are:
public String serialize(Object object)
and
public Object deserialize(String s)