I'm trying to use a Scala library in my Java program and I have some difficulties to convert a complex Scala Map to Java.
My used Scala object method has the following return type: scala.collection.mutable.Map<String, Map<Object, Seq<Object>>>
How do I convert that to a Java equivalent of Map<String, Map<Object, List<Object>>> ?
I already played around with the JavaConversions and JavaConvertors packages but no luck :(
public void getPartitionAssignmentForTopics(final List<String> topics) {
final Seq<String> seqTopics = scala.collection.JavaConversions.asScalaBuffer(topics).toList();
scala.collection.mutable.Map<String, Map<Object, Seq<Object>>> map2 = zkUtils
.getPartitionAssignmentForTopics(seqTopics);
val map:scala.collection.mutable.Map[String, Map[Object, Seq[Object]]] = scala.collection.mutable.Map()
map:
collection.mutable.Map[String, Map[Object, Seq[Object]]] =Map()
map.mapValues(_.mapValues(_.asJava).asJava).asJava
res2:
java.util.Map[String, java.util.Map[Object, java.util.List[Object]]] ={}
}
This does not compile :)
With playing around I meant that I use the following code to convert from Scala Seq to Java List:
scala.collection.JavaConversions.seqAsJavaList(zkUtils.getAllTopics());
I ended up with the following code. Not really nice :D
public java.util.Map<String, java.util.Map<Integer, java.util.List<Integer>>> getPartitionAssignmentForTopics(final List<String> topics) {
final scala.collection.Seq<String> seqTopics = scala.collection.JavaConversions.asScalaBuffer(topics).toList();
scala.collection.mutable.Map<String, scala.collection.Map<Object, scala.collection.Seq<Object>>> tmpMap1 =
zkUtils.getPartitionAssignmentForTopics(seqTopics);
final java.util.Map<String, java.util.Map<Integer, java.util.List<Integer>>> result = new HashMap<>();
java.util.Map<String, Map<Object, Seq<Object>>> tmpMap2 = JavaConversions.mapAsJavaMap(tmpMap1);
tmpMap2.forEach((k1, v1) -> {
String topic = (String)k1;
java.util.Map<Object, Seq<Object>> objectSeqMap = JavaConversions.mapAsJavaMap(v1);
java.util.Map<Integer, List<Integer>> tmpResultMap = new HashMap<>();
objectSeqMap.forEach((k2, v2) -> {
Integer tmpInt = (Integer)k2;
List<Integer> tmpList = (List<Integer>)(Object)JavaConversions.seqAsJavaList(v2);
tmpResultMap.put(tmpInt, tmpList);
});
result.put(topic, tmpResultMap);
});
return result;
}
Related
I have an object with structure
List<Map<String(k1), Map<String(k2), String(v2)>>>
I need to convert the above list to
List<Map<String(k2), Map<String(k1), String(v2)>>>
I am stuck on how do i get the nested map using construct like
serviceResults.stream().map((k, v) -> ????)
that will allow me to swap the keys. Is it possible to do it in a way without using loops using Java 8 streams?
Additional Info
This is the code that uses loop construct
List<Map<String, Map<String, String>>> serviceResults = new ArrayList<>();
//Populate the above list
Map<String, Map<String, String>> swpMapOuter = new HashMap<>();
Map<String, String> swpMapInner = new HashMap<>();
for (Map<String, Map<String, String>> stringMapMap : serviceResults) {
for (Map.Entry<String, Map<String, String>> s : stringMapMap.entrySet()) {
String key1 = s.getKey();
Map<String, String> value1 = s.getValue();
for (Map.Entry<String, String> s1 : value1.entrySet()) {
String key2 = s1.getKey();
String value2 = s1.getValue();
swpMapInner.put(key1, value2);
swpMapOuter.put(key2, swpMapInner);
}
}
}
System.out.println("swpMapOuter " + swpMapOuter);
Below is the code with forEach, instead of for loops, but was wondering, if it could be implemented using Stream constructs
Map<String, Map<String, String>> swpMapOuter2 = new HashMap<>();
Map<String, String> swpMapInner2 = new HashMap<>();
serviceResults.forEach((stringMapMap) -> {
stringMapMap.entrySet().forEach((s) -> {
String key1 = s.getKey();
Map<String, String> value1 = s.getValue();
value1.entrySet().forEach((s1) -> {
String key2 = s1.getKey();
String value2 = s1.getValue();
swpMapInner2.put(key1, value2);
swpMapOuter2.put(key2, swpMapInner2);
});
});
});
System.out.println("swpMapOuter2 " + swpMapOuter2);
I have a list of Maps as below:
List<Map<String,Object>> someObjectsList = new ArrayList<Map<String,Object>>();
I am storing the following data in each HashMap
key value
2017-07-21 2017-07-21-07.33.28.429340
2017-07-24 2017-07-24-01.23.33.591340
2017-07-24 2017-07-24-01.23.33.492340
2017-07-21 2017-07-21-07.33.28.429540
I want to iterate through the list of HashMaps and check if the key matches with the first 10 characters of any of the HashMap value, then I want to store those keys and values in the following format. i.e. by using the telemeter 'comma'. The ultimate aim is to group the unique keys of the HashMaps and their relative values (if the key matches with the first 10 characters of any of the HashMap value) in a new HashMap.
key value
2017-07-21 2017-07-21-07.33.28.429340,2017-07-21-07.33.28.429540
2017-07-24 2017-07-24-01.23.33.591340,2017-07-24-01.23.33.492340
I am trying with following java code using StringJoiner, but not getting the results as expected. Any clue on how to frame the logic here?
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.StringJoiner;
public class SampleOne {
public static void main(String[] args) {
// TODO Auto-generated method stub
List<Map<String, Object>> someObjectsList = new ArrayList<Map<String, Object>>();
Map<String, Object> mapOne = new HashMap<String, Object>();
mapOne.put("2017-07-21", "2017-07-21-07.33.28.429340");
Map<String, Object> mapTwo = new HashMap<String, Object>();
mapTwo.put("2017-07-24", "2017-07-24-01.23.33.591340");
Map<String, Object> mapThree = new HashMap<String, Object>();
mapThree.put("2017-07-24", "2017-07-24-01.23.33.492340");
Map<String, Object> mapFour = new HashMap<String, Object>();
mapFour.put("2017-07-21", "2017-07-21-07.33.28.429540");
someObjectsList.add(mapOne);
someObjectsList.add(mapTwo);
someObjectsList.add(mapThree);
someObjectsList.add(mapFour);
for (Map map : someObjectsList) {
StringJoiner sj = new StringJoiner(",");
for (Object key : map.keySet()) {
String value = ((String) map.get(key));
String date = value.substring(0, Math.min(value.length(), 10));
//System.out.println(str);
//System.out.println(value);
if(key.equals(date)) {
sj.add(value);
System.out.println(sj.toString());
}
}
}
}
}
output:
2017-07-21-07.33.28.429340
2017-07-24-01.23.33.591340
2017-07-24-01.23.33.492340
2017-07-21-07.33.28.429540
Make use of the .merge function:
Map<String, Object> finalMap = new HashMap<String, Object>();
for (Map map : someObjectsList) {
for (Object key : map.keySet()) {
String value = ((String) map.get(key));
finalMap.merge((String) key, value, (k, v) -> k + "," + v);
}
}
which outputs:
{2017-07-21=2017-07-21-07.33.28.429340,2017-07-21-07.33.28.429540,
2017-07-24=2017-07-24-01.23.33.591340,2017-07-24-01.23.33.492340}
The same can be achieved by the following one-liner:
someObjectsList.stream()
.flatMap(i -> i.entrySet().stream())
.collect(Collectors.toMap(Entry::getKey, Entry::getValue,
(k, v) -> k + "," + v));
On your code, you are using different StringJoiner on each map. So, it's creating a new instance of it.
You can save your keys on a map. An example code:
(Edit: I did not remove your StringJoiner part.)
public static void main(String[] args) {
// TODO Auto-generated method stub
List<Map<String, Object>> someObjectsList = new ArrayList<Map<String, Object>>();
Map<String, Object> mapOne = new HashMap<String, Object>();
mapOne.put("2017-07-21", "2017-07-21-07.33.28.429340");
Map<String, Object> mapTwo = new HashMap<String, Object>();
mapTwo.put("2017-07-24", "2017-07-24-01.23.33.591340");
Map<String, Object> mapThree = new HashMap<String, Object>();
mapThree.put("2017-07-24", "2017-07-24-01.23.33.492340");
Map<String, Object> mapFour = new HashMap<String, Object>();
mapFour.put("2017-07-21", "2017-07-21-07.33.28.429540");
someObjectsList.add(mapOne);
someObjectsList.add(mapTwo);
someObjectsList.add(mapThree);
someObjectsList.add(mapFour);
Map<String, Object> outputMap = new HashMap<String, Object>();
for (Map map : someObjectsList) {
StringJoiner sj = new StringJoiner(",");
for (Object key : map.keySet()) {
String value = ((String) map.get(key));
String date = value.substring(0, Math.min(value.length(), 10));
//System.out.println(str);
//System.out.println(value);
if(key.equals(date)) {
sj.add(value);
System.out.println(sj.toString());
if(outputMap.containsKey(key)) {
String str = (String) map.get(key);
str = str + "," + value;
outputMap.put((String)key, str);
} else {
outputMap.put((String)key, value);
}
}
}
}
for (String map : outputMap.keySet()) {
System.out.println(map + " " + outputMap.get(map));
}
}
You are looking for the grouping behavior of processing a List. You can use the advantage of java-stream since java-8. In any case, you need a new Map to store the values in order to print them. :
someObjectsList.stream()
.flatMap(i -> i.entrySet().stream()) // flatmapping to entries
.collect(Collectors.groupingBy(Entry::getKey)) // grouping them using the key
In case you want to use for-loops. In this case it is harder since the more entries might appear in each List item:
final Map<String, List<Object>> map = new HashMap<>();
for (Map<String, Object> m: someObjectsList) { // iterate List<Map>
for (Entry<String, Object> entry: m.entrySet()) { // iterate entries of each Map
List<Object> list;
final String key = entry.getKey(); // key of the entry
final Object value = entry.getValue(); // value of the entry
if (map.containsKey(key)) { // if the key exists
list = map.get(key); // ... use it
} else {
list = new ArrayList<>(); // ... or else create a new one
}
list.add(value); // add the new value
map.put(key, list); // and add/update the entry
}
}
Printing out of Map<String, List<Object>> map in both cased will produce the following output:
2017-07-21=[2017-07-21-07.33.28.429340, 2017-07-21-07.33.28.429540],
2017-07-24=[2017-07-24-01.23.33.591340, 2017-07-24-01.23.33.492340]
Any reason you're using Object over String and avoiding safety checks? That said, it's not "the first 10 characters", you want to see if value starts with key full-stop (all your keys are 10 characters). So in that case you can just do if (value.startsWith(key)) { ... }. Don't forget your newlines if the stringjoiner wasn't full. Lastly, you don't need a List, a Map can hold multiple keys at once. An alternative way of doing it:
//LinkedHashMap will preserve our insertion order
Map<String, String> map = new LinkedHashMap<>();
map.put("2017-07-21", "2017-07-21-07.33.28.429340");
map.put("2017-07-24", "2017-07-24-01.23.33.591340");
//note duplicates are overwritten, but no value change here
map.put("2017-07-24", "2017-07-24-01.23.33.492340");
map.put("2017-07-21", "2017-07-21-07.33.28.429540");
// You can also use Java 8 streams for the concatenation
// but I left it simple
List<String> matches = map.entrySet()
.filter(e -> e.getValue().startsWith(e.getKey())
.collect(Collectors.toList());
String concatenated = String.join("\n", matches);
If you wanted to generate that string without streams, it would look like this (again, not using #entrySet for simplicity, but it would be more efficient here):
List<String> matches = new ArrayList<>();
StringJoiner joiner = new StringJoiner("\n");
for (String key : map.keySet()) {
String value = map.get(key);
if (value.startsWith(key)) {
joiner.add(value);
}
}
//joiner#toString will give the expected result
I'm trying to use Spark (Java API) to take an in-memory Map (that potentially contains other nested Maps as its values) and convert it into a dataframe. I think I need something along these lines:
Map myMap = getSomehow();
RDD myRDD = sparkContext.makeRDD(myMap); // ???
DataFrame df = sparkContext.read(myRDD); // ???
But I'm having a tough time seeing the forest through the trees here...any ideas? Again this might be a Map<String,String> or a Map<String,Map>, where there could be several nested layers of maps-inside-of-maps-inside-of-maps, etc.
So I tried something, not sure if this is the most efficient option to do it, but I do not see any other right now.
SparkConf sf = new SparkConf().setAppName("name").setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(sf);
SQLContext sqlCon = new SQLContext(sc);
Map map = new HashMap<String, Map<String, String>>();
map.put("test1", putMap);
HashMap putMap = new HashMap<String, String>();
putMap.put("1", "test");
List<Tuple2<String, HashMap>> list = new ArrayList<Tuple2<String, HashMap>>();
Set<String> allKeys = map.keySet();
for (String key : allKeys) {
list.add(new Tuple2<String, HashMap>(key, (HashMap) map.get(key)));
};
JavaRDD<Tuple2<String, HashMap>> rdd = sc.parallelize(list);
System.out.println(rdd.first());
List<StructField> fields = new ArrayList<>();
StructField field1 = DataTypes.createStructField("String", DataTypes.StringType, true);
StructField field2 = DataTypes.createStructField("Map",
DataTypes.createMapType(DataTypes.StringType, DataTypes.StringType), true);
fields.add(field1);
fields.add(field2);
StructType struct = DataTypes.createStructType(fields);
JavaRDD<Row> rowRDD = rdd.map(new Function<Tuple2<String, HashMap>, Row>() {
#Override
public Row call(Tuple2<String, HashMap> arg0) throws Exception {
return RowFactory.create(arg0._1, arg0._2);
}
});
DataFrame df = sqlCon.createDataFrame(rowRDD, struct);
df.show();
In this scenario I assumed that the Map in the Dataframe is of Type (String, String). Hope this helps!
Edit: Obviously you can delete all the prints. I did this for visualization purposes!
I am trying to map a meterId to a list of MeterBlinks that have that Id. I'm mainly confused on how to build the list up for the HashMap.put() call. Code below:
Map<String, List<MeterBlink>> IdToMetersMap = new HashMap<>();
for (MeterBlink meterBlink : createData()) {
List<MeterBlink> meterBlinkList = new ArrayList<>();
meterBlinkList.add(meterBlink);
String meterId = meterBlink.getMeterId();
idToMetersMap.put(meterId, meterBlinkList)
}
I think the issue is that I am creating a new list each time I iterate through but I am not sure how to resolve this.
Use the computeIfAbsent method added in jre 8:
Map<String, List<MeterBlink>> idToMetersMap = new HashMap<>();
for (MeterBlink meterBlink : createData()) {
String meterId = meterBlink.getMeterId();
idToMetersMap.computeIfAbsent(meterId, k -> new ArrayList<>()).add(meterBlinks);
}
Another option in java 8:
Map<String, List<MeterBlink>> idToMetersMap = createData().stream()
.collect(Collectors.groupingBy(MeterBlink::getMeterId));
I like the java8 answer, but here without java8 (without lambda expressions):
Map<String, List<MeterBlink>> idToMetersMap = new HashMap<>();
for (MeterBlink meterBlink : createData()) {
String meterId = meterBlink.getMeterId();
List<MeterBlink> meterBlinkList = idToMetersMap.get(meterId);
//if List doesn't exist create it and put in Map
if (meterBlinkList == null) {
meterBlinkList = new ArrayList<>();
idToMetersMap.put(meterId, meterBlinksList)
}
meterBlinkList.add(meterBlink);
}
I am trying to convert my Java code to scala in Spark, but found it very complicated. Is it possible to convert the following Java code to scala? Thanks!
JavaPairRDD<String,Tuple2<String,String>> newDataPair = newRecords.mapToPair(new PairFunction<String, String, Tuple2<String, String>>() {
private static final long serialVersionUID = 1L;
#Override
public Tuple2<String, Tuple2<String, String>> call(String t) throws Exception {
MyPerson p = (new Gson()).fromJson(t, MyPerson.class);
String nameAgeKey = p.getName() + "_" + p.getAge() ;
Tuple2<String, String> value = new Tuple2<String, String>(p.getNationality(), t);
Tuple2<String, Tuple2<String, String>> kvp =
new Tuple2<String, Tuple2<String, String>>(nameAgeKey.toLowerCase(), value);
return kvp;
}
});
I tried the following, but I am sure I have missed many things. And actually it is not clear to me how to do the override function in scala ... Please suggest or share some examples. Thank you!
val newDataPair = newRecords.mapToPair(new PairFunction<String, String, Tuple2<String, String>>() {
#Override
public val call(String t) throws Exception {
val p = (new Gson()).fromJson(t, MyPerson.class);
val nameAgeKey = p.getName() + "_" + p.getAge() ;
val value = new Tuple2<String, String>(p.getNationality(), t);
val kvp =
new Tuple2<String, Tuple2<String, String>>(nameAgeKey.toLowerCase(), value);
return kvp;
}
});
Literal translations from Spark-Java to Spark-Scala typically don't work because Spark-Java introduces many artifacts to cope with the limited type system in Java. Examples in this case: mapToPair in Java is just map in Scala. Tuple2 has a more terse syntax (a,b)
Applying that (and some more) to the snippet:
val newDataPair = newRecords.map{t =>
val p = (new Gson()).fromJson(t, classOf[MyPerson])
val nameAgeKey = p.getName + "_" + p.getAge
val value = (p.getNationality(), t)
(nameAgeKey.toLowerCase(), value)
}
It could be made a bit more concise but I wanted to keep the same structure as the Java counterpart to facilitate the understanding of it.