Develop Reference

Scala : Transforming a List into a nested Map

I have an ConfigEntry class defined as
case class ConfigEntry(
key: String,
value: String
)
and a list:
val list: List[ConfigEntry] = List(
ConfigEntry("general.first", "general first value"),
ConfigEntry("general.second", "general second value"),
ConfigEntry("custom.first", "custom first value"),
ConfigEntry("custom.second", "custom second value")
)
Given a list of ConfigEntry, I want a map from property -> map of entries that satisfy that property
As an example, if I have
def getConfig: Map[String, Map[String, String]] = {
def getKey(key: String, index: Int): String = key.split("\\.")(index)
list.map { config =>
getKey(config.key, 0) -> Map(getKey(config.key, 1) -> config.value)
}.toMap
}
I get result
res0: Map[String,Map[String,String]] =
Map(
"general" ->
Map("second" -> "general second value"),
"custom" ->
Map("second" -> "custom second value")
)
and it should be
res0: Map[String,Map[String,String]] =
Map(
"general" ->
Map(
"first" -> "general first value",
"second" -> "general second value"
),
"custom" ->
Map(
"first" -> "custom first value",
"second" -> "custom second value"
)
)
The first record from the list is missing. It's probably through .toMap
How can I do this?
Thank you for any help given

You can do something like this:
final case class ConfigEntry(
key: String,
value: String
)
type Config = Map[String, Map[String, String]]
def getConfig(data: List[ConfigEntry]): Config =
data
.view
.map(e => e.key.split('.').toList -> e.value)
.collect {
case (k1 :: k2 :: Nil, v) => k1 -> (k2 -> v)
}.groupMap(_._1)(_._2)
.view
.mapValues(_.toMap)
.toMap
Or something like this:
def getConfig(data: List[ConfigEntry]): Config = {
#annotation.tailrec
def loop(remaining: List[ConfigEntry], acc: Config): Config =
remaining match {
case ConfigEntry(key, value) :: xs =>
val newAcc = key.split('.').toList match {
case k1 :: k2 :: Nil =>
acc.updatedWith(k1) {
case Some(map) =>
val newMap = map.updatedWith(k2) {
case Some(v) =>
println(s"Overwriting previous value ${v} for the key: ${key}")
// Just overwrite the previous value.
Some(value)
case None =>
Some(value)
}
Some(newMap)
case None =>
Some(Map(k2 -> value))
}
case _ =>
println(s"Bad key: ${key}")
// Just skip this key.
acc
}
loop(remaining = xs, newAcc)
case Nil =>
acc
}
loop(remaining = data, acc = Map.empty)
}
I leave the handling of errors like duplicated keys or bad keys to the reader.
BTW, since this is a config, have you considered using a Config library?

Your map will only produce a 1 to 1 result. To do what you want you will need an accumulator (existing map) to do this.
Working with your existing code, if you're especially tied to how you're parsing your primary and secondary keys via getKey you can apply foldLeft to your list instead, with an empty map as an initial value.
list.foldLeft(Map.empty[String, Map[String, String]]) { (configs, configEntry) =>
val primaryKey = getKey(configEntry.key, 0)
val secondaryKey = getKey(configEntry.key, 1)
configs.get(primaryKey) match {
case None =>
configs.updated(primaryKey, Map(secondaryKey -> configEntry.value))
case Some(configMap) =>
configs.updated(primaryKey, configMap.updated(secondaryKey, configEntry.value))
}
}

Simply:
list.map { ce =>
val Array(l, r) = ce.key.split("\\.")
l -> (r -> ce.value)
} // List[(String, (String, String))]
.groupBy { case (k, _) => k } // Map[String, List[(String, (String, String))]]
.view.mapValues(_.map { case (_, v) => v }.toMap) // MapView[String, List[(String, String)]]
.toMap // Map[String, Map[String, String]]

Spark Accumulator

I am new to accumulators in Spark . I have created an accumulator which gathers the information of sum and count of all columns in a dataframe into a Map.
Which is not functioning as expected , so I have a few doubts.
When I run this class( pasted below) in local mode , I can see that the accumulators getting updated but the final value is still empty.For debug purposes, I added a print Statement in add() .
Q1) Why is the final accumulable not being updated when the accumulator is being added ?
For reference , I studied the CollectionsAccumulator where they have made use of SynchronizedList from Java Collections.
Q2) Does it need to be a synchronized/concurrent collection for an accumulator to update ?
Q3) Which collection will be best suited for such purpose ?
I have attached my execution flow along with spark ui snapshot for analysis.
Thanks.
EXECUTION:
INPUT DATAFRAME -
+-------+-------+
|Column1|Column2|
+-------+-------+
|1 |2 |
|3 |4 |
+-------+-------+
OUTPUT -
Add - Map(Column1 -> Map(sum -> 1, count -> 1), Column2 -> Map(sum -> 2, count -> 1))
Add - Map(Column1 -> Map(sum -> 4, count -> 2), Column2 -> Map(sum -> 6, count -> 2))
TestRowAccumulator(id: 1, name: Some(Test Accumulator for Sum&Count), value: Map())
SPARK UI SNAPSHOT -
CLASS :
class TestRowAccumulator extends AccumulatorV2[Row,Map[String,Map[String,Int]]]{
private var colMetrics: Map[String, Map[String, Int]] = Map[String , Map[String , Int]]()
override def isZero: Boolean = this.colMetrics.isEmpty
override def copy(): AccumulatorV2[Row, Map[String,Map[String,Int]]] = {
val racc = new TestRowAccumulator
racc.colMetrics = colMetrics
racc
}
override def reset(): Unit = {
colMetrics = Map[String,Map[String,Int]]()
}
override def add(v: Row): Unit = {
v.schema.foreach(field => {
val name: String = field.name
val value: Int = v.getAs[Int](name)
if(!colMetrics.contains(name))
{
colMetrics = colMetrics ++ Map(name -> Map("sum" -> value , "count" -> 1 ))
}else
{
val metric = colMetrics(name)
val sum = metric("sum") + value
val count = metric("count") + 1
colMetrics = colMetrics ++ Map(name -> Map("sum" -> sum , "count" -> count))
}
})
}
override def merge(other: AccumulatorV2[Row, Map[String,Map[String,Int]]]): Unit = {
other match {
case t:TestRowAccumulator => {
colMetrics.map(col => {
val map2: Map[String, Int] = t.colMetrics.getOrElse(col._1 , Map())
val map1: Map[String, Int] = col._2
map1 ++ map2.map{ case (k,v) => k -> (v + map1.getOrElse(k,0)) }
} )
}
case _ => throw new UnsupportedOperationException(s"Cannot merge ${this.getClass.getName} with ${other.getClass.getName}")
}
}
override def value: Map[String, Map[String, Int]] = {
colMetrics
}
}

After a bit of debug , I found that merge function is being called .
It had erroneous code so the accumulable value was Map()
EXECUTION FlOW OF ACCUMULATOR (LOCAL MODE) :
ADD
ADD
MERGE
Once I corrected the merge function , accumulator worked as expected

Get LinkedHashMapValue with Dynamic Key

I want to get the value from LinkedHashmap with dynamic key like below.
def map = [Employee1: [Status: 'Working', Id: 1], Employee2: [Status: 'Resigned', Id: 2]]
def keys = "Employee1.Status"
def keyPath = "";
def keyList = keys.tokenize(".");
keyList.eachWithIndex() { key, i ->
keyPath += "$key"
if(i != keyList.size() - 1){ keyPath += "." }
}
println keyPath //Employee1.Status
println map.keyPath //Always null
println map.'Employee1'.'Status' //Working
println map.Employee1.Status //Working
Here map.keyPath always returning null. How to get the value with dynamic key?

I think you can simply do this:
def tmpMap = map;
keyList.subList(0, keyList.size - 1).each {key ->
tmpMap = map[key]
}
println tmpMap[keyList[keyList.size - 1]]
This will extract the sub maps until the actual value key is reached. To make this more stable you should add some logic to check if the value associated with the current key is actually a map.

With curiosity I try to use just Your code.
keyPath == 'Employee1.Status' not 'Employee1'.'Status'
So to do this you can make something like this:
def map = [
Employee1:
[Status: 'Working', Id: 1],
Employee2:
[Status: 'Resigned', Id: 2]
]
def keys = "Employee1.Status"
def keyPath = "";
def keyList = keys.tokenize(".");
keyList.eachWithIndex() { key, i ->
keyPath += "$key"
if(i != keyList.size() - 1){ keyPath += '.' }
}
println keyPath //Employee1.Status
//tokenize it and get elements as a[0] and a[1]
a = keyPath.tokenize(".");
println map.(a[0]).(a[1]) //Working
println map.'Employee1'.'Status' //Working
println map.Employee1.Status //Working

How to write a custom serializer for Java 8 LocalDateTime

I have a class named Child1 which I want to convert into JSON using Lift Json. Everything is working fine i was using joda date time but now i want to use Java 8 LocalDateTime but i am unable to write custom serializer for this here is my code
import org.joda.time.DateTime
import net.liftweb.json.Serialization.{ read, write }
import net.liftweb.json.DefaultFormats
import net.liftweb.json.Serializer
import net.liftweb.json.JsonAST._
import net.liftweb.json.Formats
import net.liftweb.json.TypeInfo
import net.liftweb.json.MappingException
class Child1Serializer extends Serializer[Child1] {
private val IntervalClass = classOf[Child1]
def deserialize(implicit format: Formats): PartialFunction[(TypeInfo, JValue), Child1] = {
case (TypeInfo(IntervalClass, _), json) => json match {
case JObject(
JField("str", JString(str)) :: JField("Num", JInt(num)) ::
JField("MyList", JArray(mylist)) :: (JField("myDate", JInt(mydate)) ::
JField("number", JInt(number)) ::Nil)
) => {
val c = Child1(
str, num.intValue(), mylist.map(_.values.toString.toInt), new DateTime(mydate.longValue)
)
c.number = number.intValue()
c
}
case x => throw new MappingException("Can't convert " + x + " to Interval")
}
}
def serialize(implicit format: Formats): PartialFunction[Any, JValue] = {
case x: Child1 =>
JObject(
JField("str", JString(x.str)) :: JField("Num", JInt(x.Num)) ::
JField("MyList", JArray(x.MyList.map(JInt(_)))) ::
JField("myDate", JInt(BigInt(x.myDate.getMillis))) ::
JField("number", JInt(x.number)) :: Nil
)
}
}
Object Test extends App {
case class Child1(var str:String, var Num:Int, MyList:List[Int], myDate:DateTime) {
var number: Int=555
}
val c = Child1("Mary", 5, List(1, 2), DateTime.now())
c.number = 1
println("number" + c.number)
implicit val formats = DefaultFormats + new Child1Serializer
val ser = write(c)
println("Child class converted to string" + ser)
var obj = read[Child1](ser)
println("object of Child is "+ obj)
println("str" + obj.str)
println("Num" + obj.Num)
println("MyList" + obj.MyList)
println("myDate" + obj.myDate)
println("number" + obj.number)
}
now i want to use Java 8 LocalDateTime like this
case class Child1(var str: String, var Num: Int, MyList: List[Int], val myDate: LocalDateTime = LocalDateTime.now()) {
var number: Int=555
}
what modification do i need to make in my custom serializer class Child1Serializer i tried to do it but i was unable to do it please help me

In the serializer, serialize the date like this:
def serialize(implicit format: Formats): PartialFunction[Any, JValue] = {
case x: Child1 =>
JObject(
JField("str", JString(x.str)) :: JField("Num", JInt(x.Num)) ::
JField("MyList", JArray(x.MyList.map(JInt(_)))) ::
JField("myDate", JString(x.myDate.toString)) ::
JField("number", JInt(x.number)) :: Nil
)
}
In the deserializer,
def deserialize(implicit format: Formats): PartialFunction[(TypeInfo, JValue), Child1] = {
case (TypeInfo(IntervalClass, _), json) => json match {
case JObject(
JField("str", JString(str)) :: JField("Num", JInt(num)) ::
JField("MyList", JArray(mylist)) :: (JField("myDate", JString(mydate)) ::
JField("number", JInt(number)) ::Nil)
) => {
val c = Child1(
str, num.intValue(), mylist.map(_.values.toString.toInt), LocalDateTime.parse(myDate)
)
c.number = number.intValue()
c
}
case x => throw new MappingException("Can't convert " + x + " to Interval")
}
}
The LocalDateTime object writes to an ISO format using toString and the parse factory method should be able to reconstruct the object from such a string.

You can define the LocalDateTime serializer like this.
class LocalDateTimeSerializer extends Serializer[LocalDateTime] {
private val LocalDateTimeClass = classOf[LocalDateTime]
def deserialize(implicit format: Formats): PartialFunction[(TypeInfo, JValue), LocalDateTime] = {
case (TypeInfo(LocalDateTimeClass, _), json) => json match {
case JString(dt) => LocalDateTime.parse(dt)
case x => throw new MappingException("Can't convert " + x + " to LocalDateTime")
}
}
def serialize(implicit format: Formats): PartialFunction[Any, JValue] = {
case x: LocalDateTime => JString(x.toString)
}
}
Also define your formats like this
implicit val formats = DefaultFormats + new LocalDateTimeSerializer + new FieldSerializer[Child1]
Please note the usage of FieldSerializer to serialize the non-constructor filed, number

Is there a cleaner way to do this Group Query in MongoDB from Groovy?

I'm working on learning MongoDB. Language of choice for the current run at it is Groovy.
Working on Group Queries by trying to answer the question of which pet is the most needy one.
Below is my first attempt and it's awful. Any help cleaning this up (or simply confirming that there isn't a cleaner way to do it) would be much appreciated.
Thanks in advance!
package mongo.pets
import com.gmongo.GMongo
import com.mongodb.BasicDBObject
import com.mongodb.DBObject
class StatsController {
def dbPets = new GMongo().getDB('needsHotel').getCollection('pets')
//FIXME OMG THIS IS AWFUL!!!
def index = {
def petsNeed = 'a walk'
def reduce = 'function(doc, aggregator) { aggregator.needsCount += doc.needs.length }'
def key = new BasicDBObject()
key.put("name", true)
def initial = new BasicDBObject()
initial.put ("needsCount", 0)
def maxNeeds = 0
def needyPets = []
dbPets.group(key, new BasicDBObject(), initial, reduce).each {
if (maxNeeds < it['needsCount']) {
maxNeeds = it['needsCount']
needyPets = []
needyPets += it['name']
} else if (maxNeeds == it['needsCount']) {
needyPets += it['name']
}
}
def needyPet = needyPets
[petsNeedingCount: dbPets.find([needs: petsNeed]).count(), petsNeed: petsNeed, mostNeedyPet: needyPet]
}
}

It should be possible to be change the whole method to this (but I don't have MongoDB to test it)
def index = {
def petsNeed = 'a walk'
def reduce = 'function(doc, aggregator) { aggregator.needsCount += doc.needs.length }'
def key = [ name: true ] as BasicDBObject
def initial = [ needsCount: 0 ] as BasicDBObject
def allPets = dbPets.group( key, new BasicDBObject(), initial, reduce )
def maxNeeds = allPets*.needsCount.collect { it as Integer }.max()
def needyPet = allPets.findAll { maxNeeds == it.needsCount as Integer }.name
[petsNeedingCount: dbPets.find([needs: petsNeed]).count(), petsNeed: petsNeed, mostNeedyPet: needyPet]
}