I have a written a method in Scala that is using a method written in Java - processSale() method takes util.List<Sale> as a parameter.
But after groupByKey() I'm getting an RDD[(String, Iterable[Sale])]. I've tried to import scala.collection.JavaConverters._ and do SaleParser.processSale(a.asJava).
However it gives me an Iterable[Sale]. How is it possible to convert it into a Java util.List?
val parseSales: RDD[(String, Sale)] = rawSales
.map(sale => sale.Id -> sale)
.groupByKey()
.mapValues(a => SaleParser.processSale(???))
a.toSeq.asJava
Note that if this Iterable is actually a Seq, toSeq just returns the same object.
See API doc for the complete list of conversions.
Related
I am new to apache spark and trying to run the wordcount example . But intellij editor gives me the error at line 47 Cannot resolve method 'flatMap()' error.
Edit :
This is the line where I am getting the error
JavaRDD<String> words = lines.flatMap(s -> Arrays.asList(SPACE.split(s)).iterator());
It looks like you're using an older version of Spark that expects Iterable rather than Iterator from the flatMap() function. Try this:
JavaRDD<String> words = lines.flatMap(s -> Arrays.asList(SPACE.split(s)));
See also Spark 2.0.0 Arrays.asList not working - incompatible types
Stream#flatMap is used for combining multiple streams into one, so the supplier method you provided must return a Stream result.
you can try like this:
lines.stream().flatMap(line -> Stream.of(SPACE.split(line)))
.map(word -> // map to JavaRDD)
flatMap method take a FlatMapFunctionas parameter which is not annotated with #FunctionalInterface. So indeed you can not use it as a lambda.
Just build a real FlatMapFunctionobject as parameter and you will be sure of it.
flatMap() is Java 8 Stream API. I think you should check the IDEA compile java version.
compile java version
Is there any accepted technique of writing Scala code against a Java-8 API which uses Java #FunctionalInterface / SAM / lambda expressions?
While Java lambda expressions inter-op is available under a flag in M2 http://www.scala-lang.org/news/2.12.0-M2, I was rather hoping that a type class / AnyVal solution might work together with scala.FunctionX traits.
Unfortunately though, scala.FunctionX extends AnyRef and not Any so one cannot use/mix these traits into an implicit AnyVal class implementation.
Added: I'm not entirely sure that I though out how I would achieve my aim even if scala.FunctionX were global traits (extending from Any). My use case is this though:
In a project of mine, I've chosen to provide a Java-8 API with FunctionalInterfaces like the Java Stream interfaces & classes so as to cater for the widest possible audience of JVM-based client languages e.g. Closure, Scala, Kotlin. For each client language using my Java-8 API, I will write appropriate bindings (if necessary) to use language-specific idioms if in event of accessing Java-8 API feels clunky in that language.
btw. I would be interested in any comments with this question taken in a Kotlin-Java interop context also.
This Scala program demonstrates one side of the coin for my question, that is, how to get Scala functions to masquerade as Java 8 Lambdas.
Syntactically and idiomatically this seems to work fine by creating some implicit Scala functions to convert Scala functions to their Java 8 FunctionalInterface counterpart types.
The caveat is, of course, that this method does not take advantage of Java 8's ability to optimize lambda creation via invokedynamic.
Accordingly this approach results in a JVM object being created for the Scala function instance and this may impact upon memory usage and performance compared with Java 8 native lambdas.
For the flip side of the coin, that is, how to get Java 8 Lambdas to masquerade as Scala functions, I guess one would have to write some Java code to interop with Scala (if one's aim was to have a Scala API that was callable from Java).
Justin Johansson,
Microblogging about my Project Clockwork,
A new implementation of XPath/XQuery on the JVM,
as #MartianOdyssey on Twitter
https://twitter.com/MartianOdyssey
/**
* Scala Functions masquerading as Java 8 Lambdas.
*
* (C) Justin Johansson 2015.
*
* Microblogging about my Project Clockwork, a
* new implementation of XPath/XQuery on the JVM,
* as #MartianOdyssey on Twitter (https://twitter.com/MartianOdyssey).
*
* Permission to use this code is granted under Apache License,
* Version 2.0 and providing attribution is afforded to author,
* Justin Johansson.
*/
package lab
import scala.language.implicitConversions
import java.util.{ Arrays => JArrays, List => JList }
import java.util.function.{ Consumer => JConsumer, Function => JFunction, Predicate => JPredicate }
import java.util.stream.{ Stream => JStream }
object JLambda extends App {
println("JLambda: Scala to Java 8 lambda test")
implicit def func1ToJConsumer[T](func: T => Unit): JConsumer[T] = {
new JConsumer[T] {
def accept(arg: T): Unit = func(arg)
}
}
implicit def func1ToJFunction[T, R](func: T => R): JFunction[T, R] = {
new JFunction[T, R] {
def apply(arg: T): R = func(arg)
}
}
implicit def func1ToJPredicate[T](func: T => Boolean): JPredicate[T] = {
new JPredicate[T] {
def test(arg: T): Boolean = func(arg)
}
}
val myList = JArrays.asList("cake", "banana", "apple", "coffee")
println(s"myList = $myList")
val myListFiltered: JStream[String] = myList.stream
.filter { x: String => x.startsWith("c") }
val myListFilteredAndMapped: JStream[String] = myListFiltered
.map { x: String => x.toUpperCase }
myListFilteredAndMapped.forEach { x: String => println(s"x=$x") }
}
/*
Outputs:
JLambda: Scala to Java 8 lambda test
myList = [cake, banana, apple, coffee]
x=CAKE
x=COFFEE
*/
btw. I would be interested in any comments with this question taken in a Kotlin-Java interop context also.
Kotlin's FunctionX interfaces are SAM's, so there's no need to do anything extra to make Java 8 understand them
I'm using Mule ESB (Java Based) and I have some scala components that modify and create data. My Data is represented in Case Classes. I'm trying to convert them to Java, however Just getting them to convert to Scala types is a challenge. Here's a simplified example of what I'm trying to do:
package com.echostar.ese.experiment
import scala.collection.JavaConverters
case class Resource(guid: String, filename: String)
case class Blackboard(name: String, guid:String, resource: Resource)
object CCC extends App {
val res = Resource("4alskckd", "test.file")
val bb = Blackboard("Test", "123asdfs", res)
val myMap = getCCParams(bb)
val result = new java.util.HashMap[String,Object](myMap)
println("Result:"+result)
def getCCParams(cc: AnyRef) =
(Map[String, Any]() /: cc.getClass.getDeclaredFields) {(a, f) =>
f.setAccessible(true)
val value = f.get(cc) match {
// this covers tuples as well as case classes, so there may be a more specific way
case caseClassInstance: Product => getCCParams(caseClassInstance): Map[String, Any]
case x => x
}
a + (f.getName -> value)
}
}
Current Error: Recursive method needs return type.
My Scala Foo isn't very strong. I grabbed this method from another answer here
and basically know what it's doing, but not enough to change this to java.util.HashMap and java.util.List
Expected Output:
Result:{"name"="Test", "guid"="123asdfs", "resource"= {"guid"="4alskckd", "filename"="test.file"}}
UPDATE1:
1. Added getCCParams(caseClassInstance): Map[String, Any] to line 22 Above per #cem-catikkas. IDE syntax error still says "recursive method ... needs result type" and "overloaded method java.util.HashMap cannot be applied to scala.collection.immutable.Map".
2. Changed java.util.HashMap[String, Object]
You should follow what the error tells you. Since getCCParams is a recursive method you need to declare its return type.
def getCCParams(cc: AnyRef): Map[String, Any]
Answering this in case anyone else going through the issue ends up here (as happened to me).
I believe the error you were getting had to do with the fact that the return type was being declared at method invocation (line 22), however the compiler was expecting it at the method's declaration (in your case, line 17). The below seems to have worked:
def getCCParams(cc: AnyRef): Map[String, Any] = ...
Regarding the conversion from Scala Map to Java HashMap, by adding the ._ wildcard to the JavaConverters import statement, you manage to import all the methods of the object as single identifiers, which is a requirement for implicit conversions. This will include the asJava method which can then be used to convert the Scala Map to a Java one, and then this can be passed to the java.util.HashMap(Map<? extends K,? extends V> m) constructor to instantiate a HashMap:
import scala.collection.JavaConverters._
import java.util.{HashMap => JHashMap}
...
val myMap = getCCParams(bb)
val r = myMap.asJava // converting to java.util.Map[String, Any]
val result: JHashMap[String,Any] = new JHashMap(r)
I wonder if you've considered going at it the other way around, by implementing the java.util.Map interface in your case class? Then you wouldn't have to convert back and forth, but any consumers downstream that are using a Map interface will just work (for example if you're using Groovy's field dot-notation).
I'm new to Scala and our project mixes Java and Scala code together (using the Play Framework). I'm trying to write a Scala method that can take a nested Java Map such as:
LinkedHashMap<String, LinkedHashMap<String, String>> groupingA = new LinkedHashMap<String, LinkedHashMap<String,String>>();
And have that java object passed to a Scala function that can loop through it. I have the following scala object definition to try and support the above Java nested map:
Seq[(String, Seq[(String,String)])]
Both the Java file and the Scala file compile fine individually, but when my java object tries to create a new instance of my scala class and pass in the nested map, I get a compiler error with the following details:
[error] ..... overloaded method value apply with alternatives:
[error] (options: java.util.List[String])scala.collection.mutable.Buffer[(String, String)] <and>
[error] (options: scala.collection.immutable.List[String])List[(String, String)] <and>
[error] (options: java.util.Map[String,String])Seq[(String, String)] <and>
[error] (options: scala.collection.immutable.Map[String,String])Seq[(String, String)] <and>
[error] (options: (String, String)*)Seq[(String, String)]
[error] cannot be applied to (java.util.LinkedHashMap[java.lang.String,java.util.LinkedHashMap[java.lang.String,java.lang.String]])
Any ideas here on how I can pass in a nested Java LinkedHashMap such as above into a Scala file where I can generically iterate over a nested collection? I'm trying to write this generic enough that it would also work for a nested Scala collection in case we ever switch to writing our play framework controllers in Scala instead of Java.
Seq is a base trait defined in the Scala Collections hierarchy. While java and scala offer byte code compatibility, scala defines a number of its own types including its own collection library. The rub here is if you want to write idiomatic scala you need to convert your java data to scala data. The way I see it you have a few options.
You can use Richard's solution and convert the java types to scala types in your scala code. I think this is ugly because it assumes your input will always be coming from java land.
You can write beautiful, perfect scala handler and provide a companion object that offers the ugly java conversion behavior. This disentangles your scala implementation from the java details.
Or you could write an implicit def like the one below genericizing it to your heart's content.
.
import java.util.LinkedHashMap
import scala.collection.JavaConversions.mapAsScalaMap
object App{
implicit def wrapLhm[K,V,G](i:LinkedHashMap[K,LinkedHashMap[G,V]]):LHMWrapper[K,V,G] = new LHMWrapper[K,V,G](i)
def main(args: Array[String]){
println("Hello World!")
val lhm = new LinkedHashMap[String, LinkedHashMap[String,String]]()
val inner = new LinkedHashMap[String,String]()
inner.put("one", "one")
lhm.put("outer",inner);
val s = lhm.getSeq()
println(s.toString())
}
class LHMWrapper[K,V,G](value: LinkedHashMap[K,LinkedHashMap[G,V]]){
def getSeq():Seq[ (K, Seq[(G,V)])] = mapAsScalaMap(value).mapValues(mapAsScalaMap(_).toSeq).toSeq
}
}
Try this:
import scala.collections.JavaConversions.mapAsScalaMap
val lhm: LinkedHashMap[String, LinkedHashMap[String, String]] = getLHM()
val scalaMap = mapAsScalaMap(lhm).mapValues(mapAsScalaMap(_).toSeq).toSeq
I tested this, and got a result of type Seq[String, Seq[(String, String)]]
(The conversions will wrap the original Java object, rather than actually creating a Scala object with a copy of the values. So the conversions to Seq aren't necessary, you could leave it as a Map, the iteration order will be the same).
Let me guess, are you processing query parameters?
How do I convert java.util.Map[String, Object] to scala.collection.immutable.Map[String, Any], so that all values in the original map (integers, booleans etc.) are converted to the right value to work well in Scala.
As VonC says, scala.collections.JavaConversion supports mutable collections only, but you don't have to use a separate library. Mutable collections are derived from TraversableOnce which defines a toMap method that returns an immutable Map:
import scala.collection.JavaConversions._
val m = new java.util.HashMap[String, Object]()
m.put("Foo", java.lang.Boolean.TRUE)
m.put("Bar", java.lang.Integer.valueOf(1))
val m2: Map[String, Any] = m.toMap
println(m2)
This will output
Map(Foo -> true, Bar -> 1)
The JavaConversions package of Scala2.8 deals only with mutable collections.
The scalaj-collection library might help here.
java.util.Map[A, B] #asScala: scala.collection.Map[A, B]
#asScalaMutable: scala.collection.mutable.Map[A, B]
#foreach(((A, B)) => Unit): Unit
In order to convert convert java.util.Map[String, Object] to scala.collection.immutable.Map[String,Object] , you need to simple import below statement in Scala Project and clean build.
import collection.JavaConversions._
Refer to below code:
var empMap= Map[String.Object]()
var emp= new Employee(empMap) // Employee is java POJO in which,passing scala map to overloaded constructor for setting default values.