Develop Reference

Scala interop with Java SAM without 2.12 M2 flag

Is there any accepted technique of writing Scala code against a Java-8 API which uses Java #FunctionalInterface / SAM / lambda expressions?
While Java lambda expressions inter-op is available under a flag in M2 http://www.scala-lang.org/news/2.12.0-M2, I was rather hoping that a type class / AnyVal solution might work together with scala.FunctionX traits.
Unfortunately though, scala.FunctionX extends AnyRef and not Any so one cannot use/mix these traits into an implicit AnyVal class implementation.
Added: I'm not entirely sure that I though out how I would achieve my aim even if scala.FunctionX were global traits (extending from Any). My use case is this though:
In a project of mine, I've chosen to provide a Java-8 API with FunctionalInterfaces like the Java Stream interfaces & classes so as to cater for the widest possible audience of JVM-based client languages e.g. Closure, Scala, Kotlin. For each client language using my Java-8 API, I will write appropriate bindings (if necessary) to use language-specific idioms if in event of accessing Java-8 API feels clunky in that language.
btw. I would be interested in any comments with this question taken in a Kotlin-Java interop context also.

This Scala program demonstrates one side of the coin for my question, that is, how to get Scala functions to masquerade as Java 8 Lambdas.
Syntactically and idiomatically this seems to work fine by creating some implicit Scala functions to convert Scala functions to their Java 8 FunctionalInterface counterpart types.
The caveat is, of course, that this method does not take advantage of Java 8's ability to optimize lambda creation via invokedynamic.
Accordingly this approach results in a JVM object being created for the Scala function instance and this may impact upon memory usage and performance compared with Java 8 native lambdas.
For the flip side of the coin, that is, how to get Java 8 Lambdas to masquerade as Scala functions, I guess one would have to write some Java code to interop with Scala (if one's aim was to have a Scala API that was callable from Java).
Justin Johansson,
Microblogging about my Project Clockwork,
A new implementation of XPath/XQuery on the JVM,
as #MartianOdyssey on Twitter
https://twitter.com/MartianOdyssey
/**
* Scala Functions masquerading as Java 8 Lambdas.
*
* (C) Justin Johansson 2015.
*
* Microblogging about my Project Clockwork, a
* new implementation of XPath/XQuery on the JVM,
* as #MartianOdyssey on Twitter (https://twitter.com/MartianOdyssey).
*
* Permission to use this code is granted under Apache License,
* Version 2.0 and providing attribution is afforded to author,
* Justin Johansson.
*/
package lab
import scala.language.implicitConversions
import java.util.{ Arrays => JArrays, List => JList }
import java.util.function.{ Consumer => JConsumer, Function => JFunction, Predicate => JPredicate }
import java.util.stream.{ Stream => JStream }
object JLambda extends App {
println("JLambda: Scala to Java 8 lambda test")
implicit def func1ToJConsumer[T](func: T => Unit): JConsumer[T] = {
new JConsumer[T] {
def accept(arg: T): Unit = func(arg)
}
}
implicit def func1ToJFunction[T, R](func: T => R): JFunction[T, R] = {
new JFunction[T, R] {
def apply(arg: T): R = func(arg)
}
}
implicit def func1ToJPredicate[T](func: T => Boolean): JPredicate[T] = {
new JPredicate[T] {
def test(arg: T): Boolean = func(arg)
}
}
val myList = JArrays.asList("cake", "banana", "apple", "coffee")
println(s"myList = $myList")
val myListFiltered: JStream[String] = myList.stream
.filter { x: String => x.startsWith("c") }
val myListFilteredAndMapped: JStream[String] = myListFiltered
.map { x: String => x.toUpperCase }
myListFilteredAndMapped.forEach { x: String => println(s"x=$x") }
}
/*
Outputs:
JLambda: Scala to Java 8 lambda test
myList = [cake, banana, apple, coffee]
x=CAKE
x=COFFEE
*/

btw. I would be interested in any comments with this question taken in a Kotlin-Java interop context also.
Kotlin's FunctionX interfaces are SAM's, so there's no need to do anything extra to make Java 8 understand them

Get Java reflection representation of Scala type

This seems like a simple question, but it's very challenging to search for, so I'm asking a new question. My apologies if it's already been asked.
Due to the compiler bug described here Scala 2.11.5 compiler crash with type aliases and manifests (also here https://issues.scala-lang.org/browse/SI-9155), I need to use scala TypeTags and friends for discovery of type parameters to methods. However, I then need to use that type information in a Java library that uses java.lang.Class and java.lang.reflect.Type.
How can I convert a scala.reflect.runtime.universe Type into a java.lang.reflect.Type or java.lang.Class?
Put concretely, how would I fill out the body of this method:
def typeFor[T](implicit tag: TypeTag[T]): java.lang.reflect.Type = ...
or, if that's not possible:
def typeFor[T](implicit tag: TypeTag[T]): java.lang.Class[T] = ...
And note, due to the bug posted above, I cannot use scala.reflect.Manifest.

The short answer is no, but you can try to do something similar to this SO question. However there is an open ticket....

This may have some limitations I'm not aware of, but you could drop down to Java reflection and try something like:
import scala.util.control.Exception._
def typeMe[T](implicit t: TypeTag[T]) = {
catching(classOf[Exception]) opt Class.forName(t.tpe.typeSymbol.asClass.fullName)
}
println(typeMe[String])
println(typeMe[ClassTag[_]])
Results in:
Some(class java.lang.String)
Some(interface scala.reflect.ClassTag)

The way I solved it with manifests, was:
private def typeFromManifest(m: Manifest[_]): Type = {
if (m.typeArguments.isEmpty) { m.runtimeClass }
else new ParameterizedType {
def getRawType = m.runtimeClass
def getActualTypeArguments = m.typeArguments.map(typeFromManifest).toArray
def getOwnerType = null
}
}
Right now I'm trying to solve this using something other than Manifest which should be removed from scala runtime.

How to provide a codec to the SaveAsSequenceFile method in Spark?

I am trying to figure out how to pass a codec to the saveAsSequenceFile method in Apache Spark. Below is the code I am trying to run. I am running Scala 2.10.4, Spark 1.0.0, Java 1.7.60, and Apache Hadoop 2.4.0.
val rdd:RDD[(String, String)] = sc.sequenceFile(secPath,
classOf[Text],
classOf[Text]
).map { case (k,v) => (k.toString, v.toString)}
val sortedOutput = rdd.sortByKey(true, 1)
sortedOutput.saveAsSequenceFile(secPathOut)
My issue is that I am new to Spark and Scala. I do not understand what the javadoc means for the codec variable passed to the saveAsSequenceFile method.
def saveAsSequenceFile(path: String, codec: Option[Class[_ <: CompressionCodec]] = None): Unit
What does the <: mean? I get that the codec is optional, because when I run the above code it works. Could someone please show an example of a properly formatted codec call to this method?
Thanks!

The <: indicates that the class you pass in should extend org.apache.hadoop.io.compress.CompressionCodec (read this), spark uses a lot of HDFS features and is pretty heavily integrated with it at this point. This means you can pass the class of any of the following as the codec, BZip2Codec, DefaultCodec, GzipCodec, there are likely also other extensions of CompressionCodec not built into hadoop. Here is an example of calling the method
sc.parallelize(List((1,2))).saveAsSequenceFile("path",Some(classOf[GzipCodec]))
The Option[...] is used in scala in favor of java's null even though null exists in scala. Option can be Some(...) or None

How do I write a JSON Format for an object in the Java library that doesn't have an apply method?

I've been stuck on this particular problem for about a week now, and I figure I'm going to write this up as a question on here to clear out my thoughts and get some guidance.
So I have this case class that has a java.sql.Timestamp field:
case class Request(id: Option[Int], requestDate: Timestamp)
and I want to convert this to a JsObject
val q = Query(Requests).list // This is Slick, a database access lib for Scala
printList(q)
Ok(Json.toJson(q)) // and this is where I run into trouble
"No Json deserializer found for type List[models.Request]. Try to implement an implicit Writes or Format for this type." Okay, that makes sense.
So following the Play documentation here, I attempt to write a Format...
implicit val requestFormat = Json.format[Request] // need Timestamp deserializer
implicit val timestampFormat = (
(__ \ "time").format[Long] // error 1
)(Timestamp.apply, unlift(Timestamp.unapply)) // error 2
Error 1
Description Resource Path Location Type overloaded method value format with alternatives:
(w: play.api.libs.json.Writes[Long])(implicit r: play.api.libs.json.Reads[Long])play.api.libs.json.OFormat[Long]
<and>
(r: play.api.libs.json.Reads[Long])(implicit w: play.api.libs.json.Writes[Long])play.api.libs.json.OFormat[Long]
<and>
(implicit f: play.api.libs.json.Format[Long])play.api.libs.json.OFormat[Long]
cannot be applied to (<error>, <error>)
Apparently importing like so (see the documentation "ctrl+F import") is getting me into trouble:
import play.api.libs.json._ // so I change this to import only Format and fine
import play.api.libs.functional.syntax._
import play.api.libs.json.Json
import play.api.libs.json.Json._
Now that the overloading error went away, I reach more trubbles: not found: value __ I imported .../functional.syntax._ already just like it says in the documentation! This guy ran into the same issue but the import fixed it for him! So why?! I thought this might just be Eclipse's problem and tried to play run anyway ... nothing changed. Fine. The compiler is always right.
Imported play.api.lib.json.JsPath, changed __ to JsPath, and wallah:
Error 2
value apply is not a member of object java.sql.Timestamp
value unapply is not a member of object java.sql.Timestamp
I also try changing tacks and writing a Write for this instead of Format, without the fancy new combinator (__) feature by following the original blog post the official docs are based on/copy-pasted from:
// I change the imports above to use Writes instead of Format
implicit val timestampFormat = new Writes[Timestamp]( // ERROR 3
def writes(t: Timestamp): JsValue = { // ERROR 4 def is underlined
Json.obj(
/* Returns the number of milliseconds since
January 1, 1970, 00:00:00 GMT represented by this Timestamp object. */
"time" -> t.getTime()
)
}
)
ERROR 3: trait Writes is abstract, cannot be instantiated
ERROR 4: illegal start of simple expression
At this point I'm about at my wits' end here, so I'm just going back to the rest of my mental stack and report from my first piece of code
My utter gratefulness to anybody who can put me out of my coding misery

It's not necessarily apply or unapply functions you need. It's a) a function that constructs whatever the type you need given some parameters, and b) a function that turns an instance of that type into a tuple of values (usually matching the input parameters.)
The apply and unapply functions you get for free with a Scala case class just happen to do this, so it's convenient to use them. But you can always write your own.
Normally you could do this with anonymous functions like so:
import java.sql.Timestamp
import play.api.libs.functional.syntax._
import play.api.libs.json._
implicit val timestampFormat: Format[Timestamp] = (
(__ \ "time").format[Long]
)((long: Long) => new Timestamp(long), (ts: Timestamp) => (ts.getTime))
However! In this case you fall foul of a limitation with the API that prevents you from writing formats like this, with only one value. This limitation is explained here, as per this answer.
For you, a way that works would be this more complex-looking hack:
import java.sql.Timestamp
import play.api.libs.functional.syntax._
import play.api.libs.json._
implicit val rds: Reads[Timestamp] = (__ \ "time").read[Long].map{ long => new Timestamp(long) }
implicit val wrs: Writes[Timestamp] = (__ \ "time").write[Long].contramap{ (a: Timestamp) => a.getTime }
implicit val fmt: Format[Timestamp] = Format(rds, wrs)
// Test it...
val testTime = Json.obj("time" -> 123456789)
assert(testTime.as[Timestamp] == new Timestamp(123456789))

Trouble using java class within jruby

I'm trying to use Java Opencl from within jruby, but am encountering a problem which I can't solve, even with much google searching.
require 'java'
require 'JOCL-0.1.7.jar'
platforms = org.jocl.cl_platform_id.new
puts platforms.class
org.jocl.CL.clGetPlatformIDs(1, platforms, nil)
when I run this code using: jruby test.rb
I get the following error, when the last line is uncommented:
#<Class:0x10191777e>
TypeError: cannot convert instance of class org.jruby.java.proxies.ConcreteJavaP
roxy to class [Lorg.jocl.cl_platform_id;
LukeTest at test.rb:29
(root) at test.rb:4
Just wondering whether anyone has an idea on how to solve this problem?
EDIT:
ok so I think I've solved the first part of this problem by making platforms an array:
platforms = org.jocl.cl_platform_id[1].new
but that led to this error when adding the next couple of lines:
context_properties = org.jocl.cl_context_properties.new()
context_properties.addProperty(org.jocl.CL::CL_CONTEXT_PLATFORM, platforms[0])
CodegenUtils.java:98:in `human': java.lang.NullPointerException
from CodegenUtils.java:152:in `prettyParams'
from CallableSelector.java:462:in `argumentError'
from CallableSelector.java:436:in `argTypesDoNotMatch'
from RubyToJavaInvoker.java:248:in `findCallableArityTwo'
from InstanceMethodInvoker.java:66:in `call'
from CachingCallSite.java:332:in `cacheAndCall'
from CachingCallSite.java:203:in `call'
from test.rb:36:in `module__0$RUBY$LukeTest'
from test.rb:-1:in `module__0$RUBY$LukeTest'
from test.rb:4:in `__file__'
from test.rb:-1:in `load'
from Ruby.java:679:in `runScript'
from Ruby.java:672:in `runScript'
from Ruby.java:579:in `runNormally'
from Ruby.java:428:in `runFromMain'
from Main.java:278:in `doRunFromMain'
from Main.java:198:in `internalRun'
from Main.java:164:in `run'
from Main.java:148:in `run'
from Main.java:128:in `main'
for some reason when I print the class of platforms[0] it's listed as NilClass!?

You are overlooking a very simple mistake. You write
platforms = org.jocl.cl_platform_id.new
but that line creates a single instance of the class org.jocl.cl_platform_id. You then pass that as the second parameter to org.jocl.CL.clGetPlatformIDs in
org.jocl.CL.clGetPlatformIDs(1, platforms, nil)
and that doesn't work, because the second argument of the method requires an (empty) array of org.jocl.cl_platform_id objects.
What the error says is: "I have something that is a proxy for a Java object and I can't turn it into an an array of org.jocl.cl_platform_id objects, as you are asking me to do.
If you just say
platforms = []
and pass that in, it might just work :).