spark.sqlContext.udf.register('dataSource', (data: scala.collection.mutable.WrappedArray$ofRef, ofType : String) => {
var flag = ''
try{
val array: Seq[String] = data.flatten(x => Seq(x.toString()))
array.foreach(s => if(s.contains(ofType)) { flag = ofType })
}
catch {
println("Exception :" + x.getMessage)
flag = ''
}
flag;
})
It is not required to convert this Scala code into Java code. In Spark you can register a UDF in any of the languages and also make use of it in either Java or Scala as long as it is used in the same SparkSession or Context.
Related
I Have a file json named production_2.json
[
{
"v":{
"id":"rep_01564526",
"name":"tuttoverde.pgmx",
"type":"PRODUCTION_STARTED",
"ute":"CDL",
"ver":"1.0",
"status":"EXE"
},
"ts":"2020-11-19T08:00:00Z"
},
{
"v":{
"id":"rep_01564526",
"name":"tuttoverde.pgmx",
"type":"PRODUCTION_ENDED",
"ute":"CDL",
"ver":"1.0",
"status":"EXE"
},
"ts":"2020-11-19T17:00:00Z"
}
]
And have the folling Karate code, that:
Read the file production_2.json
and for each element of the array send a topic
I * def sendtopics =
"""
function(i){
var topic = "data." + machineID + ".Production";
var payload = productions[i];
karate.log('topic: ', topic )
karate.log('payload: ', payload )
return mqtt.sendMessage(payload, topic);
}
"""
* def productions = read('this:productions_json/production_2.json')
* def totalProductionEvents = productions.length
* def isTopicWasSent = karate.repeat(totalProductionEvents, sendtopics)
* match each isTopicWasSent == true
The function
mqtt.sendMessage(payload, topic);
is a function in java, that have the following segnature
public Boolean sendMessage(String payload, String topic) {
System.out.println("Publishing message: ");
System.out.println("payload " + payload);
System.out.println("topic " + topic);
return true;
}
the problem is that the value of the "payload" inside the javascript function is correct and is printed correctly, while inside the "sendMessage" function the value of the payload is formatted incorrectly.
For example here is what it prints inside karate.log('payload: ', payload )
payload: {
"v": {
"id": "rep_01564526",
"name": "tuttoverde.pgmx",
"type": "PRODUCTION_STARTED",
"ute": "CDL",
"ver": "1.0",
"status": "EXE"
},
"ts": "2021-01-08T08:00:00Z"
}
And Here instead what is printed on the function "sendMessage" of the java class
payload {v={id=rep_01564526, name=tuttoverde.pgmx, type=PRODUCTION_STARTED, ute=CDL, ver=1.0, status=EXE, ts=2021-01-08T08:00:00Z}
I don't understand why the payload is formatted incorrectly (= instead of : ) and is it not a string. I also tried using the following solution and it doesn't work for me
* def sendtopics =
"""
function(i){
var topic = "data." + machineID + ".Production";
var payload = productions[i];
var payload2 = JSON.stringify(payload);
return mqtt.sendMessage(payload2, topic);
}
"""
How do I convert an object inside javascript to a string so I can pass it to java?
You are doing some really advanced stuff in Karate. I strongly suggest you start looking at the new version (close to release) and you can find details here: https://github.com/intuit/karate/wiki/1.0-upgrade-guide
The reason is because the async and Java interop has some breaking changes - and there are some new API-s you can call on the karate object in JS to do format conversions:
var temp = karate.fromString(payload);
And karate.log() should work better and not give you that odd formatting you are complaining about. With the current version you can try karate.toJson() if that gives you the conversion you expect.
Given your advanced usage, I recommend you start using the new version and provide feedback on anything that may be still needed.
The example is C# code. I am using RxJava to use lambda in Android.
var dex = device.OrderBy(x => x.Connection1).ThenBy(x => x.EventName1).GroupBy(x => new { x.Connection1, x.EventName1 }, y => new { EventNames = y.EventName1 }).
Select(x => new { Connector = x.Key.Connection1, EventName = x.Key.EventName1, Counter = x.Select(y => y.EventNames).Count() });
So i need this code written in Java with the help of RxJava does anyone know how to use a multible group by in RxJava ???
I ran into the following Java compiler warning issue (using IntelliJ 14):
I have the following class in my Scala library:
object ScalatoJava {
def asJava[A,B](map: Map[A,B]): java.util.Map[A,B] = { map.asJava }
def asJava[A](list: List[A]): java.util.Collection[A] = {list.asJavaCollection}
}
I use it to convert few collections coming from Scala in my Java program.
Now in my Java program I have the following code for which IntelliJ is underlying in red stating that it can't resolve things:
private final java.util.Map<String, FieldSettings> poolpartyfieldsettings =
ScalatoJava.asJava(poolpartyConnectorSettings.fieldsSettingsMap());
It says it can not resolve the method asJava(scala.collection.immutable.Map<String, org.....FieldSettings>).
While if I write either
private final java.util.Map <String, FieldSettings> poolpartyfieldsettings =
ScalatoJava.asJava((scala.collection.immutable.Map<String, FieldSettings>)poolpartyConnectorSettings.fieldsSettingsMap());
or
scala.collection.immutable.Map<String, FieldSettings> poolpartyfieldsettings =
poolpartyConnectorSettings.fieldsSettingsMap();
ScalatoJava.asJava(poolpartyfieldsettings)
the compiler does not complain.
Here is the PoolpartyConnectorSettings Scala class:
trait DspacePoolPartyConnectorSettings {
def poolpartyServerSettings : PoolpartySettings
def fieldsSettingsList : List[FieldSettings]
def fieldsSettingsMap : Map[String, FieldSettings]
}
case class DspaceDspacePoolPartyConnectorSettingImpl (configUri: String) extends DspacePoolPartyConnectorSettings {
private val config = ConfigFactory.parseURL(new URL(configUri)).resolve()
private val configFieldSettinglist = config.getConfigList("PoolPartyConnectorSettings.FieldSettings").asScala.toList
private val configPoolPartySetting = config.getConfig("PoolPartyConnectorSettings.PoolPartySettings")
val poolpartyServerSettings = PoolpartySettings( configPoolPartySetting.getString("apirootEndpoint"),
configPoolPartySetting.getString("thesaurusapiEndpoint"),
configPoolPartySetting.getString("extratorapiEndpoint"),
configPoolPartySetting.getString("coreProjectId"),
configPoolPartySetting.getString("coreThesaurusUri"),
configPoolPartySetting.getString("jelProjectId"),
configPoolPartySetting.getString("jelThesaurusUri"))
val fieldsSettingsList = configFieldSettinglist map { e =>
FieldSettings(e.getString("fieldname"), e.getBoolean("treeBrowser"),
e.getBoolean("multilanguage"), e.getStringList("languages").asScala.toList,
e.getBoolean("closed"), e.getString("scheme"), e.getString("poolpartyProjectId"))
}
val fieldsSettingsMap = Map(fieldsSettingsList map {e => (e.fieldName, e)}: _*)
}
Any idea what the problem might be?
Let's say I have a function
writeToFileOrStdout(fname: String = Nil) = { ... }
If the user passes a string value for fname, then I'd like to open a file with that name and write to it; otherwise, I'd like to print to stdout. I could always just write an if statement to take care of this, but how would I write a case statement on fname and open the correct corresponding outputStream?
val outStream = fname match {
case Nil => ???
case _ => new java.io.FileOutputStream(new java.io.File(fname))
}
outStream.write( ... )
Thanks!
Why not rewrite the function as:
def writeToFileOrStdout(fname: Option[String] = None) = {
val outStream = fname match{
case Some(name) => new java.io.FileOutputStream(new java.io.File(name))
case None => System.out
}
...
}
It's always a good idea to use Option for an optional input as opposed to using null. That's basically what it's there for. In good scala code, you will not see explicit references to null.
In fact, your code doesn't even compile for me. Nil is used to represent an empty list, not a null or non supplied String.
To augment cmbaxter's response...
Mapping a String with a possible null value to Option[String] is trivial: Option(stringValue) will return None where stringValue is null, and Some(stringValue) where non-null.
Thus, you can either:
writeToFileOrStdout(Option(stringValue)), or
If you're stuck on String (and possibly a null value) as the parameter to writeToFileOrStdout, then internally use Option(fname) and match to what it returns::
def writeToFileOrStdout(fname: String = null) = {
val outStream = Option(fname) match{
case Some(name) => new java.io.FileOutputStream(new java.io.File(name))
case None => System.out
}
...
}
To further augment cmbaxter's response, you might consider writing this:
def filenameToOutputStream(name: String) =
new java.io.FileOutputStream(new java.io.File(name))
def writeToFileOrStdout(fname: Option[String] = None) = {
val outStream = fname map filenameToOutputStream getOrElse System.out
...
}
As the post Idiomatic Scala: Your Options Do Not Match suggests, this might be more idiomatic Scala.
Can anyone assist me on how do I load the object FooBar dynamically and call the roquet function (dynamically) ?
I have this snippet in Perl:
#!/usr/bin/perl -w
use Inline Java => << 'End_Of_Java_Code';
class FooBar {
public FooBar(){}
public void roquet() {
System.out.println("HELLO!");
}
}
End_Of_Java_Code
use Data::Dumper;
use Class::Sniff;
my $sniff = Class::Sniff->new({class=>'FooBar'});
my $num_methods = $sniff->methods;
my $num_classes = $sniff->classes;
my #methods = $sniff->methods;
my #classes = $sniff->classes;
my #unreachable = $sniff->unreachable;
foreach my $method (#methods) {
if ( $method eq "roquet" ) {
print "$method\n";
}
}
I tried the following approaches and variations:
a. approach:
use Module::Load
my $package = "main::FooBar";
load $package;
$package->new();
$package->$rflmethod;//where rflmethod is the variable: $rflmethod='roquet';
b. approach:
no strict 'refs';
use Package::Stash;
my $stash = Package::Stash->new('main::FooBar');
my $coderef = $stash->get_symbol('&__new');
$coderef->()
This works:
my ($class, $method) = qw(FooBar roquet);
my $f = $class->new;
$f->$method;