How to use Or operation inside when function in Spark Java API. I want something like this but I get a compiler error.
Dataset<Row> ds = ds1.withColumn("Amount2", when(ds2.col("Type").equalTo("A") Or ds2.col("Type").equalTo("B"), "Amount1").otherwise(0))
Can somebody guide me please with a sample expression.
You should use or method:
ds2.col("Type").equalTo("A").or(ds2.col("Type").equalTo("B"))
With equalTo isin should work as well:
ds2.col("Type").isin("A", "B")
Related
I'm trying to write this java code in scala, but I am getting a compile error.
Document<String, String> doc = Document.<String, String>builder().id("newDocId").score(1d).build();
I am trying:
val doc = Document.<String, String>builder().id("newDocId").score(1d).build();
How do I convert this java generic usage?
I also tried Document[String, String] but I get an error sayingDocument is not a value.
Try this:
val doc = Document.builder[String, String]().id("newDocId").score(1d).build()
Scala uses square brackets for generics (and semicolons are optional). Also, the type parameters go to the method, not the object.
Using the AWS DataPipeline API, I'm trying to programmatically evaluate an Expression like the following:
sometext-#{format(#scheduledStartTime, 'YYYYMMddHHmmss')
To evaluate the expression, I'm using a PipelineObject that looks something like the following:
Id:#MyPipelineObject_2018-08-26T01:00:00
Name:#MyPipelineObject_2018-08-26T01:00:00
- Key:#scheduledStartTime
- StringValue:2018-08-26T01:00:00
- Key:#scheduledEndTime
- StringValue:2018-08-27T01:00:00
How can I evaluate the expression, given that I know the pipelineId and pipelineObjectId? I'm using the Java AWS SDK, and creating an EvaluateExpressionRequest like so:
String expressionToBeEvaluated = "sometext-#{format(#scheduledStartTime, 'YYYYMMddHHmmss')";
String myPipelineObjectId = "#MyPipelineObject_2018-08-26T01:00:00";
EvaluateExpressionRequest evaluateExpressionRequest = new EvaluateExpressionRequest()
.withPipelineId(myPipelineId)
.withExpression(expressionToBeEvaluated)
.withObjectId(myPipelineObjectId);
However, from the docs it's not clear to me how to actually issue the request with the EvaluateExpressionRequest object. I've looked at EvaluateExpressionResult but the setEvaluatedExpression method only takes a String as input.
Am I doing something wrong, missing something fundamental, or does the SDK just not support what I'm trying to do?
Any input or suggestions would be really appreciated. Thanks!
So I figured it out a few minutes after posting my question. It turns out the answer is very simple and I've just been looking at this stuff for too long. The DataPipeline object has the method evaluateExpression() which takes an EvaluateExpressionRequest and returns the EvaluateExpressionResult. You get the evaluated result by calling getEvaluatedExpression on the returned object.
EvaluateExpressionRequest evaluateExpressionRequest = new EvaluateExpressionRequest()
.withPipelineId(myPipelineId)
.withExpression(expressionToBeEvaluated)
.withObjectId(myPipelineObjectId);
dataPipeline.evaluateExpression(evaluateExpressionRequest).getEvaluatedExpression(); //evaluates to sometext-20180826010000
Hope that helps anyone with a similar affliction!
I'm trying to filter a Spark DataFrame using a list in Java.
java.util.List<Long> selected = ....;
DataFrame result = df.filter(df.col("something").isin(????));
The problem is that isin(...) method accepts Scala Seq or varargs.
Passing in JavaConversions.asScalaBuffer(selected) doesn't work either.
Any ideas?
Use stream method as follows:
df.filter(col("something").isin(selected.stream().toArray(String[]::new))))
A bit shorter version would be:
df.filter(col("something").isin(selected.toArray()));
what are the changes that I need to make if I am using jdk 7 and want to use lambda expression?
I am comparing 2 xml files and want to ignore specific nodes hence using this expression
final Diff documentDiff = DiffBuilder
.compare(expectedSource)
.withTest(actualSource)
.withNodeFilter(node -> !node.getNodeName().equals(someName))
.build();
error: Syntax error on token '-',-- expected
Try this.
final Diff documentDiff = DiffBuilder
.compare(expectedSource)
.withTest(actualSource)
.withNodeFilter(new Predicate<Node>() {
#Override
public boolean test(Node node) {
return !node.getNodeName().equals(someName);
}
})
.build();
This is redundant, but JDK7 will accept it.
I don't know if you can realize what you want to do with this.
lambda expression can not be used in java 7. The syntax itself is only allowed in java8. However the functionality you can achieve by writing more code or with out using lamda expression. You need to write a filter method which checks the name of node with someName if it returns true then proceed with building document Difference. You will need to write multiple statement and if case to check name equality.
You can also achieve this using xslt. which is very fast for long xml files but then you will need to write a lot of code as xslt is declarative and based upon functional programming.
Like
String r = SomeThing.toExecString("new Object().toString()");
And when executed the value of r would be:
"new Object().toString() = java.lang.Object#c5e3974"
Is this even possible at all? Would it need a bunch of reflection? A built in compiler maybe?
ScriptEngine engine = new ScriptEngineManager().getEngineByName("beanshell");
Object result = engine.eval("new Object().toString();");
System.out.println(result);
You may get close to what you want using BeanShell. I ran the above code with Java 6 with BeanShell 2.0b4 and the JSR 223-based bsh-engine.jar engine on the classpath.
There is a great post here:
Generating Static Proxy Classes - http://www.javaspecialists.eu/archive/Issue180.html
Part one is enough for what you asked, I think
Don't know if you really wanted this. But your problem would be solved with this method:
String toExecString( String code ) {
return String.format(
"\"%s\" = %s#%x",
code,
code.getClass().getName(),
code.hashCode()
);
}