I'm running into issues with Java Spark Dataset's groupByKey method. The following code, when run locally in a test environment (Spark 2.1.0, spark-core_2.11, spark-sql_2.11), throws the following exception:
java.lang.Exception: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 43, Column 21: No applicable constructor/method found for zero actual parameters; candidates are: "public int org.package.example.ExampleTest$1ExampleClass.getX()
Code is:
class ExampleClass implements Serializable {
private int x;
private int y;
public ExampleClass() {}
public ExampleClass(int x, int y) {this.x = x; this.y = y;}
public int getX() {return x;}
public void setX(int x) {
this.x = x;
}
public int getY() {
return y;
}
public void setY(int y) {
this.y = y;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ExampleClass that = (ExampleClass) o;
if (x != that.x) return false;
return y == that.y;
}
#Override
public int hashCode() {
int result = x;
result = 31 * result + y;
return result;
}
}
ExampleClass exampleClass1 = new ExampleClass(1, 1);
ExampleClass exampleClass2 = new ExampleClass(1, 2);
ExampleClass exampleClass3 = new ExampleClass(1, 3);
List<ExampleClass> exampleClasses = Lists.newArrayList(
exampleClass1,
exampleClass2,
exampleClass3
);
Dataset<ExampleClass> dataset = spark.createDataset(exampleClasses, Encoders.bean(ExampleClass.class));
KeyValueGroupedDataset<Integer, ExampleClass> grouped = dataset.groupByKey(
(MapFunction<ExampleClass, Integer>) ExampleClass::getX,
Encoders.INT()
);
}
It looks like it can't find the default parameter-less constructor, or I'm leaving something out. Also interesting, if I change the ints to boxed Integers, it says the candidate is getY() not getX().
Any help is much appreciated!
EDIT: After messing around with it some more, moving ExampleClass out of a nested class in my test and into its own file fixes the problem. I still don't know why though so any answers are still appreciated.
I had the same issue. The problem in my case and I suspect in yours, is that your class isn't declared with the public modifier.
public class ExampleClass implements Serializable
Kudos to Andy Grove: https://www.mail-archive.com/user#spark.apache.org/msg55084.html
Related
I am trying to make a videogame that is similar to "Turing Complete". In this game you are supposed to connect circuit pieces to create a computer by the end. So I have a question on inheritance. I have a class called "Circuitware" and every object extends this as follows
public Circuitware implements connectable {
protected int x;
protected int y;
protected int allowedConnetions;
protected Ciruitware[] connections;
private int totalConnections;
public Circuitware(int x, int y, int allowedConnections, Circuitware[] connections) {
this.x = x;
this.y = y;
this.allowedConnections = allowedConnections;
this.connections = new Array[allowedConnections];
for (Object O: connections) {
if (O != null) {
this.totalConnections += 1;
}
}
}
public int getX() {
return this.x;
}
public int getY() {
return this.y;
}
public void setX(int x) {
this.x = x;
}
public void setY(int y) {
this.y = y;
}
public<T> void addConnection(T connection) {
if (this.totalConnections<this.allowedConnections) {
this.connections.add(connection);
}
}
}
Now an example of a subclass is
public class LogicGate extends Circuitware {
String type;
public LogicGate(int x, int y, int allowedConnections, Circuitware[] connections, String type) {
super(x, y, allowedConnections, connections);
this.type = type;
}
public boolean output() {
//Depending on the gate and input we output things
}
}
Now the thing is that the number of allowed connections depends on the type of the circuit, for example logic gates in my game can have 1 connection if it is an NOT gate, 2 connections if it is an AND, XOR, NOR gates or 3 connections if it is and extended OR, extended AND gates, where are switches can only have 0. Is there are better way of doing this?
I also need help understanding that if I have an array of type Circuitware can I add objects from its subclasses? Because the output of logic gates and inputs of other logic gates while they can be inputs to clocks.
Would it be better, design-wise, if I got rid of the "Circuitware" class altogether? But then wouldn't that make connections between different classes harder?
This question already has answers here:
What's wrong with overridable method calls in constructors?
(8 answers)
Closed 2 years ago.
I have the follow classes:
public abstract class MyAbstractClass {
protected int x;
protected int number;
public MyAbstractClass(int x) {
this.x = x;
this.number = this.generateNumber();
}
public abstract int generateNumber(); // This class is called in the super constructor and elsewhere
}
public class MySubClass extends MyAbstractClass {
private int y;
public MySubClass(int x, int y) {
super(x);
this.y = y;
}
#Override
public int generateNumber() {
return this.x + this.y; // this.y has not been initialized!
}
}
My issue is that MySubClass's y property has to be initialized before the super constructor is run, because a method using y is run in the super constructor.
I am aware that this may not be possible even with some sneaky workaround, but I am yet to find an alternative solution.
Also please keep in mind that I will have many more derived classes, with different values being passed into their constructors.
You can defer the number calculation until it is needed.
public abstract class MyAbstractClass {
protected int x;
protected Integer number;
public MyAbstractClass(int x) {
this.x = x;
}
public int getNumber() {
if (number == null) {
number = generateNumber();
}
return number.intValue();
}
protected abstract int generateNumber();
}
I want to use a map in Groovy where the keys will be instances of an unmutable class.
This is something I do often in Java and it works fine, like in this example class:
public class TestMap {
static final class Point {
final int x; final int y;
public Point(int x, int y) {this.x = x;this.y = y;}
}
public static void main(String[] args) {
Map<Point, String> map = new HashMap<>();
final Point origin = new Point(0, 0);
map.put(origin, "hello world !" );
if(!map.containsKey(origin))
throw new RuntimeException("can't find key origin in the map");
if(!map.containsKey(new Point(0,0))) {
throw new RuntimeException("can't find new key(0,0) in the map");
}
}
}
But when I try to achieve the same thing with Groovy, it doesn't work.
Why ?
Here is a sample non working example in Groovy:
class Point {
final int x; final int y
Point(int x, int y) { this.x = x; this.y = y }
public String toString() { return "{x=$x, y=$y}" }
}
def origin = new Point(0, 0)
def map = [(origin): "hello"]
map[(new Point(1,1))] = "world"
map.put(new Point(2,2), "!")
assert map.containsKey(origin) // this works: when it's the same ref
assert map.containsKey(new Point(0,0))
assert map.containsKey(new Point(1,1))
assert map.containsKey(new Point(2,2))
assert !map.containsKey(new Point(3,3))
You need to have an equals and hashCode method on your Point class so that the instances can be found as keys in the HashMap
You can do this quickly by adding an annotation in Groovy:
import groovy.transform.*
#EqualsAndHashCode
class Point {
final int x; final int y
Point(int x, int y) { this.x = x; this.y = y }
public String toString() { return "{x=$x, y=$y}" }
}
I have a class Vertex which implements Comparable and overrides equals.
public class Vertex implements Comparable<Vertex>{
private final int x;
private final int y;
private final char c;
public Vertex(int x, int y, char c) {
this.x = x;
this.y = y;
this.c=c;
}
public double heuristic(Vertex goal){
double dx = Math.abs(x - goal.x);
double dy = Math.abs(y - goal.y);
return Math.sqrt(Math.pow(dx, 2)+Math.pow(dy, 2));
}
#Override
public String toString() {
return ("(" + x + "," + y + ")"+"["+c+"]");
}
#Override
public int compareTo(Vertex v) {
// TODO Auto-generated method stub
if(this.heuristic(end)<v.heuristic(end)) return -1;
return 1;
}
public boolean equals(Vertex v){
if(this.x==v.x && this.y==v.y) return true;
return false;
}
/*
public boolean equals(Object o){
if(o.getClass().getName()=="Vretex"){
Vertex v=(Vertex)o;
return this.equals(v);
}
return false;
}
*/
}
When I use PriorityQueue and check if one object (different objects) that equal functionality according the equal function is in the PriorityQueue I get false.
Also tried the one in comment without success.
Your equals is wrong, it should be
#Override public boolean equals (Object other)...
i.e. the parameter must be an Object. Also you should implement hashCode too. See http://java67.blogspot.com.br/2013/04/example-of-overriding-equals-hashcode-compareTo-java-method.html for more information.
(sorry for the pun)
Say one wants to define a generic builder, like this:
public abstract class GenericBuilder<T extends Product> {
int x;
int y;
<K extends GenericBuilder<T>> K setX(int x) {
this.x = x;
return (K)this;
}
<K extends GenericBuilder<T>> K setY(int y) {
this.y = y;
return (K) this;
}
abstract T build();
}
abstract class Product {
int x;
int y;
}
class ConcreteProduct extends Product {
int z;
}
class ConcreteBuilder extends GenericBuilder<ConcreteProduct>{
int z;
<K extends GenericBuilder<ConcreteProduct>> K setZ(int z) {
this.z = z;
return (K) this;
}
#Override
ConcreteProduct build() {
ConcreteProduct cp = new ConcreteProduct();
cp.x = x;
cp.y = y;
cp.z = z;
return cp;
}
public static void main(String[] args) {
new ConcreteBuilder().setX(1).setY(2).setZ(3);
}
}
When calling ConcreteBuilder.setZ(), it fails during compilation.
Why is that? Is it due erasure? Or the generics, say, don't carry information about its generic parameters?
EDIT:
Any ideas how to avoid using second generic parameter in:
public class ConcreteBuilder extends GenericBuilder<ConcreteProduct, ConcreteBuilder>
i.e. <..., ConcreteBuilder>, which seems to be a little clumsy? I guess it's not possible. Are there other languages (C# maybe?) which allow to do that?
Break your code this way and you will understand that your class GenericBuilder<ConcreteProduct> doesn't have any setZ() method defined.
GenericBuilder<ConcreteProduct> setY = new ConcreteBuilder().setX(1).setY(2);
setY.setZ(3);
In your GenericBuilder your functions return a GenericBuilder when you don't specify the type argument of the function. In your main function the call to setX returns a GenericBuilder and you loose the information that you are actually using a ConcreteBuilder. To succesfully make the calls, you have to specify the generic parameters for the setters:
new ConcreteBuilder().<ConcreteBuilder>setX(1).<ConcreteBuilder>setY(2).setZ(3);
Alternative
You can add a second type parameter to GenericBuilder:
public abstract class GenericBuilder<T extends Product, K extends GenericBuilder<T, K>> {
int x;
int y;
K setX(int x) {
this.x = x;
return (K)this;
}
K setY(int y) {
this.y = y;
return (K) this;
}
abstract T build();
}
and change ConcreteBuilder to this:
public class ConcreteBuilder extends GenericBuilder<ConcreteProduct, ConcreteBuilder> {
int z;
ConcreteBuilder setZ(int z) {
this.z = z;
return this;
}
#Override
public ConcreteProduct build() {
ConcreteProduct cp = new ConcreteProduct();
cp.x = x;
cp.y = y;
cp.z = z;
return cp;
}
public static void main(String[] args) {
new ConcreteBuilder().setX(1).setY(2).setZ(3);
}
}