Develop Reference

Spark SQL - Encoders for Tuple Containing a List or Array as an Element

Using Spark 2.2 + Java 1.8
I have two custom data types "Foo" and "Bar". Each one implements serializable.'Foo' has a one to many relationship with 'Bar' so their relationship is represented as a Tuple:
Tuple2<Foo, List<Bar>>
Typically, when I have a 1:1 relationship, I can encode to my custom types like so:
Encoder<Tuple2<Foo,Bar>> fooBarEncoder = Encoders.tuple(Encoders.bean(Foo.class),Encoders.bean(Bar.class));
and then use to encode my Dataset
Dataset<Tuple2<Foo,Bar>> fooBarSet = getSomeData().as(fooBarEncoder);
But I am having trouble finding a way to encode for the scenario when I have a list (or an array) as a Tuple2 element. What I would like to be able to do is to provide an encoder for the second element like this:
Encoder<Tuple2<Foo,List<Bar>>> fooBarEncoder = Encoders.tuple(Encoders.bean(Foo.class), List<Bar>.class);
and then encode to my dataset:
Dataset<Tuple2<Foo,List<Bar>>> fooBarSet = getSomeData().as(fooBarEncoder)
But obviously I cannot invoke .class on a parameterized type like List
I know that for String and primitive types, arrays are supported by spark implicits e.g.:
sparkSession.implicits().newStringArrayEncoder()
But how would I create an encoder for a List or Array of a custom class type?

I'm not sure how well this method could be implemented within your setup but here goes. Create a wrapper class for your list and try it out.
public class BarList implements Serializable {
List<Bar> list;
public List<Bar> getList() {
return list;
}
public void setList(List<Bar> l) {
list = l;
}
}

I'm don't know if it's possible. I tried the following Scala, trying to help, figuring that I could build up the encoder by first teaching spark how to encode X, then List[X] and finally a tuple containing List[X] (not shown below):
import org.apache.spark.sql.Encoders
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import scala.beans.BeanProperty
class X(#BeanProperty var field: String) extends Serializable
case class Z(field: String)
implicit val XEncoder1 = Encoders.bean(classOf[X])
implicit val ZEncoder = Encoders.product[Z]
val listXEncoder = ExpressionEncoder[List[X]] // doesn't work
val listZEncoder = ExpressionEncoder[List[Z]]
listZEncoder works fine
Switching to use
implicit val XEncoder2 = org.apache.spark.sql.Encoders.kryo[X]
Still doesn't work for listXEncoder
The error ends up at a place in catalyst ScalaReflection, which is beyond me.

Google Protocol Buffers checking if a field has been set

I am iterating over the fields of GPB message and have to find out the list of fields which have been set in the message
public void printFields(Person person){
Builder builder = person.toBuilder();
Descriptor descriptor = Person.getDescriptor();
List<FieldDescriptor> fields = descriptor.getFields();
for(FieldDescriptor fd : fields){
Object value = builder.get(fd); //this gives the value
// how to check if this field is set or not
}
}
Checking value for null doesnot help as there can be primitive types in the message as well.
I have read about hasXXX() methods on the message class which tells whether XXX has default value or explicit value, but how to invoke these using builder/descriptor/fieldDescriptor.

If you are using Java 8, you can find the defined fields using the streaming api:
List<Descriptors.FieldDescriptor> definedPersonFields = Person.getDescriptor()
.getFields()
.stream()
.filter(Person::hasField)
.collect(Collectors.toList())

It is late, but may be useful for other people..
The below code will print modified attribute name and its value. you can change it identify populated attributes in proto
Map<Descriptors.FieldDescriptor, Object> modifiedFields = proto.getAllFields();
for(Descriptors.FieldDescriptor fieldDescriptor : modifiedFields.keySet())
{
int protNum = fieldDescriptor.toProto().getNumber();
Descriptors.FieldDescriptor.Type fieldType = fieldDescriptor.getType();
Object value = modifiedFields.get(fieldDescriptor);
System.out.println(fieldDescriptor.getFullName());
System.out.println(value);
}

Why do you need to use descriptor/fieldDescriptor? They are even not available if you define "option optimize_for = LITE_RUNTIME".
Anyway, you can do it already on a Person.Builder or just Person, e.g.:
Builder builder = person.toBuilder();
if (builder.hasXXX()){
XXX xxx = builder.getXXX();
}

References in properties change in both JTree

Before all I'm sorry for my bad english and tell me if anything is not understandable.
I have 2 Jtree. Each tree apparently has the same information. The only thing that changes in them are the names of the properties that has each node.
Eg
JTree1 has an ID and a ParentID. These properties have as a name and value. Name: ID_Tree1. Value: TESTID1 / / Name: ParentID_Tree1. Value: TESTPID1
In JTree2 has the same values ​​as in the JTree1 but the names are different.
There is a moment in which I transfer a node from JTree1 to JTree2 to create it. The transfer/creation is correct but when I read the nodes, it has a different property name architecture(Jtree1 arch.) and can't be read because need to have the JTree2 architecture. I have the function changeAttributesNamesFromDOORSToTC() to solve the problem because it just change the name to the correct name and understandable for JTree2
The real problem: The function make the change in the node of JTree2 but at the same time it change the values name of the same node in JTree1. It makes reference data instead of assignments I think.
How can I solve this!?
Thanks!
JTree treeDOORSCode; //JTree1
JTree treeTCCode; //JTree2
Main Code:
//ACTUAL NODE
DefaultMutableTreeNode selectedTreeNode = (DefaultMutableTreeNode) CurrentSelection.getLastPathComponent();
NodeClass actualNode = (NodeClass)selectedTreeNode.getUserObject();
//ACTUAL PARENT NODE
DefaultMutableTreeNode selectedParentTreeNode = (DefaultMutableTreeNode) selectedTreeNode.getParent();
NodeClass parentNode = (NodeClass) selectedParentTreeNode.getUserObject();
DefaultMutableTreeNode parent = findNode(NodeClass.getNodeParentIdentifierAttrDOORS(parentNode), treeTCCode);
//NEW NODE
DefaultMutableTreeNode newSelectedTreeNode = selectedTreeNode;
//NEW PART
NodeClass newNode = new NodeClass();
newNode = insertNodeInfo(actualNode);
//Create the Model and insert the node
DefaultTreeModel treeModelTC = (DefaultTreeModel)treeTCCode.getModel();
treeModelTC.insertNodeInto(newSelectedTreeNode, parent, 0);
//NEW PART
newNode .changeAttributesNamesFromDOORSToTC();
newSelectedTreeNode.setUserObject(newNode);
Function which change the attr Name values:
public void changeAttributesNamesFromDOORSToTC(){
for (int i = 0; i < this.attributes.size(); i++) {
if (this.attributes.get(i).attributeName.equals(DOORS_ID)){
if (this.tag.equals(TYPE_NAME_CASE)){
this.attributes.get(i).attributeName = TC_IDCASE;
}
if (this.tag.equals(TYPE_NAME_FOLDER)){
this.attributes.get(i).attributeName = TC_IDFOLDER;
}
if (this.tag.equals(TYPE_NAME_FEATURE)){
this.attributes.get(i).attributeName = TC_IDFEATURE;
}
}
if (this.attributes.get(i).attributeName.equals(DOORS_PARENTID)){
this.attributes.get(i).attributeName = TC_PARENTID;
}
if (this.attributes.get(i).attributeName.equals(DOORS_SRS)){
this.attributes.get(i).attributeName = TC_SRS;
}
}
}
Attributes Class:
NodeAttributesClass (String attributeName, String attributeValue)
{
this.attributeName = attributeName;
this.attributeValue = attributeValue;
}
Let me know if need more info!

Object assignment in java is actually a reference copy.
NodeClass newActualNode = actualNode;
That line doesn't mean "put the values of object actualNode into object newActualNode", because actually the variable actualNode is not an instance of NodeClass, but a reference to an instance of NodeClass. So when you do NodeClass newActualNode = actualNode; you are copying the reference and now both variables effectively point to the same instance.
Then when you change the attribute names in one "the other" also changes, because there is no such other, it's the same place in memory.
Now, what you need to do is to create a new instance of NodeClass with the values you want. There are several ways to do so, it's hard for me to know wich one is more suitable as I don't know the internal structures, but in the end you need to:
Create a new NodeClass instance for the newActualNode variable
Put the values (fields) of actualNode into the newActualNode
Assign the attribute names that you want in newActualNode
So, it could be something like this:
NodeClass newActualNode = new NodeClass(actualNode); //copy constructor NodeClass(NodeClass anotherNode);
newActualNode.changeAttributesNamesFromDOORSToTC(); //assuming the constructor doesn't put them right
or you could use a flag in the NodeClass construtors to indicate what kind of attribute names you want.
and the copy constructor should look like this:
public NodeClass(NodeClass anotherNode)
{
this(anotherNode.someFields); //call to your "normal" constructor, with whatever params you need
//copy the values into the new instance this, if you didn't do it in the above line
this.field1 = anotherNode.field1;
this.field2 = anotherNode.field2;
//...
this.fieldn = anotherNode.fieldn;
}
You can take this code with a grain of salt, there are several ways to do it, the differences are subtle if any. The important thing is that you need to have another instance for the other tree.
EDIT
If that doesn't work my guess would be, that either you need to do newSelectedTreeNode.setUserObject(newNode); before the insert, like this:
NodeClass newNode = new NodeClass();
newNode = insertNodeInfo(actualNode);
newNode.changeAttributesNamesFromDOORSToTC();
DefaultMutableTreeNode newSelectedTreeNode = new DefaultMutableTreeNode();
newSelectedTreeNode.setUserObject(newNode);
treeModelTC.insertNodeInto(newSelectedTreeNode, parent, 0);
or that parent is not properly calculated, in particular that you are getting the parent node of the treeDOORSCode, and not the one of the treeTCCode.

java.lang.IllegalArgumentException: invalid ObjectId [0]

I'm getting an java.lang.IllegalArgumentException: invalid ObjectId [0] when I try to do similar to the code below in one of my class.
String s = "0_abc";
--------
private ObjectId obj; [Instance variable]
public NewClass(String s){
String[] sarray = s.split("_");
obj = new ObjectId(sarray[0]);
}
What could be the possible reason for it to fail?

I assume ObjectId class is (org.bson.types.ObjectId) and it is used in the MongoDB Java driver.
What this exception means is that "0" cannot be pass as a value for the ObjectId construction.
Not every string is a correct ObjectId value. When ObjectId is initialized with the given string, validation is done. For example ObjectId requires string to have 24 characters. There are more constrains and you can find the complete validation source code in GitHub repo, in the method isValid().
If you want to create an ObjectId object with correct value you can create it using available factory method like this:
ObjectId obj = ObjectId.get()
This will create ObjectId with auto-generated value.

Unfortunately you have not sent the stacktrace but I think that the exception is thrown from constructor of your class ObjectId that does not allow sending 0 as an argument.

Formatting a string in Java using class attributes

I have a class with an attribute and getter method:
public Class MyClass
{
private String myValue = "foo";
public String getMyValue();
}
I would like to be able to use the value of foo in a formatted string as such:
String someString = "Your value is {myValue}."
String result = Formatter.format(someString, new MyClass());
// result is now "Your value is foo."
That is, I would like to have some function like .format above which takes a format string specifying properties on some object, and an instance with those properties, and formats the string accordingly.
Is it possible to do accomplish this feat in Java?

you could use JUEL for this. it's an implementation of the Java Expression Language. the code is rather compact and looks like this:
ExpressionFactory factory = new ExpressionFactoryImpl();
// create a context and add a Person object to the context, this variable will be used
// in the property replacement
// objects of type Person have two fields: firstName and lastName
SimpleContext context = new SimpleContext();
Person person = new Person("John", "Doe");
context.setVariable("person", factory.createValueExpression(person, Person.class));
// create the expression
String expr = "My name is ${person.firstName} ${person.lastName}";
ValueExpression e = factory.createValueExpression(context, expr, String.class);
// evaluate the expression
System.out.println(e.getValue(context));
which prints 'My name is John Doe'
note that it's also possible to use an expression like this: '${firstName}' instead of '${person.firstName}', but then you will have to write and provide a custom resolver (javax.el.ELResolver) for the variable and property resolution

(My other answer's probably only useful if you're already using struts.)
Similar to sdb's answer, there is apache JEXL.
The UnifiedJEXL class provides template-like functionality, so you can write (as shown in javadocs):
JexlEngine jexl = new JexlEngine();
UnifiedJEXL ujexl = new UnifiedJEXL(jexl);
UnifiedJEXL.Expression expr = ujexl.parse("Hello ${user}");
String hello = expr.evaluate(context, expr).toString();
(The expr not only looks strange being passed as a parameter to a method on itself, but is indeed not needed as a parameter)
The context setup is shown earlier in the same page:
// Create a context and add data
JexlContext jc = new MapContext();
jc.set("foo", new Foo() );
You'll also need either commons-logging, or you can configure JEXL to use your own logger.
So to get close to what you asked, you can create:
public class Formatter {
public static String format(String format, Object ... inputs) {
JexlContext context = new MapContext();
for (int i=0;i<inputs.length;i++) {
context.set("_" + (i+1), inputs[i] );
}
JexlEngine jexl = new JexlEngine();
UnifiedJEXL ujexl = new UnifiedJEXL(jexl);
UnifiedJEXL.Expression expr = ujexl.parse(format);
return expr.evaluate(context).toString();
}
}
and call it with
String someString = "Your value is ${_1.myValue}.";
String result = Formatter.format(someString, new MyClass());
At which point, result is "Your value is foo."

It's in theory possible with a stackbased parser to determine the valueholders in the string, in combination with reflection (or better, a Javabean inspection API, such as Commons BeanUtils) to get the bean property values.
Unfortunately no ready-made nor 3rd party API comes to mind, if you were looking for that. It's an interesting question however.

You could create one with struts2/xwork/OGNL, similar to the below (copied from an email from Vlad)
public static String translateOgnl(String message, Map<Object, Object> args) {
OgnlValueStack stack = new OgnlValueStack();
stack.push(args);
return TextParseUtil.translateVariables(message, stack);
}
The javadocs for TextParseUtil.translateVariables() say
Converts all instances of ${...} in expression to the value returned by a call to ValueStack.findValue(java.lang.String). If an item cannot be found on the stack (null is returned), then the entire variable ${...} is not displayed, just as if the item was on the stack but returned an empty string.