I am coming from Java background and trying to understand how to model Domain classes/POJOs in Scala.
I am trying to Deserialize JSON response from a RestAPI and my Java POJOs are as follows:
#Data
public class ColumnResponse {
private String id;
private String name;
private String type;
...
}
k
#Data
public class DataSetGetResponse {
private String id;
private List<ColumnResponse> columns;
...
}
Now I have created following Case Classes
case class DataSetGetResponse (id: String,
columns: List[ColumnResponse]
.... )
case class ColumnResponse (id: String,name: String ...)
I am trying to use https://sttp.readthedocs.io/en/latest/json.html#json4s library for HTTP communication and json4s for deserialization.
Questions:
1) In the DataSetGetResponse case class, field "columns" is a List.By default this is an immutable list. How the Deserialization library add new DataColumnGetResponse objects to this immutable list? Do I have to declare this as mutable ?
2) There is a field called 'type' field in the ColumnResponse POJO. In Scala 'type' is a reserved keyword.How to handle this case?
Answer the first one:
An immutable object can be mutated with the copy function:
dataSet.copy(columns = newResp :: dataSet.columns)
For more complex tasks you can use Lenses see for example here: enter link description here
Answer the second one:
If it is a reserved word you can do it like
case class ColumnResponse (id: String, name: String, `type`: String)
This answer addresses the following aspect of the question:
How the Deserialization library add new DataColumnGetResponse objects
to this immutable list?
Let us consider a simplified version of the problem:
JsonMethods.parse("""[1,2,3]""").extract[List[Int]]
How does json4s deserialise [1,2,3] into immutable List[Int]? First it parses the raw string into an intermediary AST (abstract syntax tree) data structure where it represents the list like so
case class JArray(arr: List[JValue]) extends JValue
We see here that arr is an immutable list. The key line that builds it up after parse executes is in JsonParser
def newValue(v: JValue): Unit = {
...
case a: JArray => vals.replace(JArray(v :: a.arr))
...
}
Note how the operator :: in v :: a.arr adds an element at the beginning of this list and returns a new list with v added in. This means since there are three elements in [1,2,3] the following three lists are created by json4s in the process of deserialisation
JArray(List(JInt(1))
JArray(List(JInt(2), JInt(1)))
JArray(List(JInt(3), JInt(2), JInt(1)))
Again note these are three separate lists.
Next, after internal AST is created, actual deserialisation to List[Int] takes place by calling extract[List[Int]]. The key component that does this for lists is CollectionBuilder
private class CollectionBuilder(json: JValue, tpe: ScalaType)(implicit formats: Formats) {
...
val array: Array[_] = json match {
case JArray(arr) => arr.map(extractDetectingNonTerminal(_, typeArg)).toArray
...
}
Note how we simply map over AST arr built up during parsing step and convert each element to the model of type typeArg, which in our simple case is Int but in your case would be DataColumnGetResponse.
Related
Using Spark 2.2 + Java 1.8
I have two custom data types "Foo" and "Bar". Each one implements serializable.'Foo' has a one to many relationship with 'Bar' so their relationship is represented as a Tuple:
Tuple2<Foo, List<Bar>>
Typically, when I have a 1:1 relationship, I can encode to my custom types like so:
Encoder<Tuple2<Foo,Bar>> fooBarEncoder = Encoders.tuple(Encoders.bean(Foo.class),Encoders.bean(Bar.class));
and then use to encode my Dataset
Dataset<Tuple2<Foo,Bar>> fooBarSet = getSomeData().as(fooBarEncoder);
But I am having trouble finding a way to encode for the scenario when I have a list (or an array) as a Tuple2 element. What I would like to be able to do is to provide an encoder for the second element like this:
Encoder<Tuple2<Foo,List<Bar>>> fooBarEncoder = Encoders.tuple(Encoders.bean(Foo.class), List<Bar>.class);
and then encode to my dataset:
Dataset<Tuple2<Foo,List<Bar>>> fooBarSet = getSomeData().as(fooBarEncoder)
But obviously I cannot invoke .class on a parameterized type like List
I know that for String and primitive types, arrays are supported by spark implicits e.g.:
sparkSession.implicits().newStringArrayEncoder()
But how would I create an encoder for a List or Array of a custom class type?
I'm not sure how well this method could be implemented within your setup but here goes. Create a wrapper class for your list and try it out.
public class BarList implements Serializable {
List<Bar> list;
public List<Bar> getList() {
return list;
}
public void setList(List<Bar> l) {
list = l;
}
}
I'm don't know if it's possible. I tried the following Scala, trying to help, figuring that I could build up the encoder by first teaching spark how to encode X, then List[X] and finally a tuple containing List[X] (not shown below):
import org.apache.spark.sql.Encoders
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import scala.beans.BeanProperty
class X(#BeanProperty var field: String) extends Serializable
case class Z(field: String)
implicit val XEncoder1 = Encoders.bean(classOf[X])
implicit val ZEncoder = Encoders.product[Z]
val listXEncoder = ExpressionEncoder[List[X]] // doesn't work
val listZEncoder = ExpressionEncoder[List[Z]]
listZEncoder works fine
Switching to use
implicit val XEncoder2 = org.apache.spark.sql.Encoders.kryo[X]
Still doesn't work for listXEncoder
The error ends up at a place in catalyst ScalaReflection, which is beyond me.
I have a pojo similar to below pojo
public class TestData {
int number;
String name;
//Getters, setters, rest of class
}
The multiple object of this pojo is inserted in the rule engine. Now I want a list having name attribute collected from all inserted pojo through rule.
i.e. List<String> for the name from the inserted pojo.
Thanks
Shorav
This very simple rule collects all names in TestData facts:
rule "collect names"
when
accumulate( TestData( $n: name ); $names: collectList( $n ) )
then
// ... $names is a List containing String objects
end
Note that the List $names is compiled as a List<?>. Processing on the right hand side may have to cast list elements to String.
An API my application is communicating with sends responses that look like:
{
Code: 200,
Message: "HELLO",
Data: []
}
The Data field is always an array of SOMETHING. But that something could be a single node of text, another array of something else, or any other of an assortment of different objects.
In the below example, the data node is an array of an array of car objects.
Data: [ [ {car:1}, {car:2} ] ]
Another return type could be an array of insect objects:
Data: [ {insect : spider} ]
I would like to design a Gson object to handle this and was wondering what the best way would be.
My first thought is to have an abstract class that holds the Code and Message fields, and then have many sub-types that all have their own Data field. Then I would just call .fromJson() passing it the sub-class.
Is there a more optimal way to design it so that Gson would handle the differences?
I figured out what I believe is the best answer. Fairly straightforward!
Make the class generic and supply the type by creating a TypeToken before passing to Gson:
public class Response<T> {
private String code;
private String message;
private List<T> data;
}
Then when using Gson:
Type myCarListResponse = new TypeToken<Response<List<Car>>>(){}.getType();
Response<List<Car>> response = gson.fromJson(json, myCarListResponse);
Replace > with the type you are expecting from the Data node. The above example satisfies the first example from the original post.
To satisfy the second example:
Type myInsectResponse = new TypeToken<Response<Insect>>(){}.getType();
Response<Insect> response = gson.fromJson(json, myInsectResponse);
In Jackson, you can use #JsonAnyGetter/Setter to achieve this.
Refer http://www.cowtowncoder.com/blog/archives/2011/07/entry_458.html, http://wiki.fasterxml.com/JacksonFeatureAnyGetter
consider following classes:
class Basic{
String id;
Double val;
//some other member variables
}
class NodeBO{
List<String> id;
Type type;
// list of id from objects of Basic class in data below
Map<ChEnum, Basic> data;
addBeans(NodeBO nodeBO, Node node){
// in transaction...
node.setProperty("priperties", nodeBO.toString());
// is it ok to convert to array? or should be converted to JSON string?
node.setProperty(GraphElementProps.id,toArray(nodeBO.id));
node.setProperty(GraphElementProps.type, nodeBO.type);
}
#override
toString(){
//return json of this object
}
}
enum ChEnum{
CH1(1), CH2(2);
// constructor and some methods
}
nodes are indexed using autoIndexer:
AutoIndexer<Node> nodeAutoIndexer = GRAPH_DB.index().getNodeAutoIndexer();
nodeAutoIndexer.startAutoIndexingProperty(GraphElementProps.id);
nodeAutoIndexer.setEnabled(true);
GRAPH_NODE_AUTO_INDEX = nodeAutoIndexer.getAutoIndex();
Here I'm storing GraphElementProps.id as node property (by converting to array). Does it take array (of string) as property? Or should I convert list to JSON-string and then store?
I want to be able to query on this array given with queryId. e.g. query on node-index to get nodes in which node.getProperty(GraphElementProps.id) contain given queryId? i.e. something like:
// how to do this?
GRAPH_NODE_AUTO_INDEX.get(/*Nodes whose id contain queryId*/);
Or is it (somehow) possible to make id property of Basic class indexable and searchable? How to index such properties, if possible? and how to query them?
I'm unable to understand but is it something related to Spring-data-neo4j? I'm completely new to Spring-data-neo4j.
I think the best solution is to use Spring-data-neo4j. This will allow to index embedded fields and query on them.
I have some difficulties with json deserialization using GSon and I hope somebody can help me.
I want to deserialize the following json snippet:
{
"fieldA": "valueA",
"myCollection": {
"AnotherClass": [
{
"objectAfieldA": "valueB",
"objectAfieldB": "valueC"
},
{
"objectAfieldA": "valueD",
"objectAfieldB": "valueE"
}
]
}
}
the corresponding overall class has following fields:
...
String fieldA;
List<AnotherClass> = new ArrayList<AnotherClass>();
....
Now, my problem is that when I deserialize, using fromJson(jsonSample, resultContainer.class), without the List<T> element, everything is good, but I get a NullPointerException when I include the contained list. I've read about how to deal with collections of generic types and the use of TypeToken, but I can't apply this knowledge when my collection is part of another class…
I really would appreciate any help to solve this.
The solution for deserealizing the unnamed JSON array is quite simple:
List<resultContainer> lres = gson.fromJson(new FileReader("input.json"), new TypeToken<List<resultContainer>>(){}.getType());
When deserializing, you only need to use the TypeToken if the outer-most structure to be deserialized into is a generic collection. This is not the case for the example in the original question. So, use of a TypeToken is unnecessary.
The issue appears to be that the JSON structure does not match the Java structure attempting to be bound to.
The JSON structure defines
an object with two elements
element 1 is a string named "fieldA",
element 2 is an object named "myCollection", which has one element
the one element is an array named "AnotherClass", composed of objects with two elements
element 1 is a string named "objectAfieldA",
element 2 is a string named "objectAfieldB"
So, define a Java data structure to match that, and deserialization will work very simply, without any custom processing necessary. If such a matching Java structure is not provided, then custom deserialization is necessary.
Here is such a working example using the names and types from the original question.
import java.io.FileReader;
import com.google.gson.Gson;
public class Foo
{
public static void main(String[] args) throws Exception
{
Gson gson = new Gson();
resultContainer result = gson.fromJson(new FileReader("input.json"), resultContainer.class);
System.out.println(gson.toJson(result));
}
}
class resultContainer
{
String fieldA;
MyCollectionContainer myCollection;
}
class MyCollectionContainer
{
SomeOtherClass[] AnotherClass;
}
class SomeOtherClass
{
String objectAfieldA;
String objectAfieldB;
}