I have an object that is a list of 'Level' objects and I'm testing transferring them with Spring Boot Rest Controller in 2 ways:
with JSON, in Rest Controller I use something like:
#RequestMapping(value = "/api/v1/layers/{layername}", method = RequestMethod.GET, produces = "application/json")
public #ResponseBody List<Level> query(#PathVariable String layername,
#RequestParam("northEastLat") Float northEastLat,
#RequestParam("northEastLng") Float northEastLng,
#RequestParam("northWestLat") Float northWestLat,
#RequestParam("northWestLng") Float northWestLng,
#RequestParam("southEastLat") Float southEastLat,
#RequestParam("southEastLng") Float southEastLng,
#RequestParam("southWestLat") Float southWestLat,
#RequestParam("southWestLng") Float southWestLng
) {
List<Level> poligons=levelService.obtainLevels(layername,southWestLng,southWestLat,northWestLng,northWestLat,northEastLng,northEastLat,southEastLng,southEastLat);
int i=1;
for (Level p : poligons) {
System.out.println("poligon" + i++ + " is:" + p.toString());
}
return poligons;
}
With Protostuff Protobuf format, I use something like:
#RequestMapping(value = "/api/v1/layers/{layername}", method = RequestMethod.GET,produces = "text/plain")
public String query(#PathVariable String layername,
#RequestParam("northEastLat") Float northEastLat,
#RequestParam("northEastLng") Float northEastLng,
#RequestParam("northWestLat") Float northWestLat,
#RequestParam("northWestLng") Float northWestLng,
#RequestParam("southEastLat") Float southEastLat,
#RequestParam("southEastLng") Float southEastLng,
#RequestParam("southWestLat") Float southWestLat,
#RequestParam("southWestLng") Float southWestLng
) {
List<Level> poligons=levelService.obtainLevels(layername,southWestLng,southWestLat,northWestLng,northWestLat,northEastLng,northEastLat,southEastLng,southEastLat);
LevelList list = new LevelList(poligons);
byte[] bytes;
int i=1;
for (Level p : poligons) {
System.out.println("poligon" + i++ + " is:" + p.toString());
}
Schema<LevelList> schema = RuntimeSchema.getSchema(LevelList.class);
LinkedBuffer buffer = LinkedBuffer.allocate();
try
{
bytes = ProtostuffIOUtil.toByteArray(list, schema, buffer);
}
finally
{
buffer.clear();
}
return new String(bytes);
}
The Level object format is :
[{"wkb_geometry":"{"type":"Polygon","coordinates":[[[24.446822,45.34997],[24.706508,45.352485]]]}","id":199,"level":"3","type":null}
The Level object is :
#Entity(name = "Level")
#Table(name="Level2G")
#SecondaryTables({
#SecondaryTable(name="Level3G"),
#SecondaryTable(name="Level4G")
})
public class Level implements Serializable {
private static final long serialVersionUID = 1L;
// #Column(name = "wkb_geometry",columnDefinition="Geometry")
//#Type(type = "org.hibernate.spatial.GeometryType")
#Column(name="wkb_geometry")
private /*Geometry */ String wkb_geometry;
#Id
#Column(name="id")
private Integer id;
#Column(name="level")
private String level;
#Transient
private String type;
public Level() {
}
public Level(String wkb_geometry, Integer id, String level) {
this.wkb_geometry = wkb_geometry;
this.id = id;
this.level = level;
this.type = "Feature";
}
public Level(String wkb_geometry, Integer id, String level, String type) {
this.wkb_geometry = wkb_geometry;
this.id = id;
this.level = level;
this.type = type;
}
public Object getWkb_geometry() {
return wkb_geometry;
}
public void setWkb_geometry(String wkb_geometry) {
this.wkb_geometry = wkb_geometry;
}
public Integer getId() {
return id;
}
public void setId(Integer id) {
this.id = id;
}
public String getLevel() {
return level;
}
public void setLevel(String level) {
this.level = level;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
#Override
public String toString() {
return "Level{" +
"wkb_geometry=" + wkb_geometry +
", id=" + id +
", level='" + level + '\'' +
", type='" + type + '\'' +
'}';
}
}
The LevelList object is just a List of Level objects
The problem is that with Protostuff I get a bigger payload (26 kb) comparing to JSON (3.7kb). Why?
Also for second option I also tried setting "application/octet-stream" to return bytes directly but still the same result. Also I compared speed for both JSON and protobuf; protobuf has the better performance even with a bigger payload. Any idea why?
Protostuff and Protobuf are not the same thing. Protostuff is a wrapper library that can use many different serialization formats. It also supports runtime schema generation, which you appear to be using. That runtime schema requires sending extra metadata along with the message to tell the receiver about the message's schema. I would guess that the large message you're seeing is mostly from this runtime schema data.
With standard Protobuf, the schema is not sent with the message because it is assumed that the sender and recipient already agree on a schema provided by a .proto file compiled into both programs. If you use Protobuf with a standard .proto file, you'll find that the messages it produces are much smaller than JSON.
You have at least one problem in your test.
This transformation from byte array to String is not valid:
bytes = ProtostuffIOUtil.toByteArray(list, schema, buffer);
return new String(bytes);
This constructor of String will try to parse byte array as a UTF-8 string (most probably; depends on your locale settings), but given data by definition is not valid UTF-8 string.
If you want to make better size comparison, you should write a test in a following form:
LevelList source = testData();
byte[] jsonData = generateJson(source);
byte[] protobufData = generateProtobuf(source);
System.out.println("JSON=" + jsonData.size() + " Protobuf=" + protobufData.size());
Main point here is to make your test reproducible, so that other people can repeat it.
Related
I am trying to parsing a Soap response string to a JAVA object to get those parameters.
Here's the response string:
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
<S:Body>
<ns0:submitCdus1Response xmlns:ns0="http://example.com/">
<MessageId>D421425215</MessageId>
<NoOfDay>14</NoOfDay>
<Status>Y</Status>
<LastControlPoint>SBC</LastControlPoint>
<LastEntryDate>20210415</LastEntryDate>
<ReplyDateTime>20210427114126848</ReplyDateTime>
<TypeOfTravel>A</TypeOfTravel>
</ns0:submitCdus1Response>
</S:Body>
</S:Envelope>
My Object class:
#XmlAccessorType(XmlAccessType.FIELD)
#XmlRootElement(name="submitCdus1Response", namespace="http://example.com/" )
public class TravelHistoryResponseDTO implements Serializable {
/**
*
*/
private static final long serialVersionUID = 1L;
#XmlAttribute(name = "MessageId")
private String MessageId;
#XmlAttribute(name = "NoOfDay")
private Integer NoOfDay;
#XmlAttribute(name = "Status")
private String Status; //Y , N ,
#XmlAttribute(name = "LastControlPoint")
private String LastControlPoint;
#XmlAttribute(name = "TypeOfTravel")
private String TypeOfTravel;// A
#XmlAttribute(name = "LastEntryDate")
private Date LastEntryDate;
#XmlAttribute(name = "ReplyDateTime")
private Date ReplyDateTime;
public String getMessageId() {
return MessageId;
}
public void setMessageId(String messageId) {
MessageId = messageId;
}
public Integer getNoOfDay() {
return NoOfDay;
}
public void setNoOfDay(Integer noOfDay) {
NoOfDay = noOfDay;
}
public String getStatus() {
return Status;
}
public void setStatus(String status) {
Status = status;
}
public String getLastControlPoint() {
return LastControlPoint;
}
public void setLastControlPoint(String lastControlPoint) {
LastControlPoint = lastControlPoint;
}
public String getTypeOfTravel() {
return TypeOfTravel;
}
public void setTypeOfTravel(String typeOfTravel) {
TypeOfTravel = typeOfTravel;
}
public Date getLastEntryDate() {
return LastEntryDate;
}
public void setLastEntryDate(Date lastEntryDate) {
LastEntryDate = lastEntryDate;
}
public Date getReplyDateTime() {
return ReplyDateTime;
}
public void setReplyDateTime(Date replyDateTime) {
ReplyDateTime = replyDateTime;
}
#Override
public String toString() {
return "TravelHistoryResponseDTO [MessageId=" + MessageId + ", NoOfDay=" + NoOfDay + ", Status=" + Status
+ ", LastControlPoint=" + LastControlPoint + ", TypeOfTravel=" + TypeOfTravel + ", LastEntryDate="
+ LastEntryDate + ", ReplyDateTime=" + ReplyDateTime + "]";
}
}
My code:
SOAPMessage message = MessageFactory.newInstance().createMessage(null,
new ByteArrayInputStream(responseString.getBytes()));
Unmarshaller unmarshaller = JAXBContext.newInstance(TravelHistoryResponseDTO.class).createUnmarshaller();
TravelHistoryResponseDTO dto = (TravelHistoryResponseDTO)unmarshaller.unmarshal(message.getSOAPBody().extractContentAsDocument());
But I am getting
javax.xml.bind.UnmarshalException: unexpected element (uri:http://example.com/", local:"submitCdus1Response"). Expected elements are <{}submitCdus1Response>
Anyway to solve this and able to map the parameters to the java object?
There is an error in the JAXB object mapping : you are using #XmlAttribute for the various fields of your object, but the XML data you are trying to read are not XML attributes (<ns0:submitCdus1Response MessageId="xxx">) but XML elements (<MessageId>xxx</MessageId>).
Therefore all you have to is switch from #XmlAttribute to #XmlElement to match the mapping object to your actual XML.
As a note : if this comes really out of a SOAP call, chances are that you have a XSD Schema somewhere that defines the XML structure that is used here. JAXB comes with a code generation tool that will create the JAXB Objects (e.g. TravelHistoryResponseDTO) for you, deriving the data from the XSD. The tool is called xjc (you can google / search for it), it comes with the JDK and or can be found in maven plugins, (and more). This is the kind of mistake the tool would have prevented.
I am trying to get Avro Schema at runtime with the following method:
private Schema getSchema(Class clazz) {
Schema s = ReflectData.get().getSchema(clazz);
AvroSchema avroSchema = new AvroSchema(s);
return avroSchema.getAvroSchema();
}
But since my POJO class contains generics as below:
public abstract class Data<T> implements Serializable {
private static final long serialVersionUID = 1L;
private String dataType;
private T id;
public Data() {
}
public Data(String dataType) {
this.dataType = dataType;
}
public Data(String dataType, T id) {
this.dataType = dataType;
this.id = id;
}
}
I get the following exception:
Exception in thread "main" org.apache.avro.AvroRuntimeException: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroTypeException: Unknown type: T
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:227)
I understand that Avro will not support generic Types. Is there a way I can omit certain class fields from my class during schema generation at runtime?
private <T> String writePojoToParquet(List<T> pojos, String fileKey){
String fileName = fileKey + ".parquet";
Path path = new Path(fileName.replace("/", "_"));
//No matter what delete file always.
String strPath = path.toString();
FileUtils.delete(strPath);
FileUtils.delete(strPath + ".crc");
logger.debug("Writing data to parquet file {}", strPath);
Configuration conf = new Configuration();
try (ParquetWriter<T> writer =
AvroParquetWriter.<T>builder(path)
.withSchema(ReflectData.AllowNull.get().getSchema(pojos.get(0).getClass()))
.withDataModel(ReflectData.get())
.withConf(conf)
.withCompressionCodec(CompressionCodecName.SNAPPY)
.withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
.enableValidation()
.enableDictionaryEncoding()
.build()) {
for (T p : pojos) {
writer.write(p);
}
return strPath;
} catch (IOException e) {
logger.error("Error while writing data to parquet file {}.", strPath, e);
}
return null;
}
I declared a custom class in my project:
public class LocationData {
private Location location;
private LocalDateTime localDateTime;
private int heartRate;
private int calories;
private int ropeMoves;
private int dumbbellMoves;
private int pedalRotations;
private int wheelRotations;
private int numberOfSteps;
public LocationData(Location location, LocalDateTime localDateTime, int heartRate, int calories, int ropeMoves, int dumbbellMoves, int pedalRotation, int wheelRotations, int numberOfSteps) {
this.location = location;
this.localDateTime = localDateTime;
this.heartRate = heartRate;
this.calories = calories;
this.ropeMoves = ropeMoves;
this.dumbbellMoves = dumbbellMoves;
this.pedalRotations = pedalRotations;
this.wheelRotations = wheelRotations;
this.numberOfSteps = numberOfSteps;
}
public Location getLocation() {
return location;
}
public LocalDateTime getLocalDateTime() {
return localDateTime;
}
// Other getters
#Override
public String toString() {
return "LocationData[" +
"\n\tlocation=" + location +
"\n\tlocalDateTime=" + localDateTime +
"\n\tcalories=" + calories +
"\n\theartRate=" + heartRate +
"\n\tropeMoves=" + ropeMoves +
"\n\tdumbbellMoves=" + dumbbellMoves +
"\n\tpedalRotations=" + pedalRotations +
"\n\twheelRotations=" + wheelRotations +
"\n\tnumberOfSteps=" + numberOfSteps +
"]";
}
}
It represents a location plus some infos. Then I save a List of LocationData to create a "route".
I need to save this List (called location) to a file because the user, in the future, will ask to retrieve it to create a GPX file.
I think that the best solution is make the LocationData class serializable, but I don't know how to serialize (then deserialize) it. So... I need to understand how:
Serialize LocationData class
Deserialize LocationData class
Create a List of serialized LocationData
Write the list in a file
Read the list from a file
You need to add implements Serializable to the class definition, but you don't have to implement Serializable's methods - Object already has a default implementation, and the Serializable interface is used as a declarative.
You also need to ensure that Location and LocalDateTime are both Serializable.
Finally, once that's all in place you can use ObjectInputStream / ObjectOutputStream to read and write your objects
I try to deserialize object that contains null-properties and have the JsonMappingException.
What I do:
String actual = "{\"#class\" : \"PersonResponse\"," +
" \"id\" : \"PersonResponse\"," +
" \"result\" : \"Ok\"," +
" \"message\" : \"Send new person object to the client\"," +
" \"person\" : {" +
" \"id\" : 51," +
" \"firstName\" : null}}";
ObjectMapper mapper = new ObjectMapper();
mapper.readValue(new StringReader(json), PersonResponse.class); //EXCEPTION!
BUT: if to throw away "firstName = null" property - all works fine!
I mean pass the next string:
String test = "{\"#class\" : \"PersonResponse\"," +
" \"id\" : \"PersonResponse\"," +
" \"result\" : \"Ok\"," +
" \"message\" : \"Send new person object to the client\"," +
" \"person\" : {" +
" \"id\" : 51}}";
ObjectMapper mapper = new ObjectMapper();
mapper.readValue(new StringReader(json), PersonResponse.class); //ALL WORKS FINE!
Question:
How to avoid this exception or to pledge Jackson ignore null-values during serialization?
Throws:
Message:
com.fasterxml.jackson.databind.MessageJsonException:
com.fasterxml.jackson.databind.JsonMappingException:
N/A (through reference chain: person.Create["person"]->Person["firstName"])
cause:
com.fasterxml.jackson.databind.MessageJsonException:
com.fasterxml.jackson.databind.JsonMappingException:
N/A (through reference chain: prson.Create["person"]->Person["firstName"])
cause: java.lang.NullPointerException
Sometimes this problem occurs when accidentally using a primitive type as return type of the getter of a non-primitive field:
public class Item
{
private Float value;
public float getValue()
{
return value;
}
public void setValue(Float value)
{
this.value = value;
}
}
Notice the "float" instead of "Float" for the getValue()-method, this can lead to a Null Pointer Exception, even when you have added
objectMapper.setSerializationInclusion(Include.NON_NULL);
If you don't want to serialize null values, you can use the following setting (during serialization):
objectMapper.setSerializationInclusion(Include.NON_NULL);
Hope this solves your problem.
But the NullPointerException you get during deserialization seems suspicious to me (Jackson should ideally be able to handle null values in the serialized output). Could you post the code corresponding to the PersonResponse class?
I also faced the same issue.
I just included a default constructor in the model class along with the other constructor with parameters.
It worked.
package objmodel;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
public class CarModel {
private String company;
private String model;
private String color;
private String power;
public CarModel() {
}
public CarModel(String company, String model, String color, String power) {
this.company = company;
this.model = model;
this.color = color;
this.power = power;
}
#JsonDeserialize
public String getCompany() {
return company;
}
public void setCompany(String company) {
this.company = company;
}
#JsonDeserialize
public String getModel() {
return model;
}
public void setModel(String model) {
this.model = model;
}
#JsonDeserialize
public String getColor() {
return color;
}
public void setColor(String color) {
this.color = color;
}
#JsonDeserialize
public String getPower() {
return power;
}
public void setPower(String power) {
this.power = power;
}
}
Add JsonProperty annotation to your attribute in TO class, as below
#JsonProperty
private String id;
HI, i am trying to use the above spring hibernate temnplate method do a simple query based on a specific ID from the database but the problem is that the query doesnt replace the ":" character from the sql string below into the value contained in "id".
i thought that this method replaces ":" with the given parameter i set in the method bit it doesnt?
Code is below:
private static final String SQL_GET_FILE = "select new FileObject(filename, size, id, type, file) from FileObject where id = : limit 1";
FileObject file = (FileObject) hbTemplate.findByNamedParam(SQL_GET_FILE, "id", id);
//File object POJO:
package com.kc.models;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.sql.Blob;
import java.sql.SQLException;
import org.hibernate.Hibernate;
public class FileObject {
private String filename;
private String type;
private double size;
private Blob file;
private int id;
public FileObject() {
}
public FileObject(String name, double size, int id, String type) {
this.filename = name;
this.type = type;
this.size = size;
this.id = id;
}
public FileObject(String name, double size, int id, String type, Blob file) {
this.filename = name;
this.type = type;
this.size = size;
this.id = id;
this.file = file;
}
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getFilename() {
return filename;
}
public void setFilename(String fileName) {
this.filename = fileName;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public double getSize() {
return size;
}
public void setSize(double size) {
this.size = size;
}
public Blob getFile() {
return file;
}
public void setFile(Blob file) {
this.file = file;
}
}
The exception i get is basically this:
org.hibernate.hql.ast.QuerySyntaxException: unexpected token: 1 near line 1, column 104 [select new FileObject(filename, size, id, type, file) from com.kc.models.FileObject where id = : limit 1]
org.hibernate.hql.ast.QuerySyntaxException.convert(QuerySyntaxException.java:31)
org.hibernate.hql.ast.QuerySyntaxException.convert(QuerySyntaxException.java:24)
org.hibernate.hql.ast.ErrorCounter.throwQueryException(ErrorCounter.java:59)
org.hibernate.hql.ast.QueryTranslatorImpl.parse(QueryTranslatorImpl.java:258)
org.hibernate.hql.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:157)
org.hibernate.hql.ast.QueryTranslatorImpl.compile(QueryTranslatorImpl.java:111)
org.hibernate.engine.query.HQLQueryPlan.<init>(HQLQueryPlan.java:77)
org.hibernate.engine.query.HQLQueryPlan.<init>(HQLQueryPlan.java:56)
org.hibernate.engine.query.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:72)
org.hibernate.impl.AbstractSessionImpl.getHQLQueryPlan(AbstractSessionImpl.java:133)
org.hibernate.impl.AbstractSessionImpl.createQuery(AbstractSessionImpl.java:112)
org.hibernate.impl.SessionImpl.createQuery(SessionImpl.java:1623)
org.springframework.orm.hibernate3.HibernateTemplate$31.doInHibernate(HibernateTemplate.java:949)
org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
org.springframework.orm.hibernate3.HibernateTemplate.findByNamedParam(HibernateTemplate.java:947)
org.springframework.orm.hibernate3.HibernateTemplate.findByNamedParam(HibernateTemplate.java:938)
com.kc.models.DbFileHelper.getFile(DbFileHelper.java:81)
com.kc.models.FileHelper.getFileFromDb(FileHelper.java:195)
com.kc.Controllers.DownloadAppController.handle(DownloadAppController.java:48)
org.springframework.web.servlet.mvc.AbstractCommandController.handleRequestInternal(AbstractCommandController.java:84)
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:501)
javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
Now i have temporarily done a quick fix by simply doing the above
private static final String SQL_GET_FILE = "select new FileObject(filename, size, id, type, file) from FileObject where id = ";
List<FileObject> file = hbTemplate.find(SQL_GET_FILE+id);
But i dont like the idea of joining a query string with +.
it would get tedius if i have a sql looking something like this:
SQL_GET_FILE = "select new FileObject(filename, size, id, type, file)
from FileObject where id = 10 && size < 1000 && type = jpg";
cheers in advance
You should give the parameter a name, not just a colon:
where id = :id
Also, don't use LIMIT - use template.setMaxResults()
Actually, I would advise for using the hibernate Session directly - the HibernateTemplate is something that the guys at Hibernate criticize a lot - see here the comments by Gaving King.
You can still use HibernateTemplate, but for features (like setFirstResult(..)) you can use the Session.
Finally, I think using EntityManager is the best choice. Spring offers very good JPA integration as well.