Develop Reference

Flink Collector issue when Collection Object with Map of Object class

I am facing a issue where when i collecting object from flink flatmap collector than i am not getting value collected correctly. I am getting object reference and its not giving me actual value.
dataStream.filter(new FilterFunction<GenericRecord>() {
#Override
public boolean filter(GenericRecord record) throws Exception {
if (record.get("user_id") != null) {
return true;
}
return false;
}
}).flatMap(new ProfileEventAggregateFlatMapFunction(aggConfig))
.map(new MapFunction<ProfileEventAggregateEmittedTuple, String>() {
#Override
public String map(
ProfileEventAggregateEmittedTuple profileEventAggregateEmittedTupleNew)
throws Exception {
String res=null;
try {
ObjectMapper mapper = new ObjectMapper();
mapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);
res= mapper.writeValueAsString(profileEventAggregateEmittedTupleNew);
} catch (Exception e) {
e.printStackTrace();
}
return res;
}
}).print();
public class ProfileEventAggregateFlatMapFunction extends
RichFlatMapFunction<GenericRecord, ProfileEventAggregateEmittedTuple> {
private final ProfileEventAggregateTupleEmitter aggregator;
ObjectMapper mapper = ObjectMapperPool.getInstance().get();
public ProfileEventAggregateFlatMapFunction(String config) throws IOException {
this.aggregator = new ProfileEventAggregateTupleEmitter(config);
}
#Override
public void flatMap(GenericRecord event,
Collector<ProfileEventAggregateEmittedTuple> collector) throws Exception {
try {
List<ProfileEventAggregateEmittedTuple> aggregateTuples = aggregator.runAggregates(event);
for (ProfileEventAggregateEmittedTuple tuple : aggregateTuples) {
collector.collect(tuple);
}
}}
Debug Results:
tuple that i am collecting in collector
tuple = {ProfileEventAggregateEmittedTuple#7880}
profileType = "userprofile"
key = "1152473"
businessType = "keyless"
name = "consumer"
aggregates = {ArrayList#7886} size = 1
0 = {ProfileEventAggregate#7888} "geo_id {geo_id=1} {keyless_select_destination_cnt=1, total_estimated_distance=12.5}"
entityType = "geo_id"
dimension = {LinkedHashMap#7891} size = 1
"geo_id" -> {Integer#7897} 1
key = "geo_id"
value = {Integer#7897} 1
metrics = {LinkedHashMap#7892} size = 2
"keyless_select_destination_cnt" -> {Long#7773} 1
key = "keyless_select_destination_cnt"
value = {Long#7773} 1
"total_estimated_distance" -> {Double#7904} 12.5
key = "total_estimated_distance"
value = {Double#7904} 12.5
This i get in my map function .map(new MapFunction<ProfileEventAggregateEmittedTuple, String>()
profileEventAggregateEmittedTuple = {ProfileEventAggregateEmittedTuple#7935}
profileType = "userprofile"
key = "1152473"
businessType = "keyless"
name = "consumer"
aggregates = {GenericData$Array#7948} size = 1
0 = {ProfileEventAggregate#7950} "geo_id {geo_id=java.lang.Object#863dce2} {keyless_select_destination_cnt=java.lang.Object#7cdb4bfc, total_estimated_distance=java.lang.Object#52e81f57}"
entityType = "geo_id"
dimension = {HashMap#7952} size = 1
"geo_id" -> {Object#7957}
key = "geo_id"
value = {Object#7957}
Class has no fields
metrics = {HashMap#7953} size = 2
"keyless_select_destination_cnt" -> {Object#7962}
key = "keyless_select_destination_cnt"
value = {Object#7962}
Class has no fields
"total_estimated_distance" -> {Object#7963}
Please help me to understand what is happening why i am not getting correct data.
public class ProfileEventAggregateEmittedTuple implements Cloneable, Serializable {
private String profileType;
private String key;
private String businessType;
private String name;
private List<ProfileEventAggregate> aggregates = new ArrayList<ProfileEventAggregate>();
private long startTime;
private long endTime;
public String getProfileType() {
return profileType;
}
public void setProfileType(String profileType) {
this.profileType = profileType;
}
public String getKey() {
return key;
}
public void setKey(String key) {
this.key = key;
}
public String getBusinessType() {
return businessType;
}
public void setBusinessType(String businessType) {
this.businessType = businessType;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public List<ProfileEventAggregate> getAggregates() {
return aggregates;
}
public void addAggregate(ProfileEventAggregate aggregate) {
this.aggregates.add(aggregate);
}
public void setAggregates(List<ProfileEventAggregate> aggregates) {
this.aggregates = aggregates;
}
public long getStartTime() {
return startTime;
}
public void setStartTime(long startTime) {
this.startTime = startTime;
}
public long getEndTime() {
return endTime;
}
public void setEndTime(long endTime) {
this.endTime = endTime;
}
#Override
public ProfileEventAggregateEmittedTuple clone() {
ProfileEventAggregateEmittedTuple clone = new ProfileEventAggregateEmittedTuple();
clone.setProfileType(this.profileType);
clone.setKey(this.key);
clone.setBusinessType(this.businessType);
clone.setName(this.name);
for (ProfileEventAggregate aggregate : this.aggregates) {
clone.addAggregate(aggregate.clone());
}
return clone;
}
public class ProfileEventAggregate implements Cloneable, Serializable {
private String entityType;
private Map<String, Object> dimension =new LinkedHashMap<String, Object>();
private Map<String, Object> metrics = new LinkedHashMap<String, Object>();
public Map<String, Object> getDimension() {
return dimension;
}
public void setDimension(Map<String, Object> dimension) {
this.dimension.putAll(dimension);
}
public void addDimension(String dimensionKey, Object dimensionValue) {
this.dimension.put(dimensionKey, dimensionValue);
}
public Map<String, Object> getMetrics() {
return metrics;
}
public void addMetric(String metricKey, Object metricValue) {
this.metrics.put(metricKey, metricValue);
}
public void setMetrics(Map<String, Object> metrics) {
this.metrics.putAll(metrics);
}
public String getEntityType() {
return entityType;
}
public void setEntityType(String entityType) {
this.entityType = entityType;
}
#Override
public ProfileEventAggregate clone() {
ProfileEventAggregate clone = new ProfileEventAggregate();
clone.setEntityType(this.entityType);
clone.getDimension().putAll(this.getDimension());
clone.getMetrics().putAll(this.metrics);
return clone;
}

When you don't enableObjectReuse, objects are copied with your configured serializer (seems to be Avro?).
In your case, you use Map<String, Object> where you cannot infer a plausible schema.
The easiest fix would be to enableObjectReuse. Else make sure your serializer matches your data. So you could add a unit test where you use AvroSerializer#copy and make sure your POJO is properly annotated if you want to stick with Avro reflect or even better go with a schema first approach, where you generate your Java POJO with a Avro schema and use specific Avro.
Let's discuss some alternatives:
Use GenericRecord. Instead of converting it to a Java type, directly access GenericRecord. This is usually the only way when the full record is flexible (e.g. your job takes any input and writes it out to S3).
Denormalize schema. Instead of having some class Event { int id; Map<String, Object> data; } you would use class EventInformation { int id; String predicate; Object value; }. You would need to group all information for processing. However, you will run into the same type issues with Avro.
Use wide-schema. Looking at the previous approach, if the different predicates are known beforehand, then you can use that to craft a wide-schema class Event { int id; Long predicate1; Integer predicate2; ... String predicateN; } where all oft he entries are nullable and most of them are indeed null. To encode null is very cheap.
Ditch Avro. Avro is fully typed. You may want to use something more dynamic. Protobuf has Any to support arbitrary submessages.
Use Kryo. Kryo can serialize arbitrary object trees at the cost of being slower and having more overhead.
If you want to write the data, you also need to think about a solution where the type information is added for proper deserialization. For an example, check out this JSON question. But there are more ways to implement it.

How to create String Format template to generate Json for Eclipse autogenerated toString

Eclipse gave option to auto generate the toString method for every class.
Further leverage this facility, I am creating String Format Template to give as Json format when eclipse generate toString Method.
I used following String Format Template:
{ ${member.name()}:"${member.value}", ${otherMembers}}
now i generated toString method as following POJO but When i run this program i got result as and not a VALID JSON.
{ name:"null", reportees:"[1, 2, 3]", department:"[retail, banking, finance]", owns:"null", supplimentary:"null}
Code
public class TestPojo {
private String name;
private List<String> reportees;
private String[] department;
private Machine owns;
private List<Machine> supplimentary;
public static void main(String arg[]) {
TestPojo aTestPojo = new TestPojo();
aTestPojo.department = new String[] { "retail", "banking", "finance" };
aTestPojo.reportees = new ArrayList<String>() {
{
add("1");
add("2");
add("3");
}
};
System.out.print(aTestPojo);
}
public static class Machine {
private String machineName;
private String duties;
public String getMachineName() {
return machineName;
}
public void setMachineName(String machineName) {
this.machineName = machineName;
}
public String getDuties() {
return duties;
}
public void setDuties(String duties) {
this.duties = duties;
}
#Override
public String toString() {
StringBuilder builder = new StringBuilder();
builder.append("{ machineName:\"").append(machineName).append("\", duties:\"").append(duties).append("}");
return builder.toString();
}
}
#Override
public String toString() {
StringBuilder builder = new StringBuilder();
builder.append("{ name:\"").append(name).append("\", reportees:\"").append(reportees).append("\", department:\"").append(Arrays.toString(department)).append("\", owns:\"").append(owns).append("\", supplimentary:\"").append(supplimentary).append("}");
return builder.toString();
}
}

With the help of #dvlcube idea. I built a "Eclipse Custom toString() builder" to generate toString method code to return a JSON formatted string of current object.
Follow the github for this solution Click [Eclipse toString_Builder for JSON](https://github.com/djaganathan/EclipseToStringBuilderForJson/blob/master/src/main/java/com/github/djaganathan/opensource/eclipse/util/JsonToStringBuilder.java,"Custom Eclipse toString() Builder")
Sample Testing Code
import com.google.common.collect.Maps;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.lang3.time.StopWatch;
import org.elasticsearch.common.collect.Lists;
public class TestPojo {
private String name;
private List<String> reportees;
private String[] department;
private Machine owns;
private List<Machine> supplimentary;
private int i = 10;
private Map<String, Machine> machineList = Maps.newConcurrentMap();
public static void main(String arg[]) {
TestPojo aTestPojo = new TestPojo();
aTestPojo.department = new String[] { "retail", "banking", "finance"};
aTestPojo.reportees = new ArrayList<String>() {
{
add("1");
add("2");
add("3");
}
};
Machine aMachine = new Machine("Train", "travel");
Machine aMachine1 = new Machine("Lorry", "Carrier");
aTestPojo.supplimentary = Lists.newArrayList(aMachine, aMachine1);
aTestPojo.machineList.put("Train", aMachine);
aTestPojo.machineList.put("Lorry", aMachine1);
System.out.print(aTestPojo);
}
public static class Machine {
private String machineName;
private String duties;
public Machine(String machineName, String duties) {
super();
this.machineName = machineName;
this.duties = duties;
}
public String getMachineName() {
return machineName;
}
public void setMachineName(String machineName) {
this.machineName = machineName;
}
public String getDuties() {
return duties;
}
public void setDuties(String duties) {
this.duties = duties;
}
#Override
public String toString() {
JsonToStringBuilder builder = new JsonToStringBuilder(this);
builder.append("machineName", machineName);
builder.append("duties", duties);
return builder.build();
}
}
#Override
public String toString() {
JsonToStringBuilder builder = new JsonToStringBuilder(this);
builder.append("name", name);
builder.append("reportees", reportees);
builder.append("department", department);
builder.append("owns", owns);
builder.append("supplimentary", supplimentary);
builder.append("i", i);
builder.append("machineList", machineList);
String value = builder.build();
return value;
}
}
While running this program i got the following JSON output
{"name": null,"reportees": ["1","2","3"],"department": ["retail","banking","finance"],"owns": null,"supplimentary": [{"machineName": "Train","duties": "travel"},{"machineName": "Lorry","duties": "Carrier"}],"i": 10,"machineList": {"Lorry": {"machineName": "Lorry","duties": "Carrier"},"Train": {"machineName": "Train","duties": "travel"}}}

It's not valid JSON because of the way arrays and collections are being printed ("[1,2,3]" instead of ["1","2","3"]).
Also, it wouldn't pass strict JSON validation because the field names should be quoted as well.
Eclipse's String Format Template can be very useful, but for full control it's better to create a builder class.
Here's a gist for doing just that. You can expand on it, and it works for your example class out of the box.
You can use this class to generate the toString() methods in Eclipse.

Recommended only for String values. Below expression gives you as a JSON toString generation.
{"${member.name()}":"${member.value}", "${otherMembers}"}

JAX-WS For Json request and response [duplicate]

Is it possible that a jax-ws soap-webservice can output json format instead of xml?
#Component
#WebService
public class HRSService {
#WebMethod
public String test(String value) {
return value; //returned as XML. JSON possible?
}
}

Apparently it's possible by following the instructions indicated at https://jax-ws-commons.java.net/json/ (Archive version)
Summing up:
#BindingType(JSONBindingID.JSON_BINDING)
public class MyService {
public Book get(#WebParam(name="id") int id) {
Book b = new Book();
b.id = id;
return b;
}
public static final class Book {
public int id = 1;
public String title = "Java";
}
}
You just need jaxws-json.jar in your WEB-INF/lib for this to work.
I hope it helps!

This is late. I recently returned to programming in Java, but for those who will be visiting this page in the future. The example in the JAXWS metro documents works only in javascript. I used the following together with JSONObject:
#WebServiceProvider
#ServiceMode(value = Service.Mode.MESSAGE)
#BindingType(value=HTTPBinding.HTTP_BINDING)
then implement Provider(DataSource), as in example:
public class clazz implements Provider<DataSource>
{ ...
#Override
public DataSource invoke(DataSource arg)
{
...
String emsg = "Request Parameter Error.";
String sret = create_error_response(emsg);
return getDataSource(sret);
}
}
private DataSource getDataSource(String sret)
{
ByteArrayDataSource ds = new ByteArrayDataSource(sret.getBytes(), "application/json");
return ds;
}
public String create_error_response(String msg)
{
JSONObject json = new JSONObject();
json.put("success", new Boolean(false));
json.put("message", msg);
return json.toString();
}

JSON output of a JAX-WS webservice?

Is it possible that a jax-ws soap-webservice can output json format instead of xml?
#Component
#WebService
public class HRSService {
#WebMethod
public String test(String value) {
return value; //returned as XML. JSON possible?
}
}

Apparently it's possible by following the instructions indicated at https://jax-ws-commons.java.net/json/ (Archive version)
Summing up:
#BindingType(JSONBindingID.JSON_BINDING)
public class MyService {
public Book get(#WebParam(name="id") int id) {
Book b = new Book();
b.id = id;
return b;
}
public static final class Book {
public int id = 1;
public String title = "Java";
}
}
You just need jaxws-json.jar in your WEB-INF/lib for this to work.
I hope it helps!

This is late. I recently returned to programming in Java, but for those who will be visiting this page in the future. The example in the JAXWS metro documents works only in javascript. I used the following together with JSONObject:
#WebServiceProvider
#ServiceMode(value = Service.Mode.MESSAGE)
#BindingType(value=HTTPBinding.HTTP_BINDING)
then implement Provider(DataSource), as in example:
public class clazz implements Provider<DataSource>
{ ...
#Override
public DataSource invoke(DataSource arg)
{
...
String emsg = "Request Parameter Error.";
String sret = create_error_response(emsg);
return getDataSource(sret);
}
}
private DataSource getDataSource(String sret)
{
ByteArrayDataSource ds = new ByteArrayDataSource(sret.getBytes(), "application/json");
return ds;
}
public String create_error_response(String msg)
{
JSONObject json = new JSONObject();
json.put("success", new Boolean(false));
json.put("message", msg);
return json.toString();
}

BeanWriter Disable outputting empty lists

I have the following bean:
public class ContractBean {
private List<String> listNd;
private String nd;
public List<String> getListNd() {
return listNd;
}
public void setListNd(final List<String> listNd) {
this.listNd = listNd;
}
public String getNd() {
return nd;
}
public void setNd(final String nd) {
this.nd= nd;
}
}
I use apache Betwixt to output XML from my bean.
final BeanWriter beanWriter = new BeanWriter(outputWriter);
beanWriter.getXMLIntrospector().getConfiguration().setAttributesForPrimitives(false);
beanWriter.getBindingConfiguration().setMapIDs(false);
beanWriter.enablePrettyPrint();
beanWriter.setWriteEmptyElements(false);
beanWriter.getBindingConfiguration().setObjectStringConverter(new CustomObjectStringConverter());
beanWriter.write(obj);
The listND attribute of my bean is null, but i still get:
<contract>
<listNd/>
<nd>22222</nd>
</contract>
How can I remove empty lists from the output XML ?