Develop Reference

unable to parse CSV with `CsvParser` abstract class mapping but works with JSON mapper

Unable to parse CSV files to Object type,
CsvMapper not working with polymorphism, while JSON mapper works with jsonString.
#Getter
#JsonTypeInfo(use = JsonTypeInfo.Id.NAME,
property = "type", visible = true,
include = JsonTypeInfo.As.PROPERTY)
#JsonSubTypes({
#Type(value = FHR.class, name = "FHR"),
#Type(value = BHR.class, name = "BHR")})
public class PaymentBatchRecord {
protected String type;
}
#Getter
#Setter
#JsonPropertyOrder({
// "type",
"transmit_id",
"password",
"creation_date",
"creation_time",
"file_format_code",
"file_reference_code"
})
class FHR extends PaymentBatchRecord implements Serializable {
private final static long serialVersionUID = -584359005702082280L;
// #JsonProperty("type")
// private String type;
#JsonProperty("transmit_id")
private String transmitId;
#JsonProperty("password")
private String password;
#JsonProperty("creation_date")
private String creationDate;
#JsonProperty("creation_time")
private String creationTime;
#JsonProperty("file_format_code")
private String fileFormatCode;
#JsonProperty("file_reference_code")
private String fileReferenceCode;
}
#Setter
#Getter
#JsonPropertyOrder({
// "type",
"transaction_type",
"merchant_id",
"merchant_name",
"batch_entry_description",
"batch_reference_code",
"batch_number"
})
class BHR extends PaymentBatchRecord implements Serializable {
private final static long serialVersionUID = 1650905882208990490L;
// #JsonProperty("type")
// private String type;
#JsonProperty("transaction_type")
private String transactionType;
#JsonProperty("merchant_id")
private String merchantId;
#JsonProperty("merchant_name")
private String merchantName;
#JsonProperty("batch_entry_description")
private String batchEntryDescription;
#JsonProperty("batch_reference_code")
private String batchReferenceCode;
#JsonProperty("batch_number")
private Integer batchNumber;
}
and here is I'm trying to de-serialize
CsvMapper mapper = new CsvMapper();
// uncomment it to run but with all the null values
// mapper.enable(CsvParser.Feature.IGNORE_TRAILING_UNMAPPABLE)
;
CsvSchema sclema = mapper.schemaFor(PaymentBatchRecord.class)
.withoutHeader();
MappingIterator<PaymentBatchRecord> iterator = mapper
.readerFor(PaymentBatchRecord.class)
.with(sclema)
.readValues(in);
List<PaymentBatchRecord> ppojos = iterator.readAll();
and here is the sample csv input
FHR,BILLER_1,"biller1pwd","20200224","091503","CSV","202002240915031"
BHR,"PMT","BILLER_1","BILLER 1 NAME","UTILITY BILL",,1
Exception I got:
Exception in thread "main" com.fasterxml.jackson.dataformat.csv.CsvMappingException: Too many entries: expected at most 1 (value #1 (8 chars) "BILLER_1")
at [Source: (com.fasterxml.jackson.dataformat.csv.impl.UTF8Reader); line: 1, column: 5]
at com.fasterxml.jackson.dataformat.csv.CsvMappingException.from(CsvMappingException.java:28)
at com.fasterxml.jackson.dataformat.csv.CsvParser._reportCsvMappingError(CsvParser.java:1246)
at com.fasterxml.jackson.dataformat.csv.CsvParser._handleExtraColumn(CsvParser.java:1001)
at com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:862)
at com.fasterxml.jackson.dataformat.csv.CsvParser.nextToken(CsvParser.java:609)
at com.fasterxml.jackson.core.util.JsonParserSequence.switchAndReturnNext(JsonParserSequence.java:234)
at com.fasterxml.jackson.core.util.JsonParserSequence.nextToken(JsonParserSequence.java:152)
at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:861)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:295)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:189)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:161)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1196)
at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68)
at com.fasterxml.jackson.databind.MappingIterator.nextValue(MappingIterator.java:280)
at com.fasterxml.jackson.databind.MappingIterator.readAll(MappingIterator.java:320)
at com.fasterxml.jackson.databind.MappingIterator.readAll(MappingIterator.java:306)

Map String to Object using Jackson with inheritance

I have the QueueContent class that it has is a superclass of two others.
I get a String in JSON format that contains the information I need to extract. The super class is:
#Data
#JsonIgnoreProperties(ignoreUnknown = true)
public class QueueContent {
private String empresa;
private String empresa_cor;
private String empresa_contato;
private String empresa_url;
private String empresa_telefone;
private String empresa_idioma;
public QueueContent(String empresa, String empresa_cor, String empresa_contato, String empresa_url, String empresa_telefone, String empresa_idioma) {
this.empresa = empresa;
this.empresa_cor = empresa_cor;
this.empresa_contato = empresa_contato;
this.empresa_url = empresa_url;
this.empresa_telefone = empresa_telefone;
this.empresa_idioma = empresa_idioma;
}
public QueueContent() {
}
}
I'm using Lombok to generate Getters / Setters)
This is the child class:
#Data
public class EmailCameraOffline extends QueueContent {
private Timestamp camera_last_online;
private String camera_nome;
private String empresa_url_plataforma;
public EmailCameraOffline(String empresa, String empresa_cor, String empresa_contato, String empresa_url, String empresa_telefone, String empresa_idioma, Timestamp camera_last_online, String camera_nome, String empresa_url_plataforma) {
super(empresa, empresa_cor, empresa_contato, empresa_url, empresa_telefone, empresa_idioma);
this.camera_last_online = camera_last_online;
this.camera_nome = camera_nome;
this.empresa_url_plataforma = empresa_url_plataforma;
}
public EmailCameraOffline() {
}
}
So I've done:
EmailCameraOffline infosEmail = new ObjectMapper().readValue(content, EmailCameraOffline.class);
System.out.println(infosEmail);
And the output is:
EmailCameraOffline (camera_last_online = 2020-03-12 03: 01: 45.0, camera_nome = Pier Cam 1, empresa_url_platform = null)
How do I get my EmailCameraOffline object to have the superclass attributes initialized?

Everything should be loaded and initialized just fine, so calling:
System.out.println(infosEmail.getEmpresa());
should give expected value.
Problem
The problem is in the default implementation of toString() method (done via #Data) at EmailCameraOffline class, which does not include inherited fields.
Solution
To fix this you can "override" #Data's toString() implementation to include inherited fields as well using Lombok as:
#Data
#ToString(callSuper = true)
public class EmailCameraOffline extends QueueContent {
...
}

Using Lombok RequiredArgsConstructor as JsonCreator

I'd love to use this:
#Getter
#ToString
#RequiredArgsConstructor(onConstructor_ = {#JsonCreator(mode = JsonCreator.Mode.PROPERTIES)})
private static class RestErrorObject {
private final String error; // optional
private final String message; // optional
private final String path; // optional
private final String status; // optional
private final String timestamp; // optional
}
But instead, I have to use this:
#Getter
#ToString
private static class RestErrorObject {
private final String error; // optional
private final String message; // optional
private final String path; // optional
private final String status; // optional
private final String timestamp; // optional
#JsonCreator
RestErrorObject(#JsonProperty("error") String error, #JsonProperty("message") String message,
#JsonProperty("path") String path, #JsonProperty("status") String status,
#JsonProperty("timestamp") String timestamp) {
this.error = error;
this.message = message;
this.path = path;
this.status = status;
this.timestamp = timestamp;
}
}
Is there any way I can use Lombok's RequiredArgsConstructor annotation with Jackson's JsonCreator? The problem appears to be the age-old Jackson requirement that each parameter in a multi-arg constructor be annotated with #JsonProperty. I understand this is a Java thing (or at least a Java 8 thing) that Jackson can't determine the argument names via reflection so the annotations must exist so Jackson knows where to pass each field from the json into the constructor. It just seems sadly redundant.

I had the same issue that you, found the solution here
https://projectlombok.org/features/constructor
To put annotations on the generated constructor, you can use
onConstructor=#__({#AnnotationsHere}), but be careful; this is an
experimental feature. For more details see the documentation on the
onX feature.
#Getter
#ToString
#RequiredArgsConstructor(onConstructor=#__(#JsonCreator(mode = JsonCreator.Mode.PROPERTIES)))
private static class RestErrorObject {
private final String error; // optional
private final String message; // optional
private final String path; // optional
private final String status; // optional
private final String timestamp; // optional
}
Even that I found no reference to this #__(...), I assume it converts the annotation to a constant for the compiler.

How to create Jackson XML POJO class for a JsonObject of JsonObjects

I'm trying to create POJOs for the following JSON structure. The Fields node is easy enough to wire up, but I'm unsure how to use annotations to wire up the Description node. If I had been defining the JSON structure for that node, I'd have create an JsonArray of JsonObjects, which would make the java class easy, but since I didn't, I need to figure out how to serialize the structure below:
{
"Fields": {
"Required": ["ftp.hostname"],
"Optional": ["ftp.rootDirectory"]
},
"Description": {
"ftp.hostname": {
"label": "SFTP Hostname",
"description": "SFTP server hostname or IP address"
},
"ftp.rootDirectory": {
"label": "Root Directory",
"description": "The root path on the Data Store accessible by this connector"
}
}
}
Note that the nodes in the Description object have names that correlate to the values defined in the Fields node, which means their node names can vary from payload to payload.
The class for the Fields node:
public class FieldDetails {
public static final String REQUIRED = "Required";
public static final String OPTIONAL = "Optional";
#JsonProperty(value = REQUIRED, required = true)
private List<String> required;
#JsonProperty(value = OPTIONAL, required = true)
private List<String> optional;
}
And what I have so far for the entire object:
public class FieldDefinitions {
public static final String FIELDS = "Fields";
public static final String DESCRIPTION = "Description";
#JsonProperty(value = FIELDS, required = true)
private FieldDetails fields;
#JsonProperty(value = DESCRIPTION , required = true)
private ??? descriptions;
}

Generally, you can always map any JSON object to Map<String, Object>. If JSON is complicated with many nested objects, Jackson will automatically pick correct type: Map for objects and List for arrays.
You can also declare class like below for Description properties.
class Description {
private String label;
private String description;
// getters, setters, toString
}
The whole Description is a big JSON which you can map to Map<String, Description>. So, it could look like below:
class FieldDefinitions {
public static final String FIELDS = "Fields";
public static final String DESCRIPTION = "Description";
#JsonProperty(value = FIELDS, required = true)
private FieldDetails fields;
#JsonProperty(value = DESCRIPTION, required = true)
private Map<String, Description> descriptions;
// getters, setters, toString
}
Rest is the same. Example app:
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.File;
import java.util.List;
import java.util.Map;
public class JsonApp {
public static void main(String[] args) throws Exception {
File json = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper mapper = new ObjectMapper();
FieldDefinitions fields = mapper.readValue(json, FieldDefinitions.class);
System.out.println("Required");
fields.getFields().getRequired().forEach(r ->
System.out.println(r + " = " + fields.getDescriptions().get(r)));
System.out.println("Optional");
fields.getFields().getOptional().forEach(r ->
System.out.println(r + " = " + fields.getDescriptions().get(r)));
}
}
For given JSON payload prints:
Required
ftp.hostname = Description{label='SFTP Hostname', description='SFTP server hostname or IP address'}
Optional
ftp.rootDirectory = Description{label='Root Directory', description='The root path on the Data Store accessible by this connector'}

That's the structure.
public class FieldDefinitions {
#JsonProperty("Fields")
public FieldDetails fields = new FieldDetails();
#JsonProperty("Description")
public Map<String, Property> properties = new HashMap<>();
}
public class FieldDetails {
#JsonProperty("Required")
public List<String> required = new ArrayList<>();
#JsonProperty("Optional")
public List<String> optional = new ArrayList<>();
}
public class Property {
public String label;
public String description;
}

About an exception org.springframework.core.convert.ConverterNotFoundException in spring-data-cassandra when i try to insert a row

i am quite newwbee with spring-data-cassandra and i am facing problems when i try to create one row within a cassandra table.
This is the exception when i try to run the test, setUp method is never executed:
org.springframework.core.convert.ConversionFailedException: **Failed to convert from type [java.util.HashSet<?>] to type [java.lang.String] for value '[unicon.matthews.entity.DataSync#79135a38[**
id=data_sync_id
orgId=identifier
tenantId=_tenand_id
syncDateTime=2017-09-25T13:35:14.153
syncType=all
syncStatus=fully_completed
]]'; nested exception is org.springframework.core.convert.ConverterNotFoundException: No converter found capable of converting from type [unicon.matthews.entity.DataSync] to type [java.lang.String]
...
Caused by: org.springframework.core.convert.ConverterNotFoundException: No converter found capable of converting from type [unicon.matthews.entity.DataSync] to type [java.lang.String]
at org.springframework.core.convert.support.GenericConversionService.handleConverterNotFound(GenericConversionService.java:324)
at org.springframework.core.convert.support.GenericConversionService.convert(GenericConversionService.java:206)
at org.springframework.core.convert.support.CollectionToStringConverter.convert(CollectionToStringConverter.java:71)
at org.springframework.core.convert.support.ConversionUtils.invokeConverter(ConversionUtils.java:37)
... 60 more
This is the test:
#RunWith(SpringRunner.class)
#SpringBootTest(classes = unicon.matthews.oneroster.service.repository.CassandraConfiguration.class)
public class CassandraOrgRepositoryTests {
final String _userName = UUID.randomUUID().toString();
final String _orgName = UUID.randomUUID().toString();
final String _sourceId = UUID.randomUUID().toString();
final String _id = UUID.randomUUID().toString();
final String _api_key = UUID.randomUUID().toString();
final String _api_secret = UUID.randomUUID().toString();
final String _tenant_id = "_tenand_id";
final Status _status = Status.inactive;
final OrgType _org_type = OrgType.school;
final String _org_identifier = UUID.randomUUID().toString();
#ClassRule public final static CassandraKeyspace CASSANDRA_KEYSPACE = CassandraKeyspace.onLocalhost().atLeast(Version.parse("3.0"));
#Autowired CassandraOrgRepository repository;
#Before
public void setUp() throws Exception {
repository.deleteAll();
OrgCassandraTable aPojo = new OrgCassandraTable();
aPojo.setTenantId(_tenant_id );
Org.Builder myOrgBuilder = Org.Builder.class.newInstance();
Map<String, String> metadata = new TreeMap<String,String>();
metadata.put("key","value");
Org myOrgPojo = myOrgBuilder.withIdentifier("identifier")
.withDateLastModified(LocalDateTime.now())
.withMetadata(metadata)
.withName(_orgName)
.withSourcedId(_sourceId)
.withStatus(_status)
.withType(_org_type)
.build();
aPojo.setSourcedId(_sourceId);
// active 0,
// inactive 1,
// tobedeleted 2;
aPojo.setStatus("1");
aPojo.setDateLastModified(LocalDateTime.now() );
aPojo.setName(_orgName);
aPojo.setType(_org_type.toString());
aPojo.setIdentifier(_org_identifier);
aPojo.setTenantId(_tenant_id);
// THIS MUST BE THE PROBLEM!
Set<DataSync> _dataSyncSet = new HashSet<DataSync>();
DataSync.Builder _dataSyncBuilder = DataSync.Builder.class.newInstance();
DataSync new_data_sync=_dataSyncBuilder.withId("data_sync_id")
.withOrgId(myOrgPojo.getIdentifier())
.withSyncDateTime(LocalDateTime.now())
.withSyncStatus(DataSync.DataSyncStatus.fully_completed)
.withSyncType(DataSync.DataSyncType.all)
.withTenantId(_tenant_id)
.build();
_dataSyncSet.add(new_data_sync);
aPojo.setDataSyncs(_dataSyncSet);
aPojo.setApiSecret(_api_secret);
aPojo.setApiKey(_api_key);
aPojo.setId(_id);
repository.save(aPojo);
assertTrue(repository.count() > 0);
System.out.println("Created a org with fake data...");
}
#Test
public void testFindbyId() {
Optional<WrapperOrg> loaded = repository.findById(_id);
Assert.assertNotNull(loaded);
Assert.assertEquals("something went wrong...",_id,loaded.get().getId());
}
}
This is the repository:
import java.util.Optional;
import org.springframework.data.cassandra.repository.CassandraRepository;
import org.springframework.data.cassandra.repository.Query;
// this repo must implement something that paginates rows, because ALLOW FILTERING must not be used
public interface CassandraOrgRepository extends CassandraRepository<OrgCassandraTable> {
#Query("SELECT * FROM org WHERE id = ?0")
Optional<WrapperOrg> findById(final String id);
#Query("SELECT * FROM org WHERE api_key = ?0 AND api_secret = ?1 ALLOW FILTERING")
Optional<WrapperOrg> findByApiKeyAndApiSecret(final String apiKey, final String apiSecret);
#Query("SELECT * FROM org WHERE api_key = ?0 ALLOW FILTERING")
Optional<WrapperOrg> findByApiKey(final String apiKey);
}
This is the CassandraConfiguration class that i mention in the test class. I suspect that i will have to do something here:
import java.util.ArrayList;
import java.util.List;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.cassandra.core.keyspace.CreateKeyspaceSpecification;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.cassandra.config.SchemaAction;
import org.springframework.data.cassandra.config.java.AbstractCassandraConfiguration;
import org.springframework.data.cassandra.repository.config.EnableCassandraRepositories;
#Configuration
#EnableAutoConfiguration
public class CassandraConfiguration {
#Configuration
#EnableCassandraRepositories
static class CassandraConfig extends AbstractCassandraConfiguration {
private static final String KEYSPACE = "example";
#Override
public String getKeyspaceName() {
return KEYSPACE;
}
#Override
public SchemaAction getSchemaAction() {
return SchemaAction.RECREATE_DROP_UNUSED;
}
protected List<CreateKeyspaceSpecification> getKeyspaceCreations() {
List<CreateKeyspaceSpecification> createKeyspaceSpecifications = new ArrayList<>();
createKeyspaceSpecifications.add(getKeySpaceSpecification());
return createKeyspaceSpecifications;
}
// Below method creates KEYSPACE if it doesnt exist.
private CreateKeyspaceSpecification getKeySpaceSpecification() {
CreateKeyspaceSpecification pandaCoopKeyspace = new CreateKeyspaceSpecification();
pandaCoopKeyspace.name(KEYSPACE);
pandaCoopKeyspace.ifNotExists(true)
.createKeyspace();
return pandaCoopKeyspace;
}
#Override
public String getContactPoints() {
return "localhost";
}
#Override
public String[] getEntityBasePackages() {
return new String[] {"unicon.matthews.oneroster.service.repository"};
}
}
}
This is the Entity pojo class:
import java.io.Serializable;
import java.time.LocalDateTime;
import java.util.Map;
import java.util.Set;
import org.springframework.cassandra.core.PrimaryKeyType;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.cassandra.mapping.CassandraType;
import org.springframework.data.cassandra.mapping.Column;
import org.springframework.data.cassandra.mapping.Indexed;
import org.springframework.data.cassandra.mapping.PrimaryKeyColumn;
import org.springframework.data.cassandra.mapping.Table;
import com.datastax.driver.core.DataType;
import unicon.matthews.entity.DataSync;
import unicon.matthews.oneroster.Org;
import unicon.matthews.oneroster.OrgType;
import unicon.matthews.oneroster.Status;
#Table(value=OrgCassandraTable.tableName)
public class OrgCassandraTable implements Serializable{
#org.springframework.data.annotation.Transient
public static final String tableName = "org";
#PrimaryKeyColumn(name = "id", ordinal = 0, type = PrimaryKeyType.PARTITIONED)
#CassandraType(type = DataType.Name.TEXT)
#Column("id")
private String id;
#Indexed
#CassandraType(type = DataType.Name.TEXT)
#Column("tenant_id")
private String tenantId;
#Indexed
#CassandraType(type = DataType.Name.TEXT)
#Column("api_key")
private String apiKey;
#Indexed
#CassandraType(type = DataType.Name.TEXT)
#Column("api_secret")
private String apiSecret;
#Indexed
#CassandraType(type = DataType.Name.TEXT)
#Column("org_source_id")
private String sourcedId;
#CassandraType(type = DataType.Name.TEXT)
#Column("org_status")
private String status;
#Column("org_metadata")
private Map<String, String> metadata;
#Column("org_dateLastModified")
#LastModifiedDate
private LocalDateTime dateLastModified;
#Column("org_name")
#CassandraType(type = DataType.Name.TEXT)
private String name;
// ojito que esto es un enum
#Column("org_type")
#CassandraType(type = DataType.Name.TEXT)
private String type;
#Column("org_identifier")
#CassandraType(type = DataType.Name.TEXT)
#Indexed
private String identifier;
// THIS FIELD LOOKS TO BE THE PROBLEM!
#Column("org_data_syncs")
#CassandraType(type = DataType.Name.TEXT)
private Set<DataSync> dataSyncs;
public OrgCassandraTable(){
}
This is DataSync class. It belongs to a third party library, i do not have the code. What do am i doing wrong?
public class DataSync implements Serializable {
private static final long serialVersionUID = 1L;
private String id;
private String orgId;
private String tenantId;
private LocalDateTime syncDateTime;
private DataSync.DataSyncType syncType;
private DataSync.DataSyncStatus syncStatus;
...getters, setters, equals, hashCode, toString methods
}
...
// getters, setters, hashCode, equals, toString methods.
}

Cassandra is a column-oriented store – Spring Data Cassandra maps each domain class to a single table, there are no relations, and there is no (not yet, but might come) support for embedded objects. Embedded objects in the sense of flattening the data structure to the columns of the table the enclosing object maps to.
However, there is support for user-defined types via #UserDefinedType on the object class representing the data structure. Adding #UserDefinedType requires having control over the class/code.
If you want to stick to the class, then you still have an option to serialize the data yourself, e.g., using Jackson and storing the JSON inside a single Cassandra column:
static class DataSyncWriteConverter implements Converter<DataSync, String> {
public String convert(DataSync source) {
try {
return new ObjectMapper().writeValueAsString(source);
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
}
You should be able to work with collection types as well, meaning, that you can persist a Set<DataSync> within a set<varchar> column in Cassandra with this approach.
One last thing: Using 3rd-party classes comes at the risk of changes to the external classes where you don't have control over. Creating an own data structure by replicating all fields and mapping the data to the 3rd-party-class give you control over the lifecycle of changes.
References:
Saving using a registered Spring Converter.