Develop Reference

Avro Schema is being generated in wrong field order (alphabetical)

I have a class the I need to converto to an avro schema:
public class MonitorStateSchema {
private MetaResponse c;
private Header a;
private MonitorStateEnvelope b;
public MonitorStateSchema(MetaResponse c, Header a, MonitorStateEnvelope b) {
this.c = c;
this.a = a;
this.b = b;
}
}
I use a generic method to get the schema from it
public static <D> void getFromAvro(Class<D> schemaType) {
Schema schema = ReflectData.get().getSchema(schemaType);
// other stuff
}
After doing it, I got a different order in the result than the expected:
EXPECTED:
{
"type": "record",
"name": "MonitorSchema",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "c",
"type": {
"type": "record",
"name": "MetaResponse",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "uuid",
"type": "string"
},
{
"name": "code",
"type": "int"
},
{
"name": "message",
"type": "string"
}
]
}
},
{
"name": "a",
"type": {
"type": "record",
"name": "Header",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "apiKey",
"type": "string"
},
{
"name": "signature",
"type": "string"
},
{
"name": "nonce",
"type": "int"
}
]
}
},
{
"name": "b",
"type": {
"type": "record",
"name": "MonitorEnvelope",
"fields": [
{
"name": "fields",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "Field",
"fields": [
{
"name": "name",
"type": "string"
},
{
"name": "value",
"type": "string"
}
]
},
"java-class": "[Lmypackage.monitor.Field;"
}
}
]
}
}
]
}
ACTUAL RESULT:
{
"type": "record",
"name": "MonitorStateSchema",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "a",
"type": {
"type": "record",
"name": "Header",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "apiKey",
"type": "string"
},
{
"name": "nonce",
"type": "int"
},
{
"name": "signature",
"type": "string"
}
]
}
},
{
"name": "b",
"type": {
"type": "record",
"name": "MonitorStateEnvelope",
"fields": [
{
"name": "fields",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "Field",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "name",
"type": "string"
},
{
"name": "value",
"type": "string"
}
]
},
"java-class": "[Lmypackage.monitor.Field;"
}
}
]
}
},
{
"name": "c",
"type": {
"type": "record",
"name": "MetaResponse",
"namespace": "mypackage.monitor",
"fields": [
{
"name": "code",
"type": "int"
},
{
"name": "message",
"type": "string"
},
{
"name": "uuid",
"type": "string"
}
]
}
}
]
}
Seems that it is ordering by alphabetical order on the name of the field and it is breaking the application when deserializing the byte array. Is there any reason for this to happen?

Avro version 1.10.0 changed ReflectData to sort fields into alphabetical order because the Class.getDeclaredFields() method API specification does not mandate a specific order of the fields that get returned. Some Java implementations return the fields in the order they appear in the Java source file. Some people complained that was a breaking change.
If you don't want the fields sorted into alphabetical order, then you must stay at Avro version 1.9.2 or lower.

Consumer reads data twice from avro schema

I have a streaming app where it listens to some data and then transforming the data by pushing the data into a new topic. I use avro schema for both to read/write my data into topics. The problem is when i consume the data from the final destination by using the command in the below. However, my data is a little complex with some array and json inside of it and i suspect that my avro schemas might not be correct for my purpose. There is no error or anything, I can see all my data on my final topic but the "Pets" field are duplicated for some reason and i can't understand why. In fact, i only add one new field (job_id) to my existing data in the avro schema, i don't make big changes on it when i transform it.
./bin/kafka-console-consumer --topic my_topic \
--bootstrap-server localhost:9092 \
Here's the json data i have
{
"Person":{
"id":"104440",
"Name":"William",
"LastName":"Dorsey",
"archived":false,
"Timezone":"America/Los_Angeles",
"brandCompanyName":"Twitter",
"brandID":"cf545a7b",
"creatorID":"1234",
"currency":"USD",
"dateCreated":"2020-09-07T02:56:22Z",
"dateModified":"2020-09-07T02:57:24Z",
"disabled":false,
"endDate":"2020-11-29T19:51:00-08:00",
"startDate":"2020-08-31T20:55:00-07:00",
"totalBudget":0
},
"Pets":[
{
"Name":"Pawny",
"Id":"4214",
"budget":"0",
"adoptionDate":"2020-09-07T02:56:22Z",
"year":"2",
"type":"Golden",
"gender":"male"
}
],
"CreationTime":"1604036638"
}
my avro schema
{
"name": "MyClass",
"type": "record",
"namespace": "com.acme.avro",
"fields": [
{
"name": "Person",
"type": {
"name": "Person",
"type": "record",
"fields": [
{
"name": "id",
"type": "string"
},
{
"name": "Name",
"type": "string"
},
{
"name": "LastName",
"type": "string"
},
{
"name": "archived",
"type": "boolean"
},
{
"name": "Timezone",
"type": "string"
},
{
"name": "brandCompanyName",
"type": "string"
},
{
"name": "brandID",
"type": "string"
},
{
"name": "creatorID",
"type": "string"
},
{
"name": "currency",
"type": "string"
},
{
"name": "dateCreated",
"type": "int",
"logicalType": "date"
},
{
"name": "dateModified",
"type": "int",
"logicalType": "date"
},
{
"name": "disabled",
"type": "boolean"
},
{
"name": "endDate",
"type": "int",
"logicalType": "date"
},
{
"name": "startDate",
"type": "int",
"logicalType": "date"
},
{
"name": "totalBudget",
"type": "int"
}
]
}
},
{
"name": "Pets",
"type": {
"type": "array",
"items": {
"name": "Pets_record",
"type": "record",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "Id",
"type": "string"
},
{
"name": "budget",
"type": "string"
},
{
"name": "adoptionDate",
"type": "int",
"logicalType": "date"
},
{
"name": "year",
"type": "string"
},
{
"name": "type",
"type": "string"
},
{
"name": "gender",
"type": "string"
}
]
}
}
},
{
"name": "CreationTime",
"type": "string"
},
{
"name":"jobID",
"type":"string"
}
]
}
the output in my topic when i consume the topic - the pets field are duplicated for some reason? I can't figure out why
{
"id":"104440",
"Name":"William",
"LastName:"Dorsey",
"archived":false,
"Timezone":"America/Los_Angeles",
"brandCompanyName":"Twitter",
"brandID":"cf545a7b",
"creatorID":"1234",
"currency":"USD",
"dateCreated":"2020-09-07T02:56:22Z",
"dateModified":"2020-09-07T02:57:24Z",
"disabled":false,
"endDate":"2020-11-29T19:51:00-08:00",
"startDate":"2020-08-31T20:55:00-07:00",
"totalBudget":0,
"Pets":[
{
"Name":"Pawny",
"Id":"4214",
"budget":"0",
"adoptionDate":2020-09-07T02:56:22Z",
"year":"2",
"type":"Golden",
"gender":"male"
}
],
"CreationTime":1604036638,
"jobID":12512,
"pets":[
{
"Name":"Pawny",
"Id":"4214",
"budget":"0",
"adoptionDate":2020-09-07T02:56:22Z",
"year":"2",
"type":"Golden",
"gender":"male"
}
]
}

It's because i was using Uppercase name in my field names... Wandering in endless loops for 24 hours, i was finally able to figure out this if anyone ran into same issue. Please read here and use lowercase name for your fieldnames. When i changed my field name to "pet". The duplicates are gone

Problem with checking values of JSON array using Rest Assured

I'm trying to add some tests with Rest-Assured to my application, but I can't figure out how to assert some nested values. The error message is :
Expected: (a collection containing "json")
Actual: [[json, spring, gulp, path etc...]]
Here is the code :
when().
get("/api/personsByID/{id}/{count}", 262, 2).
then().
statusCode(200).
body("personDependencies.name", hasItems("json"));
And here is the JSON file that is returned by rest controller:
[
{
"id": 346,
"verified": true,
"displayName": "eda656a2c3cb59ae840e40a28ba4ab50bfb9de0185abcb901c6af6dc59d6668f",
"emails": [
{
"email": "16a23f2e5477df0bbcad718c3abc235b2cb8a1b6648d14f58d42a7be13df2b6e"
}
],
"personDependencies": [
{
"name": "json"
},
{
"name": "spring"
},
{
"name": "gulp"
},
{
"name": "path"
},
{
"name": "junit"
},
{
"name": "activemq"
},
{
"name": "hibernate"
},
{
"name": "jstl"
},
{
"name": "phantomjs"
},
{
"name": "activiti"
},
{
"name": "commons"
},
{
"name": "h2"
},
{
"name": "joda"
},
{
"name": "log4j"
},
{
"name": "exec"
},
{
"name": "admin"
},
{
"name": "coveralls"
},
{
"name": "cxf"
},
{
"name": "cglib"
},
{
"name": "camel"
},
{
"name": "sugaronrest"
},
{
"name": "tslint"
},
{
"name": "httpclient"
},
{
"name": "guava"
},
{
"name": "inventory"
},
{
"name": "jackson"
},
{
"name": "gson"
},
{
"name": "event"
},
{
"name": "OTRS"
},
{
"name": "maven"
},
{
"name": "karma"
},
{
"name": "slf4j"
},
{
"name": "postgresql"
},
{
"name": "typescript"
},
{
"name": "jasmine"
},
{
"name": "spa"
},
{
"name": "javax.servlet"
}
],
"countries": [],
"member_of": [],
"projects": [],
"employee_type": [],
"languages": [
{
"language": "reStructuredText",
"sum": 575
},
{
"language": "JSON",
"sum": 21
},
{
"language": "JavaScript",
"sum": 4467
},
{
"language": "Java",
"sum": 7958
},
{
"language": "Python",
"sum": 2
},
{
"language": "XML",
"sum": 477
},
{
"language": "Plain Text",
"sum": 41
}
],
"distance": 0.6028837702084446
}
]
I have no idea how to make proper assertions, any help would be great. Thanks!

If I am reading your question right you need to check if a certain values are present in a list that is returned for a particular ID
The below should work for you
given().when().get().then().body("find {it.id == 346}.personDependencies.name", hasItems("json", "jackson"));

The first problem you don't need to check the presence of an item with hasItems, you should use hasItem
when().
get("/api/personsByID/{id}/{count}", 262, 2).
then().
statusCode(200).
body("personDependencies.name", hasItem("json"));
Then if you need to add more message to the assertion when the test fails you can do such way:
when().
get("/api/personsByID/{id}/{count}", 262, 2).
then().
statusCode(200).
body("personDependencies.name", describedAs("Array not containing the provided item",hasItem("json")));
In your case you can validate such a way:
when().
get("/api/personsByID/{id}/{count}", 262, 2).
then().
statusCode(200).
body("personDependencies[*].name", hasItem("json"));

Generalised solution to edit a json for SCIM apis using the attributes and excludedAttributes

Below is the original response:
{
"emails": [
{
"type": "work",
"value": "bjensen#example.com"
}
],
"id": "2819c223-7f76-453a-919d-413861904646",
"phoneNumbers": [
{
"type": "work",
"value": "555-555-8377"
},
{
"type": "business",
"value": "555-555-8377"
}
],
"schemas": [
"urn:scim:schemas:core:1.0"
],
"userName": "bjensen"
}
And in the above response I would pass excludedAttributes=phoneNumbers.type and the response should be like below:
{
"emails": [
{
"type": "work",
"value": "bjensen#example.com"
}
],
"id": "2819c223-7f76-453a-919d-413861904646",
"phoneNumbers": [
{
"value": "555-555-8377"
},
{
"value": "555-555-8377"
}
],
"schemas": [
"urn:scim:schemas:core:1.0"
],
"userName": "bjensen"
}

How to implement an autocomplete search field (suggestor) with an existing ElasticSearch index?

The ES index consists of 2 types that are implicitly mapped (default mapping). One type is "person" or an author, the 2nd type is "document".
The index has some 500k entries.
What I have to do is: implement an autocomplete (suggestions) functionality where only the fields "title", "classification" (document) and "name" (author) are relevant for the suggestions shown to the user.
Could it be done without changing the 500k docs in the index?
I found some tutorials that suggest preparing a specific mapping and also altering the documents (this I want to avoid if possible) and so on but I am new to this and I am not sure how to go about the this problem?
Below is the JSON for the index, and how the documents look:
//a Document
{
"rawsource": "Phys.Rev. D67 (2003) 084031",
"pubyear": 2003,
"citedFrom": 19,
"topics": [
{
"name": "General Relativity and Quantum Cosmology"
}
],
"cited": [
{
"ref": 0,
"id": "PN132433"
},
{
"ref": 1,
"id": "PN206900"
}
],
"id": "PN120001",
"collection": "PN",
"source": "Phys Rev D",
"classification": "Physics",
"title": "Observables in causal set cosmology",
"url": "http://arxiv.org/abs/gr-qc/0210061",
"authors": [
{
"name": "Brightwell, Graham"
},
{
"name": "Dowker, H. Fay"
},
{
"name": "Garcia, Raquel S."
},
{
"name": "Henson, Joe"
},
{
"name": "Sorkin, Rafael D."
}
]
}
//a Person (author)
{
"name": "Terasawa, M.",
"documents": [
{
"citedFrom": 0,
"id": "PN039187"
}
],
"coAuthors": [
{
"name": "Famiano, M. A.",
"count": "1"
},
{
"name": "Boyd, R. N.",
"count": "1"
}
],
"topics": [
{
"name": "Astrophysics",
"count": "1"
}
]
}
//the mapping (implicit/default)
{
"dlsnew": {
"aliases": {
},
"mappings": {
"person": {
"properties": {
"coAuthors": {
"properties": {
"count": {
"type": "string"
},
"name": {
"type": "string"
}
}
},
"documents": {
"properties": {
"citedFrom": {
"type": "long"
},
"id": {
"type": "string"
}
}
},
"name": {
"type": "string"
},
"referenced": {
"properties": {
"count": {
"type": "string"
},
"id": {
"type": "string"
}
}
},
"topics": {
"properties": {
"count": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
},
"document": {
"properties": {
"abstract": {
"type": "string"
},
"authors": {
"properties": {
"name": {
"type": "string"
}
}
},
"cited": {
"properties": {
"id": {
"type": "string"
},
"ref": {
"type": "long"
}
}
},
"citedFrom": {
"type": "long"
},
"classification": {
"type": "string"
},
"collection": {
"type": "string"
},
"id": {
"type": "string"
},
"pubyear": {
"type": "long"
},
"rawsource": {
"type": "string"
},
"source": {
"type": "string"
},
"title": {
"type": "string"
},
"topics": {
"properties": {
"name": {
"type": "string"
}
}
},
"url": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1454247029258",
"number_of_shards": "5",
"uuid": "k_CyQaxwSAaae67wW98HyQ",
"version": {
"created": "1050299"
},
"number_of_replicas": "1"
}
},
"warmers": {
}
}
}
The implementation is to be done using JAVA and the Vaadin Framework (this is not relevant at this point, but examples in Java/Vaadin will be most welcomed).
Thanks.

So, I think I solved my problem on the Elasticsearch side or at least to a good enough extend for me and the task at hand. I followed this ruby example.
I had to re-index all documents to accommodate the new settings for my index and to change my mapping explicitly.
They key is in defining proper analyzers and an edgeNGram filter in this case, like so:
"settings": {
"index": {
"analysis": {
"filter": {
"def_ngram_filter": {
"min_gram": "1",
"side": "front",
"type": "edgeNGram",
"max_gram": "16"
}
},
"analyzer": {
"def_search_analyzer": {
"filter": [
"lowercase",
"asciifolding"
],
"type": "custom",
"tokenizer": "def_tokenizer"
},
"def_ngram_analyzer": {
"filter": [
"lowercase",
"asciifolding",
"def_ngram_filter"
],
"type": "custom",
"tokenizer": "def_tokenizer"
},
"def_shingle_analyzer": {
"filter": [
"shingle",
"lowercase",
"asciifolding"
],
"type": "custom",
"tokenizer": "def_tokenizer"
},
"def_default_analyzer": {
"filter": [
"lowercase",
"asciifolding"
],
"type": "custom",
"tokenizer": "def_tokenizer"
}
},
"tokenizer": {
"def_tokenizer": {
"type": "whitespace"
}
}
}
}
}
and the use these in the mapping for the fields to be searched, like so:
"mappings": {
"person": {
"properties": {
"coAuthors": {
"properties": {
"count": {
"type": "string"
},
"name": {
"type": "string"
}
}
},
"documents": {
"properties": {
"citedFrom": {
"type": "long"
},
"id": {
"type": "string"
}
}
},
"name": {
"type": "string",
"analyzer": "def_default_analyzer",
"fields": {
"ngrams": {
"type": "string",
"index_analyzer": "def_ngram_analyzer",
"search_analyzer": "def_search_analyzer"
},
"shingles": {
"type": "string",
"analyzer": "def_shingle_analyzer"
},
"stemmed": {
"type": "string",
"analyzer": "def_snowball_analyzer"
}
}
},
"referenced": {
"properties": {
"count": {
"type": "string"
},
"id": {
"type": "string"
}
}
},
"topics": {
"properties": {
"count": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
},
"document": {
"properties": {
"abstract": {
"type": "string"
},
"authors": {
"properties": {
"name": {
"type": "string",
"analyzer": "def_default_analyzer",
"fields": {
"ngrams": {
"type": "string",
"index_analyzer": "def_ngram_analyzer",
"search_analyzer": "def_search_analyzer"
},
"shingles": {
"type": "string",
"analyzer": "def_shingle_analyzer"
},
"stemmed": {
"type": "string",
"analyzer": "def_snowball_analyzer"
}
}
}
}
},
"cited": {
"properties": {
"id": {
"type": "string"
},
"ref": {
"type": "long"
}
}
},
"citedFrom": {
"type": "long"
},
"classification": {
"type": "string"
},
"collection": {
"type": "string"
},
"id": {
"type": "string"
},
"pubyear": {
"type": "long"
},
"rawsource": {
"type": "string"
},
"source": {
"type": "string"
},
"title": {
"type": "string",
"analyzer": "def_default_analyzer",
"fields": {
"ngrams": {
"type": "string",
"index_analyzer": "def_ngram_analyzer",
"search_analyzer": "def_search_analyzer"
},
"shingles": {
"type": "string",
"analyzer": "def_shingle_analyzer"
},
"stemmed": {
"type": "string",
"analyzer": "def_snowball_analyzer"
}
}
},
"topics": {
"properties": {
"name": {
"type": "string",
"analyzer": "def_default_analyzer",
"fields": {
"ngrams": {
"type": "string",
"index_analyzer": "def_ngram_analyzer",
"search_analyzer": "def_search_analyzer"
},
"shingles": {
"type": "string",
"analyzer": "def_shingle_analyzer"
},
"stemmed": {
"type": "string",
"analyzer": "def_snowball_analyzer"
}
}
}
}
},
"url": {
"type": "string"
}
}
}
}
then querying the index with the following works as expected:
curl -XGET "http://localhost:9200/_search " -d'
{
"size": 5,
"query": {
"multi_match": {
"query": "physics",
"type": "most_fields",
"fields": [
"document.title^10",
"document.title.shingles^2",
"document.title.ngrams",
"person.name^10",
"person.name.shingles^2",
"person.name.ngrams",
"document.topics.name^10",
"document.topics.name.shingles^2",
"document.topics.name.ngrams"
],
"operator": "and"
}
}
}'
Hope this will help someone, it is probably not the best example as I am a complete noob to this, but it worked for me.

There exist different Autocomplete components for Vaadin.
Have a look at this link.
Depending on which Add-On you choose, the databinding is done differently, but you have to "connect" it to your index.