Develop Reference

How to validate against an OpenAPI schema with $refs using json-schema-validator

One caveat here is that the OpenAPI Schema is not being used for the purposes of validating requests and responses. The OpenAPI schema can't be changed to a regular JSON Schema.
Say we have an OpenAPI schema in the form
TestSchema.json
{
"components": {
"schemas": {
"book": {
"type": "object",
"properties": {
"title": {
"type": "string"
},
"author": {
"$ref": "#/components/schemas/author"
}
}
},
"author": {
"type": "object",
"properties": {
"name": {
"type": "string"
}
}
}
}
}
}
Now imagine we have some JSON taking the form of a "book" as defined in the above schema
{
"title": "LOTR",
"author": {
"name": "Tolkien"
}
}
Using the json-schema-validator library we've written code to perform this validation as follows
public class SchemaValidator {
private final ObjectMapper mapper = new ObjectMapper();
private final JsonSchema schema;
public SchemaValidator(FileReader schemaFile) throws IOException {
schema = getJsonSchemaFromJsonNode(mapper.readTree(schemaFile));
}
public Set<ValidationMessage> validate(String json) throws IOException {
return schema.validate(mapper.readTree(json));
}
private JsonSchema getJsonSchemaFromJsonNode(JsonNode node) {
return JsonSchemaFactory.getInstance(VersionFlag.V6).getSchema(node);
}
}
Usage:
#Test
void validate_invalidTitle() throws IOException {
var schemaValidator = new SchemaValidator(
new FileReader("src/test/resources/TestSchema.json")
);
var json = "{\"title\":5,\"author\":{\"name\":\"Tolkien\"}}";
assertEquals(
"$.title: integer found, string expected",
schemaValidator.validate(json).stream().findAny().orElseThrow().getMessage()
);
}
Since the OpenAPI schema has two extra nodes, "components" and "schemas" this code doesn't perform validation because we can't do a 1-1 validation between our "book" json and the schema.
A way around this is to re-write the constructor with a .findValue("book")
public SchemaValidator(FileReader schemaFile) throws IOException {
schema = getJsonSchemaFromJsonNode(mapper.readTree(schemaFile).findValue("book"));
}
But this results in an error of
#/properties/author/$ref: Reference #/components/schemas/author cannot be resolved
As we have now broken the reference path.
One way around this to get the test to pass is to adjust the $ref in the schema to #/author but the schema itself is then invalid.
Am I missing a tool in this library or building the objects incorrectly? What needs to be done to make this validation work?
I dug around in the open-api-validator which ends up using the json-schema-validator library to perform validation but the request/response validation is not a step I need.

OpenAPI Custom Generator - How do I prevent "AllOf" Class Generation in OpenApi

I'm building a custom generator to generate TypeScript/Angular models for an angular application. For a starting point, I copied the code from https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/java/org/openapitools/codegen/languages/TypeScriptAngularClientCodegen.java
I'm trying to figure out how to prevent the generator from generating "AllOf" and from extending models from "AllOf" models.
The generator already generates a model with everything it needs without needing to extend from *AllOf.
I've been able to modify the .java file to prevent it from importing classes that with with "AllOf", but I can't find any documentation or examples to restrict extending from classes that end with "AllOf"
Am I missing something? It seems like there should be a way to tell the generator to just not import or create "AllOf" classes.
// what I get:
import { ValidatePassword } from './validate-password.model';
export interface ChangePassword extends ChangePasswordAllOf, ValidatePassword {
...
}
// what I want
import { ValidatePasswordModel } from './validate-password.model';
export interface ChangePassword extends ValidatePassword {
...
}
Here's my modelGeneric.mustache template:
export interface {{classname}}{{#allOf}}{{#-first}} extends {{/-first}}{{{.}}}{{^-last}}, {{/-last}}{{/allOf}} { {{>modelGenericAdditionalProperties}}
{{#vars}}
{{#description}}
/**
* {{{.}}}
*/
{{/description}}
{{#isReadOnly}}readonly {{/isReadOnly}}{{{name}}}{{^required}}?{{/required}}: {{#isEnum}}{{{datatypeWithEnum}}}{{/isEnum}}{{^isEnum}}{{{dataType}}}{{/isEnum}};
{{/vars}}
}{{>modelGenericEnums}}
Here's the relevant schema sample:
...
"ChangePassword": {
"allOf": [
{
"$ref": "#/components/schemas/ValidatePassword"
},
{
"type": "object",
"additionalProperties": false,
"required": [
"OldPassword"
],
"properties": {
"OldPassword": {
"title": "Current password",
"type": "string",
"minLength": 1
}
}
}
]
},
...

According to this: https://github.com/OpenAPITools/openapi-generator/issues/3100
Could you try:
"ChangePassword": {
"type": "object",
"additionalProperties": false,
"required": [
"OldPassword"
],
"properties": {
"OldPassword": {
"title": "Current password",
"type": "string",
"minLength": 1
}
},
"allOf": [{
"$ref": "#/components/schemas/ValidatePassword"
}]
}

Is there way to get required field array in draft-07 json schema through java code

I want to generate json schema of draft-04 or draft-07 with required array for mandatory fields?
I am new to JSON schema, so able to generate draft-07 schema with victools:
Sharing code for same:
SchemaGeneratorConfigBuilder configBuilder = new SchemaGeneratorConfigBuilder(SchemaVersion.DRAFT_7, OptionPreset.PLAIN_JSON);
SchemaGeneratorConfig config = configBuilder.build();
SchemaGenerator generator = new SchemaGenerator(config);
JsonNode jsonSchema = generator.generateSchema("MyClassName".class);
System.out.println(jsonSchema.toString());
The o/P i got is:
Json schema for draft-07 like this:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveOrHistoricCurrencyAndAmount": {
"type": "object",
"properties": {
"ccy": { "type": "string" },
"value": { "type": "number" }
}
}
}
}
what i wanted is:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveOrHistoricCurrencyAndAmount": {
"type": "object",
"properties": {
"ccy": { "type":"string" },
"value": { "type":"number" }
},
"required": ["ccy", "value"]
}
}
}
I wanted required array for mandatory fields also , so how to generate this using java?

victools generator 4.17.0 currently does not support the required attribute of the Jackson annotation #JsonProperty(required=true) out of the box . It is scheduled for the next version, and will probably be enabled with the option JacksonOption.RESPECT_JSONPROPERTY_ORDER set to true.
see victools changelog
Until then, you could customize the SchemaGeneratorConfigBuilder to support this attribute like this :
SchemaGeneratorConfigBuilder configBuilder = new SchemaGeneratorConfigBuilder(SchemaVersion.DRAFT_7, OptionPreset.PLAIN_JSON);
configBuilder.forFields()
.withRequiredCheck(field -> {
JsonProperty jsonProperty = field.getAnnotationConsideringFieldAndGetter(JsonProperty.class) ;
if ( jsonProperty == null )
return false; // No #JsonProperty ? => field not required.
else
return jsonProperty.required(); // let's respect what the 'required' says
});
SchemaGeneratorConfig config = configBuilder.build();
SchemaGenerator generator = new SchemaGenerator(config);
JsonNode jsonSchema = generator.generateSchema("MyClassName".class);
(disclaimer: I recently submitted the PR for this feature)

ElasticSearch Rest High Level Client remapping wrong

I'm trying to create a class which will write automatically to ElasticSearch through the Rest High Level Client with the operations (create, createBatch, remove, removeBatch, update, updateBatch) and those operations all work and my test cases all succeed. To add a bit more flexibility, I wanted to implement the following method: (find, findAll, getFirsts(n), getLasts(n)). find(key) and findAll() both work perfectly fine but getFirsts(n) and getLasts(n) don't at all.
Here is the context:
Before each test case -> Ensure that index "test" exists and create it if it doesn't
After each test case -> Delete index "test"
For getFirsts(n) and getLasts(n) I call create to have a few items in ElasticSearch and then search according to the uniqueKey.
Here is the mapping for my Test Object:
{
"properties": {
"date": { "type": "long" },
"name": { "type": "text" },
"age": { "type": "integer" },
"uniqueKey": { "type": "keyword" }
}
}
Here is my test case:
#Test
public void testGetFirstByIds() throws BeanPersistenceException {
List<StringTestDataBean> beans = new ArrayList<>();
StringTestDataBean bean1 = new StringTestDataBean();
bean1.setName("Tester");
bean1.setAge(22);
bean1.setTimeStamp(23213987321712L);
beans.add(elasticSearchService.create(bean1));
StringTestDataBean bean2 = new StringTestDataBean();
bean1.setName("Antonio");
bean1.setAge(27);
bean1.setTimeStamp(2332321117321712L);
beans.add(elasticSearchService.create(bean2));
Assert.assertNotNull("The beans created should not be null", beans);
Assert.assertEquals("The uniqueKeys of the fetched list should match the existing",
beans.stream()
.map(ElasticSearchBean::getUniqueKey)
.sorted((b1,b2) -> Long.compare(Long.parseLong(b2),Long.parseLong(b1)))
.collect(Collectors.toList()),
elasticSearchService.getFirstByIds(2).stream()
.map(ElasticSearchBean::getUniqueKey)
.collect(Collectors.toList())
);
}
Here is getFirstByIds(n):
#Override
public Collection<B> getFirstByIds(int entityCount) throws BeanPersistenceException {
assertBinding();
FilterContext filterContext = new FilterContext();
filterContext.setLimit(entityCount);
filterContext.setSort(Collections.singletonList(new FieldSort("uniqueKey",true)));
return Optional.ofNullable(find(filterContext)).orElseThrow();
}
Here is the find(filterContext):
#Override
public List<B> find(FilterContext filter) throws BeanPersistenceException {
assertBinding();
BoolQueryBuilder query = QueryBuilders.boolQuery();
List<FieldFilter> fields = filter.getFields();
StreamUtil.ofNullable(fields)
.forEach(fieldFilter -> executeFindSwitchCase(fieldFilter,query));
SearchSourceBuilder builder = new SearchSourceBuilder().query(query);
builder.from((int) filter.getFrom());
builder.size(((int) filter.getLimit() == -1) ? FILTER_LIMIT : (int) filter.getLimit());
SearchRequest request = new SearchRequest();
request.indices(index);
request.source(builder);
List<FieldSort> sorts = filter.getSort();
StreamUtil.ofNullable(sorts)
.forEach(fieldSort -> builder.sort(SortBuilders.fieldSort(fieldSort.getField()).order(
fieldSort.isAscending() ? SortOrder.ASC : SortOrder.DESC)));
try {
if (strict)
client.indices().refresh(new RefreshRequest(index), RequestOptions.DEFAULT);
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
SearchHits hits = response.getHits();
List<B> results = new ArrayList<>();
for (SearchHit hit : hits)
results.add(objectMapper.readValue(hit.getSourceAsString(), clazz));
return results;
}
catch(IOException e){
logger.error(e.getMessage(),e);
}
return null;
}
The issue happens if I run the test case more than one time. The first time, the test passes fine but whenever we reach the second test, I get an exception :
ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]];
After looking around for over a day, I've realized that the map gets changed from the original mapping (map specified at the beginning) and it gets automatically created with this :
"test": {
"aliases": {},
"mappings": {
"properties": {
"age": {
"type": "long"
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timeStamp": {
"type": "long"
},
"uniqueKey": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
As I can see, the mapping changes automatically and throws the error.
Thanks for any help!

Elastic creates dynamic mapping only when no mapping exist for a field when documents are inserted. Check to see if the put mapping call happens before documents are added to index. If the mappings are applied statically, be sure the documents are inserted to the right index.

Altering dynamic mapping Elasticsearch 5.3

Many of the string fields in my application need to be mapped dynamically in elasticsearch 5.3. All new fields that end in id or ids should be mapped and indexed automatically by elastic as such:
"_my_propertyId":
{
"type": "keyword"
}
I defined a dynamic template for the index/type like this
"mappings": {
"my_type": {
"dynamic_templates": [
{
"id_as_keywords": {
"match": "*id|*Id|*Ids",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
Yet, elastic still creates the properties like this:
"_someIds": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
I'm not sure what i'm doing wrong or why this is the default mapping for dynamic string fields now. However, I need to be able dynamically map all properties that end in id or ids as keywords, without ignore_above and fully indexed so I can search for them using the searchAPI. Ideas? Why is this the default string mapping now (I understand the introduction of keyword/text, but still)?
Update
Found a good article on these default settings:
Strings

You can use match_pattern parameter to have more control on match parameter. Find the updated dynamic template below:
"dynamic_templates": [
{
"id_as_keywords": {
"match_mapping_type": "string",
"match_pattern": "regex",
"match": ".*(id|Id|Ids)",
"mapping": {
"type": "keyword"
}
}
}
]
You can read more about match_pattern here.