Supposing we have documents in a Mongo collection with the following format:
{
"_id" : "5fb3c5ce9997c61e15a9108c",
"stages" : {
"stage1" : {
"type" : "RandomType"
},
"stage2" : {
"type" : "RandomType2"
},
"arbitaryStage" : {
"type" : "RandomType3"
},
// Possibly many other stages
},
// Fields omitted
}
How can I query a collection of such documents where any stages.X.type is equal to a predefined value? My application doesn't know what X is and doesn't care about it, it should only know that the type of at least one of the stages is equal to a given value. I am trying to do that in Morphia however plain JS would guide me if that's possible to do with the given data format.
For reference the Class from which this entity is originated from is the following
#Entity(value = "stages_collection", noClassnameStored = true)
public class StackOverflowQ {
#Id
private ObjectId id;
#Embedded
private Map<String, Stage> stages;
// Rest of fields/setters/getters omitted
}
public class Stage {
private String type;
// Rest of fields/setters/getters omitted
}
Even if you employ a blanket check, if the system cannot pick the right field then basically the best we can do is search the whole collection, even if you find someway to do it it won't be a very efficient.
A simple change in schema would be better when you expect random fields in the data. In your case.
{
stageName : "Stage 1",
type : "RandomType"
}
You will be able to utilise indexes properly here as well when you scale and the flexibility remains in your hands for future additions. No need to change things in the code when a new stage is required.
Related
I'm quite new to elastic search, I'm not able to set mapping via field annotation. I'm using spring data elastic 4.3.4. I'm adding settings via #Setting annotation which is actually working. But if I set Field type to Keyword it is not getting updated, and elastic dynamically maps the field type to text. My requirement is to add a normaliser to enable alphabetic sort on specific fields. Please find my set-up below, I really appreciate your help.
Configuration:
Elastic Search version :"7.11.1",
Spring data elastic 4.3.4.
Sample code
#Document(indexName = "#{#environment.getProperty('elastic.index.prefix')}-test")
#Setting(settingPath = "/elasticsearch/analyzer.json")
public class ElasticTest {
#Id
String id;
#Field(type = FieldType.Keyword, normalizer="sort_normalizer")
private String name;
#Field
private String createdDate;
#Field(type=FieldType.Object)
private CustomerType customerType;
=============================================================================
So once the index is created, I can see the settings added
"creation_date" : "1664385255792",
"analysis" : {
"normalizer" : {
"sort_normalizer" : {
"filter" : [
"lowercase",
"asciifolding"
],
"type" : "custom",
"char_filter" : [ ]
}
},
"analyzer" : {
"custom_pattern_analyzer" : {
"lowercase" : "true",
"pattern" : """\W|_""",
"type" : "pattern"
}
}
},
Mapping:
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
Note: Since I'm working locally I have to drop and recreate the index multiple times. I'm able to set these field types via curl/kibana.
Update: Here we create the index by:
if (!searchTemplate.indexOps(ElasticContract.class).exists()) {
searchTemplate.indexOps(ElasticContract.class).create();
}
And we also use ElasticsearchRepository for querying.
How are the index and the mapping created?
If you use Spring Data Elasticsearch repositories, then the index with the setting and mapping will be created if it does not yet exist on application startup.
If you do not use a repository, but use ElasticsearchOperations, you will need to create the index by yourself:
ElasticsearchOperations operations;
IndexOperations indexOps = operations.indexOps(ElasticTest.class);
indexOps.createWithMapping();
If you do not create the index but just insert some data, then Elasticsearch will automatically create the index and the mapping. The mapping you show is the typical autocreated one for a String field.
The #Field annotation you use is correct, we have a similar setup in one of the tests that tests exactly this behaviour, see https://github.com/spring-projects/spring-data-elasticsearch/blob/main/src/test/java/org/springframework/data/elasticsearch/core/index/MappingBuilderIntegrationTests.java#L126-L145
I'm having some trouble properly indexing dynamic numeric fields, it seems they are always indexed as strings.
In my understanding, when indexing dynamic numeric fields I must use dynamic templates:
PUT /com.product.product
{
"mappings": {
"com.product.Product": {
"dynamic_templates": [
{
"numeric_sort": {
"match_mapping_type": "*",
"match_pattern": "regex",
"match": "^sort_num_.*",
"mapping": {
"type": "double"
}
}
}
]
}
}
}
That i'm uploading in an event listener:
#Configuration
#Transactional
public abstract class DynamicTemplateConfig {
#EventListener
public void addDynamicTemplates(ContextRefreshedEvent event) {
if (this.searchIndexingIsActive) {
this.addDynamicTemplates();
}
}
...
}
And i'm indexing the properties in a field bridge:
public class PropertyValueFieldBridge implements FieldBridge {
...
private void indexBigDecimalProperties(Document document, LuceneOptions luceneOptions, PropertyBigDecimal property) {
String fieldName = PREFIX_SORT + NUMERIC + DELIMITER + property.getProperty().getCode();
Double indexedValue = property.getValue().doubleValue();
luceneOptions.addNumericFieldToDocument(
fieldName,
indexedValue,
document);
}
}
After indexing these BigDecimal properties I always end with a string property indexed:
"_source": {
"id": "1",
"sort_id": 1,
"filter_id": 1,
"sort_num_quantity": "115.0"
}
And when i try to sort through this property i have the following exception:
org.hibernate.search.exception.SearchException: Cannot automatically determine the field type for field 'sort_num_quantity'. Use byField(String, Sort.Type) to provide the sort type explicitly.
at org.hibernate.search.query.dsl.sort.impl.SortFieldStates.getCurrentSortFieldTypeFromMetamodel(SortFieldStates.java:177) ~[hibernate-search-engine-5.11.5.Final.jar:5.11.5.Final]
at org.hibernate.search.query.dsl.sort.impl.SortFieldStates.determineCurrentSortFieldTypeAutomaticaly(SortFieldStates.java:150) ~[hibernate-search-engine-5.11.5.Final.jar:5.11.5.Final]
at org.hibernate.search.query.dsl.sort.impl.ConnectedSortContext.byField(ConnectedSortContext.java:42) ~[hibernate-search-engine-5.11.5.Final.jar:5.11.5.Final]
I'm trying to avoid using the byField(String, Sort.Type) since it requires the explicity verification of every property, which i might not know the name and type.
Am i doing something wrong in the indexing process?
Thanks in advance
I don't think you're doing anything wrong. The experimental Elasticsearch integration in Hibernate Search 5 does not really support dynamic fields. You can't specify the type of fields in advance, and it apparently defaults to the String type for dynamic fields.
Upgrading to Hibernate Search 6 (currently in the Candidate Release phase) would be a solution, since it supports dynamic fields through field templates
The Hibernate Search 6 API is different, however, so migrating may require significant work.
tl;dr
Atempting to add an ArrayList in which Object may be an ArrayList to Persistance.
Tried to add an AttributeConverter > Failed
Plz Help
I have no idea what I am doing.
How stupid am I?
The Problem
Dependencies
spring-boot-starter-data-jpa 2.0.0
spring-boot-starter-data-mongodb 2.0.0
eclipselink 2.7.1 <- Probably don't need this one, not sure.
So here is my problem I am trying to add persistence in a Spring Boot Application for a MongoDB in this case I am using tables, the problem comes exactly on the TableRaw bean (a striped down version of Table just for persistance).
Document(collection = "rule_tables")
public class TableRaw {
#Id
private String _id;
private String key;
private String name;
private String returns;
private ArrayList<AxisRaw> axis;
private ArrayList<Object> values = new ArrayList<>();
}
Everything else is just the default constructor (without _id) and getsetters.
So everything works fine with the exception of the values ArrayList. It works fine if it just a simple ArrayList with number and whatnot however in my case I want something like what I am inserting into the database (this is done every time it runs for testing purposes and the values inserted are using the MongoRepository, it works fine)
{
"_id":"5ac20c8b8ee6e6360c8947be",
"key":"1",
"name":"Table 1",
"returns":"Number",
"axis":[
{
"name":"potato",
"values":[
{
"_id":"BottomEdge","value":0
},{
"_id":"Range",
"value":[1,2]
},{
"_id":"TopEdge",
"value":3
}
]
}
],
"values":[
[1,2,3],
[1,2,3],
[1,2,3]
],
"_class":"pt.i2s.gm.gm.rulehandler.tables.model.TableRaw"
}
(For usage in the code the axis length and number of axis matters but in this case it is completely irrelevant.)
Anyway as stated previously it inserts fine into MongoDB but when attempting to get the value the following error is presented.
org.springframework.data.mapping.MappingException: Cannot convert [1, 2, 3] of type class java.util.ArrayList into an instance of class java.lang.Object! Implement a custom Converter<class java.util.ArrayList, class java.lang.Object> and register it with the CustomConversions. Parent object was: [empty]
First thing first I don't exactly know what Parent object was: [empty] means.
Second I tried creating an AttributeConverter as such:
#Component
#Converter(autoApply = true)
public class ArrayList2ObjectConverter implements
AttributeConverter<ArrayList<Object>,Object> {
#Override
public Object convertToDatabaseColumn(ArrayList<Object> attribute) {
return attribute;
}
#SuppressWarnings("unchecked") //If you don't like it suppress it
#Override
public ArrayList<Object> convertToEntityAttribute(Object dbData) {
System.out.println("Converting...");
return (ArrayList<Object>)dbData;
}
}
And adding #Convert(converter = ArrayList2ObjectConverter.class) above the values attribute. However this wasn't even called.
For some reason I couldn't find any answers to this problem, possibly due to my bad coding and making something that is just stupid to do so nobody would do it like this cause it doesn't work.
So how do I do this? And thank you for reading.
Update regarding the Axis and Value amounts
thomi sugested something that would work if I knew from the get go what type of values the table added. I apreciate the answere however some clarification should be made regarding this.
I do not know how many Axis, and therefore nested arrays I will have, it may be 1 it may be 30.
I do not know what the class type of objects will be, it may be numbers, Strings, Booleans, dates, etc. the options are limited but still extensive.
Possible Solution Which I Do Not don't want to use
I could simply create an Object that held a string and an ArrayList which would probably work fine, however I wanted to avoid this resolution, as I don't want to add irrelevant information to the database.
Adopted Solution
By request of #user_531 I will add the solution to this problem.
As this was not working I altered my aproach to the utilization of a new object called ValueList which is simply a wrapper class for a single Object
private ArrayList<ValueList> values;
ValueList Class
public class ValueList {
public Object value;
}
This allows me to add any type of object I want to the list, this does result however in tables looking like this:
{
"key":1,
...... (Same as above)
"values": [
{
"value": [
{
"value":1
},
{
"value":2
}
]
},
{
"value": [
{
"value":3
},
{
"value":4
}
]
}
]
}
Which does look hidious but it doesn't fail anymore and allows me to read values relativelly consistently by calling the "getValue()" method or "getValueList()" method acording to the result from "isValueList()".
I think you should not map something to an object. In your DB, you will surely have an idea of what datatype there will be in your Array, In your case, try and replace with:
#Document(collection = "rule_tables")
public class TableRaw {
#Id
private String _id;
private String key;
private String name;
private String returns;
private ArrayList<AxisRaw> axis;
private List<List<Integer>> values; // no initialization.
}
This should map your structure just fine.
MongoItemReader provided by spring batch has method setFields:
public void setFields(java.lang.String fields)
JSON defining the fields to be returned from the matching documents by
MongoDB.
Parameters:
fields - JSON string that identifies the fields to sort by.
I have a class:
public class Raw {
private String id;
private String version;
private String client;
private String appName;
private String os;
// getters & setters
}
And I have data in mongodb like that:
{
"_id" : ObjectId("58a3373e1e041a1191cd5d6d"),
"Version" : "123",
"Client" : "SomeClient",
"MobilePlatform" : "iphoneos",
"AppName" : "MyAppName",
"Os" : "Windows 10"
}
- so as you can see all fields names start with capital letter.
Now I need to read data from mongo with spring batch.
And I need to map somehow fields in my Raw class to data in mongo DB so I will be able to fetch data.
I suspect that setFields method is just for such cases.
But I am relatively new to mongo and spring batch also,
so I would like to ask how to do that?
Which JSON should I put into setFields method?
Or probably there are some other options?
Any help is greatly appreciated.
I found the anwer by myself in documentation: http://docs.spring.io/spring-data/data-mongo/docs/1.4.2.RELEASE/reference/html/mapping-chapter.html
#Field annotation can be used and it works.
I am hitting a RESTful 3rd party API that always sends JSON in the following format:
{
"response": {
...
}
}
Where ... is the response object that needs to be mapped back to a Java POJO. For instance, sometimes the JSON will contain data that should be mapped back to a Fruit POJO:
{
"response": {
"type": "orange",
"shape": "round"
}
}
...and sometimes the JSON will contain data that should be mapped back to an Employee POJO:
{
"response": {
"name": "John Smith",
"employee_ID": "12345",
"isSupervisor": "true",
"jobTitle": "Chief Burninator"
}
}
So depending on the RESTful API call, we need these two JSON results mapped back to one of the two:
public class Fruit {
private String type;
private String shape;
// Getters & setters for all properties
}
public class Employee {
private String name;
private Integer employeeId;
private Boolean isSupervisor;
private String jobTitle;
// Getters & setters for all properties
}
Unfortunately, I cannot change the fact that this 3rd party REST service always sends back a { "response": { ... } } JSON result. But I still need a way to configure a mapper to dynamically map such a response back to either a Fruit or an Employee.
First, I tried Jackson with limited success, but it wasn't as configurable as I wanted it to be. So now I am trying to use XStream with its JettisonMappedXmlDriver for mapping JSON back to POJOs. Here's the prototype code I have:
public static void main(String[] args) {
XStream xs = new XStream(new JettisonMappedXmlDriver());
xs.alias("response", Fruit.class);
xs.alias("response", Employee.class);
// When XStream sees "employee_ID" in the JSON, replace it with
// "employeeID" to match the field on the POJO.
xs.aliasField("employeeID", Employee.class, "employee_ID");
// Hits 3rd party RESTful API and returns the "*fruit version*" of the JSON.
String json = externalService.getFruit();
Fruit fruit = (Fruit)xs.fromXML(json);
}
Unfortunately when I run this I get an exception, because I have xs.alias("response", ...) mapping response to 2 different Java objects:
Caused by: com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$UnknownFieldException: No such field me.myorg.myapp.domain.Employee.type
---- Debugging information ----
field : type
class : me.myorg.myapp.domain.Employee
required-type : me.myorg.myapp.domain.Employee
converter-type : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
path : /response/type
line number : -1
version : null
-------------------------------
So I ask: what can I do to circumvent the fact that the API will always send back the same "wrapper" response JSON object? The only thing I can think of is first doing a String-replace like so:
String json = externalService.getFruit();
json = json.replaceAll("response", "fruit");
...
But this seems like an ugly hack. Does XStream (or another mapping framework) provide anything that would help me out in this particular case? Thansk in advance.
There are two ways with Jackson:
test manually that the wanted keys are there (JsonNode has the necessary methods);
use JSON Schema; there is one API in Java: json-schema-validator (yes, that is mine), which uses Jackson.
Write a schema matching your first object type:
{
"type": "object",
"properties": {
"type": {
"type": "string",
"required": true
},
"shape": {
"type": "string",
"required": true
}
},
"additionalProperties": false
}
Load this as a schema, validate your input against it: if it validates, you know you need to deserialize against your fruit class. Otherwise, make the schema for the second item type, validate against it as a security measure, and deserialize using the other class.
There are code examples for the API, too (version 1.4.x)
If you do know the actual type, it should be relatively straight-forward with Jackson.
You need to use a generic wrapper type like:
public class Wrapper<T> {
public T response;
}
and then the only trick is to construct type object to let Jackson know what T there is.
If it is statically available, you just do:
Wrapper<Fruit> wrapped = mapper.readValue(input, new TypeReference<Wrapper<Fruit>>() { });
Fruit fruit = wrapped.response;
but if it is more dynamically generated, something like:
Class<?> rawType = ... ; // determined using whatever logic is needed
JavaType actualType = mapper.getTypeFactory().constructGenericType(Wrapper.class, rawType);
Wrapper<?> wrapper = mapper.readValue(input, actualType);
Object value = wrapper.response;
but either way it "should just work". Note that in latter case you may be able to use base types ("? extends MyBaseType"), but in general dynamic type can't be specified.