I have several indices where I save products:
product-example1
product-example2
product-example3
product-example4
product-example5
I have the a document in elastic search that has the same structure and it can used for different indices:
#Data
#org.springframework.data.elasticsearch.annotations.Document(indexName = "", type = "", createIndex = false)
public class ProductDocument {
#Id
private String id;
private String title;
private String seller;
private String releaseDate;
....
}
So basically I want to use same settings, same mappings same search service.
So I made indexName and type parametric in spring boot java, instead of creating 5 classes extending ProductDocument ?
#Autowired
private final ElasticsearchTemplate elasticsearchTemplate;
this.elasticsearchTemplate.createIndex("product-example1", loadFile("/files/settings.json"));
this.elasticsearchTemplate.putMapping("product-example1", "product-type1", loadFile("/files/mapping.json"));
this.elasticsearchTemplate.createIndex("product-example2", loadFile("/files/settings.json"));
this.elasticsearchTemplate.putMapping("product-example2", "product-type2", loadFile("/files/mapping.json"));
......
Now I want to create a ProductRepository but I don't have a class with defined index name. If I use generic class:
public interface DocumentRepository extends ElasticsearchRepository<ProductDocument, String> {
}
I get the error which is totally understable cause I created the index names in dynamic way:
lang.IllegalArgumentException: Unknown indexName. Make sure the indexName is defined. e.g #ProductDocument(indexName="foo")
So is it possible somehow to create repository for indexes that I created in dynamic way as described above, and pass the index name and type as parameter ?
Please help! I'm stuck.
Related
I have kind of a combination-follow up question to [1] and [2].
I have a POJO with a field I want to persist in - and read from - Elasticsearch:
#Document
public class MyPojo {
private String level3;
// getters/setters...
}
For convenience and because the property is also being persisted (flatly) into postgres, the property level3 should be a String, however it should be written into ES as a nested object (because the ES index is defined elsewhere).
The current solution is unsatisfactory:
#Document
#Entity
public class MyPojo {
#Column(name = "level3)
#Field(name = "level3", type = FieldType.Keyword)
#ValueConverter(MyConverter.class)
private String level3;
// getters/setters...
}
with the object path "level1.level2.level3" hardcoded within MyConverter, which converts from Map<String, Object> to String (read) and from String to Map<String, Object> (write). Because we potentially need to do this on multiple fields, this is not a really viable solution.
I'd rather do something like this:
#Document
#Entity
public class MyPojo {
#Column(name = "level3)
#Field(name = "level1.level2.level3", type = FieldType.Keyword)
#ValueConverter(MyConverter2.class)
private String level3;
// getters/setters...
}
which does not work (writing works fine, while reading we get the "null is not a map" error from [2]).
Is this at all possible (if I understood [2] correctly, no)? If not, is there another way to achieve what I want without hardcoding and an extra converter per field?
Can I somehow access the #Field annotation within MyConverter (e.g. the name), or can I somehow supply additional arguments to MyConverter?
[1] Spring data elasticsearch embedded field mapping
[2] Spring Elasticsearch: Adding fields to working POJO class causes IllegalArgumentException: null is not a Map
I have a requirement of using a java model which will have couple of attributes but some of them will have values and some of them will not have and this is not fixed. For example if it has 4 attribute 3 may have values while passing it to the controller method or it may so happen 2 of them will have values but then rest of the attributes will be null. So to handle this i choose to use Query by example of spring , but i am getting
java.lang.IllegalArgumentException : "Should not reach the end of iterator"
I am trying to fetch data from a Azure CosmosDB. Below is the code i have used
ExampleMatcher macther = ExampleMatcher.matching().withIgnoreNullValues();
Example<RequestModel> exampleQ = Example.of(new RequestModel(
req.getEmp(), // these are the attributes which can have alternatively values or can be empty
req.getBase(),
req.getSeat(),
req.getRent()
),matcher);
sampleRepo.findByEmpOrBaseOrSeatOrRent(exampleQ ); // here i am getting the exception
The Repository
public interface SampleRepo extends CosmosRepository<TableA,String>,BaseContainerRepo{
}
The container
#Container(containerName= "${container-tableA}")
public class TableA extends BaseContainer{
}
Base model class
public class BaseContainer{
#Id
private String id;
private Inetger emp;
#PartitionKey
private String key;
private String base;
private String eqp;
}
The base container repo
public interface BaseContainerRepo{
List<BaseContainer> findByEmpOrBaseOrSeatOrRent(Example<RequestModel> exampleQ);
}
Can anyone please let me know where i am doing it wrong .
My domain:
public class Moral {
private String moralId;
private String socialReason;
private Framework framework;
}
public class Framework {
private String externalId;
private Set<String> identifiers;
}
public class Lab extends Framework {
private String system;
private String availability;
}
My DTO:
public class CreateLabRequest {
private String socialReason;
private Set<String> identifiers;
private String system;
private String availability;
}
My Mapper for this looks like:
#Mapping(source = "system", target = "framework.system")
#Mapping(source = "availability", target = "framework.availability")
#Mapping(source = "identifiers", target = "framework.identifiers")
Moral createLabRequestToMoral (CreateLabRequest createLabRequest);
However, I get the following error:
Unknown property "system" in type Framework for target name
"framework.system". Did you mean "framework.externalId"? Unknown
property "availability" in type Framework for target name
"framework.availability". Did you mean "framework.externalId"?
Simply, It is not Possible !
Maybe you wanted to make Framework inherits from Map ?!
Otherwise, the problem is due that you want to access some field in a class that doesn't have it !
public class Framework {
private String externalId;
private Set<String> identifiers;
}
public class Lab extends Framework {
private String system;
private String availability;
}
As it says, extends means that your Lab class inherits from Framework, that means that Lab inherits all fields that Framework has, and not the opposite.
So with that being said :
"framework.system" // cannot be accessed
Since there is no field named "system" in the framework class
However :
"lab.externalId" // CAN be accessed
Since Lab class inherits it from its parent class "Framework" eventhough there is no such field named "system" in the Lab class
More explanations about JAVA inheritance can be found here : https://www.geeksforgeeks.org/inheritance-in-java/
This is possible as follows:
#Mapping( source = ".", target = "framework" )
Moral createLabRequestToMoral( CreateLabRequest createLabRequest );
Lab createLabFramework( CreateLabRequest createLabRequest )
Since Lab extends Framework mapstruct will use createLabFramework( CreateLabRequest createLabRequest ) since it is an user defined method.
Also since all the fields are called the same it is not needed to add those #Mapping annotations.
Edit: expanding a bit about the error.
Unknown property "system" in type Framework for target name "framework.system". Did you mean "framework.externalId"? Unknown property "availability" in type Framework for target name "framework.availability". Did you mean "framework.externalId"?
This is basically caused by MapStruct not knowing that there is a Lab class available that extends Framework. MapStruct can also not know that you want to use the Lab class instead of the Framework class.
As shown above, one of the methods is manually defining an additional mapping method to notify MapStruct about this situation.
I have following entity-model which I'll save to my MongoDB:
#Document(collection = "googleplaygames")
#Component
public final class GooglePlayGame implements Serializable {
#Id
private String title;
private String genre;
private String price;
private LocalDate dateOfLastUpdate;
...
This code allows me to save duplicates Game objects. I found annotation #Indexed and rewrote code:
#Document(collection = "googleplaygames")
#Component
public final class GooglePlayGame implements Serializable {
#Indexed(unique=true)
private String title;
private String genre;
private String price;
private LocalDate dateOfLastUpdate;
...
Now if I'll try to save entity with same title, I'll receive org.springframework.dao.DuplicateKeyException. Fair enough.
And I found this "error" in logs while Spring Boot app is starting:
2020-06-26 13:26:52,303 WARN [restartedMain] org.springframework.data.mongodb.core.index.JustOnceLogger: Automatic index creation will be disabled by default as of Spring Data MongoDB 3.x.
Please use 'MongoMappingContext#setAutoIndexCreation(boolean)' or override 'MongoConfigurationSupport#autoIndexCreation()' to be explicit.
However, we recommend setting up indices manually in an application ready block. You may use index derivation there as well.
> -----------------------------------------------------------------------------------------
> #EventListener(ApplicationReadyEvent.class)
> public void initIndicesAfterStartup() {
>
> IndexOperations indexOps = mongoTemplate.indexOps(DomainType.class);
>
> IndexResolver resolver = new MongoPersistentEntityIndexResolver(mongoMappingContext);
> resolver.resolveIndexFor(DomainType.class).forEach(indexOps::ensureIndex);
> }
> -----------------------------------------------------------------------------------------
But I want to update the enity, if any of other fields are different. For example, if I have an entity with title "Dead Cells" and version "1.0" in my db, I want to update this entity if version now is "1.1". But code above doesn't allow me to do this.
So what is this error? And how to update entity (doesnt allow duplicate by title field, but allow to rewrite entity, if other fields were changed).
According to how you put the question, if you are updating a document, you shouldn't receive the DuplicateKeyException, this only occurs if you have another document with the index value duplicated. Check you are applying the save() method of your repository (as I presume you do) on and existent instance of a Mongo document.
If you want to control duplications, you could use a compound index. A compound index includes more than one field in its definition, so it's faster to make searches for including in the criteria the fields of the index. Including the unicity constraint you'll be able to forbid the duplication of such values in another documents.
Let's say, according to your example you want to avoid titles and genre duplication, then you could define an index as:
#Document(collection = "googleplaygames")
#CompoundIndex(def = "{'title':1,'genre':1}", unique = true)
#Component
public final class GooglePlayGame implements Serializable {
private String title;
private String genre;
private String price;
private LocalDate dateOfLastUpdate;
...
With this index, if you have a document:
{"title":"Pacman","genre":"arcade"}
And try to create a new document with the same values for title and genre you will get the DuplicateKeyException.
If you try to create a document:
{"title":"Pacman","genre":"mobile"}
The you will have two documents.
I am experimenting with spring data elasticsearch by implementing a cluster which will host multi-tenant indexes, one index per tenant.
I am able to create and set settings dynamically for each needed index, like
public class SpringDataES {
#Autowired
private ElasticsearchTemplate es;
#Autowired
private TenantIndexNamingService tenantIndexNamingService;
private void createIndex(String indexName) {
Settings indexSettings = Settings.builder()
.put("number_of_shards", 1)
.build();
CreateIndexRequest indexRequest = new CreateIndexRequest(indexName, indexSettings);
es.getClient().admin().indices().create(indexRequest).actionGet();
es.refresh(indexName);
}
private void preapareIndex(String indexName){
if (!es.indexExists(indexName)) {
createIndex(indexName);
}
updateMappings(indexName);
}
The model is created like this
#Document(indexName = "#{tenantIndexNamingService.getIndexName()}", type = "movies")
public class Movie {
#Id
#JsonIgnore
private String id;
private String movieTitle;
#CompletionField(maxInputLength = 100)
private Completion movieTitleSuggest;
private String director;
private Date releaseDate;
where the index name is passed dynamically via the SpEl
#{tenantIndexNamingService.getIndexName()}
that is served by
#Service
public class TenantIndexNamingService {
private static final String INDEX_PREFIX = "test_index_";
private String indexName = INDEX_PREFIX;
public TenantIndexNamingService() {
}
public String getIndexName() {
return indexName;
}
public void setIndexName(int tenantId) {
this.indexName = INDEX_PREFIX + tenantId;
}
public void setIndexName(String indexName) {
this.indexName = indexName;
}
}
So, whenever I have to execute a CRUD action, first I am pointing to the right index and then to execute the desired action
tenantIndexNamingService.setIndexName(tenantId);
movieService.save(new Movie("Dead Poets Society", getCompletion("Dead Poets Society"), "Peter Weir", new Date()));
My assumption is that the following dynamically index assignment, will not work correctly in a multi-threaded web application:
#Document(indexName = "#{tenantIndexNamingService.getIndexName()}"
This is because TenantIndexNamingService is singleton.
So my question is how achieve the right behavior in a thread save manner?
I would probably go with an approach similar to the following one proposed for Cassandra:
https://dzone.com/articles/multi-tenant-cassandra-cluster-with-spring-data-ca
You can have a look at the related GitHub repository here:
https://github.com/gitaroktato/spring-boot-cassandra-multitenant-example
Now, since Elastic has differences in how you define a Document, you should mainly focus in defining a request-scoped bean that will encapsulate your tenant-id and bind it to your incoming requests.
Here is my solution. I create a RequestScope bean to hold the indexes per HttpRequest
how does singleton bean handle dynamic index