I have a document with many fields (some nested) indexed on elasticsearch. For example:
{
"id" : 1,
"username" : "...",
"name" : "...",
"surname" : "...",
"address" : "...",
"age": 42,
...
"bookmarks" : [{...}, {...}],
"tags" : [{...}, {...}]
}
Only some filed is mapped in my entity (I don't want to map the entire document):
#Document(indexName = "...", type = "...")
public class User {
#Id
private int id;
private String username;
private String address;
// getter/setter methods
}
In the service class I would like to do a partial update with ElasticsearchRepository, without mapping all document's fields in the entity:
public class UserServiceClass {
#Autowired
private UserElasticsearchRepository userElasticsearchRepository;
public void updateAddress(int id, String updatedAddress) {
User user = userElasticsearchRepository.findOne(id);
user.setAddress(updatedAddress);
userElasticsearchRepository.save(user);
}
}
but save method overwrites the entire document:
{
"id" : 1,
"username" : "...",
"address" : "..."
}
Partial udpdate seems not supported by ElasticsearchRepository. So I used ElasticsearchTemplate, to make a partial update, for example:
public class UserServiceClass {
#Autowired
private UserElasticsearchRepository userElasticsearchRepository;
#Autowired
private ElasticsearchTemplate elasticsearchTemplate;
public void updateAddress(int id, String updatedAddress) {
User user = userElasticsearchRepository.findOne(id);
if (user.getUsername().equals("system")) {
return;
}
IndexRequest indexRequest = new IndexRequest();
indexRequest.source("address", updatedAddress);
UpdateQuery updateQuery = new UpdateQueryBuilder().withId(user.getId()).withClass(User.class).withIndexRequest(indexRequest).build();
elasticsearchTemplate.update(updateQuery);
}
}
but seems a bit redundant to have two similar references (repository and ElasticsearchTemplate).
Can anyone suggest me a better solution?
Instead of having both ElasticsearchTemplate and UserElasticsearchRepository injected into your UserServiceClass, you can implement your own custom repository and let your existing UserElasticsearchRepository extend it.
I assume that your existing UserElasticsearchRepository look something like this.
public interface UserElasticsearchRepository extends ElasticsearchRepository<User, String> {
....
}
You have to create new interface name UserElasticsearchRepositoryCustom. Inside this interface you can list your custom query method.
public interface UserElasticsearchRepositoryCustom {
public void updateAddress(User user, String updatedAddress);
}
Then implement your UserElasticsearchRepositoryCustom by create a class called UserElasticsearchRepositoryImpl and implement your custom method inside with injected ElasticsearchTemplate
public class UserElasticsearchRepositoryImpl implements UserElasticsearchRepositoryCustom {
#Autowired
private ElasticsearchTemplate elasticsearchTemplate;
#Override
public void updateAddress(User user, String updatedAddress){
IndexRequest indexRequest = new IndexRequest();
indexRequest.source("address", updatedAddress);
UpdateQuery updateQuery = new UpdateQueryBuilder().withId(user.getId()).withClass(User.class).withIndexRequest(indexRequest).build();
elasticsearchTemplate.update(updateQuery);
}
}
After that, just extends your UserElasticsearchRepository with UserElasticsearchRepositoryCustom so it should look like this.
public interface UserElasticsearchRepository extends ElasticsearchRepository<User, String>, UserElasticsearchRepositoryCustom {
....
}
Finally, you service code should look like this.
public class UserServiceClass {
#Autowired
private UserElasticsearchRepository userElasticsearchRepository;
public void updateAddress(int id, String updatedAddress) {
User user = userElasticsearchRepository.findOne(id);
if (user.getUsername().equals("system")) {
return;
}
userElasticsearchRepository.updateAddress(user,updatedAddress);
}
}
You can also move your user finding logic into the custom repository logic as well so that you can passing only user id and address in the method. Hope this is helpful.
You can use ElasticSearchTemplate also to get your User object instead of repository interface. you can use NativeSearchQueryBuilder and other classes to build your query. With this you can avoid two similar reference from your class. Let me know if this solves your problem.
Related
I have a list of documents called customers that I retrieved using mongotemplate, bellow some of the documents:
{"customer": {"entityPerimeter": "abp", "name": "ZERZER", "siren": "6154645", "enterpriseId": "546456", "ic01": "", "marketingOffer": "qlksdjf", "irType": "Router", "offerSpecificationOfferLabel": "2Mb"}}
{"customer": {"entityPerimeter": "sdf", "name": "qazer", "siren": "156", "enterpriseId": "546456", "ic01": "", "marketingOffer": "qlksdjddddsqf", "irType": "Ruter", "offerSpecificationOfferLabel": "2Mb"}}
{"customer": {"entityPerimeter": "zer", "name": "fazdsdfsdgg", "siren": "sdfs", "enterpriseId": "1111", "ic01": "", "marketingOffer": "qsdfqsd", "irType": "Router", "offerSpecificationOfferLabel": "2Mb"}}
That what I did in mongodb to have this result:
public List<DBObject> findAllCustomersByExtractionDateMongo(LocalDate extractionDate) {
Aggregation aggregation = newAggregation(
match(Criteria.where(EXTRACTION_DATE).is(extractionDate)),
project(CUSTOMER).andExclude("_id"),
group().addToSet("$customer").as("distinct_customers"),
unwind("distinct_customers"),
project().andExclude("_id").and("distinct_customers").as("customer"),
project().andExclude("distinct_customers")
);
return template
.aggregate(aggregation, COLLECTION, DBObject.class)
.getMappedResults();
}
Now what I really want is to map those Documents to a Class called Customer:
#Data
#NoArgsConstructor
#AllArgsConstructor
#Builder
public class Customer {
private String entityPerimeter;
private String name;
private String siren;
private String enterpriseId;
private String ic01;
private String marketingOffer;
private String product;
private String irType;
}
I tried to do that by creating a DTO interface:
public interface DocumentToCustomerMapper {
String NULL = "null";
static Customer getFilter(DBObject document) {
var customer = new Customer();
customer.setSiren(Optional.ofNullable((String) document.get(CustomerAttributes.SIREN.value())).orElse(NULL));
customer.setEnterpriseId(Optional.ofNullable((String) document.get(CustomerAttributes.ENTERPRISE_ID.value())).orElse(NULL));
customer.setEntityPerimeter(Optional.ofNullable((String) document.get(CustomerAttributes.ENTITY_PERIMETER.value())).orElse(NULL));
customer.setName(Optional.ofNullable((String) document.get(CustomerAttributes.NAME.value())).orElse(NULL));
customer.setIc01(Optional.ofNullable((String) document.get(CustomerAttributes.IC_01.value())).orElse(NULL));
customer.setMarketingOffer(Optional.ofNullable((String) document.get(CustomerAttributes.MARKETING_OFFER.value())).orElse(NULL));
customer.setProduct(Optional.ofNullable((String) document.get(CustomerAttributes.PRODUCT.value())).orElse(NULL));
customer.setIrType(Optional.ofNullable((String) document.get(CustomerAttributes.IR_TYPE.value())).orElse(NULL));
return customer;
}
}
Then in the findAllCystomersByExtractionDateMongo() I'm doing this:
public List<Customer> findAllCustomersByExtractionDateMongo(LocalDate extractionDate) {
Aggregation aggregation = newAggregation(
match(Criteria.where(EXTRACTION_DATE).is(extractionDate)),
project(CUSTOMER).andExclude("_id"),
group().addToSet("$customer").as("distinct_customers"),
unwind("distinct_customers"),
project().andExclude("_id").and("distinct_customers").as("customer"),
project().andExclude("distinct_customers")
);
final Converter<DBObject, Customer> converter = DocumentToCustomerMapper::getFilter;
MongoCustomConversions cc = new MongoCustomConversions(List.of(converter));
((MappingMongoConverter) template.getConverter()).setCustomConversions(cc);
return template
.aggregate(aggregation, COLLECTION, Customer.class)
.getMappedResults();
}
But unfortunately it's giving me an exception:
Couldn't resolve type arguments for class com.obs.dqsc.api.repository.mongo_template.CustomerRepositoryImpl$$Lambda$1333/0x00000008012869a8!
I tried to remove this code:
final Converter<DBObject, Customer> converter = DocumentToCustomerMapper::getFilter;
MongoCustomConversions cc = new MongoCustomConversions(List.of(converter));
((MappingMongoConverter) template.getConverter()).setCustomConversions(cc);
Then all what I'm getting is some null values in my customer objects:
Customer(entityPerimeter=null, name=null, siren=null, enterpriseId=null, ic01=null, marketingOffer=null, product=null, irType=null)
Customer(entityPerimeter=null, name=null, siren=null, enterpriseId=null, ic01=null, marketingOffer=null, product=null, irType=null)
Customer(entityPerimeter=null, name=null, siren=null, enterpriseId=null, ic01=null, marketingOffer=null, product=null, irType=null)
Note: for performance issues, I don't want to do any mapping in the java side, also I don't want to use a global converter in my mongo configuration.
The problem is that you are using a method reference to express your converter:
final Converter<DBObject, Customer> converter = DocumentToCustomerMapper::getFilter;
(Expanding the method reference to a lambda won't work either.)
Try rewriting that snippet to something else (such as an anonymous inner class).
Here is a very similar issue reported, including info on how to work around this problem: https://github.com/arangodb/spring-data/issues/120
I am trying this, it works for only one id at a time. I want to add list of ids.
public interface collectionRepository extends MongoRepository<collection, String> {
List<collection> findByIds(List<UUID> id);
}
Could anyone suggest some ideas?Thank you for your help in advance!
This works for a case where the POJO class is like:
public class User {
private String id;
private String firstName;
private String lastName;
// constructors (including default), get/set methods, etc.
}
And, the documents are stored in a collection user as, for example:
{ "_id" : ObjectId("604827bf8187ce707fb88681"), "firstName" : "John", "lastName" : "Doe", "_class" : "com.example.demo.User" }
The repository with the method to fetch the objects with the supplied list of ids:
public interface UserRepository extends MongoRepository<User, String> {
#Aggregation(pipeline = { " { '$match': { '_id': { '$in': ?0 } } }" } )
List<User> findByIdsIn(List<String> ids);
}
The call to the repository method:
List<String> inputIds = Arrays.asList("604827d13de5773133374acc", "604827617a40155f5111b9ff");
List<User> outputList = userRepository.findByIdsIn(inputIds);
The outputList has the two documents matching the ids from the variable inputIds.
following is Developer Entity.
Developer
#Entity
#Getter
#Setter
public class Developer {
#Id
#GeneratedValue
#Column(name="DEVELOPER_ID")
private Long id;
private String nickname;
private String name;
private String imageURI;
#OneToMany(mappedBy = "developer", cascade={CascadeType.PERSIST})
private List<Article> articleList = new ArrayList<>();
public void addArticle(Article article) {
this.articleList.add(article);
if ( article.getDeveloper() != this )
article.setDeveloper(this);
}
}
and following is Article Entity.
Article
#Entity
#Getter
#Setter
public class Article {
#Id
#GeneratedValue
#Column(name="ARTICLE_ID")
private Long id;
private String subject;
#ElementCollection
private List<String> contents = new ArrayList<>();
public void addContent(String content) {
this.contents.add(content);
}
#ManyToOne
#JoinColumn(name="DEVELOPER_ID")
private Developer developer;
public void setDeveloper(Developer developer) {
this.developer = developer;
if ( !developer.getArticleList().contains(this) )
developer.getArticleList().add(this);
}
}
view request to RestController (/about/developers/rest/add) via ajax.
and following is Request Body's json.
request body json
[
{"articleList":[
{"contents":["article line","article line"],
"subject":"article subject"},
{"contents":["article line","article line","article line"],
"subject":"article subject"}
],
"nickname":"dev nickname",
"name":"(dev name)",
"imageURI":"default.png"
},
{"articleList":[
{"contents":["article line","article line"],
"subject":"article subject"},
{"contents":["article line"],
"subject":"article subject"}
],
"nickname":"dev nickname",
"name":"(dev name)",
"imageURI":"default.png"}
]
RestController that including #RequestBody
RestController
#RestController
#RequestMapping("/about")
public class AboutRestController {
#Autowired
private DeveloperService developerService;
#PostMapping("/developers/rest/add")
public void developersRegisterAPI(#RequestBody List<Developer> developerList) {
/** Not Working **/
// Saving Developer only (except Associated Articles)
for (Developer developer : developerList) {
developerService.save(developer);
}
/** Working **/
// Saving Developers and Associated Articles
for (Developer developer : developerList) {
Developer newDeveloper = new Developer();
newDeveloper.setName(developer.getName());
newDeveloper.setImageURI(developer.getImageURI());
newDeveloper.setNickname(developer.getNickname());
for (Article article : developer.getArticleList()) {
Article newArticle = new Article();
newArticle.setSubject(article.getSubject());
newArticle.setContents(article.getContents());
newDeveloper.addArticle(newArticle);
}
developerService.save(newDeveloper);
}
}
}
when i printed #RequestBody List<Developer> developerList, each Developer object has associated Article objects.
in first case (not working properly, developerService.save(developer);), developerService.findAll()'s result including only Developer Objects. there are not Article objects in retrieved Developer objects.
in second case (working properly, developerService.save(newDeveloper);), developerService.findAll()'s result including both Developer Objects and associated Article objects.
in fact, both cases are saving associated Articles in RestController's method.
but in web controller's method, when i used developerService.findAll(), there aren't articles. so if i passed retrieved Developer Objects via Model, there aren't Associated Article and view can not print these objects.
and following is DeveloperService
DeveloperService and DeveloperRepository
#Service
public class DeveloperService {
#Autowired
DeveloperRepository developerRepository;
public void save(Developer developer) {
developerRepository.save(developer);
}
public void delete(Developer developer) {
developerRepository.delete(developer);
}
public List<Developer> findAll() {
return developerRepository.findAll();
}
}
#Repository
#Transactional
public interface DeveloperRepository extends JpaRepository<Developer, Long> {
}
and following is ARTICLE Table in h2-console
ARTICLE Table in h2-console (click me for image)
as you can see, DEVELOPER_ID is null.
can i fix this only using
for (Developer developer : developerList) {
developerService.save(developer);
}
?
first will not work because in Articles list in json does not have Developer Object bound. the second case is working because of this line
newDeveloper.addArticle(newArticle);
in which you are adding
if ( article.getDeveloper() != this )
article.setDeveloper(this);
try to bind developer object in your article list in the first loop and your code will work. i mean try to bind developer object in each article object from the list
I have something similar to these two classes:
public class User {
#JsonView(UserView.IdOnly.class)
int userID;
String name;
}
public class Project {
#JsonView(ProjectView.IdOnly.class)
int projectID;
#JsonView(ProjectView.Summary.class)
// JPA annotations ommitted
User user;
}
And the following View classes:
public class UserView extends View {}
public class ProjectView extends View {
public interface Summary extends IdOnly {}
}
public class View {
public interface IdOnly {}
}
My controller is as follows:
#JsonView(ProjectView.Summary.class)
#RequestMapping(value="/project/", method = RequestMethod.GET)
public List getProjects() {
return repository.findAll();
}
The JSON output, as you can see, wraps the userID inside the User object:
[
{
"projectID": 1,
"user": {
"userID": 1
}
}
]
This works as expected, and I can have my clients work with it, but it doesn't seem finished... I would like to get rid of the "user" wrapper:
[
{
"projectID": 1,
"userID": 1
}
]
Is there a way to do this cleanly? Preferably by using another annotation. I don't have any custom serializers yet and I would hate to have to start using them. If this can't be done with JsonViews, is there an alternative?
I know one solution would be to add a userID field to the Project class, but the setter would need a call to the repository (to also update the user field) which would mess up my class diagram.
Looks like there's an annotation called #JsonUnwrapped that removes the object wrapper, and all properties within it are included in the parent object.
http://fasterxml.github.io/jackson-annotations/javadoc/2.0.0/com/fasterxml/jackson/annotation/JsonUnwrapped.html
Hope this helps.
I'm using spring-data-elasticsearch and for the beginning everything works fine.
#Document( type = "products", indexName = "empty" )
public class Product
{
...
}
public interface ProductRepository extends ElasticsearchRepository<Product, String>
{
...
}
In my model i can search for products.
#Autowired
private ProductRepository repository;
...
repository.findByIdentifier( "xxx" ).getCategory() );
So, my problem is - I've the same Elasticsearch type in different indices and I want to use the same document for all queries. I can handle more connections via a pool - but I don't have any idea how I can implement this.
I would like to have, something like that:
ProductRepository customerRepo = ElasticsearchPool.getRepoByCustomer("abc", ProductRepository.class);
repository.findByIdentifier( "xxx" ).getCategory();
Is it possible to create a repository at runtime, with an different index ?
Thanks a lot
Marcel
Yes. It's possible with Spring. But you should use ElasticsearchTemplate instead of Repository.
For example. I have two products. They are stored in different indices.
#Document(indexName = "product-a", type = "product")
public class ProductA {
#Id
private String id;
private String name;
private int value;
//Getters and setters
}
#Document(indexName = "product-b", type = "product")
public class ProductB {
#Id
private String id;
private String name;
//Getters and setters
}
Suppose if they have the same type, so they have the same fields. But it's not necessary. Two products can have totally different fields.
I have two repositories:
public interface ProductARepository extends ElasticsearchRepository<ProductA, String> {
}
public interface ProductBRepository
extends ElasticsearchRepository<ProductB, String> {
}
It's not necessary too. Only for testing. The fact that ProductA is stored in "product-a" index and ProductB is stored in "product-b" index.
How to query two(ten, dozen) indices with the same type?
Just build custom repository like this
#Repository
public class CustomProductRepositoryImpl {
#Autowired
private ElasticsearchTemplate elasticsearchTemplate;
public List<ProductA> findProductByName(String name) {
MatchQueryBuilder queryBuilder = QueryBuilders.matchPhrasePrefixQuery("name", name);
//You can query as many indices as you want
IndicesQueryBuilder builder = QueryBuilders.indicesQuery(queryBuilder, "product-a", "product-b");
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(builder).build();
return elasticsearchTemplate.query(searchQuery, response -> {
SearchHits hits = response.getHits();
List<ProductA> result = new ArrayList<>();
Arrays.stream(hits.getHits()).forEach(h -> {
Map<String, Object> source = h.getSource();
//get only id just for test
ProductA productA = new ProductA()
.setId(String.valueOf(source.getOrDefault("id", null)));
result.add(productA);
});
return result;
});
}
}
You can search as many indices as you want and you can transparently inject this behavior into ProductARepository adding custom behavior to single repositories
Second solution is to use indices aliases, but you had to create custom model or custom repository too.
We can use the withIndices method to switch the index if needed:
NativeSearchQueryBuilder nativeSearchQueryBuilder = nativeSearchQueryBuilderConfig.getNativeSearchQueryBuilder();
// Assign the index explicitly.
nativeSearchQueryBuilder.withIndices("product-a");
// Then add query as usual.
nativeSearchQueryBuilder.withQuery(allQueries)
The #Document annotation in entity will only clarify the mapping, to query against a specific index, we still need to use above method.
#Document(indexName="product-a", type="_doc")