I'm using the Spring framework to perform an aggregation on my mongodb. However, the lookup keeps failing and I can't understand why. Here's the query:
Aggregation aggregation = newAggregation(
match(Criteria.where("idOfUser").is(loggedInAccount.getId())),
group("imgID"),
new CustomAggregationOperation(
new BasicDBObject("$lookup",
new BasicDBObject("from","img")
.append("localField","_id")
.append("foreignField","_id")
.append("as","uniqueImgs")
)
),
limit(pageable.getPageSize()),
skip(pageable.getPageSize()*pageable.getPageNumber())
);
AggregationResults aggregationResults = mongo.aggregate(aggregation, "comment", String.class); //Using String at the moment just to see the output clearly.
CustomAggregationOperation is as follows:
public class CustomAggregationOperation implements AggregationOperation {
private DBObject operation;
public CustomAggregationOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
The Spring MongoDB version of lookup isn't recognised which is why I'm using this CustomAggregationOperation. AFAIK it shouldn't affect it.
Ideally what I want to happen is:
Get all the comments of the user.
Make sure that the imgID is distinct for the comments (so there are only the id's of the imgs that have been commented on)
Get the actual img objects related to these ids.
Paginate the returned imgs.
At the moment, step 3 doesn't work, and I think 4 wouldn't work either since limit and skip won't be applied to the objects in "uniqueImgs".
What is returned is:
[{ "_id" : "570e2f5cb1b9125510a443f5" , "uniqueImgs" : [ ]}]
How can I fix this?
EDIT
the imgID stored isn't an ObjectID whereas the _id in the img collection is. Would that have any effect?
The current release (at the time of writing 1.9.5) has support for the $lookup operator and can be implemented as (untested):
LookupOperation lookupOperation = LookupOperation.newLookup()
.from("img")
.localField("_id")
.foreignField("_id")
.as("uniqueImgs");
Aggregation agg = newAggregation(
match(Criteria.where("idOfUser").is(loggedInAccount.getId())),
group("imgID"),
lookupOperation,
limit(pageable.getPageSize()),
skip(pageable.getPageSize()*pageable.getPageNumber())
);
AggregationResults aggregationResults = mongo.aggregate(agg, "comment", String.clas);
Related
I am trying to find how to use mongo Atlas search indexes, from java application, which is using spring-data-mongodb to query the data, can anyone share an example for it
what i found was as code as below, but that is used for MongoDB Text search, though it is working, but not sure whether it is using Atlas search defined index.
TextQuery textQuery = TextQuery.queryText(new TextCriteria().matchingAny(text)).sortByScore();
textQuery.fields().include("cast").include("title").include("id");
List<Movies> movies = mongoOperations
.find(textQuery, Movies.class);
I want smaple java code using spring-data-mongodb for below query:
[
{
$search: {
index: 'cast-fullplot',
text: {
query: 'sandeep',
path: {
'wildcard': '*'
}
}
}
}
]
It will be helpful if anyone can explain how MongoDB Text Search is different from Mongo Atlas Search and correct way of using Atalas Search with the help of java spring-data-mongodb.
How to code below with spring-data-mongodb:
Arrays.asList(new Document("$search",
new Document("index", "cast-fullplot")
.append("text",
new Document("query", "sandeep")
.append("path",
new Document("wildcard", "*")))),
new Document())
Yes, spring-data-mongo supports the aggregation pipeline, which you'll use to execute your query.
You need to define a document list, with the steps defined in your query, in the correct order. Atlas Search must be the first step in the pipeline, as it stands. You can translate your query to the aggregation pipeline using the Mongo Atlas interface, they have an option to export the pipeline array in the language of your choosing. Then, you just need to execute the query and map the list of responses to your entity class.
You can see an example below:
public class SearchRepositoryImpl implements SearchRepositoryCustom {
private final MongoClient mongoClient;
public SearchRepositoryImpl(MongoClient mongoClient) {
this.mongoClient = mongoClient;
}
#Override
public List<SearchEntity> searchByFilter(String text) {
// You can add codec configuration in your database object. This might be needed to map
// your object to the mongodb data
MongoDatabase database = mongoClient.getDatabase("aggregation");
MongoCollection<Document> collection = database.getCollection("restaurants");
List<Document> pipeline = List.of(new Document("$search", new Document("index", "default2")
.append("text", new Document("query", "Many people").append("path", new Document("wildcard", "*")))));
List<SearchEntity> searchEntityList = new ArrayList<>();
collection.aggregate(pipeline, SearchEntity.class).forEach(searchEntityList::add);
return searchEntityList;
}
}
I am trying to make the below elasticsearch query to work with spring data. The intent is to return unique results for the field "serviceName". Just like a SELECT DISTINCT serviceName FROM table would do comparing to a SQL database.
{
"aggregations": {
"serviceNames": {
"terms": {
"field": "serviceName"
}
}
},
"size":0
}
I configured the field as a keyword and it made the query work perfectly in the index_name/_search api as per the response snippet below:
"aggregations": {
"serviceNames": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "service1",
"doc_count": 20
},
{
"key": "service2",
"doc_count": 8
},
{
"key": "service3",
"doc_count": 8
}
]
}
}
My problem is the same query doesn't work in Spring data when I try to run with a StringQuery I get the error below. I am guessing it uses a different api to run queries.
Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"parsing_exception","reason":"no [query] registered for [aggregations]","line":2,"col":19}],"type":"parsing_exception","reason":"no [query] registered for [aggregations]","line":2,"col":19} , message : null
I have tried using the SearchQuery type to achieve the same results, no duplicates and no object loading, but I had no luck. The below sinnipet shows how I tried doing it.
final TermsAggregationBuilder aggregation = AggregationBuilders
.terms("serviceName")
.field("serviceName")
.size(1);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices("index_name")
.withQuery(matchAllQuery())
.addAggregation(aggregation)
.withSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.withSourceFilter(new FetchSourceFilter(new String[] {"serviceName"}, new String[] {""}))
.withPageable(PageRequest.of(0, 10000))
.build();
Would someone know how to achieve no object loading and object property distinct aggregation on spring data?
I tried many things without success to print queries on spring data, but I could not, maybe because I am using the com.github.vanroy.springdata.jest.JestElasticsearchTemplate implementation.
I got the query parts with the below:
logger.info("query:" + searchQuery.getQuery());
logger.info("agregations:" + searchQuery.getAggregations());
logger.info("filter:" + searchQuery.getFilter());
logger.info("search type:" + searchQuery.getSearchType());
It prints:
query:{"match_all":{"boost":1.0}}
agregations:[{"serviceName":{"terms":{"field":"serviceName","size":1,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}]
filter:null
search type:DFS_QUERY_THEN_FETCH
I figured out, maybe can help someone. The aggregation don't come with the query results, but in a result for it self and is not mapped to any object. The Objects results that comes apparently are samples of the query elasticsearch did to run your aggregation (not sure, maybe).
I ended up by creating a method which can do a simulation of what would be on the SQL SELECT DISTINCT your_column FROM your_table, but I think this will work only on keyword fields, they have a limitation of 256 characters if I am not wrong. I explained some lines in comments.
Thanks #Val since I was only able to figure it out when debugged into Jest code and check the generated request and raw response.
public List<String> getDistinctField(String fieldName) {
List<String> result = new ArrayList<>();
try {
final String distinctAggregationName = "distinct_field"; //name the aggregation
final TermsAggregationBuilder aggregation = AggregationBuilders
.terms(distinctAggregationName)
.field(fieldName)
.size(10000);//limits the number of aggregation list, mine can be huge, adjust yours
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices("your_index")//maybe can be omitted
.addAggregation(aggregation)
.withSourceFilter(new FetchSourceFilter(new String[] { fieldName }, new String[] { "" }))//filter it to retrieve only the field we ar interested, probably we can take this out.
.withPageable(PageRequest.of(0, 1))//can't be zero, and I don't want to load 10 results every time it runs, will always return one object since I found no "size":0 in query builder
.build();
//had to use the JestResultsExtractor because com.github.vanroy.springdata.jest.JestElasticsearchTemplate don't have an implementation for ResultsExtractor, if you use Spring defaults, you can probably use it.
final JestResultsExtractor<SearchResult> extractor = new JestResultsExtractor<SearchResult>() {
#Override
public SearchResult extract(SearchResult searchResult) {
return searchResult;
}
};
final SearchResult searchResult = ((JestElasticsearchTemplate) elasticsearchOperations).query(searchQuery,
extractor);
final MetricAggregation aggregations = searchResult.getAggregations();
final TermsAggregation termsAggregation = aggregations.getTermsAggregation(distinctAggregationName);//this is where your aggregation results are, in "buckets".
result = termsAggregation.getBuckets().parallelStream().map(TermsAggregation.Entry::getKey)
.collect(Collectors.toList());
} catch (Exception e) {
// threat your error here.
e.printStackTrace();
}
return result;
}
Let's say we have a entity "Device" it contains other entity "DeviceInfo", and we have a entity "Site" which contains a List of "DeviceInfo" entities, and "DeviceInfo" has a "Device" and a "Site" in its properties.
My task was to find all "Device"s which are in one "Site". To some endpoint I would send a "Site" id and page number and size of page (since it has to be pageable). I have made it work by creating a JPA specification
public static Specification<Device> bySearchRequest(final DeviceSearchRequest searchRequest) {
return (root, query, cb) -> {
final Join<Device, DeviceInfo> deviceInfo
= root.join(Device_.deviceInfo, JoinType.LEFT);
final Join<DeviceInfo, Site> site
= deviceInfo.join(DeviceInfo_.site, JoinType.LEFT);
return cb.and(cb.equal(site.get(Site.id), searchRequest.getSiteId()));
};
}
And then using I would convert the "Device"s to "IndexDevice"s which is in ES.
deviceRepository.findAll(currentUser,
DeviceRepository.Specs.bySearchRequest(searchRequest),
new PageRequest(searchRequest.getPage(), searchRequest.getSize()))
.getContent().stream().map(x ->indexedDeviceConverter.convert(x)).collect(Collectors.toList());
That is it. It works. But here I am fetching the data from DB, and I already have everything in Elasticsearch. Is there a way to make this same query to fetch the data directly from ES (with paging) ?
Only difference is that in ES "IndexedDevice" has a direct relation with a "IndexedSite" (there is no "IndexedDeviceInfo").
IndexedDevice
{
"id":"3eba5104-0c7a-4564-8270-062945cc8f5e",
"name":"D4",
"site":{
"id":"46e7ada4-3f34-4962-b849-fac59c8fe8ad",
"name":"SomeSite",
"displayInformation":"SomeSite",
"subtitle":""
},
"suggest":{
"input":[]
},
"displayInformation":"D4",
"subtitle":""
}
IndexedSite
{
"id": "46e7ada4-3f34-4962-b849-fac59c8fe8ad",
"name": "SomeSite",
"displayInformation": "SomeSite",
"subtitle": ""
}
I managed to do it. At the end it was really simple. I used ElasticsearchRepository (org.springframework.data.elasticsearch.repository).
elasticsearchRepositoy.search(QueryBuilders.termsQuery
("site.id",
searchRequest.getSite()),
new PageRequest(searchRequest.getPage(),
searchRequest.getSize()));
I'm new in mongodb. I have following data as a JSON format in mongodb. I need to search the bookLabel or the shortLabel for the book and it should show me all the information about the book. For example: if I query for 'Cosmos' it'll show all the description about the book, like: bookLabel, writer, yearPublish, url. How can I do that in java? Need query, please help.
"Class":"Science",
"Description":[
{
"bookLabel":"Cosmos (Mass Market Paperback)",
"shortLabel":"Cosmos",
"writer":"Carl Sagan",
"yearPublish":[
"2002"
],
"url":"https://www.goodreads.com/book/show/55030.Cosmos"
},
{
"bookLabel":"The Immortal Life of Henrietta Lacks",
"shortLabel":"Immortal Life",
"writer":"Rebecca Skloot",
"yearPublish":[
"2010, 2011"
],
"url":"https://www.goodreads.com/book/show/6493208-the-immortal-life-of-henrietta-lacks"
}
],
"Class":"History",
"Description":[
{
"bookLabel":"The Rise and Fall of the Third Reich",
"shortLabel":"Rise and Fall",
"writer":"William L. Shirer",
"yearPublish":[
"1960"
],
"url":"https://www"
}
]
}
With MongoDB Java Driver v3.2.2 you can do something like this:
FindIterable<Document> iterable = collection.find(Document.parse("{\"Description.shortLabel\": {$regex: \"Cosmos\"}"));
This returns all documents containing Cosmos in the Description.shortLabel nested field. For an exact match, try this {"Description.shortLabel": "Cosmos"}. Replace shortLabel with bookLabelto search the bookLabel field. Then you can do iterable.forEach(new Block<Document>()) on the returned documents. To search both bookLabel and shortLabel, you can do a $or{}. My syntax could be wrong so check the MongoDB manual. But this is the general idea.
For this, you can use MongoDB's Text Search Capabilities. You'll have to create a text index on your collection for that.
First of all create a text index on your collection on fields bookLabel and shortLabel.
db.books.createIndex({ "Description.bookLabel" : "text", "Description.shortLabel" : "text" })
Note that this is done in the Mongo shell
Then
DBObject command = BasicDBObjectBuilder
.start("text", "books")
.append("search", "Cosmos")
.get();
CommandResult result = db.command(command);
BasicDBList results = (BasicDBList) result.get("results");
for(Object o : results) {
DBObject dbo = (DBObject) ((DBObject) o).get("obj");
String id = (String) dbo.get("_ID");
System.out.println(id);
}
Haven't really tested this. But just give it a try. Should work.
I have the following query in MongoDB:
db.cache.find({objectKey: { $regex: 'Bos*'}, cacheVersionString:'08/03/15_11:05:09'},
{ _id:0, objectData:0, lastModified :0, productCode:0})
Also, I would like to calculate number of records, that is possible in Mongo shell by using count
db.cache.find({objectKey: { $regex: 'Bos*'}, cacheVersionString:'08/03/15_11:05:09'},
{ _id:0, objectData:0, lastModified :0, productCode:0}).count()
How to write the analogous queries in Java using Java MongoDB driver, version 3?
Constructing queries is really just creating BSON document representation, which is basically just the same interface as standard HashMap or List interfaces as appropriate:
Document query = new Document("objectKey",new Document( "$regex","Bos"))
.append("cacheVersionString","08/03/15_11:05:09");
Document projection = new Document("_id",0)
.append("objectData",0)
.append("lastModified",0)
.append("productCode",0);
MongoCursor<Document> cursor = collection.find(query).projection(projection).iterator();
Where that is basically identical to how you are structuring queries in the MongoDB shell.
Alternately you can use builder interfaces if that seems more logical to you:
QueryBuilder builder = QueryBuilder.start();
builder.and("objectKey").regex(Pattern.compile("box"));
builder.and("cache_version_string").is("08/03/15_11:05:09");
BasicDBObject query = (BasicDBObject)builder.get();
Bson projection = Projections.exclude(
"_id",
"obectdata",
"lasModified",
"productCode"
);
MongoCursor<Document> cursor = collection.find(query).projection(projection).iterator();
while (cursor.hasNext()) {
Document doc = cursor.next();
System.out.println(doc.toJson());
}
Both forms essentially contruct the BSON for both the "query" and "projection" components and issue them as arguments to the .find() method. There are also class type definitions if that suits you as well.