Want to convert following query to Java Mongo template aggregation but don't know how to write multiple fields in group operator with max operator condition.
db.getCollection('mycollection').aggregate([
{
$group: {
_id: "$somefield.$id",
xyz: {
"$max": "$_id"
}
}
}
])
tried to find on mongodb forums, and other website but no solution.
Simply use:
Aggregation.group("somefield.id").max("_id").as("xyz");
And you can get the data with something like this:
// Create aggregation step
AggregateOperation groupOperation = Aggregation.group("somefield.id").max("_id").as("xyz");
// Create aggregation object
Aggregation aggregation = Aggregation.newAggregation(groupOperation);
// Run the aggregate query
AggregationResults<YourClass> result = mongoTemplate.aggregate(aggregation, "mycollection", YourClass.class);
// Get result as list
List<YourClass> listResult = result.getMappedResults();
Related
Vertx mongo client expects query parameter as a JSON object. So far mostly I have done something similar to below
JsonObject queryParam = new JsonObject().put("id", 123);
mongoClient.find("collection", queryParam, asyncResult -> {
if (asyncResult.succeeded()) {
// Do something
promise.complete();
} else {
promise.fail(asyncResult.cause());
}
});
I would build up the above query param according to my needs.
Now I have a service that returns me a list of IDs. I have to query the collection for those matching IDs. Ideally in a mongo shell I would do db.getCollection("collection").find( id: { $in: [1, 2, 3, 4] } )
I checked vertx documents and even searched a bit in vertx mongoClient.java file. Google search didn't yield me solutions to do this in vertx. How can I achieve this without writing a heavy code?
Okay, it's pretty straight forward and I just needed to clear my head a bit to notice that. The complex query can be broken down into a Json Object.
JsonObject queryParam = new JsonObject()
.put("id", new JsonObject().put("$in", listOfIds)); // ArrayList<Long>
I can go ahead and use simple find query with this queryParam.
mongoClient.find("collection", queryParam, asyncResult -> {
// do something
});
I want to perform aggregation using multiple criteria. The problem is that I don't know how to pass multiple criteria. Do I declare multiple Match operation like below?
MatchOperation matchOperation1 = Aggregation.match(criteria);
MatchOperation matchOperation2 = Aggregation.match(criteria2);
And if yes, then how do I pass them to the aggregation method? I thought that it should be possible to create a MatchOperation that adheres in multiple criteria but I have not found such an example online.
Do I declare multiple Match operation like below?
MatchOperation matchOperation1 = Aggregation.match(criteria);
MatchOperation matchOperation2 = Aggregation.match(criteria2);
The Criteria class has an and method, which allows combining two conditions. For example, consider the three documents:
{ _id: 1, size: 10, color: "blue" }
{ _id: 2, size: 12, color: "red" }
{ _id: 3, size: 8, color: "blue" }
The aggregation match stage is defined as follows:
Aggregation.match(Criteria.where("size").gt(new Integer(8))
.and("color").is("blue")
)
This returns the document with _id: 1.
You can try something like this
final Criteria firstMatchCriteria = Criteria.where("fielname").is(someValue)
final Criteria secondMatchCriteria = Criteria.where("fielname").is(someValue)
final Aggregation aggregation = Aggregation.newAggregation( match(firstMatchCriteria),unwind("FIELD_"),match(secondMatchCriteria),project())
You can also use list of criteria in MatchOperation , when you want filter dynamically:
List<Criteria> criterias = new ArrayList<>();
for (String key : mongoRequest.getFilterQuery().keySet()) {
Criteria ci= new Criteria();
List<Object> listOfValues=new ArrayList<>();
for(Object value:mongoRequest.getFilterQuery().get(key)){
listOfValues.add(value);
}
ci = Criteria.where(key).in(listOfValues);
criterias.add(ci);
}
MatchOperation match =new MatchOperation(!criterias.isEmpty()?new Criteria().andOperator(criterias.toArray(new Criteria[criterias.size()])):new Criteria());
you can also refer this link link.
I have a better solution
Criteria result = criteria1.andOperator(criteria2, cri2teria,......)
That worked for me after all.
Criteria criterias = new Criteria().andOperator(Criteria.where(Force.AMOUNT).gte(minForceAmount)
.and(ForceType.TYPE).is(ForceTypeEnum.MAGNETIC_FORCE)
.and("createdAt").gte(startDate).lte(endDate));
MatchOperation matchOperation = Aggregation.match(criteria);
I have a typed dataset of a custom class and use groupbykey method on it. You know that it results a KeyValueGroupedDataset. I want to filter this new dataset but there is no filter method for this type of dataset. So, My question is: How can I filter on this type of dataset? (Java solution is needed. spark version: 2.3.1).
sampleData:
"id":1,"fname":"Gale","lname":"Willmett","email":"gwillmett0#nhs.uk","gender":"Female"
"id":2,"fname":"Chantalle","lname":"Wilcher","email":"cwilcher1#blinklist.com","gender":"Female"
"id":3,"fname":"Polly","lname":"Grandisson","email":"pgrandisson2#linkedin.com","gender":"Female"
"id":3,"fname":"Moshe","lname":"Pink","email":"mpink3#twitter.com","gender":"Male"
"id":2,"fname":"Yorke","lname":"Ginnelly","email":"yginnelly4#apple.com","gender":"Male"
And What I did:
Dataset<Person> peopleDS = spark.read().format("parquet").load("\path").as(Encoders.bean(Person.class));
KeyValueGroupedDataset<String, Person> KVDS = peopleDS.groupByKey( (MapFunction<Person, String> ) f -> f.getGender() , Encoders.STRING());
//How Can I filter on KVDS's id field?
Update1 (use of flatMapGroups):
Dataset<Person> persons = KVDS.flatMapGroups((FlatMapGroupsFunction <String,Person,Person>) (f,k) -> (Iterator<Person>) k , Encoders.bean(Person.class));
Update2 (use of MapGroups)
Dataset<Person> peopleMap = KVDS.mapGroups((MapGroupsFunction <String,Person,Person>) (f,g) -> {
while (g.hasNext()) {
//What can I do here?
}
},Encoders.bean(Person.Class);
Update3 : I want to filter those groups that distinct of their ids is greater than 1. for example in below picture: I want just Female groups because distinct of their ids is greater that 1 (first field is id. Others are fname,lname,email and gender).
Update4: I did What I want with "RDD", but I want to do exactly this part of code with "Dataset":
List<Tuple2<String, Iterable<Person>>> f = PersonRDD
.mapToPair(s -> new Tuple2<>(s.getGender(), s)).groupByKey()
.filter(t -> ((Collection<Person>) t._2()).stream().mapToInt(e -> e.getId).distinct().count() > 1)
.collect();
Why don't you filter on id before grouping ? GroupByKey is an expensive action, it should be faster to filter first.
If you really want to group first, you may have to then use .flatMapGroups with identity function.
Not sure about java code but scala version would be something as follow:
peopleDS
.groupByKey(_.gender)
.mapGroups { case (gender, persons) => persons.filter(your condition) }
But again, you should filter first :). Specially since your ID field is already available before grouping.
Grouping is used for aggregation functions, you can find functions like "agg" in "KeyValueGroupedDataset" class. If you apply aggregation function for ex. "count", you will get "Dataset", and "filter" function will be available.
"groupBy" without aggregation function looks strange, other function, for ex. "distinct" can be used.
Filtering example with "FlatMapGroupsFunction":
.flatMapGroups(
(FlatMapGroupsFunction<String, Person, Person>) (f, k) -> {
List<Person> result = new ArrayList<>();
while (k.hasNext()) {
Person value = k.next();
// filter condition here
if (value != null) {
result.add(value);
}
}
return result.iterator();
},
Encoders.bean(Person.class))
Im trying to retrieve a document with a specific '_id' and a single embedded document with another specific '_id'.
my document is represent a catalog and it contains an array of courses.
example data:
'_id': ObjectId('1111'),
'name': 'example catalog',
...
...
'courses': [
{
'_id': ObjectId('2222'),
'name': 'my course',
...
},
{
....
}
In mongod I run this aggregation query, and get back what I wish for:
db.getCollection('catalogs').aggregate(
{ $match: { '_id': ObjectId('58e8da206ca4f710bab6ef74') } },
{ $unwind: '$courses' },
{ $match: { 'courses._id': ObjectId('58d65541495c851c1703c57f') } })
As I mentioned earlier, I've get back I single catalog instance with a single course instance within.
In my java repo, I was trying to do the same:
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where(Catalog.ID_FIELD).is(catalogId)),
Aggregation.unwind(Catalog.COURSES_FIELD, true),
Aggregation.match(Criteria.where(Catalog.COURSES_FIELD + '.' + Course.ID_FIELD).is(embeddedCourseId))
);
AggregationResults<Catalog> results = mongoTemplate.aggregate(aggregation,
Catalog.class, Catalog.class);
List<Catalog> catalog = results.getMappedResults();
But unfortunately, I've got an instance of my 'example catalog' with empty array of courses.
While debugging, I've found that inside results, there are two props that returns back.
first one is what I've used, called mappedResults (represents the converted object returning from mongoDB) - contains an empty array of courses.
the other one is the rawResults, (represents the data as DBObject) - contains the specific course I query for
my Catalog class contains an ArrayList (if that make any difference).
Please help and let me know what should I do to convert the results properly, or if I did something wrong in my code.
You can try below options. The key is to preserve the structure when mapping the response.
Regular Queries:
Using $positional projection
Query query = new Query();
query.addCriteria(Criteria.where("id").is(new ObjectId("58e8da206ca4f710bab6ef74")).and("courses.id").is(new ObjectId("58d65541495c851c1703c57f")));
query.fields().include("name").position("courses", 1);
List<Course> courses = mongoTemplate.find(query, Course.class);
Using $elemMatch projection
Query query = new Query();
query.addCriteria(Criteria.where("id").is(new ObjectId("58e8da206ca4f710bab6ef74")));
query.fields().include("name").elemMatch("courses", Criteria.where("_id").is(new ObjectId("58d65541495c851c1703c57f") ) );
List<Course> Course = mongoTemplate.find(query, Course.class);
Aggregation
Mongo Version >= 3.4 & Spring 1.5.2 Boot / Spring 1.10.1 Mongo.
You can use $addFields stage which will overwrite the courses field with the $filter value while keeping all the existing properties. I couldn't find any addFields builder in current spring version. So I have to use AggregationOperation to create a new one.
AggregationOperation addFields = new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext aggregationOperationContext) {
DBObject dbObject =
new BasicDBObject("courses",
new BasicDBObject("$filter",
new BasicDBObject("input", "$$courses").
append("as", "course").
append("cond",
new BasicDBObject("$eq", Arrays.<Object>asList("$$course._id", new ObjectId("58d65541495c851c1703c57f"))))));
return new BasicDBObject("$addFields", dbObject);
}
};
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("_id").is(new ObjectId("58e8da206ca4f710bab6ef74"))),
addFields
);
Mongo Version = 3.2 & Spring 1.5.2 Boot / Spring 1.10.1 Mongo..
The idea is still same as above but this pipeline uses $project so you'll have to add all the fields that you want to keep in final response. Also used spring helper methods to create the $filter pipeline.
Aggregation aggregation = newAggregation(
Aggregation.match(Criteria.where("id").is(new ObjectId("58e8da206ca4f710bab6ef74"))),
Aggregation.project("name")
.and(ArrayOperators.Filter.filter("courses").as("course")
.by(ComparisonOperators.Eq.valueOf("course._id").equalToValue(new ObjectId("58d65541495c851c1703c57f")))
).as("courses")
);
Mongo Version <= 2.6
You'll have to use $unwind and add a course field to have spring map it correctly.
The problem that you have here is that your Catalog class has a courses field which maps to a List/ArrayList. But when your aggregation query unwinds the courses array, it is going to output the courses field as a sub-document. The Spring mapper doesn't know how to deal with that because it doesn't match your Catalog object structure.
You haven't fully defined your problem here, but what would probably make more sense is if you had the aggregation return a Course object rather than a Catalog object. In order to do that you're going to need to add a projection stage to your aggregation pipeline so that the result looks exactly like a single Course object. The key is that the data coming back from MongoDB needs to match your object structure.
I want to execute a query in java where path and _id are two fields of the mongo document.
I want to get results list where these two fields are equal in the document.
I have tried using the following query.But could not retrieve the results properly.Received empty list which is not the case.
List<Metadata> MetadataList= ops.find(new Query(Criteria.where("path").is("_id")), Metadata.class);
How to get results where two field values are equal in mongo.
What you are looking for is the $where operator in MongoDB. Standard query operations do not compare the values of one field against another. In order to do this, you need to employ the JavaScript evaluation server side which can actually compare the two field values:
BasicQuery query = new BasicQuery(
new BasicDBObject("$where", "return this._id == this.path")
);
<Metadata> MetadataList = ops.find(query, Metadata.class);
Or you can do the same thing with native operators through the $redact pipeline stage available to the aggregation framework.
Pretty sure there is no $redact support in spring mongo as yet, but you can wrap the aggregation operation with a class to do so:
public class CustomAggregationOperation implements AggregationOperation {
private DBObject operation;
public CustomAggregattionOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
And use it like this:
Aggregation aggregation = newAggregation(
new CustomAggregationOperation(
new BasicDBObject(
"$redact",
new BasicDBObject("$cond",
new BasicDBObject()
.append("if", new BasicDBObject(
"$eq", Arrays.asList("$_id", "$path")
))
.append("then", "$$KEEP")
.append("else", "$$PRUNE")
)
)
)
);
AggregationResults<Metadata> results = ops.aggregate(
(TypedAggregation<Metadata>) aggregation, Metadata.class);
So basic MongoDB query operations do not compare field values against each other. To do this you need to follow one of the methods here.
You can use BasicDBObject to add condition.
Try something
BasicDBObject query = new BasicDBObject("path", new BasicDBObject("$eq", "_id");
collection.find(query);
Please refer the below link for more information
http://mongodb.github.io/mongo-java-driver/2.13/getting-started/quick-tour/