I have a collection where the documents look like this:
{
_id: "545b9fa0dd5318a4285f7ce7",
owner: "admin",
messages: [
{
id: "100",
status: "sent",
note: ""
},
{
id: "100",
status: "pending",
note: ""
},
{
id: "101",
status: "sent",
note: ""
},
{
id: "102",
status: "sent",
note: ""
},
{
id: "101",
status: "done",
note: ""
}
]
}
(This is just a short example, in my case the sub array is very large)
I need to query the collection and get some statistics for a specific document.
So in this example if I query the doucment that has id: "545b9fa0dd5318a4285f7ce7"
I should get this result:
{
sent: 3,
pending: 1,
done: 1
}
How can I do such aggregation with spring mongotemplate ?
To do this sort of thing you need the $cond operator in the aggregation framework. Spring Data MongoDB does not do that yet, and there are many things lacking from common $group operations that are even only implemented under $project.
The issue for tracking implementation of any $cond support is here:
https://jira.spring.io/browse/DATAMONGO-861
For the rest of the world it looks like this:
db.collection.aggregate([
{ "$match": { "_id": ObjectId("545b9fa0dd5318a4285f7ce7") } },
{ "$unwind": "$messages" },
{ "$group": {
"_id": "$_id",
"sent": {
"$sum": {
"$cond": [
{ "$eq": [ "$mesages.status", "sent" ] },
1,
0
]
}
},
"pending": {
"$sum": {
"$cond": [
{ "$eq": [ "$messages.status", "pending" ] },
1,
0
]
}
},
"done": {
"$sum": {
"$cond": [
{ "$eq": [ "$messages.status", "done" ] },
1,
0
]
}
}
}}
])
To get that sort of thing to work under a mongotemplate aggregation you need a class extending aggregation operations that can be built from a DBObject:
public class CustomGroupOperation implements AggregationOperation {
private DBObject operation;
public CustomGroupOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
Then you can define the "$group" as a DBObject and implement in the aggregation pipeline:
DBObject myGroup = (DBObject)new BasicDBObject(
"$group", new BasicDBObject(
"_id","$_id"
).append(
"sent", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$messages.status", "sent"}
),
1,
0
}
)
)
).append(
"pending", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$messages.status", "pending"}
),
1,
0
}
)
)
).append(
"done", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$messages.status", "done"}
),
1,
0
}
)
)
)
);
ObjectId myId = new ObjectId("545b9fa0dd5318a4285f7ce7");
Aggregation aggregation = newAggregation(
match(Criteria.where("_id").is(myId)),
unwind("messges"),
new CustomGroupOperation(myGroup)
);
That allows you to come up with a pipeline that is basically the same as the shell representation above.
So it would seem that for now, where certain operations and sequences are not supported the best case is to implement a class on the AgggregationOperation interface that can be fed a DBObject or otherwise internally construct one from your own custom methods.
You can use following aggregation :
db.collection.aggregate(
{ $match : { "_id" : ObjectId("545b9fa0dd5318a4285f7ce7") } },
{ $unwind : "$messages" },
{ $group : { "_id" : "$messages.status", "count" : { $sum : 1} } }
)
it will give you the count of status for which message are available, all other status count should be consider 0.
Related
Hi everyone I have a collection of documents like bellow. I want to directly get "rights" from roles array for params: _id, groups._id, roles._id using java mongo driver.
{
"_id": 1000002,
"groups": [
{
"_id": 1,
"roles": [
{
"rights": 3,
"_id": 1
},
{
"rights": 7,
"_id": 2
},
{
"rights": 3,
"_id": 3
}
]
}
],
"timestamp": {
"$date": {
"$numberLong": "1675267318028"
}
},
"users": [
{
"accessProviderId": 1,
"rights": 1,
"_id": 4
},
{
"accessProviderId": 1,
"rights": 3,
"_id": 5
}
]
}
I have AccessListItem class which represents this document and I have used Bson filters to get it from mongo, but after fetching i had to get information through java function.. I want to get int value directly from mongo base.
Bson fileFilter = Filters.eq("_id", itemId);
Bson groupFilter = Filters.elemMatch("groups", Document.parse("{_id:"+groupId+"}"));
Bson roleFilter = Filters.elemMatch("groups.roles", Document.parse("{_id:"+role+"}"));
Bson finalFilter = Filters.and(fileFilter, Filters.and(groupFilter,roleFilter));
MongoCollection<AccessListItem> accessListItemMongoCollection = MongoUtils.getAccessCollection(type);
AccessListItem accessListItem = accessListItemMongoCollection.find(finalFilter).first();
The short answer is you can't.
MongoDB is designed for returning documents, that is, objects containing key-value pairs. There is no mechanism for a MongoDB query to return just a value, i.e. it will never return just 3 or [3].
You could use aggregation with a $project stage at the end to give you a simplified object like:
{ rights: 3}
In javascript that might look like:
db.collection.aggregate([
{$match: {
_id: itemId,
"groups._id": groupId,
"groups.roles._id": role
}},
{$project: {
_id: 0,
group: {
$first: {
$filter: {
input: "$groups",
cond: {$eq: ["$$this._id",groupId]}
}
}
}
}},
{$project: {
"roles": {
$first: {
$filter: {
input: "$group.roles",
cond: { $eq: [ "$$this._id",role]}
}
}
}
}},
{$project: {
rights: "$roles.rights"
}}
])
Example: Playground
I'm not familiar with spring boot, so I'm not sure what that would look like in Java.
I have a json
{"id": 2,"name": "Chethan","address":"Banglore"}
Trying to groupby two fields id and name,
List<String> statFields = new ArrayList();
statFields.add("name");
statFields.add("id");
// 2. bootstrap the query
SearchRequestBuilder search = client.prepareSearch("student")
.setSize(0).setFrom(0)
.setQuery(QueryBuilders.matchAllQuery());
// 3. add a stats aggregation for each of your fields
for (String field : statFields) {
search.addAggregation(AggregationBuilders.terms(field+"_stats").field(field));
}
// 4. execute the query
SearchResponse response = search.execute().actionGet();
for(String field : statFields) {
Terms termAgg = (Terms) response.getAggregations().get(field+"_stats");
for (Terms.Bucket entry : termAgg.getBuckets()) {
System.out.println(entry.getKey() + " **** " + entry.getDocCount()); // Doc count
}
}
Below is the response
chethan**** 2
Raj**** 1
Mohan**** 1
1 **** 1
2 **** 1
3 **** 1
But I need combined response like sql,
name id count
chethan 1 1
is it possible through elasticsearch java api
You should have used subAggregation plus use keyword type for aggregations.
Java Rest High-Level Client
Assuming your mappings look like:
PUT student
{
"mappings": {
"doc": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"id": {
"type": "keyword"
},
"address": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}
In order to group by name and id you should use this query (low level query):
GET student/_search
{
"size": 0,
"aggs": {
"name": {
"terms": {
"field": "name.keyword",
"size": 10
},"aggs": {
"id": {
"terms": {
"field": "id",
"size": 10
}
}
}
}
}
}
In java the query above is similar to:
SearchResponse response = client.search(new SearchRequest("student")
.source(new SearchSourceBuilder()
.size(0)
.aggregation(
AggregationBuilders.terms("by_name").field("name.keyword")
.subAggregation(AggregationBuilders.terms("by_id")
.field("id")
)
);
If you want to use your code, probably something like that :
// 2. bootstrap the query
SearchRequestBuilder search = client.prepareSearch("student")
.setSize(0).setFrom(0)
.setQuery(QueryBuilders.matchAllQuery());
// 3. add a stats aggregation for each of your fields
TermsAggregationBuilder aggregation = AggregationBuilders.terms("name_stats").field("name.keyword");
aggregation.subAggregation(AggregationBuilders.terms("id_stats").field("id"));
search.aggregation(aggregation);
// 4. execute the query
SearchResponse response = search.execute().actionGet();
Terms termAgg = (Terms)response.getAggregations().get("name_stats");
for (Terms.Bucket entry: termAgg.getBuckets()) {
if (entry.getDocCount() != 0) {
Terms terms =entry.getAggregations().get("id");
Collection<Terms.Bucket> buckets = terms.getBuckets();
for (Bucket sub : buckets ) {
System.out.println((int) sub.getDocCount());
System.out.println(sub.getKeyAsString());
}
}
}
I removed the for loop. you should design your own structure now that you have to use sub-aggregations.
UPDATE
Is this what you want?
GET student/_search
{
"size": 0,
"aggs" : {
"name_id" : {
"terms" : {
"script" : {
"source": "doc['name.keyword'].value + '_' + doc['id'].value",
"lang": "painless"
}
}
}
}
}
I hope this is what you aimed for.
How to use mongodb java driver to compare dayOfYear of two ISODate objects?
Here are my docs
{"name": "hello", "count": 4, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "hello", "count": 5, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "goodbye", "count": 6, "TIMESTAMP": ISODate("2017-10-01T02:00:35.098Z")}
{"name": "foo", "count": 6, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
I want to compare the day in "TIMESTAMP" to perform some aggregation
Bson match = Aggregates.match(eq("name": "hello"));
Bson group = Aggregates.group(new Document("name", "$name"), Accumulators.sum("total", 1));
collection.aggregate(Arrays.asList(match, group))
Now I am not sure how to do this aggregation for all the records that belongs to particular day?
so my expected result for "2017-10-02" is
[{"_id": {"name":"hello"}, "total": 9}, {"_id": {"name":"foo"}, "total": 6}]
Given the following documents:
{"name": "hello", "count": 4, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "hello", "count": 5, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "goodbye", "count": 6, "TIMESTAMP": ISODate("2017-10-01T02:00:35.098Z")}
{"name": "foo", "count": 6, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
The following command ...
db.getCollection('dayOfYear').aggregate([
// project dayOfYear as an attribute
{ $project: { name: 1, count: 1, dayOfYear: { $dayOfYear: "$TIMESTAMP" } } },
// match documents with dayOfYear=275
{ $match: { dayOfYear: 275 } },
// sum the count attribute for the selected day and name
{ $group : { _id : { name: "$name" }, total: { $sum: "$count" } } }
])
... will return:
{
"_id" : {
"name" : "foo"
},
"total" : 6
}
{
"_id" : {
"name" : "hello"
},
"total" : 9
}
I think this meets the requirement expressed in your OP.
Here's the same command expressed using the MongoDB Java driver:
MongoCollection<Document> collection = mongoClient.getDatabase("stackoverflow").getCollection("dayOfYear");
Document project = new Document("name", 1)
.append("count", 1)
.append("dayOfYear", new Document("$dayOfYear", "$TIMESTAMP"));
Document dayOfYearMatch = new Document("dayOfYear", 275);
Document grouping = new Document("_id", "$name").append("total", new Document("$sum", "$count"));
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(
new Document("$project", project),
new Document("$match", dayOfYearMatch),
new Document("$group", grouping)
));
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Update based on this comment:
One of the problems with project is that it only include fields you specify . The above input is just an example. I have 100 fields in my doc I can't sepecify every single one so if I use project I have to specify all 100 fields in addition to "dayOfYear" field. – user1870400 11 mins ago
You can use the following command to return the same output but without a $project stage:
db.getCollection('dayOfYear').aggregate([
// ignore any documents which do not match dayOfYear=275
{ "$redact": {
"$cond": {
if: { $eq: [ { $dayOfYear: "$TIMESTAMP" }, 275 ] },
"then": "$$KEEP",
"else": "$$PRUNE"
}
}},
// sum the count attribute for the selected day
{ $group : { _id : { name: "$name" }, total: { $sum: "$count" } } }
])
Here's that command in its 'Java form':
MongoCollection<Document> collection = mongoClient.getDatabase("stackoverflow").getCollection("dayOfYear");
Document redact = new Document("$cond", new Document("if", new Document("$eq", Arrays.asList(new Document("$dayOfYear", "$TIMESTAMP"), 275)))
.append("then", "$$KEEP")
.append("else", "$$PRUNE"));
Document grouping = new Document("_id", "$name").append("total", new Document("$sum", "$count"));
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(
new Document("$redact", redact),
new Document("$group", grouping)
));
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Note: Depending on the size of your collection/your non functional requirements/etc you may want to consider the performance of these solutions and either (a) add a match stage before you start projecting/redacting or (b) extract dayOfYear into its own attribute so that you can avoid this complexity entirely.
I have a MongoDB storing data from different sensors. It has the following structure:
{
"_id" : 1,
"sensorName" : "Heart Rate",
"samplePeriod" : 1000,
"data" : [
{
"timestamp" : NumberLong("1483537204046"),
"dataPoints" : [ 68 70 ]
},
{
"timestamp" : NumberLong("1483537206046"),
"dataPoints" : [ 68 70 ]
}
]
}
{
"_id" : 2,
"sensorName" : "Ambient Light",
"samplePeriod" : 500,
"data" : [
{
"timestamp" : NumberLong("1483537204058"),
"dataPoints" : [ 56, 54, 54, 54 ]
},
{
"timestamp" : NumberLong("1483537206058"),
"dataPoints" : [ 56, 54, 54, 54 ]
}
]
}
Now for example i need the "Heart Rate" - document with all of its fields and those of its "data" - subdocuments matching the condition "timestamp between 1483537204000 and 1483537214000".
I already got the answer on how to do this in the mongo shell in another Question. See this code:
aggregate([{
$match: {
"_id": 1
}
}, {
"$project": {
"_id": 1,
"sensorName": 1,
"samplePeriod": 1,
"data": {
"$filter": {
"input": "$data",
"as": "result",
"cond": {
$and: [{
$gte: ["$$result.timestamp", 1483537204000]
}, {
$lte: ["$$result.timestamp", 1483537214000]
}]
}
}
}
}
}])
But how do I do this in java spring-data? It seems there is nothing like $filter in spring-data. Is there a workaround?
How efficient is $filter anyway?
Can you think of a more efficient/practical way of structuring this kind of data in mongodb?
Thanks in advance!
You'll need to make use of MongoTemplate provided in the spring mongo data dependency. There is no out of box support for $filter in the current release version. Make use of AggressionExpression. Include below projection in project. Use 1.8.5 spring mongo data version.
Aggregation aggregation = newAggregation(
match(Criteria.where("_id").is(1)),
project( "_id", "sensorName", "samplePeriod").and(new AggregationExpression() {
#Override
public DBObject toDbObject(AggregationOperationContext aggregationOperationContext) {
DBObject filter = new BasicDBObject("input", "$data").append("as", "result").append("cond",
new BasicDBObject("$and", Arrays.<Object> asList(new BasicDBObject("$gte", Arrays.<Object> asList("$$result.timestamp", 1483537204000L)),
new BasicDBObject("$lte", Arrays.<Object> asList("$$result.timestamp", 1483537214000L)))));
return new BasicDBObject("$filter", filter);
}
}).as("data")
);
List<BasicDBObject> dbObjects = monoTemplate.aggregate(aggregation, "collectionname", BasicDBObject.class).getMappedResults();
I think the same can be achieved by the use of unwind and an extra match. Spring mongo driver does provide support for unwind and it looks bit cleaner.
aggregate([{
$match: {
"_id": 1
}
}, {
$unwind : "$data"
},{
$match : {'data.timestamp' : {$gte : 1483537204000, $lte : 1483537214000}}
}, {
$group : {
_id : $_id,
data : {$push:$data}
}
}])
The Spring Data MongoDB 1.10 RC1 (Ingalls), 2.0 M2 (Kay) releases (as of writing) have added support for $filter and this can be implemented as follows:
Aggregation.newAggregation(Entity.class,
match(Criteria.where("_id").is(1)),
project("sensorName", "samplePeriod")
.and(
filter("data")
.as("item")
.by(
GTE.of(field("item.timestamp"), 1483537204000)
.LTE.of(field("item.timestamp"), 1483537214000)
)
).as("data")
)
I have a survey document in mongodb, each survey have surveyRefId for unique identification. I am not able to understand how to find sub-documents having questionType = hard in documents whose surveyRefid = 377 or 360.
Here is a sample document:
{
"json": {
"surveyRefId": 377,
"surveyName": "survey on sociology",
"questionsVoList": [
{
"questionId": "556708425215763c64b8af3d",
"questionText": "question no 1",
"questionTitle": "",
"questionType": "hard",
"preQuestion": true,
"questionOptions": [
{
"questionRefId": 0,
"optionType": "RADIOBUTTON",
"isanswer": true,
"optionText": "ch1"
}
]
},
{
"questionId": "556708425215763c64b8af3d",
"questionText": "question no 2",
"questionTitle": "",
"questionType": "simple",
"question": true,
"questionOptions": [
{
"questionRefId": 0,
"optionType": "RADIOBUTTON",
"isanswer": true,
"optionText": "ch1"
}
],
},
{
"questionId": "556708425215763c64b8af3d",
"questionText": "question no 3",
"questionTitle": "",
"questionType": "hard",
"questionOptions": [
{
"questionRefId": 0,
"optionType": "RADIOBUTTON",
"isanswer": true,
"optionText": "ch1"
},
{
"questionRefId": 0,
"optionType": "RADIOBUTTON",
"isanswer": false,
"optionText": "ch2"
}
],
}
]
}
}
EDIT-- Solution by using Java driver as per Sylvain Leroux
BasicDBObject matchSurvey = new BasicDBObject();
matchSurvey.put("$match", new BasicDBObject("json.surveyRefId", new BasicDBObject("$in", new Integer[]{377,360})));
BasicDBObject unwind = new BasicDBObject();
unwind.put("$unwind", "$json.questionsVoList");
BasicDBObject matchQuestion = new BasicDBObject();
matchQuestion.put("$match", new BasicDBObject("json.questionsVoList.questionType", "hard"));
HashMap map = new HashMap();
map.put("_id", "$_id");
map.put("questionsVoList", new BasicDBObject("$push", "$json.questionsVoList"));
BasicDBObject group = new BasicDBObject();
group.put("$group",map);
List<BasicDBObject> list = new ArrayList<BasicDBObject>();
list.add(matchSurvey);
list.add(unwind);
list.add(matchQuestion);
list.add(group);
AggregateIterable output = collection.aggregate(list, DBObject.class);
"find sub-documents having questionType = "hard"" can be understood in three different ways:
All documents having a "hard" query
If you only want all documents having an "hard query, you will use find and $elemMatch:
db.test.find({"json.surveyRefId": { $in: [377, 360]},
"json.questionsVoList": {$elemMatch: {"questionType":"hard"}}})
First "hard" query of a document
If you need to find the first "hard" query of a document, you will combine the above query with the $ projection operator:
db.test.find({"json.surveyRefId": { $in: [377, 360]},
"json.questionsVoList": {$elemMatch: {"questionType":"hard"}}}
{"json.surveyRefId":1, "json.questionsVoList.$":1})
All hard queries
If you need to find all "hard" queries of the document, you will have to use the aggregation framework:
db.test.aggregate({$match: { "json.surveyRefId": { $in: [377, 360]} }},
{$unwind: "$json.questionsVoList"},
{$match: { "json.questionsVoList.questionType": "hard"}},
{$group: {_id: "$_id", questionsVoList: {$push: "$json.questionsVoList"}}}
)
The first $match step will filter out unwanted documents based on their surveyRefId
Then $unwind will produce a document for each sub-document
An other $match filters out unwanted documents based on questionType
Finally, a $group will combine all sub-documents back as one for a given _id
Producing:
{
"_id" : ObjectId("556828d002509ae174742d11"),
"questionsVoList" : [
{
"questionId" : "556708425215763c64b8af3d",
"questionText" : "question no 1",
"questionTitle" : "",
"questionType" : "hard",
"preQuestion" : true,
"questionOptions" : [
{
"questionRefId" : 0,
"optionType" : "RADIOBUTTON",
"isanswer" : true,
"optionText" : "ch1"
}
]
},
{
"questionId" : "556708425215763c64b8af3d",
"questionText" : "question no 3",
"questionTitle" : "",
"questionType" : "hard",
"questionOptions" : [
{
"questionRefId" : 0,
"optionType" : "RADIOBUTTON",
"isanswer" : true,
"optionText" : "ch1"
},
{
"questionRefId" : 0,
"optionType" : "RADIOBUTTON",
"isanswer" : false,
"optionText" : "ch2"
}
]
}
]
}