Consider collection with whitespace in field, in DB if there is white space if we group them "Ravi", " Ravi " , consider as total 2 values. but it should consider as single value. So I have trim in Group. I had performed in DB. But I don't know how can I trim and group value in Springboot. We can use Aggregation for this in Springboot. but don't how to use trim. Kindly help on this
Sample Collection:
[
{
"name": "ravi",
"DOB": "04-02-2000",
"Blood": "A1+"
},
{
"name": "ravi ",
"DOB": "05-03-2000",
"Blood": "A1+"
},
{
"name": "kumar ",
"DOB": "02-04-2000",
"Blood": "A1+"
},
{
"name": "kumar",
"DOB": "03-05-2000",
"Blood": "A1+"
}
]
MongoDB Operation:
db.collection.aggregate({
"$group": {
_id: {
$trim: {
input: "$name"
}
},
doc: {
"$first": "$$ROOT",
}
}
},
{
"$replaceRoot": {
"newRoot": "$doc"
}
})
Output:
[
{
"Blood": "A1+",
"DOB": "04-02-2000",
"_id": ObjectId("5a934e000102030405000000"),
"name": "ravi"
},
{
"Blood": "A1+",
"DOB": "02-04-2000",
"_id": ObjectId("5a934e000102030405000002"),
"name": "kumar "
}
]
Not possible via standard API.
Workaround: We need to add extra $addFields to trim the name field before applying $group stage.
#Autowired
private MongoTemplate template;
...
Aggregation agg = Aggregation.newAggregation(
Aggregation.addFields()
.addFieldWithValue("name", Trim.valueOf("name")).build(),
Aggregation.group("name")
.first("$$ROOT").as("doc"),
Aggregation.replaceRoot("doc")
);
...
template.aggregate(agg, inputType, outputType);
Note: Since you were manipulating the name field in the first stage, MongoDB wouldn't use indexes, so we can add an extra stage to the pipeline.
Related
Hi everyone I have a collection of documents like bellow. I want to directly get "rights" from roles array for params: _id, groups._id, roles._id using java mongo driver.
{
"_id": 1000002,
"groups": [
{
"_id": 1,
"roles": [
{
"rights": 3,
"_id": 1
},
{
"rights": 7,
"_id": 2
},
{
"rights": 3,
"_id": 3
}
]
}
],
"timestamp": {
"$date": {
"$numberLong": "1675267318028"
}
},
"users": [
{
"accessProviderId": 1,
"rights": 1,
"_id": 4
},
{
"accessProviderId": 1,
"rights": 3,
"_id": 5
}
]
}
I have AccessListItem class which represents this document and I have used Bson filters to get it from mongo, but after fetching i had to get information through java function.. I want to get int value directly from mongo base.
Bson fileFilter = Filters.eq("_id", itemId);
Bson groupFilter = Filters.elemMatch("groups", Document.parse("{_id:"+groupId+"}"));
Bson roleFilter = Filters.elemMatch("groups.roles", Document.parse("{_id:"+role+"}"));
Bson finalFilter = Filters.and(fileFilter, Filters.and(groupFilter,roleFilter));
MongoCollection<AccessListItem> accessListItemMongoCollection = MongoUtils.getAccessCollection(type);
AccessListItem accessListItem = accessListItemMongoCollection.find(finalFilter).first();
The short answer is you can't.
MongoDB is designed for returning documents, that is, objects containing key-value pairs. There is no mechanism for a MongoDB query to return just a value, i.e. it will never return just 3 or [3].
You could use aggregation with a $project stage at the end to give you a simplified object like:
{ rights: 3}
In javascript that might look like:
db.collection.aggregate([
{$match: {
_id: itemId,
"groups._id": groupId,
"groups.roles._id": role
}},
{$project: {
_id: 0,
group: {
$first: {
$filter: {
input: "$groups",
cond: {$eq: ["$$this._id",groupId]}
}
}
}
}},
{$project: {
"roles": {
$first: {
$filter: {
input: "$group.roles",
cond: { $eq: [ "$$this._id",role]}
}
}
}
}},
{$project: {
rights: "$roles.rights"
}}
])
Example: Playground
I'm not familiar with spring boot, so I'm not sure what that would look like in Java.
I have an existing collection, containing several documents.
[{
"_id": "...1",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"value": "sub element 1.1"
},
{
"value": "sub element 1.2"
},
{
"value": "sub element 1.3"
}
]
}, {
"_id": "...2",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"value": "sub element 2.1"
},
{
"value": "sub element 2.2"
}
]
}, // many others here...
]
For each root document, I would like to add an _id property of type ObjectId on each sub-element of someArray. So, after I run my command, the content of the collection is the following:
[{
"_id": "...1",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"_id": ObjectId("..."),
"value": "sub element 1.1"
},
{
"_id": ObjectId("..."),
"value": "sub element 1.2"
},
{
"_id": ObjectId("..."),
"value": "sub element 1.3"
}
]
}, {
"_id": "...2",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"_id": ObjectId("..."),
"value": "sub element 2.1"
},
{
"_id": ObjectId("..."),
"value": "sub element 2.2"
}
]
}, // ...
]
Each ObjectId being, of course, unique.
The closer I got was with this:
db.getCollection('myCollection').updateMany({}, { "$set" : { "someArray.$[]._id" : ObjectId() } });
But every sub-element of the entire collection ends up with the same ObjectId value...
Ideally, I need to get this working using Java driver for MongoDB. The closest version I got is this (which presents the exact same problem: all the ObjectId created have the same value).
database
.getCollection("myCollection")
.updateMany(
Filters.ne("someArray", Collections.emptyList()), // do not update empty arrays
new Document("$set", new Document("someArray.$[el]._id", "ObjectId()")), // set the new ObjectId...
new UpdateOptions().arrayFilters(
Arrays.asList(Filters.exists("el._id", false)) // ... only when the _id property doesn't already exist
)
);
With MongoDB v4.4+, you can use $function to use javascript to assign the _id in the array.
db.collection.aggregate([
{
"$addFields": {
"someArray": {
$function: {
body: function(arr) {
return arr.map(function(elem) {
elem['_id'] = new ObjectId();
return elem;
})
},
args: [
"$someArray"
],
lang: "js"
}
}
}
}
])
Here is the Mongo playground for your reference. (It's slightly different from the code above as playground requires the js code to be in double quote)
For older version of MongoDB, you will need to use javascript to loop the documents and update them one by one.
db.getCollection("...").find({}).forEach(function(doc) {
doc.someArray = doc.someArray.map(function(elem) {
elem['_id'] = new ObjectId();
return elem;
})
db.getCollection("...").save(doc);
})
Here is what I managed to write in the end:
MongoCollection<Document> collection = database.getCollection("myCollection");
collection
.find(Filters.ne("someArray", Collections.emptyList()), MyItem.class)
.forEach(item -> {
item.getSomeArray().forEach(element -> {
if( element.getId() == null ){
collection.updateOne(
Filters.and(
Filters.eq("_id", item.getId()),
Filters.eq("someArray.value", element.getValue())
),
Updates.set("someArray.$._id", new ObjectId())
);
}
});
});
The value property of sub-elements had to be unique (and luckily it was). And I had to perform separate updateOne operations in order to obtain a different ObjectId for each element.
I have a bit of a complex query of view creation using 3 collections. The query is written in the native level. I need that query to be executed from Java and is there any way that I can execute these types of queries from Java level. Maybe a function that takes a MongoDB native query as a string and executes that on the database level
db.createView('TARGET_COLLECTION', 'SOURCE_COLLECTION_1', [
{
$facet: {
SOURCE_COLLECTION_1: [
{$match: {}},
{ $project: { "sourceId": {$toString: "$_id"}, "name": 1, "image": "$logo" }}
],
SOURCE_COLLECTION_2: [
{$limit: 1},
{
$lookup: {
from: 'SOURCE_COLLECTION_2',
localField: '__unexistingfield',
foreignField: '__unexistingfield',
as: '__col2'
}
},
{$unwind: '$__col2'},
{$replaceRoot: {newRoot: '$__col2'}},
{ $project: { "sourceId": {$toString: "$_id"}, "name": 1, "image": 1 }}
],
SOURCE_COLLECTION_3: [
{$limit: 1},
{
$lookup: {
from: 'SOURCE_COLLECTION_3',
localField: '__unexistingfield',
foreignField: '__unexistingfield',
as: '__col2'
}
},
{$unwind: '$__col2'},
{$replaceRoot: {newRoot: '$__col2'}},
{ $project: { "sourceId": {$toString: "$_id"}, "name": 1, "image": "$logo" }}
]
},
},
{$project: {data: {$concatArrays: ['$SOURCE_COLLECTION_1', '$SOURCE_COLLECTION_2', '$SOURCE_COLLECTION_3']}}},
{$unwind: '$data'},
{$replaceRoot: {newRoot: '$data'}}
])
An example:
Consider a document in a collection:
{ _id: 1234, name: "J. Doe", colors: [ "red", "black" ] }
And the following aggregation from the mongo shell:
db.collection.agregate( [
{ $project: { _id: 0, colors: 1 } }
] )
This returns: { "colors" : [ "red", "black" ] }
This can also be run with the following command:
db.runCommand( {
aggregate: "collection",
pipeline: [ { $project: { _id: 0, colors: 1 } } ],
cursor: { }
} )
And, its translation using Spring Data's MongoTemplate:
String jsonCommand = "{ aggregate: 'collection', pipeline: [ { $project: { _id: 0, colors: 1 } } ], cursor: { } }";
Document resultDoc = mongoTemplate.executeCommand(jsonCommand);
The output document resultDoc has a format like the following:
{
"cursor" : {
"firstBatch" : [
{
"colors" : [
"red",
"black"
]
}
],
"id" : NumberLong(0),
"ns" : "test.colors"
},
"ok" : 1
}
To know more about the db.runCommand(...) method see MongoDB documentation at: Database Commands and Database Command Aggregate.
I started working with MongoDB. I prepared some basic training JSON:
{
"header": {
"Hotel": {
"data": [
{
"name": "Hilton",
"Id": "1231213421"
}
]
},
"standard": "5",
"priceStage": "4"
},
"data": {
"http": {
"strean": {}
}
}
}
and I wrote a query like this:
db.hotel.find( "data": { Id: "1231213421"})
Why query does not return anything?
You're trying to match on an element within an array and you don't need to match on the entire array. So,something like the following will work:
db.hotel.find({"header.Hotel.data": {"$elemMatch": {"Id": "1231213421"}}} );
I have a collection of events its structure is as follows :
{
"_id" : ObjectId("537b3ff288f4ca2f471afcae"),
"Name" : "PREMISES MAP DELETED",
"ScreenName" : "AccessPointActivity",
"Timestamp" : NumberLong("1392113758000"),
"EventParams" : "null",
"TracInfo" : {
"ApplicationId" : "fa41f204bfc711e3b9f9c8cbb8c502c4",
"DeviceId" : "2_1VafJVPu4yfdbMWO1XGROjK6iQZhq4hAVCQL837W",
"UserId" : "pawan",
"SessionId" : "a8UHE16mowNwNGyuLXbW",
"WiFiAP" : "null",
"WiFiStrength" : 0,
"BluetoothID" : "null",
"BluetoothStrength" : 0,
"NetworkType" : "null",
"NetworkSubType" : "null",
"NetworkCarrier" : "Idea",
"Age" : 43,
"Gender" : "Female",
"OSVersion" : "16",
"Manufacturer" : "samsung",
"Resolution" : "600*976",
"Platform" : "Android",
"Latitude" : 40.42,
"Longitude" : -74,
"City" : "Monmouth County",
"CityLowerCase" : "monmouth county",
"Country" : "United States",
"CountryLowerCase" : "united states",
"Region" : "New Jersey",
"RegionLowerCase" : "new jersey",
"Time_zone" : "null",
"PinCode" : "07732",
"Locale" : ", Paradise Trailer Park",
"Accuracy" : 0,
"Timestamp" : NumberLong("1392113758000")
}
}
their are many event on different screens.
My expected output is as follows :
{
ApplicationId:"fa41f204bfc711e3b9f9c8cbb8c502c4",
EventName:"PREMISES MAP DELETED",
Eventcount:300,
ScreenviewCount:20,
DeviceCount:10,
UserCount:3
}
EventCount : It is count of EventName
ScreenviewCount : It is the count of distinct screenName distinct per session
DeviceCount : It is the count of distinct deviceId
UserCount : It is the count of distinct userCount
Their will be multiple event on multiple screens(ScreenName).
Currently i' am using following approach :
Using aggregation to get each event name and it count
eg :
{
_id:
{
ApplicationId:"fa41f204bfc711e3b9f9c8cbb8c502c4",
EventName:"PREMISES MAP DELETED"
}
EventCount:300
}
For each event name from above aggregation result I call following queries in while loop until aggregation output has documents:
a) Distinct query using eventName from aggregation output for screenview count(on event collection).
b) Distinct query eventName from aggregation output for device count(on event collection).
c) Distinct query eventName from aggregation output for user count(on event collection).
And the problem is its slow as it has 3 distinct queries on each result of aggregation output.
Is their any way to do it in single aggregation call or something else.
Thank you in advance!!!
The general case here that you seemed to have missed is that to get the "distinct" values of various fields in your document under the "event" totals you can use the $addToSet operator.
A "set" by definition has all of it's values "unique/distinct", so you just want to hold all those possible values in a "set" for your grouping level and then get the "size" of the array produced, which is exactly what the $size operator introduced in MongoDB 2.6 does.
db.collection.aggregate([
{ "$group": {
"_id": {
"ApplicationId": "$TracInfo.ApplicationId",
"EventName": "$Name",
},
"oScreenViewCount": {
"$addToSet": {
"ScreenName": "$ScreenName",
"SessionId": "$TracInfo.SessionId",
}
},
"oDeviceCount": { "$addToSet": "$TracInfo.DeviceId" },
"oUserCount": { "$addToSet": "$TracInfo.UserId" },
"oEventcount": { "$sum": 1 }
}},
{ "$project": {
"_id": 0,
"ApplicationId": "$_id.ApplicationId",
"EventName": "$_id.EventName",
"EventCount": "$oEventCount",
"ScreenViewCount": { "$size": "$oScreenViewCount" },
"DeviceCount": { "$size": "$oDeviceCount" },
"UserCount": { "$size": "$oUserCount" }
}}
])
Versions pre MongoDB 2.6 require a little more work, using $unwind and $group to count the arrays:
db.collection.aggregate([
{ "$group": {
"_id": {
"ApplicationId": "$TracInfo.ApplicationId",
"EventName": "$Name",
},
"oScreenviewCount": {
"$addToSet": {
"ScreenName": "$ScreenName",
"SessionId": "$TracInfo.SessionId",
}
},
"oDeviceCount": { "$addToSet": "$TracInfo.DeviceId" },
"oUserCount": { "$addToSet": "$TracInfo.UserId" },
"oEventcount": { "$sum": 1 }
}},
{ "$unwind": "$oScreeenviewCount" },
{ "$group": {
"_id": "$_id",
"oScreenviewCount": { "$sum": 1 },
"oDeviceCount": { "$first": "$oDeviceCount" },
"oUserCount": { "$first": "$oUserCount" },
"oEventcount": { "$first": "$oEventCount" }
}},
{ "$unwind": "$oDeviceCount" },
{ "$group": {
"_id": "$_id",
"oScreenviewCount": { "$first": "$oScreenViewCount" },
"oDeviceCount": { "$sum": "$oDeviceCount" },
"oUserCount": { "$first": "$oUserCount" },
"oEventcount": { "$first": "$oEventCount" }
}},
{ "$unwind": "$oUserCount" },
{ "$group": {
"_id": "$_id",
"oScreenviewCount": { "$first": "$oScreenViewCount" },
"oDeviceCount": { "$first": "$oDeviceCount" },
"oUserCount": { "$sum": "$oUserCount" },
"oEventcount": { "$first": "$oEventCount" }
}},
{ "$project": {
"_id": 0,
"ApplicationId": "$_id.ApplicationId",
"EventName": "$_id.EventName",
"EventCount": "$oEventCount",
"ScreenViewCount": "$oScreenViewCount",
"DeviceCount": "$oDeviceCount",
"UserCount": "$oUserCount"
}}
])
The end usage of $project in the second listing and all the general usage of the "o" prefixed names is really just for prettying up the result at the end and making sure the output field order is the same as in your sample result.
As a general disclaimer, your question lacks the information to determine the exact fields or combinations that are used for these totals, but the principles and approach are sound and should be near enough to the same implementation.
So essentially, you are getting the "distinct" values within the "group" by using $addToSet for whatever the field or combination is and then you are determining the "count" of those "sets" by whatever means is available to you.
Much better than issuing many queries and merging results in client code.