Develop Reference

How can I find the number of duplicates for a field using MongoDB Java?

How can I find the number of duplicates in each document in Java-MongoDB
I have collection like this.
Collection example:
{
"_id": {
"$oid": "5fc8eb07d473e148192fbecd"
},
"ip_address": "192.168.0.1",
"mac_address": "00:A0:C9:14:C8:29",
"url": "https://people.richland.edu/dkirby/141macaddress.htm",
"datetimes": {
"$date": "2021-02-13T02:02:00.000Z"
}
{
"_id": {
"$oid": "5ff539269a10d529d88d19f4"
},
"ip_address": "192.168.0.7",
"mac_address": "00:A0:C9:14:C8:30",
"url": "https://people.richland.edu/dkirby/141macaddress.htm",
"datetimes": {
"$date": "2021-02-12T19:00:00.000Z"
}
}
{
"_id": {
"$oid": "60083d9a1cad2b613cd0c0a2"
},
"ip_address": "192.168.1.5",
"mac_address": "00:0A:05:C7:C8:31",
"url": "www.facebook.com",
"datetimes": {
"$date": "2021-01-24T17:00:00.000Z"
}
}
example query:
BasicDBObject whereQuery = new BasicDBObject();
DBCursor cursor = table1.find(whereQuery);
while (cursor.hasNext()) {
DBObject obj = cursor.next();
String ip_address = (String) obj.get("ip_address");
String mac_address = (String) obj.get("mac_address");
Date datetimes = (Date) obj.get("datetimes");
String url = (String) obj.get("url");
System.out.println(ip_address, mac_address, datetimes, url);
}
in Java, How I can know count duplicated data of "url". And how many of duplicated.

in mongodb you can solve this problem with "Aggregation Pipelines". You need to implement this pipeline in "Mongodb Java Driver". It gives only duplicated results with their duplicates count.
db.getCollection('table1').aggregate([
{
"$group": {
// group by url and calculate count of duplicates by url
"_id": "$url",
"url": {
"$first": "$url"
},
"duplicates_count": {
"$sum": 1
},
"duplicates": {
"$push": {
"_id": "$_id",
"ip_address": "$ip_address",
"mac_address": "$mac_address",
"url": "$url",
"datetimes": "$datetimes"
}
}
}
},
{ // select documents that only duplicates count higher than 1
"$match": {
"duplicates_count": {
"$gt": 1
}
}
},
{
"$project": {
"_id": 0
}
}
]);
Output Result:
{
"url" : "https://people.richland.edu/dkirby/141macaddress.htm",
"duplicates_count" : 2.0,
"duplicates" : [
{
"_id" : ObjectId("5fc8eb07d473e148192fbecd"),
"ip_address" : "192.168.0.1",
"mac_address" : "00:A0:C9:14:C8:29",
"url" : "https://people.richland.edu/dkirby/141macaddress.htm",
"datetimes" : {
"$date" : "2021-02-13T02:02:00.000Z"
}
},
{
"_id" : ObjectId("5ff539269a10d529d88d19f4"),
"ip_address" : "192.168.0.7",
"mac_address" : "00:A0:C9:14:C8:30",
"url" : "https://people.richland.edu/dkirby/141macaddress.htm",
"datetimes" : {
"$date" : "2021-02-12T19:00:00.000Z"
}
}
]
}

If I understand your question correctly you're trying to find the amount of duplicate entries for the field url. You could iterate over all your documents and add them to a Set. A Set has the property of only storing unique values. When you add your values, the ones that are already in the Set will not be added again. Thus the difference of the number of entries in the Set to the number of documents is the amount of duplicate entries for the given field.
If you wanted to know which URLs are non-unique, you could evaluate the return value from Set.add(Object) which will tell you, whether or not the given value has been in the Set beforehand. If it has, you got yourself a duplicate.

Need to update only particular element in an array of mongo in java

I want to update a particular element in an array of mongo with filters.
I have already tried having using BasicDBObject, but than not able to use filter as i have collection of type MongoCollection.
MongoCollection collection = db.getCollection(tnId);
Even tried:
FindIterable<Document> document = collection.find(new BasicDBObject("DocumentName", documentName) .append("Attributes.name", "Party 1" ).append("Attributes.value", 12) .append("Attributes.actualValue.initialValue", USA));
But in this case What I get is the whole record and than I have to iterate through each Attribute again.
mongo = new MongoClient("localhost", 27017);
MongoDatabase db = mongo.getDatabase(companyName);
MongoCollection collection = db.getCollection(tnId);
Actual data which I am passing is
{
"DocumentName" : "doc1",
"Attributes" : [
{
"name" : "Party 1",
"value" : 12,
"actualValue" : {
"initialValue" : "USA"
}
},
{
"name" : "Party 1",
"value" : 16,
"actualValue" : {
"initialValue" : "SYSTEM"
}
}
]
}
and I want to search attribute where actualValue is "USA" and Value of attribute is 12 and updated data should look like
{
"DocumentName" : "doc1",
"Attributes" : [
{
"name" : "Party 1",
"value" : 12,
"actualValue" : {
"initialValue" : "USA"
},
"updatedvalue" : {
"initialValue" : "USA",
"changedValue" : "Europe"
}
},
{
"name" : "Party 1",
"value" : 16,
"actualValue" : {
"initialValue" : "SYSTEM"
}
}
]
}

db.collection.findOneAndUpdate({ "Attributes.actualValue.initialValue": "USA" },
{ $set: { "Attributes.$.updatedvalue.initialValue": "USA", "Attributes.$.updatedvalue.changedValue": "Europe" } })
db.collection.findOneAndUpdate({ "Attributes.actualValue.initialValue": "USA" },
{ $set: { "Attributes.$.updatedvalue.initialValue": "India", "Attributes.$.updatedvalue.changedValue": "India" } })

Try with this one
db.collection.updateMany(
{ "Attributes.actualValue.initialValue": "USA" },
{ $set: { "Attributes.$.updatedvalue" : {
"initialValue" : "USA",
"changedValue" : "Europe"
} } }
)

How I did is. I made a document where i added teh required data which i want to update it with.Say
Document valueDocument = new Document();
valueDocument.put("InitialValue", USA);
BasicDBObject query = new BasicDBObject();
query.put("Attributes", att);
BasicDBObject data = new BasicDBObject();
data.put("Attributes.$.updatedvalue" + qCLevel, valueDocument);
BasicDBObject command = new BasicDBObject();
command.put("$set", data);
collection.updateOne(query, command);
It goes and insert the data in the right place.

How to use MongoDB Java driver to group by dayOfYear on ISODate attributes?

How to use mongodb java driver to compare dayOfYear of two ISODate objects?
Here are my docs
{"name": "hello", "count": 4, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "hello", "count": 5, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "goodbye", "count": 6, "TIMESTAMP": ISODate("2017-10-01T02:00:35.098Z")}
{"name": "foo", "count": 6, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
I want to compare the day in "TIMESTAMP" to perform some aggregation
Bson match = Aggregates.match(eq("name": "hello"));
Bson group = Aggregates.group(new Document("name", "$name"), Accumulators.sum("total", 1));
collection.aggregate(Arrays.asList(match, group))
Now I am not sure how to do this aggregation for all the records that belongs to particular day?
so my expected result for "2017-10-02" is
[{"_id": {"name":"hello"}, "total": 9}, {"_id": {"name":"foo"}, "total": 6}]

Given the following documents:
{"name": "hello", "count": 4, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "hello", "count": 5, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "goodbye", "count": 6, "TIMESTAMP": ISODate("2017-10-01T02:00:35.098Z")}
{"name": "foo", "count": 6, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
The following command ...
db.getCollection('dayOfYear').aggregate([
// project dayOfYear as an attribute
{ $project: { name: 1, count: 1, dayOfYear: { $dayOfYear: "$TIMESTAMP" } } },
// match documents with dayOfYear=275
{ $match: { dayOfYear: 275 } },
// sum the count attribute for the selected day and name
{ $group : { _id : { name: "$name" }, total: { $sum: "$count" } } }
])
... will return:
{
"_id" : {
"name" : "foo"
},
"total" : 6
}
{
"_id" : {
"name" : "hello"
},
"total" : 9
}
I think this meets the requirement expressed in your OP.
Here's the same command expressed using the MongoDB Java driver:
MongoCollection<Document> collection = mongoClient.getDatabase("stackoverflow").getCollection("dayOfYear");
Document project = new Document("name", 1)
.append("count", 1)
.append("dayOfYear", new Document("$dayOfYear", "$TIMESTAMP"));
Document dayOfYearMatch = new Document("dayOfYear", 275);
Document grouping = new Document("_id", "$name").append("total", new Document("$sum", "$count"));
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(
new Document("$project", project),
new Document("$match", dayOfYearMatch),
new Document("$group", grouping)
));
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Update based on this comment:
One of the problems with project is that it only include fields you specify . The above input is just an example. I have 100 fields in my doc I can't sepecify every single one so if I use project I have to specify all 100 fields in addition to "dayOfYear" field. – user1870400 11 mins ago
You can use the following command to return the same output but without a $project stage:
db.getCollection('dayOfYear').aggregate([
// ignore any documents which do not match dayOfYear=275
{ "$redact": {
"$cond": {
if: { $eq: [ { $dayOfYear: "$TIMESTAMP" }, 275 ] },
"then": "$$KEEP",
"else": "$$PRUNE"
}
}},
// sum the count attribute for the selected day
{ $group : { _id : { name: "$name" }, total: { $sum: "$count" } } }
])
Here's that command in its 'Java form':
MongoCollection<Document> collection = mongoClient.getDatabase("stackoverflow").getCollection("dayOfYear");
Document redact = new Document("$cond", new Document("if", new Document("$eq", Arrays.asList(new Document("$dayOfYear", "$TIMESTAMP"), 275)))
.append("then", "$$KEEP")
.append("else", "$$PRUNE"));
Document grouping = new Document("_id", "$name").append("total", new Document("$sum", "$count"));
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(
new Document("$redact", redact),
new Document("$group", grouping)
));
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Note: Depending on the size of your collection/your non functional requirements/etc you may want to consider the performance of these solutions and either (a) add a match stage before you start projecting/redacting or (b) extract dayOfYear into its own attribute so that you can avoid this complexity entirely.

How to push json data into an arrya object in MONGO DB by java

"termsrelation" : [
{
"rel": "RT",
"terms": [
{
"objid": "55dc25083d2cbcb8b0dc48c8",
"source": null,
"scopeNote": [
{
"sourceType": "source",
"source": "abc"
}
],
"formated_name": "Milkbuns",
"name": "Milkbuns",
"type": null,
"url": "test.com",
"nodeid": "14050"
},
{
"objid": "552cae1940feb9123e3f5fb6",
"source": null,
"scopeNote": [
{
"sourceType": "source",
"source": "xyz"
}
],
"formated_name": "Milkchocolate",
"name": "Milkchocolate",
"type": null,
"url": "test.com",
"nodeid": "193570"
}
]
}
]
I need to push
{
"sourceType" : "source",
"source" : "def"
}
json to scopeNote object where name = Milkchocolate by java, Please help me out
I tried this query { "$push" : { "termsrelation.$.terms.$.scopeNote" : { "sourceType" : "source" , "source" : "ddd" }}} with search query { "$and" : [ { "id" : "4003"} , { "termsrelation.rel" : "RT"} , { "termsrelation.terms.scopeNote.sourceType" : "source"} , { "termsrelation.terms.scopeNote.source" : "xyz"} , { "termsrelation.terms.name" : "Milk chocolate"}]} but didn't work

Step 1. First get the position for element there which you want to insert
Step 2. You got the position
then use this query
List<BasicDBObject> andQuery2 = new ArrayList<BasicDBObject>();
andQuery2.add(new BasicDBObject("id", id));
andQuery2.add(new BasicDBObject("termsrelation.rel", relation));
andQuery2.add(new BasicDBObject("termsrelation.terms.name", name));
BasicDBObject searchObj2 = new BasicDBObject();
searchObj2.put("$and", andQuery2);
BasicDBObject basicDBObject = new BasicDBObject();
basicDBObject.put("sourceType", type);
basicDBObject.put("source", source);
BasicDBObject updateQuery = new BasicDBObject();
updateQuery.append("$push", new BasicDBObject().append("termsrelation.$.terms."+position+".scopeNote", basicDBObject));
coll.update(searchObj2, updateQuery).getError();

mongotemplate aggregation with condition

I have a collection where the documents look like this:
{
_id: "545b9fa0dd5318a4285f7ce7",
owner: "admin",
messages: [
{
id: "100",
status: "sent",
note: ""
},
{
id: "100",
status: "pending",
note: ""
},
{
id: "101",
status: "sent",
note: ""
},
{
id: "102",
status: "sent",
note: ""
},
{
id: "101",
status: "done",
note: ""
}
]
}
(This is just a short example, in my case the sub array is very large)
I need to query the collection and get some statistics for a specific document.
So in this example if I query the doucment that has id: "545b9fa0dd5318a4285f7ce7"
I should get this result:
{
sent: 3,
pending: 1,
done: 1
}
How can I do such aggregation with spring mongotemplate ?

To do this sort of thing you need the $cond operator in the aggregation framework. Spring Data MongoDB does not do that yet, and there are many things lacking from common $group operations that are even only implemented under $project.
The issue for tracking implementation of any $cond support is here:
https://jira.spring.io/browse/DATAMONGO-861
For the rest of the world it looks like this:
db.collection.aggregate([
{ "$match": { "_id": ObjectId("545b9fa0dd5318a4285f7ce7") } },
{ "$unwind": "$messages" },
{ "$group": {
"_id": "$_id",
"sent": {
"$sum": {
"$cond": [
{ "$eq": [ "$mesages.status", "sent" ] },
1,
0
]
}
},
"pending": {
"$sum": {
"$cond": [
{ "$eq": [ "$messages.status", "pending" ] },
1,
0
]
}
},
"done": {
"$sum": {
"$cond": [
{ "$eq": [ "$messages.status", "done" ] },
1,
0
]
}
}
}}
])
To get that sort of thing to work under a mongotemplate aggregation you need a class extending aggregation operations that can be built from a DBObject:
public class CustomGroupOperation implements AggregationOperation {
private DBObject operation;
public CustomGroupOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
Then you can define the "$group" as a DBObject and implement in the aggregation pipeline:
DBObject myGroup = (DBObject)new BasicDBObject(
"$group", new BasicDBObject(
"_id","$_id"
).append(
"sent", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$messages.status", "sent"}
),
1,
0
}
)
)
).append(
"pending", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$messages.status", "pending"}
),
1,
0
}
)
)
).append(
"done", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$messages.status", "done"}
),
1,
0
}
)
)
)
);
ObjectId myId = new ObjectId("545b9fa0dd5318a4285f7ce7");
Aggregation aggregation = newAggregation(
match(Criteria.where("_id").is(myId)),
unwind("messges"),
new CustomGroupOperation(myGroup)
);
That allows you to come up with a pipeline that is basically the same as the shell representation above.
So it would seem that for now, where certain operations and sequences are not supported the best case is to implement a class on the AgggregationOperation interface that can be fed a DBObject or otherwise internally construct one from your own custom methods.

You can use following aggregation :
db.collection.aggregate(
{ $match : { "_id" : ObjectId("545b9fa0dd5318a4285f7ce7") } },
{ $unwind : "$messages" },
{ $group : { "_id" : "$messages.status", "count" : { $sum : 1} } }
)
it will give you the count of status for which message are available, all other status count should be consider 0.