How to create a super set of fields(except _id) present in all the documents in a single collection of MongoDB assuming that all documents are of same type but different number of fields.
Example :
doc1 - {"_id":"test1", "firstName":"sample1", "age":24, "state":"Kansas"}
doc2 - {"_id":"test2", "lastName":"sample2", "age":24, "country":"US"}
Super set would be : {"firstName", "lastName", "age", "state", "country"}
You could try running a mapReduce operation that will return all the given document keys in the input collection as the _id key of the resulting document. This will be in an output collection where you can then apply the distinct command on the _id field to retrieve the superset of the fields.
The example that follows shows this concept:
// Run mapReduce on collectionName
String map = "function () { for (var key in this) { emit(key, null); } }";
String reduce = "function () {}";
MapReduceResults<ValueObject> results = mongoTemplate.mapReduce(
"collectionName",
map,
reduce,
new MapReduceOptions().outputCollection("col_out"),
ValueObject.class
);
// Get the distinct keys from output collection col_out ---
List<String> fieldsSuperset = mongoTemplate.getCollection("col_out").distinct("_id");
Related
I am a little confused as to how to add an element to an array in an exisiting mongodb document, or why my results are not showing properly and how I expect.
There is only one document in the collection and will only ever be one. The mongo document looks like when I do a db.collection-name.find.pretty() command in a mongo session on the command line:
{
"_id" : ObjectID("1234567890"),
"details" : {
...
},
"calculations" : [
{
"count" : 1,
"total" : 10,
"mean" : 2.5
},
{
"count" : 2,
"total" : 20,
"mean" : 6.4
}
]
}
I want to add another object to the calculations list.
The Java code I am running is based upon THIS example:
// Get the database and collection
MongoDatabase database = mongo.getDatabase(dataBaseName);
MongoCollection<Document> collection = database.getCollection(collectionName);
Document document = collection.find().first(); // will only ever be one document
// The object comes in as a Map
Map<String, Object> incomingMap = new HashMap<>();
incomingMap.put("count", 3);
incomingMap.put("total", 4);
incomingMap.put("mean", 7.9);
// convert to a Document
Document newDocument = new Document();
incomingMap.forEach((k, v) -> {
newDocument.append(k, v);
});
// append this to the collection - this is where I am confused
// for this example just hardcoding the _id value for simplicity
collection.updateOne(new Document("_id", "1234567890"), Updates.push("calculations", newDocument));
However when I do a System.out.println(collection.find().first()) in the code after this or db.collection-name.find.pretty() in a mongo session the new document has not been added. There are no errors thrown and completes fine.
What I am wondering is
Is the line collection.updateOne(new Document("_id", "1234567890"), Updates.push("calculations", newDocument)); correct?
Has it been added but not been saved - if so how do I save?
Can I do this at a document level, for example document.update(new Documen(Updates.push("calculations", newDocument)); or similar?
I have also tried collection.findAndUpdateOne(new Document("_id", "1234567890"), Updates.push("calculations", newDocument)); with the same result
Is how I am getting/hardcoding the document ID incorrect?
You have filter condition issue (your _id is ObjectId type)
new Document("_id", ObjectId("1234567890"))`
Always make sure your documents updated correctly. Look code fagment:
UpdateResult result = collection.updateOne(filter, update);
log.info("Update with date Status : " + result.wasAcknowledged());
log.info("Nº of Record Modified : " + result.getModifiedCount());
https://api.mongodb.com/java/3.1/com/mongodb/client/result/UpdateResult.html
I got a JSON string that looks something like this:
String tmp = "[
{
"ID":"12",
"Date":"2018-02-02",
"ObjData":[
{
"Name":"AAA",
"Order":"12345",
"Extra1":{
"Temp":"3"
},
"Extra2":{
"Temp":"5"
}
},
{
"Name":"BBB",
"Order":"54321",
"Extra1":{
"Temp":"3"
},
"Extra2":{
"Temp":"5"
}
}
]
}
]"
I would like to remove for example the the document where ´Order´ equals "54321" from ´ObjData´. I got the following code:
Document doc = new Document();
doc = Document.parse(tmp);
Document fields = new Document("ID", "12")
.append("ObjData", Arrays.asList(new Document("Order", "54321")));
Document update = new Document("$pull", fields);
coll.updateOne(doc, update);
I am trying to use the ´pull´ method to remove the entire document from the array where the ´Order´ equals 54321 but for some reason it's not working, I am probably doing something wrong. Could someone point out the issue please?
Also, what would be the best way to keep count of the documents within the array so that once all documents are pulled the entire document is deleted from the database? Would it be good to add some kind of ´size´ attribute and keep track of the size and decrease it after each pull?
To remove document with Order=54321 from internal array from any document (if you don't know ID) you can use empty filter like:
Document filter = new Document();
Document update = new Document("$pull", new Document("ObjData", new Document("Order", "54321")));
coll.updateOne(filter, update);
Updating records to remove values from ObjData array
The first parameter to the updateOne method is a query to find the document you want to update, not the full document.
So for your code, assuming ID is a unique value and that there's an item in your collection with an ID of "12":
// { ID: "12" }
Document query = new Document("ID", "12");
// { ObjData: { $pull: { Order: "54321" } } }
Document update = new Document("ObjData",
new Document("$pull",
new Document("Order", "54321")
)
);
coll.updateOne(query, update);
Alternatively, if you want to remove the order from all documents in the database, just replace query with an empty Document, i.e.:
// { <empty> }
Document query = new Document();
Deleting records with empty ObjData array
As for removing records when the size reaches zero, you can use a filter with $size:
db.myColl.deleteMany({ ObjData: { $size: 0 } })
This is also doable using the Java driver:
// { ObjData: { $size: 0 } }
Document query = new Document("ObjData",
new Document("$size", 0)
);
coll.deleteMany(query);
Note that for large collections (i.e. where myColl is large, not the ObjData array), this may not perform very well. If this is the case, then you may want to track the size separately (as you hinted at in your question) and index it to make it faster to search on since you can't create an index on array size in MongoDB.
References
updateOne documentation for updating documents using the Java driver
deleteOne documentation for deleting documents using the Java driver
$pull documentation for removing documents from an array
$size documentation for filtering documents based on the size of an array
I have a mongo collection with objects like these:
[
{
"_id" : "a2d",
"entityType" : "Location",
"type" : "STRING",
},
{
"_id" : "a1_order",
"entityType" : "Order",
"type" : "STRING",
}
]
Trying to append the _entityType to all document's id where it is not present at the end id the id (the first object in the above case).
Using mongo with Spring, but I'm already stuck with the first step, to get all the objects with no entityType in id.
Thinking about something like this, with regex, but I'm not sure how should it look like:
Query query = new Query();
query.addCriteria( Criteria.where( "id" ).regex( "here i need the entity type of the current document" ) );
You can build your regex by '^' ('starts with' Regex).
So you need a function who point in all documents and check this filter
List<Document> result = new ArrayList<Document>();
StringBuilder idPrefix = new StringBuilder();
idPrefix.append("^");
idPrefix.append(idCode);
idPrefix.append("_");
List<Bson> filters = new ArrayList<Bson>();
filters.add(Filters.regex("_id", keyPrefix.toString()));
for (Document d : yourCollections.find(Filters.and(filters)))
list.add(d);
You actually want a "reverse regex" here, as you need to use the data in the document in order to match on another field.
Presently you can really only do this with MongoDB using $where, which evaluates JavaScript on the server. So for spring mongo, you need the BasicQuery instead, so we can construct from BasicDBObject and Code primatives:
BasicDBObject basicDBObject = new BasicDBObject("$where",
new Code("!RegExp('_' + this.entityType + '$','i').test(this.id)"));
BasicQuery query = new BasicQuery(basicDBObject);
That will test the "id" field in the document to see if it matches the value from entityType at the "end of the string" and without considering "case". The ! is a Not condition, so the "reverse" of the logic is applied to "not match" where the field actually did end that way.
The new AWS DynamoDB document API allows 2 new data types that correspond directly to the underlying JSON representation: Map (aka JSON object) and List (aka JSON array).
However, I can't find a way to update attributes of these data types without completely overwriting them. In contrast, a Number attribute can be updated by ADDing another number, so in Java you can do something like:
new AttributeUpdate("Some numeric attribute").addNumeric(17);
Similarly you can addElements to an attribute of a Set data type. (In the old API you would use AttributeAction.ADD for both purposes.)
But for a Map or a List, it seems you must update the previous value locally, then PUT it instead of that value, for example in Java:
List<String> list = item.getList("Some list attribute");
list.add("new element");
new AttributeUpdate("Some list attribute").put(list);
This is much less readable, and under some circumstances much less efficient.
So my questions are:
Is there a way to update an attribute of a Map or a List data type without overwriting the previous value? For example, to add an element to a List, or to put an element in a Map?
How would you implement it using the Java API?
Do you know of plans to support this in the future?
Please take a look at UpdateExpression in the UpdateItem API
For example given an item with a list:
{
"hashkey": {"S" : "my_key"},
"my_list" : {"L":
[{"N":"3"},{"N":"7"} ]
}
You can update the list with code like the following:
UpdateItemRequest request = new UpdateItemRequest();
request.setTableName("myTableName");
request.setKey(Collections.singletonMap("hashkey",
new AttributeValue().withS("my_key")));
request.setUpdateExpression("list_append(:prepend_value, my_list)");
request.setExpressionAttributeValues(
Collections.singletonMap(":prepend_value",
new AttributeValue().withN("1"))
);
dynamodb.updateItem(request);`
You can also append to the list by reversing the order of the arguments in the list_append expression.
An expression like: SET user.address.zipcode = :zip would address a JSON map element combined with expression attribute values {":zip" : {"N":"12345"}}
Base on DynamoDB examples, this also work (scala)
val updateItemSpec:UpdateItemSpec = new UpdateItemSpec()
.withPrimaryKey("hashkey", my_key)
.withUpdateExpression("set my_list = list_append(:prepend_value, my_list)")
.withValueMap(new ValueMap()
.withList(":prepend_value", "1"))
.withReturnValues(ReturnValue.UPDATED_NEW)
println("Updating the item...")
val outcome: UpdateItemOutcome = table.updateItem(updateItemSpec)
println("UpdateItem succeeded:\n" + outcome.getItem.toJSONPretty)
A generic function to add or update a key/value pairs. attribute updateColumn should be of type map.
Update tableName attribute name should be passed as attributeName under key:value pairs where primaryKey = primaryKeyValue
public boolean insertKeyValue(String tableName, String primaryKey, String
primaryKeyValue, String attributeName, String newKey, String newValue) {
//Configuration to connect to DynamoDB
Table table = dynamoDB.getTable(tableName);
boolean insertAppendStatus = false;
try {
//Updates when map is already exist in the table
UpdateItemSpec updateItemSpec = new UpdateItemSpec()
.withPrimaryKey(primaryKey, primaryKeyValue)
.withReturnValues(ReturnValue.ALL_NEW)
.withUpdateExpression("set #columnName." + newKey + " = :columnValue")
.withNameMap(new NameMap().with("#columnName", attributeName))
.withValueMap(new ValueMap().with(":columnValue", newValue))
.withConditionExpression("attribute_exists("+ attributeName +")");
table.updateItem(updateItemSpec);
insertAppendStatus = true;
//Add map column when it's not exist in the table
} catch (ConditionalCheckFailedException e) {
HashMap<String, String> map = new HashMap<>();
map.put(newKey, newValue);
UpdateItemSpec updateItemSpec = new UpdateItemSpec()
.withPrimaryKey(primaryKey,primaryKeyValue)
.withReturnValues(ReturnValue.ALL_NEW)
.withUpdateExpression("set #columnName = :m")
.withNameMap(new NameMap().with("#columnName", attributeName))
.withValueMap(new ValueMap().withMap(":m", map));
table.updateItem(updateItemSpec);
insertAppendStatus = true;
} catch(Exception e) {
e.printStackTrace();
}
return insertAppendStatus;
}
I want to define a document in MongoDB in order to keep a list of key-value pairs in addition to some more information . I need to query on keys and extract just values , not whole the document. Let’s say it looks like:
{ title :” Do not stop me now”
Artist: “Queen”
Info :{
Metadata: [
{key: “genre”, value: “Rock” },
{key: “bps”, value: 120}
]
}
}
I selected this format based on http://java.dzone.com/articles/indexing-schemaless-documents
I want to query like select “genre” from song where artist is “Queen”
My current code is:
BasicDBObject eleMatch = new BasicDBObject();
eleMatch.put("key","genre");
BasicDBObject up = new BasicDBObject();
up.put("$elemMatch",eleMatch);
BasicDBObject query = new BasicDBObject();
query.put("info.Metadata", up);
query.put(“Artist”,”Queen”);
BasicDBObject fields = new BasicDBObject("info.Metadata.$",1).append("_id", false);
DBObject object =collection.findOne(query,fields);
I tried to extract value like:
System.out.println( (((BasicBSONList) ((BasicBSONObject) object.get("info")).get("Metadata")).get("value")).toString());
But I cannot get access to "value"
How I can solve it?
The positional operator return an array of one element. So, what you are converting to string is an array and of course you will get a kind of java id.
Cast it to array and get its first element