I am writting a java service where I am building the document for output.
But My structure should be : OutPut Doc is the top level doc. Inside that i want to have another Doc say Intermediate doc and in this intermediate doc i want to have Key values.
But My question is how can i insert one doc to another. I see the IDataUtil has put method which ask for key as string and value can be object.
My code is IDataUtil.put(idcvalueDoc, "Body", FullValue.toString());
But this Body should not be string it should be document .I want to insert one Doc to another.
Please help me
To accomplish what you're after, you will need to do the following:
Create an intermediateDoc IData object
Add key value tuples to the intermediateDoc as required
Create an outputDoc IData object
Add the intermediateDoc as a key value tuple to the outputDoc
Add the outputDoc to the pipeline
The following is an example Java service that demonstrates this (note the key value tuples added to the intermediateDoc are hard-coded here for convenience):
public static final void exampleService(IData pipeline) throws ServiceException {
IDataCursor pipelineCursor = pipeline.getCursor();
try {
// create an intermediateDoc IData object
IData intermediateDoc = IDataFactory.create();
// create a cursor to use to add key value tuples to the intermediateDoc
IDataCursor intermediateCursor = intermediateDoc.getCursor();
// add key value tuples as required to the intermediateDoc
IDataUtil.put(intermediateCursor, "key1", "value1");
IDataUtil.put(intermediateCursor, "key2", "value2");
// ...
// destroy the intermediateCursor when done adding key value tuples
intermediateCursor.destroy();
// create an outputDoc IData object
IData outputDoc = IDataFactory.create();
// create a cursor to use to add key value tuples to the outputDoc
IDataCursor outputCursor = outputDoc.getCursor();
// add the intermediateDoc to the outputDoc
IDataUtil.put(outputCursor, "intermediateDoc", intermediateDoc);
// destroy the outputCursor when done adding key value tuples
outputCursor.destroy();
// add the outputDoc to the pipeline
IDataUtil.put(pipelineCursor, "outputDoc", outputDoc);
} finally {
// destroy the pipelineCursor
pipelineCursor.destroy();
}
}
Assumption and Design Input/Output
Assuming the
Document of ValuesInput[] are the input, which are same as Values[] the output.
Document of columnValue under Document of ValuesInput[] has string variable named additionalString (which it would be makes no sense if there are no variable inside/under the Document)
So overall would be like this:
Of course you could generate the code after designing the input/output by right-clicking your mouse and generate code -> For implementing this service.
But, instead of using the generated code, I'm trying to give an example with IDataMap which you can find in webMethods Javadoc com.softwareag.util.IDataMap. Which is very convenient to use
IDataMap
IDataMap combines the functionality of IData, IDataCursor, IDataUtil,
and IDataFactory. IDataMap implements the java.util.Map interface from
the Java Collections Framework, providing a familiar and simple
interface. IDataMap expands the Map interface, adding getAs<Type>
methods, which convert the returned value to a specific type.
And it goes like this:
public static final void mapDocument(IData pipeline) throws ServiceException
{
// pipeline input by IDataMap
IDataMap pipelineMap = new IDataMap(pipeline);
// extracting Values input into IData[] variable array
IData[] ValuesInput = pipelineMap.getAsIDataArray("ValuesInput");
// Initiate OutDoc.Values length based on ValuesInput length
IData[] Values = new IData[ValuesInput.length];
// OutDoc.Values
// Iterate and copying all ValuesInputDoc into OutDoc.Values
for (int i = 0; i < ValuesInput.length; i++)
{
Values[i] = IDataUtil.clone(ValuesInput[i]);
}
// OutDoc
IData OutDoc = IDataFactory.create();
IDataMap outDocMap = new IDataMap(OutDoc);
// OutDoc IDataMap
String TableName = "TableName is Never assigned";
outDocMap.put("TableName", TableName);
// OutDoc.Values
outDocMap.put("Values", Values);
// Wrap the OutDoc into pipeline
pipelineMap.put("OutDoc", OutDoc);
}
And the result
With the wmboost-data library you can write this:
public static final void exampleService(IData pipeline) throws ServiceException {
Document outputDoc = Documents.create();
Document intermediateDoc = outputDoc.docEntry("intermediateDoc").putNew();
intermediateDoc.entry("key1").put("value1");
intermediateDoc.entry("key2").put("value2");
Documents.wrap(pipeline).entry("outputDoc").put(outputDoc);
}
The code:
Creates the top-level document, outputDoc
Creates intermediateDoc as nested document (an alternative is to create it independently and to attach it to its parent later)
Assigns entry values to intermediateDoc
Adds outputDoc to the pipeline
Disclaimer: I'm the author of wmboost-data.
Just check if any of the following in the WmPublic package would help:
pub.list:appendToDocumentList
or
pub.document:insertDocument
Related
I have a set of simple xpaths involving only tags and attributes, no predicates. My XML input has a size of several MB so I want to use a streaming XML parser.
How can I match the streaming XML parser against the set of xapths to retrieve one value for each xpath?
The crux seems to build the right data structure from the set of xpaths so it can be evaluated based on the xml events.
This seems like a fairly common task but I couldn't find any readily available solutions.
To match a streaming XML parser against a set of simple xpaths, you can use the following steps:
Create a Map<String, String> to store the xpaths and their corresponding values. Initialize the values to null.
Create a Stack<String> to keep track of the current path of the XML elements.
Create a SAXParser and a DefaultHandler to parse the XML input.
In the startElement method of the handler, push the element name to the stack and append it to the current path. Then, check if the current path matches any of the xpaths in the map. If yes, set a flag to indicate that the value should be extracted.
In the endElement method of the handler, pop the element name from the stack and remove it from the current path. Then, reset the flag to indicate that the value should not be extracted.
In the characters method of the handler, check if the flag is set. If yes, append the character data to the value of the matching xpath in the map.
After parsing the XML input, return the map with the xpaths and their values.
Explanation
A streaming XML parser, such as SAXParser, reads the XML input sequentially and triggers events when it encounters different parts of the document, such as start tags, end tags, text, etc. It does not build a tree structure of the document in memory, which makes it more efficient for large XML inputs.
An xpath is a syntax for selecting nodes from an XML document. It consists of a series of steps, separated by slashes, that describe the location of the desired node. For example, /bookstore/book/title selects the title element of the book element of the bookstore element.
A simple xpath involves only tags and attributes, no predicates. For example, /bookstore/book[#lang='en']/title selects the title element of the book element that has an attribute lang with value en.
To match a streaming XML parser against a set of simple xpaths, we need to keep track of the current path of the XML elements as we parse the input, and compare it with the xpaths in the set. If we find a match, we need to extract the value of the node and store it in a map. We also need to handle the cases where the node value spans across multiple character events, or where the node has multiple occurrences in the document.
Example
Suppose we have the following XML input:
<bookstore>
<book lang="en">
<title>Harry Potter and the Philosopher's Stone</title>
<author>J. K. Rowling</author>
<price>10.99</price>
</book>
<book lang="fr">
<title>Le Petit Prince</title>
<author>Antoine de Saint-Exupéry</author>
<price>8.50</price>
</book>
</bookstore>
And the following set of simple xpaths:
/bookstore/book/title
/bookstore/book/author
/bookstore/book[#lang='fr']/price
We can use the following Java code to match the streaming XML parser against the set of xpaths:
import java.io.*;
import java.util.*;
import javax.xml.parsers.*;
import org.xml.sax.*;
import org.xml.sax.helpers.*;
public class XPathMatcher {
public static Map<String, String> match(InputStream xmlInput, Set<String> xpaths) throws Exception {
// Create a map to store the xpaths and their values
Map<String, String> map = new HashMap<>();
for (String xpath : xpaths) {
map.put(xpath, null);
}
// Create a stack to keep track of the current path
Stack<String> stack = new Stack<>();
// Create a SAXParser and a DefaultHandler to parse the XML input
SAXParserFactory factory = SAXParserFactory.newInstance();
SAXParser parser = factory.newSAXParser();
DefaultHandler handler = new DefaultHandler() {
// A flag to indicate if the value should be extracted
boolean extract = false;
// A variable to store the current path
String currentPath = "";
// A variable to store the matching xpath
String matchingXPath = "";
#Override
public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
// Push the element name to the stack and append it to the current path
stack.push(qName);
currentPath += "/" + qName;
// Check if the current path matches any of the xpaths in the map
for (String xpath : map.keySet()) {
// If the xpath has an attribute, extract the attribute name and value
String attrName = "";
String attrValue = "";
if (xpath.contains("[#")) {
int start = xpath.indexOf("[#") + 2;
int end = xpath.indexOf("=");
attrName = xpath.substring(start, end);
start = end + 2;
end = xpath.indexOf("]");
attrValue = xpath.substring(start, end - 1);
}
// If the xpath matches the current path, and either has no attribute or has a matching attribute, set the flag and the matching xpath
if (xpath.startsWith(currentPath) && (attrName.isEmpty() || attrValue.equals(attributes.getValue(attrName)))) {
extract = true;
matchingXPath = xpath;
break;
}
}
}
#Override
public void endElement(String uri, String localName, String qName) throws SAXException {
// Pop the element name from the stack and remove it from the current path
stack.pop();
currentPath = currentPath.substring(0, currentPath.length() - qName.length() - 1);
// Reset the flag and the matching xpath
extract = false;
matchingXPath = "";
}
#Override
public void characters(char[] ch, int start, int length) throws SAXException {
// Check if the flag is set
if (extract) {
// Append the character data to the value of the matching xpath in the map
String value = map.get(matchingXPath);
if (value == null) {
value = "";
}
value += new String(ch, start, length);
map.put(matchingXPath, value);
}
}
};
// Parse the XML input
parser.parse(xmlInput, handler);
// Return the map with the xpaths and their values
return map;
}
public static void main(String[] args) throws Exception {
// Create an input stream from the XML file
InputStream xmlInput = new FileInputStream("bookstore.xml");
// Create a set of simple xpaths
Set<String> xpaths = new HashSet<>();
xpaths.add("/bookstore/book/title");
xpaths.add("/bookstore/book/author");
xpaths.add("/bookstore/book[#lang='fr']/price");
// Match the streaming XML parser against the set of xpaths
Map<String, String> map = match(xmlInput, xpaths);
// Print the results
for (String xpath : map.keySet()) {
System.out.println(xpath + " = " + map.get(xpath));
}
}
}
The output of the code is:
/bookstore/book/title = Harry Potter and the Philosopher's StoneLe Petit Prince
/bookstore/book/author = J. K. RowlingAntoine de Saint-Exupéry
/bookstore/book[#lang='fr']/price = 8.50
I got a JSON string that looks something like this:
String tmp = "[
{
"ID":"12",
"Date":"2018-02-02",
"ObjData":[
{
"Name":"AAA",
"Order":"12345",
"Extra1":{
"Temp":"3"
},
"Extra2":{
"Temp":"5"
}
},
{
"Name":"BBB",
"Order":"54321",
"Extra1":{
"Temp":"3"
},
"Extra2":{
"Temp":"5"
}
}
]
}
]"
I would like to remove for example the the document where ´Order´ equals "54321" from ´ObjData´. I got the following code:
Document doc = new Document();
doc = Document.parse(tmp);
Document fields = new Document("ID", "12")
.append("ObjData", Arrays.asList(new Document("Order", "54321")));
Document update = new Document("$pull", fields);
coll.updateOne(doc, update);
I am trying to use the ´pull´ method to remove the entire document from the array where the ´Order´ equals 54321 but for some reason it's not working, I am probably doing something wrong. Could someone point out the issue please?
Also, what would be the best way to keep count of the documents within the array so that once all documents are pulled the entire document is deleted from the database? Would it be good to add some kind of ´size´ attribute and keep track of the size and decrease it after each pull?
To remove document with Order=54321 from internal array from any document (if you don't know ID) you can use empty filter like:
Document filter = new Document();
Document update = new Document("$pull", new Document("ObjData", new Document("Order", "54321")));
coll.updateOne(filter, update);
Updating records to remove values from ObjData array
The first parameter to the updateOne method is a query to find the document you want to update, not the full document.
So for your code, assuming ID is a unique value and that there's an item in your collection with an ID of "12":
// { ID: "12" }
Document query = new Document("ID", "12");
// { ObjData: { $pull: { Order: "54321" } } }
Document update = new Document("ObjData",
new Document("$pull",
new Document("Order", "54321")
)
);
coll.updateOne(query, update);
Alternatively, if you want to remove the order from all documents in the database, just replace query with an empty Document, i.e.:
// { <empty> }
Document query = new Document();
Deleting records with empty ObjData array
As for removing records when the size reaches zero, you can use a filter with $size:
db.myColl.deleteMany({ ObjData: { $size: 0 } })
This is also doable using the Java driver:
// { ObjData: { $size: 0 } }
Document query = new Document("ObjData",
new Document("$size", 0)
);
coll.deleteMany(query);
Note that for large collections (i.e. where myColl is large, not the ObjData array), this may not perform very well. If this is the case, then you may want to track the size separately (as you hinted at in your question) and index it to make it faster to search on since you can't create an index on array size in MongoDB.
References
updateOne documentation for updating documents using the Java driver
deleteOne documentation for deleting documents using the Java driver
$pull documentation for removing documents from an array
$size documentation for filtering documents based on the size of an array
How to create a super set of fields(except _id) present in all the documents in a single collection of MongoDB assuming that all documents are of same type but different number of fields.
Example :
doc1 - {"_id":"test1", "firstName":"sample1", "age":24, "state":"Kansas"}
doc2 - {"_id":"test2", "lastName":"sample2", "age":24, "country":"US"}
Super set would be : {"firstName", "lastName", "age", "state", "country"}
You could try running a mapReduce operation that will return all the given document keys in the input collection as the _id key of the resulting document. This will be in an output collection where you can then apply the distinct command on the _id field to retrieve the superset of the fields.
The example that follows shows this concept:
// Run mapReduce on collectionName
String map = "function () { for (var key in this) { emit(key, null); } }";
String reduce = "function () {}";
MapReduceResults<ValueObject> results = mongoTemplate.mapReduce(
"collectionName",
map,
reduce,
new MapReduceOptions().outputCollection("col_out"),
ValueObject.class
);
// Get the distinct keys from output collection col_out ---
List<String> fieldsSuperset = mongoTemplate.getCollection("col_out").distinct("_id");
The new AWS DynamoDB document API allows 2 new data types that correspond directly to the underlying JSON representation: Map (aka JSON object) and List (aka JSON array).
However, I can't find a way to update attributes of these data types without completely overwriting them. In contrast, a Number attribute can be updated by ADDing another number, so in Java you can do something like:
new AttributeUpdate("Some numeric attribute").addNumeric(17);
Similarly you can addElements to an attribute of a Set data type. (In the old API you would use AttributeAction.ADD for both purposes.)
But for a Map or a List, it seems you must update the previous value locally, then PUT it instead of that value, for example in Java:
List<String> list = item.getList("Some list attribute");
list.add("new element");
new AttributeUpdate("Some list attribute").put(list);
This is much less readable, and under some circumstances much less efficient.
So my questions are:
Is there a way to update an attribute of a Map or a List data type without overwriting the previous value? For example, to add an element to a List, or to put an element in a Map?
How would you implement it using the Java API?
Do you know of plans to support this in the future?
Please take a look at UpdateExpression in the UpdateItem API
For example given an item with a list:
{
"hashkey": {"S" : "my_key"},
"my_list" : {"L":
[{"N":"3"},{"N":"7"} ]
}
You can update the list with code like the following:
UpdateItemRequest request = new UpdateItemRequest();
request.setTableName("myTableName");
request.setKey(Collections.singletonMap("hashkey",
new AttributeValue().withS("my_key")));
request.setUpdateExpression("list_append(:prepend_value, my_list)");
request.setExpressionAttributeValues(
Collections.singletonMap(":prepend_value",
new AttributeValue().withN("1"))
);
dynamodb.updateItem(request);`
You can also append to the list by reversing the order of the arguments in the list_append expression.
An expression like: SET user.address.zipcode = :zip would address a JSON map element combined with expression attribute values {":zip" : {"N":"12345"}}
Base on DynamoDB examples, this also work (scala)
val updateItemSpec:UpdateItemSpec = new UpdateItemSpec()
.withPrimaryKey("hashkey", my_key)
.withUpdateExpression("set my_list = list_append(:prepend_value, my_list)")
.withValueMap(new ValueMap()
.withList(":prepend_value", "1"))
.withReturnValues(ReturnValue.UPDATED_NEW)
println("Updating the item...")
val outcome: UpdateItemOutcome = table.updateItem(updateItemSpec)
println("UpdateItem succeeded:\n" + outcome.getItem.toJSONPretty)
A generic function to add or update a key/value pairs. attribute updateColumn should be of type map.
Update tableName attribute name should be passed as attributeName under key:value pairs where primaryKey = primaryKeyValue
public boolean insertKeyValue(String tableName, String primaryKey, String
primaryKeyValue, String attributeName, String newKey, String newValue) {
//Configuration to connect to DynamoDB
Table table = dynamoDB.getTable(tableName);
boolean insertAppendStatus = false;
try {
//Updates when map is already exist in the table
UpdateItemSpec updateItemSpec = new UpdateItemSpec()
.withPrimaryKey(primaryKey, primaryKeyValue)
.withReturnValues(ReturnValue.ALL_NEW)
.withUpdateExpression("set #columnName." + newKey + " = :columnValue")
.withNameMap(new NameMap().with("#columnName", attributeName))
.withValueMap(new ValueMap().with(":columnValue", newValue))
.withConditionExpression("attribute_exists("+ attributeName +")");
table.updateItem(updateItemSpec);
insertAppendStatus = true;
//Add map column when it's not exist in the table
} catch (ConditionalCheckFailedException e) {
HashMap<String, String> map = new HashMap<>();
map.put(newKey, newValue);
UpdateItemSpec updateItemSpec = new UpdateItemSpec()
.withPrimaryKey(primaryKey,primaryKeyValue)
.withReturnValues(ReturnValue.ALL_NEW)
.withUpdateExpression("set #columnName = :m")
.withNameMap(new NameMap().with("#columnName", attributeName))
.withValueMap(new ValueMap().withMap(":m", map));
table.updateItem(updateItemSpec);
insertAppendStatus = true;
} catch(Exception e) {
e.printStackTrace();
}
return insertAppendStatus;
}
The following code haven't worked for me:
public void addFieldWithValueToDoc(String DBName, String collName, String docID, String key, String value) {
BasicDBObject setNewFieldQuery = new BasicDBObject().append("$set", new BasicDBObject().append(key, value));
mongoClient.getDB(DBName).getCollection(collName).update(new BasicDBObject().append("_id", docID), setNewFieldQuery);
}
Where mongoClient variable's type is MongoClient.
It's inspired by Add new field to a collection in MongoDB .
What's wrong and how to do it right?
Thanks.
I've written a JUnit test to prove that your code does work:
#Test
public void shouldUpdateAnExistingDocumentWithANewKeyAndValue() {
// Given
String docID = "someId";
collection.save(new BasicDBObject("_id", docID));
assertThat(collection.find().count(), is(1));
// When
String key = "newKeyName";
String value = "newKeyValue";
addFieldWithValueToDoc(db.getName(), collection.getName(), docID, key, value);
// Then
assertThat(collection.findOne().get(key).toString(), is(value));
}
public void addFieldWithValueToDoc(String DBName, String collName, String docID, String key, String value) {
BasicDBObject setNewFieldQuery = new BasicDBObject().append("$set", new BasicDBObject().append(key, value));
mongoClient.getDB(DBName).getCollection(collName).update(new BasicDBObject().append("_id", docID), setNewFieldQuery);
}
So your code is correct, although I'd like to point out some comments on style that would make it more readable:
Parameters and variables should start with a lower-case letter. DBName should be dbName,
You don't need new BasicDBObject().append(key, value) use new BasicDBObject(key, value)
This code does the same thing as your code, but is shorter and simpler:
public void addFieldWithValueToDoc(String dbName, String collName, String docID, String key, String value) {
mongoClient.getDB(dbName).getCollection(collName).update(new BasicDBObject("_id", docID),
new BasicDBObject("$set", new BasicDBObject(key, value)));
}
To update existing documents in a collection, you can use the collection’s updateOne() or updateMany methods.
updateOne method has the following form:
db.collection.updateOne(filter, update, options)
filter - the selection criteria for the update. The same query selectors as in the find() method are available.
Specify an empty document { } to update the first document returned in
the collection.
update - the modifications to apply.
So, if you want to add one more field using Mongodb Java driver 3.4+, it will be:
collection.updateOne(new Document("flag", true),
new Document("$set", new Document("title", "Portable Space Ball")));
The following operation updates a single document where flag:true
Or in the same logic:
collection.updateOne(eq("flag", true),
new Document("$set", new Document("title", "Portable Space Ball")));
If the title field does not exist, $set will add a new field with the specified value, provided that the new field does not violate a type constraint. If you specify a dotted path for a non-existent field, $set will create the embedded documents as needed to fulfill the dotted path to the field.