Develop Reference

Is it possible to create a map in runtime mode, which will be filled in passing through all documents and returned at the end ES

For ex: I have 2 documents with this body:
{
"id": "doc_one",
"name": "test_name",
"date_creation": "some_date_cr_1",
"date_updation": "some_date_up_1"
}
And the second doc:
{
"id": "doc_two",
"name": "test_name",
"date_creation": "some_date_cr_2",
"date_updation": "some_date_up_2"
}
What I want to do: to create two runtime field or Map('data_creation',count_of_doc_where_field_not_null_AND_the_condition_is_met).
For ex: I've got the 1st doc, there is date_creation IS NOT NULL and the condition startDate<=date_creation<=endDate is met, so, I create some field count = 0 and when I've got this case I do count++. When I will get all the docs I will set finally count value from map as result: Map('data_creation',final_count) and the same for another field but in the same map.
I tried to use script, but there is return Map for each doc, for ex:
{
"_index": "my_index_001",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"fields": {
"my_doubled_field": [
{
"NEW": 2
}
]
}
},
{
"_index": "my_index_001",
"_type": "_doc",
"_id": "2",
"_score": 1.0,
"fields": {
"my_doubled_field": [
{
"NEW": 2
}
]
}
}

I have index below 3 documents to index where one document dont have date_creation field:
POST sample/_doc
{
"id": "doc_two",
"name": "test_name",
"date_updation": "some_date_up_2"
}
POST sample/_doc
{
"id": "doc_one",
"name": "test_name",
"date_creation": "some_date_cr_1",
"date_updation": "some_date_up_1"
}
POST sample/_doc
{
"id": "doc_two",
"name": "test_name",
"date_creation": "some_date_cr_2",
"date_updation": "some_date_up_2"
}
Now you can use filter aggregation from elasticsearch as shown below:
{
"size": 0,
"aggs": {
"date_creation": {
"filter": {
"range": {
"date_creation": {
"gte": "2020-01-09T10:20:10"
}
}
}
},
"date_updation": {
"filter": {
"range": {
"date_updation": {
"gte": "2020-01-09T10:20:10"
}
}
}
}
}
}
Response:
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"date_updation" : {
"meta" : { },
"doc_count" : 3
},
"date_creation" : {
"meta" : { },
"doc_count" : 2
}
}
You can see date_updation field is available in 3 doc so it is showing count as 3 and date_creation field is available in 2 doc so it is showing count as 2.

MongoDB Java - Create a new ObjectId for each element of an existing array

I have an existing collection, containing several documents.
[{
"_id": "...1",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"value": "sub element 1.1"
},
{
"value": "sub element 1.2"
},
{
"value": "sub element 1.3"
}
]
}, {
"_id": "...2",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"value": "sub element 2.1"
},
{
"value": "sub element 2.2"
}
]
}, // many others here...
]
For each root document, I would like to add an _id property of type ObjectId on each sub-element of someArray. So, after I run my command, the content of the collection is the following:
[{
"_id": "...1",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"_id": ObjectId("..."),
"value": "sub element 1.1"
},
{
"_id": ObjectId("..."),
"value": "sub element 1.2"
},
{
"_id": ObjectId("..."),
"value": "sub element 1.3"
}
]
}, {
"_id": "...2",
"prop1": "...",
"prop2": "...",
"someArray": [
{
"_id": ObjectId("..."),
"value": "sub element 2.1"
},
{
"_id": ObjectId("..."),
"value": "sub element 2.2"
}
]
}, // ...
]
Each ObjectId being, of course, unique.
The closer I got was with this:
db.getCollection('myCollection').updateMany({}, { "$set" : { "someArray.$[]._id" : ObjectId() } });
But every sub-element of the entire collection ends up with the same ObjectId value...
Ideally, I need to get this working using Java driver for MongoDB. The closest version I got is this (which presents the exact same problem: all the ObjectId created have the same value).
database
.getCollection("myCollection")
.updateMany(
Filters.ne("someArray", Collections.emptyList()), // do not update empty arrays
new Document("$set", new Document("someArray.$[el]._id", "ObjectId()")), // set the new ObjectId...
new UpdateOptions().arrayFilters(
Arrays.asList(Filters.exists("el._id", false)) // ... only when the _id property doesn't already exist
)
);

With MongoDB v4.4+, you can use $function to use javascript to assign the _id in the array.
db.collection.aggregate([
{
"$addFields": {
"someArray": {
$function: {
body: function(arr) {
return arr.map(function(elem) {
elem['_id'] = new ObjectId();
return elem;
})
},
args: [
"$someArray"
],
lang: "js"
}
}
}
}
])
Here is the Mongo playground for your reference. (It's slightly different from the code above as playground requires the js code to be in double quote)
For older version of MongoDB, you will need to use javascript to loop the documents and update them one by one.
db.getCollection("...").find({}).forEach(function(doc) {
doc.someArray = doc.someArray.map(function(elem) {
elem['_id'] = new ObjectId();
return elem;
})
db.getCollection("...").save(doc);
})

Here is what I managed to write in the end:
MongoCollection<Document> collection = database.getCollection("myCollection");
collection
.find(Filters.ne("someArray", Collections.emptyList()), MyItem.class)
.forEach(item -> {
item.getSomeArray().forEach(element -> {
if( element.getId() == null ){
collection.updateOne(
Filters.and(
Filters.eq("_id", item.getId()),
Filters.eq("someArray.value", element.getValue())
),
Updates.set("someArray.$._id", new ObjectId())
);
}
});
});
The value property of sub-elements had to be unique (and luckily it was). And I had to perform separate updateOne operations in order to obtain a different ObjectId for each element.

Create a nested map from Json using streams

I have the below Json that I'm reading into nested POJOs with the same structure.
{
"employees": [
{
"name": "John",
"age": "30",
"proData": [
{
"year": "1",
"idList": [
"234342",
"532542",
"325424",
"234234"
]
},
{
"year": "2",
"idList": [
"234342",
"532542",
"325424",
"234234"
]
},
{
"year": "3",
"idList": [
"234342",
"532542",
"325424",
"234234"
]
}
]
},
{
"name": "Scott",
"age": "32",
"proData": [
{
"year": "1",
"idList": [
"234342",
"532542",
"325424",
"234234"
]
},
{
"year": "2",
"idList": [
"234342",
"532542",
"325424",
"234234"
]
},
{
"year": "3",
"idList": [
"234342",
"532542",
"325424",
"234234"
]
}
]
}
]
}
Now I wanted to map this to a structure like below, the ProData can be initialized using each of the string in the idList.
Map<String,Map<String,List<ProData>>> finalMap
I have written something like the below and it works.
Map<String,Map<String,List<ProData>>> finalMap = new HashMap<>();
for(Employee employee:root.getEmployees()){
Map<String,List<ProData>> proDataMap = new HashMap<>();
for(ProData proData: employee.getProData()){
List<ProData> finalObjs = new ArrayList<>();
for(String id:proData.getIdList()){
finalObjs.add(new ProData(id));
}
proDataMap.put(proData.getYear(),finalObjs);
}
finalMap.put(employee.getName(),proDataMap);
}
I wanted to make a better version using the stream API.

The end result is a map, so use the toMap collector. The maps' keys are the employee names (assuming no duplicates), and the map values require a little bit more work.
root.getEmployees().stream().collect(
Collectors.toMap(
Employee::getName,
Employee::getProDataMap
)
}
Now let's try writing getProDataMap in Employee. Again we use the toMap collector. The keys are the years (assuming no duplicates), and the values are the id lists mapped to ProData using the constructor.
public Map<String, List<ProData>> getProDataMap() {
return this.getProData().stream().collect(
Collectors.toMap(
ProData::getYear,
proData -> proData.getIdList().stream()
.map(ProData::new)
.collect(Collectors.toList())
)
)
}

should be something similar to,
root.stream().forEach(employee -> {
Map<String,List<ProData>> proDataMap = new HashMap<>();
employee.getProdData().stream().forEach(data -> {
List<ProData> finalObjs = new ArrayList<>();
data.getIdList().stream().forEach(data -> {
finalObjs.add(new ProData(id));
});
});
});
but regardless you can also use Gson to parse it
https://mkyong.com/java/how-to-parse-json-with-gson/

Dynamic Json to CSV in java. looking for Generic code that works with any json, irrespective of any number of node in json file

hi i need to convert all json elements into csv . json is dynamic file, number of field and names will change from file to file.
i tried different methods but most cases i need to mention field names in scripts to pull data into csv
JSON file
[
{
"system": "Application",
"id": "12345",
"version": 1,
"event": "NEW",
"keywords": {
"ProductType": "ALL",
"Business": "USA",
},
"product": {
"type": "INS",
"startDate": 20190102,
"endDate": 20190104,
"cash": 100000.00,
"sub": {
"type": "Life",
"productId": 987,
"maturityDate": 20260421,
},
"paymentCalendar": [
"Monthly"
],
"duration": "20Y",
"Amount": 1000.00,
"cashFlows": [
{
"startDate": 20190102,
"endDate": 20190104,
"paymentDate": 20190104,
}
],
"principalFlows": [
{
"startDate": 20190102,
"endDate": 20190104,
"paymentDate": 20190102,
"currency": "USA",
"amount": 400.0
},
{
"startDate": 20190104,
"endDate": 20190104,
"paymentDate": 20190104,
"currency": "USA",
"amount": 600.0
}
]
},
"EventDate": 20190108,
"maturityDate": 20190104
}
]
above fields are not constant, all filed will keep changing.
expected output is below

Using Jackson ObjectMapper and Apache Commons CSV you can implement the functionality you require by reading the JSON and then visiting all the nodes.
If the node is a collection then visit all its children with the field or array index appended to the prefix
Note that arrays and objects need to be handled independently
If the node is a not a collection then add it to the CSV output
public void jsonToCsv(String json, Appendable appendable) throws IOException {
JsonNode root = new ObjectMapper().reader().readTree(json);
CSVPrinter printer = CSVFormat.DEFAULT.print(appendable);
appendNode(root.get(0), "", printer);
}
private void appendNode(JsonNode node, String prefix, CSVPrinter printer) throws IOException {
if (node.isArray()) {
for (int i = 0; i < node.size(); ++i) {
appendNode(node.get(i), String.format("%s/%d", prefix, i), printer);
}
} else if (node.isContainerNode()) {
Iterator<Map.Entry<String, JsonNode>> fields = node.fields();
while (fields.hasNext()) {
Map.Entry<String, JsonNode> field = fields.next();
appendNode(field.getValue(), String.format("%s/%s", prefix, field.getKey()), printer);
}
} else {
printer.printRecord(prefix.substring(1), node.asText());
}
}

Merge data together based on an ID Java streams

I currently have a set of data I pull back from an API in my java app. The data returned is along the lines of the following:
{
"id": 1,
"receiptId": "123456",
"selections": [
{
"name": "Apple",
"price": "£1"
}
]
},
{
"id": 2,
"receiptId": "678910",
"selections": [
{
"name": "Pear",
"price": "£0.5"
}
]
},
{
"id": 3,
"receiptId": "123456",
"selections": [
{
"name": "Banana",
"price:": "£2.00"
}
]
}
As you can see, two of the receiptId's are the same, I would like to merge any duplicate receiptId's data together to become one block. I.e:
{
"id": 1,
"receiptId": "123456",
"selections": [
{
"name": "Apple",
"price": "£1"
},
{
"name": "Banana",
"price": "£2.00"
}
]
},
{
"id": 2,
"receiptId": "678910",
"selections": [
{
"name": "Pear",
"price": "£0.5"
}
]
},
Currently I am just streaming the data into a map by doing the following:
List<String> data = data.getData()
.stream()
.map(this::dataToReadable)
.collect(Collectors.toList());
dataToReadable does the following:
private String dataToReadable(List data) {
return new DataBuilder().fromData(data)
.buildData();
}
fromData does the following:
public DataBuilder fromData(List data) {
this.withId(data.getId())
this.withSelections(data.getSelections())
this.withReceiptId(data.getReceiptId())
return this;
}

See if this works.
data.getData()
.stream()
.collect(Collectors.groupingBy(Data::getRecieptId))
.entrySet()
.stream()
.map(item -> dataToReadable(item.getValue()))
.collect(Collectors.toList());