I want to transform following input JSON to output JSON format
INPUT JSON:
[
{
"orderNumber": "201904-000000001",
"items": [
{
"itemPrice": 40000,
"itemQuantity": 11,
"item": {
"external_id": "IPHONE"
}
},
{
"itemPrice": 25000,
"itemQuantity": 22,
"item": {
"external_id": "ONEPLUS"
}
},
{
"itemPrice": 35000,
"itemQuantity": 33,
"item": {
"external_id": "SAMSUNGS10"
}
}
]
}
]
OUTPUT JSON:
[{
"orderNumber" : "201904-000000001",
"items" : [ {
"itemQuantity" : 11,
"external" : "IPHONE"
} ]
},
{
"orderNumber" : "201904-000000001",
"items" : [ {
"itemQuantity" : 22,
"external" : "ONEPLUS"
} ]
},
{
"orderNumber" : "201904-000000001",
"items" : [ {
"itemQuantity" : 33,
"external" : "SAMSUNGS10"
} ]
}]
I have tried following spec which is not working...could someone guide me about spec I should use and explain each step if possible if nested arrays and objects are even deeper how to convert
SPEC I HAVE USED:
[
{
"operation": "shift",
"spec": {
"*": {
"orderNumber": "[&1].orderNumber",
"items": {
"*": {
"itemQuantity": "[&1].items[].itemQuantity",
"item": {
"external_id": "[&1].items[].external"
}
}
}
}
}
}
]
Thanks guys, Following spec did work for me after trying different combinations. If anyone comes across this question please explain to me the answer for not trying combinations next time
[
{
"operation": "shift",
"spec": {
"*": {
"orderNumber": "[&1].orderNumber",
"items": {
"*": {
"itemQuantity": "[&3].items[&1].itemQuantity",
"item": {
"external_id": "[&4].items[&2].external"
}
}
}
}
}
}
]
Related
I am not able to transform the jolt properly using jolt spec. Thanks in photos array elements can be any like in above two are there, so it can be 3 or 4 or 5 any.
My jolt spec:
Input JSON :
{
"Entity": {
"card": {
"name": "RAM",
"lastName": "ABU"
},
"Photos": [
{
"Id": "327703",
"Caption": "TEST>> photo 1",
"Url": "http://bob.com/0001/327703/photo.jpg"
},
{
"Id": "327704",
"Caption": "TEST>> photo 2",
"Url": "http://bob.com/0001/327704/photo.jpg"
}
]
}
}
Jolt Spec :
[
{
"operation": "shift",
"spec": {
"Entity": {
"card": {
"name": "Photos[0].no",
"lastName": "Photos[0].caption2."
},
"Photos": {
"*": {
"Id": "Photos[&1].no",
"Caption": "Photos[&1].caption2"
}
}
}
}
}
]
Current output :
{
"Photos" : [ {
"no" : [ "RAM", "327703" ],
"caption2" : [ "ABU", "TEST>> photo 1" ]
}, {
"no" : "327704",
"caption2" : "TEST>> photo 2"
} ]
}
Expected output :
{
"Photos" : [
{
"no" : "Ram",
"caption2" : "ABU"
},
{
"no" : "327703" ,
"caption2" : "TEST>> photo 1"
},
{
"no" : "327704",
"caption2" : "TEST>> photo 2"
}
]
}
You can construct a shift transformation spec which will rearrange the desired elements to remain as two individual arrays nested within an object tagged Photos, and then put them into their respective (three) objects such as
[
{
"operation": "shift",
"spec": {
"Entity": {
"card": {
"name": "Photos.no",
"lastName": "Photos.caption2"
},
"Photos": {
"*": {
"Id": "Photos.no",
"Caption": "Photos.caption2"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"#": "&3[&1].&2" // grouped by "Photos" which is grabbed after going the tree 3 leves up(&3) and binded as array by their respective indexes([&1] which yields 0,1,2) and thier key names(&2)
}
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
I have a complex Json where i wanted to flatten the Json using Jolt or any other ways to flatten in Nifi is also fine
can anyone please help to flatten the below JSON
{
"veread": "ndjdjjywuieopppqpppwaghhzuii",
"debittanceInformation": [
{
"internalDocumentInformation": [
{
"number": "8",
"lineDetails": [
{
"identifications": [
{
"number": "44",
"model": {
"code": "xdd",
"proprietary": "ddy"
},
"relatedDate": "3/4"
}
],
"quantity": 136,
"unitOfMeasure": "each",
"money": {
"duePayablemoney": 44,
"discountAppliedmoneys": [
{
"model": {
"code": "dddddd",
"proprietary": "rddddy"
},
"money": 216.2
}
],
"valuemoneys": [
{
"model": {
"code": "ddddd",
"proprietary": "dddd"
},
"money": 391.41
}
],
"adjustmentmoneyAndReasons": [
{
"money": 824.54,
"reason": "dd"
}
],
"debittedmoney": 660.7,
"creditNotemoney": 985.63
}
}
],
"internalDocumentmoney": {
"duePayablemoney": 567.45,
"discountAppliedmoneys": [
{
"model": {
"code": "DD",
"proprietary": "EE"
},
"money": 5.64
}
],
"valuemoneys": [
{
"model": {
"code": "z",
"proprietary": "a"
},
"money": 145
}
],
"adjustmentmoneyAndReasons": [
{
"money": 678,
"reason": "tyuiop"
}
],
"debittedmoney": 123,
"creditNotemoney": 456
}
}
],
"supplierReferenceInformation": {
"model": {
"code": "x",
"proprietary": "y"
},
"issuer": "issuer",
"reference": "hfhjfdozkdfl;fdko;k"
},
"billr": {
"name": "jdjaiofjiosJOfjaiodjoiwaj",
"companyIdentifications": [
{
"identification": "74747435438",
"model": "jdidfjdsjf"
}
]
},
"bille": {
"name": "djhsaijdwjadfja;dddF",
"companyIdentifications": [
{
"identification": "dddddd",
"model": "kksjdadawx"
}
]
},
"additionaldebittanceInformation": [
"dd",
"xnjasndfjafnlkNDflkhlfkacmakjhfncasklnf"
]
}
],
"chargesInformation": []
}
The above is My Json which I need to flatten in to simple JSON so that I can process it in Hive .
i have tried using Flatten JSON processor but it is not working so now I an trying using jolt spec
can any one please guide me with above issue using either Jolt transformation or in Nifi.
my expected output should be as below
{
"veread": "ndjdjjywuieopppqpppwaghhzuii",
"ddddwww_internalDocumentInformation_number": "8",
"ddddwww_internalDocumentInformation_lineDetails_identifications_number": "1025",
"ddddwww_internalDocumentInformation_lineDetails_identifications_model.code": "x",
"ddddwww_internalDocumentInformation_lineDetails_identifications_model.proprietary": "y",
"ddddwww_internalDocumentInformation_lineDetails_identifications_relatedDate": "3/23",
"ddddwww_internalDocumentInformation_lineDetails_quantity": 136,
"ddddwww_internalDocumentInformation_lineDetails_unitOfMeasure": "each",
"ddddwww_internalDocumentInformation_lineDetails_money.duePayablemoney": 957.86,
"ddddwww_internalDocumentInformation_lineDetails_money.discountAppliedmoneys_model.code": "dvbnqwe",
"ddddwww_internalDocumentInformation_lineDetails_money.discountAppliedmoneys_model.proprietary": "rqazloy",
"ddddwww_internalDocumentInformation_lineDetails_money.discountAppliedmoneys_money": 216.2,
"ddddwww_internalDocumentInformation_lineDetails_money.valuemoneys_model.code": "YUIO",
"ddddwww_internalDocumentInformation_lineDetails_money.valuemoneys_model.proprietary": "BHRTY",
"ddddwww_internalDocumentInformation_lineDetails_money.valuemoneys_money": 391.41,
"ddddwww_internalDocumentInformation_lineDetails_money.adjustmentmoneyAndReasons_money": 824.54,
"ddddwww_internalDocumentInformation_lineDetails_money.adjustmentmoneyAndReasons_reason": "BONUS",
"ddddwww_internalDocumentInformation_lineDetails_money.debittedmoney": 660.7,
"ddddwww_internalDocumentInformation_lineDetails_money.creditNotemoney": 985.63,
"ddddwww_internalDocumentInformation_internalDocumentmoney.duePayablemoney": 567.45,
"ddddwww_internalDocumentInformation_internalDocumentmoney.discountAppliedmoneys_model.code": "DD",
"ddddwww_internalDocumentInformation_internalDocumentmoney.discountAppliedmoneys_model.proprietary": "EE",
"ddddwww_internalDocumentInformation_internalDocumentmoney.discountAppliedmoneys_money": 5.64,
"ddddwww_internalDocumentInformation_internalDocumentmoney.valuemoneys_model.code": "z",
"ddddwww_internalDocumentInformation_internalDocumentmoney.valuemoneys_model.proprietary": "a",
"ddddwww_internalDocumentInformation_internalDocumentmoney.valuemoneys_money": 145,
"ddddwww_internalDocumentInformation_internalDocumentmoney.adjustmentmoneyAndReasons_money": 678,
"ddddwww_internalDocumentInformation_internalDocumentmoney.adjustmentmoneyAndReasons_reason": "tyuiop",
"ddddwww_internalDocumentInformation_internalDocumentmoney.debittedmoney": 123,
"ddddwww_internalDocumentInformation_internalDocumentmoney.creditNotemoney": 456,
"ddddwww_supplierReferenceInformation.model.code": "x",
"ddddwww_supplierReferenceInformation.model.proprietary": "y",
"ddddwww_supplierReferenceInformation.issuer": "issuer",
"ddddwww_supplierReferenceInformation.reference": "hfhjfdozkdfl;fdko;k",
"ddddwww_billr.name": "jdjaiofjiosJOfjaiodjoiwaj",
"ddddwww_billr.companyIdentifications_identification": "74747435438",
"ddddwww_billr.companyIdentifications_model": "jdidfjdsjf",
"ddddwww_bille.name": "djhsaijdwjadfja;lMEjknEAKDLJOJRF",
"ddddwww_bille.companyIdentifications_identification": "mkskfsajdwiojdijRUWa",
"ddddwww_bille.companyIdentifications_model": "kksjdadawx",
"ddddwww_additionalddddwww[0]": "JDSJAHDJKLJHRFQWJESJqiorwjd",
"ddddwww_additionalddddwww[1]": "xnjasndfjafnlkNDflkhlfkacmakjhfncasklnf",
"chargesInformation": []
}
I have updated the request
Use FlattenJson processor with properties:
Separator: _
Flatten Mode: normal
Output json:
{
"transactionId" : "ndjdjjywuieopppqpppwaghhzuii",
"debittanceInformation[0]_internalDocumentInformation[0]_number" : "8",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_identifications[0]_number" : "1025",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_identifications[0]_model_code" : "x",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_identifications[0]_model_proprietary" : "y",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_identifications[0]_relatedDate" : "3\/23",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_quantity" : 136,
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_unitOfMeasure" : "each",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_duePayablemoney" : 957.86,
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_discountAppliedmoneys[0]_model_code" : "dvbnqwe",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_discountAppliedmoneys[0]_model_proprietary" : "rqazloy",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_discountAppliedmoneys[0]_money" : 216.2,
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_valuemoneys[0]_model_code" : "YUIO",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_valuemoneys[0]_model_proprietary" : "BHRTY",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_valuemoneys[0]_money" : 391.41,
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_adjustmentmoneyAndReasons[0]_money" : 824.54,
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_adjustmentmoneyAndReasons[0]_reason" : "BONUS",
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_debittedmoney" : 660.7,
"debittanceInformation[0]_internalDocumentInformation[0]_lineDetails[0]_money_creditNotemoney" : 985.63,
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_duePayablemoney" : 567.45,
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_discountAppliedmoneys[0]_model_code" : "DD",
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_discountAppliedmoneys[0]_model_proprietary" : "EE",
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_discountAppliedmoneys[0]_money" : 5.64,
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_valuemoneys[0]_model_code" : "z",
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_valuemoneys[0]_model_proprietary" : "a",
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_valuemoneys[0]_money" : 145,
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_adjustmentmoneyAndReasons[0]_money" : 678,
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_adjustmentmoneyAndReasons[0]_reason" : "tyuiop",
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_debittedmoney" : 123,
"debittanceInformation[0]_internalDocumentInformation[0]_internalDocumentmoney_creditNotemoney" : 456,
"debittanceInformation[0]_supplierReferenceInformation_model_code" : "x",
"debittanceInformation[0]_supplierReferenceInformation_model_proprietary" : "y",
"debittanceInformation[0]_supplierReferenceInformation_issuer" : "issuer",
"debittanceInformation[0]_supplierReferenceInformation_reference" : "hfhjfdozkdfl;fdko;k",
"debittanceInformation[0]_billr_name" : "jdjaiofjiosJOfjaiodjoiwaj",
"debittanceInformation[0]_billr_companyIdentifications[0]_identification" : "74747435438",
"debittanceInformation[0]_billr_companyIdentifications[0]_model" : "jdidfjdsjf",
"debittanceInformation[0]_bille_name" : "djhsaijdwjadfja;lMEjknEAKDLJOJRF",
"debittanceInformation[0]_bille_companyIdentifications[0]_identification" : "mkskfsajdwiojdijRUWa",
"debittanceInformation[0]_bille_companyIdentifications[0]_model" : "kksjdadawx",
"debittanceInformation[0]_additionaldebittanceInformation[0]" : "JDSJAHDJKLJHRFQWJESJqiorwjd",
"debittanceInformation[0]_additionaldebittanceInformation[1]" : "xnjasndfjafnlkNDflkhlfkacmakjhfncasklnf",
"chargesInformation" : [ ]
}
You can use the shift transformation below in a JoltTransformJSON processor
[
{
"operation": "shift",
"spec": {
"*": "&",
"debittanceInformation": {
"*": {
"internalDocumentInformation": {
"*": {
"number": "&4_&2_&",
"lineDetails": {
"*": {
"identifications": {
"*": {
"*": {
"#": "&9_&7_&5_&3_&"
},
"model": {
"*": {
"#": "&10_&8_&6_&4_&2\\.&"
}
}
}
},
"quantity": "&6_&4_&2_&",
"unitOfMeasure": "&6_&4_&2_&",
"money": {
"*": {
"#": "&8_&6_&4_&2\\.&"
},
"discountAppliedmoneys": {
"*": {
"model": {
"*": {
"#": "&11_&9_&7_&5\\.&4_&2\\.&"
}
},
"*": "&9_&7_&5_&3\\.&2_&"
}
},
"valuemoneys": {
"*": {
"model": {
"*": {
"#": "&11_&9_&7_&5\\.&4_&2\\.&"
}
},
"*": "&9_&7_&5_&3\\.&2_&"
}
},
"adjustmentmoneyAndReasons": {
"*": {
"*": "&9_&7_&5_&3\\.&2_&"
}
}
}
}
},
"internalDocumentmoney": {
"*": {
"#": "&6_&4_&2\\.&"
},
"discountAppliedmoneys": {
"*": {
"model": {
"*": "&8_&6_&4_&3_&1\\.&"
},
"*": "&7_&5_&3\\.&2_&"
}
},
"valuemoneys": {
"*": {
"model": {
"*": "&8_&6_&4\\.&3_&1\\.&"
},
"*": "&7_&5_&3\\.&2_&"
}
},
"adjustmentmoneyAndReasons": {
"*": {
"*": "&7_&5_&3\\.&2_&"
}
}
}
}
},
"supplierReferenceInformation": {
"model": {
"*": "&4_&2\\.&"
},
"*": "&3_&1\\.&"
},
"bill*": {
"companyIdentifications": {
"*": {
"*": "&5_&3\\.&2_&"
}
},
"*": "&3_&1\\.&"
},
"additionaldebittanceInformation": {
"*": "&3_&1\\[&\\]"
}
}
}
}
}
]
The Java code of flattening JSON records of an indefinite number of levels to a two-dimensional table is extremely long.
You can use SPL, an open-source Java package, to get this done. It is convenient and four lines of code are sufficient:
img
SPL offers JDBC driver to be invoked by Java. Just store the above SPL script as flattenjson.splx and invoke it in Java as you call a stored procedure:
…
Class.forName("com.esproc.jdbc.InternalDriver");
con= DriverManager.getConnection("jdbc:esproc:local://");
st = con.prepareCall("call flattenjson()");
st.execute();
…
I have a data like below:
{
"resourceType": "Immunization",
"id": "example",
"protocolApplied": [
{
"series": "2-dose",
"authority": {
"reference": "Organization/org1",
"type": "Organization",
"display": "xyz organization"
},
"targetDisease": [
{
"coding": [
{
"system": "http://snomed.info/sct",
"code": "40468003"
}
]
}
],
"doseNumberPositiveInt": 1,
"seriesDosesPositiveInt": 10
},
{
"series": "3-dose",
"targetDisease": [
{
"coding": [
{
"system": "http://snomed.info/sct",
"code": "66071002"
}
]
}
],
"doseNumberString": "one",
"seriesDosesString": "ten"
}
]
}
I need to transform it to get the below output:
[ {
"resourceType" : "Immunization",
"series" : "2-dose",
"reference" : "Organization/org1",
"type" : "Organization",
"display" : "xyz organization",
"targetDiseaseCodingSystem":"http://snomed.info/sct"
"targetDiseaseCode":"40468003"
"doseNumberPositiveInt" : 1,
"seriesDosesPositiveInt" : 10
}, {
"resourceType" : "Immunization",
"series" : "3-dose",
"targetDiseaseCodingSystem":"http://snomed.info/sct"
"targetDiseaseCode": "66071002"
"doseNumberString" : "one",
"seriesDosesString" : "ten"
} ]
Below is my spec:
[
{
"operation": "shift",
"spec": {
"protocolApplied": {
"*": {
"#(2,resourceType)": "[#2].resourceType",
"authority": {
// "reference": "reference"
"*": "[#3].&"
},
"targetDisease": {
"*": {
"coding": {
"*": {
//"*": "[#2].&"
"#(2,system)": "[#2].targetDiseaseCodingSystem"
}
}
}
},
"*": "[#2].&"
}
}
}
}
]
I am getting below output after applying above spec:
[ {
"resourceType" : "Immunization",
"series" : "2-dose",
"reference" : "Organization/org1",
"type" : "Organization",
"display" : "xyz organization",
"doseNumberPositiveInt" : 1,
"seriesDosesPositiveInt" : 10
}, {
"resourceType" : "Immunization",
"series" : "3-dose",
"doseNumberString" : "one",
"seriesDosesString" : "ten"
} ]
Where targetcodingsystem and targetdiseasecode is not populated. Please help me in this.
Below is spec would help,
Use [&n], to shift the value n object above.
[
{
"operation": "shift",
"spec": {
"protocolApplied": {
"*": {
"#(2,resourceType)": "[&1].resourceType",
"authority": {
"*": "[&2].&"
},
"targetDisease": {
"*": {
"coding": {
"*": {
"system": "[&5].targetDiseaseCodingSystem",
"code": "[&5].targetDiseaseCode"
}
}
}
},
"*": "[&1].&"
}
}
}
}
]
Hello everyone my json input is this:
{
"chequesCollateral": [
{
"accountNum": "0026.0002.62.0300162968",
"agreement": "0026.5501.90.0490520505",
"checkno": "229425941 ",
"amount": 20000,
"issueBank": "0026",
"branch": "0154",
"currency": "EUR",
"expDate": "2019-09-20"
},
{
"accountNum": "0026.0002.62.0300162968",
"agreement": "0026.5501.90.0490520505",
"checkno": "322108888 ",
"amount": 2500,
"issueBank": "0011",
"branch": "0335",
"currency": "EUR",
"expDate": "2019-10-26"
},
{
"accountNum": "0026.0002.62.0300162968",
"agreement": "0026.5501.90.0490520505",
"checkno": "321979826 ",
"amount": 3964.77,
"issueBank": "0011",
"branch": "0104",
"currency": "EUR",
"expDate": "2019-10-31"
}
]
}
I use this transform to produce this:
[
{
"operation": "shift",
"spec": {
"chequesCollateral": {
"*": {
"issueBank": "distinctBinData.&0"
}
}
}
}
]
and this is produced:
{
"distinctBinData" : {
"issueBank" : [ "0026", "0011", "0011" ]
}
}
I want to produce this :
{
"distinctBinData" : [ "0026", "0011", "0011" ]
}
What should i do?
You don't have to specify the &0 part in the destination key. &0 refers to the current level JSON key, which is issueBank in your case. So when you specified the destination key as distinctBinData.&0, it resolves to distinctBinData.issueBank. So just use distinctBinData as the destination key as follows.
[
{
"operation": "shift",
"spec": {
"chequesCollateral": {
"*": {
"issueBank": "distinctBinData"
}
}
}
}
]
I have this collection of documents:
[
{
"name": "name1",
"data": [
{
"numbers": ["1","2","3"]
}
]
},
{
"name": "name2",
"data": [
{
"numbers": ["2","5","3"]
}
]
},
{
"name": "name3",
"data": [
{
"numbers": ["1","5","2"]
}
]
},
{
"name": "name4",
"data": [
{
"numbers": ["1","4","3"]
}
]
},
{
"name": "name5",
"data": [
{
"numbers": ["1","2"]
}
]
}
]
I want to get all documents of this collection when an array passed as a parameter is a subset of data.numbers.
This is the aggregation that I'm using.
db.testing.aggregate(
[
{ "$match" : { "data.numbers" : { "$exists" : true } } },
{ "$project" : { "is_subset" : { "$filter" : { "input" : "$data", "as" : "d", "cond" : { "$setIsSubset" :[ ["1"],"$$d.numbers"] } } } } },
{ "$match" : { "is_subset.0" : { "$exists" : true } } }]
);
I'm trying to reproduce the above aggregation in Spring Data MongoDB.
How to pass an array as parameter in $filter and $setIsSubset functions?
operations.aggregate(
newAggregation(Testing.class,
match(where("data.numbers").exists(true)),
project().and(
filter("data")
.as("d")
.by(???))
.as("is_subset"),
match(where("is_subset.0").exists(true))
), Testing.class);
I solve my issue.
operations.aggregate(
newAggregation(Testing.class,
match(where("data.numbers").exists(true)),
project("id", "name").and(
filter("data")
.as("d")
.by(context -> new Document("$setIsSubset", Arrays.asList(numbers, "$$d.numbers"))))
.as("is_subset"),
match(where("is_subset.0").exists(true))
), Testing.class);
I created a Document with the content that I needed in the $filter condition.
new Document("$setIsSubset", Arrays.asList(numbers, "$$d.numbers"))