Develop Reference

Recursively flatten a map of nested objects

I have a list of map of fields from ElasticSearch in a JSON structure. I need to extract the keys from the fields into a name.value list to be used as search terms.
For example, the response I get from ElasticSearch looks like:
{
"orange": {
"type": "keyword"
},
"apple": {
"type": "keyword"
},
"banana": {
"type": "keyword"
},
"pineapple": {
"properties": {
"color": {
"type": "text"
},
"size": {
"type": "text"
}
}
},
"vegetables": {
"properties": {
"potato": {
"properties": {
"quality": {
"type": "keyword"
},
"price": {
"type": "keyword"
},
"location": {
"type": "keyword"
}
}
}
}
}
}
I need to transform this into a list of
[
"orange",
"apple",
"banana",
"pineapple.color",
"pineapple.size",
"vegetables.potato.quality",
"vegetables.potato.price",
"vegetables.potato.location",
"vegetables.cabbage"
]
I'm a bit lost as to where to start so I end up with something that will work no matter how deep the "object" + "properties" key ends up being.
edit:
I have a couple of methods I'm trying to do this with, but I keep ending up with nested loops instead
private static String process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final Map<String, Object> value = toSourceMap(entry.getValue());
if (value.containsKey("properties")) {
final Map<String, Object> properties = toSourceMap(value.get("properties"));
process(entry); // ??
}
return fieldName;
}
And a small helper method I'm using which casts the unknown object to a map
private static Map<String, Object> toSourceMap(final Object sourceMap) {
try {
final Map<String, Object> map = (Map) sourceMap;
return map;
} catch (final Exception e) {
return Map.of();
}
}
And I'm calling this
final List<String> fieldName = new ArrayList<>();
for (final Map.Entry<String, Object> entry : properties.entrySet()) {
fieldName.add(process(entry));
}
Trying to get a list of each value from the process method
edit 2:
I can get something that works for one level deep, but this won't capture the deeper objects like vegetables.potato.quality
private static List<String> process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final Map<String, Object> value = toSourceMap(entry.getValue());
final List<String> fields = new ArrayList<>();
if (value.containsKey("properties")) {
final Map<String, Object> properties = toSourceMap(value.get("properties"));
properties.keySet().stream().map(s -> fieldName + "." + s).forEach(fields::add);
} else {
fields.add(fieldName);
}
return fields;
}
and the caller
final List<String> fieldName = new ArrayList<>();
for (final Map.Entry<String, Object> entry : properties.entrySet()) {
fieldName.addAll(process(entry));
}

I'm sure theres a cleaner and better way to do it, but I was able to achieve what I needed by doing the following
private static List<String> process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final List<String> fields = new ArrayList<>();
final Map<String, Object> value = toSourceMap(entry.getValue());
if (value.containsKey(MAPPING_PROPERTIES)) {
toSourceMap(value.get(MAPPING_PROPERTIES)).entrySet()
.stream()
.map(FieldMappingFactory::process)
.map(nestedFields -> nestedFields.stream().map(f -> "%s.%s".formatted(fieldName, f)).toList())
.forEach(fields::addAll);
} else {
fields.add(fieldName);
}
return fields;
}

Here's a solution based on the Depth first search tree-traversal algorithm.
Since it's iterative, you can use it to process even deeply nested massive JSON without a risk of getting a StackOverFlowError.
To implement DFS, we need a Stack and a Map is needed to store the paths associated with a node that are being explored.
As the first step, we need to read the whole JSON tree and the obtained JsonNode on the stack.
Then, until the stack is not empty, need to examine each node it contains by pulling them out from the stack. If the node happens to be an ArrayNode or ObjectNode, then all its children-nodes which could be obtained via JsonNode.fields() should be added on the stack.
String json = // the source JSON
JsonNode node = new ObjectMapper().readTree(json);
List<String> results = new ArrayList<>();
Deque<JsonNode> stack = new ArrayDeque<>();
Map<JsonNode, List<String>> pathByNode = new HashMap<>();
pathByNode.put(node, Collections.emptyList());
Set<String> keysToExclude = Set.of("type", "properties"); // add more if you need
stack.push(node);
while (!stack.isEmpty()) {
JsonNode current = stack.pop();
List<String> path = pathByNode.get(current);
if (current instanceof ArrayNode || current instanceof ObjectNode) {
for (Iterator<Map.Entry<String, JsonNode>> it = current.fields(); it.hasNext(); ) {
Map.Entry<String, JsonNode> next = it.next();
stack.push(next.getValue());
String propertyName = next.getKey();
List<String> newPath;
if (!keysToExclude.contains(propertyName)) {
newPath = new ArrayList<>(path);
newPath.add(propertyName);
results.add(String.join(".", newPath)); // list of path should be updated
} else {
newPath = path;
}
pathByNode.put(next.getValue(), newPath);
}
}
}
results.forEach(System.out::println);
Output:
orange
apple
banana
pineapple
vegetables
vegetables.potato
vegetables.potato.quality
vegetables.potato.price
vegetables.potato.location
pineapple.color
pineapple.size

This is a simpler solution.
private static void process(List<String> paths, String path, JsonNode node, Set<String> excludeKeys) {
if (node.isValueNode()) {
paths.add(path);
} else if (node.isObject()) {
node.fields().forEachRemaining(elem -> {
String key = excludeKeys.contains(elem.getKey()) ? path : (path == null ? "" : path + ".") + elem.getKey();
process(paths, key, elem.getValue(), excludeKeys);
});
} else { // This part is not required if there is no array inside the source JSON object
for (int i = 0; i < node.size(); i++) {
process(paths, String.format("%s[%d]", path, i), node.get(i), excludeKeys);
}
}
}
Caller
JsonNode node = new ObjectMapper().readTree(// JSON string //);
List<String> paths = new ArrayList<>();
process(paths, null, node, Set.of("properties", "type"));
paths.forEach(System.out::println);
Output
orange
apple
banana
pineapple.color
pineapple.size
vegetables.potato.quality
vegetables.potato.price
vegetables.potato.location

Java json object replace keys names

I have Json like this:
{
"_id" : ObjectId("5e99f6d16cbddf7dad26557f"),
"channel_id" : 49066,
"timestamp" : NumberLong(1580982302003),
"values" : {
"some id" : "81151501",
"some title" : "Some title",
"some address" : "https://www.some-address.com",
"new hash" : {
"some value" : "5",
"other value" : " 54.10 BRL"
},
"wrong values" : "This text have wrong & values & and netx is wrong too & and this &"
},
"null value" : null,
"zero integer" : 0
}
I need to loop through each key and replace spaces with snake_case, for example from other value to other_value
Additionally, I wanted to check every value in the loop by replacing the character & with _, for example:
from This text have wrong & values & and netx is wrong too & and this & to This text have wrong _ values _ and netx is wrong too _ and this _
My json object is made from:
JSONobject jsonObject = new JSONobject(jsonString)

You could iterate over the keys, normalize the key and recursively continue as long as the value is a JSONObject. If it's not, then you could normalize the value as well. So this would look something like this:
static JSONObject normalize(JSONObject object) throws JSONException {
JSONObject result = new JSONObject();
Iterator iterator = object.keys();
while (iterator.hasNext()) {
String key = (String) iterator.next();
String normalizedKey = key.replace(" ", "_");
Object inner = object.get(key);
if (inner instanceof JSONObject) {
result.put(normalizedKey, normalize((JSONObject) inner));
} else if (inner instanceof String) {
result.put(normalizedKey, object.getString(key).replace("&", "_"));
} else {
result.put(normalizedKey, inner);
}
}
return result;
}
Latest version of the library also provides the ability to obtain a keyset, which would allow for a slightly cleaner looping of the keys:
static JSONObject normalized(JSONObject object) {
JSONObject result = new JSONObject();
object.keySet().forEach(key -> {
String normalizedKey = key.replace(" ", "_");
Object value = object.get(key);
if (value instanceof JSONObject) {
result.put(normalizedKey, normalized((JSONObject) value));
} else if (value instanceof String) {
result.put(normalizedKey, ((String) value).replace("&", "_"));
} else {
result.put(normalizedKey, value);
}
});
return result;
}

JSONObject from dot seperated strings

I am struggling with a specific problem, that I cannot think of correctly. The following is the problem
I have a map with key value like the following, i just used strings here
String key = "activate.message.success"
String value = "success"
String key1 = "activate.title"
String value1 = "Good Title"
String key2 = "activate.message.error"
String value2 = "error"
String key3 = "activate.message.short.poll"
String value3 = "This is short poll"
I need to build a json like the following
{
"activate":{
"message":{
"success":"success",
"error":"error",
"short":{
"poll":"This is short poll"
}
},
"title":"Good Title"
}
}
I could not think of a proper solution for this use case and struggling for 3 hours. I thought of using recursion, but i dont how exactly i could do. Please help with this. I am using java for this, I should use generic JSONObject to solve as there is not POJO mappings. So far I have just splitted the strings using separtor and stored in an another map like the following
public Map<String, Object> getJsonObjectFromKeyValueMap(Map<String, String> stringValueMap,
Map<String, Object> stringObjectMap) {
for (Entry entry : stringValueMap.entrySet()) {
String[] keyValueArrayString = entry.getKey().toString().split("\\.");
int sizeOfMap = keyValueArrayString.length;
int i = 0;
String concatString = "";
for (String processKey : keyValueArrayString) {
if (i < sizeOfMap - 1) {
concatString += processKey + ".";
stringObjectMap.put(concatString, (Object) new JSONObject());
} else {
concatString += processKey;
stringObjectMap.put(concatString, entry.getValue());
concatString = "";
}
i++;
}
}
return stringObjectMap;
}

First, let's update your data into a proper map :
Map<String, String> data = new HashMap<>();
data.put("activate.message.success", "success");
data.put("activate.title", "Good Title");
data.put("activate.message.error", "error");
data.put("activate.message.short.poll", "This is short poll");
Then, your logic is pretty close, for each node but the last, you create a new JSONObject, for the last, you insert the value.
If you try to build a JSONObject instead of the map directly, you would get a pretty good result already, well somewhat of a result.
The following will iterate a Map<String, String> of data.
For each entry, we split the key to getting the nodes.
Then, we just need to move in the json, if a node doesn't exist, we create it.
Then, for the last value, create the value.
public static JSONObject build(Map<String, String> data) {
JSONObject json = new JSONObject();
//Iterate the map entries
for (Entry<String, String> e : data.entrySet()) {
String[] keys = e.getKey().split("\\.");
// start from the root
JSONObject current = json;
for (int i = 0; i < keys.length; ++i) {
String key = keys[i];
//Search for the current node
try {
//If it exist, do nothing
current = current.getJSONObject(key);
} //If it does not exist
catch (JSONException ex) {
//Is it the last node, create the value
if (i == keys.length - 1) {
current.put(key, e.getValue());
} //Not the last node, create a new JSONObject
else {
JSONObject tmp = new JSONObject();
current.put(key, tmp);
current = tmp; //Always replace current with the last node to go deeped each iteration
}
}
}
}
return json;
}
And the example :
public static void main(String[] args) {
Map<String, String> data = new HashMap<>();
data.put("activate.message.success", "success");
data.put("activate.title", "Good Title");
data.put("activate.message.error", "error");
data.put("activate.message.short.poll", "This is short poll");
JSONObject json = build(data);
System.out.println(json.toString(4));
}
Ouptut:
{"activate": {
"message": {
"success": "success",
"short": {"poll": "This is short poll"},
"error": "error"
},
"title": "Good Title"
}}
Note : I used an exception to check for the existance of the key, if the map is huge, this could have some impact so you can simply use :
if(current.isNull(key)){
if (i == keys.length - 1) {
current.put(key, e.getValue());
} else {
JSONObject tmp = new JSONObject();
current.put(key, tmp);
current = tmp;
}
} else {
current = current.getJSONObject(key);
}
This was created using org.json/json

How to check if a JsonNode is a single element or an Array in Java?

I am traversing a json response with JsonNode (com.fasterxml.jackson.databind.JsonNode)
How can I check if a given JsonNode is a single element or an Array?, because I need to traverse it deeper, and update some values (for example, the name value)
I can have a json response like this: (with a single element)
{ person: {
name: "name1",
address: "address1"
}
}
or I can have a json response like this: (with a )
{ "person": [
{
"name": "name1",
"address": "address1"
},
{
"name": "name2",
"address": "address2"
}
]
}
For an single element, I have this code:
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(jsonString);
JsonNode personNode = root.findPath("person");
if(!personRootNode.isMissingNode())
((ObjectNode)nameNode).put("name","UPDATED NAME");
And, for an array element, I have this code
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(jsonString);
JsonNode personNode = root.findPath("person");
if(!personNode .isMissingNode())
for(JsonNode node: personRootNode){
if(!node.isMissingNode()) {
((ObjectNode)node).put("name","UPDATED NAME");
}
}
I want to mix the logic in a single place, because the unique difference is the for loop
I can wrap the replace logic in a function/method. But how do I check if the current node is an element or an array?

You can call isArray() function on JsonNode object. For example :
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(s2);
JsonNode personNode = root.findPath("person");
if(personNode.isArray()) {
///array found
} else {
// non-array element
}

use the below method. Enter the JSONObject to check and node name.(used the org.json.JSONObject)
private void checkNodeStatus(JSONObject jsonObject, String node) {
if (jsonObject.optJSONArray(node) != null) {
} else if (jsonObject.optString(node) != null) {
JSONArray array = new JSONArray();
array.put(jsonObject.getJSONObject(node));
jsonObject = jsonObject.put(node, array);
} else {
System.out.println("error in checkNodeStatus > node : " + node);
}
}
if It is a JSONObject, It converts to JSONArray.

Parse json with mixed child objects via gson and send back as list

I'm facing a problem related to parsing json which have mixed arrays of child classes, and I need to provide that as java list back to client.
Sample JSON:
{
"status": "OK",
"results": [
{
"type": "one",
"Id": "2170676",
"count": "456",
"title": "title",
"description": "description",
"oneMemberOne": "11",
"oneMemberTwo": "12",
}
{
"type": "two",
"Id": "2170677",
"count": "123",
"title": "title",
"description": "description",
"twoMemberOne": "21",
"twoMemberTwo": "22",
}
]
}
I created one Parent class and two child class from this:
Num : type, Id, count, title, description fields
One extends Num : oneMemberOne, oneMemberTwo
Two extends Num : twoMemberOne, twoMemberTwo
Now my question:
I have a method for asking results. Say it List<Num> getResults()
I parse the data properly like this:
List<Num> result = new ArrayList<Num>();
JsonParser parser = new JsonParser();
JsonObject jsonObject = parser.parse(lastResponsePayload).getAsJsonObject()
JsonArray results = jsonObject.get("results").getAsJsonArray();
for (JsonElement element : results) {
JsonObject trs = element.getAsJsonObject();
String type = trs.get("type").getAsString();
if (type.equals("one") {
One one = new Gson().fromJson(element, One.class);
result.add(product);
} else if (type.equals("two") {
Two two = new Gson().fromJson(element, Two.class);
result.add(metaproduct);
}
return result;
Now, on client side, after I get the list, i have to do this:
List<Num> results = getResults();
List<One> oness = new ArrayList<One>();
List<Two> twoss = new ArrayList<Two>();
for(Num n : results) {
if(n.type.equals("one")
oness.add((One) n);
else
twoss.add((Two) n);
Is this a good design for this scenario ?
User of this API has to downcast everytime based on the type field of parent class. Because webservice gives me mixed array of child classes, I have to do this. Is there any better approach to this problem ?
One another approach in my mind is to create a Result class which contains two members One and Two and provide my a List<Result> instead of List<Num>,
but then user has to check whether member is null or not and then take appropriate steps.
Thank you in advance.

I will suggest that you have another class something like this. It prevents client code from spinning through list again and parsing out records. Please note I have not run this code to test it.
public class Result {
private List<One> oness;
private List<Two> twoss;
public List<One> getOness() {
return oness;
}
public void setOness(List<One> oness) {
this.oness = oness;
}
public List<Two> getTwoss() {
return twoss;
}
public void setTwoss(List<Two> twoss) {
this.twoss = twoss;
}
}
And change
List<Num> getResults()
To
Result getResults()
also user will not have to check for nulls if you modify your parsing logic. Please see that in case we don't have results we are not returning null list but EmptyList.
Result result = new Result();
JsonParser parser = new JsonParser();
JsonObject jsonObject = parser.parse(lastResponsePayload).getAsJsonObject()
JsonArray results = jsonObject.get("results").getAsJsonArray();
List<One> oness = null;
List<Two> twoss = null;
for (JsonElement element : results) {
JsonObject trs = element.getAsJsonObject();
String type = trs.get("type").getAsString();
if (type.equals("one")) {
if(oness == null) {
oness = new ArrayList<One>();
result.setOness(oness);
}
One one = new Gson().fromJson(element, One.class);
oness.add(product);
} else if (type.equals("two")) {
if(twoss == null) {
twoss = new ArrayList<Two>();
result.setTwoss(twoss);
}
Two two = new Gson().fromJson(element, Two.class);
twoss.add(metaproduct);
}
if(oness == null) {
result.setOness(Collections.<One>EMPTY_LIST);
}
if(twoss == null) {
result.setTwoss(Collections.<Two>EMPTY_LIST);
}
return result;
Hope it helps :)