I have a list of map of fields from ElasticSearch in a JSON structure. I need to extract the keys from the fields into a name.value list to be used as search terms.
For example, the response I get from ElasticSearch looks like:
{
"orange": {
"type": "keyword"
},
"apple": {
"type": "keyword"
},
"banana": {
"type": "keyword"
},
"pineapple": {
"properties": {
"color": {
"type": "text"
},
"size": {
"type": "text"
}
}
},
"vegetables": {
"properties": {
"potato": {
"properties": {
"quality": {
"type": "keyword"
},
"price": {
"type": "keyword"
},
"location": {
"type": "keyword"
}
}
}
}
}
}
I need to transform this into a list of
[
"orange",
"apple",
"banana",
"pineapple.color",
"pineapple.size",
"vegetables.potato.quality",
"vegetables.potato.price",
"vegetables.potato.location",
"vegetables.cabbage"
]
I'm a bit lost as to where to start so I end up with something that will work no matter how deep the "object" + "properties" key ends up being.
edit:
I have a couple of methods I'm trying to do this with, but I keep ending up with nested loops instead
private static String process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final Map<String, Object> value = toSourceMap(entry.getValue());
if (value.containsKey("properties")) {
final Map<String, Object> properties = toSourceMap(value.get("properties"));
process(entry); // ??
}
return fieldName;
}
And a small helper method I'm using which casts the unknown object to a map
private static Map<String, Object> toSourceMap(final Object sourceMap) {
try {
final Map<String, Object> map = (Map) sourceMap;
return map;
} catch (final Exception e) {
return Map.of();
}
}
And I'm calling this
final List<String> fieldName = new ArrayList<>();
for (final Map.Entry<String, Object> entry : properties.entrySet()) {
fieldName.add(process(entry));
}
Trying to get a list of each value from the process method
edit 2:
I can get something that works for one level deep, but this won't capture the deeper objects like vegetables.potato.quality
private static List<String> process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final Map<String, Object> value = toSourceMap(entry.getValue());
final List<String> fields = new ArrayList<>();
if (value.containsKey("properties")) {
final Map<String, Object> properties = toSourceMap(value.get("properties"));
properties.keySet().stream().map(s -> fieldName + "." + s).forEach(fields::add);
} else {
fields.add(fieldName);
}
return fields;
}
and the caller
final List<String> fieldName = new ArrayList<>();
for (final Map.Entry<String, Object> entry : properties.entrySet()) {
fieldName.addAll(process(entry));
}
I'm sure theres a cleaner and better way to do it, but I was able to achieve what I needed by doing the following
private static List<String> process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final List<String> fields = new ArrayList<>();
final Map<String, Object> value = toSourceMap(entry.getValue());
if (value.containsKey(MAPPING_PROPERTIES)) {
toSourceMap(value.get(MAPPING_PROPERTIES)).entrySet()
.stream()
.map(FieldMappingFactory::process)
.map(nestedFields -> nestedFields.stream().map(f -> "%s.%s".formatted(fieldName, f)).toList())
.forEach(fields::addAll);
} else {
fields.add(fieldName);
}
return fields;
}
Here's a solution based on the Depth first search tree-traversal algorithm.
Since it's iterative, you can use it to process even deeply nested massive JSON without a risk of getting a StackOverFlowError.
To implement DFS, we need a Stack and a Map is needed to store the paths associated with a node that are being explored.
As the first step, we need to read the whole JSON tree and the obtained JsonNode on the stack.
Then, until the stack is not empty, need to examine each node it contains by pulling them out from the stack. If the node happens to be an ArrayNode or ObjectNode, then all its children-nodes which could be obtained via JsonNode.fields() should be added on the stack.
String json = // the source JSON
JsonNode node = new ObjectMapper().readTree(json);
List<String> results = new ArrayList<>();
Deque<JsonNode> stack = new ArrayDeque<>();
Map<JsonNode, List<String>> pathByNode = new HashMap<>();
pathByNode.put(node, Collections.emptyList());
Set<String> keysToExclude = Set.of("type", "properties"); // add more if you need
stack.push(node);
while (!stack.isEmpty()) {
JsonNode current = stack.pop();
List<String> path = pathByNode.get(current);
if (current instanceof ArrayNode || current instanceof ObjectNode) {
for (Iterator<Map.Entry<String, JsonNode>> it = current.fields(); it.hasNext(); ) {
Map.Entry<String, JsonNode> next = it.next();
stack.push(next.getValue());
String propertyName = next.getKey();
List<String> newPath;
if (!keysToExclude.contains(propertyName)) {
newPath = new ArrayList<>(path);
newPath.add(propertyName);
results.add(String.join(".", newPath)); // list of path should be updated
} else {
newPath = path;
}
pathByNode.put(next.getValue(), newPath);
}
}
}
results.forEach(System.out::println);
Output:
orange
apple
banana
pineapple
vegetables
vegetables.potato
vegetables.potato.quality
vegetables.potato.price
vegetables.potato.location
pineapple.color
pineapple.size
This is a simpler solution.
private static void process(List<String> paths, String path, JsonNode node, Set<String> excludeKeys) {
if (node.isValueNode()) {
paths.add(path);
} else if (node.isObject()) {
node.fields().forEachRemaining(elem -> {
String key = excludeKeys.contains(elem.getKey()) ? path : (path == null ? "" : path + ".") + elem.getKey();
process(paths, key, elem.getValue(), excludeKeys);
});
} else { // This part is not required if there is no array inside the source JSON object
for (int i = 0; i < node.size(); i++) {
process(paths, String.format("%s[%d]", path, i), node.get(i), excludeKeys);
}
}
}
Caller
JsonNode node = new ObjectMapper().readTree(// JSON string //);
List<String> paths = new ArrayList<>();
process(paths, null, node, Set.of("properties", "type"));
paths.forEach(System.out::println);
Output
orange
apple
banana
pineapple.color
pineapple.size
vegetables.potato.quality
vegetables.potato.price
vegetables.potato.location
Related
I am struggling with a specific problem, that I cannot think of correctly. The following is the problem
I have a map with key value like the following, i just used strings here
String key = "activate.message.success"
String value = "success"
String key1 = "activate.title"
String value1 = "Good Title"
String key2 = "activate.message.error"
String value2 = "error"
String key3 = "activate.message.short.poll"
String value3 = "This is short poll"
I need to build a json like the following
{
"activate":{
"message":{
"success":"success",
"error":"error",
"short":{
"poll":"This is short poll"
}
},
"title":"Good Title"
}
}
I could not think of a proper solution for this use case and struggling for 3 hours. I thought of using recursion, but i dont how exactly i could do. Please help with this. I am using java for this, I should use generic JSONObject to solve as there is not POJO mappings. So far I have just splitted the strings using separtor and stored in an another map like the following
public Map<String, Object> getJsonObjectFromKeyValueMap(Map<String, String> stringValueMap,
Map<String, Object> stringObjectMap) {
for (Entry entry : stringValueMap.entrySet()) {
String[] keyValueArrayString = entry.getKey().toString().split("\\.");
int sizeOfMap = keyValueArrayString.length;
int i = 0;
String concatString = "";
for (String processKey : keyValueArrayString) {
if (i < sizeOfMap - 1) {
concatString += processKey + ".";
stringObjectMap.put(concatString, (Object) new JSONObject());
} else {
concatString += processKey;
stringObjectMap.put(concatString, entry.getValue());
concatString = "";
}
i++;
}
}
return stringObjectMap;
}
First, let's update your data into a proper map :
Map<String, String> data = new HashMap<>();
data.put("activate.message.success", "success");
data.put("activate.title", "Good Title");
data.put("activate.message.error", "error");
data.put("activate.message.short.poll", "This is short poll");
Then, your logic is pretty close, for each node but the last, you create a new JSONObject, for the last, you insert the value.
If you try to build a JSONObject instead of the map directly, you would get a pretty good result already, well somewhat of a result.
The following will iterate a Map<String, String> of data.
For each entry, we split the key to getting the nodes.
Then, we just need to move in the json, if a node doesn't exist, we create it.
Then, for the last value, create the value.
public static JSONObject build(Map<String, String> data) {
JSONObject json = new JSONObject();
//Iterate the map entries
for (Entry<String, String> e : data.entrySet()) {
String[] keys = e.getKey().split("\\.");
// start from the root
JSONObject current = json;
for (int i = 0; i < keys.length; ++i) {
String key = keys[i];
//Search for the current node
try {
//If it exist, do nothing
current = current.getJSONObject(key);
} //If it does not exist
catch (JSONException ex) {
//Is it the last node, create the value
if (i == keys.length - 1) {
current.put(key, e.getValue());
} //Not the last node, create a new JSONObject
else {
JSONObject tmp = new JSONObject();
current.put(key, tmp);
current = tmp; //Always replace current with the last node to go deeped each iteration
}
}
}
}
return json;
}
And the example :
public static void main(String[] args) {
Map<String, String> data = new HashMap<>();
data.put("activate.message.success", "success");
data.put("activate.title", "Good Title");
data.put("activate.message.error", "error");
data.put("activate.message.short.poll", "This is short poll");
JSONObject json = build(data);
System.out.println(json.toString(4));
}
Ouptut:
{"activate": {
"message": {
"success": "success",
"short": {"poll": "This is short poll"},
"error": "error"
},
"title": "Good Title"
}}
Note : I used an exception to check for the existance of the key, if the map is huge, this could have some impact so you can simply use :
if(current.isNull(key)){
if (i == keys.length - 1) {
current.put(key, e.getValue());
} else {
JSONObject tmp = new JSONObject();
current.put(key, tmp);
current = tmp;
}
} else {
current = current.getJSONObject(key);
}
This was created using org.json/json
I want store and access nested json value by key such as arrayOfObjectList.objectKey1 objectKey.objectKey1 and others. Now I store data in HashMap(String,Object) and I don't know how to access it by key
because json has indeterminate layer and inner data can be arrayList, object and string. Please introduce me to accomplish this task.
I am a beginner, sorry for my algorithm
code
Gson gson = new Gson();
HashMap<String,Object> param = new HashMap<>();
param = (HashMap<String, Object>) gson.fromJson(jsonData, param.getClass());
for (Map.Entry resourceItem : param.entrySet()) {
checkAndParseInternalLayer(param,resourceItem.getKey().toString());
}
method
void checkAndParseInternalLayer(HashMap<String, Object> data, String key) {
Object obj = data.get(key);
if (obj instanceof ArrayList) {
List<Object> internalList = new ArrayList<>();
for (Object dataObj : (ArrayList)obj) {
if (dataObj instanceof LinkedTreeMap) {
// get all key in LinkedTreeMap
Set<String> keySet = ((LinkedTreeMap) dataObj).keySet();
HashMap<String, Object> internalMap = new HashMap<>();
// set HashMap
for (String keyStr : keySet) {
AFLog.d("set map : "+keyStr+" = "+((LinkedTreeMap) dataObj).get(key));
internalMap.put(keyStr, ((LinkedTreeMap) dataObj).get(keyStr));
}
// check internal layer
for (HashMap.Entry entry : internalMap.entrySet()) {
checkAndParseInternalLayer(internalMap, entry.getKey().toString());
}
internalList.add(internalMap);
}
else if (dataObj instanceof String) {
internalList.add(dataObj);
}
}
data.put(key, internalList);
} else if (obj instanceof LinkedTreeMap) {
// get all key in LinkedTreeMap
Set<String> keySet = ((LinkedTreeMap) obj).keySet();
HashMap<String, Object> internalMap = new HashMap<>();
// set HashMap
for (String keyStr : keySet) {
internalMap.put(keyStr, ((LinkedTreeMap) obj).get(keyStr));
}
// check internal layer
for (HashMap.Entry entry : internalMap.entrySet()) {
checkAndParseInternalLayer(internalMap, entry.getKey().toString());
}
data.put(key, internalMap);
}
}
json data
{
"stringKey": "stringValue",
"valueStringList": [
"value1",
"value2"
],
"arrayOfObjectList": [
{
"objectKey1": "objectValue1",
"objectKey2": "objectValue2"
},
{
"objectKey1": "objectValue1",
"objectKey2": "objectValue2"
}
],
"objectKey": {
"objectKey1": "objectValue1",
"objectKey2": "objectValue2"
}
}
I have a Json String representing a object which further has another nested objects. Also, I have a list of keys which I need to remove from this Json String. These keys can be at any nested level of object inside this string. Finally I need to compare this edited Json string to another string and output the differences. I need to remove those key-value pairs from first Json string because I need to ignore those keys during comparison. Currently, I am converting the Json String to LinkedTreeMap provided by Gson API and then doing Map.difference() to compare. Please suggest a solution to this.
I did it by traversing recursively inside the Nested LinkedTreeMap till I find the field and remove it if it exists. The full path of Key needs to be provide to get the exact key-value location inside the Object (like "objects.desc" in the below Json Sample to remove desc from the Json String)
Json Sample:
{
"message": "MSG",
"code": "COD001",
"objects": [
{
"resource": "Student",
"field": "StudentId",
"desc": "Student Description"
}
]
}
Code Sample:
public MapDifference<String, Object> getMapDifference(String jsonString1, String jsonString2) {
MapDifference<String, Object> mapDifference = null;
Gson gson = new Gson();
Type mapType = new TypeToken<Map<String, Object>>() {
}.getType();
Map<String, Object> firstMap = gson.fromJson(jsonString1, mapType);
Map<String, Object> secondMap = gson.fromJson(jsonString2, mapType);
firstMap = CollectionUtils.isEmpty(firstMap) ? new HashMap<>() : firstMap;
secondMap = CollectionUtils.isEmpty(secondMap) ? new HashMap<>() : secondMap;
//This contains the List of keys that is required to be filtered out from Json Strings before comparision like {"message", "objects.desc"}
List<String> firstIgnoreList = getIgnoreList1();
List<String> secondIgnoreList = getIgnoreList2();
filterKeys(firstMap, firstIgnoreList);
filterKeys(secondMap, secondIgnoreList);
mapDifference = Maps.difference(firstMap, secondMap);
return mapDifference;
}
private void filterKeys(Map<String, Object> keyMap, List<String> ignoreList) {
if (!(CollectionUtils.isEmpty(keyMap) || CollectionUtils.isEmpty(ignoreList))) {
ignoreList.stream().parallel().forEach(key -> recursiveRemove(keyMap, key));
}
}
private static void recursiveRemove(Map<String, Object> keyMap, String key) {
List<String> path = Arrays.asList(StringUtils.split(key.trim(), "."));
int size = path.size();
int index = 0;
List<LinkedTreeMap> treeMapList = new ArrayList<LinkedTreeMap>();
treeMapList.add((LinkedTreeMap) keyMap);
while (index != size - 1) {
int i = index++;
List<LinkedTreeMap> treeMapListTemp = new ArrayList<LinkedTreeMap>();
treeMapList.stream().parallel().forEach(treeMap -> {
Object obj = treeMap.get(path.get(i));
if (obj instanceof List) {
treeMapListTemp.addAll((List<LinkedTreeMap>) obj);
} else if (obj instanceof LinkedTreeMap) {
treeMapListTemp.add((LinkedTreeMap) obj);
}
});
treeMapList = treeMapListTemp;
}
treeMapList.stream().parallel().forEach(treeMap -> treeMap.remove(path.get(size - 1)));
}
I have a json structure similar to this
{
"Byc-yes": { // < code
"Updated": "week", // < period
"Time": "12pm" // < time
},
"Uop-thp":{
"Updated": "week",
"Time": "12pm
} ,
...
I want to deserialize it to a Java class
class Updating {
private String code;
private String period;
private String time;
}
There any native JACKSON mappers to do this, or will I need to create my own custom deserializer for this?
I will read it as Map.class and then iterate through keyset to extract values.
ObjectMapper objectMapper = new ObjectMapper();
Map map = objectMapper.readValue(s, Map.class);
for (Object o : map.keySet()) {
String key = (String) o;
System.out.println(key);//prints Byc-yes for first
Map<String, String> value = (Map<String, String>) map.get(key);
System.out.println(value); //prints {Updated=week, Time=12pm} for first
Updating updating = new Updating(key, value.get("Updated"), value.get("Time"));
System.out.println(updating);
}
Assuming Updated and Time are fixed keys.
I have this json response:
{
"Name": "tardis",
"Driver": "custom",
"Mountpoint": "/var/lib/docker/volumes/tardis/_data",
"Status": {
"hello": "world"
},
"Labels": {
"com.example.some-label": "some-value",
"com.example.some-other-label": "some-other-value"
},
"Scope": "local"
}
From this response, I want to parse the key-value pairs inside "Labels" key. Currently my method to extract those looks like this:
//"json" is the JSON response as a string
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(json);
Iterator<Entry<String, JsonNode>> it = root.fields();
while (it.hasNext())
{
Map.Entry<String, JsonNode> entry = it.next();
JsonNode n = entry.getValue();
String nlabel = entry.getKey();
if (nlabel != null && nlabel.equals(key))
{
Iterator<Entry<String, JsonNode>> itr = n.fields();
while (itr.hasNext())
{
Map.Entry<String, JsonNode> it2 = itr.next();
String labelKey = it2.getKey();
String labelValue = it2.getValue().toString();
System.out.println();
}
}
}
I am able to retrieve those key value pairs using this logic, but I want to make it simpler because I want to make it work when there is a deeper nesting involved.
Is there a simpler, better way to do it using jackson libs?