Develop Reference

Recursively flatten a map of nested objects

I have a list of map of fields from ElasticSearch in a JSON structure. I need to extract the keys from the fields into a name.value list to be used as search terms.
For example, the response I get from ElasticSearch looks like:
{
"orange": {
"type": "keyword"
},
"apple": {
"type": "keyword"
},
"banana": {
"type": "keyword"
},
"pineapple": {
"properties": {
"color": {
"type": "text"
},
"size": {
"type": "text"
}
}
},
"vegetables": {
"properties": {
"potato": {
"properties": {
"quality": {
"type": "keyword"
},
"price": {
"type": "keyword"
},
"location": {
"type": "keyword"
}
}
}
}
}
}
I need to transform this into a list of
[
"orange",
"apple",
"banana",
"pineapple.color",
"pineapple.size",
"vegetables.potato.quality",
"vegetables.potato.price",
"vegetables.potato.location",
"vegetables.cabbage"
]
I'm a bit lost as to where to start so I end up with something that will work no matter how deep the "object" + "properties" key ends up being.
edit:
I have a couple of methods I'm trying to do this with, but I keep ending up with nested loops instead
private static String process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final Map<String, Object> value = toSourceMap(entry.getValue());
if (value.containsKey("properties")) {
final Map<String, Object> properties = toSourceMap(value.get("properties"));
process(entry); // ??
}
return fieldName;
}
And a small helper method I'm using which casts the unknown object to a map
private static Map<String, Object> toSourceMap(final Object sourceMap) {
try {
final Map<String, Object> map = (Map) sourceMap;
return map;
} catch (final Exception e) {
return Map.of();
}
}
And I'm calling this
final List<String> fieldName = new ArrayList<>();
for (final Map.Entry<String, Object> entry : properties.entrySet()) {
fieldName.add(process(entry));
}
Trying to get a list of each value from the process method
edit 2:
I can get something that works for one level deep, but this won't capture the deeper objects like vegetables.potato.quality
private static List<String> process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final Map<String, Object> value = toSourceMap(entry.getValue());
final List<String> fields = new ArrayList<>();
if (value.containsKey("properties")) {
final Map<String, Object> properties = toSourceMap(value.get("properties"));
properties.keySet().stream().map(s -> fieldName + "." + s).forEach(fields::add);
} else {
fields.add(fieldName);
}
return fields;
}
and the caller
final List<String> fieldName = new ArrayList<>();
for (final Map.Entry<String, Object> entry : properties.entrySet()) {
fieldName.addAll(process(entry));
}

I'm sure theres a cleaner and better way to do it, but I was able to achieve what I needed by doing the following
private static List<String> process(final Map.Entry<String, Object> entry) {
final String fieldName = entry.getKey();
final List<String> fields = new ArrayList<>();
final Map<String, Object> value = toSourceMap(entry.getValue());
if (value.containsKey(MAPPING_PROPERTIES)) {
toSourceMap(value.get(MAPPING_PROPERTIES)).entrySet()
.stream()
.map(FieldMappingFactory::process)
.map(nestedFields -> nestedFields.stream().map(f -> "%s.%s".formatted(fieldName, f)).toList())
.forEach(fields::addAll);
} else {
fields.add(fieldName);
}
return fields;
}

Here's a solution based on the Depth first search tree-traversal algorithm.
Since it's iterative, you can use it to process even deeply nested massive JSON without a risk of getting a StackOverFlowError.
To implement DFS, we need a Stack and a Map is needed to store the paths associated with a node that are being explored.
As the first step, we need to read the whole JSON tree and the obtained JsonNode on the stack.
Then, until the stack is not empty, need to examine each node it contains by pulling them out from the stack. If the node happens to be an ArrayNode or ObjectNode, then all its children-nodes which could be obtained via JsonNode.fields() should be added on the stack.
String json = // the source JSON
JsonNode node = new ObjectMapper().readTree(json);
List<String> results = new ArrayList<>();
Deque<JsonNode> stack = new ArrayDeque<>();
Map<JsonNode, List<String>> pathByNode = new HashMap<>();
pathByNode.put(node, Collections.emptyList());
Set<String> keysToExclude = Set.of("type", "properties"); // add more if you need
stack.push(node);
while (!stack.isEmpty()) {
JsonNode current = stack.pop();
List<String> path = pathByNode.get(current);
if (current instanceof ArrayNode || current instanceof ObjectNode) {
for (Iterator<Map.Entry<String, JsonNode>> it = current.fields(); it.hasNext(); ) {
Map.Entry<String, JsonNode> next = it.next();
stack.push(next.getValue());
String propertyName = next.getKey();
List<String> newPath;
if (!keysToExclude.contains(propertyName)) {
newPath = new ArrayList<>(path);
newPath.add(propertyName);
results.add(String.join(".", newPath)); // list of path should be updated
} else {
newPath = path;
}
pathByNode.put(next.getValue(), newPath);
}
}
}
results.forEach(System.out::println);
Output:
orange
apple
banana
pineapple
vegetables
vegetables.potato
vegetables.potato.quality
vegetables.potato.price
vegetables.potato.location
pineapple.color
pineapple.size

This is a simpler solution.
private static void process(List<String> paths, String path, JsonNode node, Set<String> excludeKeys) {
if (node.isValueNode()) {
paths.add(path);
} else if (node.isObject()) {
node.fields().forEachRemaining(elem -> {
String key = excludeKeys.contains(elem.getKey()) ? path : (path == null ? "" : path + ".") + elem.getKey();
process(paths, key, elem.getValue(), excludeKeys);
});
} else { // This part is not required if there is no array inside the source JSON object
for (int i = 0; i < node.size(); i++) {
process(paths, String.format("%s[%d]", path, i), node.get(i), excludeKeys);
}
}
}
Caller
JsonNode node = new ObjectMapper().readTree(// JSON string //);
List<String> paths = new ArrayList<>();
process(paths, null, node, Set.of("properties", "type"));
paths.forEach(System.out::println);
Output
orange
apple
banana
pineapple.color
pineapple.size
vegetables.potato.quality
vegetables.potato.price
vegetables.potato.location

How to add JSON values to an ArrayList in Java

I have the following JSON file:
{
"meta" : {
"stock" : "AWS",
"date modified" : 90
},
"roles" : [ "Member", "Admin" ],
"name" : "John Doe",
"admin" : true,
"email" : "john.doe#example.com"
}
I wanted to both read the values of the keys and add them to an Array List.
try {
// create object mapper instance
ObjectMapper mapper = new ObjectMapper();
// convert JSON file to map
Map<?, ?> map = mapper.readValue(Paths.get("user.json").toFile(), Map.class);
ArrayList<String> data = new ArrayList<String>();
// print map entries
for (Map.Entry<?, ?> entry : map.entrySet()) {
System.out.println((entry.getClass()) + " " + entry.getValue());
data.add((String)entry.getValue()); // trying to add entry values to arraylist
}
} catch (Exception ex) {
ex.printStackTrace();
}
I'm able to print out the data type of the value along with the value itself. All the values are part of class java.util.LinkedHashMap$Entry. I'm not able to cast the values to a String to add them to an ArrayList. How should I go about doing this? Thanks

From the jackson-databind documentation you can convert your json to a Map<String, Object> map with the following line (you have boolean, list, number and string values in your json) :
Map<String, Object> map = mapper.readValue(json, Map.class);
// it prints {meta={stock=AWS, date modified=90}, roles=[Member, Admin], name=John Doe, admin=true, email=john.doe#example.com}
System.out.println(map);
If you want to save your map values string representation into an ArrayList data you can iterate over them with a loop :
List<String> data = new ArrayList<>();
for (Object value : map.values()) {
data.add(value.toString());
}
//it will print [{stock=AWS, date modified=90}, [Member, Admin], John Doe, true, john.doe#example.com]
System.out.println(data);

Your data type of entries will be like:
meta: Map<String:Object>
roles: List<String>
admin: Boolean
So you will get an exception when casting to string for each entry value.
You should handle different data type and convert it according to your request:
Object value = entry.getValue();
I highly recommend you write more few functions to check and convert map/list/primitive variables to expected data (String):
boolean isList(Object obj);
boolean isMap(Object obj);
...
public List<String> convertMap(Map<String,Object> map);
...

Parsing Json file and create Hashmap<k,v>

I am finding a solution for get each single key value of a JSONObject dynamicaly.
I have a JSON file like this:
{
"RESPONSE":{
"A":"test",
"B":{
"C":"0",
"D":"1"
},
"E":{
"F":"2",
"G":"3"
}
}
}
I wish to create a Hashmap that contains the key and the value of the objects, like this example:
legend:
key = value
"A" = "test"
"B" = "{"C":"0", "D":"1"}"
"B.C" = "0"
"B.D" = "1"
"E" = "{"F":"2","G":"3"}"
"E.F" = "2"
"E.G" = "3"
I have tried writing this code:
private static HashMap<String, Object> createJsonObjectsHashMapOfConfigFile(String jsonFilePath,
String JsonObjectToIterate) throws Exception {
HashMap<String, Object> jHashMap = new HashMap<>();
JSONObject jsonConfigFile = SimulatorUtil.readJsonFromFile(jsonFilePath); // Returns the Json file
JSONObject jsonConfigObj = jsonConfigFile.getJSONObject(JsonObjectToIterate); // JsonObjectToIterate -->
// "RESPONSE"
Iterator<?> configIter = jsonConfigObj.keys();
String currentDynamicKey = "";
while (configIter.hasNext()) {
currentDynamicKey = (String) configIter.next();
Object currentDynamicValue = jsonConfigObj.get(currentDynamicKey);
jHashMap.put(currentDynamicKey, currentDynamicValue);
logger.info(" ---> " + jHashMap.toString());
}
return jHashMap;
}
but it get only the first level key ("A", "B", "E").
I must use org.json.
Can anyone help me?
Thanks,
Luca

Parsing unnamed nested arrays with minimal-json?

So I'm working on a fairly simple Java program which grabs market data from cryptocurrency exchanges and displays information to the user. I am using the minimal-json library.
Here is my current code:
public class Market {
static JsonArray arrayBittrex;
public static void startTimer(){
Timer timer = new Timer();
timer.scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
String url = "https://bittrex.com/api/v1.1/public/getmarketsummaries";
try {
URL url2 = new URL(url);
URLConnection con = url2.openConnection();
InputStream in = con.getInputStream();
String encoding = "UTF-8";
String body = IOUtils.toString(in, encoding);
arrayBittrex = Json.parse(body).asObject().get("result").asArray();
}
catch(MalformedURLException e) {}
catch(IOException e) {}
}
}, 0,5000);
}
public static float getPrice(String exchange, String market) {
for (JsonValue item : arrayBittrex) {
float last = item.asObject().getFloat("Last", 0);
System.out.println(last);
return last;
}
return 0;
}
}
This code works with simple json, for example (from https://bittrex.com/api/v1.1/public/getmarketsummary?market=btc-ltc):
{
"success" : true,
"message" : "",
"result" : [{
"MarketName" : "BTC-LTC",
"High" : 0.01350000,
"Low" : 0.01200000,
"Volume" : 3833.97619253,
"Last" : 0.01349998
}
]
}
It will properly return the "Last" value in the array.
However, this cant work when the json has multiple arrays (like in https://bittrex.com/api/v1.1/public/getmarketsummaries):
{
"success" : true,
"message" : "",
"result" : [{
"MarketName" : "BTC-888",
"High" : 0.00000919,
"Low" : 0.00000820,
"Volume" : 74339.61396015,
"Last" : 0.00000820
}, {
"MarketName" : "BTC-A3C",
"High" : 0.00000072,
"Low" : 0.00000001,
"Volume" : 166340678.42280999,
"Last" : 0.00000005
}
]
}
So my question is: how can I get the "Last" value by searching for the array by the "MarketName" value?

Here is a direct & null-safe way to tackle this using Java 8 library Dynamics. We're going to parse the json into a Map, read that map dynamically to what we want.
So first we can use Jackson, Gson or something to convert json -> map.
// com.fasterxml.jackson.core:jackson-databind json -> map
Map jsonMap = new ObjectMapper()
.enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS)
.readValue(jsonStringOrInputSourceEtc, Map.class);
We can now get a Dynamic instance. And, for example, grab the BTC-A3C - Last value.
Dynamic json = Dynamic.from(jsonMap);
BigDecimal a3cLast = json.get("result").children()
.filter(data -> data.get("MarketName").asString().equals("BTC-A3C"))
.findAny()
.flatMap(data -> data.get("Last").maybe().convert().intoDecimal())
.orElse(BigDecimal.ZERO);
// 5E-8
Or perhaps convert the whole lot into a map of MarketName -> Last value
Map<String, BigDecimal> marketNameLastValue = json.get("result").children()
// assume fields are always present, otherwise see #maybe() methods
.collect(toMap(
data -> data.get("MarketName").asString(),
data -> data.get("Last").convert().intoDecimal()
));
// {BTC-A3C=5E-8, BTC-888=0.00000820}
See more examples https://github.com/alexheretic/dynamics

retrieve array from mongodb using java with mongodb api

I understand that there are many questions which as for the same and they are answered well. The problem is all those questions use MongoDBObject, MongoDBList to retrieve arrays. My problem is I am using http://api.mongodb.org/java/3.0/index.html?overview-summary.html api and I am having hard time retrieving array and adding elements to it. I have to use MongoCollection, MongoDatabase and MongoClient. I am trying to solve an assignment from mongodb course. The problem statement is to find an array and update it in mongod.
Here is what I have tried
Document post = null; Bson filter = new Document("permalink",
permalink); Bson projection = new Document("comments", true);
List<Document> comments = postsCollection.find(filter)
.projection(projection).into(new ArrayList<Document>());
System.out.println(comments);
post = postsCollection.find(Filters.eq("permalink",
permalink)).first();
Document newComment = new Document();
newComment.append("author", name); newComment.append("body", body);
if (email != null && (!email.equals(""))) {
newComment.append("email", email); }
comments.add(newComment);
Bson filter2 = new Document("_id", post.get("_id"));
System.out.println(comments); post =
postsCollection.find(filter).first();
postsCollection.updateOne(filter2, new Document("$unset",new
Document("comments",true))); postsCollection.updateOne(filter2, new
Document("$set", new Document( "comments", comments)));
This does not create a new comment. Instead, it creates another comments array in comments array itself. THe array should be updated in student
Here is the json data
{
"_id" : ObjectId("55d965eee60dd20c14e8573e"),
"title" : "test title",
"author" : "prasad",
"body" : "test body",
"permalink" : "test_title",
"tags" : [
"test",
"teat"
],
"date" : ISODate("2015-08-23T06:19:26.826Z"),
"comments" : [
{
"_id" : ObjectId("55d965eee60dd20c14e8573e"),
"comments" : [
{
"_id" : ObjectId("55d965eee60dd20c14e8573e"),
"comments" : []
},
{
"author" : "commented",
"body" : "something in comment",
"email" : "some#thing.com"
}
]
},
{
"author" : "commented",
"body" : "something in comment",
"email" : "some#thing.com"
}
]
}

To avoid unchecked casts and linter warnings, along with writing your own loop, use the libary's get(final Object key, final Class<T> clazz) method:
List<Document> comments = posts.get("comments", docClazz)
where docClazz is something that you create once:
final static Class<? extends List> docClazz = new ArrayList<Document>().getClass();

You need not write to this much code. Please check following code,
public void addPostComment(final String name, final String email, final String body,
final String permalink) {
Document post = findByPermalink(permalink);
List<Document> comments = null;
Document comment = new Document();
if(post != null){
comments = (List<Document>)post.get("comments");
comment.append("author",name).append("body", body);
if(email != null){
comment.append("email", email);
}
comments.add(comment);
postsCollection.updateOne(new Document("permalink",permalink),
new Document("$set",new Document("comments",comments)));
}
}

This is much simplified here!
version - mongo-java-driver-3.12.5.jar
comments = post.getList("comments", Document.class);

If you're forced to use older version of mongo driver and you can't use the method the MKP has mentioned, then you can copy the method itself.
Here it is as a Kotlin extension
import org.bson.Document
import java.lang.String.format
fun <T> Document.getList(key: String, clazz: Class<T>, defaultValue: List<T>): List<T> {
val list = this.get(key, List::class.java)
if (list == null) {
return defaultValue
}
list.forEach {
if (!clazz.isAssignableFrom(it!!::class.java)) {
throw ClassCastException(format("List element cannot be cast to %s", clazz.getName()))
}
}
return list as List<T>
}