I was wondering if anyone can help me or hint me towards how to edit the attached dummy JSON file in Java.
As you can see I have a head object that contains many values and children that follow the same pattern.
I wanted to know if there was a way to remove all the keys where the value is -1.
Following is what I was trying based on many websites using jackson:
try {
// create object mapper instance
ObjectMapper mapper = new ObjectMapper();
// convert JSON file to map
Map<?, ?> map = mapper.readValue(Paths.get("test.json").toFile(), Map.class);
// print map entries
for (Map.Entry<?, ?> entry : map.entrySet()) {
isInteger = main.isObjectInteger(entry.getValue());
// System.out.println("if value is all: " + entry.getKey() + "=" + entry.getValue());
//
The above code will display the structure of the file, however my problem is reaching the -1 values inside the children and removing them.
Using the .getClass and .simpleName methods I know that it is an arrayList but I am confused as to how to search through it.
Any help will be greatly appreciated!
In Jackson you can read whole JSON payload as JsonNode and iterate over all properties check given condition. In case condition is met you can remove given field. To do that you need to implement recursive method. Take a look on below example:
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.io.File;
import java.io.IOException;
import java.util.Iterator;
import java.util.Map;
import java.util.Objects;
import java.util.function.Predicate;
public class JsonRemoveSomeFieldsApp {
public static void main(String[] args) throws IOException {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper mapper = new ObjectMapper();
mapper.enable(SerializationFeature.INDENT_OUTPUT);
JsonNode root = mapper.readTree(jsonFile);
JsonCleaner jsonCleaner = new JsonCleaner(root, (node) -> node.isNumber() && node.numberValue().intValue() == -1);
JsonNode result = jsonCleaner.removeAll();
// write to file
mapper.writeValue(System.out, result);
}
}
class JsonCleaner {
private final JsonNode root;
private final Predicate<JsonNode> toRemoveCondition;
JsonCleaner(JsonNode node, Predicate<JsonNode> toRemoveCondition) {
this.root = Objects.requireNonNull(node);
this.toRemoveCondition = Objects.requireNonNull(toRemoveCondition);
}
public JsonNode removeAll() {
process(root);
return root;
}
private void process(JsonNode node) {
if (node.isObject()) {
ObjectNode object = (ObjectNode) node;
Iterator<Map.Entry<String, JsonNode>> fields = object.fields();
while (fields.hasNext()) {
Map.Entry<String, JsonNode> field = fields.next();
JsonNode valueToCheck = field.getValue();
if (valueToCheck.isContainerNode()) {
process(valueToCheck);
} else if (toRemoveCondition.test(valueToCheck)) {
fields.remove();
}
}
} else if (node.isArray()) {
ArrayNode array = (ArrayNode) node;
array.elements().forEachRemaining(this::process);
}
}
}
For below JSON payload:
{
"name": "Head",
"missed": -1,
"covered": -1,
"children": [
{
"name": "project1",
"missed": -1,
"covered": -1,
"children": [
{
"name": "project1",
"missed": 10,
"covered": 11
}
]
},
{
"name": "project1",
"missed": -1,
"covered": 12,
"children": [
{
"name": "project1",
"missed": 10,
"covered": -1
}
]
}
]
}
above code prints:
{
"name" : "Head",
"children" : [ {
"name" : "project1",
"children" : [ {
"name" : "project1",
"missed" : 10,
"covered" : 11
} ]
}, {
"name" : "project1",
"covered" : 12,
"children" : [ {
"name" : "project1",
"missed" : 10
} ]
} ]
}
See also:
Flattening a 3 level nested JSON string in java
There are two main techniques to parse and generate JSON data (as well as many other formats like XML etc): object mapping and event/token/stream-oriented processing. The second way is the best way for many cases, including filtering. Props:
the file/data doesn't require to be loaded entirely into memory, you
can process megs/gigs with no problems
it works much more faster, especially for large files
it's easy to implement any custom type/rule of transformation with this pattern
Both Gson and Jackson supports stream-oriented processing. To illustrate the idea here is just an example using a tiny parser/generator https://github.com/anatolygudkov/green-jelly
import org.green.jelly.AppendableWriter;
import org.green.jelly.JsonBufferedWriter;
import org.green.jelly.JsonEventPump;
import org.green.jelly.JsonNumber;
import org.green.jelly.JsonParser;
import java.io.StringWriter;
public class UpdateMyJson {
private static final String jsonToUpdate = "{\n" +
"\"name\": \"Head\",\n" +
"\"missed\": -1,\n" +
"\"children\": [\n" +
" {\n" +
" \"name\": \"project1\",\n" +
" \"fixes\": 0,\n" +
" \"commits\": -1,\n" +
" },\n" +
" {\n" +
" \"name\": \"project2\",\n" +
" \"fixes\": 20,\n" +
" \"commits\": 5,\n" +
" }\n" +
"]\n" +
"}";
public static void main(String[] args) {
final StringWriter result = new StringWriter(); // you can use FileWriter
final JsonParser parser = new JsonParser();
parser.setListener(new MyJsonUpdater(new AppendableWriter<>(result)));
parser.parseAndEoj(jsonToUpdate); // if you read a file with a buffer,
// to don't load the whole file into memory,
// call parse() several times (part by part) in a loop until EOF
// and then call .eoj()
System.out.println(result);
}
static class MyJsonUpdater extends JsonEventPump {
MyJsonUpdater(final JsonBufferedWriter output) {
super(output);
}
#Override
public boolean onNumberValue(final JsonNumber number) {
if (number.mantissa() == -1 && number.exp() == 0) {
return true; // return immediately
}
return super.onNumberValue(number); // otherwise pump the value to the result JSON
}
}
}
Related
I want to remove blob columns from JSON objects.
I need to check if any of the object has "#type": "blob", the entire column should be dropped.
Ex. following is a record from a DB. 'experience', 'hitpoints', 'name', 'uuid', 'image' (optional) are the columns. since the record has a blob column i.e image. It should be dropped.
Sample I/P:
{
"experience": 14248,
"hitpoints": 9223372036854775807,
"name": "Aaron1",
"uuid": "78edf902-7dd2-49a4-99b4-1c94ee286a33",
"image": {
"#type": "blob",
"content_type": "image/jpeg",
"digest": "sha1–4xlj1AKFgLdzcD7a1pVChrVTJIc=",
"length": 3888349
}
},
{
"experience": 14252,
"hitpoints": 92233720368512345,
"name": "Aaron2",
"uuid": "78edf902-7dd2-49a4-99b4-1a94ff286a45",
}
Sample O/P:
{
"experience": 14248,
"hitpoints": 9223372036854775807,
"name": "Aaron1",
"uuid": "78edf902-7dd2-49a4-99b4-1c94ee286a33",
},
{
"experience": 14252,
"hitpoints": 92233720368512345,
"name": "Aaron2",
"uuid": "78edf902-7dd2-49a4-99b4-1a94ff286a45",
}
Is there a way to achieve this by using optimized JSON parsing.
Currently, my logic follows the steps:
I'm parsing through the entire object using a function where I'm looping through the node to read the object.
Calling the 'blobChecker' function on every object.
Assigning null to the node if it contains blob.
Skipping the null node in the original function that invokes 'blobChecker'
The original function to parseJSON:
parseJsonNode(JsonNode node){
blobNodeChecker(node);
if(node!=null)
//The funtionality
}
The blobNodeChecker Function:
blobNodeChecker(JsonNode node) {
Boolean isBlob = false;
String blobNode = null;
Iterator<Map.Entry<String, JsonNode>> fields = node.fields();
while (fields.hasNext()) {
Map.Entry<String, JsonNode> next = fields.next();
String key = next.getKey();
String val = next.getValue().toString().toLowerCase();
if (key.equals("#type")) {
if (val.contains("blob")) {
isBlob = true;
break;
}
}
}
if (isBlob) {
node = null;
}
return node;
}
How about something like below. You can directly read a path and depending upon that delete a node. No need to loop all keys.
String tt = " {" +
" \"experience\": 14248," +
" \"hitpoints\": 9223372036854775807," +
" \"name\": \"Aaron1\"," +
" \"uuid\": \"78edf902-7dd2-49a4-99b4-1c94ee286a33\"," +
" \"image\": {" +
" \"#type\": \"blob\"," +
" \"content_type\": \"image/jpeg\"," +
" \"digest\": \"sha1–4xlj1AKFgLdzcD7a1pVChrVTJIc=\"," +
" \"length\": 3888349" +
" }" +
" }";
ObjectMapper mapper = new ObjectMapper();
mapper.enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT);
JsonFactory factory = mapper.getFactory();
JsonParser createParser = factory.createParser(tt);
JsonNode actualObj1 = mapper.readTree(createParser);
JsonNode path = actualObj1.path("image").path("#type");
if( path != null && "blob".equalsIgnoreCase(path.asText())) {
((ObjectNode)actualObj1).remove("image");
}
System.out.println(actualObj1.toString());
I have a JSON that I'm getting from some external system. I need to convert that JSON into key value based on my system mapping. For instance:
JSON from external system:
[{
"name": "Tim",
"address": "New York",
"education" : {
"university": "XYZ"
}
},
{
"name": "Steve",
"address": "UK"
}]
I have following mapping that we need to use:
{
"name": "firstName",
"address": "location",
"university": "college"
}
i.e name to be mapped to firstName and address to be mapped to location. And finally, my processed mapped will look like:
[{
"firstName": "Tim",
"location": "New York"
"education" : {
"college": "XYZ"
}
},
{
"firstName": "Steve",
"location": "UK"
}]
What can be the best way to achieve this? Should I use normal hashmap operations or is there any other efficient way. I was checking JSONNode for this purpose, but the approach was similar to hash map. Is there any utility I can use to traverse tree like json map and replace the key?
I'm a fan of Jackson which you can use to traverse the JSON. As you do so, populate a list of maps, using your system mapping to replace whatever keys are encountered that have a mapping, leaving the others as is. At the end, dump the list of maps back out as JSON.
Edit: adding code example below
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import java.io.IOException;
import java.util.*;
public class ReplaceKeys {
private static final ObjectMapper mapper = new ObjectMapper();
private static Map<String, String> keymap;
static {
try {
// Create the key map based on the json provided by OP
keymap = mapper.readValue("{\n" +
"\"name\": \"firstName\",\n" +
"\"address\": \"location\",\n" +
"\"university\": \"college\"\n" +
"}", Map.class);
} catch (IOException e) {
System.out.println("whoops");
}
}
public static String mapKeys(String input) throws IOException {
// Assume the input is an array and therefore so is the output.
List<Map<String, Object>> output = new ArrayList<>();
ArrayNode inputArray = (ArrayNode) mapper.readTree(input);
for (JsonNode node : inputArray) {
output.add(mapKeys(node));
}
return mapper.writeValueAsString(output);
}
private static Map<String, Object> mapKeys(JsonNode node) {
Map<String, Object> map = new HashMap<>();
for (Iterator<String> iterator = node.fieldNames(); iterator.hasNext(); ) {
String key = iterator.next();
key = keymap.containsKey(key) ? keymap.get(key) : key;
for (JsonNode child : node) {
if (node.isValueNode()) {
// This is coercing everything to a String. You could dig into using
// proper types based on node.getNodeType().
map.put(key, node.asText());
} else {
map.put(key, mapKeys(child));
}
}
}
return map;
}
}
I have the following JSON source:
{
"my-systems": [{
"SYSTEM_A": [{
"parameter-name": "a_param1",
"parameter-display-name": "parameter 1",
"optional": "true"
},
{
"parameter-name": "a_param2",
"parameter-display-name": "Parameter 2",
"optional": "false"
}
]
},
{
"SYSTEM_B": [{
"parameter-name": "b_param1",
"parameter-display-name": "Parameter 1",
"optional": "true"
},
{
"parameter-name": "b_param2",
"parameter-display-name": "Parameter 2",
"optional": "false"
}
]
}
]
}
I try to read it into a map of Map<String, SystemParameter[]>.
I have this code which I'm really not sure if it's the best approach for my goal.
ArrayNode systemsArr = (ArrayNode)jsonDoc.get("my-systems");
if(systemsArr!= null && !systemsArr.isEmpty()){
for(JsonNode systemNode : systemsArr ){
ObjectNode systemObj = (ObjectNode)systemNode;
System.out.println(systemObj .textValue());
}
}
Is it a valid approach? How do I get the name of the system (SYSTEM_A, SYSTEM_B) and convert the contained parameters into a parameters objects array?
You just need to use jackson-databind and jackson-annotations jar in your dependency and you should be able to run below code
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class JsonParsing {
public static void main(String[] args) throws IOException {
String jsonFilePath = "src/main/resources/source.json"; // JSON File Path
MySystems mySystems = new ObjectMapper().readValue(new File(jsonFilePath),MySystems.class);
Map<String,SystemParameter[]> outputMap = new HashMap<>();
for (Map<String,List<SystemParameter>> map :mySystems.mySystems) {
for (String key :map.keySet()) {
outputMap.put(key, map.get(key).toArray(new SystemParameter[0]));
}
}
System.out.println(outputMap);
}
}
class MySystems {
#JsonProperty("my-systems")
List<Map<String,List<SystemParameter>>> mySystems;
}
class SystemParameter {
#JsonProperty("parameter-name")
String paramName;
#JsonProperty("parameter-display-name")
String paramDispName;
#JsonProperty("optional")
String optional;
}
Using an array 'my-systems' is of little use if all you're doing is keeping your keys unique. (I am assuming your SYSTEM_A may be different).
Instead, I suggest you format the JSON data in the following way:
{
"my-systems": [
{
"system-name" : {
"name":"System_A",
"parameters": [
{
"parameter-name": "a_param1",
"parameter-display-name": "parameter 1",
"optional": "true"
},
{
"parameter-name": "a_param2",
"parameter-display-name": "Parameter 2",
"optional": "false"
}
]
}
}
]
}
This method is clean, and allows you to catch the name of the system under the property 'system-name' and it's parameters are nested inside. You can simply declare a model and Jackson (or gson) will simply take care of everything.
If you'd rather parse the response separately or you have no control over the response, you can opt to go with your implementation but you do not need to convert to ObjectNode. You can use the samples below inside your for loop:
String f2Str = jsonNode.get("f2").asText();
double f2Dbl = jsonNode.get("f2").asDouble();
int f2Int = jsonNode.get("f2").asInt();
long f2Lng = jsonNode.get("f2").asLong();
where 'f2' is your key, and you can get the key using node.fieldNames().next() which is actually getting the property from an iterator.
You may also try the ones below, they seem better to handle.
JsonNode parent = ...;
for (Iterator<String> it = parent.fieldNames() ; it.hasNext() ; ) {
String field = it.next();
System.out.println(field + " => " + parent.get(field));
}
for (Iterator<Map.Entry<String,JsonNode>> it = parent.fields() ;
it.hasNext() ; ) {
Map.Entry<String,JsonNode> e = it.next();
System.out.println(e.getKey() + " => " + e.getValue());
}
The latest examples are taken from here. Hope this helps.
I have a JSON file from which I need to remove node objects which contain a data value of null. Can this be done? I'm using Jackson.
In the sample JSON below, I need to remove the object where its "v" tag has a null value.
Example:
{
"tags" : [ {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:27:36",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90WN1OUT",
"data" : [ {
"ts" : "2019-03-20T15:27:36",
"v" : null,
"q" : "0"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:20",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90WN1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:20",
"v" : null,
"q" : "0"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:37",
"v" : "96.2581177",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90WN1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:37",
"v" : null,
"q" : "0"
} ]
} ]
}
I need it to look like this:
{
"tags" : [ {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:27:36",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:20",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:37",
"v" : "96.2581177",
"q" : "3"
} ]
} ]
}
Can this be done? Please show me how. Fairly new to JSON manipulation, I've seen another post that kind of shows how to remove an element from a node, but I think my case is a little different. I've tried chasing down documentation to no avail, maybe looking in the wrong places.
Thank you in advance.
JSONPath
For JSON manipulation and filtering you can also use JsonPath library. It has a great web tool where you can try different filters and options. We can filter all nodes wit not null values using below path:
$.tags[?(#.data[0].v != null)]
Example application which does the same:
import com.jayway.jsonpath.JsonPath;
import net.minidev.json.JSONArray;
import net.minidev.json.JSONObject;
import java.io.File;
public class JsonPathApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
JSONArray filtered = JsonPath.parse(jsonFile).read("$.tags[?(#.data[0].v != null)]");
// Create root object
JSONObject root = new JSONObject();
root.appendField("tags", filtered);
// Get JSON
String json = root.toString();
// Write JSON on console or file
System.out.println(json);
}
}
Above code prints:
{"tags":[{"tagId":"G1.A_90LT1OUT","data":[{"ts":"2019-03-20T15:27:36","v":"96.2427826","q":"3"}]},{"tagId":"G1.A_90LT1OUT","data":[{"ts":"2019-03-20T15:29:20","v":"96.2427826","q":"3"}]},{"tagId":"G1.A_90LT1OUT","data":[{"ts":"2019-03-20T15:29:37","v":"96.2581177","q":"3"}]}]}
Jackson
The same with Jackson we can achieve in that way:
Read JSON as tree
Go to tags array
Iterate over array
For each item find v key in 0-index element
In case it is null - remove it
Example implementation:
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import java.io.File;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class JsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(jsonFile);
ArrayNode tags = (ArrayNode) root.get("tags");
Iterator<JsonNode> elements = tags.elements();
while (elements.hasNext()) {
JsonNode item = elements.next();
ArrayNode data = (ArrayNode) item.get("data");
JsonNode v = data.get(0).get("v");
if (v.isNull()) {
elements.remove();
}
}
System.out.println(root);
}
}
I have to read a JSON without knowing the keys.
My JSON is like this:
{
"hello": {
"bye": {
"a": "1",
"b": "2",
"c": "3"
},
"d": "4",
"e": "5"
},
"hey": {
"qwer": {
"asdf": "1",
"fda": "2",
"bc": "3"
},
"dq": "4"
}
}
just with more data and more elements.
I would like to access them like a map using a path, just with e.g.
get() for hello&hey,
get("hello") for bye,d&e,
get("hello","bye") for a,b&c
...
I hope you understood me, I usually use Gson to parse JSON but passing a corresponding class (like AClass c = new Gson.fromJson(myJson, AClass.class);), but without knowing the JSON structure, I cannot use this approach.
It would be also great if I could easily remove and add entries somewhere.
To do what you need, you can still use Gson, but you need a more "low level" class like the JsonParser.
Here I provide you an example on how you can navigate the JsonElement tree generated by the parser to reach the requested node. It works only on syntaxes like you requested (node1/node2/..) but you can easily adapt also to cases like this: node1/3/node3/4/2 where the number indicates the child.
When you reach the node, you can parse using a Gson if you need, since you can pass to Gson::fromJson a JsonElement also. Here's the code:
package stackoverflow.questions;
import java.util.*;
import com.google.gson.*;
public class Q20883087 {
public static JsonElement get(JsonElement je, String... args){
if(args.length == 0)
return je;
if (je != null && je.isJsonObject()){
String[] newArgs = Arrays.copyOfRange(args, 1, args.length);
return get(je.getAsJsonObject().get(args[0]), newArgs);
}
}
public static void main(String[] args) {
String json ="{ "+
" \"hello\": { "+
" \"bye\": { "+
" \"a\": \"1\", "+
" \"b\": \"2\", "+
" \"c\": \"3\" "+
" }, "+
" \"d\": \"4\", "+
" \"e\": \"5\" "+
" }, "+
" \"hey\": { "+
" \"qwer\": { "+
" \"asdf\": \"1\", "+
" \"fda\": \"2\", "+
" \"bc\": \"3\" "+
" }, "+
" \"dq\": \"4\" "+
" } "+
" } ";
JsonElement jsonElement = new JsonParser().parse(json);
System.out.println(get(jsonElement, "hello","bye"));
System.out.println(get(jsonElement, "hey","qwer"));
}
}
The result is:
{"a":"1","b":"2","c":"3"}
{"asdf":"1","fda":"2","bc":"3"}
Pay attention to the recursive nature of the get method, since Json is recursive by nature, the best way to manage it is using a recursive method.