I am working on a use case where I need to parse different JSON strings and query for specific fields basing on the "type" of the JSON. The "type" is a field in the JSON string.
I am using Jackson API to perform this task after going through the blogs and benchmarks, as it is the fastest.
I am able to parse the JSON and achieve what I want but the issue is with the performance.
public String generate(String inputJson, List<String> idParams,final String seperator) throws Exception {
JsonNode node;
StringBuilder sb = new StringBuilder();
try {
node = new ObjectMapper().readTree(new StringReader(inputJson));
idParams.forEach(e -> {
String value = node.findValue(e).asText();
sb.append(value).append(seperator);
});
} catch (Exception e) {
throw e;
}
return sb.toString();
}
In the above method, I am getting the field details as a List. With the help of forEach(), i am able to fetch the values by finding using the fields.
The culprit is the list iterator as I have to search the whole json tree to find the value for each element. Is there a better approach to optimize. I would also like to get the inputs on other JSON parsing libraries which can improve the performance here.
I am also thinking of parsing the whole json once and writing the Keys and Values to a HashMap. But, I have very few fields which i really care about the remaining fields are not needed.
take a look at JsonPath . It offers xpath-like rich query language that allows for search and retrieval of individual or few elements from the JSON tree.
Consider using Jackson Streaming API -
https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi
Take a look at this example -
http://www.baeldung.com/jackson-streaming-api
Related
I have an API request from my CRM that can either return a jsonObject if there is only one result, or a jsonArray if there are multiple results. Here are what they look like in JSON Viewer
JsonObject:
JsonArray:
Before you answer, this is not my design, it's my CRM's design, I don't have any control over it, and yes, I don't like how it is designed either. The only reason I am not storing the records in my own database and just parsing that, which would be MUCH easier, is because my account is having issues not running some workflows that would allow me to auto add the records. Is there any way to figure out if the result is an object or an array using java? This is for an android app by the way, I need it to display the records on the phone.
You should use OPT command instead of GET
JSONObject potentialObject=response.getJsonObject("resuslt")
.getJsonObject("Potentials");
// here use opt. if the object is null, it means its not the type you specified
JSONObject row=potentialObject.optJsonObject("row");
if(row==null){
// row is json array .
JSONArray rowArray=potentialObject.getJsonArray("row");
// do whatever you like with rowArray
} else {
// row is json object. do whatever you like with it
}
ONE
You can use instanceof keyword to check the instances as in
if(json instanceof JSONObject){
System.out.println("object");
}else
System.out.println("array");
TWO
BUT I think a better way to do this is choose to use only JSONArray so that the format of your results can be predicated and catered for. JSONArrays can contain JSONObjects. That is they can cover the scope of JSONObject.
For example when you get the response (either in a JSONObject or a JSONArray), you need to store that in an instance. What instance are you going to store it in? So to avoid issues use JSONArray to store the response and provide statements to handle that.
THREE
Also you can try method overloading in java or Generics
Simplest way is to use Moshi, so that you dont have to parse, even in the case of the Model changing later, you have to change your pojo and it will work.
Here is the snippet from readme
String json = ...;
Moshi moshi = new Moshi.Builder().build();
JsonAdapter<BlackjackHand> jsonAdapter = moshi.adapter(BlackjackHand.class);
BlackjackHand blackjackHand = jsonAdapter.fromJson(json);
System.out.println(blackjackHand);
https://github.com/square/moshi/blob/master/README.md
Since my very first days of Java + JSON I tried to extract just some certain parts of a JSON.
But no matter if which of the libraries I used:
Gson
json-simple
javax.json
it never was possible to make it quick and comfortable. Mostly for easy task or even prototyping. It already cost me many hours of different approaches.
Going trough the hierarchy of an JSON
Object jsonObject = gson.fromJson(output, Object.class);
JsonElement jsonTree = gson.toJsonTree(jsonObject);
JsonArray commitList = jsonTree.getAsJsonArray();
JsonElement firstElement = commitList.get(0);
JsonObject firstElementObj = firstElement.getAsJsonObject();
System.out.println(firstElementObj.get("sha"));
JsonElement fileList = firstElementObj.get("files");
This is dirty code for a reason. It shows how many early approaches looks like and how many people cannot achieve it to do it better early.
Deserializing JSON to a Java Object
Your have to analyse the complete JSON to create an complete Java-Object representation just to get access to some single memebers of it. This is a way I never wanted to do for prototyping
JSON is an easy format. But using libraries like that is quite difficult and often an problem for beginner. I've found several different answers via Google and even StackOverflow. But most were quite big larged which required to create a own specific class for the whole JSON-Object.
What is the best approach to make it more beginner-friendly?
or
What is the best beginner-friendly approach?
Using Jackson (which you tagged), you can use JsonPointer expressions to navigate through a tree object:
ObjectMapper mapper = new ObjectMapper();
JsonNode tree = mapper
.readTree("[ { \"sha\": \"foo\", \"files\": [ { \"sha\": \"bar\" }, { \"sha\": \"quux\" } ] } ]");
System.out.println(tree.at("/0/sha").asText());
for (JsonNode file : tree.at("/0/files")) {
System.out.println(file.get("sha").asText());
}
You could also use the ObjectMapper to convert just parts of a tree to your model objects, if you want to start using that:
for (JsonNode fileNode : tree.at("/0/files")) {
FileInfo fileInfo = mapper.convertValue(fileNode, FileInfo.class);
System.out.println(fileInfo.sha);
}
If your target class (FileInfo) specifies to ignore unknown properties (annotate target class with #JsonIgnoreProperties(ignoreUnknown = true) or disable DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES on the mapper), then you can simply declare the properties you are interested in.
"Best" is whatever works to get you going.
Generate Plain Old Java Objects from JSON or JSON-Schema
One little helper I found via my research was an Online-Tool like
http://www.jsonschema2pojo.org/
This is a little help, when you know about that. But the negative side I mentioned at point 2 is still there.
You can use JsonSurfer to selectively extract value or object from big json with streaming JsonPath processor.
JsonSurfer jsonSurfer = JsonSurfer.gson();
System.out.println(jsonSurfer.collectOne(json, "$[0].sha"));
System.out.println(jsonSurfer.collectOne(json, "$[0].files"));
I'm a beginner of Scala, and I have JSON data formatted like below:
{
"index_key": {
"time":"12938473",
"event_detail": {
"event_name":"click",
"location":"US"
}
}
}
I'm trying to get the content of "index_key" and extract the content of second level as a new JSON objet and initiate a class based on the second level data.
{
"time":"12938473",
"event_detail": {
"event_name":"click",
"location":"US"
}
}
I tried to use json4s to extract from the above json to be a Event class, but how to get rid of the "index_key" which is the first level key?
case class Detail(event_name: String, location: String)
case class Event(time: String, event_detail: Detail)
json.extract[Event]
I've read json4s documentation, and also http://www.scala-lang.org/api/2.10.3/index.html#scala.util.parsing.json.JSON$, but still don't quite get it, as it seems the pre-defined json should be fit for the parser?
Could anyone please tell me how to get the second level data (or any lower level) of the json structure?
You can use \ to get to the object you want to extract:
val json = parse(str) \ "index_key"
json.extract[Event]
I'm starting out to collect logs with logstash. The current setup consist of a Java server using logback as logging mechanism and logstash-logback-encoder, outputting the data in a neat JSON representation. The basics work just fine.
I would like to separate additional data in JSON format in separate fields (so each key of the JSON ends up in its own field). logstash-logback-encoder provides a mechanism for that to output such data in a json_mesage field. However this JSON string is placed into a JSON array. See here a sample formatted for better reading.
{
"#timestamp":"2014-03-25T19:34:11.586+01:00",
"#version":1,
"message":"Message{\"activeSessions\":0}",
"logger_name":"metric.SessionMetrics",
"thread_name":"scheduler-2",
"level":"INFO",
"level_value":20000,
"HOSTNAME":"stage-01",
"json_message":["{\"activeSessions\":0}"],
"tags":[]
}
I tried to parse the incoming JSON using a simple JSON filter. See here my configuration:
input {
lumberjack {
<snipped>
codec => "json"
}
}
filter {
json {
source => "json_message"
}
}
output {
elasticsearch {
<snipped>
}
}
However this leads to following error in the logstash log. The JSON string in an array simply can't be handled.
{:timestamp=>"2014-03-25T19:43:13.232000+0100",
:message=>"Trouble parsing json",
:source=>"json_message",
:raw=>["{\"activeSessions\":0}"],
:exception=>#<TypeError: can't convert Array into String>,
:level=>:warn}
Is there a way to extract the JSON string from the array prior to parsing? Any help is greatly appreciated. Thanks.
Actually, it is quite simple and plays along the lines of common programming languages. Though, I did not find the answer in the docs.
Just add an index to the field in the filter:
filter {
json {
source => "json_message[0]"
}
}
I have successfully created an index using elasticsearch, and can serialized those exact json payloads back to my java application.
for (SearchHit searchHit : searchResponse.getHits()) {
try {
result.getItems().add(objectMapper.readValue(searchHit.getSourceRef().streamInput(), Program.class));
} catch (IOException e) {
throw new IllegalArgumentException("Cannot marshall json", e);
}
}
The payload size that I export to elasticsearch is very large, but the response, I want to be very small. I also want to allow the client to dynamically include or exclude some fields. So I did this, where fields is an array of fields I want to include. This works well in that only the fields I ask for are returned, however the searchHit.getSourceRef is now null. Is there any way to get it to just copy the fields that I included via Jackson? Or must I always return the entire source object? Or do I have to write some sort of mapping code to translate (I would really like to avoid this) ?
SearchResponse searchResponse = transportClient.prepareSearch("programs")
.addFields(fields.toArray(new String[fields.size()]))
.setTypes("program")
.setQuery(query).setFrom(start).setSize(pageSize)
.execute().actionGet();
however the searchHit.getSourceRef is now null.
It is null because searchHit.getSource() is also null. As far as I know you have to add "_source" to your fields list when you do the search. Something like this:
ArrayList<String> fields = new ArrayList<String>();
fields. add("field1");
fields.add("field2");
fields.add("_source"); // add this field
SearchResponse response = transportClient.prepareSearch("programs")
.addFields(fields.toArray(new String[fields.size()]))
.execute().actionGet();