I want to parse the JSON objects from my log file. For using JSON Parser my complete files has to be in JSON format which is not the case with me. Is there any way I can parse the file line by line and get the JSON objects.
Below is my log file format:
2015-10-19 11:24:35:701 INFO BrokerTcpClient:28 - Set destination
2015-10-19 11:24:35:929 DEBUG BrokerTcpClient:32 - received data: {type=data, payload={
"core" : [ {
"id" : {
"datatype" : "http://www.w3.org/2001/hk#long",
"type" : "gh",
"value" : "gh"
},
"entity" : {
"type" : "uri",
"value" : "http://fg.fg.com/ext/g/fg"
},
"Sno" : {
"type" : "literal",
"value" : "fg"
}]
2015-10-19 11:24:35:701 INFO BrokerTcpClient:28 - Set destination
2015-10-19 11:24:35:929 DEBUG BrokerTcpClient:32
"core" : [ {
"id" : {
"datatype" : "http://www.w3.org/2001/hk#long",
"type" : "gh",
"value" : "gh"
},
"entity" : {
"type" : "uri",
"value" : "http://fg.fg.com/ext/g/fg"
},
"Sno" : {
"type" : "literal",
"value" : "fg"
}]
Can any one please help how should I get my JSON objects. When I am trying to parse a single line of JSON objects throwing an exception.
Here is a solution that works for the sample log that is posted.
import java.io.*;
import com.fasterxml.jackson.databind.*;
public class JSONTest {
public static void main(String[] args) {
String logFilename = "C://Temp/sample.log";
String line, json = "";
try (BufferedReader br = new BufferedReader(new FileReader(logFilename))) {
while ((line = br.readLine()) != null) {
if (isLogLine(line)) {
if (!json.isEmpty()) {
parseJson(json);
json = "";
}
} else {
json += line;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
public static boolean isLogLine(String line) {
return line.matches("^\\d{4}\\-\\d{2}\\-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}:\\d{3}.+$");
}
public static void parseJson(String json) throws Exception {
if (!json.startsWith("{") && !json.endsWith("}")) json = "{" + json + "}";
ObjectMapper om = new ObjectMapper();
System.out.println(om.readValue(fixJson(json), Object.class));
}
public static String fixJson(String json) {
return "{" + json.replace("}]", "}}]") + "}";
}
}
Notes:
the regex that identifies log line is checking for timestamp at the
begining of the line. this will not work in case the log message
spans multiple lines (for example if the message contains new line)
there is a method that attempts to "fix" the incomplete json from the log file, it may need additional logic, if there are more cases of incomplete json along the way
I used Jackson Json parser and let ObjectMapper figure out what data structure to parse into
Have you looked into logstash (https://www.elastic.co/products/logstash)? It can solve this problem and perhaps other problems you will come across when trying to parse logs.
Related
I have a JSON file from which I need to remove node objects which contain a data value of null. Can this be done? I'm using Jackson.
In the sample JSON below, I need to remove the object where its "v" tag has a null value.
Example:
{
"tags" : [ {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:27:36",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90WN1OUT",
"data" : [ {
"ts" : "2019-03-20T15:27:36",
"v" : null,
"q" : "0"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:20",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90WN1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:20",
"v" : null,
"q" : "0"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:37",
"v" : "96.2581177",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90WN1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:37",
"v" : null,
"q" : "0"
} ]
} ]
}
I need it to look like this:
{
"tags" : [ {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:27:36",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:20",
"v" : "96.2427826",
"q" : "3"
} ]
}, {
"tagId" : "G1.A_90LT1OUT",
"data" : [ {
"ts" : "2019-03-20T15:29:37",
"v" : "96.2581177",
"q" : "3"
} ]
} ]
}
Can this be done? Please show me how. Fairly new to JSON manipulation, I've seen another post that kind of shows how to remove an element from a node, but I think my case is a little different. I've tried chasing down documentation to no avail, maybe looking in the wrong places.
Thank you in advance.
JSONPath
For JSON manipulation and filtering you can also use JsonPath library. It has a great web tool where you can try different filters and options. We can filter all nodes wit not null values using below path:
$.tags[?(#.data[0].v != null)]
Example application which does the same:
import com.jayway.jsonpath.JsonPath;
import net.minidev.json.JSONArray;
import net.minidev.json.JSONObject;
import java.io.File;
public class JsonPathApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
JSONArray filtered = JsonPath.parse(jsonFile).read("$.tags[?(#.data[0].v != null)]");
// Create root object
JSONObject root = new JSONObject();
root.appendField("tags", filtered);
// Get JSON
String json = root.toString();
// Write JSON on console or file
System.out.println(json);
}
}
Above code prints:
{"tags":[{"tagId":"G1.A_90LT1OUT","data":[{"ts":"2019-03-20T15:27:36","v":"96.2427826","q":"3"}]},{"tagId":"G1.A_90LT1OUT","data":[{"ts":"2019-03-20T15:29:20","v":"96.2427826","q":"3"}]},{"tagId":"G1.A_90LT1OUT","data":[{"ts":"2019-03-20T15:29:37","v":"96.2581177","q":"3"}]}]}
Jackson
The same with Jackson we can achieve in that way:
Read JSON as tree
Go to tags array
Iterate over array
For each item find v key in 0-index element
In case it is null - remove it
Example implementation:
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import java.io.File;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class JsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(jsonFile);
ArrayNode tags = (ArrayNode) root.get("tags");
Iterator<JsonNode> elements = tags.elements();
while (elements.hasNext()) {
JsonNode item = elements.next();
ArrayNode data = (ArrayNode) item.get("data");
JsonNode v = data.get(0).get("v");
if (v.isNull()) {
elements.remove();
}
}
System.out.println(root);
}
}
I am having some trouble trying to figure out how to parse a line in a json file so that it only returns part of the line as a string. I will illustrate below:
public String GetDistance(String origin, String destination) throws MalformedURLException, IOException {
//URL url = new URL("https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins" + origin + ",UK+destination=" + destination + ",UK&key=mykey");
URL url = new URL("https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=Cornwall,UK&destinations=London,UK&key=mykey");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
String line, outputString = "";
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null) {
if (line.contains("distance")) {
outputString = reader.readLine().trim();
return outputString;
}
}
return outputString;
}
What this function does is create a json file in my browser using Google Maps API:
{
"destination_addresses" : [ "London, UK" ],
"origin_addresses" : [ "Cornwall, UK" ],
"rows" : [
{
"elements" : [
{
"distance" : {
"text" : "284 mi",
"value" : 456443
},
"duration" : {
"text" : "4 hours 52 mins",
"value" : 17530
},
"status" : "OK"
}
]
}
],
"status" : "OK"
}
Currently the "outputString" returns the line: "text" : "284 mi". However, the desired output is to just return the miles, "284".
I know this is most likely a re post, however I have been searching around for a solution to this and have been unsuccessful in implementing something that works.
Any help on this would be greatly appreciated, Cheers.
You can have 2 solutions:
Parse the JSON and treat it as an object and then just return the String you're looking for.
Split the line as follows:
outputString = reader.readLine().trim();
That line above returns "text" : "284 mi"
Then you need to split the line by ::
outputString.split(":")
That should create an array with 2 Strings: "text" and "284 mi"
Then take the second String and split it by a space and take the first String:
outputString.split("\\s")
Now you have: "284
Then just return it from subindex 1 till the end, see docs:
outputString.substring(1)
And then just put it all together:
return outputString.split(":")[1].split("\\s")[0].substring(1)
I haven't tried above code but it should work.
BTW follow Java Naming Conventions
firstWordLowerCaseVariable
firstWordLowerCaseMethod()
FirstWordUpperCaseClass
ALL_WORDS_UPPER_CASE_CONSTANT
And use them consistently
my JSON is :
{
"errorMessages" : [ {
"key" : "QUIESCE_FAILURE",
"code" : "12345",
"description" : "User already exists aaa",
"reason" : "Username {user} is already added aaa",
"resolution" : "Please provide a different username aaa",
"more_information" : "more info aaa",
"type" : "error aaa"
}, {
"key" : "DUPLICATE_USERS",
"code" : "3114587",
"description" : "Failed to quiesce database(s)",
"reason" : "Database {database} on host {host} is not accessible",
"resolution" : "Please check that the database is accessible",
"more_information" : "kkkk",
"type" : "error"
} ]
}
i want to delete json with code 3114587 using jackson.
may i get any suggestion?
Thanks in advance.
I suppose you have your JSON contents as a File object.
If you have it as an URL, InputStream, Reader or String,
then just modify the code below by taking another readValue method.
With the classes from packages com.fasterxml.jackson.databind
and com.fasterxml.jackson.databind.node
you can achieve your goal in a straight-forward way.
Read the JSON input, find and remove the array element, and write the JSON output:
File file = ...;
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.enable(SerializationFeature.INDENT_OUTPUT); // for pretty print
JsonNode jsonNode = objectMapper.readValue(file, JsonNode.class);
ArrayNode errorMessages = (ArrayNode) jsonNode.get("errorMessages");
for (int i = 0; i < errorMessages.size(); i++) {
ObjectNode errorMessage = (ObjectNode) errorMessages.get(i);
String code = errorMessage.get("code").asText();
if (code.equals("3114587")) {
errorMessages.remove(i); // remove the element
}
}
objectMapper.writeValue(System.out, jsonNode);
delete json[key];
Store your Json object in a variable name json. Find the key that you want to delete and it should work.
I am not very experienced in this so need help.I need following things to be done :
1. Want to traverse a folder structure in a content management repository.
2. Want to create JSON Object of the folder in child-node form.
3. Json data will be used to create tree using jQuery
Json format:
var data = [
{
text: "Parent 1",
nodes: [
{
text: "Child 1",
nodes: [
{
text: "Grandchild 1"
},
{
text: "Grandchild 2"
}
]
},
{
text: "Child 2"
}
]
},
{
text: "Parent 2"
},
{
text: "Parent 3"
},
{
text: "Parent 4"
},
{
text: "Parent 5"
}
];
My Java method is like this:
public static void displayIt(File node){
System.out.println(node.getAbsoluteFile());
if(node.isDirectory()){
String[] subNote = node.list();
for(String filename : subNote){
displayIt(new File(node, filename));
}
}
}
I am Struggling to frame strategy to create JSON object and Array to depict this.
Please can you help me.
If you want to read all files or folder from content management repository using recursion than use below function :
public static MyFolder readFiles(File file,List<MyFolder> myfolder,MyFolder childObj)
{
try
{
List<MyFolder> childArray = new ArrayList<MyFolder>();
if(file.isDirectory())
{
File[] file_array = file.listFiles();
if(file_array.length == 0 ){
childObj.text = file.getAbsolutePath();
myfolder.add(childObj);
childObj = new MyFolder();
}else{
childObj.text = file.getAbsolutePath();
childArray = childObj.nodes;
if(childArray == null)
childArray = new ArrayList<MyFolder>();
}
for(File tempFile : file_array)
{
if(tempFile.isDirectory())
{
childObj = readFiles(tempFile,myfolder,childObj);
if(childObj.text != null)
myfolder.add(childObj);
childObj = new MyFolder();
}
else
{
MyFolder obj = new MyFolder();
obj.text = tempFile.getAbsolutePath();
childArray.add(obj);
}
}
childObj.nodes = childArray;
}
else
{
childObj.text = file.getAbsolutePath();
myfolder.add(childObj);
childObj = new MyFolder();
}
}
catch(Exception ex)
{
ex.printStackTrace();
}
return childObj;
}
MyFolder Class :
class MyFolder
{
String text;
List<MyFolder> nodes;
}
You need some JSON utility or API to convert your class into json string. I have used GSON google API to convert my List<MyFolder> to JSON string.
Below is my test Class to test above :
List<MyFolder> myFolder = new ArrayList<MyFolder>();
File file = new File("D:/test");
MyFolder childArray = new MyFolder();
readFiles(file,myFolder,childArray);
Gson json = new Gson();
System.out.println(json.toJson(myFolder));
Output is :
[
{
"text": "D:\\test\\test1\\test12",
"nodes": [
{
"text": "D:\\test\\test1\\test12\\test12.txt"
},
{
"text": "D:\\test\\test1\\test12\\test12_2.txt"
}
]
},
{
"text": "D:\\test\\test2"
},
{
"text": "D:\\test\\test3"
}
]
Last thing remaining is just passed this to client side and process JSON to generate tree stucture.
May this will help you.
There's a way in JAVA to produce a JSON object from different collection types - Flexjson. Here you can find most essential information about Flexjson. In short, the command looks like this:
String jsonString = new flexjson.JSONSerializer().deepSerialize(myCollection);
Jquery perfectly understands the resulting JSON object and you can easily access the contents in JQuery like this:
jsonString["column1"]
So, you can just save data in an Arra/Map/whatewer is suitable for your situation and then translate in to JSON.
Hope it helps, I use this method every time I need a JSON string and works perfectly for me.
I'm using the package org.json to parse a JSONArray (I have the json strings saved in a database). However, I don't succeed in parsing it when the same key could have associated a String or a JSONObject, depending on the context.
For example, see the following JSON code...
[ { "cssClass" : "input_text",
"required" : "undefined",
"values" : "First Name"
},
{ "cssClass" : "checkbox",
"required" : "undefined",
"title" : "What's on your pizza?",
"values" : { "2" : { "baseline" : "undefined",
"value" : "Extra Cheese"
},
"3" : { "baseline" : "undefined",
"value" : "Pepperoni"
}
}
}
]
In the code above, the key "values" has 2 possibilities...
A String with value "First Name"
A JSONObject with value {"2":{"value":"Extra Cheese","baseline":"undefined"},"3":{"value":"Pepperoni","baseline":"undefined"}}.
How am I able to process this correctly when the value could be 2 different data types?
You'll probably still need to detect whether it is a JSONObject or a String, so that you can process it further, but perhaps something here might help...
You could try something like this...
String cssClass = myJson.getString("cssClass");
if (cssClass.equals("input_text")){
// Read it as a String
String values = myJson.getString("values");
}
else if (cssClass.equals("checkbox")){
// Read it as a JSONObject
JSONObject values = myJson.JSONObject("values");
// further processing here
}
Or maybe something like this...
String cssClass = myJson.getString("cssClass");
String values = myJson.getString("values");
if (cssClass.equals("input_text")){
// do nothing - it's already a String
}
else if (cssClass.equals("checkbox")){
// Parse the String into a JSONObject
JSONObject valuesObject = new JSONObject(values);
// further processing here
}
Think it this way in js or java duplicate variable creation under same scope is invalid,so to avoid ambiguity put them in separate json object with different variable names before putting it to the json array.