Forgive me if this is trivial or not possible but I'm having a Monday morning moment here.
I'd like to create a method that implements some methods from the Gson library to loaded some settings Objects. Basically, I have a bunch of different settings objects but I don't want to habe to override the load method for each class to I'd like to have something like:
public class ConfigLoader {
public static void main(final String[] args) {
final ConfigurationSettings loadedConfigSettigs =
load("testSettings.json", ConfigurationSettings.class);
final AlternativeConfigurationSettings alternativeConfigSettigs =
load("testSettings2.json", AlternativeConfigurationSettings .class);
}
public T load(final InputStream inputStream, final Class<T> clazz) {
try {
if (inputStream != null) {
final Gson gson = new Gson();
final BufferedReader reader =
new BufferedReader(new InputStreamReader(inputStream));
return gson.fromJson(reader, clazz);
}
} catch (final Exception e) {
}
return null;
}
}
where I can pass in the InputStream and the class of the object I want to return. Is there a simple way to do this (I don't want to have to create a method for each Class I want to be able to load, nor do I want to have to create a specific loader for each class)?
The following code works (requires Java 1.5 or above):
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import com.google.gson.Gson;
public class ConfigLoader {
public static void main(final String[] args) {
final ConfigurationSettings loadedConfigSettigs = load(new FileInputStream(new File("testSettings.json")),
ConfigurationSettings.class);
final AlternativeConfigurationSettings alternativeConfigSettigs = load(new FileInputStream(new File("testSettings2.json")),
AlternativeConfigurationSettings.class);
}
public static <T> T load(final InputStream inputStream, final Class<T> clazz) {
try {
if (inputStream != null) {
final Gson gson = new Gson();
final BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
return gson.fromJson(reader, clazz);
}
} catch (final Exception e) {
}
return null;
}
}
Related
I have a problem with huge JSON file (1GB) which contains an array of JSON objects but all of them are in one line. So it looks like...
[{JsonObject1},{JsonObject1},{JsonObject1},...,{JsonObject999999}]
I'm not able to save the content into the memory so I wanted to do this using streaming. I know how to stream line by line but if I have everything in one line only how can I stream JSON object one after another from this one line array?
I tried to browse the internet but I couldn't find anything :(
Using google gson, you can whip up something that operates lazily:
Maintained as a Gist here:
<script src="https://gist.github.com/smac89/bdcb9b08fcdf9d055150824d57ab3513.js"></script>
import com.google.gson.Gson;
import com.google.gson.stream.JsonReader;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.reflect.Type;
import java.net.URL;
import java.util.Iterator;
import java.util.Spliterators;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
public class ReadJsonFile {
private static final class JsonIterator<T> implements Iterator<T> {
private final Gson gson;
private final Type objectType;
private final JsonReader reader;
private JsonIterator(JsonReader reader, Gson gson, Type objectType) {
this.gson = gson;
this.objectType = objectType;
this.reader = reader;
}
#Override
public boolean hasNext() {
try {
return reader.hasNext();
} catch (IOException ioe) {
return false;
}
}
#Override
public T next() {
return gson.fromJson(reader, objectType);
}
}
public static <J> Stream<J> readJsonFromFile(Gson gson, URL jsonFile, Type type) throws IOException {
JsonReader reader = new JsonReader(
new BufferedReader(new InputStreamReader(jsonFile.openStream())));
reader.beginArray();
if (!reader.hasNext()) {
return Stream.empty();
}
return StreamSupport.stream(Spliterators.spliteratorUnknownSize(
new JsonIterator<J>(reader, gson, type), 0), false).onClose(() -> {
try {
reader.close();
} catch (IOException ioe) {
}
});
}
}
data.json
<script src="https://gist.github.com/smac89/15fc3bffa4f965d18587eec2db4972dd.js"></script>
Been googling a while but haven't found anything useful since keywords are too common.
Currently in my Java code method toString() (called on some JSONObject object) produces output like this:
{"key_name":<numerical_value>}
while what I need (due to the fact that parser on the other side of the project is imperfect) is:
{"key_name": <numerical_value>}
The only difference is that extra space after colon. Is there any JSONObject builtin way to do it or do I need some handwritten simple string operation for it?
I can only imagine building a custom JSONStringer. In fact, it could be a JSONWriter built using a StringWriter. Then you override all the value methods to add a space before calling parent class method.
class JSONSpaceStringer extends JSONWriter {
private StringWriter sw;
JSONSpaceStringer(StringWriter sw) {
parent(sw); // initialize parent
this.sw = sw;
}
#Override
public JSONWriter value(long l) throws JSONException {
sw.write(" "); // add an initial space
parent.value(l); // and let parent class do the job...
}
// same for all other value methods
//...
}
use the below code. it will help you to solve the problem.
package com.javamad.utils;
import java.io.IOException;
import org.apache.log4j.Logger;
import org.codehaus.jackson.JsonGenerationException;
import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
public class JsonUtils {
private static Logger logger = Logger.getLogger(JsonUtils.class.getName());
public static <T> T jsonToJavaObject(String jsonRequest, Class<T> valueType)
throws JsonParseException, JsonMappingException, IOException {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(org.codehaus.jackson.map.DeserializationConfig.Feature.UNWRAP_ROOT_VALUE,false);
T finalJavaRequest = objectMapper.readValue(jsonRequest, valueType);
return finalJavaRequest;
}
public static String javaToJson(Object o) {
String jsonString = null;
try {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(org.codehaus.jackson.map.DeserializationConfig.Feature.UNWRAP_ROOT_VALUE,true);
jsonString = objectMapper.writeValueAsString(o);
} catch (JsonGenerationException e) {
logger.error(e);
} catch (JsonMappingException e) {
logger.error(e);
} catch (IOException e) {
logger.error(e);
}
return jsonString;
}
}
I want to write a json file in java, but it doesn't work, I get this warning:
I want to know how to do this, because I am going to convert a cfg file that is tabbed to json.
Type safety: The method add(Object) belongs to the raw type ArrayList. References to generic type ArrayList<E> should be parameterized
and I have this code:
package json;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
public class JsonWriter {
public static void main(String[] args) {
JSONObject countryObj = new JSONObject();
countryObj.put("Name", "India");
countryObj.put("Population", new Integer(1000000));
JSONArray listOfStates = new JSONArray();
listOfStates.add("Madhya Pradesh");
listOfStates.add("Maharastra");
listOfStates.add("Rajasthan");
countryObj.put("States", listOfStates);
try {
// Writing to a file
File file=new File("JsonFile.json");
file.createNewFile();
FileWriter fileWriter = new FileWriter(file);
System.out.println("Writing JSON object to file");
System.out.println("-----------------------");
System.out.print(countryObj);
fileWriter.write(countryObj.toJSONString());
fileWriter.flush();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I would suggest that you just make a simple ArrayList with your objects, and then serialize them into JSON with a serializer (Using the Jacksoin library in the example below). It would look something like this:
First, define your model in a class (Made without incapsulations for readability):
public class Country{
public String name;
public Integer population;
public List<String> states;
}
Then you can go ahead and create it, and populate the list:
import java.io.File;
import java.io.IOException;
import org.codehaus.jackson.JsonGenerationException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
public class JsonWriter {
public static void main(String[] args) {
Country countryObj = new Country();
countryObj.name = "India";
countryObj.population = 1000000;
List<String> listOfStates = new ArrayList<String>();
listOfStates.add("Madhya Pradesh");
listOfStates.add("Maharastra");
listOfStates.add("Rajasthan");
countryObj.states = listOfStates ;
ObjectMapper mapper = new ObjectMapper();
try {
// Writing to a file
mapper.writeValue(new File("c:\\country.json"), countryObj );
} catch (IOException e) {
e.printStackTrace();
}
}
}
I am using java spark API to write some test application . I am using a class which doesn't extends serializable interface . So to make the application work I am using kryo serializer to serialize the class . But the problem which I observed while debugging was that during the de-serialization the returned class object becomes null and in turn throws a null pointer exception . It seems to be closure problem where things are going wrong but not sure.Since I am new to this kind of serialization I don't know where to start digging.
Here is the code I am testing :
package org.apache.spark.examples;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.net.InetAddress;
import java.net.UnknownHostException;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
/**
* Spark application to test the Serialization issue in spark
*/
public class Test {
static PrintWriter outputFileWriter;
static FileWriter file;
static JavaSparkContext ssc;
public static void main(String[] args) {
String inputFile = "/home/incubator-spark/examples/src/main/scala/org/apache/spark/examples/InputFile.txt";
String master = "local";
String jobName = "TestSerialization";
String sparkHome = "/home/test/Spark_Installation/spark-0.7.0";
String sparkJar = "/home/test/TestSerializationIssesInSpark/TestSparkSerIssueApp/target/TestSparkSerIssueApp-0.0.1-SNAPSHOT.jar";
SparkConf conf = new SparkConf();
conf.set("spark.closure.serializer","org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", "org.apache.spark.examples.MyRegistrator");
// create the Spark context
if(master.equals("local")){
ssc = new JavaSparkContext("local", jobName,conf);
//ssc = new JavaSparkContext("local", jobName);
} else {
ssc = new JavaSparkContext(master, jobName, sparkHome, sparkJar);
}
JavaRDD<String> testData = ssc.textFile(inputFile).cache();
final NotSerializableJavaClass notSerializableTestObject= new NotSerializableJavaClass("Hi ");
#SuppressWarnings({ "serial", "unchecked"})
JavaRDD<String> classificationResults = testData.map(
new Function<String, String>() {
#Override
public String call(String inputRecord) throws Exception {
if(!inputRecord.isEmpty()) {
//String[] pointDimensions = inputRecord.split(",");
String result = "";
try {
FileWriter file = new FileWriter("/home/test/TestSerializationIssesInSpark/results/test_result_" + (int) (Math.random() * 100));
PrintWriter outputFile = new PrintWriter(file);
InetAddress ip;
ip = InetAddress.getLocalHost();
outputFile.println("IP of the server: " + ip);
result = notSerializableTestObject.testMethod(inputRecord);
outputFile.println("Result: " + result);
outputFile.flush();
outputFile.close();
file.close();
} catch (UnknownHostException e) {
e.printStackTrace();
}
catch (IOException e1) {
e1.printStackTrace();
}
return result;
} else {
System.out.println("End of elements in the stream.");
String result = "End of elements in the input data";
return result;
}
}
}).cache();
long processedRecords = classificationResults.count();
ssc.stop();
System.out.println("sssssssssss"+processedRecords);
}
}
Here is the KryoRegistrator class
package org.apache.spark.examples;
import org.apache.spark.serializer.KryoRegistrator;
import com.esotericsoftware.kryo.Kryo;
public class MyRegistrator implements KryoRegistrator {
public void registerClasses(Kryo kryo) {
kryo.register(NotSerializableJavaClass.class);
}
}
Here is the class I am serializing :
package org.apache.spark.examples;
public class NotSerializableJavaClass {
public String testVariable;
public NotSerializableJavaClass(String testVariable) {
super();
this.testVariable = testVariable;
}
public String testMethod(String vartoAppend){
return this.testVariable + vartoAppend;
}
}
This is because spark.closure.serializer only supports the Java serializer. See http://spark.apache.org/docs/latest/configuration.html about spark.closure.serializer
my research on google-search and stackoverflow regarding this problem didn't resolve it.
i'd like to show you a snipped of my Datastructure:
there's a class, called "SequenceHolder" => this one carries an:
ArrayList<SequenceData> data;
within the Sequenceholder, there is a function to call the serialization:
public void writeSequenceList() throws FileNotFoundException, IOException {
FileOutputStream fout = new FileOutputStream(path);
ObjectOutputStream oout = new ObjectOutputStream(fout);
oout.writeObject(data);
oout.close();
fout.close();
}
The class SequenceObject has following fields: (this one is on the top, where i start the serialization)
private ArrayList<SequenceModel> recordedSequenceData;
private String sequenceUrl;
while the SequenceModel is defined like this:
private Object sequenceRawData;
private boolean isProcessedByRequest;
The sequenceRawdata objects are basically two other classes (containing Strings only)!
every class of this "trail" implements the interface "Serializable".
this is the deserialization:
public ArrayList<SequenceData> loadSequenceList() throws FileNotFoundException, IOException, ClassNotFoundException {
FileInputStream fileIn = new FileInputStream(path);
ObjectInputStream in = new ObjectInputStream(fileIn);
this.data = (ArrayList<SequenceData>) in.readObject();
in.close();
fileIn.close();
return data; // load from de-serialization
}
after a deserialization of the SequenceObject, i'll only retrieve the "sequenceUrl", but no recordedSequenceData.
Is there a trick to do this?!
It came just up to my mind, to extend some classes with the ObjectOutputStream and call the writingprocess with "this" explicitly in every class - but yeah, i am not sure if thats a good idead.
What do you mean by "The sequenceRawdata objects are basically two other classes (containing Strings only)!" because when I try to run the same program :
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.util.ArrayList;
class SequenceModel implements Serializable
{
public SequenceModel(Object a, boolean b)
{
sequenceRawData = a;
isProcessedByRequest = b;
}
public String toString()
{
return (String)sequenceRawData + isProcessedByRequest + " SeqModel ";
}
private Object sequenceRawData;
private boolean isProcessedByRequest;
}
class SequenceData implements Serializable
{
public SequenceData(ArrayList<SequenceModel> a, String b)
{
recordedSequenceData = a;
sequenceUrl = b;
}
public String toString()
{
return recordedSequenceData + sequenceUrl + " SeqData ";
}
private ArrayList<SequenceModel> recordedSequenceData;
private String sequenceUrl;
}
class SequenceHolder implements Serializable
{
ArrayList<SequenceData> data;
public String toString()
{
return data + " SeqHol ";
}
public SequenceHolder(ArrayList<SequenceData> a)
{
data = a;
}
public void writeSequenceList() throws FileNotFoundException, IOException {
FileOutputStream fout = new FileOutputStream(Test.file);
ObjectOutputStream oout = new ObjectOutputStream(fout);
oout.writeObject(data);
oout.close();
fout.close();
}
public ArrayList<SequenceData> loadSequenceList() throws FileNotFoundException, IOException, ClassNotFoundException {
FileInputStream fileIn = new FileInputStream(Test.file);
ObjectInputStream in = new ObjectInputStream(fileIn);
this.data = (ArrayList<SequenceData>) in.readObject();
in.close();
fileIn.close();
return data; // load from de-serialization
}
}
public class Test
{
public static File file = new File("abc.txt");
public static void main(String[] args)
{
SequenceModel obj = new SequenceModel("abc", false);
ArrayList list = new ArrayList(); list.add(obj);
SequenceData obh = new SequenceData(list, "str");
ArrayList l2 = new ArrayList();
l2.add(obh);
SequenceHolder obi = new SequenceHolder(l2);
try {
obi.writeSequenceList();
System.out.println(obi.loadSequenceList());
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
it is able to serialize and deserialize both and there is no problem.
Output it is printing is : [[abcfalse SeqModel ]str SeqData ] which includes everything as desired.
Please let me know if I am missing anything.