I have a microservice/application that accepts a JSON and convert to java POJO that is processed further in that application. Say the structure is:
{
"header": {
"msgTs": "2020-02-20T11:00:00"
}
"hazmat": {
"name": "a"
}
// some other properties
}
public class Hazmat {
private String hazmatName;
}
public class POJO {
private Header header;
private Hazmat hazmet;
}
Here the "hazmat" could be present or absent. For all test data, the value of header and msgTs doesn't matter. So we can hard code the value;
Now to test it I would like to create a test data builder. The builder will be like this:
public class PojoBulider {
private String hazmatName;
public PojoBuilder withHamatName(String hazmatName) { this.hazmatname = hazmatName; return this; }
public String build() {
// generate test data from the captured inputs eg "hazmatName".
}
}
for generating the test data the user input / feature file column value will work with a few selected properties (eg hazmatName). The rest can be hard coded.
Now the problem is how can we hard code the fixed value such as msgTs? I can have a template json:
{
"header": {
"msgTs": "2020-02-20T11:00:00"
}
"hazmat": {
"name": "%%hazmatName%%"
}
}
Then have:
public void build() {
String templateJson; // load from file;
templateJson = templateJson.replaceAll("%%hazmatName%%", hazmatName);
return templateJson;
}
The thing is that the JSON can have properties that are present and absent. So I cannot have multiple templateJson with different possible value ie one templateJson:
{
"header": {
"msgTs": "2020-02-20T11:00:00"
}
"hazmat": {
"name": "%%hazmatName%%"
}
}
another template json:
{
"header": {
"msgTs": "2020-02-20T11:00:00"
}
"toy": {
"name": "%%toyName%%"
}
}
I can't have one template json as:
{
"header": {
"msgTs": "2020-02-20T11:00:00"
}
"toy": {
"name": "%%toyName%%"
}
"hazmat": {
"name": "%%hazmatName%%"
}
}
Because not test data will have both "toy" and "hazmat" section.
So I am stuck. I can probably have multiple template json like one for hazmat, hazmat.json:
"hazmat": {
"name": "%%hazmatName%%"
}
and another for toy:
"toy": {
"name": "%%toyName%%"
}
and combine them like:
public String build() {
if (hazmatName != null) {
// load hazmatJson
// append to templateJson
}
if (toyName != null) {
// load toyJson
// append to templateJson
}
return templateJson;
}
I would like to use the POJO instead:
public String build() [
Pojo pojo = new Pojo();
Hazmat hazmat = new Hazmat();
hazmat.setHazmatName(this.hazmatName);
pojo.setHazmat(hazmat);
return new ObjectMapper().writeValueAsString(pojo);
}
but the problem is how can I populate the header, etc value that don't change for all json. I can have another json:
{
"header": { "msgTs": "..." } }
and then load it using ObjectMapper load:
public String build() {
Header header = new ObjectMapper().load("header.json");
pojo.setHeader(header);
}
Is there a pure java based approach without using any json file altogether (ie header.json, toy.json, hazmat.json)? how do normally people generate test data in this case?
As you commented that you prefer to modify fields over Key-based mappings, I write this answer explaining what I would do under that constraint (not saying that this is the best solution, as I would use constant keys with maps).
Still assuming, that you ultimately want to generate a json-like String as input for your test candidate, I would create a class that prints itself to json format. Be aware, that I only tested for Object-arrays, not primitive arrays! List, Map and Set types are also not included, but you can add them yourself, if you like.
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public abstract class JSON {
public static final Object UNDEFINED = new Object();
public String toJsonThrowRTE() {
try {
return toJsonString();
} catch (Throwable t) {
throw new RuntimeException("I found thiz. May I keep it? :)", t);
}
}
public String toJsonString() throws NoSuchMethodException, SecurityException,
IllegalArgumentException, IllegalAccessException, InvocationTargetException {
StringBuilder sb = new StringBuilder();
boolean first = true;
for (Field field : getAllDeclaredFields()) {
String name = field.getName();
// This prevents endless recursion.
if (name.startsWith("this")) {
continue;
}
Class<?> fieldType = field.getType();
if (fieldType.isPrimitive()) {
// No need for the first field to have a ',' before it.
if (first) {
first = false;
} else {
sb.append(",\n");
}
// This is cumbersome, but I have no cleaner solution.
sb.append(name).append(": ");
switch (fieldType.getTypeName()) {
case "int":
sb.append(field.getInt(this));
break;
case "long":
sb.append(field.getLong(this));
break;
case "boolean":
sb.append(field.getBoolean(this));
break;
case "char":
sb.append("'" + field.getChar(this) + "'");
break;
case "double":
sb.append(field.getDouble(this));
break;
case "float":
sb.append(field.getFloat(this));
break;
case "byte":
sb.append(field.getByte(this));
break;
case "short":
sb.append(field.getShort(this));
break;
}
} else {
String value = valueToString(field.get(this));
if (value != null) {
// No need for the first field to have a ',' before it.
if (first) {
first = false;
} else {
sb.append(",\n");
}
sb.append(name).append(": ").append(value);
}
}
}
return "{\n" + indent(sb.toString()) + "\n}";
}
public String valueToString(Object value) throws NoSuchMethodException, SecurityException,
IllegalArgumentException, IllegalAccessException, InvocationTargetException {
if (value == null) {
return "null";
} else if (value == UNDEFINED) {
return null;
} else if (value instanceof JSON) {
JSON childJson = (JSON) value;
return childJson.toJsonString();
} else if (value instanceof String) {
return "\"" + value + "\"";
} else if (value instanceof Character) {
return "'" + value + "'";
} else if (value.getClass().isArray()) {
Object[] array = (Object[]) value;
StringBuilder sb = new StringBuilder();
boolean first = true;
for (Object o : array) {
if (first) {
first = false;
} else {
sb.append(", ");
}
sb.append(valueToString(o));
}
return "[ " + sb.toString() + " ]";
}
// Everything else is just toString.
return value.toString();
}
public List<Field> getAllDeclaredFields() {
List<Field> allFields = new ArrayList<>();
// We start with the current class, and stop at JsonStringBuilder.
// After that, the fields are Object fields, and we don't want those.
Class<?> currentClass = getClass();
while (JSON.class.isAssignableFrom(currentClass)) {
// We want to order the fields from the most general to the most specific,
// so we must add the new fields at the start.
List<Field> declaredFields = Arrays.asList(currentClass.getDeclaredFields());
List<Field> childFields = allFields;
allFields = new ArrayList<>(childFields.size() + declaredFields.size());
allFields.addAll(declaredFields);
allFields.addAll(childFields);
currentClass = currentClass.getSuperclass();
}
return allFields;
}
public String indent(String toIndent) {
return toIndent.replaceAll("(?:^|(\\n))", "$1\t");
}
}
All objects you derive from this class will print themselves to json on command. This is the most powerful I can come up with on a short notice, but I think it should do.
You can now create any POJO with constructor and such, so you can set up your test, by creating you default object and then adding the required test values on demand. Objects that are JSON.UNDEFINED are not printed.
Here is an example on how to use it:
#Test
public void test() {
JSON topLevel = new JSON() {
Object msgTs = "2020-02-20T11:00:00";
Object name = "a";
Object[] someArray = { 7, "String intermix", 'k', 5d };
Object iAmNull = null;
Object undefined = JSON.UNDEFINED;
Object someChild = new JSON() {
int someInt = 42;
char myChar = 'j';
};
};
System.out.println(topLevel.toJsonThrowRTE());
}
The above "test" will print this String to console:
{
msgTs: "2020-02-20T11:00:00",
name: "a",
someArray: [ 7, "String intermix", 'k', 5.0 ],
iAmNull: null,
someChild: {
someInt: 42,
myChar: 'j'
}
}
Now you just have to insert this string into your Json parser or whatever you want to test. This would be your base object, which you create the same way with the #Before or #BeforeAll, depending on your junit version. And from there you can modify the object per test. Remember to declare fields that may or may not be used as JSON.UNDEFINED, so that you can assign a useful value, but they are not printed, unless you set them!
JSON message = new JSON() {
Object header = new JSON() {
String msgTs = "2020-02-20T11:00:00";
};
Object hazmet = JSON.UNDEFINED;
Object toy = JSON.UNDEFINED;
// ...
};
Related
I am trying to convert the large JSON/JSON-LD file to XML. The JSON file will contain a list of events (not all events are the same and each may have different information/data). I would like to read the events one by one and store the information in a single Object (rather than having different objects for each event). I would like to store it in the form of Parent and Children. As soon as I read one event I would like to convert it to JSON then go to the next one.
I am looking into JACKSON and seems like it fits this. However, I am looking into this particular class TokenBuffer which is actually kind of doing the same. It's reading the JSON file line by line and then trying to see if the incoming element is Array or Object based on that it's making various decisions.
I wanted to confirm if I can use this Class directly to pass my JsonFile and get the Parent and child relationship which will be stored in JsonWriteContext later I can convert it to XML.
Basically, my sample file will have many events in the JSON which I would like to read one by one and then get the element info and its parent-children relationship in a Single Object such as maybe HashMap.
After trying some of the things following is my code which will contain the Parent and Child elements from Json:
#Getter
#Setter
#NoArgsConstructor
public class JsonContext {
private JsonContext parent;
private String key;
private String value;
private int childCount;
private Map<String, List<JsonContext>> children = new HashMap<String, List<JsonContext>>();
// Constructor for Children Object
JsonContext(String key, String value) {
this.key = key;
this.value = value;
}
// Constructor for Parent Object
JsonContext(JsonContext parent, String key, String value) {
this(key, value);
this.parent = parent;
}
// Add child based on incoming element
public JsonContext addChild(JsonContext context) {
List<JsonContext> childValues = children.get(context.getKey());
if (childValues == null) {
childValues = new ArrayList<JsonContext>();
}
childValues.add(context);
children.put(context.key, childValues);
childCount++;
return context;
}
// Get the parent and their subsequent children (test purpose only)
#Override
public String toString() {
String s = key + children.entrySet().stream().map(e -> e.getKey() + " -- " + String.join(", ", e.getValue().stream().map(v -> v.toString())
.collect(Collectors.toList()))).collect(Collectors.toList());
if (value.length() > 0) {
final String chars = value.toString().trim();
if (chars.length() > 0) {
s = s + " = " + chars;
}
}
return s;
}
}
The above is the context file that will store the information. Below is the JSON parser file.
import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.Arrays;
import com.converter.xml2jsonld.test.JSONParser;
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
public class JsonEventsParser {
private JsonContext context = null;
private final String[] eventTypes = new String[] { "event1", "event2", "event3", "event4",
"event5" };
public void jsonFileIterator() throws URISyntaxException, JsonParseException, IOException {
// Get the JSON Factory and parser Object
JsonFactory jsonFactory = new JsonFactory();
JsonParser jsonParser = jsonFactory.createParser(new File(JSONParser.class.getClassLoader().getResource("inputJsonFile.json").toURI()));
JsonToken current = jsonParser.nextToken();
// Check the first element is Object
if (current != JsonToken.START_OBJECT) {
throw new IllegalStateException("Expected content to be an array");
}
// Call the method to loop until the end of the events file
FileNavigator(jsonParser);
}
private void FileNavigator(JsonParser jsonParser) throws IOException {
JsonToken current = jsonParser.getCurrentToken();
// Loop until the end of the EPCIS events file
while (jsonParser.nextToken() != JsonToken.END_OBJECT) {
final JsonToken token = jsonParser.nextToken();
final String name = jsonParser.getCurrentName();
// Handling the fields with direct key value pairs
if ((token == JsonToken.FIELD_NAME || token == JsonToken.VALUE_STRING)) {
writeFieldName(jsonParser, token);
}
// Handling the Object
if (token == JsonToken.START_OBJECT) {
writeObjectFields(jsonParser, token);
}
// Handling the Array
if (token == JsonToken.START_ARRAY) {
writeArrayFields(jsonParser, token);
}
if (context != null) {
if (context.getParent() != null) {
context = context.getParent();
}
}
}
System.out.println(context.getChildren().toString());
}
// Method to obtain the STRING field and write into Context
private void writeFieldName(JsonParser jsonParser, JsonToken token) throws IOException {
final String key = jsonParser.getCurrentName();
final String value = jsonParser.getValueAsString();
// Check for the eventType
if (context == null && Arrays.asList(eventTypes).contains(value)) {
context = new JsonContext(key, value);
} else if (context != null) {
context = context.addChild(new JsonContext(context, key, value));
}
}
// Method to obtain the OBJECT and write its children into Context
private void writeObjectFields(JsonParser jsonParser, JsonToken token) throws IOException {
final String objectParent = jsonParser.getCurrentName() == null ? context.getParent().getKey() : jsonParser.getCurrentName();
// Add the name of the OBJECT
if (context == null) {
context = new JsonContext(jsonParser.getCurrentName(), "Object");
} else if (context != null) {
context = context.addChild(new JsonContext(context, objectParent, "Object"));
}
token = jsonParser.nextToken();
// Loop through all elements within OBJECT and add them to its parent
while (token != JsonToken.END_OBJECT) {
final String key = jsonParser.getCurrentName();
token = jsonParser.nextToken();
// Check for incoming tokens within array and process accordingly
switch (token) {
case START_ARRAY:
writeArrayFields(jsonParser, token);
break;
case START_OBJECT:
writeObjectFields(jsonParser, token);
break;
default:
final String value = jsonParser.getText();
context = context.addChild(new JsonContext(context, key, value));
break;
// throw new RuntimeException("Object : Elements does not match the type
// (Method: writeObjectFields)");
}
context = context.getParent();
token = jsonParser.nextToken();
}
}
// Method to Obtain the ARRAY and write its children into Context
private void writeArrayFields(JsonParser jsonParser, JsonToken token) throws IOException {
final String arrayField = jsonParser.getCurrentName();
// Add the name of the ARRAY
if (context == null) {
context = new JsonContext(arrayField, "Array");
} else if (context != null) {
context = context.addChild(new JsonContext(context, arrayField, "Array"));
}
token = jsonParser.nextToken();
// Loop through all ARRAY elements
while (token != JsonToken.END_ARRAY) {
switch (token) {
case START_OBJECT:
writeObjectFields(jsonParser, token);
break;
case VALUE_STRING:
context = context.addChild(new JsonContext(context, arrayField, jsonParser.getText()));
break;
case START_ARRAY:
writeArrayFields(jsonParser, token);
break;
default:
throw new RuntimeException("Array : Elements does not match the type (Method: writeArrayFields)");
}
context = context.getParent();
token = jsonParser.nextToken();
}
}
}
I am trying to write some Parquet records that contain LogicalTypes to JSON. I do this via AvroParquetReader, which gives me an Avro GenericRecord:
GenericData.get().addLogicalTypeConversion(new TimeConversions.TimeMillisConversion());
try (ParquetReader<GenericRecord> parquetReader =
AvroParquetReader.<GenericRecord>builder(new LocalInputFile(this.path))
.withDataModel(GenericData.get())
.build()) {
GenericRecord record = parquetReader.read();
record.toString();
}
record.toString() produces:
{"universe_member_id": 94639, "member_from_dt": 2001-08-31T00:00:00Z, "member_to_dt": 2200-01-01T00:00:00Z}
Notice that this is invalid JSON - the dates are correctly converted as per their LogicalType, but are not surrounded by quotes.
So instead I tried the JsonEncoder:
GenericData.get().addLogicalTypeConversion(new TimeConversions.TimeMillisConversion()); //etc
OutputStream stringOutputStream = new StringOutputStream();
try (ParquetReader<GenericRecord> parquetReader =
AvroParquetReader.<GenericRecord>builder(new LocalInputFile(this.path))
.withDataModel(GenericData.get())
.build()) {
GenericRecord record = parquetReader.read();
DatumWriter<GenericRecord> writer = new GenericDatumWriter<>(record.getSchema());
JsonEncoder encoder = EncoderFactory.get().jsonEncoder(record.getSchema(), stringOutputStream);
writer.write(record, encoder);
encoder.flush();
}
but this doesn't convert the date fields at all and bakes the datatype into every record:
{"universe_member_id":{"long":94639},"member_from_dt":{"long":999216000000000},"member_to_dt":{"long":7258118400000000}}
The output I'm looking for is:
{"universe_member_id": 94639, "member_from_dt": "2001-08-31T00:00:00Z", "member_to_dt": "2200-01-01T00:00:00Z"}
How can I correctly write a GenericRecord to JSON?
As you have indicated, the method toString() in class GenericRecord will give you a nearly valid JSON representation.
As you can see in the source code of the GenericData class, the GenericData.Record toString method just invoke the GenericData toString(Object) method in its implementation.
If you want a valid JSON representation of the record, you can take that code and, with minimal modifications, obtain the information that you need.
For instance, we can define an utility class like the following:
package stackoverflow.parquetavro;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.Collection;
import java.util.HashMap;
import java.util.IdentityHashMap;
import java.util.Map;
import java.util.function.Function;
import org.apache.avro.LogicalType;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericContainer;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericEnumSymbol;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.IndexedRecord;
public class GenericRecordJsonEncoder {
Map<LogicalType, Function<Object, Object>> logicalTypesConverters = new HashMap<>();
public void registerLogicalTypeConverter(LogicalType logicalType, Function<Object, Object> converter) {
this.logicalTypesConverters.put(logicalType, converter);
}
public Function<Object, Object> getLogicalTypeConverter(Schema.Field field) {
Schema fieldSchema = field.schema();
LogicalType logicalType = fieldSchema.getLogicalType();
return getLogicalTypeConverter(logicalType);
}
public Function<Object, Object> getLogicalTypeConverter(LogicalType logicalType) {
if (logicalType == null) {
return Function.identity();
}
return logicalTypesConverters.getOrDefault(logicalType, Function.identity());
}
public String serialize(GenericRecord value) {
StringBuilder buffer = new StringBuilder();
serialize(value, buffer, new IdentityHashMap<>(128) );
String result = buffer.toString();
return result;
}
private static final String TOSTRING_CIRCULAR_REFERENCE_ERROR_TEXT =
" \">>> CIRCULAR REFERENCE CANNOT BE PUT IN JSON STRING, ABORTING RECURSION <<<\" ";
/** Renders a Java datum as JSON. */
private void serialize(final Object datum, final StringBuilder buffer, final IdentityHashMap<Object, Object> seenObjects) {
if (isRecord(datum)) {
if (seenObjects.containsKey(datum)) {
buffer.append(TOSTRING_CIRCULAR_REFERENCE_ERROR_TEXT);
return;
}
seenObjects.put(datum, datum);
buffer.append("{");
int count = 0;
Schema schema = getRecordSchema(datum);
for (Schema.Field f : schema.getFields()) {
serialize(f.name(), buffer, seenObjects);
buffer.append(": ");
Function<Object, Object> logicalTypeConverter = getLogicalTypeConverter(f);
serialize(logicalTypeConverter.apply(getField(datum, f.name(), f.pos())), buffer, seenObjects);
if (++count < schema.getFields().size())
buffer.append(", ");
}
buffer.append("}");
seenObjects.remove(datum);
} else if (isArray(datum)) {
if (seenObjects.containsKey(datum)) {
buffer.append(TOSTRING_CIRCULAR_REFERENCE_ERROR_TEXT);
return;
}
seenObjects.put(datum, datum);
Collection<?> array = getArrayAsCollection(datum);
buffer.append("[");
long last = array.size()-1;
int i = 0;
for (Object element : array) {
serialize(element, buffer, seenObjects);
if (i++ < last)
buffer.append(", ");
}
buffer.append("]");
seenObjects.remove(datum);
} else if (isMap(datum)) {
if (seenObjects.containsKey(datum)) {
buffer.append(TOSTRING_CIRCULAR_REFERENCE_ERROR_TEXT);
return;
}
seenObjects.put(datum, datum);
buffer.append("{");
int count = 0;
#SuppressWarnings(value="unchecked")
Map<Object,Object> map = (Map<Object,Object>)datum;
for (Map.Entry<Object,Object> entry : map.entrySet()) {
serialize(entry.getKey(), buffer, seenObjects);
buffer.append(": ");
serialize(entry.getValue(), buffer, seenObjects);
if (++count < map.size())
buffer.append(", ");
}
buffer.append("}");
seenObjects.remove(datum);
} else if (isString(datum)|| isEnum(datum)) {
buffer.append("\"");
writeEscapedString(datum.toString(), buffer);
buffer.append("\"");
} else if (isBytes(datum)) {
buffer.append("{\"bytes\": \"");
ByteBuffer bytes = ((ByteBuffer) datum).duplicate();
writeEscapedString(StandardCharsets.ISO_8859_1.decode(bytes), buffer);
buffer.append("\"}");
} else if (((datum instanceof Float) && // quote Nan & Infinity
(((Float)datum).isInfinite() || ((Float)datum).isNaN()))
|| ((datum instanceof Double) &&
(((Double)datum).isInfinite() || ((Double)datum).isNaN()))) {
buffer.append("\"");
buffer.append(datum);
buffer.append("\"");
} else if (datum instanceof GenericData) {
if (seenObjects.containsKey(datum)) {
buffer.append(TOSTRING_CIRCULAR_REFERENCE_ERROR_TEXT);
return;
}
seenObjects.put(datum, datum);
serialize(datum, buffer, seenObjects);
seenObjects.remove(datum);
} else {
// This fallback is the reason why GenericRecord toString does not
// generate a valid JSON representation
buffer.append(datum);
}
}
// All these methods are also copied from the GenericData class source
private boolean isRecord(Object datum) {
return datum instanceof IndexedRecord;
}
private Schema getRecordSchema(Object record) {
return ((GenericContainer)record).getSchema();
}
private Object getField(Object record, String name, int position) {
return ((IndexedRecord)record).get(position);
}
private boolean isArray(Object datum) {
return datum instanceof Collection;
}
private Collection getArrayAsCollection(Object datum) {
return (Collection)datum;
}
private boolean isEnum(Object datum) {
return datum instanceof GenericEnumSymbol;
}
private boolean isMap(Object datum) {
return datum instanceof Map;
}
private boolean isString(Object datum) {
return datum instanceof CharSequence;
}
private boolean isBytes(Object datum) {
return datum instanceof ByteBuffer;
}
private void writeEscapedString(CharSequence string, StringBuilder builder) {
for(int i = 0; i < string.length(); i++){
char ch = string.charAt(i);
switch(ch){
case '"':
builder.append("\\\"");
break;
case '\\':
builder.append("\\\\");
break;
case '\b':
builder.append("\\b");
break;
case '\f':
builder.append("\\f");
break;
case '\n':
builder.append("\\n");
break;
case '\r':
builder.append("\\r");
break;
case '\t':
builder.append("\\t");
break;
default:
// Reference: http://www.unicode.org/versions/Unicode5.1.0/
if((ch>='\u0000' && ch<='\u001F') || (ch>='\u007F' && ch<='\u009F') || (ch>='\u2000' && ch<='\u20FF')){
String hex = Integer.toHexString(ch);
builder.append("\\u");
for(int j = 0; j < 4 - hex.length(); j++)
builder.append('0');
builder.append(hex.toUpperCase());
} else {
builder.append(ch);
}
}
}
}
}
In this class you can register converters for the logical types that you need. Consider the following example:
GenericRecordJsonEncoder encoder = new GenericRecordJsonEncoder();
// Register as many logical types converters as you need
encoder.registerLogicalTypeConverter(LogicalTypes.timestampMillis(), o -> {
final Instant instant = (Instant)o;
final String result = DateTimeFormatter.ISO_INSTANT.format(instant);
return result;
});
String json = encoder.serialize(genericRecord);
System.out.println(json);
This will provide you the desired result.
I have int, float, boolean and string from Properties file. Everything has loaded in Properties. Currently, I am parsing values as I know expected value for particular key.
Boolean.parseBoolean("false");
Integer.parseInt("3")
What is better way of setting these constants values, If I don't know what could be primitive value datatype for a key.
public class Messages {
Properties appProperties = null;
FileInputStream file = null;
public void initialization() throws Exception {
appProperties = new Properties();
try {
loadPropertiesFile();
} catch (Exception e) {
throw new Exception(e.getMessage(), e);
}
}
public void loadPropertiesFile() throws IOException {
String path = "./cfg/message.properties";
file = new FileInputStream(path);
appProperties.load(file);
file.close();
}
}
Properties File.
messassge.properties
SSO_URL = https://example.com/connect/token
SSO_API_USERNAME = test
SSO_API_PASSWORD = Uo88YmMpKUp
SSO_API_SCOPE = intraday_api
SSO_IS_PROXY_ENABLED = false
SSO_MAX_RETRY_COUNT = 3
SSO_FLOAT_VALUE = 3.0
Constant.java
public class Constants {
public static String SSO_URL = null;
public static String SSO_API_USERNAME = null;
public static String SSO_API_PASSWORD = null;
public static String SSO_API_SCOPE = null;
public static boolean SSO_IS_PROXY_ENABLED = false;
public static int SSO_MAX_RETRY_COUNT = 0;
public static float SSO_FLOAT_VALUE = 0;
}
If you have a class of configuration values, like your Constants class, and you want to load all values from a configuration (properties) file, you can create a little helper class and use reflection:
public class ConfigLoader {
public static void load(Class<?> configClass, String file) {
try {
Properties props = new Properties();
try (FileInputStream propStream = new FileInputStream(file)) {
props.load(propStream);
}
for (Field field : configClass.getDeclaredFields())
if (Modifier.isStatic(field.getModifiers()))
field.set(null, getValue(props, field.getName(), field.getType()));
} catch (Exception e) {
throw new RuntimeException("Error loading configuration: " + e, e);
}
}
private static Object getValue(Properties props, String name, Class<?> type) {
String value = props.getProperty(name);
if (value == null)
throw new IllegalArgumentException("Missing configuration value: " + name);
if (type == String.class)
return value;
if (type == boolean.class)
return Boolean.parseBoolean(value);
if (type == int.class)
return Integer.parseInt(value);
if (type == float.class)
return Float.parseFloat(value);
throw new IllegalArgumentException("Unknown configuration value type: " + type.getName());
}
}
Then you call it like this:
ConfigLoader.load(Constants.class, "/path/to/constants.properties");
You can extend the code to handle more types. You can also change it to ignore missing properties, instead of failing like it does now, such that assignments in the field declaration will remain unchanged, i.e. be the default.
If you know the type of constant, you can use Apache Commons Collections.
For example, you can use some utilities method based on type of your constant.
booelan SSO_IS_PROXY_ENABLED = MapUtils.getBooleanValue(appProperties, "SSO_IS_PROXY_ENABLED", false);
String SSO_URL = MapUtils.getString(appProperties, "SSO_URL", "https://example.com/connect/token");
You can even use default values to avoid errors.
Dambros is right, every thing you store inside a Properties file is as a String value.
You can track your different primitive data types after retrieving properties value as below like ref. -
Java Properties File: How to Read config.properties Values in Java?
package crunchify.com.tutorial;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.Date;
import java.util.Properties;
/**
* #author Crunchify.com
*
*/
public class CrunchifyGetPropertyValues {
String result = "";
InputStream inputStream;
public String getPropValues() throws IOException {
try {
Properties prop = new Properties();
String propFileName = "config.properties";
inputStream = getClass().getClassLoader().getResourceAsStream(propFileName);
if (inputStream != null) {
prop.load(inputStream);
} else {
throw new FileNotFoundException("property file '" + propFileName + "' not found in the classpath");
}
Date time = new Date(System.currentTimeMillis());
// get the property value and print it out
String user = prop.getProperty("user");
String company1 = prop.getProperty("company1");
String company2 = prop.getProperty("company2");
String company3 = prop.getProperty("company3");
result = "Company List = " + company1 + ", " + company2 + ", " + company3;
System.out.println(result + "\nProgram Ran on " + time + " by user=" + user);
} catch (Exception e) {
System.out.println("Exception: " + e);
} finally {
inputStream.close();
}
return result;
}
}
and later convert to primitive -
How to convert String to primitive type value?
I suggest you to track your data types value by putting the key values inside String type switch statement and later retrieve the related data type value by using key name cases.
String type switch case is possible after Java 7.
Not entirely sure whether I exactly understand the problem but a possibility could be to include the type of the property value in the (String) value. So for example the properties you showed would become something like:
SSO_URL = URL:https://example.com/connect/token
SSO_API_USERNAME = STRING:test
SSO_API_PASSWORD = STRING:Uo88YmMpKUp
SSO_API_SCOPE = STRING:intraday_api
SSO_IS_PROXY_ENABLED = BOOLEAN:false
SSO_MAX_RETRY_COUNT = INTEGER:3
SSO_FLOAT_VALUE = FLOAT:3.0
During the parsing of the property values you first determine the type of the property by looking at the part before : and use the part after for the actual parsing.
private static Object getValue(Properties props, String name) {
String propertyValue = props.getProperty(name);
if (propertyValue == null) {
throw new IllegalArgumentException("Missing configuration value: " + name);
} else {
String[] parts = string.split(":");
switch(parts[0]) {
case "STRING":
return parts[1];
case "BOOLEAN":
return Boolean.parseBoolean(parts[1]);
....
default:
throw new IllegalArgumentException("Unknown configuration value type: " + parts[0]);
}
}
}
Follow the dropwizard configuration pattern where you define your constants using YAML instead of Properties and use Jackson to deserialize it into your Class. Other than type safety, dropwizard's configuration pattern goes one step further by allowing Hibernate Validator annotations to validate that the values fall into your expected ranges.
For dropwizard's example...
http://www.dropwizard.io/0.9.2/docs/getting-started.html#creating-a-configuration-class
For more information about the technology involved...
github.com/FasterXML/jackson-dataformat-yaml
hibernate.org/validator/
Spring Boot has ready to use and feature reach solution for type-safe configuration properties.
Definitely, use of the Spring just for this task is overkill but Spring has a lot of cool features and this one can attract you to right side ;)
You can define your configurable parameters as 'static' in your class of choice, and from a static init call a method that loads the parameter values from a properties file.
For example:
public class MyAppConfig {
final static String propertiesPath="/home/workspace/MyApp/src/config.properties";
static String strParam;
static boolean boolParam;
static int intParam;
static double dblParam;
static {
// Other static initialization tasks...
loadParams();
}
private static void loadParams(){
Properties prop = new Properties();
try (InputStream propStream=new FileInputStream(propertiesPath)){
// Load parameters from config file
prop.load(propStream);
// Second param is default value in case key-pair is missing
strParam=prop.getProperty("StrParam", "foo");
boolParam=Boolean.parseBoolean(prop.getProperty("boolParam", "false"));
intParam= Integer.parseInt(prop.getProperty("intParam", "1"));
dblParam=Double.parseDouble(prop.getProperty("dblParam", "0.05"));
} catch (IOException e) {
logger.severe(e.getMessage());
e.printStackTrace();
}
}
}
This might help:
props.getProperty("name", Integer.class);
I'm developing an Android App which uses JSON for the server communication and I've got a weird problem when I'm trying to parse my json file.
This is my json from the server
{
"street2": null,
"province": null,
"street1": null,
"postalCode": null,
"country": null,
"city": null
}
I'm getting the value for City by calling String city = address.optString("city", "") on my address Json-object. For this situation I'm expecting cityto be empty (that's what optString is here for isn't it?) but in fact it contains the String "null". So further null- or isEmpty-checks will return false as the String contains text. If I call address.isNull("city") it returns true which is correct. Only optString fails.
I couldn't find anything on Google or Stackoverflow for this problem. I don't really understand how it can happen as I thought optString would do exactly what I expected. Anybody knows what's going wrong here?
You're not alone in running into this problem and scratching your head, thinking "Could they really have meant this?" According to an AOSP issue, the Google engineers did consider this a bug, but they had to be compatible with the org.json implementation, even bug-compatible.
If you think about it, it makes sense, because if the same code which uses the same libraries run in other Java environments behaves differently in Android, there would be major compatibility problems when using 3rd party libraries. Even if the intentions were good and it truly fixed bugs, it would open up a whole new can of worms.
According to the AOSP issue:
The behavior is intentional; we went out of our way to be bug-compatible with org.json. Now that that's fixed, it's unclear whether we should fix our code as well. Applications may have come to rely on this buggy behavior.
If this is causing you grief, I recommend you workaround by using a different mechanism to test for null, such as json.isNull().
Here's a simple method to help you out:
/** Return the value mapped by the given key, or {#code null} if not present or null. */
public static String optString(JSONObject json, String key)
{
// http://code.google.com/p/android/issues/detail?id=13830
if (json.isNull(key))
return null;
else
return json.optString(key, null);
}
You basically have 2 choices:
1) Send a JSON payload with null values
{
"street2": "s2",
"province": "p1",
"street1": null,
"postalCode": null,
"country": null,
"city": null
}
You will have to check for null values and parse them accordingly:
private String optString_1(final JSONObject json, final String key) {
return json.isNull(key) ? null : json.optString(key);
}
2) Do not send the keys with null values and use optString(key, null) directly (should save you bandwidth).
{
"street2": "s2",
"province": "p1"
}
Got rid off this situation by simply replacing "null" with "".
String city = address.optString("city").replace("null", "");
Using Matt Quigley's answer as a basis, here is the code if you desire to mimic the full functionality of optString, including the fallback portion, written in Kotlin and Java.
Kotlin:
fun optString(json: JSONObject, key: String, fallback: String?): String? {
var stringToReturn = fallback
if (!json.isNull(key)) {
stringToReturn = json.optString(key, null)
}
return stringToReturn
}
Java:
public static String optString(JSONObject json, String key, String fallback) {
String stringToReturn = fallback;
if (!json.isNull(key)) {
stringToReturn = json.optString(key, null);
}
return stringToReturn;
}
Simply pass in null for the fallback parameter if you don't need the fallback.
I ended up creating a utility class for this purpose:
import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import org.json.JSONException;
import org.json.JSONObject;
final public class JsonUtil {
#Nullable
public static String optString(
#NonNull JSONObject object,
#NonNull String key
) throws JSONException {
if (object.has(key) && !object.isNull(key)) {
return object.getString(key);
}
return null;
}
}
I do like this...
String value;
if(jsonObject.get("name").toString().equals("null")) {
value = "";
}else {
value = jsonObject.getString("name");
}
if (json != null && json.getString(KEY_SUCCESS) != null){
// PARSE RESULT
}else{
// SHOW NOTIFICIATION: URL/SERVER NOT REACHABLE
}
that is for checking json null with there key word.
JSONObject json = new JSONObject("{\"hello\":null}");
json.getString("hello");
this you get is String "null" not null.
your shoud use
if(json.isNull("hello")) {
helloStr = null;
} else {
helloStr = json.getString("hello");
}
first check with isNull()....if cant work then try belows
and also you have JSONObject.NULL to check null value...
if ((resultObject.has("username")
&& null != resultObject.getString("username")
&& resultObject.getString("username").trim().length() != 0)
{
//not null
}
and in your case also check
resultObject.getString("username").trim().eqauls("null")
If you must parse json first and handle object later, let try this
Parser
Object data = json.get("username");
Handler
if (data instanceof Integer || data instanceof Double || data instanceof Long) {
// handle number ;
} else if (data instanceof String) {
// hanle string;
} else if (data == JSONObject.NULL) {
// hanle null;
}
My Josn parser was long and had to create a new class to fix that,
then just had to add 1 extra line in each method and rename current JSONObject property name, so all other calls were referencing to my new class instead to JSONObject.
public static ArrayList<PieceOfNews> readNews(String json) {
if (json != null) {
ArrayList<PieceOfNews> res = new ArrayList<>();
try {
JSONArray jsonArray = new JSONArray(json);
for (int i = 0; i < jsonArray.length(); i++) {
//before JSONObject jo = jsonArray.getJSONObject(i);
JSONObject joClassic = jsonArray.getJSONObject(i);
//facade
FixJsonObject jo = new FixJsonObject(joClassic);
PieceOfNews pn = new PieceOfNews();
pn.setId(jo.getInt("id"));
pn.setImageUrl(jo.getString("imageURL"));
pn.setText(jo.getString("text"));
pn.setTitle(jo.getString("title"));
pn.setDate(jo.getLong("mills"));
res.add(pn);
}
return res;
} catch (JSONException e) {
e.printStackTrace();
}
}
return null;
}
Here is my class with the methods I needed, you can add more
public class FixJsonObject {
private JSONObject jsonObject;
public FixJsonObject(JSONObject jsonObject) {
this.jsonObject = jsonObject;
}
public String optString(String key, String defaultValue) {
if (jsonObject.isNull(key)) {
return null;
} else {
return jsonObject.optString(key, defaultValue);
}
}
public String optString(String key) {
return optString(key, null);
}
public int optInt(String key) {
if (jsonObject.isNull(key)) {
return 0;
} else {
return jsonObject.optInt(key, 0);
}
}
public double optDouble(String key) {
return optDouble(key, 0);
}
public double optDouble(String key, double defaultValue) {
if (jsonObject.isNull(key)) {
return 0;
} else {
return jsonObject.optDouble(key, defaultValue);
}
}
public boolean optBoolean(String key, boolean defaultValue) {
if (jsonObject.isNull(key)) {
return false;
} else {
return jsonObject.optBoolean(key, defaultValue);
}
}
public long optLong(String key) {
if (jsonObject.isNull(key)) {
return 0;
} else {
return jsonObject.optLong(key, 0);
}
}
public long getLong(String key) {
return optLong(key);
}
public String getString(String key) {
return optString(key);
}
public int getInt(String key) {
return optInt(key);
}
public double getDouble(String key) {
return optDouble(key);
}
public JSONArray getJSONArray(String key) {
if (jsonObject.isNull(key)) {
return null;
} else {
return jsonObject.optJSONArray(key);
}
}
}
If values for key is null like below
{
"status": 200,
"message": "",
"data": {
"totalFare": null,
},
}
check with "isNull" , for Eg:
String strTotalFare;
if (objResponse.isNull("totalFare"))
{
strTotalFare = "0";
} else {
strTotalFare = objResponse.getString("totalFare");
}
if value is "null" for key "totalFare", above function will enter in if and assign value zero else it will get actual value from key.
I'm not at all sure that this is even a solvable problem, but supposing that I have a freemarker template, I'd like to be able to ask the template what variables it uses.
For my purposes, we can assume that the freemarker template is very simple - just "root level" entries (the model for such a template could be a simple Map). In other words, I don't need to handle templates that call for nested structures, etc.
one other way to get the variables from java. This just tries to process the template and catch the InvalidReferenceException to find all the variables in a freemarker-template
/**
* Find all the variables used in the Freemarker Template
* #param templateName
* #return
*/
public Set<String> getTemplateVariables(String templateName) {
Template template = getTemplate(templateName);
StringWriter stringWriter = new StringWriter();
Map<String, Object> dataModel = new HashMap<>();
boolean exceptionCaught;
do {
exceptionCaught = false;
try {
template.process(dataModel, stringWriter);
} catch (InvalidReferenceException e) {
exceptionCaught = true;
dataModel.put(e.getBlamedExpressionString(), "");
} catch (IOException | TemplateException e) {
throw new IllegalStateException("Failed to Load Template: " + templateName, e);
}
} while (exceptionCaught);
return dataModel.keySet();
}
private Template getTemplate(String templateName) {
try {
return configuration.getTemplate(templateName);
} catch (IOException e) {
throw new IllegalStateException("Failed to Load Template: " + templateName, e);
}
}
This is probably late, but in case anyone else encountered this problem : you can use 'data_model' and 'globals' to inspect the model - data_model will only contain values provided by the model while globals will also contain any variables defined in the template.
You need to prepend the special variables with a dot - so to access globals, use ${.globals}
For other special variables see http://freemarker.sourceforge.net/docs/ref_specvar.html
I had the same task to get the list of variables from template on java side and don't found any good approaches to that except using reflection. I'm not sure whether there is a better way to get this data or not but here's my approach:
public Set<String> referenceSet(Template template) throws TemplateModelException {
Set<String> result = new HashSet<>();
TemplateElement rootTreeNode = template.getRootTreeNode();
for (int i = 0; i < rootTreeNode.getChildCount(); i++) {
TemplateModel templateModel = rootTreeNode.getChildNodes().get(i);
if (!(templateModel instanceof StringModel)) {
continue;
}
Object wrappedObject = ((StringModel) templateModel).getWrappedObject();
if (!"DollarVariable".equals(wrappedObject.getClass().getSimpleName())) {
continue;
}
try {
Object expression = getInternalState(wrappedObject, "expression");
switch (expression.getClass().getSimpleName()) {
case "Identifier":
result.add(getInternalState(expression, "name").toString());
break;
case "DefaultToExpression":
result.add(getInternalState(expression, "lho").toString());
break;
case "BuiltinVariable":
break;
default:
throw new IllegalStateException("Unable to introspect variable");
}
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new TemplateModelException("Unable to reflect template model");
}
}
return result;
}
private Object getInternalState(Object o, String fieldName) throws NoSuchFieldException, IllegalAccessException {
Field field = o.getClass().getDeclaredField(fieldName);
boolean wasAccessible = field.isAccessible();
try {
field.setAccessible(true);
return field.get(o);
} finally {
field.setAccessible(wasAccessible);
}
}
Sample project that I made for demonstrating template introspection can be found on github: https://github.com/SimY4/TemplatesPOC.git
I solved this for my very simple usecase (only using flat datastructure, no nesting (for example: ${parent.child}), lists or more specific) with dummy data provider:
public class DummyDataProvider<K, V> extends HashMap<K, V> {
private static final long serialVersionUID = 1;
public final Set<String> variables = new HashSet<>();
#SuppressWarnings("unchecked")
#Override
public V get(Object key) {
variables.add(key.toString());
return (V) key;
}
}
You can give this for processing to a template and when it finishes Set variables contains your variables.
This is very simplistic approach, which certainly needs improvement, but you get the idea.
public static Set<String> getNecessaryTemplateVariables(String templateName) throws TemplateModelException {
Set<String> result = new HashSet<>();
TemplateElement rootTreeNode = getTemplate(templateName).getRootTreeNode();
if ("IteratorBlock".equals(rootTreeNode.getClass().getSimpleName())) {
introspectFromIteratorBlock(result, rootTreeNode);
return result;
}
for (int i = 0; i < rootTreeNode.getChildCount(); i++) {
TemplateModel templateModel = rootTreeNode.getChildNodes().get(i);
if (!(templateModel instanceof StringModel)) {
continue;
}
Object wrappedObject = ((StringModel) templateModel).getWrappedObject();
if ("DollarVariable".equals(wrappedObject.getClass().getSimpleName())) {
introspectFromDollarVariable(result, wrappedObject);
} else if ("ConditionalBlock".equals(wrappedObject.getClass().getSimpleName())) {
introspectFromConditionalBlock(result, wrappedObject);
} else if ("IfBlock".equals(wrappedObject.getClass().getSimpleName())) {
introspectFromIfBlock(result, wrappedObject);
} else if ("IteratorBlock".equals(wrappedObject.getClass().getSimpleName())) {
introspectFromIteratorBlock(result, wrappedObject);
}
}
return result;
}
private static void introspectFromIteratorBlock(Set<String> result, Object wrappedObject) {
try {
Object expression = getInternalState(wrappedObject, "listExp");
result.add(getInternalState(expression, "name").toString());
} catch (NoSuchFieldException | IllegalAccessException ignored) {
}
}
private static void introspectFromConditionalBlock(Set<String> result, Object wrappedObject)
throws TemplateModelException {
try {
Object expression = getInternalState(wrappedObject, "condition");
if (expression == null) {
return;
}
result.addAll(dealCommonExpression(expression));
String nested = getInternalState(wrappedObject, "nestedBlock").toString();
result.addAll(getNecessaryTemplateVariables(nested));
} catch (NoSuchFieldException | IllegalAccessException ignored) {
}
}
private static Set<String> dealCommonExpression(Object expression)
throws NoSuchFieldException, IllegalAccessException {
Set<String> ret = Sets.newHashSet();
switch (expression.getClass().getSimpleName()) {
case "ComparisonExpression":
String reference = dealComparisonExpression(expression);
ret.add(reference);
break;
case "ExistsExpression":
reference = dealExistsExpression(expression);
ret.add(reference);
break;
case "AndExpression":
ret.addAll(dealAndExpression(expression));
default:
break;
}
return ret;
}
private static String dealComparisonExpression(Object expression)
throws NoSuchFieldException, IllegalAccessException {
Object left = getInternalState(expression, "left");
Object right = getInternalState(expression, "right");
String reference;
if ("Identifier".equals(left.getClass().getSimpleName())) {
reference = getInternalState(left, "name").toString();
} else {
reference = getInternalState(right, "name").toString();
}
return reference;
}
private static String dealExistsExpression(Object expression) throws NoSuchFieldException, IllegalAccessException {
Object exp = getInternalState(expression, "exp");
return getInternalState(exp, "name").toString();
}
private static Set<String> dealAndExpression(Object expression) throws NoSuchFieldException,
IllegalAccessException{
Set<String> ret = Sets.newHashSet();
Object lho = getInternalState(expression, "lho");
ret.addAll(dealCommonExpression(lho));
Object rho = getInternalState(expression, "rho");
ret.addAll(dealCommonExpression(rho));
return ret;
}
private static void introspectFromIfBlock(Set<String> result, Object wrappedObject)
throws TemplateModelException {
result.addAll(getNecessaryTemplateVariables(wrappedObject.toString()));
}
private static void introspectFromDollarVariable(Set<String> result, Object wrappedObject)
throws TemplateModelException {
try {
Object expression = getInternalState(wrappedObject, "expression");
switch (expression.getClass().getSimpleName()) {
case "Identifier":
result.add(getInternalState(expression, "name").toString());
break;
case "DefaultToExpression":
result.add(getInternalState(expression, "lho").toString());
break;
case "BuiltinVariable":
break;
default:
break;
}
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new TemplateModelException("Unable to reflect template model");
}
}
private static Object getInternalState(Object o, String fieldName) throws NoSuchFieldException, IllegalAccessException {
Field [] fieldArray = o.getClass().getDeclaredFields();
for (Field field : fieldArray) {
if (!field.getName().equals(fieldName)) {
continue;
}
boolean wasAccessible = field.isAccessible();
try {
field.setAccessible(true);
return field.get(o);
} finally {
field.setAccessible(wasAccessible);
}
}
throw new NoSuchFieldException();
}
private static Template getTemplate(String templateName) {
try {
StringReader stringReader = new StringReader(templateName);
return new Template(null, stringReader, null);
} catch (IOException e) {
throw new IllegalStateException("Failed to Load Template: " + templateName, e);
}
}
I optimized SimY4's answer, and supported <#list> and <#if> block. Code is not fully tested
Execute following regex on the template:
(?<=\$\{)([^\}]+)(?=\})
(?<=\$\{) Matches everything followed by ${
([^\}]+) Matches any string not containing }
(?=\}) Matches everything before }
I had the same problem and none of posted solution made sense to me. What I came with is plugging custom implementation of
TemplateExceptionHandler. Example:
final private List<String> missingReferences = Lists.newArrayList();
final Configuration cfg = new Configuration(Configuration.VERSION_2_3_23);
cfg.setTemplateExceptionHandler(new TemplateExceptionHandler() {
#Override
public void handleTemplateException(TemplateException arg0, Environment arg1, Writer arg2) throws TemplateException {
if (arg0 instanceof InvalidReferenceException) {
missingReferences.add(arg0.getBlamedExpressionString());
return;
}
throw arg0;
}
}
Template template = loadTemplate(cfg, templateId, templateText);
StringWriter out = new StringWriter();
try {
template.process(templateData, out);
} catch (TemplateException | IOException e) {
throw new RuntimeException("oops", e);
}
System.out.println("Missing references: " + missingReferences);
Read more here: https://freemarker.apache.org/docs/pgui_config_errorhandling.html