How to parse newline delimited Json response in Android? - java

Sample of NdJson data:
{"type":"data","id":"xyz"}
{"type":"value","id":"xcf"}
....
....
Here is my Retrofit and RxJava code which is working fine for fetching data with the limit=1 i.e. {"type":"data","id":"xyz"}.
adapter.create(API.class).getData()
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Observer<APIData>() {
#Override
public void onCompleted() {}
#Override
public void onError(Throwable e) {}
#Override
public void onNext(APIData apidata) {}
});
My model class
Model class have only two parameter:
public class APIData(){
private String type;
private String id;
..// Getter and setter for above two fields
}
Api class
public interface WarehouseAPI {
#GET("/search?limit=1")
public Observable<APIData> getdata ();
}
The error I am getting while replacing #GET("/search?limit=1") with #GET("/search") is Malformed JSON: Syntax error.
Question
How to proper parse the NdJson ?
Is there any way to store the response in List<APIData>
Edit-1
Now I am trying to accept the generic Response in observer:
adapter.create(APIData.class).getdata()
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Observer<Response>() {
#Override
public void onCompleted() {}
#Override
public void onNext(Response response) {}
});
But getting no suitable method found for subscribe(<anonymous Observer<Response>>)
Edit-2
However, I was doing some silly mistake and the error what I am getting in "Edit-1" section is fixed now. Now I am getting
Error:retrofit.RetrofitError: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 2 column 2 path $

To make things work I had to add Custom converter implements by Retrofit.Converter:
public class CustomConverter implements Converter {
#Override
public Object fromBody(TypedInput body, Type type) throws ConversionException {
String text = null;
try {
text = fromStream(body.in());
} catch (IOException e) {
e.printStackTrace();
}
return text;
}
#Override
public TypedOutput toBody(Object object) {
return null;
}
// Custom method to convert stream from request to string
public static String fromStream(InputStream in) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder out = new StringBuilder();
String newLine = System.getProperty("\n");
String line;
while ((line = reader.readLine()) != null) {
out.append(line);
out.append(newLine);
}
return out.toString();
}
}
And it works perfectly!

I assume that server response has the following format
{"type":"data","id":"xyz"}\n{"type":"data","id":"xyz"}
The basic idea is to receive the response from the server as the String. Split it into array like response.split("\n"). Iterate through an Array creating a new json object for every array element.
I do realize it's quite time consuming as you've described in the comments. You can also try to play with String replaceAll method to transform each line into an array element and parse the whole string. like
String myResponse = "[" + response.replaceAll("/\n/", ",") + "]";
Gson gson = new Gson();
MyEntity[] arr = gson.fromJson(myResponse, MyEntity[].class);
In case of Retrofit. You will have to use a custom Response Converter.
I won't write up the complete solution since you have found the way to get it done using custom Converter.

For starters, your json is not json that Gson understand, so you can't use the Gson converter in Retrofit directly. So
public Observable<APIData> getdata ();
must be
public Observable<Response> getdata ();
Then, you have
adapter.create(WarehouseAPI.class).getdata()
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.map((Response res) -> new BufferedStreamReader(res.body.body().bytesStream()))
.flatMap(stream -> {
return Observable.create(subscriber -> {
String line;
while((line = stream.readLine() != null) {
subscriber.onNext(gson.fromJson(line, APIData.class));
}
subscriber.onCompleted();
});
});
You can subscribe to that and receive each individual item of your json list.
This is not tested, and error cases are not handled (and validity of subscription is not tested, for shortness).
It should mostly be in the right direction though.

Related

How to get parameters from DialogFlow in Java (Android Studio)

I need to get parameters from DialogFlow to my Android app.
I tried using getQueryResult().getParameters().getFieldsMap()
but the result is the following.
{type=list_value {
values {
string_value: "pizza"
}
}
, ristorante=string_value: ""
}
I would like to get just the string value "pizza" and not the entire FieldMap.
I have already seen this topic, but it didn't help me, because I don't know what protobuf is and seems a bit complicated.
Is there a simple way to get a parameter's value?
I see two possibilities:
Try to access the Map values directly.
The getFieldsMap() method returns a java.util.Map class. You can try to retrieve the values by getting first a collection of Values, then iterate:
Collection colletion = <Detect_Intent_Object>.getQueryResult().getParameters().getFieldsMap().values():
for (iterable_type iterable_element : collection)
From my humble point of view the bucle is necesary because there could be more than one parameter.
Transform the protobuf response into a json and access the values.
Sample code:
import com.google.protobuf.util.JsonFormat;
String jsonString = JsonFormat.printToString(<Detect_Intent_Object>.getQueryResult().getParameters());
// Then use a json parser to obtain the values
import org.json.*;
JSONObject obj = new JSONObject(jsonString);
JSONArray jsonnames = obj.names();
Method names() will let you know the string names you want to access.
If you use Dialogflowv2
public String getParameter(GoogleCloudDialogflowV2WebhookRequest request, String parameterName) {
try {
GoogleCloudDialogflowV2QueryResult queryResult = request.getQueryResult();
Map<String, Object> parameters = queryResult.getParameters();
String parameter = (String) parameters.get(parameterName);
if(parameter != null && !parameter.equals("")) {
return parameter;
}
} catch (ClassCastException e) {
logger.error("Error");
}
return null;
}
If you use GoogleActions
public String getParameter(ActionRequest request, String parameterName) {
try {
Map<String, Object> parameters = request.getWebhookRequest().getQueryResult().getParameters();
String parameter = (String) parameters.get(parameterName);
if(parameter != null && !parameter.equals("")) {
return parameter;
}
} catch (ClassCastException e) {
logger.error("Error");
}
return null;
}

Odd behaviour with hibernate JSON converter

I am having some trouble with converting a Set of POJO's to a JSON string so I can store it in a database column. The conversion to JSON works as expected but when the conversion from JSON to a Set< Qualification > happens it always returns a LinkedHashSet<LinkedHashMap> which is causing issues.
The weird thing is that inside my converter the JSON string is converted successfully to a Set<Qualification>. When I debug in my IDE and step through the execution I can see that after it calls a deepCopy method in the MutableMutabilityPlan abstract class. At this point the data is of type LinkedHashSet<LinkedHashMap> and not Set<Qualification> from the conversion.
Here is my converter.
public class SetToStringConverter implements AttributeConverter<Set<Qualification>, String> {
private final ObjectMapper mapper = new ObjectMapper();
private final String errorMessage = "converter.invalid";
#Override
public String convertToDatabaseColumn(final Set<Qualification> items) {
try {
return mapper.writeValueAsString(items);
} catch (JsonProcessingException e) {
throw new ConverterFailureException(errorMessage);
}
}
#Override
public Set<Qualification> convertToEntityAttribute(final String data) {
try {
if (data != null) {
final Set<Qualification> s = mapper.readValue(data, new TypeReference<Set<Qualification>>() {});
return s; // Debugging here I have the correct type
}
return new HashSet<>();
} catch (IOException e) {
throw new ConverterFailureException(errorMessage);
}
}
}
I have done some research and have tried various approaches but the result is always the same.
Has anyone run into this issue before or can see anything wrong with my converter. Let me know if anything isn't clear so I can provide more information.
Thanks very much for the help.

Android - waiting for json to get value before continuing with method

I am using this library in order to get data from server. The data is decode in the server into JSONObject. The method I made will call the url and return the number of rows inside a mysql table.
The method is working however, I cannot return the value properly from my method:
ParseLevels.java
static public int countLevels() {
final int[] count = {0};
AndroidNetworking.get("https://example.com/gameLevels.php?option=count")
.setPriority(Priority.LOW)
.build()
.getAsJSONObject(new JSONObjectRequestListener() {
#Override
public void onResponse(JSONObject response) {
// responce is {count:20}
try {
count[0] = response.getInt("count"); // this is getting the value of 20
} catch (JSONException e) {
e.printStackTrace();
}
Log.d("responce_app", String.valueOf(count[0]));
}
#Override
public void onError(ANError error) {
// handle error
Log.d("responce_app", String.valueOf(error)); //Logs out 20
}
});
return count[0]; // return always 0
}
When I call from my MainActivity
Log.d("responce_app", String.valueOf(ParseLevels.countLevels()));
the method returns always 0.
I understand that the return is fired before the jsonObject is fetched however, how can I wait for the method to fetch the jsonObject and after return the value?
In iOS I use something like:
static func getLevels(feedsArray: (levelsCount : [Int]) -> Void) {
}
how could convert this into Java?
How can I wait for the method to fetch the jsonObject and after return
the value?
Two options:
1. Do code inside onResponse method which want to execute according to result of request.
2. Create event listener using interface and implement it in countLevels method caller class to execute block of code when onResponse method execution done
You should look at the the Making Synchronous Request example from your library's README:
https://github.com/amitshekhariitbhu/Fast-Android-Networking
ANRequest request = AndroidNetworking.get("https://fierce-cove-29863.herokuapp.com/getAllUsers/{pageNumber}")
.addPathParameter("pageNumber", "0")
.addQueryParameter("limit", "3")
.build();
ANResponse<List<User>> response = request.executeForParsed(new TypeToken<List<User>>() {});
if (response.isSuccess()) {
List<User> users = responseTwo.getResult();
} else {
//handle error
}
You would return your count in the response.isSuccess() conditional.

What security issues come from calling methods with reflection?

I'm working on a project that has hosts and clients, and where hosts can send commands to clients (via sockets).
I'm determined that using JSON to communicate works the best.
For example:
{
"method" : "toasty",
"params" : ["hello world", true]
}
In this example, when this JSON string is sent to the client, it will be processed and a suitable method within the client will be run as such:
public abstract class ClientProcessor {
public abstract void toasty(String s, boolean bool);
public abstract void shutdown(int timer);
private Method[] methods = getClass().getDeclaredMethods();
public void process(String data) {
try {
JSONObject json = new JSONObject(data);
String methodName = (String) json.get("method");
if (methodName.equals("process"))
return;
for (int i = 0; i < methods.length; i++)
if (methods[i].getName().equals(methodName)) {
JSONArray arr = json.getJSONArray("params");
int length = arr.length();
Object[] args = new Object[length];
for (int i2 = 0; i2 < length; i2++)
args[i2] = arr.get(i2);
methods[i].invoke(this, args);
return;
}
} catch (Exception e) {}
}
}
And using the ClientProcessor:
public class Client extends ClientProcessor {
#Override
public void toasty(String s, boolean bool) {
//make toast here
}
#Override
public void shutdown(int timer) {
//shutdown system within timer
}
public void processJSON(String json) {
process(json);
}
}
The JSON is sent by the server to the client, but the server could be modified to send different JSONs.
My questions are:
Is this a safe way of running methods by processing JSON?
Is there a better way to do this? I'm thinking that using reflection is terribly slow.
There's a 100 and 1 ways you can process a JSON message so that some processing occurs, but they'll all boil down to:
parse message
map message to method
invoke method
send response
While you could use a reflective call (performance-wise it would be fine for most cases) to invoke a method, that, imho, would be a little too open - a malicious client could for example crash your system by issuing wait calls.
Reflection also opens you up to having to correctly map the parameters, which is more complicated than the code you've shown in your question.
So don't use Reflection.
Would you could do is define a simple interface, implementations of which would understand how to process the parameters and have your processor (more commonly referred to as a Controller) invoke that, something like this:
public interface ServiceCall
{
public JsonObject invoke(JsonArray params) throws ServiceCallException;
}
public class ServiceProcessor
{
private static final Map<String, ServiceCall> SERVICE_CALLS = new HashMap<>();
static
{
SERVICE_CALLS.put("toasty", new ToastCall());
}
public String process(String messageStr)
{
try
{
JsonObject message = Json.createReader(new StringReader(messageStr)).readObject();
if (message.containsKey("method"))
{
String method = message.getString("method");
ServiceCall serviceCall = SERVICE_CALLS.get(method);
if (serviceCall != null)
{
return serviceCall.invoke(message.getJsonArray("params")).toString();
}
else
{
return fail("Unknown method: " + method);
}
}
else
{
return fail("Invalid message: no method specified");
}
}
catch (Exception e)
{
return fail(e.message);
}
}
private String fail(String message)
{
return Json.createObjectBuilder()
.add("status", "failed")
.add("message", message)
.build()
.toString();
}
private static class ToastCall implements ServiceCall
{
public JsonObject invoke(JsonArray params) throws ServiceCallException
{
//make toast here
}
}
}
Map method names to int constants and just switch(case) on these constants to invoke appropriate method.
"toasty" : 1
"shutdown": 2
switch()
case 1: toasty()
case 2: shutdown()
I believe you are trying to convert JSON string to Java object and vice versa... if that is the requirement then this would not be the right approach...
Try any open source API like Gson...
it is the API by Google for conversin of Java to JSON and vice versa.
Please check ...
https://google-gson.googlecode.com/svn/trunk/gson/docs/javadocs/com/google/gson/Gson.html
Let me know if you have any further questions...

Looking for Json-path/(any API) to update any value in given json string in Java

Inshort : I am trying to find some api that could just change the value by taking first parameter as jsonString , second parameter as JSONPath and third will be new value of that parameter. But, all I found is this..
https://code.google.com/p/json-path/
This api allows me to find any value in JSON String. But, I am not finding easy way to update the value of any key. For example, Here is a book.json.
{
"store":{
"book":[
{
"category":"reference",
"author":"Nigel Rees",
"title":"Sayings of the Century",
"price":8.95
},
{
"category":"fiction",
"author":"Evelyn Waugh",
"title":"Sword of Honour",
"price":12.99,
"isbn":"0-553-21311-3"
}
],
"bicycle":{
"color":"red",
"price":19.95
}
}
}
I can access color of bicycle by doing this.
String bicycleColor = JsonPath.read(json, "$.store.bicycle.color");
But I am looking for a method in JsonPath or other api some thing like this
JsonPath.changeNodeValue(json, "$.store.bicycle.color", "green");
String bicycleColor = JsonPath.read(json, "$.store.bicycle.color");
System.out.println(bicycleColor); // This should print "green" now.
I am excluding these options,
Create a new JSON String.
Create a JSON Object to deal with changing value and convert it back to jsonstring
Reason: I have about 500 different requests for different types of service which return different json structure. So, I do not want to manually create new JSON string always. Because, IDs are dynamic in json structure.
Any idea or direction is much appreciated.
Updating this question with following answer.
Copy MutableJson.java.
copy this little snippet and modify as per you need.
private static void updateJsonValue() {
JSONParser parser = new JSONParser();
JSONObject jsonObject = new JSONObject();
FileReader reader = null;
try {
File jsonFile = new File("path to book.json");
reader = new FileReader(jsonFile);
jsonObject = (JSONObject) parser.parse(reader);
} catch (Exception ex) {
System.out.println(ex.getLocalizedMessage());
}
Map<String, Object> userData = null;
try {
userData = new ObjectMapper().readValue(jsonObject.toJSONString(), Map.class);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
MutableJson json = new MutableJson(userData);
System.out.println("Before:\t" + json.map());
json.update("$.store.book[0].author", "jigish");
json.update("$.store.book[1].category", "action");
System.out.println("After:\t" + json.map().toString());
}
Use these libraries.
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.codehaus.jackson.map.ObjectMapper;
The thing is that the functionality you want is already an undocumented feature of JsonPath. Example using your json structure:
String json = "{ \"store\":{ \"book\":[ { \"category\":\"reference\", \"author\":\"Nigel Rees\", \"title\":\"Sayings of the Century\", \"price\":8.95 }, { \"category\":\"fiction\", \"author\":\"Evelyn Waugh\", \"title\":\"Sword of Honour\", \"price\":12.99, \"isbn\":\"0-553-21311-3\" } ], \"bicycle\":{ \"color\":\"red\", \"price\":19.95 } } }";
DocumentContext doc = JsonPath.parse(json).
set("$.store.bicycle.color", "green").
set("$.store.book[0].price", 9.5);
String newJson = new Gson().toJson(doc.read("$"));
Assuming that parsed JSON can be represented in memory as a Map, you can build an API similar to JsonPath that looks like:
void update(Map<String, Object> json, String path, Object newValue);
I've quickly done a gist of a dirty implementation for simple specific paths (no support for conditions and wildcards) that can traverse json tree, E.g. $.store.name, $.store.books[0].isbn. Here it is: MutableJson.java. It definitely needs improvement, but can give a good start.
Usage example:
import java.util.*;
public class MutableJson {
public static void main(String[] args) {
MutableJson json = new MutableJson(
new HashMap<String, Object>() {{
put("store", new HashMap<String, Object>() {{
put("name", "Some Store");
put("books", Arrays.asList(
new HashMap<String, Object>() {{
put("isbn", "111");
}},
new HashMap<String, Object>() {{
put("isbn", "222");
}}
));
}});
}}
);
System.out.println("Before:\t" + json.map());
json.update("$.store.name", "Book Store");
json.update("$.store.books[0].isbn", "444");
json.update("$.store.books[1].isbn", "555");
System.out.println("After:\t" + json.map());
}
private final Map<String, Object> json;
public MutableJson(Map<String, Object> json) {
this.json = json;
}
public Map<String, Object> map() {
return json;
}
public void update(String path, Object newValue) {
updateJson(this.json, Path.parse(path), newValue);
}
private void updateJson(Map<String, Object> data, Iterator<Token> path, Object newValue) {
Token token = path.next();
for (Map.Entry<String, Object> entry : data.entrySet()) {
if (!token.accept(entry.getKey(), entry.getValue())) {
continue;
}
if (path.hasNext()) {
Object value = token.value(entry.getValue());
if (value instanceof Map) {
updateJson((Map<String, Object>) value, path, newValue);
}
} else {
token.update(entry, newValue);
}
}
}
}
class Path {
public static Iterator<Token> parse(String path) {
if (path.isEmpty()) {
return Collections.<Token>emptyList().iterator();
}
if (path.startsWith("$.")) {
path = path.substring(2);
}
List<Token> tokens = new ArrayList<>();
for (String part : path.split("\\.")) {
if (part.matches("\\w+\\[\\d+\\]")) {
String fieldName = part.substring(0, part.indexOf('['));
int index = Integer.parseInt(part.substring(part.indexOf('[')+1, part.indexOf(']')));
tokens.add(new ArrayToken(fieldName, index));
} else {
tokens.add(new FieldToken(part));
}
};
return tokens.iterator();
}
}
abstract class Token {
protected final String fieldName;
Token(String fieldName) {
this.fieldName = fieldName;
}
public abstract Object value(Object value);
public abstract boolean accept(String key, Object value);
public abstract void update(Map.Entry<String, Object> entry, Object newValue);
}
class FieldToken extends Token {
FieldToken(String fieldName) {
super(fieldName);
}
#Override
public Object value(Object value) {
return value;
}
#Override
public boolean accept(String key, Object value) {
return fieldName.equals(key);
}
#Override
public void update(Map.Entry<String, Object> entry, Object newValue) {
entry.setValue(newValue);
}
}
class ArrayToken extends Token {
private final int index;
ArrayToken(String fieldName, int index) {
super(fieldName);
this.index = index;
}
#Override
public Object value(Object value) {
return ((List) value).get(index);
}
#Override
public boolean accept(String key, Object value) {
return fieldName.equals(key) && value instanceof List && ((List) value).size() > index;
}
#Override
public void update(Map.Entry<String, Object> entry, Object newValue) {
List list = (List) entry.getValue();
list.set(index, newValue);
}
}
A JSON string can be easily parsed into a Map using Jackson:
Map<String,Object> userData = new ObjectMapper().readValue("{ \"store\": ... }", Map.class);
Just answering for folks landing on this page in future for reference.
You could consider using a Java implementation of jsonpatch. RFC can be found here
JSON Patch is a format for describing changes to a JSON document. It can be used to avoid sending a whole document when only a part has changed. When used in combination with the HTTP PATCH method it allows partial updates for HTTP APIs in a standards compliant way.
You can specify the operation that needs to be performed (replace, add....), json path at which it has to be performed, and the value which should be used.
Again, taking example from the RFC :
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]
For Java implementation, I have not used it myself, but you can give a try to https://github.com/fge/json-patch
So in order to change a value within a JSon string, there are two steps:
Parse the JSon
Modify the appropriate field
You are trying to optimize step 2, but understand that you are not going to be able to avoid step 1. Looking at the Json-path source code (which, really, is just a wrapper around Jackson), note that it does do a full parse of the Json string before being able to spit out the read value. It does this parse every time you call read(), e.g. it is not cached.
I think this task is specific enough that you're going to have to write it yourself. Here is what I would do:
Create an object that represents the data in the parsed Json string.
Make sure this object has, as part of it's fields, the Json String pieces that you do not expect to change often.
Create a custom Deserializer in the Json framework of your choice that will populate the fields correctly.
Create a custom Serializer that uses the cached String pieces, plus the data that you expect to change
I think the exact scope of your problem is unusual enough that it is unlikely a library already exists for this. When a program receives a Json String, most of the time what it wants is the fully deserialized object - it is unusual that it needs to FORWARD this object on to somewhere else.

Categories

Resources