I have my Apache Flink program:
import org.apache.flink.api.scala._
import scala.util.parsing.json._
object numHits extends App {
val env = ExecutionEnvironment.getExecutionEnvironment
val data=env.readTextFile("file:///path/to/json/file")
val j=data.map { x => ("\"\"\""+x+"\"\"\"") }
/*1*/ println( ((j.first(1).collect())(0)).getClass() )
/*2*/ println( ((j.first(1).collect())(0)) )
/*3*/ println( JSON.parseFull((j.first(1).collect())(0)) )
}
I want to parse the input JSON file into normal scala Map and for that I am using the default scala.util.parsing.json._ library.
The output of the first println statement is class java.lang.String which is required by the JSON parsing function.
Output of the second println function is the actual JSON string appended and prepended by "\"\"\"" which is also required by the JSON parser.
Now at this point if I copy the output of the second println command printed in the console and pass it to the JSON.parseFull() function, it properly parses it.
Therefore the third println function should properly parse the same string passed to it but it does not as it outputs a "None" string which means it failed.
Why does this happen and how can I make it work?
Output of the second println function is the actual JSON string appended and prepended by "\"\"\"" which is also required by the JSON parser.
No, of course it isn't. This produces a string like """{}""", which isn't valid JSON and this properly rejected by the parser. When you write """{}""" in Scala code, the quotes aren't part of the string itself, they just delimit the literal: the content of the string is {}, which is valid JSON.
You will have to just change
val j=data.map { x => ("\"\"\""+x+"\"\"\"") }
to
val j=data.map { x => x.replaceAll("\"", "\\\"") }
But the above code is not required as the code below will work:
val data=env.readTextFile("file:///path/to/json").flatMap( line => JSON.parseFull(line) )
Related
I have yaml:
- flowId: "2021:6:10:20:22:7"
flowType: "BIG"
summary: "Description"
flowStatus: "NEW"
createdDate: "2021-06-10"
lastModifiedDate: "2021-06-10"
class Flow{
...
FlowType flowType;
...
}
Enum FlowType{
SMALL,MEDIUM, BIG;
}
Parsing file using YamlSlurper:
def flowList = new YamlSlurper().parseText(new File(myFile).text).collect { it as Flow }
error: java.lang.ClassCastException: java.lang.String cannot be cast to model.FlowType
Is there a way to solve this?
The YAML slurper is a cute tool to quickly read a YAML file or string
and deal with it to the degree, that you would use the other slurpers:
get some basic data types inside lists and maps and just use that.
Your attempt to cast the map to the object only works for very basic
objects. The cast basically gets unrolled to something like:
[a: 42] as X
becomes
def x = new X()
map.each{ k, v ->
x."$k" = v
}
This does:
not coerce/convert types
fails if keys in the map, that are not set-able properties in the
resulting object
If you need proper object mapping, the slurpers are not directly useful
most of the time. You would rather switch to using something made for
that task -- e.g. like Jackson.
Lucky for us, the YAML slurper just uses Jackson (it actually just
transforms the YAML into JSON and then uses the JSON slurper to give you
your data back).
Following an example of how to load the YAML data into objects using
Jackson (all deps are already there, if you already see YamlSlurper):
import com.fasterxml.jackson.core.type.TypeReference
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import com.fasterxml.jackson.dataformat.yaml.YAMLMapper
def yamlStr = """\
- flowId: "2021:6:10:20:22:7"
flowType: "BIG"
summary: "Description"
flowStatus: "NEW"
createdDate: "2021-06-10"
lastModifiedDate: "2021-06-10"
"""
enum FlowType{ SMALL,MEDIUM, BIG }
#groovy.transform.ToString
class Flow {
FlowType flowType
}
def flows = new ObjectMapper(new YAMLFactory()).with{
configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
readValue(yamlStr, new TypeReference<List<Flow>>(){})
}
println flows
// → Flow(BIG)
I've parsed the Below Json file and retrieve the username value.
"LogInFunctionTest": [
{
"TestCaseID": "Login-TC_02",
"TestScenario": "Negative Case - Login with unregistered username and Password",
"TestData": [
{
"UserName": "usernameX",
"Password": "passwordX"
}
]
Using the below code to retrieve the UserName Value.
def InputJSON = new JsonSlurper().parse(new File(fileName))
def testDataItem = InputJSON.LogInFunctionTest.find { it.TestCaseID == Login-TC_02 }.TestData.UserName
Output - [usernameX]
After this, I want to remove the brackets from the above.
'Remove brackets from arraylist'
testDataItem = testDataItem.substring(1, testDataItem.length() - 1)
I'm getting the below exception
org.codehaus.groovy.runtime.InvokerInvocationException: groovy.lang.MissingMethodException: No signature of method: java.util.ArrayList.length() is applicable for argument types: () values: []
Possible solutions: last(), last(), init(), init(), get(int), get(int)
Anyone guide us on how to remove the brackets from the output?
testDataItem is a list of UserNames as TestData contains a list
That's why when it's displayed to you, it has [] round it...
If you just want the first one in the list, then you can do:
def testDataItem = InputJSON
.LogInFunctionTest
.find { it.TestCaseID == 'Login-TC_02' }
.TestData
.UserName
.first()
(ie: put a call to first() at the end)
Obviously, if there's two of them, you'll only be getting the first one
You could also get the first one with .UserName[0], but .first() is more descriptive
testDataItem = testDataItem.get(0)
might do the job.
Looks like you're reading a list of Strings, not a String.
Technically you referred to 'TestData' as a List with the brackets above so proper reference should be:
TestData[0].UserName
... or since it's all contained in the 'LogInFunctionTest' List:
LogInFunctionTest[0].TestData[0].UserName
That is if you are not going to loop/iterate through them.
I'm currently rewriting one of our applications, which uses PostgreSQL and now should use Mongo.
The architecture is pretty simple
db => case class => rest api
we are using scala jackson for it and everything works fine, except for some minor annoyance and I'm just looking for the right approach.
consider this case class to understand my annoyance
case class Test(val id:String, val value:Any)
the value in our application can be a number or a string at this point, number either an Integer or a Double.
so when we receive JSON like this, from our legacy application:
{ id:"a",value:"test"}
it gets mapped correctly and results in the expected types of String, String.
But assuming the following:
{ id:"b", value:"1"}
we would like to have this mapped instead to String,Integer. But obviously jackson thinks it's a String and maps it to String, String.
Is there some transparent way todo this? My thinking, would be something like an interceptor for jackson would be nice, which simple tests
if type.isDouble == true return double value
else if type.isInteger == true return integer value
else return string value
so we would end up with
String,Double
String,Integer
String,String
in this example.
Obviously I can write a generic parser and bring the dataformat into the correct form beforehand, but I rather would have this done transparency, since we never know when user will submit the legacy JSON format, with this bug and so possibly corrupt the system.
thx
Ok it looks like this is easier than exspected, with writing a customer serializer and annotating our one field which needs it
Solution found here
actual example:
class NumberDeserializer extends JsonDeserializer[Any] {
override def deserialize(jsonParser: JsonParser, deserializationContext: DeserializationContext): Any = {
val jsonNode: JsonNode = jsonParser.getCodec.readTree(jsonParser)
val content = jsonNode.textValue
try {
content.toInt
} catch {
case e: NumberFormatException => try {
content.toDouble
}
catch {
case e2: NumberFormatException => content
}
}
}
}
usage:
case class Test(
#JsonDeserialize(using = classOf[NumberDeserializer])
value: Any
)
Assume I have following DTO:
class C {
String a;
String b;
}
And I have the JSON:
{
"c" : {
"a" : "aaa",
"b" : "bbb"
}
}
What I want to do is, accomplish following test:
C expected = new C("aaa","bbb");
mockMvc.perform(get("url"))
.andExpect(jsonPath("$.c", is(expected)));
It fails. If I first serialize expected to JSON and then try to match, it again fails because it's a string. Is this possible?
Always remember: There is no such thing as a "JSON object". JSON is a serialization format for objects. JSON is always a string. You can convert from object to JSON and back (and hence from object to string and back). But
{ "a": "b" }
is a JavaScript object, not JSON (even if it looks very similar).
This in fact is the answer to your question: When you serialize expected, you get JSON (the transport format, i.e. the string). This isn't what jsonPath() checks. jsonPath() validates against JavaScript types.
This blog post suggests that you need to check each field individually:
.andExpect(jsonPath("$.c.a", is(expected.a)))
.andExpect(jsonPath("$.c.b", is(expected.b)));
which is tedious. What you need is
a) to configure your JSON framework to use a mapping system that sorts keys and
b) you need to figure out what type jsonPath("$.c", ...) returns - it's probably the type which your JSON framework uses to represent generic JavaScript objects.
The check then looks like this:
C c = new C("aaa","bbb");
String serialized = JSON.serialize(c); // to string
JSObject expected = JSON.parse(serialized); // to generic JavaScript object
mockMvc.perform(get("url"))
.andExpect(jsonPath("$.c", is(expected)));
Note that this only works if JSObject has a proper implementation for equals().
If you can afford to modify your "C" class to add it an "equals" operator and to modify slightly your JSON file, I would suggest you to transform your JSON string into an instance of "C". This can be done with a good JSON-ifier (Jackson or GSON). Then you just have to compare the 2 instances.
Some examples with GSON:
class C {
String a;
String b;
public boolean equals(C obj) { return a.equals(obj.a) && b.equals(obj.b); }
}
// Your JSON file should look like that
{
"a" : "aaa",
"b" : "bbb"
}
// So the test is simple
C expected = new C("aaa","bbb");
C json = gson.fromJson(jsonString, C.class);
if (expected.equals(json)) {
// Do whatever you want here
}
If you cannot afford to change the JSON file, just create another class to contains your main class, like this:
class Wrapper {
C c;
}
Wrapper jsonW = gson.fromJson(jsonString, Wrapper.class);
C json = jsonW.c;
...
If you cannot afford the addition of the equals operator, I suggest to create JSON string based on the 2 "C" instance objects and compare the strings. Your jsonString becomes a real "C" object (json) before ending into a new string (jsonStr).
String expectedStr = gson.toJson(expected);
String jsonStr = gson.toJSON(json);
if (expectedStr.equals(jsonStr)) {
// Do whatever you want here
}
I'm building a json object in java. I need to pass a function into my javascript and have it validated with jquery $.isFunction(). The problem I'm encountering is I have to set the function in the json object as a string, but the json object is passing the surrounding quotes along with object resulting in an invalid function. How do I do this without having the quotes appear in the script.
Example Java
JSONObject json = new JSONObject();
json.put("onAdd", "function () {alert(\"Deleted\");}");
Jquery Script
//onAdd output is "function () {alert(\"Deleted\");}"
//needs to be //Output is function () {alert(\"Deleted\");}
//in order for it to be a valid function.
if($.isFunction(onAdd)) {
callback.call(hidden_input,item);
}
Any thoughts?
You can implement the JSONString interface.
import org.json.JSONString;
public class JSONFunction implements JSONString {
private String string;
public JSONFunction(String string) {
this.string = string;
}
#Override
public String toJSONString() {
return string;
}
}
Then, using your example:
JSONObject json = new JSONObject();
json.put("onAdd", new JSONFunction("function () {alert(\"Deleted\");}"));
The output will be:
{"onAdd":function () {alert("Deleted");}}
As previously mentioned, it's invalid JSON, but perhaps works for your need.
You can't. The JSON format doesn't include a function data type. You have to serialise functions to strings if you want to pass them about via JSON.
Running
onAdd = eval(onAdd);
should turn your string into a function, but it's buggy in some browsers.
The workaround in IE is to use
onAdd = eval("[" + onAdd + "]")[0];
See Are eval() and new Function() the same thing?