I'm using a combination of Java Play Framework, MongoDB and Jongo as my go between for a basic web CRUD app. I keep receiving a JSON parse exception even though my string doesn't contain any illegal characters. It's actually failing on closing curly bracket at the end of the statement. Below is my error and code. The query string is just a string builder, searching if an object is empty or has a value, if it has a value it's appended to a string.
Jongo method:
public static Iterable<OneDomain> findWithQueryString(String queryString){
return domains().find("{#}", queryString).as(OneDomain.class);
}
Controller Methods:
String builder example:
if(queryStringBuilder.toString().equalsIgnoreCase("")){
queryStringBuilder.append("date: {$gte : " + searchObj.dateFrom + ", $lt: " + searchObj.dateTo + "}");
}else{
queryStringBuilder.append(" , ");
queryStringBuilder.append("date: {$gte : " + searchObj.dateFrom + ", $lt: " + searchObj.dateTo + "}");
}
String queryString = queryStringBuilder.toString();
Iterable<OneDomain> filteredIterable = OneDomain.findWithQueryString(queryString);
Gives me this error:
Caused by: com.mongodb.util.JSONParseException:
{"source : Online Lists , blacklist : true , vetted : true , is : false , isNot : true"}
^
with the error on the "}" ending it.
In addition to that, if I try to escape the characters by putting in a \" so it becomes \"date\" it will parse and error out like so:
Caused by: com.mongodb.util.JSONParseException:
{"\"source\" : \"Online Lists\" , \"blacklist\" : true , \"vetted\" : true , \"is\" : false , \"isNot\" : true"}
You're building JSON by hand, and doing it wrong. You need to learn the basic JSON syntax requirements
A basic JSON-encoded object is
{"key1":"value1","key2":"value with \" escaped internal quote"}
Note all of the quotes. Your json string is a single very long object key with no associated value, which is not permitted. All keys must have values.
Related
I am trying to evaluate json expression using Josson, but it gives invalid function error
String json ="{\"data\":{\"B\":\"calc(348+(96*$.SelectedPump.stg*$.ModelMaster.count))\"},\"SelectedPump\":{\"stg\":10,\"ab\":200},\"ModelMaster\":{\"count\":20}}";
Josson josson = Josson.fromJsonString(json.toString());
System.out.println(josson.getNode("data.eval(B)"));
Stacktrace:
Exception in thread "main" java.lang.IllegalArgumentException: Invalid function call eval() : Invalid function call calc() : Calc syntax error.
at com.octomix.josson.FuncDispatcher.apply(FuncDispatcher.java:84)
at com.octomix.josson.JossonCore.getPathBySteps(JossonCore.java:328)
at com.octomix.josson.JossonCore.getPathBySteps(JossonCore.java:352)
at com.octomix.josson.JossonCore.getPathBySteps(JossonCore.java:249)
at com.octomix.josson.JossonCore.getPathByExpression(JossonCore.java:211)
at com.octomix.josson.JossonCore.getNodeByExpression(JossonCore.java:147)
at com.octomix.josson.JossonCore.getNodeByExpression(JossonCore.java:142)
at com.octomix.josson.Josson.getNode(Josson.java:279)
The variables inside the math equation cannot contains special characters. You need to use substitution. The variable name can whatever you want without special characters and .. So, the statement of B become...
calc(348+(96*A*B), A:SelectedPump.stg, B:ModelMaster.count)
The function eval() in query data.eval(B) works on node data. So the original evaluation statement needs $. to re-start from the root to get the value. For this revised version eval(data.B), eval() works on the root node with parameter data.B. And therefore, $. is not necessary.
String json =
"{\n" +
" \"data\": {\n" +
" \"B\": \"calc(348+(96*A*B), A:SelectedPump.stg, B:ModelMaster.count)\"\n" +
" },\n" +
" \"SelectedPump\": {\n" +
" \"stg\": 10,\n" +
" \"ab\": 200\n" +
" },\n" +
" \"ModelMaster\": {\n" +
" \"count\": 20\n" +
" }\n" +
"}";
Josson josson = Josson.fromJsonString(json);
System.out.println(josson.getNode("eval(data.B)"));
// Output: 19548.0
Answer by Raymond Choi would be the best solution.
Alright, I have found a possible solution to this using Jossons template engine.
String json = "{\"data\":{\"B\":\"calc(348+(96*{{SelectedPump->stg}}*{{ModelMaster->count}}))\"},\"SelectedPump\":{\"stg\":10,\"ab\":200},\"ModelMaster\":{\"count\":20}}";
Jossons jossons = Jossons.fromJsonString(json);
String output = jossons.fillInPlaceholder("{\"s\":\"{{data->B}}\"}");
output = jossons.fillInPlaceholder(output);
System.out.println(output);
System.out.println(Josson.fromJsonString(output).getNode("$.eval(s)"));
It is necessary to adjust the JSON string to use placeholder to access the values from a different/nested node. We also have to create a new JSON string with a template that is filled in. Also also, it does sadly not recursively replace the placeholder, making it necessary to call jossons.fillInPlaceholder() two times. The first time it fetches the calculated formula and the second time it replaces the placeholder that are now present in the formula.
The output is:
output -> {"s":"calc(348+(96*10*20))"}
Josson.fromJsonString(output).getNode("$.eval(s)") -> 19548.0
The proposed code in the question fails, because Josson is not able to resolve nested/different node values. Pretty much as soon as we have a . present in the calc() formula, the syntax check fails. It is however possible to access nodes at the same level. So a JSON like this:
{
"data" : {
"B" : "calc(348+(96*SelectedPumpStg*ModelMasterCount))",
"SelectedPumpStg" : 10,
"ModelMasterCount" : 20
}
}
can be evaluated by:
String json = "{\"data\":{\"B\":\"calc(348+(96*SelectedPumpStg*ModelMasterCount))\",\"SelectedPumpStg\":10,\"ModelMasterCount\":20}}";
Josson josson = Josson.fromJsonString(json.toString());
System.out.println(josson.getNode("data.eval(B)"));
resulting in the same: 19548.0.
Either way, the input JSON has to be adjusted to work with Josson
I am trying to bind JSON to Java POJO class using com.google.gson.Gson
like this :
MyClass data = gson.fromJson(jsonString, MyClass.class);
When I am using below mentioned it's working fine
{
"data": "{\"key1\":{\"key11\":\"192.192.1.192\",\"key12\":\"WEB\"}}"
}
However, when below-mentioned data is used, I am getting MalformedJsonException
{
"data": "{"key1":{"key11":"192.168.1.158","key12":"WEB"}}"
}
Log :
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException:
Unterminated object at line 1 column 354 path
You can not use " within a String to quote your JSON keys and values. You either have to escape them (like you did in your first example) or you have use single quotes '.
You are effectively trying to do String concatenation without using +.
This looks for the compiler like a list of Strings with variables in between:
"{"key1":{"key11":"192.168.1.158","key12":"WEB"}}"
The compiler would expect something like this:
"{" + key1 + ":{" + key11 + ":" + 192.168.1.158 + "," + key12 + ":" + WEB + "}}";
If you look at the String this way you immediately see the problem. That's why you should either escape the quotes or use single quotes:
"{\"key1\":{\"key11\":\"192.168.1.158\",\"key12\":\"WEB\"}}"
"{'key1':{'key11':'192.168.1.158','key12':'WEB'}}"
I've been spending the last couple of days trying to format a JSON string into a JSON object, but this is not quite the usual json string. Im receiving it from an url and have no problem converting it into a string but when im going to create the JSON object Ive found a whole lot of issues.
The JSON looks like this:
header {
gtfs_realtime_version: "1.0"
incrementality: FULL_DATASET
timestamp: 1511789066
}
entity {
id: "294-131-39-562-4732025"
vehicle {
trip {
trip_id: "131-39-562-4732025"
route_id: "131"
}
position {
latitude: 3.44351
longitude: -76.52622
}
timestamp: 1511789065
stop_id: "501450"
vehicle {
id: "21002"
label: "MC21002"
license_plate: "VCQ452"
}
}
}
entity {
id: "1087-431-55-35-4732025"
vehicle {
trip {
trip_id: "431-55-35-4732025"
route_id: "431"
}
position {
latitude: 3.3767517
longitude: -76.54276
}
timestamp: 1511789065
stop_id: "502150"
vehicle {
id: "31038"
label: "MC31038"
}
}
}
as you can see there is no array, nor commas separating the element that I need to process (entity), I cant even get the JSONObject from the root.
Any suggestions?
This is the last block of code I've tried in which I remove the header, insert all the entity objects into the array buses, and separate them by commas, but is no use
//getRespuesta(); gives me the string
//Log.d("LIVE", "RAW JSon: " + getRespuesta());
String[] parts = getRespuesta().split("[}]", 2);
String[] splitter = parts[1].split("(?=entity)");
String finalString = splitter[1];
for (int t = 2; t < splitter.length; t++) {
finalString = finalString + ",\n" + splitter[t];
}
Log.d("LIVEL", "SIZE: " + splitter[1]);
String JSonChunk = "{ \n \"buses\": [ \n " + finalString + "\n ] \n }";
//String JSonChunk = "{ \n " + parts[1] + "\n }";
try {
// Log.d("LIVE", "CHUNKED JSon: " + JSonChunk);
JSONObject json = new JSONObject(JSonChunk);
}catch........
As was mentioned in the comments and as you've seen you can't use a JSON parser on non-JSON. I'd take the following approach:
If this meets the specifications of a known format, someone has probably written a JSON converter you can run it through.
If it is a custom format but each line is a predictable format and does not rely on anything before or after, you can run each line through a custom converter and give it the 'JSON-ness.' (add commas, etc) Process each line at at time rather than trying to split up the whole thing at once.
If none of the above are true, you need to go into regular expression hell. Sorry :(
From my point of view you must transform it to a valid JSON String
You can try this regex:
https://regex101.com/r/15dHwd/1
That means, for each group that capture tou replace it with ": {"
That way: "position {" will re replaced with "position: {"
Now you have other issue,
"entity" appears twice
If you are 100% sure that you will get always that format, after the first replace:
Start with appending "{" on the beginning of the string
Get a regex to find the "entity" String
Replace the first match with "array: [ entity"
List item
On the end of the string append: "]}"
I have a string output which I need to match and I am using a regex
String schemaName = "Amazon";
String test = "{\"data\": [], \"name\": \"Amazon\", \"title\": \"StoreDataConfig\"}";
String output= method("\\[\\]",schemaName);
Matcher n = Pattern.compile(output).matcher(test);
boolean available = n.find();
System.out.println(available);
I wanted to validate the same and passing the regex to a method as mentioned
private static String method(String data, String schemaName) throws IOException {
System.out.println(data);
return ("{\"data\": " + data + ", \"name\": " + "\"" + schemaName + "\"" + ", \"title\": \"StoreDataConfig\"}");
}
But I am always getting java.util.regex.PatternSyntaxException: Illegal repetition.
Can you let me know what is the mistake?
If I don't use a method for [] and just giving it directly, I am not getting an error
It looks like you are doing this:
Take a valid regex for matching [].
Embed the regex in some JSON
Attempt to compile the JSON-with-an-embedded-regex as if the whole lot was a valid regex.
That fails ... because the JSON-with-an-embedded-regex is not a valid regex.
For a start, the { character is a regex meta character.
But the real puzzle is .... what are you actually trying to do here?
If you simply want a regex that matches a literal string then this will do it.
Pattern p = Pattern.compile(Pattern.quote(someLiteralString)).
And you could build a regex out of sub-regexes and literal strings by using Pattern.quote to escape the literal parts and then concatenating.
If what you are ultimately trying to do here is to extract information from a JSON string using pattern matching / regexes, then ... don't. The better approach is to use a proper JSON parser, and extract the information you need from the JSON object tree.
It's because you need to escape {} characters like this "\\{"
I am trying to use ESAPI Encoder to identify and canonicalize URL-encoded query parameters. It sort of works, but not in the way the API seems to indicate. Here is my class, and below is the output it generates:
CODE
package test.test;
import org.owasp.esapi.ESAPI;
import org.owasp.esapi.Validator;
import org.owasp.esapi.errors.EncodingException;
import org.owasp.esapi.errors.IntrusionException;
import org.owasp.esapi.errors.ValidationException;
public class ESAPITester {
public static void main(String argsp[]) throws ValidationException,
IntrusionException, EncodingException {
String searchString = "-/+=_ !$*?#";
String singleEncoded = ESAPI.encoder().encodeForURL(searchString);
String doubleEncoded = ESAPI.encoder().encodeForURL(singleEncoded);
Validator validator = ESAPI.validator();
System.out.println("Searched : " + searchString);
System.out.println("Single encoded : " + singleEncoded);
System.out.println("Double encoded : " + doubleEncoded);
System.out.println("Decode from URL : " + ESAPI.encoder().decodeFromURL(singleEncoded));
System.out.println("Canonicalized : " + ESAPI.encoder().canonicalize(singleEncoded));
System.out.println("Valid input : " + validator.getValidInput("http",
searchString, "HTTPParameterValue", 100, true, true));
System.out.println("Valid from Encoded : " + validator.getValidInput("http",
singleEncoded, "HTTPParameterValue", 100, true, true));
}
}
OUTPUT
Searched : -/+=_ !$*?#
Single encoded : -%2F%2B%3D_+%21%24*%3F%40
Double encoded : -%252F%252B%253D_%2B%2521%2524*%253F%2540
Decode from URL : -/ =_ !$*?#
Canonicalized : -/+=_+!$*?#
Valid input : -/+=_ !$*?#
log4j:WARN No appenders could be found for logger (IntrusionDetector).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.owasp.esapi.errors.ValidationException: http: Invalid input. Please conform to regex ^[\p{L}\p{N}.\-/+=_ !$*?#]{0,1000}$ with a maximum length of 100
at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:144)
at org.owasp.esapi.reference.validation.StringValidationRule.checkWhitelist(StringValidationRule.java:160)
at org.owasp.esapi.reference.validation.StringValidationRule.getValid(StringValidationRule.java:284)
at org.owasp.esapi.reference.DefaultValidator.getValidInput(DefaultValidator.java:214)
at test.test.ESAPITester.main(ESAPITester.java:25)
My question is: Why does the getValidInput() not canonicalize the URL-encoded input parameter? I'm curious as to why the canonicalize() method does so, but getValidInput() with the final argument ('canonicalize') set to true doesn't.
So the question becomes:
why the 2nd validator.getValidInput() call throws an exception, when
all it is expected to do is to canonicalize the input and validate
that it matches the expected value. In other words, the direct call to
canonicalize() works, but the call to getValidInput() fails.
Something is very wrong here. In the version of HTTPParameterValue that you get from the OWASP source repo, the regex is ^[a-zA-Z0-9.\\-\\/+=#_ ]*$ Someone has manipulated the HTTPParameterValue to look more like SafeString: ^[\\s\\p{L}\\p{N}.]{0,1024}$
See line 440.
This is wrong. Changing default ESAPI values shouldn't be done, if you need custom changes, write a brand new validator.properties entry using the established pattern.
Your test will still fail however, because the string decodes to -/+=_ !$*?# and ? is a reserved character within http queries.
From an earlier spec:
3.4. Query Component
The query component is a string of information to be interpreted by
the resource.
query = *uric
Within a query component, the characters ";", "/", "?", ":", "#",
"&", "=", "+", ",", and "$" are reserved.
As to why the input fails according to the regex you're running at, ^[\\p{L}\\p{N}.\\-/+=_ !$*?#]{0,1000}$, read the code. At line 266 you'll see the affected method.
Here's what you want to look at:
public String getValid( String context, String input ) throws ValidationException
{
String data = null;
// checks on input itself
// check for empty/null
if(checkEmpty(context, input) == null)
return null;
if (validateInputAndCanonical)
{
//first validate pre-canonicalized data
// check length
checkLength(context, input);
// check whitelist patterns
checkWhitelist(context, input);
// check blacklist patterns
checkBlacklist(context, input);
// canonicalize
data = encoder.canonicalize( input );
} else {
//skip canonicalization
data = input;
}
// check for empty/null
if(checkEmpty(context, data, input) == null)
return null;
// check length
checkLength(context, data, input);
// check whitelist patterns
checkWhitelist(context, data, input);
// check blacklist patterns
checkBlacklist(context, data, input);
// validation passed
return data;
The regex gets checked before it even attempts to canonicalize your input.