I wish to translate an OData query into a MongoDB query. I have the olingo-odata4 code and have isolated the calls necessary to take an input string e.g.
Name eq 'Buzz' and (amount gt 1.99 or size lt 10)
and run it through ExpressionParser with my own ExpressionVisitor implementation to intercept the various parse states like visitLiteral and visitBinaryOperator, etc. From there it is trivial to construct the equivalent MongoDB query. My main is essentially a trimmed down copy of the odata test drivers, including the mocking of the Edm class for startup:
ExpressionParser ep = new ExpressionParser(mock(Edm.class), odata);
My challenge is that I cannot seem to set up the environment properly for the parser to recognize fields, e.g. members. I would have hoped that Name eq 'Buzz' would cause the visitMember method to be called for Name but it does not; instead, I get a parse fail with the message: Property paths must follow a structured type.
So my overall question becomes: If the $filter argument to OData shows the expression syntax as Name eq 'Buzz' then what has to happen in the server implementation to recognize Name as a member?
The parser is unfortunately too closely coupled to the data model, IMHO. The test drivers I looked at were mostly {5 eq 5} and things like that so I was led astray. The property path test driver, however, cannot mock the EDM; it must construct a model in order to declare the type (string, etc.) of the fields. This is why visitMember Member class has the getType() method. But this means you cannot use the parser to just crack the syntax and dynamically perform actions on the results.
I ended up just creating a very small grammar in ANTLR to parse the OData query expressions and it was easy to convert to MongoDB query language.
Related
In the following expression:
T(org.apache.commons.io.IOUtils).toString(T(java.lang.Runtime)
.getRuntime().exec(T(java.lang.Character).toString(105)
.concat(T(java.lang.Character).toString(100))).getInputStream())
Does the '105' in toString(105) refer to an itemized object within the Character class?
and
Why is the 'T', which I believe expresses a generic type, and is used 4 times in this expression, a necessary feature of Java?
The toString() method that seems to be invoked here is actually the toString(char) (static) method of java.lang.Character. Quoting the documentation:
public static String toString(char c)
Returns a String object representing the specified char.
The result is a string of length 1 consisting solely of the specified char.
Parameters:
c - the char to be converted
Returns:
the string representation of the specified char
Since:
1.4
Note that 100 and 105 are also valid char values where 100 == 'd' and 105 == 'i'.
Update: after knowing the context, I am now confident that this code is intended to be injected into a template for a web page. The template engine used provides special syntax for accessing static methods where T(Classname) resolves to just Classname (not Classname.class!) in the resulting Java code.
So your code would be translated to:
org.apache.commons.io.IOUtils.toString(java.lang.Runtime
.getRuntime().exec(java.lang.Character.toString(105)
.concat(java.lang.Character.toString(100))).getInputStream())
The full qualification of the class names is necessary because we do not know if those classes are imported on the attacked site (or if the template engine even allows imports or class names must always be fully qualified).
A more readable version of the code that assumes imports is
IOUtils.toString(
Runtime.getRuntime().exec(
Character.toString(105).concat(Character.toString(100))
).getInputStream()
)
And after a little de-obfuscation...
IOUtils.toString(Runtime.getRuntime().exec("id").getInputStream())
Whatever this is, it is definitely NOT meaningful Java code.
And the fact that you can provide it as as a search query on some site is not evidence that it is Java either.
I suspect that this is actually some custom (site-specific?) query language. That makes it futile to try to understand it as a Java snippet.
Your theory that T could denote a generic type parameter doesn't work. Java would not allow you to write T(...) if that was the case.
Furthermore, if we assume that org.apache.commons.io.IOUtils, java.lang.Runtime and so on are intended to refer to Java class objects, then the correct Java syntax would be org.apache.commons.io.IOUtils.class, java.lang.Runtime.class and so on.
So what does it mean?
Well a bit of Googling found me some other examples that look like yours. For instance;
https://github.com/VikasVarshney/ssti-payload
appears to generate "code" that is reminiscent of your example. This is SSTI - Server Side Template Injection, and it appears to be targeting Java EE Expression Language (EL).
And I think this particular example is an attempt to run the Linux id program ... which would output some basic information about the user and group ids for the account running your web server.
Does it matter? Well only if your site is vulnerable to SSTI attacks!
How would you know if your site is vulnerable?
By understanding the nature of SSTI with respect to EL and other potential attack vectors ... and auditing your codebase and configurations.
By using a vulnerability scanner to test your site and/or your code-base.
By employing the services of a trustworthy IT security company to do some penetration testing.
In this case, you could also try to use curl to repeat the attempted attack ... as the hacker would have done ... based on what is in your logs. Just see if it actually works. Note that running the id program does no actual damage to your system. The harm would be in the information that is leaked to a hacker ... if they succeed.
Note that if this hack did succeed, then the hacker would probably try to run other programs. These could do some damage to your system, depending on how how well your server was hardened against such things.
In a huge project with tens of thousands of Java files there are a couple of Java classes where developers may pass in strings as parameters to a constructor class I had implemented
public byte[] getProductReport(List<String> products, Date from, Date to) {
// ... do some stuff before...
List<ReportParameterDto> reportParameters = new ArrayList<>();
reportParameters.add(new ReportParameterDto("From (YYYY.MM.DD)", ParameterType.DATE, from));
reportParameters.add(new ReportParameterDto("To_(YYYY.MM.DD)", ParameterType.DATE, to));
reportParameters.add(new ReportParameterDto("Products", ParameterType.SELECT, someList));
return ReportFromCRServerHelper.downloadReport("ProductReporot", reportParameters, ReportFormat.PDF);
}
If a developer uses wrong string values downloading a requested report (from a remote Report server) will fail during runtime.
In this example I would like to have some validation checking - during compilation - in order to avoid these errors before they are found by the customer.
I have API methods to obtain parameter values from a report which I hope to use
during compilation of the above method.
In my example the compilation should fail and throw an error highlighting how parameters should look instead:
"From (JJJJ-MM)" is invalid --> should be "From_(JJJJ-MM)"
"Products" is invalid --> should be "PRODUCT_LIST"
Can I detect these parameters (used in above ReportParameterDto constructors) through JAVAX annotation processing?
The few tutorials / blogs that I found dealt with validating parameters in method signatures, not the values passed into methods.
Or are there a more elegant tools available?
A compile-time tool like the Checker Framework can validate the arguments to a method or constructor. You annotate the parameter types to indicate the permitted values, and then when javac runs it issues a warning if an argument may not be compatible with the parameter.
If there is a limited number of possible values, then you can use the Fake Enum Checker to treat strings or integers as enumerated values. (Using a Java enum is also a good idea, but may not be possible or convenient because of other code that expects a string or integer.)
If the number of possible values is unlimited, then you can use the Constant Value Checker -- for example, to supply a regular expression that any constant string argument must satisfy.
You can also define your own compile-time validation if you want different functionality than is available in the checkers that are distributed with the Checker Framework.
There is a table in PostgreSQL with ltree field. How to update this field with string value?
DSL.using(configuration)
.update(AREAS)
.set(AREAS.TREE, area.getTree());//getTree() return String
I tried several variants:
val(area.getTree()).cast(field("ltree")) throwing
A org.jooq.exception.DataAccessException has been caught,
SQL [update "public"."areas" set "tree" = cast(? as any) where "public"."areas"."id" = ?];
ERROR: syntax error at or near "any"*
inline(String.format("CAST(\'%s\' AS ltree)", area.getTree())) using excess quotes
A org.jooq.exception.DataAccessException has been caught,
SQL [update "public"."areas" set "tree" = 'CAST(''1.35.1284739'' AS ltree)' where "public"."areas"."id" = ?];
ERROR: syntax error at position 4*
val(field("CAST({0} AS ltree)", area.getTree())) throwing
A org.jooq.exception.SQLDialectNotSupportedException has been caught,
Type class org.jooq.impl.SQLField is not supported in dialect DEFAULT
Thorough solution
In the long run, the best approach to add support for vendor-specific data types is to specify a custom data type binding for it, which allows you to define how jOOQ should serialise / deserialise the type to the JDBC API, including the cast that you might need.
A quick win might be these:
field("CAST(? as ltree)", area.getTree());
field("CAST({0} as ltree)", val(area.getTree()));
This is why your various attempts didn't work:
val(area.getTree()).cast(field("ltree"))
That approach seems to make sense at first, as you're using jOOQ's built-in cast() support. However, your field("ltree") expression models a field named ltree of an unknown type, so when you pass that to cast(), jOOQ doesn't know what to cast the val to, thus: any
inline(String.format("CAST(\'%s\' AS ltree)", area.getTree()))
This doesn't really make sense in your context, because DSL.inline() creates an inline bind variable, or constant, or a string literal (all different names for the same concept). You don't want a string literal 'CAST(... AS ltree)', you want the expression CAST(... AS ltree).
val(field("CAST({0} AS ltree)", area.getTree()))
That's similar to the above, although you were close here. The inner field(...) expression creates a SQL template, which is what you wanted. But then you wrapped that in a bind variable using DSL.val(), which doesn't really make sense for two reasons:
You don't want your cast expression to be a bind variable
The cast expression (of type Field) is not a valid type to create a bind variable from. jOOQ doesn't know how to bind a value of type Field
I have a filtering application written in Java that will allow me to filter companies based on their fundamentals (e.g. pretax profit, dividend yield etc.).
I have created a filtering engine that, at the moment is hard-coded to take filters that have been given at compile time.
What I want to do is open up a web service where I can pass in JSON to run the filtering using filters defined and passed in at runtime.
To do this I need to somehow convert strings into Java code.
So for example the following string in JSON:
current.preTaxProfit > previous.preTaxProfit
into Java code like this:
return current.getPreTaxProfit() > previous.getPreTaxProfit();
The main issue I am having is parsing mathematical strings such as:
current.preTaxProfit > (previous.preTaxProfit * 1.10)
Things could get very complex very quickly, especially when it comes to inner brackets and such, and adhering to BODMAS.
Are there any libraries out there specifically for parsing mathematical strings, or does anyone know any good resources that could help me?
For example, I have found this:
Javaassist: http://davidwinterfeldt.blogspot.co.uk/2009/02/genearting-bytecode.html
Does anyone have any experience of using Javaassist? In my architecture I pass in objects ith 'callback' methods implemented, so the ability to create entire classes and methods could useful.
Two possibilities:
the Java Scripting API:
the EL, Expression Language.
The scripting API might be most fast to get results from, but EL is nicer.
Consider using of an expression language, for instance JEXL.
If you put current and previous into the context, just evaluate:
current.preTaxProfit > (previous.preTaxProfit * 1.10)
Complete example:
// create context
JexlContext jc = new MapContext();
context.set("current", someObject);
context.set("previous", anotherObject);
// create expression
String jexlExp = "current.preTaxProfit > (previous.preTaxProfit * 1.10)";
Expression e = jexl.createExpression(jexlExp);
// evaluate
Object result = e.evaluate(jc);
I want an expression language that runs on the JVM and includes support for
math expressions, including operator priority
string expressions, like substring, etc
supports named functions
this allows me to decorate and control exactly who and what functions can be executed.
read/write variables that are "typeless" / allow type conversion in a controlled manner.
does not allow arbitary java scriptlets.
it should not be possible to include constructs like new Someclass()
cannot execute arbitrary static or otherwise method
does not allow any OGNL like expressions.
I only want to functions I map to be available.
support for control constructs like if this then that is for the moment optional.
must be embeddable.
This previous stackoverflow question is similar, but:
does not really answer "how" or "what" as does the above,
allows java object expressions, throwing an exception from a SecurityManager to stop method execution, which is nasty and wrong.
java object like expressions should be an error at parse time.
jexel seem to be closest possible match, but License is a bit horrible (GPL/Commercial).
If you only want the scripts to output text, then Apache Velocity fit's your constraints quite well. It runs in an environment where it only has access to the objects you give it, but can do things like basic math.
The Apache license is a bit friendlier than GPL too.