Develop Reference

How to extract used table from SelectConditionStep<Record>

I'm extending on my last question I asked about jOOQ. In the Hibernate models the #Filter annotation gets used, and I want to apply this same 'default filter' to the jOOQ queries. As I'm passing a jOOQ query to the nativeQuery(org.jooq.Query query, Class<E> type) I was wondering if it's possible to extract the table (TableImpl<?,?>) used from the FROM clause in the jOOQ query (org.jooq.Query).
This is what I've tried:
private static <E> SelectConditionStep<Record> applyDefaultFilters(Class<E> type, SelectConditionStep<Record> query)
{
if (BaseOrganizationModel.class.isAssignableFrom(type)) {
query
.getQuery()
.addConditions(
query
.getQuery()
.asTable()
.field("organization_id", Long.class)
.eq(currentOrganization().id));
if (SoftDeletableModel.class.isAssignableFrom(type)) {
query
.getQuery()
.addConditions(query.getQuery().asTable().field("deleted", Boolean.class).eq(false));
}
}
return query;
}
The result is this SQL, which is not what I want. I want it to filter the corresponding table.
select distinct `EventGroup`.*
from `EventGroup`
where (
...
and `alias_100341773`.`organization_id` = ?
and `alias_17045196`.`deleted` = ?
)
I want this
select distinct `EventGroup`.*
from `EventGroup`
where (
...
and `EventGroup`.`organization_id` = ?
and `EventGroup`.`deleted` = ?
)
Is this possible at all? And if not, what possible other routes are there? (aside from the obvious passing the table to the function)

Using jOOQ 3.16 query object model API
jOOQ 3.16 introduced a new, experimental (as of 3.16) query object model API, which can be traversed.
On any Select, just call Select.$from() to access an unmodifiable view of the contained table list.
An alternative, dynamic SQL approach for the ad-hoc case
Every time you're trying to mutate an existing query, ask yourself, is there a more elegant way to do this using a more functional, immutable approach do dynamic SQL? Rather than appending your additional predicates to the query, why not produce predicates from a function?
private static Condition defaultFilters(Class<?> type, Table<?> table) {
Condition result = noCondition();
if (BaseOrganizationModel.class.isAssignableFrom(type)) {
result = result.and(table.field("organization_id", Long.class)
.eq(currentOrganization().id));
if (SoftDeletableModel.class.isAssignableFrom(type))
result = result.and(not(table.field("deleted", Boolean.class)))
}
return result;
}
And now, when you construct your query, you can add the filters:
ctx.select(T.A, T.B)
.from(T)
.where(T.X.eq(1))
.and(defaultFilters(myType, T))
.fetch();
A generic way to transform your SQL
If you really want to mutate your query (e.g. in a utility for all queries), then a transformation approach might be better suited. There are different ways to approach this.
Using views
Some RDBMS can access session variables in views. In Oracle, you'd be setting some SYS_CONTEXT variable to your organization_id inside of a view, and then query only the (possibly updatable) views instead of the tables directly. MySQL unfortunately can't do the equivalent thing, see Is there any equivalent to ORACLE SYS_CONTEXT('USERENV', 'OS_USER') in MYSQL?
I've described this approach here in this blog post. The advantage of this approach is that you will never forget to set the predicate (you can validate your view source code with CI/CD tests), and if you ever forget to set the session context variable, the view will just not return any data, so it's quite a secure approach.
Together with the WITH CHECK OPTION clause, you can even prevent insertions into wrong organization_id, which improves security.
Using a VisitListener in jOOQ
This is the most powerful approach to do this in jOOQ, and exactly what you want, but also quite a tricky one to get right for all edge cases. See this post about implementing row level security in jOOQ. Starting from jOOQ 3.16, there will be better ways to transform your SQL via https://github.com/jOOQ/jOOQ/issues/12425.
Note, it won't work for plain SQL templates that do not use any jOOQ query parts, nor for JDBC based queries or other queries that you may have in your system, so be careful with this approach as you might leak data from other organisations.
Of course, you could implement this step also on the JDBC layer, using jOOQ's ParsingConnection or ParsingDataSource, that way you can intercept also third party SQL and append your predicates.
This can work for all DML statements, including UPDATE, DELETE. It's a bit harder for INSERT, as you'd have to transform INSERT .. VALUES into INSERT .. SELECT, or throw an exception if someone wants to insert into the wrong organization_id.
Using a ExecuteListener in jOOQ
A bit more hackish than the above VisitListener approach, but generally easier to get right, just regex-replace the WHERE clause of all your statements by WHERE organization_id = ... AND in an ExecuteListener.
To play it safe, you could reject all queries without a WHERE clause, or do some additional trickery to add the WHERE clause at the right place in case there isn't already one.
Using jOOQ's equivalent of Hibernate's #Filter
jOOQ's equivalent of Hibernate's #Filter is the Table.where(Condition) clause. It's not an exact equivalent, you'd have to prevent direct access to T in your code base and make sure users access T only via a method that replaces T by T.where(defaultFilters(myType, T)) instead.
This approach currently loses type safety of the T table, see: https://github.com/jOOQ/jOOQ/issues/8012

Row Level Security implementation in JOOQ

I want to implement oracle row level security kind of feature in Java using JOOQ library
Here is an example JOOQ query code:
Result<Record> result = dslContext.select().from(Employee.EMPLOYEE).fetch();
The code above will generate SQL as below:
select [dbo].[Employee].Id,... from [dbo].[Employee]
I want to add a where clause to filter data specific to user security as below:
select [dbo].[Employee].Id,... from [dbo].[Employee] WHERE [dbo].[Employee].Security IN (1,2)

Explicit predicates
Unless I'm missing some nice SQL Server feature where rows / records contain a pseudo-column called .Security to implement row level security, you should be able to simply write
dslContext.select()
.from(EMPLOYEE)
.where(EMPLOYEE.SECURITY.in(1, 2))
.fetch();
For more info about jOOQ predicate building, see the manual here:
http://www.jooq.org/doc/latest/manual/sql-building/conditional-expressions
And in particular, the IN predicate:
http://www.jooq.org/doc/latest/manual/sql-building/conditional-expressions/in-predicate
General solution using jOOQ's ExecuteListener
Given your comments, you're looking for a general way to patch all SQL statements with an additional predicate, no matter what the particular programmer is typing.
You can do this with jOOQ, but beware of the fact that this will just help you enforcing the predicate, not guarantee it, if programmers bypass jOOQ. What you can do is set up an ExecuteListener, intercepting the renderStart() event in order to patch / replace the query being executed. Something along these lines:
#Override
public void renderStart(ExecuteContext ctx) {
if (ctx.query() instanceof Select) {
// Operate on jOOQ's internal query model
SelectQuery<?> select = null;
// Check if the query was constructed using the "model" API
if (ctx.query() instanceof SelectQuery) {
select = (SelectQuery<?>) ctx.query();
}
// Check if the query was constructed using the DSL API
else if (ctx.query() instanceof SelectFinalStep) {
select = ((SelectFinalStep<?>) ctx.query()).getQuery();
}
if (select != null) {
// Use a more appropriate predicate expression
// to form more generic predicates which work on all tables
select.addConditions(EMPLOYEE.SECURITY.in(1, 2));
}
}
}
Of course, there's room for improvement to the above. Feel free to discuss use-cases on the user group
General solution using jOOQ's VisitListener
If you're willing to go deep into jOOQ's internals, you an also try to implement a VisitListener and actually transform jOOQ's AST representation of your query. This is documented here:
http://www.jooq.org/doc/latest/manual/sql-building/queryparts/custom-sql-transformation/
General solution using views
While the above works, I would personally suggest you use views for this and hide the actual tables from developers. Example:
CREATE VIEW v_employee AS
SELECT a, b, c, ...
FROM t_employee
WHERE t_employee.security in (1, 2)
With appropriate grants, you can hide the tables from the developers, making sure they will only ever use the views with your desired predicate always in place

Inject attribute into JPQL SELECT clause

Let's depict the following use case: I have a JPQL Query which on the fly creates data objects using the new keyword. In the SELECT clause I would like to inject an attribute which is not known to the database but to the layer which queries it.
This could look like
EntityManager em; // Got it from somewhere
boolean editable = false; // Value might change, e.g. depending on current date
Query q = em.createQuery("SELECT new foo.bar.MyDTO(o, :editable) FROM MyObject o")
.setParameter("editable", editable);
List<MyDTO> results = (List<MyDTO>) q.getResultList();
Any ideas how this kind of attribute or parameter injection into the SELECT clause might work in JPQL? Both JPA and JPA 2.0 solutions are applicable.
Edit: Performance does not play a key role, but clarity and cleanness of code.

Have you measured a performance problem when simply iterating over the list of results and call a setter on each of the elements. I would guess that compared to
the time it takes to execute the query over the database (inter-process call, network communication)
the time it takes to transform each row into a MyObject instance using reflection
the time it takes to transform each MyObject instance into a MyDTO using reflection
your loop will be very fast.
If you're so concerned about performance, you should construct your MyDTO instances manually from the returned MyObject instances instead of relying on Hibernate and reflection to do it.
Keep is simple, safe, readable and maintainable first. Then, if you have a performance problem, measure to detect where it comes from. Then and only then, optimize.

It will not work without possible vendor extensions, because according specification:
4.6.4 Input Parameters
...
Input parameters can only be used in the
WHERE clause or HAVING clause of a query.

How to reuse a Criteria object with hibernate?

I'm trying to do query result pagination with hibernate and displaytag, and Hibernate DetachedCriteria objects are doing their best to stand in the way. Let me explain...
The easiest way to do pagination with displaytag seems to be implementing the PaginatedList interface that has, among others, the following methods:
/* Gets the total number of results. */
int getFullListSize();
/* Gets the current page of results. */
List getList();
/* Gets the page size. */
int getObjectsPerPage();
/* Gets the current page number. */
int getPageNumber();
/* Get the sorting column and direction */
String getSortCriterion();
SortOrderEnum getSortDirection();
I'm thinking of throwing my PaginatedList implementation a Criteria object and let it work along theese lines...
getFullListSize() {
criteria.setProjection(Projections.rowCount());
return ((Long) criteria.uniqueResult()).intValue();
}
getList() {
if (getSortDirection() == SortOrderEnum.ASCENDING) {
criteria.addOrder(Order.asc(getSortCriterion());
} else if (getSortDirection() == SortOrderEnum.DECENDING) {
criteria.addOrder(Order.desc(getSortCriterion());
}
return criteria.list((getPageNumber() - 1) * getObjectsPerPage(),
getObjectsPerPage());
}
But this doesn't work, because the addOrder() or the setProjection() calls modify the criteria object rendering it in-usable for the successive calls. I'm not entirely sure of the order of the calls, but the db throws an error on getFullListSize() trying to do a "select count(*) ... order by ..." which is obviously wrong.
I think I could fix this by creating an object of my own to keep track of query conditions and rebuilding the Criteria object for each call, but that feels like reinventing yet another wheel. Is there a smarter way, possibly copying the Criteria initially passed in and working on that copy?
Update:
It looks like getList is called first, and getFullListSize is called multiple times after, so, as soon as there's an ordering passed in, getFullListSize will fail. It would make sense to hit the db only once (in getList I'd say) and cache the results, with no need to copy/reset the Criteria object, but still...
Update (again):
Forget about that, once I've done the count I can't do the select, and vice versa. I really need two distinct Criteria objects.

Criteria.setProjection(null);
Criteria.setResultTransformer(Criteria.ROOT_ENTITY);
Will effectively "reset" the criteria between the rowCount projection and execution of the criteria itself.
I would make sure your Order hasn't been added before doing the rowCount, it'll slow things down. My implementation of PaginatedList ALWAYS runs a count query before looking for results, so ordering isn't an issue.

well, DetachedCriteria are Serializable, so you have built-in (if inelegant) deep clone support. You could serialize the initial criteria to a byte[] once on construction, then deserialize it each time you want to use it.

http://weblogs.asp.net/stefansedich/archive/2008/10/03/paging-with-nhibernate-using-a-custom-extension-method-to-make-it-easier.aspx
In that post I spotted a CriteriaTransformer.clone method.
That should copy the criteria object.
You can also set the projection on your getlist method.
Woops I didn't notice you were referring to java hibernate. Anyway, this http://forum.hibernate.org/viewtopic.php?t=939039
forum post should be able to answer your question.

Ugly as it may be I ended up using the serialization trick. I just serialize the DetachedCriteria object to a byte array on construction of the PaginatedList object and de-serialize it when needed. Ouch.

Another thing worth trying:
implement a generic DAO like the one suggested on hibernate's site and pass it to the PaginatedList object, along with a Restrictions object. The PaginatedList object would then do something like
Criteria.forClass(myDAO.getPersistentClass())
.add(myRestrictions)
.addOrder(<someOrder>)
and
Criteria.forClass(myDAO.getPersistentClass())
.add(myRestrictions)
.setProjection(Projections.rowCount());
Haven't tried that yet, but it should work.

There is a better and easy way to clone criteria, just simply:
ICriteria criteria = ...(your original criteria init here)...;
var criteriaClone = (ICriteria)criteria.Clone();
And getting back to Your problem. For pagination I've made a method, which gives me as a result:
1. Total rows count
2. Rows filtered by page & pageSize
In a single query to DB.
ICriteria criteria = ...(your original criteria init here)...;
var countCrit = (ICriteria)criteria.Clone();
countCrit.ClearOrders(); // avoid missing group by exceptions
var rowCount = countCrit
.SetProjection(Projections.RowCount()).FutureValue<Int32>();
var results = criteria
.SetFirstResult(pageIndex * pageSize)
.SetMaxResults(pageSize)
.Future<T>();
var resultsArray = results.GetEnumerable();
var totalCount = rowCount.Value;

public static DetachedCriteria Clone(this DetachedCriteria criteria)
{
var dummy = criteria.ToByteArray();
return dummy.FromByteArray<DetachedCriteria>();
}

How do you query object collections in Java (Criteria/SQL-like)?

Suppose you have a collection of a few hundred in-memory objects and you need to query this List to return objects matching some SQL or Criteria like query. For example, you might have a List of Car objects and you want to return all cars made during the 1960s, with a license plate that starts with AZ, ordered by the name of the car model.
I know about JoSQL, has anyone used this, or have any experience with other/homegrown solutions?

Filtering is one way to do this, as discussed in other answers.
Filtering is not scalable though. On the surface time complexity would appear to be O(n) (i.e. already not scalable if the number of objects in the collection will grow), but actually because one or more tests need to be applied to each object depending on the query, time complexity more accurately is O(n t) where t is the number of tests to apply to each object.
So performance will degrade as additional objects are added to the collection, and/or as the number of tests in the query increases.
There is another way to do this, using indexing and set theory.
One approach is to build indexes on the fields within the objects stored in your collection and which you will subsequently test in your query.
Say you have a collection of Car objects and every Car object has a field color. Say your query is the equivalent of "SELECT * FROM cars WHERE Car.color = 'blue'". You could build an index on Car.color, which would basically look like this:
'blue' -> {Car{name=blue_car_1, color='blue'}, Car{name=blue_car_2, color='blue'}}
'red' -> {Car{name=red_car_1, color='red'}, Car{name=red_car_2, color='red'}}
Then given a query WHERE Car.color = 'blue', the set of blue cars could be retrieved in O(1) time complexity. If there were additional tests in your query, you could then test each car in that candidate set to check if it matched the remaining tests in your query. Since the candidate set is likely to be significantly smaller than the entire collection, time complexity is less than O(n) (in the engineering sense, see comments below). Performance does not degrade as much, when additional objects are added to the collection. But this is still not perfect, read on.
Another approach, is what I would refer to as a standing query index. To explain: with conventional iteration and filtering, the collection is iterated and every object is tested to see if it matches the query. So filtering is like running a query over a collection. A standing query index would be the other way around, where the collection is instead run over the query, but only once for each object in the collection, even though the collection could be queried any number of times.
A standing query index would be like registering a query with some sort of intelligent collection, such that as objects are added to and removed from the collection, the collection would automatically test each object against all of the standing queries which have been registered with it. If an object matches a standing query then the collection could add/remove it to/from a set dedicated to storing objects matching that query. Subsequently, objects matching any of the registered queries could be retrieved in O(1) time complexity.
The information above is taken from CQEngine (Collection Query Engine). This basically is a NoSQL query engine for retrieving objects from Java collections using SQL-like queries, without the overhead of iterating through the collection. It is built around the ideas above, plus some more. Disclaimer: I am the author. It's open source and in maven central. If you find it helpful please upvote this answer!

I have used Apache Commons JXPath in a production application. It allows you to apply XPath expressions to graphs of objects in Java.

yes, I know it's an old post, but technologies appear everyday and the answer will change in the time.
I think this is a good problem to solve it with LambdaJ. You can find it here:
http://code.google.com/p/lambdaj/
Here you have an example:
LOOK FOR ACTIVE CUSTOMERS // (Iterable version)
List<Customer> activeCustomers = new ArrayList<Customer>();
for (Customer customer : customers) {
if (customer.isActive()) {
activeCusomers.add(customer);
}
}
LambdaJ version
List<Customer> activeCustomers = select(customers,
having(on(Customer.class).isActive()));
Of course, having this kind of beauty impacts in the performance (a little... an average of 2 times), but can you find a more readable code?
It has many many features, another example could be sorting:
Sort Iterative
List<Person> sortedByAgePersons = new ArrayList<Person>(persons);
Collections.sort(sortedByAgePersons, new Comparator<Person>() {
public int compare(Person p1, Person p2) {
return Integer.valueOf(p1.getAge()).compareTo(p2.getAge());
}
});
Sort with lambda
List<Person> sortedByAgePersons = sort(persons, on(Person.class).getAge());
Update: after java 8 you can use out of the box lambda expressions, like:
List<Customer> activeCustomers = customers.stream()
.filter(Customer::isActive)
.collect(Collectors.toList());

Continuing the Comparator theme, you may also want to take a look at the Google Collections API. In particular, they have an interface called Predicate, which serves a similar role to Comparator, in that it is a simple interface that can be used by a filtering method, like Sets.filter. They include a whole bunch of composite predicate implementations, to do ANDs, ORs, etc.
Depending on the size of your data set, it may make more sense to use this approach than a SQL or external relational database approach.

If you need a single concrete match, you can have the class implement Comparator, then create a standalone object with all the hashed fields included and use it to return the index of the match. When you want to find more than one (potentially) object in the collection, you'll have to turn to a library like JoSQL (which has worked well in the trivial cases I've used it for).
In general, I tend to embed Derby into even my small applications, use Hibernate annotations to define my model classes and let Hibernate deal with caching schemes to keep everything fast.

I would use a Comparator that takes a range of years and license plate pattern as input parameters. Then just iterate through your collection and copy the objects that match. You'd likely end up making a whole package of custom Comparators with this approach.

The Comparator option is not bad, especially if you use anonymous classes (so as not to create redundant classes in the project), but eventually when you look at the flow of comparisons, it's pretty much just like looping over the entire collection yourself, specifying exactly the conditions for matching items:
if (Car car : cars) {
if (1959 < car.getYear() && 1970 > car.getYear() &&
car.getLicense().startsWith("AZ")) {
result.add(car);
}
}
Then there's the sorting... that might be a pain in the backside, but luckily there's class Collections and its sort methods, one of which receives a Comparator...