Develop Reference

How to pass Runtime query to SqlTransform in apache beam?

I want to pass the Query dynamically while running the Dataflow job. I am using SQLTransform which works fine when I pass Query within code.
My use case requires passing the Query at Runtime, is it possible with SqlTransform in Apache Beam?
This works if I hard-code it in code.
String PQuery = "SELECT col1, max(col2) as max_watermark FROM PCOLLECTION GROUP BY col1";
PCollection<Row> rows1 = rows.apply(SqlTransform.query(PQuery));
But with valueProvider input, it gives compile time error.
PCollection<Row> rows1 = rows.apply(SqlTransform.query(options.getQuery()))
Error
The method query(String) in the type SqlTransform is not applicable for the arguments (ValueProvider<String>)

To solve your issue, you need to get the value inside the ValueProvider :
PCollection<Row> rows1 = rows.apply(SqlTransform.query(options.getQuery().get()))
The query method takes a String as parameter, that's why you need to get the String value of the ValueProvider option.

You should use FlexTemplates which allow dynamic graph construction (such as SqlTransform) uses based on template parameters.

Criteria JPA - Call Postgres CAST function

I'm trying to call a Postgres function with Criteria but it's not working. I need to use the LIKE clause in a UUID field, so I need to convert into VARCHAR first.
The result I need:
SELECT * FROM my_table WHERE cast(uuid as varchar(36)) like '%1234%';
What I'm doing in Criteria:
final Path<UUID> uuidField = from.get("uuid");
var cast = cb.function("cast", String.class, uuidField, cb.literal("as varchar(36)"));
cb.like(cast, String.format("%%%s%%", stringValue));
The query which is being generated:
HQL: select generatedAlias0 from com.MyTable as generatedAlias0 where function('cast', generatedAlias0.uuid, 'as varchar(36)') like '%1234%' order by generatedAlias0.name asc
Error:
2022-08-08 18:38:48,549 WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-9,5,main] has been blocked for 2393 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked
at antlr.ASTFactory.make(ASTFactory.java:342)
at antlr.ASTFactory.make(ASTFactory.java:352)
at org.hibernate.hql.internal.antlr.HqlBaseParser.jpaFunctionSyntax(HqlBaseParser.java:4633)
at org.hibernate.hql.internal.antlr.HqlBaseParser.primaryExpression(HqlBaseParser.java:1075)
The log is not so clear (I'm using Quarkus + Hibernate Reactive), but I suspect it crashed in database because the function('cast', generatedAlias0.uuid, 'as varchar(36)').
I think it should be something like: function('cast', generatedAlias0.uuid, as varchar(36)) (without quotes). But I don't know how to achieve this result to test my theory.
How can I call this CAST function?

After investigating some possible solutions (I'm avoiding to create custom database routines) I found something interesting in a answer from another question:
Currently JPA does not have APIs for replace() and cast(string as numeric). But you can use CriteriaBuilder.function(...) to create database native functions if database portability is not critical.
Source: JPA criteria builder: how to replace and cast a string to numeric in order-by?
I don't know if this is documented is some place, but assuming that there is no way to call CAST(x AS y) using Criteria, I tried a workaround to force the UUID to VARCHAR cast without using the probably unsupported CAST function.
I tested this direct SQL query to database:
SELECT * FROM my_table WHERE concat(uuid, '') like '%123%';
And it works. This CONCAT forces the cast to VARCHAR and the LIKE function does his job. Knowing this, I did:
final Path<UUID> uuidField = from.get("uuid");
var cast = cb.function("concat", String.class, uuidField, cb.literal(""));
cb.like(cast, String.format("%%%s%%", stringValue));
Worked perfectly. I hope this help someone else.
As #HaroldH said, it's a weird requirement, but happened in my project.

How to use a 'Hibernate Types' library type in a single native query that selects Postgres arrays?

I have a query that returns a Postgres array of UUIDs:
SELECT e.id, e.date,
ARRAY
(
SELECT cs.api_id FROM event_condition_set ecs
JOIN condition_set cs on cs.id = ecs.condition_set_id
WHERE ecs.event_id = e.id
) AS condition_set_ids,
...
And then create and run this query as a native query: Query query = entityManager.createNativeQuery(queryString);
Since Hibernate can normally not deal with these Postgres arrays, I use Vlad's Hibernate Types library.
However, currently I need to register this UUIDArrayType globally in my application:
public class PostgreSQL95CustomDialect extends PostgreSQL95Dialect {
public PostgreSQL95CustomDialect() {
super();
this.registerHibernateType(Types.ARRAY, UUIDArrayType.class.getName());
}
}
Aside from the fact this is a bit ugly, it also leaves no room for other types of arrays.
(Note I also tried registering a generic ListArrayType but this throws a NullPointerException during execution of the query.)
I have also tried registering it as a scalar type:
query.unwrap(org.hibernate.query.NativeQuery.class)
.addScalar("condition_set_ids", UUIDArrayType.INSTANCE);
But this makes the entire query only return a single UUID, which is very strange and seems bugged.
Is there a way to ONLY use this UUIDArrayType specifically in this query?
(Please don't suggest using array_agg because the performance is terrible for this case)

you can call native queries using custom Hibernate types as follows:
String myJsonbData = ...;
String[] myStringArr = ...;
final String queryStr = "select your_function(?, ?, ...)"; // ? for each param
entityManager
.createNativeQuery(queryStr)
.setParameter(1, new TypedParameterValue(JsonBinaryType.INSTANCE, myJsonbData))
.setParameter(2, new TypedParameterValue(StringArrayType.INSTANCE, myStringArr));
This is just an example, but as a rule of thumb, you need to instantiate a new TypedParameterValue.

Answering my own question here. After waiting for a while and updating to the most recent library version (I'm on 2.19.2 right now) I don't have any issues anymore with the scalar types registration as I mentioned in my question, i.e.:
query.unwrap(org.hibernate.query.NativeQuery.class)
.addScalar("condition_set_ids", UUIDArrayType.INSTANCE);
So it appears to just have been a bug and I can now avoid the global registration in favor of using scalars.

Introducing a named parameter breaks jOOQ query

To query a PostgreSQL 10.11 database, I am using jOOQ 3.12.4, which comes bundled with Spring Boot 2.2.
Let's assume I have built a query using jOOQ like this:
final String[] ids = ...;
final var query = dslContext.selectFrom(MY_TABLE).where(MY_TABLE.ID.in(ids));
final Map<String, List<MyTable>> changeDomains = query.fetch().intoGroups(MY_TABLE.ID, MyTable.class);
This code runs fine and produces the expected results. But when I refactor my query and introduce a named parameter (to reuse the query in multiple parts of my code), like this:
final String[] ids = ...;
final var query = dslContext.selectFrom(MY_TABLE).where(MY_TABLE.ID.in(param("ids")));
final Map<String, List<MyTable>> changeDomains = query.bind("ids", ids).fetch().intoGroups(MY_TABLE.ID, MyTable.class);
I suddenly start to get the following error:
org.springframework.jdbc.BadSqlGrammarException: jOOQ; bad SQL grammar ...; nested exception is org.postgresql.util.PSQLException: ERROR: operator does not exist: text = character varying[]
Hinweis: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Edit: I get the same error when I use
MY_TABLE.ID.in(param("ids", String[].class))
instead.
How can I solve or work around this problem?

A better solution to your code reuse approach
But when I refactor my query and introduce a named parameter (to reuse the query in multiple parts of my code)
While you could use jOOQ this way (be careful, when mutating and reusing jOOQ queries in a non-threadsafe way!), it is generally recommended to use jOOQ in a more functional way, see e.g.:
https://blog.jooq.org/2017/01/16/a-functional-programming-approach-to-dynamic-sql-with-jooq/
https://www.jooq.org/doc/latest/manual/sql-building/dynamic-sql/
You don't gain much by re-using a jOOQ query, specifically, there's hardly any performance gain.
So, instead of this:
final var query = dslContext.selectFrom(MY_TABLE)
.where(MY_TABLE.ID.in(param("ids")));
final Map<String, List<MyTable>> changeDomains = query
.bind("ids", ids).fetch().intoGroups(MY_TABLE.ID, MyTable.class);
Write this:
public ResultQuery<MyTableRecord> query(String[] ids) {
return dslContext.selectFrom(MY_TABLE).where(MY_TABLE.ID.in(ids));
}
// And then:
final Map<String, List<MyTable>> changeDomains = query(ids)
.fetch().intoGroups(MY_TABLE.ID, MyTable.class);
The actual problem you ran into:
jOOQ, JDBC, and SQL don't support single bind value IN lists. While it seems useful to write this:
SELECT * FROM t WHERE c IN (:bind_value)
And passing an array or list as a single bind value, this is not supported in SQL. Some APIs might pretend that this is supported (but behind the scenes replace the single bind value by multiple ?, ?, ..., ?
PostgreSQL supports the = ANY (:bind_value) operator with arrays
SELECT * FROM t WHERE c = ANY (:bind_value)
You could use it in jOOQ using
dslContext.selectFrom(MY_TABLE).where(MY_TABLE.ID.eq(any(ids)));
That way, you could call the bind() method to replace the array prior to execution. However, I still recommend you write functions returning queries dynamically.

How to use JOOQ's parser to extract table names from SQL statements [duplicate]

Using the JOOQ parser API, I'm able to parse the following query and get the parameters map from the resulting Query object. From this, I can tell that there is one parameter, and it's name is "something".
However, I haven't been able to figure out how to determine that the parameter "something" is assigned to a column named "BAZ" and that column is part of the table "BAR".
Does the parser API have a way to get the table/column metadata associated to each parameter?
String sql = "SELECT A.FOO FROM BAR A WHERE A.BAZ = :something";
DSLContext context = DSL.using...
Parser parser = context.parser();
Query query = parser.parseQuery(sql);
Map<String, Param<?>> params = query.getParams();

Starting from jOOQ 3.16
jOOQ 3.16 introduced a new, experimental (as of 3.16) query object model API, which can be traversed, see:
The manual
A blog post about traversing jOOQ expression trees
Specifically, you can write:
List<QueryPart> parts = query.$traverse(
Traversers.findingAll(q -> q instanceof Param)
);
Or, to conveniently produce exactly the type you wanted:
Map<String, Param<?>> params = query.$traverse(Traversers.collecting(
Collectors.filtering(q -> q instanceof Param,
Collectors.toMap(
q -> ((Param<?>) q).getParamName(),
q -> (Param<?>) q
)
)
));
The Collectors.toMap() call could include a mergeFunction, in case you have the same param name twice.
Pre jOOQ 3.16
As of jOOQ 3.11, the SPI that can be used to access the internal expression tree is the VisitListener SPI, which you have to attach to your context.configuration() prior to parsing. It will then be invoked whenever you traverse that expression tree, e.g. on your query.getParams() call.
However, there's quite a bit of manual plumbing that needs to be done. For example, the VisitListener will only see A.BAZ as a column reference without knowing directly that A is the renamed table BAR. You will have to keep track of such renaming yourself when you visit the BAR A expression.