I am working with Talend ESB which uses camel.
My requirement is I need to do batch insert into a table, and have to create the query dynamically for different types of input.
If I create a context say extVar and default it to
"insert into table_foo (foo, bar) values (:#foo, :#bar)"
and then use cMessagingEndpoint component with following code
"sql"+context.extVar+"?batch=true&dataSource=mysql"
it works fine.
In Talend, I am able to access context.extVar inside cProcessor so I tried to load the query dynamically from there, but I couldn't.
Since I am using named parameters, I assume that I can neither use the body for query nor can I get the property/header value inside the sql statement of cMessageEndPoint.
Is there way where I can populate the query dynamically into sql statement of cMessageEndPoint.
I was able to solve my issue. Thanks to this answer. My objective was to have a dynamic batch insert query, with cRecipientList, I was able to achieve it. I was also able to get the context variables read and be used inside the Expression of cRecipientList while loading that dynamically from cProcessor.
Update:
I just found that you can also use .toD (available from camel version 2.16) to use simple expression language inside query.
For example: you could write,
.toD("sql:${property.query}?dataSource=ds_dwh_d&batch=true")
Related
In our project we use mybatis and have a custom type handler on strings that strip html and other things from the result by default. We can override this by setting a different typehandler in the resultmap.
However I now have a situation where dynamic querys are used in a select so something like this
<select id="executeQuery" resultType="String">
${sql}
</select>
Now I dont want the result to be stripped of its html, so I want to use a custom typehandler. Normally I would do this in a result map, but the problem is I dont know the name of the column that will be used in the query. Is there a solution for this problem where I can use another typehandler on this result?
For others running into this problem. In the end I used the following workaround:
SELECT (${sql}) AS value
This has some limitation in what kind of queries can be used in the sql statement though. For example in ms sql a query with an order by without a top, offset or for xml cannot be used.
I have a use case where in I need to read rows from a file, transform them using an engine and then write the output to a database (that can be configured).
While I could write a query builder of my own, I was interested in knowing if there's already an available solution (library).
I searched online and could find jOOQ library but it looks like it is type-safe and has a code-gen tool so is probably suited for static database schema's. In the use case that I have db's can be configured dynamically and the meta-data is programatically read and made available for write-purposes (so a list of tables would be made available, user can select the columns to write and the insert script for these column needs to be dynamically created).
Is there any library that could help me with the use case?
If I understand correctly you need to query the database structure, display the result to via a GUI and have the user map data from a file to that structure?
Assuming this is the case, you're not looking for a 'library', you're looking for an ETL tool.
Alternatively, if you're set on writing something yourself, the (very) basic way to do this is:
the structure of a database using Connection.getMetaData(). The exact usage can vary between drivers so you'll need to create an abstraction layer that meets your needs - I'd assume you're just interested in the table structure here.
the format of the file needs to be mapped to a similar structure to the tables.
provide a GUI that allows the user to connect elements from the file to columns in the table including any type mapping that is needed.
create a parametrized insert statement based on file element to column mapping - this is just a simple bit of string concatenation.
loop throw the rows in the file performing a batch insert for each.
My advice, get an ETL tool, this sounds like a simple problem, but it's full of idiosyncrasies - getting even an 80% solution will be tough and time consuming.
jOOQ (the library you referenced in your question) can be used without code generation as indicated in the jOOQ manual:
http://www.jooq.org/doc/latest/manual/getting-started/use-cases/jooq-as-a-standalone-sql-builder
http://www.jooq.org/doc/latest/manual/sql-building/plain-sql
When searching through the user group, you'll find other users leveraging jOOQ in the way you intend
The setps you need to do is:
read the rows
build each row into an object
transform the above object to target object
insert the target object into the db
Among the above 4 steps, the only thing you need to do is step 3.
And for the above purpose, you can use Transmorph, EZMorph, Commons-BeanUtils, Dozer, etc.
I am using Hiberante to connect to postgres database. I am trying to insert a record into the database. I have the values for the record in a string array which I got from a csv file. This is my dao code
StringBuffer query=new StringBuffer("insert into t_wonlist values(");
for(int i=0;i<67;i++){
query.append(values[i]+",");
}
query.deleteCharAt(query.lastIndexOf(","));
query.append(");");
sessionfactory.getCurrentSession().createSQLQuery(query.toString()).executeUpdate();
System.out.println("Query executed");
sessionfactory.getCurrentSession().flush();
I am using StringBuffer, so that I can append the values into the query using a for loop.
but when I execute the query I am getting the following exception
org.postgresql.util.PSQLException: No value specified for parameter 1.
I am sure that the number of parameters is correct. Can someone help me. Thanks
You're approaching this in a bizarre and backwards manner.
The immediate problem is probably failure to escape/quote a ? in one of the input strings, so PgJDBC thinks it's a query parameter. That doesn't mean you should fix it by escaping/quoting question marks, it's a sign you're taking entirely the wrong approach.
Please read this page on SQL injection and this site.
You're using the Hibernate ORM, so you'd usually be using the JPA interface or the direct Hibernate interface to create new domain objects and persisting them. The typical approach is to new an object, then use the EntityManager.persist method (if using JPA) or the Session.save method (if using Hibernate directly) to persist the entities.
If you want to use direct JDBC instead you should be creating a JDBC PreparedStatement, setting its parameters, and then applying it. See this tutorial. Since you're loading CSV you'd usually do this in a JDBC batch, though this doesn't actually gain you much in PostgreSQL.
Better yet, since you're importing CSV you can probably just use PostgreSQL's built-in COPY command via PgJDBC's CopyManager to stream the changes efficiently into the target table.
first of all, a preface: I'm writing a java class that creates temporary tables on a database using jdbc. I'm using JSE6 and Oracle 11XE as a test DB, but the class needs to be also DB2 compliant.
The temporary tables I'm trying to create come from a bigger one, and I do some filtering and aggregations on the data. The parameters I base my filtering on are decided by the user at runtime. One simplified example of what I'm trying to do is this:
CREATE TABLE temp_table AS (
SELECT
table1.department_id,
SUM(CASE WHEN table1.number_1 < &1 THEN table1.number_1 ELSE 0 END)) AS column1
FROM
table1
GROUP BY table1.department_id
)
My problem is that I need to specify parameters to filter the data, and I need to be sure they're properly escaped/localized/typed. This would be easy using a prepared statement, but I cannot use bind variables with DDL.
The temporary solution I resorted to is to alter the query String myself, writing the parameters in the correct place, but this means I now have to implement all the checks instead of relying on a PreparedStatement object to do it for me, on top of losing all the other benefits.
I investigated other solutions, but none of them convinced me so far:
I could first create an empty temp_table and then fill it with INSERT INTO temp_table(id, column1) (SELECT ...) but it seems I might incur in performance loss, so I'd like to stick to the CREATE temp_table AS
I thought about creating a temporary statement to hold the inner SELECT query, and have it generate a properly formatted/localized/etc. query string, but I haven't found any way to obtain the final query from it (and I read it's definitely not possible here). The only option I found for this case is to use DebuggableStatement, but I'm not sure I can include it in the project (also, it seems a quite inelegant way of solving my problem)
Another solution I'm thinking of, is to simply put the queries that create the temporary tables (for each of them I'd put the whole CREATE AS (SELECT...) on the database, inside a procedure, which I'll then be able to call using CallableStatement. this way I could avoid handling typization and still have good performances, at the price of a tighter coupling with the db (I'd have to be sure the procedures are there, or manage in java their addition/removal from the db)
So, my question is: are there better alternatives than the ones I could think of?
Is this supposed to be database agnostic, or are you targeting for only Oracle? You don't have to store PL/SQL in a stored procedure to use it; just build an anonymous PL/SQL block that does what you need, and execute it. The anonymous PL/SQL block can be built dynamically so that strongly typed variables are declared in the PL/SQL to hold your parameters, and then your java code sticks the values in. The type safety wouldn't be handled by Java since you're just building a string; it would be handled by Oracle when you execute the anonymous PL/SQL block.
I can create a table and its columns in Java by using the statement:
CREATE TABLE table_name(column1 int, column2 double, etc...)
What I would like to do is to add descriptions to each of these columns with an appropriate statement, I found a stored procedure sp_addextendedproperty that looks like it can be used to accomplish this I just have no idea how to use it in java with jdbc.
Are you creating the table dynamically at runtime (e.g. as part of your application) - perhaps that's even user-driven? If that's the case, you already have that "documentation" (column comments) somewhere and I doubt the utility of adding them to SQL Server.
But if you're just trying to automate your build, take a look at LiquiBase. It's a pretty decent DB change management system that uses XML as backbone. It's written in java and integrates well with Hibernate (useful if you ever decide to use ORM instead of straight JDBC).
Update: If you do decide to go forward with calling stored procedure via JDBC, I would strongly recommend using CallableStatement to invoke it. Dynamically building SQL queries in the application should be avoided if possible.
There are a number of ways to call a stored procedure (essentially, preparing the statement and binding the variables, or sending a string of SQL), but the simplest is to just send rhe SQL statement
exec sp_addextendedproperty list, of, arguments, the, sp, needs;
Skipping your try/finally boilerplate, and assuming connection is a java.sql.Connection, that's:
connection
.createStatement()
.execute( "exec sp_addextendedproperty arguments;");
But ChssPly76 has a good point: doing this from Java isn't a good idea (unless you're developing some database manager in Java).