Develop Reference

How to change the way Talend formulates SQL queries in a JDBC connection?

In Talend Data Quality, I have configured a JDBC connection to an OpenEdge database and it's working fine.
I can pull the list of tables and select columns to analyse, but when executing analysis, I get this :
Table "DBGSS.SGSSGSS" cannot be found.
This is because it does not specify a schema, only the database name - DBGSS.
How can I make it specify database, schema and then the table name ? Or just the table name, its would work too.
Thanks !

You can use a tDBConnection component that give you the right to specify a schéma
Then , use it with the option of Use Existing connection
See below documentation , https://help.talend.com/r/en-US/7.3/db-generic/tdbconnection

Avoid ORA-00904 - invalid identifier error while doing sql query in java as the column may or may not be preset in the database

I am trying to write java code to migrate data from oracle database to other database.
My use case is that different client have different version of code and so the database columns may vary. Clients with later version have additional column.
For eg : Client with new version as COL99 in the table SAMPLE_TABLE.
While writing the migration code, if I try to select the COL99 from SAMPLE_TABLE, it will work fine for the new client. But for clients on old version, the code fails with
ORA-00904 Invalid Identifier error.
Is there a way to handle in sql query or java code such that, if the column doesn't exist in the database table, simply ignore and do not return the value instead of throwing the exception.

You should first check, whether COL99 exists for your current database connection.
For Oracle you can use a query like this:
SELECT
COL.COLUMN_ID,
COL.OWNER AS SCHEMA_NAME,
COL.TABLE_NAME,
COL.COLUMN_NAME
FROM
SYS.ALL_TAB_COLUMNS COL
INNER JOIN
SYS.ALL_TABLES T
ON COL.OWNER = T.OWNER
AND
COL.TABLE_NAME = T.TABLE_NAME
WHERE
COL.OWNER = 'SCHEMA'
AND
COL.TABLE_NAME = 'SAMPLE_TABLE'
AND
COL.COLUMN_NAME = 'COL99'
Then you create your query with or without COL99.

How to pass SQL query as a context variable in Talend Open Studio job

I am creating a job to pull data from a database to CSV file using talend open studio. There are 100 of tables, the data types and no of columns differ in the tables, I want to pull the data from database tables with a single job and customizable SQL query. I know how to create and use context variables.

If I understood you correctly you should be using tMap's reload at each row -option and defining table names in Excel sheet or in tFixedFlowInput.
tMap settings
Whole job and results
SQL Script:
"SELECT TOP(1) Name, Code from mdm." + (String)globalMap.get("row4.table")
I used Microsoft SQL Server for example but same script works as well with MySQL.

You can simply use a context-variable which you set via the --context_param argument in a tWhicheverDatabaseInput. E.g. define a context variable "my_sql" which you can set in the commandline as
my_job.sh --context_param my_sql="select a,b,c from a_test_table"
and then use context.my_sql as the SQL in you database input component.
However, as garpitmzn already mentioned, you will need dynamic schemas to actually work with this unknown structure in Talend. This feature only exists in the enterprise version.
If the enterprise version is available to you, simply declare a single column of type "Dynamic", which you can pass through the rest of your flow.

Declare a local context say query as type string.
Prepare a context file with variable query: query=select name from employee
Excecute the query: toraclecomponent use context.query
Query is throwing some error when you have where conditions with string type.
Need to investigate more on that. Otherwise it works.

Delete Stored Procedure through remote

I am creating and testing a stored procedure through remote using JUnit , my user name is test and this is my code,
String sql = "create procedure test.firstProc3 AS select GETDATE()";
cursor.execute(sql);
cursor.executeQuery("firstProc3");
ANd to drop the procedure:
String dropSQL = "DROP PROCEDURE test.firstProc3";
cursor.execute(dropSQL);
Since the Drop Procedure is not working, I am unable to run the same SP for the second time ( new session) My error is:
There is already an object named 'firstProc3' in the database.
When I gave sp_help on the server side,
the table had the row with the value
firstProc3 test stored procedure
However when I give
DROP PROCEDURE test.firstProc3 in the Query analyzer, the row is getting deleted from the table.
What could be the issue in trying to do the same operation through Junit?
Are there any permissions to be set?
PS - the user test has the db_ddladmin enabled.

I managed to finally solve the problem by using the following :
declare #object_id int
select #object_id = object_id('test.firstProc5')
EXEC sp_MSdrop_object #object_id
GO
This is removing the data from the table too.And I am able to successfully run the Testcase multiple times !
Thanks for your help :)
The link I referred to is here, hope it helps someone:
http://www.mssqlcity.com/Articles/Undoc/SQL2000UndocSP.htm

Based upon the error message, I doubt it is a permissions issue. It looks like your cursor object is running the create again when you're trying to drop, or you're never really dropping the proc before running your unit test again, or the tests are running out of logical order in some fashion. Can you instantiate a new cursor object and try the drop with that, or verify the SQL being run against the database using profiler or some debug output?
Beyond that, I would be curious to see if changing your SQL commands to check for proc existence makes the error go away (not that this is the right answer):
if object_id('test.firstProc3') is null begin
-- Dynamic SQL because create must be first in a batch
exec ('create procedure test.firstProc3 as select getdate()')
end
if object_id('test.firstProc3') is not null begin
drop procedure test.firstProc3
end

Maybe something small, and maybe it is even not the case, but have you tried to use escaping in your drop statement?
something like
drop procedure [test].[firstProc3]
I can remember that i had an issue with this when i called a drop table from c# code...

Getting an exception ORA-00942: table or view does not exist - when inserting into an existing table

I am getting below exception, when trying to insert a batch of rows to an existing table
ORA-00942: table or view does not exist
I can confirm that the table exists in db and I can insert data to that table using oracle
sql developer. But when I try to insert rows using preparedstatement in java, its throwing table does not exist error.
Please find the stack trace of error below
java.sql.SQLException: ORA-00942: table or view does not exist
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:573)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1889)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1093)
at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2047)
at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:1940)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout>>(OracleStatement.java:2709)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:589)
at quotecopy.DbConnection.insertIntoDestinationDb(DbConnection.java:591)
at quotecopy.QuoteCopier.main(QuoteCopier.java:72)
Can anyone suggest the reasons for this error ?
Update : Issue solved
There was no problem with my database connection properties or with my table or view name. The solution to the problem was very strange. One of the columns that I was trying insert was of Clob type. As I had a lot of trouble handling clob data in oracle db before, gave a try by replacing the clob setter with a temporary string setter and the same code executed with out any problems and all the rows were correctly inserted!!!.
ie. peparedstatement.setClob(columnIndex, clob)
was replaced with
peparedstatement.setString(columnIndex, "String")
Why an error table or view does exist error was throws for error in inserting clob data. Could anyone of you please explain ?
Thanks a lot for your answers and comments.

Oracle will also report this error if the table exists, but you don't have any privileges on it. So if you are sure that the table is there, check the grants.

There seems to be some issue with setCLOB() that causes an ORA-00942 under some circumstances when the target table does exist and is correctly privileged. I'm having this exact issue now, I can make the ORA-00942 go away by simply not binding the CLOB into the same table.
I've tried setClob() with a java.sql.Clob and setCLOB() with an oracle.jdbc.CLOB but with the same result.
As you say, if you bind as a string the problem goes away - but this then limits your data size to 4k.
From testing it seems to be triggered when a transaction is open on the session prior to binding the CLOB. I'll feed back when I've solved this...checking Oracle support.

There was no problem with my database connection properties or with my table or view name. The solution to the problem was very strange. One of the columns that I was trying insert was of Clob type. As I had a lot of trouble handling clob data in oracle db before, gave a try by replacing the clob setter with a temporary string setter and the same code executed with out any problems and all the rows were correctly inserted!!!.
ie. peparedstatement.setClob(columnIndex, clob)
was replaced with
peparedstatement.setString(columnIndex, "String")

#unbeli is right. Not having appropriate grants on a table will result in this error. For what it's worth, I recently experienced this. I was experiencing the exact problem that you described, I could execute insert statements through sql developer but would fail when using hibernate. I finally realized that my code was doing more than the obvious insert. Inserting into other tables that did not have appropriate grants. Adjusting grant privileges solved this for me.
Note: Don't have reputation to comment, otherwise this may have been a comment.

We experienced this issue on a BLOB column. Just in case anyone else lands on this question when encountering this error, here is how we resolved the issue:
We started out with this:
preparedStatement.setBlob(parameterIndex, resultSet.getBlob(columnName)); break;
We resolved the issue by changing that line to this:
java.sql.Blob blob = resultSet.getBlob(columnName);
if (blob != null) {
java.io.InputStream blobData = blob.getBinaryStream();
preparedStatement.setBinaryStream(parameterIndex, blobData);
} else {
preparedStatement.setBinaryStream(parameterIndex, null);
}

I found how to solve this problem without using JDBC's setString() method which limits the data to 4K.
What you need to do is to use preparedStatement.setClob(int parameterIndex, Reader reader). At least this is what that worked for me. Thought Oracle drivers converts data to character stream to insert, seems like not. Or something specific causing an error.
Using a characterStream seems to work for me. I am reading tables from one db and writing to another one using jdbc. And i was getting table not found error just like it is mentioned above. So this is how i solved the problem:
case Types.CLOB: //Using a switch statement for all columns, this is for CLOB columns
Clob clobData = resultSet.getClob(columnIndex); // The source db
if (clobData != null) {
preparedStatement.setClob(columnIndex, clobData.getCharacterStream());
} else {
preparedStatement.setClob(columnIndex, clobData);
}
clobData = null;
return;
All good now.

Is your script providing the schema name, or do you rely on the user logged into the database to select the default schema?
It might be that you do not name the schema and that you perform your batch with a system user instead of the schema user resulting in the wrong execution context for a script that would work fine if executed by the user that has the target schema set as default schema. Your best action would be to include the schema name in the insert statements:
INSERT INTO myschema.mytable (mycolums) VALUES ('myvalue')
update: Do you try to bind the table name as bound value in your prepared statement? That won't work.

It works for me:
Clob clob1;
while (rs.next()) {
rs.setString(1, rs.getString("FIELD_1"));
clob1 = rs.getClob("CLOB1");
if (clob1 != null) {
sta.setClob(2, clob1.getCharacterStream());
} else {
sta.setClob(2, clob1);
}
clob1 = null;
sta.setString(3, rs.getString("FIELD_3"));
}

Is it possible that you are doing INSERT for VARCHAR but doing an INSERT then an UPDATE for CLOB?
If so, you'll need to grant UPDATE permissions to the table in addition to INSERT.
See https://stackoverflow.com/a/64352414/1089967

Here I got the solution for the question. The problem is on glass fish if you are using it. When you create JNDI name make sure pool name is correct and pool name is the name of connection pool name that you are created.