I have Oracle database with 10 tables. Some of the tables have CLOB data text. I need to export data from these tables pro-grammatically using java. The export data should be in ANSI INSERT INTO SQL format, for example:
INSERT INTO table_name (column1, column2, column3, ...)
VALUES (value1, value2, value3, ...);
The main idea is that I need to import this data into three different databases:
ORACLE, MSSQL and MySQL. As I know all these databases support ANSI INSERT INTO. But I have not found any java API/framework for generating data SQL scripts. And I do not know how to deal with CLOB data, how to export it.
What is the best way to export data from a database with java?
UPDATE: (01.07.2018)
I guess it is impossible to insert text data more than 4000 bytes according to this answer. How to generate PL\SQL scripts using java programmatically? Or is there any other export format which supports ORACLE, MSSQL, etc?
Did you ever think about a proper ORM-Api? The first thing in my mind would come to Hibernate or more abstract JPA/JPQL. The framework does know all the main sql dialects. All what you need is to define your connections with your dialects. Than you retrieve the data from the database and its mapped into POJO's, and than you push(insert) the data to your different(other dialect) connection. Should work good i think, even if i never did this. But i know JPA is not new and widely used for the sake of changing the database even when the software is already in production. This approach is a bit inperformant since every row gets transformed into POJO and there is, afaik, no bulk insertion available.
If you are looking for SQL generation then there are many sqlbuilder libraries available, which you can use.
You can use metadata to get column names and type from the select * query and use it at insert query.
See https://github.com/jkrasnay/sqlbuilder
More about it at http://john.krasnay.ca/2010/02/15/building-sql-in-java.html
If your need is to export tables from a Oracle database to insert it back into different types of Database I would suggest a different approach.
This is the perfect use case for JPA (Java Persistence API) which allows you create a model that represent your database structure. This is the Java current solution solution for managing different types of database.
From your model you will be able to generate request compatible with all popular databases.
So my suggestion is, using Spring Boot + Spring Data + Spring Batch :
Create a first app from your model that exports the content of your tables to CSV format.
Create a second app from the same model that imports your CSV files. Depending on you jdbc url, Spring Boot will automtically trigger the appropriate dialect for your target Database and generate the right queries (this is also the case for the export).
This can be done within a reasonnable amount of time and with decent performance.
I like "pro-grammatically" very much. :)
Best way to export that data is to iterate over tables, then query each of them and output plain text with insert into statements. It can be problem if you have binary data there, since different RDBS can deal with it in a slightly different way.
Reading blob/clob at Java side means reading stream. It can be binary, or character stream. For Oracle, from the doc - you can do:
ResultSet rs = s.executeQuery(
"SELECT text FROM documents WHERE id = 1477");
while (rs.next()) {
java.sql.Clob aclob = rs.getClob(1);
java.io.InputStream ip = rs.getAsciiStream(1);
int c = ip.read();
while (c > 0) {
System.out.print((char)c);
c = ip.read();
}
System.out.print("\n");
}
From this answer, you can make it shorter:
Clob clob = resultSet.getClob("CLOB_COLUMN")
String clob_content = clob.getSubString(1, (int) clob.length());
Writing output perhaps would require dealing with: \t\n\r. Depends on your needs, content. The docs have full example - reading, writing. They use prepared statement, and that is why streams are needed at both ends. If your clob is not big - like 32k/64k - there can be other limits. If you have any example - like create table with 2-3 rows it would be much easier by anyone to write code, and provide something that works...
I need to export data from these tables programmatically using java
Come on guy! What is a matter? Java is a tool to operate data, not to migrate. If it's about ETL - please use ETL environments of the target DBMS or write an ETL code by your own.
For MSSQL and ORACLE you can use MERGE Tool syntax (and USING clause for Data) for ANSI standards:
MERGE INTO tablename USING table_reference ON (condition)
WHEN MATCHED THEN
UPDATE SET column1 = value1 [, column2 = value2 ...]
WHEN NOT MATCHED THEN
INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...]);
For MySQL there is a different syntax (On Duplicate Key Update Statement):
-- Insert new or merge into existing row.
MERGE INTO system_user target
USING (SELECT 1 AS system_user_id
, 'SYSADMIN' AS system_user_name
, 1 AS system_user_group_id
, 1 AS system_user_type
, 'Samuel' AS first_name
, 'the' AS middle_name
, 'Lamanite' AS last_name
, 1 AS created_by
, SYSDATE AS creation_date
, 1 AS last_updated_by
, SYSDATE AS last_update_date
FROM dual) SOURCE
ON (target.system_user_id = SOURCE.system_user_id)
WHEN MATCHED THEN
UPDATE SET first_name = 'Samuel'
, middle_name = 'the'
, last_name = 'Lamanite'
, last_updated_by = 1
, last_update_date = SYSDATE
WHEN NOT MATCHED THEN
INSERT
( target.system_user_id
, target.system_user_name
, target.system_user_group_id
, target.system_user_type
, target.first_name
, target.middle_name
, target.last_name
, target.created_by
, target.creation_date
, target.last_updated_by
, target.last_update_date )
VALUES
( SOURCE.system_user_id
, SOURCE.system_user_name
, SOURCE.system_user_group_id
, SOURCE.system_user_type
, SOURCE.first_name
, SOURCE.middle_name
, SOURCE.last_name
, SOURCE.created_by
, SOURCE.creation_date
, SOURCE.last_updated_by
, SOURCE.last_update_date );
AND:
-- Insert new or merge into existing row.
INSERT INTO system_user
( system_user_name
, system_user_group_id
, system_user_type
, first_name
, middle_name
, last_name
, created_by
, creation_date
, last_updated_by
, last_update_date )
VALUES
('SYSADMIN'
, 1
, 1
,'Samuel'
,'the'
,'Lamanite'
, 1
, NOW()
, 1
, NOW())
ON DUPLICATE KEY
UPDATE first_name = 'Samuel'
, middle_name = 'the'
, last_name = 'Lamanite'
, last_updated_by = 1
, last_update_date = UTC_DATE();
I would try Scriptella.
Is an open source ETL and script execution tool written in Java, on which you define in a xml file the source and target connections and transformations if needed. Connections can be JDBC or even to text files and there features for batching support. The resulting xml file can be processed programmatically with java, ant or command line.
In their two minute tutorial there are examples for copy tables to another database and working with BLOBs.
Related
I am trying to update 2 tables at the same time where the inserted index of the first should be inserted into the 2nd table.
The sql looks like this:
DECLARE #nrTable table (TXT_nr int)
IF NOT EXISTS (SELECT txt FROM tbl1 WHERE txt = (?))
INSERT INTO tbl1 (txt, new) OUTPUT INSERTED.nr INTO #nrTable VALUES((?), 1)
IF NOT EXISTS (SELECT txt FROM tbl1 WHERE txt =(?))
INSERT INTO tbl2 (TXT_nr, field1, field2)
VALUES((SELECT TXT_nr FROM #nrTable), (?), (?))
WHERE field3 = (?) AND field4 = (?)
I am trying to accomplish this using
this.jdbcTemplate.batchUpdate(sql, batch);
simply concatenating the lines in java using basic strings. This seems to only execute the first statement, though.
Now the reason I don´t want to do this transactionally is that I would have to do it using a loop just inserting one batch-object at a time, because of the ouput-clause. This would result in loads of calls to the sql-server.
Is there any known way to accomplish something like this?
You can't use batchupdate in this way. Please refer to this document http://tutorials.jenkov.com/jdbc/batchupdate.html
And to achieve your goal, if you are using a sequence in sql, then you need to get the new value in java and store it in your query. Like this :
long id = jdbcTemplace.queryForObject("select sequence_name.nextval from dual",Long.class);
It feels like I'm close, but I cannot figure out how to do something like the below in jOOq.
MERGE INTO USER_ASSIGNMENTS ua
USING (
SELECT core_object_id
FROM core_objects
WHERE exists(SELECT *
FROM LKU_CODE lc JOIN LKU_CODE_TYPE lct
ON lc.LKU_CODE_TYPE_ID = lct.LKU_CODE_TYPE_ID AND lct.CODE_TYPE = 'OBJECT_TYPE' AND
lc.CODE = 'PORTFOLIOS'
WHERE lc.LKU_CODE_ID = core_objects.OBJECT_TYPE_ID) AND object_id = 83
) "co"
ON (ua.CORE_OBJECT_ID = "co".CORE_OBJECT_ID AND USER_ID = 24 AND SECTION = 1)
WHEN MATCHED THEN UPDATE
SET create_date = sysdate, created_by = '24', capabilities = 12
WHERE capabilities <> 12
WHEN NOT MATCHED THEN INSERT
(CAPABILITIES, CORE_OBJECT_ID, CREATE_DATE, CREATED_BY, SECTION, USER_ID)
VALUES (5, "co".CORE_OBJECT_ID, sysdate, '24', 1, 24);
The big thing to note is that I'm trying to use the value returned by USING, so I have to alias it and .values() has to accept a field call. I think I can get around the .values() issue using the .values(Collection<?>) call, bundling things, including that field, into a Collection, so I think that I have that part. What concerns me is that I cannot do an .as() call after .using(). If I make the USING query a "table" via .asTable(), supplying an alias, will that let me call the field? Here's kind of what I have at the moment:
Table<Record1<BigDecimal>> usingStatement = readContext
.select(_co.CORE_OBJECT_ID)
.from(_co)
.where(DSL.exists(readContext.select(_lc.fields()).from(
_lc.join(_lct).onKey(Keys.LC_LCT___FK)
.and(_lc.CODE.equal(capability.getObjectTypeCode()))
.and(_lct.CODE_TYPE.equal(LkuCodeTypeLookup.OBJECT_TYPE))))).asTable("sdf");
...
return writeContext
.mergeInto(_ua)
.using(usingStatement)
.on(sectionalConditions.and(_ua.CORE_OBJECT_ID.equal(coidField)))
.whenMatchedThenUpdate()
.set(_ua.CREATE_DATE, time)
.set(_ua.CREATED_BY, creator)
.set(_ua.CAPABILITIES, capabilities)
.where(_ua.CAPABILITIES.notEqual(capabilities))
.whenNotMatchedThenInsert(_ua.CAPABILITIES, _ua.CORE_OBJECT_ID, _ua.CREATE_DATE,
_ua.CREATED_BY, _ua.SECTION, _ua.USER_ID)
.values(capabilities, gcoid, time, creator, section, uuid).execute();
A "straight merge" using dual is simple in jOOq, but I'd like to try to combine that select into the merge to save queries and let the DB do what it does best, so I'm trying not to have to get core_object_id in another query, if possible.
The aliasing really happens on the table (i.e. the select), not on some artefact returned by the USING clause. At least, that's how jOOQ models it. You have already correctly aliased your usingStatement variable. Now all you have to do is dereference the desired column from it, e.g.:
usingStatement.field(_co.CORE_OBJECT_ID);
This will look for the column named CORE_OBJECT_ID in the usingStatement table.
I'm doing at the moment a project for university where I'm using the Play! framework 2.4 together with Ebean.
At the moment I'm trying to realize a live user search where a user can search for other users using a fulltext text input. A possible seach string could be "Michael Lee" or just a email address. Im passing the search string to this method where I try to access the database in a smart way:
public Result searchUser(String sstring) {
String query = "WHERE MATCH (firstname,lastname,email) AGAINST ('" + sstring + "' IN BOOLEAN MODE)";
List<User> resultUsers = User.find
.setQuery(query)
.findList();
String resultString = "";
for (User user: resultUsers) {
resultString += user.getFirstname() + user.getLastname() + "<br>";
}
return ok(resultString);
}
I'm trying to search in the columns firstname, lastname and email for the keyword but play gives me this error:
[PersistenceException: Query threw SQLException:The used table type doesn't support FULLTEXT indexes Bind values:[] Query was: select t0.id c0, t0.firstname c1, t0.lastname c2, t0.birthday c3, t0.email c4, t0.password c5, t0.author c6, t0.points c7, t0.locked c8, t0.last_online c9, t0.date_created c10, t0.date_updated c11 from users t0 where MATCH (t0.firstname,t0.lastname,t0.email) AGAINST ('erik' IN BOOLEAN MODE) ]
Is it possible to tell ebean to create the table users with the required fulltext indexes? Or is there a other smart way to search in a MySQL table for a searchstring in many columns? The searchstring can consist keywords from all columns.
Thanks.
Update:
I tried to add an index like it is mentioned in the ebean api doc but unfortunally it seems like the play ebean version does not support the columnNames keyword (and any others at all...).
#Index(name = "livesearch_index", columnNames = {"firstname","lastname","email"})
So what else can I do? Is it possible to set this index externally maybe in a second evolution file which is not generated automaticly?
I found a solution using the play framework evolutions. Ebean is creating automatically a first evolution file called 1.sql which contains the full creation of the database schema.
For my users Table the UP part looks like this:
create table users (
id bigint auto_increment not null,
firstname varchar(45),
lastname varchar(45),
birthday datetime,
email varchar(60),
password varchar(32),
author tinyint(1) default 0,
points integer,
locked tinyint(1) default 0,
last_online datetime,
date_created datetime not null,
date_updated datetime not null,
constraint uq_users_email unique (email),
constraint pk_users primary key (id)
);
As you can see there is no FULLTEXT index created.
What I did is to create a second evolution file 2.sql. This adds a FULLTEXT index to the users table.
# --- !Ups
create fulltext index livesearch_index on users (firstname,lastname,email);
# --- !Downs
drop index livesearch_index on users;
I know it is not the best solution but it is a working one. I'm still interested how other people fix this problem or which solution do you use for a fulltext search.
Your solution is actually good. This is the proper way to handle schema evolutions.
What I normally do is to have the evolutions plugin enabled during development and then disable it when the application is ready - this way you will end up having only 1.sql - at least until you get another change request and decide to update the schema. More info here: https://playframework.com/documentation/2.4.x/Evolutions
Apart from this there is one other thing that you should consider - databases are bad at searching, especially when it comes down to text searches. Some months ago I was in a similar position as I had the requirement to implement such functionality (see https://softwareengineering.stackexchange.com/questions/301843/elasticsearch-and-relational-database-combination). To spare your time reading it: I ended up using ElasticSearch for handling all my search-related functionality (full text, aggregations, etc.) and I do not regret it! I know that this can be an overkill for your university project but in a real-world scenario you do not want to use your database for full text search.
I'm asking myself if it is possible to SELECT with LAST_INSERT_ID() in WHERE Clause after an batch of INSERTs without getting corrupt data in the tables? I'm thinking of the scenario that multiple users doing the same stuff at the same time. I develop an JSF Application in which this scenario can be possible.
In hard Code my SELECT after INSERTs looks like this:
preparedstatement.addBatch(
"INSERT INTO table1(all the FIELDS)"
+ "VALUES(null, ...);"
);
preparedstatement.addBatch(
"INSERT INTO table2(all the FIELDS)"
+ "VALUES(null, LAST_INSERT_ID(), ...);"
);
preparedstatement = connect.prepareStatement(
"SELECT id FROM table3 WHERE id = LAST_INSERT_ID();"
);
preparedstatement.executeBatch();
resultSet = preparedstatement.executeQuery();
Get I problems with this implementation or is there an better way?
Best Regards
You should be fine, quoting MySQL's documentation:
The ID that was generated is maintained in the server on a
per-connection basis. This means that the value returned by the
function to a given client is the first AUTO_INCREMENT value generated
for most recent statement affecting an AUTO_INCREMENT column by that
client. This value cannot be affected by other clients, even if they
generate AUTO_INCREMENT values of their own. This behavior ensures
that each client can retrieve its own ID without concern for the
activity of other clients, and without the need for locks or
transactions.
MySQL Last_insert_id
I recently rewrote a Java EE web application (running on a MySQL database) to Rails 3.1. The problem now is that the database model of the new application is not the same as the old one because I added, removed and renamed some attributes. The database table names are also different.
Is there a way of migrating this data? The only way I can imagine to do this is writing a stored procedure with many ALTER TABLE and CREATE TABLE statements to update the database to the new model.
Thanks in advanced.
Solution:
I finally used INSERT..SELECT statements in a mysql stored procedure to migrate the data. INSERT INTO new_schema.new_table SELECT FROM old_schema.old_table. I am now considering making a Rake task to call that procedure and doing other stuff.
The only way is to write a script that take the data from the old db and insert thme in the new db. Or you can in some way to connect to the two databases and then make some select and insert query, something like
insert into new_db.table as select old_db.table.field1, ....
or
insert into new_db.table (field_1, field_2) values (select old_db.table.field_1, ...)
In any way, is a manual process, also if can be automated to some extend with a script
Instead of a Store procedure you can try with rails and some sql within the rails console using the information_schema of mysql
sql = ActiveRecord::Base.connection()
old_tables = sql.execute "select table_name from information_schema.tables where table_schema = your_old_schema"
res.each do | old_table|
old_fields = sql.execute "select distinct column_name, data_type from information_schema.columns where table_name='#{old_table}' and table_schema='your_old_schema'"
new_fields = sql.execute "select distinct column_name, data_type from information_schema.columns where table_name='#{old_table}' and table_schema='your_new_schema'"
#compare fields and so on...
end