I am working on database migration tool in java. The tool is copying database tables with their data's to the destination database. But I want it to work on different databases. Copy from mysql and create in derby etc. With JDBC, we can gather enough information about the table and its columns. But I am going to ask this, if I can recreate tables on java with sql free. I mean different databases have different data types and some times they differs at sql syntax. So can JDBC or any other library (can be open source) do this job at an easy and global way?
Apache's DdlUtils is done what I need. When I am searching about crossdb found it, and it is very useful yet powerful. It can generate a database from scratch, just with the parameters. Or it can grab existing database table definitions, also with index definitions. You can use delimiter if you want (it is a deadly important option for me to use Apache Derby). You can just print out these definitions or apply them directly to source database (I haven't tried the second one yet). It translates definitions for the selected database. But one big problem is there is no good tutorial about how to start using it. I searched through the packages to find a good place to start. Here is what i have achieved, a sample code to generate full database table create sql.
DerbyPlatform dp = new DerbyPlatform();
dp.setDelimitedIdentifierModeOn(true);
Database dbs = new Database();
DerbyModelReader dmr = new DerbyModelReader(dp);
Database test = dmr.getDatabase(conn, "MyDBTest");
DerbyBuilder db = new DerbyBuilder(dp);
String testSqlDerby = dp.getCreateTablesSql(test, true, true);
System.out.println(testSqlDerby);
System.out.println("\n\n\n\n");
MySql50Platform mp = new MySql50Platform();
mp.setDelimitedIdentifierModeOn(true);
MySqlBuilder mb = new MySqlBuilder(mp);
String testSqlMysql = mp.getCreateTablesSql(test, true, true);
System.out.println(testSqlMysql);
I'm not aware of JDBC having a generic facility to do this. You probably have to make a utility library that generates the SQL for table creation.
Start with one that does ANSI SQL and test it on as many platforms as you intend to support. Remember, Java is still write-once, debug everywhere so you will need to test this on any platform you intend to support the system on. Subclass the generator if you have to make dialectic variations of the create statement for any of your platforms.
Try SchemaCrawler, which is also open source, and written in Java. It provides programmatic access to database metadata, and also allows scripting in JavaScript.
http://schemacrawler.sourceforge.net/
try to see what HIBERNATE provides for migration.
I know that H can generate model objects from database schema and a schema from model objects. So maybe you can reuse some parts from HIBERNATE. They have the notion of DIALECT that does exactly what you are saying: defining specifics of a db implementation.
Related
I am writing a JDBC program to fetch some database meta data information and as part of that I want to query all the indexes that are there in a given schema.
I had a look at some JDBC API and from DatabaseMetaData interface, can use methods like getTables to get all the tables for a given schema. I am trying to find something similar (or write using a combination of some API) to get information like all the indexes,views etc on a schema. Is there a way to get it? For ex, for index there is a method - getIndexInfo but for each of the table in a schema, I need to call this method. My database is db2.
I would use the DB2 Catalog Views to get the information.
As an example, if you want all of the indexes for a table, you'd use a query like this (I'm assuming you're using DB2 on Linux/Unix/Windows here):
SELECT *
FROM SYSCAT.INDEXES
WHERE tabname = #tablename
AND tabschema = #schema
ORDER BY indname
I did the following after trying some approaches
1. Wrote a wrapper around JDBC calls to simplify my work.
2. Queried the syscat schema like
select tabname from syscat.TABLES where tabschema = ?
Wrote some java utilities to compare the 2 sets of results returned by the 2 schema and also did some manual comparison.
If I find a better solution, will post it. Thanks a lot for all the help.
Getting data from Syscat schema is not correct. in ZOS environment Syscat may be or may not be present because while installation you have options not to install the Syscat schema. so better use Sysibm schema.
I have a use case where in I need to read rows from a file, transform them using an engine and then write the output to a database (that can be configured).
While I could write a query builder of my own, I was interested in knowing if there's already an available solution (library).
I searched online and could find jOOQ library but it looks like it is type-safe and has a code-gen tool so is probably suited for static database schema's. In the use case that I have db's can be configured dynamically and the meta-data is programatically read and made available for write-purposes (so a list of tables would be made available, user can select the columns to write and the insert script for these column needs to be dynamically created).
Is there any library that could help me with the use case?
If I understand correctly you need to query the database structure, display the result to via a GUI and have the user map data from a file to that structure?
Assuming this is the case, you're not looking for a 'library', you're looking for an ETL tool.
Alternatively, if you're set on writing something yourself, the (very) basic way to do this is:
the structure of a database using Connection.getMetaData(). The exact usage can vary between drivers so you'll need to create an abstraction layer that meets your needs - I'd assume you're just interested in the table structure here.
the format of the file needs to be mapped to a similar structure to the tables.
provide a GUI that allows the user to connect elements from the file to columns in the table including any type mapping that is needed.
create a parametrized insert statement based on file element to column mapping - this is just a simple bit of string concatenation.
loop throw the rows in the file performing a batch insert for each.
My advice, get an ETL tool, this sounds like a simple problem, but it's full of idiosyncrasies - getting even an 80% solution will be tough and time consuming.
jOOQ (the library you referenced in your question) can be used without code generation as indicated in the jOOQ manual:
http://www.jooq.org/doc/latest/manual/getting-started/use-cases/jooq-as-a-standalone-sql-builder
http://www.jooq.org/doc/latest/manual/sql-building/plain-sql
When searching through the user group, you'll find other users leveraging jOOQ in the way you intend
The setps you need to do is:
read the rows
build each row into an object
transform the above object to target object
insert the target object into the db
Among the above 4 steps, the only thing you need to do is step 3.
And for the above purpose, you can use Transmorph, EZMorph, Commons-BeanUtils, Dozer, etc.
I've done all my database development for the past few years in Ruby, mostly using ActiveRecord. Now I'm stuck using Java for a project, and it feels so verbose and hamfisted, I'm wondering if I'm doing things wrong.
In an ORM paradigm, if I want to insert into related tables, I'd so something like
# Joe Bob got a new car
p = Person.find_or_create_by_name("Joe Bob");
Car.new({:make=>"Toyota", :plate=>"ABC 123", :owner=>p});
In Java, at least using JDBC directly, I'm going to have to do the Person lookup by hand, insert if it doesn't exist, then create the Car entry.
Of course, in real life, it's more than just 2 tables and the pain scales exponentially. Surely there's a better way?
You can use ORM solutions for Java - there are various solutions available.
Links worth looking at:
Hibernate - http://www.hibernate.org/ - probably the leading Java ORM solution
SO Question - Hibernate, iBatis, Java EE or other Java ORM tool
Having said that, I've usually found that for complex applications ORM frequently causes more trouble than it is worth (and yes, this does include Ruby projects with Activerecord). Sometimes it really does make sense to just get at the data directly via SQL rather than attempt to force on object-oriented facade on top of it.
The better way is learn SQL! The ORM you like so much writes SQL for you behind the scenes.
So you can make a quick helper function that tries to select the record, and if it doesn't exist creates it for you.
In MySQL you can use INSERT IGNORE ..... which will insert the row only if it doesn't exist.
And here is a special bit of SQL you may like (MySQL only):
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE id=LAST_INSERT_ID(id), c=3;
This tries to insert the record, if it doesn't exist it returns the auto_increment like usual that you retrieve in your program.
But: If it does exist then it updates it - (only c is set to update in that case), but the cool part is it sets the LAST_INSERT_ID() just like it would on an insert.
So either way you get the ID field. And all in a single bit of SQL.
SQL is a very nice language - you should learn it and not rely on the psudo-language of orm.
If you are using JDBC you need to lookup yourself and create person if it does not exist. There is no better way if you use JDBC.
But you can use Hibernate, it will help you reduce writing the O-R mapping yourself and reduce the boilerplate.
As you come from Ruby and If you find it painful to write all the SQL queries, JDBC boilerplate then the better way is to use ORM. I recommend one of the following,
Hibernate
JPA (If you want to change the ORM implementation then use JPA)
Sormula contains an active record package. The save method will update an existing record or insert if no record exists.
See the active record example on web site.
Also see org.sormula.tests.active.SaveTest.java within the project:
SormulaTestAR record = new SormulaTestAR();
record.attach(getActiveDatabase()); // record needs to know data source
record.setId(8002);
record.setType(8);
record.setDescription("Save one AR 2");
record.save();
Looks like I'm late here, however, ActiveJDBC will do what you want in Java:
Person p = Person.findOrCreateIt("name", "Joe Bob");
Car car = Car.createIt("make", "Toyota", "plate", "ABC 123", "owner", p);
There is a ton more it can do, check out at: http://javalite.io/
I can create a table and its columns in Java by using the statement:
CREATE TABLE table_name(column1 int, column2 double, etc...)
What I would like to do is to add descriptions to each of these columns with an appropriate statement, I found a stored procedure sp_addextendedproperty that looks like it can be used to accomplish this I just have no idea how to use it in java with jdbc.
Are you creating the table dynamically at runtime (e.g. as part of your application) - perhaps that's even user-driven? If that's the case, you already have that "documentation" (column comments) somewhere and I doubt the utility of adding them to SQL Server.
But if you're just trying to automate your build, take a look at LiquiBase. It's a pretty decent DB change management system that uses XML as backbone. It's written in java and integrates well with Hibernate (useful if you ever decide to use ORM instead of straight JDBC).
Update: If you do decide to go forward with calling stored procedure via JDBC, I would strongly recommend using CallableStatement to invoke it. Dynamically building SQL queries in the application should be avoided if possible.
There are a number of ways to call a stored procedure (essentially, preparing the statement and binding the variables, or sending a string of SQL), but the simplest is to just send rhe SQL statement
exec sp_addextendedproperty list, of, arguments, the, sp, needs;
Skipping your try/finally boilerplate, and assuming connection is a java.sql.Connection, that's:
connection
.createStatement()
.execute( "exec sp_addextendedproperty arguments;");
But ChssPly76 has a good point: doing this from Java isn't a good idea (unless you're developing some database manager in Java).
For a university assignment I have been assigned I have a Prize object which contains either text, image, or video content. I would like to persist this information into a BLOB field within an Apache Derby database (which will be running on a low powered PDA). How can I add this data to the database?
Thanks in advance.
In this article Five Steps to Managing Unstructured Data with Derby
you can read how to do this.
It describes how to insert binary data into a column with the BLOB datatype in Apache Derby using JDBC.
I assume you'll be connecting via JDBC. If so, simply write your SQL and look at the setBlob method of a PreparedStatement. Should be pretty straightforward.
Serialization is the easy way to do it, however if possible you could make it look like a real database table with a structure containing id (bigint), datatype (smallint), creationdate (date) and data (blob) and specifically make the client code to save the object's data there. This way you could do searches like "get all video prizes created between January 1st 2008 and January 15th 2009" and it wouldn't break down old data if your class would change too much for the serialization to stop working.
This sort of solution would be easy to extend in the future too if there would be need for it; I understand this is a school assignment and such need most likely won't ever surface but if your teacher/professor knows his stuff, I bet he's willing to give an extra point or two for doing this excercise in this way since it takes a bit more time and shows that you can take the steps to prepare in advance for coping in the everchanging landscape of software development.
If you are using Netbeans (I assume Eclipse has similar functionality) you can setup your database schema and the create new Java entity classes from the database and it will generate the appropriate JPA classes for you.
http://hendrosteven.wordpress.com/2008/03/06/simple-jpa-application-with-netbeans/
This is nice as it allows you to focus on your code rather than the database glue code.
The best solution , is to use Derby, because it keep being a multi platform app developed via Java.