I have a MySQL database on server #1 and I need to migrate it to server #2. The database is very large and methods like mysqldump or MySQL Workbench Migration didn't work for me, so I wanted to write my own Java application, that would perform the following steps:
get all table names from the source database schema
for each table, it would select a batch of records (let's say 10.000 at a time) and insert them in the corresponding table in destination database. The schema is already prepared with correctly defined tables.
repeat until there are no rows left for current table.
repeat for each table.
The problem is, AFAIK, when using JDBC, it is needed to iterate through ResultSet and specify all column types and names, like this:
while (resultset.next()) {
System.out.println(resultset.getString("Col 1"));
System.out.println(resultset.getString("Col 2"));
System.out.println(resultset.getString("Col 3"));
System.out.println(resultset.getString("Col n"));
}
I want to do this for all tables and all their columns without specifying their names and types manually. I can't just type manually all these columns, as I have 150 tables and each of them has like 10-50 columns.
Is there any general way how to do this? Maybe taking advantage of the fact that both source and destination schemas are the same (same tables with same column names/types and same foreign keys)?
You can use ResultSetMetaData with ResultSet to get the columnNames, columnCount and few other details.
ResultSet resultSet = ps.executeQuery();
ResultSetMetaData resultSetMetaData = resultSet.getMetaData();
while (resultSet.next()) {
for (int i = 1; i <= resultSetMetaData.getColumnCount(); i++) {
System.out.println(resultSet.getString(i));
}
}
if you need to fetch the type of Column also to make the decision you could use Switch conditions based on the type returned from resultSetMetaData.getColumnTypeName(i) for using something like
resultSet.getString()
resultSet.getInt()
resultSet.getBoolean()
resultSet.getDate()
Many more
On top of #koushlendra answer, you can also use these methods for additional informations(if needed)
resultSetMetaData.getTableName(int column) Returns the column’s table name.
resultSetMetaData.getSchemaName(int column) Returns the name of the schema of the column’s table.
Related
I have a webpage which accepts a query from the user, and passes it to a servlet. This servlet calls my java functions which run the query and get the resulting table. I now have this result set, but I wish to display this result set in a table on my webpage. The problem is I don't know what the query is before hand or the schema of the table it will run on. I wanted to know if there is a way to dynamically present the result set contents on a table on my webpage. Since I don't know the schema, don't have the option of creating a java model object. I'm using HTML, JSP and obviously Java
I hope my memory about JSP is good. I suppose you have do a select with '*' to have all columns. In the result you can have the number of columns, so know how much column you have to create, and a "for" on column names, give you the column names. With this you should be able show select result without knowing in advance the column name in table. May be its only possible only on result set from JDBC. Sorry if not apply for you
Use getMetaData() on your resultset object to get the details of resultset like number of columns and names of the column. Using size() method get the size of the resultset.
Next using the size() you can compute how many rows should be there in your html table and using resultset.getMetaData().getColumnCount() you can compute number of columns in your html table. You can use the column names to be headings of your table columns.
Let me know if you need further help.
sample code retrieve metadata from you resultset.
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
ResultSetMetaData rsMetaData = rs.getMetaData();
int numberOfColumns = rsMetaData.getColumnCount();
System.out.println("resultSet MetaData column Count=" + numberOfColumns);
for (int i = 1; i <= numberOfColumns; i++) {
System.out.println("column MetaData ");
System.out.println("column number " + i);
// get the column's name.
System.out.println(rsMetaData.getColumnName(i));
}
I'm trying to get the name of the columns from the metadata of a ResultSet but in Hive I get column names if the form of table_name.column_name and I'd like to have only column_name (similarly to what I get from MySQL for example). Is it possible ?
Yes it is possible.
There are two ways by which it can be done:
1.) Using Hive JDBC connection(which you are looking for).
2.) Using HiveMetastoreClient.
Here are code snippets for the above two approaches:
First Approach:
ResultSet resultSet = <custom_class_for_hive_jdbc_connector>.executeQuery("DESCRIBE <TABLE_NAME>");
ResultSetMetaData metaData = resultSet.getMetaData();
while (resultSet.next()) {
System.out.println(" Column names : "+resultSet.getString(metaData.getColumnName(1)));
}
In this approach we need to create a resultsetmetadata object and using this object we can get the details of the query output. In your case "describe table"
Second Approach:
HiveMetaStoreClient metastoreClient = null;
HiveConf hiveConf = new HiveConf();
metastoreClient = new HiveMetaStoreClient(hiveConf);
List<FieldSchema> list = metastoreClient.getFields(<database_name>,<table_name>);
for (int i = 0; i < list.size(); i++) {
System.out.println("Column names : "+list.get(i).getName());
}
metastoreClient.close();
In this approach we are not running any query, we are simply connecting to the HiveMetastore and from there we are getting the details of a given table under given database.
Hope it helps...!!!
To prevent the Hive table names from being prepended to the column names, use this property setting in the Hive-site.xml file.
<property>
<name>hive.resultset.use.unique.column.names</name>
<value>false</value>
</property>
beginning with Hive .13, a new Hive property called hive.resultset.use.unique.column.names was created. The default value of this property is TRUE.
If the default value is used, the Hive table name is prepended to all column names. This is a change in behavior from previous versions of Hive.
This is also very important if you are trying to use Spring Jdbctemplate with Beanpropertyrowmapper to do the camel case to underscore format conversion for you.
Usually to sync a table with in the network I can write a simple sql query and run it to sync the tables, but what if the source database is different(could be db2, mssql, mysql, oracle - from where I can get the data in to destination table).
How should I go about writing the code in java to achieve this?. I know I can create dblinks in java to fetch data from external network database but dblinks works only with database of same type. I am not able to figure out the implementation in my head. Please point me in the right direction so I can take baby steps and learn the same as I am new to java
You can do this with plain JDBC. Code example, not tested:
void migrate() throws SQLException {
Connection connA = DriverManager.getConnection("jdbc:oracle:thin:#myhost:1521:orcl");
Connection connB = DriverManager.getConnection("jdbc:mysql://192.168.1.33/test");
PreparedStatement stmA = connA.prepareStatement("select * from product where 1=1");
PreparedStatement stmB = connB.prepareStatement("insert into prod values(?,?,?,?)");
ResultSet rs = stmA.executeQuery();
while (rs.next()) {
for (int i = 0; i < rs.getMetaData().getColumnCount(); i++) {
stmB.setObject(i + 1, rs.getObject(i + 1));
}
stmB.executeUpdate();
}
connA.close();
connB.close();
}
Data Migration If you want to migrate data from one database to another database of different types,
Suppose you have A and B two different database, you have to export data from A database in XML format, table name is XML parent node (create structure as per your database).
Then parse that XML's and create objects of each table present in A, after that use any XML parser which will parse your XML, using hibernate save() api put each object in another database manage if tables has relationships in them then insert master table data first then insert child items data.
By this approach you no need to be dependent on destination (B) database each time, with the help of hibernate you can easily change your database.
I've got an embedded Derby Database in my java application, and have multiple table's (that are created and deleted, so nothing is set in stone). I wanted to be able to return a list of names of all the tables currently in the database as I have to display the list in the application as well as get all the information from them.
Easiest way to do so? I don't need code just a method or methods. I'm a terrible google-fu user.
Currently my code works by grabbing a ResultSet from a specific table name entered, but it's only for testing purposes and I need to be able to display the full list of tables I have.
EDIT: My current workaround is actually different than posted. I simply have another table that holds all the table names created and updates when one is created/deleted. Obviously not the best approach but it works for me right now.
DatabaseMetaData metaData = connection.getMetaData();
ResultSet resultSet = metaData.getTables(null, "schenaName", "%" ,new String[] {"TABLE"} );
while (resultSet.next()) {
System.out.println(resultSet.getString(3));
}
Adding new answer:
Connection connection = getDBConnection();
DatabaseMetaData dbMetaData = connection.getMetaData();
//getting catalogs for mysql DB, if it is not working for your DB, try dbMetaData.getSchemas();
ResultSet catalogs = dbMetaData.getCatalogs();
while(catalogs.next()){
String catalogName = catalogs.getString(1);
//excluding table names from "mysql" schema from mysql DB.
if(!"mysql".equalsIgnoreCase(catalogName)){
ResultSet tables = dbMetaData.getTables(catalogName, null, null, null);
while(tables.next()){
System.out.println(catalogName + "::"+tables.getString(3));
}
}
}
Using metadata is the (somewhat) more portable solution. Note that you don't need the catalog stuff with Derby, as there are no catalogs. You can issue dmd.getTables(...) directly with null for the catalog. If all the tables you track are in a single schema, (and there aren't any other tables in that schema), getTables(null, "schemaName", null, null) should do the trick.
If need more fancy querying and you're not concerned about portability, you can check out
the dataBaseMetaData tool which gives you access to metadata as tables so that you can perform joins and other sophisticated queries on them.
Try this:
select tableName from sys.systables
You should get all the tables your system.
I am using an access mdb file as my database . Inside the table in my database I use id as autonumber. Now. I wanted to know how can i get the next generated auto number field in access database to put in java JTextField!
To get next number you can insert a "blank" row and use the number that was generated for it.
If your insert statement is stmt:
int nextKey = 0;
ResultSet keys = stmt.getGeneratedKeys();
if (keys.next())
{
nextKey = keys.getInt(1);
}
See statement.getGeneratedKeys()
Alternatively you could generate new id number with
SELECT MAX(id)+1 FROM yourTable
though this doesn't guarantee that the number will stay unused (by some other query) before you do anything with it.