I need to copy all the data from MS SQL server to MySQL server. I am planning to use the Quartz scheduler to perform this. This scheduler will run every night and move the data from MS-SQL-Server to MySQL server. Can anyone please tell if this is fine or is there any other better way to do this?
Update:
I need to transfer only one table with 40 columns (from MS SQL server to MySQL)
I wouldn't involve java unless I absolutely had to: java would be adding no value but would be adding extra complexity.
This is a "DBA" type task that belongs in a script scheduled with cron tab.
If I was implementing it, I would export the source database as an SQL script then import it by running it on the target.
SQL Server Management Studio's "Import Data" task (right-click on the DB name, then tasks) will do most of this for you. Run it from the database you want to copy the data into.
If the tables don't exist it will create them for you, but you'll probably have to recreate any indexes and such. If the tables do exist, it will append the new data by default but you can adjust that (edit mappings) so it will delete all existing data.
I use this all the time and it works fairly well.
by- david
I would recommend you to use http://www.talend.com for tasks like that.
UPDATE
Talend Open Studio for Data Integration is Opensource, there are some other features which are propietary details here
As PbxMan said, I would use an ETL, but I recommed Pentaho (http://wiki.pentaho.com/display/EAI/Spoon+User+Guide) which I think is far easier for such simple jobs
I agree with #bohemian - running a job to transfer a single table every night sounds like a great candidate for a cron job (or "scheduled task" if on Windows). Using a framework like Quartz for this task seems like overkill.
There are many solutions for moving data from SQL Server to MySQL, others have listed great solutions such as Talend. If it would benefit you to only transfer certain columns (for example, to avoid leaking PII), my product SQLpipe might help, as it transfers the result of a query, not an entire table.
Here is a blog post showing how to use the product to transfer data from SQL Server to MySQL.
Related
Here is my requirement:
My java service is continuous running process. When the java service start, it loads the configuration from one of the table in MySql database.
When there are any changes (can be insert/update/delete, outside the java service also possible), java service should reloaded the configuration immediately.
I saw a post in stackoverflow which is for Oracle database, but I am looking for Mysql database.
I can able to achieve this using separate thread, which polling the database table with some interval. But, polling for change is a not a good way to do. Instead, I am looking for any Watcher/Listener which will notify when there is any change in the MySql table.
Can anybody help me to achieve this?
Note: If this question is already answered somewhere, please post the link in the comment, I will remove this question.
You want to do a Change Data Capture system. Perhaps you could use one based on Kafka.
You can use kafka-connect with Debezium on it. This connector first snapshot your table, then read the mysql binlog to be able to have a consistent kafka topic with insert/modify/delete on the table.
If you don't want use it, perhaps you can fork the code to use the same thing as they do, just modifying the output to be your application listener.
We have a web-based application in dev phase where we use Spring 5, JPA(Hibernate) and Postgresql 9.4
Till this moment we were using one instance of the posgresql db for our work. Basically, we don't have any schema generation script and we simply were updating the db if we needed some new table, column etc. For the Hibernate we were generating classes from the db.
Now when we have some amount of test data and each change in the db brings a lot of trouble and confusion. We realized that we need to create and start maintaining some schema generation file along with some scripts which generate test data.
After some research, we see two options
Create two *.sql files. The first will contain the schema generation script the second one SQL to create test data. Then add a small module with a class which will execute the *.sql files using plain jdbc. Basically, we will continue developing and whenever we made some changes we quickly wipe->create->populate the db. This approach looks the most appealing to us at this point. It quick, simple, robust.
Second is to set up some tool which may help with that e.g. Liquibase
This approach also looks good in terms of versioning support and other capabilities. However, we are not in production yet, we are in an active development phase. We don't have much of the devs who do the db changes and we are not sure how frequently we will update the db schema in production, it could be rare.
The question is the following. Would the first approach be a bad practice and applying the second one will give the most benefits and it worth to use it?
Would appreciate any comments or any other suggestions!
First approach is NOT a bad practice, until this generation. But it will be considering the growth of tools like Liquibase.
If you are in the early or middle of the Development Phase, go ahead with LiquiBase, along with Spring Data. Contrarily, in the closing stages of the Development Phase, Think you real need for it.
I would suggest second approach as it will automatically find the new script as you add and execute the script on startup. Moreover, when you have tools available like liquibase and flyway why reinvent the wheel ?.
2nd approach will also reduce the un-necessary code for manually executing the *.sql files. Moreover this code also needs testing and if updated can be error prone.
Moreover 1st approach where you write manual code to execute script also has to check which scripts needs to be executed.. If you already has existing database and you are adding some new scripts you need to execute those new scripts only. These things are taken care of automatically with 2nd approach and you don't need to worry about already executed script being executed again
Hope this answers your concern. Happy coding
I am using Spring 4.1.6, and I have my service working fine with Hibernate. In the root of the project I've got my schema.sql which is being run every time I run the server. The problem is that first time I ran the server, I put some data in db, and when I restarted it, the script was executed again and I lost all that data that I loaded before restart.
So, I think that I have two options two solve this problem:
Edit sql script to execute all queries just in case they do not exist (which would be more laborious since I have to edit the script every time I export my db)
Tell hibernate, by some way, to execute sql script just in some cases. That would be great if there existed some config that executes the script just in case the data base doesn't exist.
Do you know if this is even possible? Thanks in advance.
It sounds like this is the perfect use-case for a tool called Liquibase. This is basically a version control tool for your database which allows you to define changes to your schema and/or data and ensures that these changes are only applied once.
It's incredibly useful if multiple people are changing the same database schema and ensures that your database is always valid for the version of the code that you have checked out/released etc.
Here is my scenario:
I have a java application that reads data from a table T1 of database D1, processes it and puts it in another table T2 of another database D2. This happens real time, i.e., as and when a record is inserted or updated in table T1, the application will pick the data, process it and pushes it to the destination table. I wish to monitor performance of this application using a testing(preferrably JUnit) and/or performance framework. In my test case I wish to have following
Insert and update a fixed number of records for fixed time at fixed intervals on table T1 of database D1.
After a fixed time, either check the number of records that are present in T2 of database D2 or look for existence of a specific record.
The tests that I wish to create should be
Database agnostic
Provide results that can show trends and be configurable with a CI tool like Jenkins
So, my question is, what is the best way to test this kind of scenario? Are there any available tools that will help me achieve this?
Database agnostic
In order to achieve that I would suggest to use simplest possible SQL and some low-level JDBC abstraction layer:
DBUtils
The Commons DbUtils library is a small set of classes designed to make
working with JDBC easier. JDBC resource cleanup code is mundane, error
prone work so these classes abstract out all of the cleanup tasks from
your code leaving you with what you really wanted to do with JDBC in
the first place: query and update data.
MyBatis
MyBatis is a first class persistence framework with support for custom
SQL, stored procedures and advanced mappings. MyBatis eliminates
almost all of the JDBC code and manual setting of parameters and
retrieval of results. MyBatis can use simple XML or Annotations for
configuration and map primitives, Map interfaces and Java POJOs (Plain
Old Java Objects) to database records.
Both will do the trick for you. With good attention to details you'll manage to provide flexible enough solution and test as many databases as you want.
Provide results that can show trends and be configurable with a CI tool like Jenkins
Define several KPIs and make sure you can get all values periodically. For example you can measure a throughput (records per second). Export data periodically (as CSV or properties for example) and use PlotPlugin for visualization:
You can also check relevant question: How do I plot benchmark data in a Jenkins matrix project
Proper testing
Please make sure your testing strategy is well defined and you will not miss anything:
Load testing
Stress testing
I have developed a small swing desktop application. This app needs data from other database, so for that I've created an small process using java that allows to get the info (using jdbc) from remote db and copy (using jpa) it to the local database, the problem is that this process take a lot of time. is there other way to do it in order to make faster this task ?
Please let me know if I am not clear, I'm not a native speaker.
Thanks
Diego
One good option is to use the Replication feature in MySQL. Please refer to the MySQL manual here for more information.
JPA is less suited here, as object-relational-mapping is costly, and this is bulk data transfer. Here you probably also do not need data base replication.
Maybe backup is a solution: several different approaches listed there.
In general one can also do a mysqldump (on a table for instance) on a cron task compress the dump, and retrieve it.