I am developing an application and i need to migrate data from netezza database into another database. For that i need to write an application in C++ or java to read from netezza database. Can anyone guide if there are any native API available for C++ or do we have to use ODBC. If ODBC is the way to go can anyone guide me how.
I don't think there is an API for your use case. There is a C++ API for Analytic Executables (user defined functions like special aggregates and table functions, but these aren't going to get you a result set by themselves). I've previously just made an ODBC connection to get to a Netezza instance and execute queries (albeit within C#, not C++). For data migration, especially between two disparate databases, I've typically written the data to a flat file of one sort or another (.csv, tab delimited text file, etc.) and then consumed that file again within my program and pushed that data to the table within the destination db (using a second ODBC connection, and with Bulk Copy (BCP) for SQL server if this is the destination db).
MSDN has some example code for connecting to a db via ODBC using C++ HERE.
Related
We have source Oracle database, where we have a lot of tabels (let say 100) which we need to mirror to target database. So we need to copy data increments periodically to another db tables. The target database is currently Oracle, but in the short future it will be probably changed to a different database technology.
So currently we can create a PL/SQL procedure which will dynamically generate DML (insert, update or merge statements) for each table (assuming that the source and target table have exactly the same attributes) from Oracle metadata.
But we would rather create some db technology independent solution so when we change target database to another (e.g. MS SQL or Postgres), then we will no need to change whole logic of data mirroring.
Does anyone have a suggestion how to do it differently (preferably in java)?
Thanks for every advice.
The problem you have is called CDC - continuous data capture. In case of Oracle this is complicated because Oracle is usually asking money for this.
So you can use:
PL/SQL or Java and use SQL to incrementally detect changes in data. IT requires plenty of work and performance is bad.
Use tools based on Oracle triggers, which will dects data changes and pushes them into some queue.
Use tool which can parse content of Oracle Archive logs. These are commercial products: GoldenGate (from Oracle) and Shareplex (Dell/EMC/dunno). GoldenDate also contains Java technology(XStreams) which allows you to inject Java visitor into the data stream. Those technologies also support sending data changes into Kafka stream.
There are plenty of tools like Debezium, Informatica, Tibco which can not parse Archived logs by themself, but rather they use Oracle's internal tool LogMiner. These tools usually do not scale well and can not cope with higher data volumes.
Here is quite article in as a summary. If you have money pick GoldenGate or Shareplex. If you don't pick Debezium or any other Java CDC project based on Logminer.
As the application gets complicated, one thing that change a lot is the queries, especially if they are complex queries. Wouldn't it be easier to maintain the queries in the db rather then the resources location inside the package, so that it can be enhanced easily without a code change. What are the drawbacks of this?
You can use stores procedures, to save your queries in the database. Than your Java code can just call the procedure from the database instead of building a complex query.
See wikipedia for a more detailed explanation about stored procedures:
https://en.wikipedia.org/wiki/Stored_procedure
You can find details about the implementation and usage in the documentation of your database system (MySql, MariaDb, Oracle...)
When you decide to move logic to the database, you should use a version control system for databases like liquibase: https://www.liquibase.org/get-started/quickstart
You can write the changes to you database code in xml, json or even yaml and check that in in your version control system (svn, git...). This way you have a history of the changes and can roll back to a previous version of your procedure, if something goes wrong.
You also asked, why some people use stored procedures and others keep their queries in the code.
Stored procedures can encapsulate the query and provide an interface to the data. They can be faster than queries. That is good.
But there are also problems
you distribute the buisiness logic of your application to the database and the programm code. It can realy be troublesome, if the logic is spread through all technical layers of your applicaton.
it is not so simple anymore to switch from a Oracle database to a MariaDb, if you use specific features of the database system. You have to migrate or rewrite the procedures.
you have to integrate liquibase or another system into you build pipeline, to keep track of you database changes.
So it depends on the project and it's size, if either of the solutions is better.
I'm trying copy table from one database to another(On different machines), and using JDBC Template to execute query, but this request is specific to Oracle:
COPY FROM username1/passwd1#//192.168.3.17:1521/PROD_SERVICE to username2/passwd2#//192.168.4.17:1521/SANDBOX_SERVICE INSERT TABLE_C (*) USING (SELECT * FROM TABLE_C);
And I get error:
Caused by: java.sql.SQLSyntaxErrorException: ORA-00900: invalid SQL statement
How can I use specific to Oracle syntax in JDBC?
Like some of the comments have already clarified, COPY is a sqlplus command, and it has been deprecated for a while. You cannot use it inside JAVA, because this command is not part of the SQL engine, it's just a kind of additional feature available only in sqlplus. It is still available, but only for backwards compatibility.
If you want to copy a table using Java, you need to understand first some things:
Java, or any external engine for that matter, can't connect at the same time to both databases. Either it connects to one or to the other.
You need to have a kind of bridge between both databases, so that your Java program is only acting as trigger.
Copying tables between databases is something related to the database, so you should think in using tools provided by your database engine. You have some options, like Datapump or RMAN, although I consider Datapump the best suitable for your scenario.
However, if you insist in using Java, first you need to have a database link between both databases. Then you can use Java to invoke an insert from one database to another.
https://docs.oracle.com/database/121/SQLRF/statements_5006.htm#SQLRF01205
If you don't want to depend on thsnames entries in the server, here an example of database links:
CREATE DATABASE LINK to_my_remote_user
CONNECT TO remote_user IDENTIFIED BY password
USING '(DESCRIPTION=
(ADDRESS=(PROTOCOL=TCP)(HOST=remote_server)(PORT=remote_port))
(CONNECT_DATA=(SERVICE_NAME=remote_service_name))
)';
Once you have the dblink created, then you can connect from java to the database where the link is available and copy the data to the remote database
INSERT INTO remote_user.remote_table#to_my_remote_user
select * from local_user.local_table ;
Important: Normally dblinks are not allowed on Production systems, because they increase security risks. Also remember that DDL operations over a database link require an extra step, such as using the procedure DBMS_UTILITY.EXEC_DDL_STATEMENT#dblink('create table ...);
Another option outside of Java is using SQL Developer copy feature. Although I only recommend it for small tables. If you want to use it with big tables, it will probably hang. You can read here an good example :
copy from one database to another using oracle sql developer - connection failed
I have a web application written in java and it is about course scheduling. I used file system to store my data. I serialized my collentions (objects in them) and collect them into an serialalized object. Every user has a few file for system data and backup or prepared table data. My program is reading that file (only system data) when user entered and writing it to that file when user exit.
Object design is very complex and there are many many-to-many relations between my objects. Writing data of this program requires so much join operation and well design schema in SQL. Furthermore a user generally have thousands of information containing hundreds of courses, teacher, rooms and even thousands of constraints. Beside of those, there is no connection between users.
In this program I do not see any advantage of using sql. Even my system is running very fast. However lots of developers which do not know structure of my program advised to me to carry my database system to SQL.
What is the advantage of SQL against file system (with serialized objects) especially for my extraordinary system?
To name just a few advantages of using a SQL database in general:
Scalability: You can move your data to a separate server(s) to reduce the load on your application server.
Accessibility: Should you wish to release an extension of your program in the future, that program can run independently and still access the same data without you having to worry about locking files etc.
Security: Database systems allow you to configure "privileges" as who is allowed to write data and who is allowed to read data.
Portability: Should you decide in the future to upgrade your program to a different technology, the SQL language is an industry standard that can relatively easily be ported from one database vendor to another. You can also port your main application independently of your database to another technology given both technologies has drivers / libraries / connection / plug-ins / what-ever-you-want-to-call-it to communicate with your database.
Backups: You can easily run a backup while your application is online. With a files, you might run into locking issues or alternatively have to shut down the application to do a backup.
Human resources: You can employ a database administrator to take care of your databases and reduce the responsibility of your developers/engineers so that they can focus on other tasks.
Monitoring: You can monitor or inspect your data, while your application is live.
However, usually application that uses a SQL database was designed with a relational data-structure in mind from the beginning. Without fully understanding your application, it sound like you have a very complex data structure and that it might not be that easy to migrate to a SQL database. I would recommend that you also have a look at a NoSQL database which
allows large volumes of structured, semi-structured, and unstructured data
encourages object-oriented programming
easy to use
and is much more flexible than SQL
I have developed a small swing desktop application. This app needs data from other database, so for that I've created an small process using java that allows to get the info (using jdbc) from remote db and copy (using jpa) it to the local database, the problem is that this process take a lot of time. is there other way to do it in order to make faster this task ?
Please let me know if I am not clear, I'm not a native speaker.
Thanks
Diego
One good option is to use the Replication feature in MySQL. Please refer to the MySQL manual here for more information.
JPA is less suited here, as object-relational-mapping is costly, and this is bulk data transfer. Here you probably also do not need data base replication.
Maybe backup is a solution: several different approaches listed there.
In general one can also do a mysqldump (on a table for instance) on a cron task compress the dump, and retrieve it.