I have a situation wherein as part of a online transaction, I have to save some data into other database, a slight latency (few seconds) in updating the other database is fine. Now since both databases are Oracle, I have below 3 options, I need some insight as which one is better.
Oracle Database Links: Wherein I convert the SQL into PL/SQL and make my database take care of writing into another Oracle based database for DEV env both the databases are in same server as different schema while in production they happen to be two separate ORACLE RACs separated by a few routers and switches.
Spring Batch: Use a batch job somehow to pick the transactions from my source database and process and write into another target database. This way my online transactions would not fail it other database ever goes down or hits a perf issue or face a network issue. And if they ever fail I can code for job restart ability. Is Spring batch well suited for such event publishing case? Would I hit any challenge in future?
2-Phase-Commit: I simply implement 2PC and save the data in both the database in a transaction. Or maybe make it look more future proof and save in a messaging system and my source database.
Related
I have a SpringBoot application running on a multitenant architecture.
I've two databases Admin and Client (both are MySQL) and both these databases have a User table
Client can add users to the User Table but I need them to get synchronized in the User table of Admin database.
Is there a way I can achieve this?
I've read about flyway migrations but I think it works more on database schema changes and not values.
Please ignore my mistakes as this my first question, any help would be appreciated.
This looks like a solution to your problem:
SymmetricDS is software that replicates relational database tables between multiple databases. It can also be used to replicate files and directories between multiple hosts. It uses a light-weight, web-based protocol to send and receive data, which makes it easy to work with firewalls. Replication is done in the background asynchronously, allowing data changes in offline mode. It supports most commercial and open source database platforms.
How does it work?
Triggers are installed in the database to guarantee that data changes are captured. This means that applications continue to use the database as usual without any special driver software. The triggers are written to be as small and efficient as possible. Routing and syncing of data is done outside of the database in the SymmetricDS process.
SymmetricDS supports many databases and can replicate across different databases, including Oracle, MySQL, MariaDB, PostgreSQL, MS SQL and many more.
https://www.symmetricds.org/docs/faq
You need to create some event from the flow where client adds user to the User Table.
If this "client" flow is in same java service then you can make use of Spring's Asynchronous Event Handling or have a method(which does the data copy) marked with #Async. This ensures the data copy happens in separate thread.
If the "client" flow is in different java service, then any publisher-subscriber model can be used (some opensource frameworks available are kafka, rabbitmq etc).
Now to connect to two datasources at the same time, Spring's RoutingDataSource will come handy in this scenario as it works on "lookup key" to choose the datasource. Or else you can hardcode two datasource beans in your config (since it is fixed in your case).
I am developing spring mvc application
For now I am using innodb mysql but I have to develop the application to support other databases also.
Can any one please suggest me how to handle concurrent sql update on single record.
Suppose two users are trying to update same record then how to handle such scenario.
Note: My database structure is dependent on some configuration (It can change at runtime) and my spring controller is singleton in nature.
Thanks.
Update:
Just for reference I am going to implement version like https://stackoverflow.com/a/3618445/3898076).
Transactions are the way to go when it comes to concurrent sql updates, in spring you can use a transaction manager.
As for the database structure, as far as I know MySql does not support transactions for DDL commands, that is if you change the structure concurrently with updating, you're likely to run into problems.
To handle multiple users working on the same data, you need to implement a manual "lock" or "version" field on the table to keep track of last updates.
I have a java dynamic web app. I am exposing RESTful webservices for my android application.
The thing is that there are some services that do DB updates. Now, I want to host the application on public domain. I was wondering how parallel processing works on web hosting.
Say, my service /updateDB updates the database. Now, if there are two users who hit the same service at the same time, will the two of them run concurrently, because that will cause inconsistency in data. How exactly does the whole thing work.
Do I need to take care of synchronisation in my code?
Why kind of database are you using?
Certain database engines already have mechanisms in place to allow a transaction to be completed before another request over writes data. Most web developers do not have to worry about this because the application server (websphere, weblogic) and database (Mysql,Oracle) take care of these things for you.
(I am going to overly simplify this for you.)
A request to the webservice may perform one or more actions on the DB. These actions can be clumped together and be called a transaction. A transaction can include one or more of the following INSERT, UPDATE, DELETE etc. e.g A new customer registers for your webservice. the following actions take place which can be considered into a transaction.
Insert a new customer username password in the Customer table
Insert customers address in Address table
Update total customer count in Summary table
All the above actions can be completed as one transaction. If any of this fails then all actions will be reverted back automatically. Similarly if two customers registers simultaneously then the database will take care to not over write each other as well.
We can configure the database to make sure that every transaction should be completed before another transaction can dirty the data in a row.
In a database they are called ACID properties.
A - Atomicity - Every transaction must be complete, if anything in a transaction fails, then do not complete the transaction and also revert back every previous action within that transaction.
C - Consistency - make sure that every transaction that occurs will always update the database in a predefined manner. e.g. after every customer registration make sure that all the actions within it are executed
I - Isolation - if more than one request comes in, then they get executed on the database separately
D - Durabilty - after a transaction completes, the changes done should remain forever.
For example Mysql Database with the InnoDB engine supports this. There are other databases which support this as well.
You can read more here
http://java.dzone.com/articles/beginners-guide-acid-and
This is a very vast topic in databases.
Programming language have APIS which will help you write code in this manner. But the basic take away is that databases and applications servers will do most of the work for you. You just have to make sure to design the code structure to identify transactions and commit them appropriately).
Java and other programming languages are aware of ACID properties in DB and will help you achieve that goal.
Read more here about how you use Java to achieve things we mentioned above.
http://docs.oracle.com/javase/tutorial/jdbc/basics/transactions.html
Similarly other languages have similar functionality and APIs.
In google search for "java database transaction" or "<your favorite language>database transaction"
I am developping an application that tests different WebServices, and I want it to be as generic as possible. I need to populate database to do JUnit tests, but I don't want these changes to be commited.
I know that some in-memory databases like HSQL DB allow testing on a sort of a virtual (or mock) database, but unfortunately I use oracle and I cannot change it now because of my complex data tables structure.
What is the best practice you suggest?
Thanks.
First of all, HSQL and Hibernate aren't related in any way. The question is whether you can find an embedded database which supports the same SQL as your production database (or rather the subset of SQL which your application uses).
A good candidate for this is H2 database since it emulates a lot of different SQL flavours.
On top of that: Don't test the database. Assume that the database is tested thoroughly by your vendor and just works.
In my code, I aim for:
Save and load each entity.
Generate the SQL for all the queries that I use and compare them against String literals in tests (i.e. I don't run the queries against the database all the time).
Some tests look for a System property. If it's set, then they will run the queries against the database. This happens during the night on my CI server.
The rationale for this: As long as the DB schema doesn't change, there is no point to actually run the queries. That means running them during the day while I sit in front of the computer is a huge waste of time.
To make sure that "low impact" changes don't slip through the gaps, I let a computer run them when I don't care.
Along the same lines, I have mocks for many DAOs which return various predefined results, so I don't have to query the database. The rationale here is that I want to test the processing of results from the database, not the JDBC API, the DB driver, the OS's TCP/IP stack, the network hardware (and software), or any other of the 1000 things between my code and the database records on a harddisk somewhere.
More details in my blog: http://blog.pdark.de/2008/07/26/testing-with-databases/
I have an Oracle database that runs a PL/SQl job once every week to copy data from a table in the Oracle database to another Oracle database using Oracle database links. Both systems run on Unix/Solaris. The job is scheduled and executed via the Unix Cron facility.
The remote machine is now being migrated from UNIX to a Windows platform with a MSQL Server database. I have been looking into the possibility of using Oracle database links to connect from UNIX to the MSSQL Server database which will be running on a Windows platform. It is possible but it looks to me like there is a lot of fiddling around to get it to work and there is not a lot of information on how to implement it.
The other solution i am thinking of is to implement the process as a Java daemon process that will run every week. All it will do is connect to the Oracle database read data from some tables, connect to the MSSQL Server database and run some insert commands. I think i will need two kind of drivers for this - i.e. jdbc for Oracle and jdbc for MSSQL Server.
Does Spring provide any facility that will make the above easier to implement and or maintain? (Specifically the multiple drivers and the scheduling part of the requirements)
Is Hibernate suitable for managing the multiple database connections or is that overkill?
Feel free to also suggest a better solution :)
Thanks
Spring has extensive support for scheduling tasks.
Running queries against multiple databases via different drivers is a basic JDBC feature; Spring is not really relevant there - unless you want it to happen in a single (distributed) transaction. Spring does support those via JTA and XA.
This sounds like a perfect application for Spring Batch
You should check these Pages to see what it's about:
Features
Use Cases
Spring Batch will happily accept different datasources with different drivers, but you will probably have to provide some RowMapper objects to create interim Objects.