We have a system made in java using a postgres database.
This database changes often, and once a week or less we are updating it. These changes are in the struture of the DB (DDL), usually in functions and fields to add new functionality.
For the changes in the DB we usually use navicat as follows:
1- We made the change in the structure of the DB using navicat and we copy the SQL that gives us to an XML file for each change we made.
2- When we have to update the DB in production we check files, identified by a version number, and update the DB.
3- Then we repeat this for each DB installed (30 in total)
The problem that we are having is that as the whole process is manual and is very easy to forget to copy a change to the XML so when we use it the script does not work or even worse when the system needs this change fails.
Therefore we are looking for a way to automate this task and we came with the following idea:
1- We make changes in navicat
2- Configure the postgres to LOG the changes in the DDL into a CSV file
3- Later we read the CSV file and pass the changes to the XML to update the producction DB
The problem we are having is that the LOG will save all attempts to change the structure, including errors so if we use that script to update it will fail too.
Is there some way to save only successful DDL changes in the log in postgres?
Is there a script or application to get the DDL changes and put it in script automatically?
Is there a better way to automate this process?
there are many answers for the questions above :-) i have managed rapidly changing databases using a number of schemes. one way to do it is maintain a master database (like you have). Use dbtoyaml to create a yaml description of the database. Then use yamltodb on all of the (30) targets, which will do everything necessary to make the target databases look exactly like the master. I have used this software for about 6 months, it is fantastic. pyrseas. -g
Related
We have a web-based application in dev phase where we use Spring 5, JPA(Hibernate) and Postgresql 9.4
Till this moment we were using one instance of the posgresql db for our work. Basically, we don't have any schema generation script and we simply were updating the db if we needed some new table, column etc. For the Hibernate we were generating classes from the db.
Now when we have some amount of test data and each change in the db brings a lot of trouble and confusion. We realized that we need to create and start maintaining some schema generation file along with some scripts which generate test data.
After some research, we see two options
Create two *.sql files. The first will contain the schema generation script the second one SQL to create test data. Then add a small module with a class which will execute the *.sql files using plain jdbc. Basically, we will continue developing and whenever we made some changes we quickly wipe->create->populate the db. This approach looks the most appealing to us at this point. It quick, simple, robust.
Second is to set up some tool which may help with that e.g. Liquibase
This approach also looks good in terms of versioning support and other capabilities. However, we are not in production yet, we are in an active development phase. We don't have much of the devs who do the db changes and we are not sure how frequently we will update the db schema in production, it could be rare.
The question is the following. Would the first approach be a bad practice and applying the second one will give the most benefits and it worth to use it?
Would appreciate any comments or any other suggestions!
First approach is NOT a bad practice, until this generation. But it will be considering the growth of tools like Liquibase.
If you are in the early or middle of the Development Phase, go ahead with LiquiBase, along with Spring Data. Contrarily, in the closing stages of the Development Phase, Think you real need for it.
I would suggest second approach as it will automatically find the new script as you add and execute the script on startup. Moreover, when you have tools available like liquibase and flyway why reinvent the wheel ?.
2nd approach will also reduce the un-necessary code for manually executing the *.sql files. Moreover this code also needs testing and if updated can be error prone.
Moreover 1st approach where you write manual code to execute script also has to check which scripts needs to be executed.. If you already has existing database and you are adding some new scripts you need to execute those new scripts only. These things are taken care of automatically with 2nd approach and you don't need to worry about already executed script being executed again
Hope this answers your concern. Happy coding
I am trying to create some MySQL code that will invoke a Java program from a trigger.
Here is what I have so far:
CREATE TRIGGER trig_name after insert ON studentinfo
FOR EACH ROW
BEGIN
END
The trigger content would then call the Java program. Is this possible?
Though not a standard feature this is very well possible with MySQL. You can use the SELECT .. INTO OUTFILE statement from inside the trigger to write to a named pipe (Windows) or memroy filesystem (Linux). Both of those can easily be monitored from Java code (or any other code for that matter). Using this technique you will avoid polling and because no actual disk access takes place either you will have good performance.
I have written a Java package for this actually so I'm 100% sure it is possible and performs well. Unfortunately I am not allowed to share my efforts here (my previous answer was deleted by a moderator) so you will have to code it yourself, sorry.
A direct answer: no you can't call a java method from a mysql trigger. If you had an oracle database you could, but not mysql.
To do what you want to do with mysql you can
make the code that updates the database also notify the swing application. Or you can
make the trigger accumulate data on pending operations in a separate table that you read periodically from the swing app.
Calling a java method from an SQL database isn't a standard feature. The Informix DB can call a shell script from a stored procedure, but I don't know of a feature like this in MySQL (I'm not an expert on mysql).
The closest thing that works with all databases would be to have a thread and periodically poll the database for new records.
SELECT * FROM studentinfo WHERE id > last_seen_id
Or you could use a timestamp:
SELECT * FROM studentinfo WHERE create_date >= last_seen_create_date
In this case you would have to filter duplicated rows which have already loaded from the previous run.
I am using Spring 4.1.6, and I have my service working fine with Hibernate. In the root of the project I've got my schema.sql which is being run every time I run the server. The problem is that first time I ran the server, I put some data in db, and when I restarted it, the script was executed again and I lost all that data that I loaded before restart.
So, I think that I have two options two solve this problem:
Edit sql script to execute all queries just in case they do not exist (which would be more laborious since I have to edit the script every time I export my db)
Tell hibernate, by some way, to execute sql script just in some cases. That would be great if there existed some config that executes the script just in case the data base doesn't exist.
Do you know if this is even possible? Thanks in advance.
It sounds like this is the perfect use-case for a tool called Liquibase. This is basically a version control tool for your database which allows you to define changes to your schema and/or data and ensures that these changes are only applied once.
It's incredibly useful if multiple people are changing the same database schema and ensures that your database is always valid for the version of the code that you have checked out/released etc.
i have a java project with mysql database
i am using advance installer to create a setup file...
i can embed jre to run the software(Without installing java in the system).
like wise,i want to embed the mysql database (system doesn't contains mysql )...
.There is any software to embed mysql database in my project setup...
MySQL is very difficult to embed correctly and there are a number of failure states that might occur if it is not shut down using the proper procedure. SQLite is a much better engine for this sort of thing and is used by a number of applications as a persistent backing store. While not as powerful as MySQL, it is much more resilient. It also has the advantage of not requiring a separate process.
SQLite's storage method is to persist things into a file that can be copied, moved, or backed-up without any issues. MySQL involves many such files, some of which are in an inconsistent state unless the correct FLUSH is called.
The best you can do with MySQL is bundle it, not embed it, but then you'll be responsible for setting it up on the host system, configuring it correctly, running the appropriate maintenance procedures, and providing some kind of back-up facility for the database itself.
I am trying to create some MySQL code that will invoke a Java program from a trigger.
Here is what I have so far:
CREATE TRIGGER trig_name after insert ON studentinfo
FOR EACH ROW
BEGIN
END
The trigger content would then call the Java program. Is this possible?
Though not a standard feature this is very well possible with MySQL. You can use the SELECT .. INTO OUTFILE statement from inside the trigger to write to a named pipe (Windows) or memroy filesystem (Linux). Both of those can easily be monitored from Java code (or any other code for that matter). Using this technique you will avoid polling and because no actual disk access takes place either you will have good performance.
I have written a Java package for this actually so I'm 100% sure it is possible and performs well. Unfortunately I am not allowed to share my efforts here (my previous answer was deleted by a moderator) so you will have to code it yourself, sorry.
A direct answer: no you can't call a java method from a mysql trigger. If you had an oracle database you could, but not mysql.
To do what you want to do with mysql you can
make the code that updates the database also notify the swing application. Or you can
make the trigger accumulate data on pending operations in a separate table that you read periodically from the swing app.
Calling a java method from an SQL database isn't a standard feature. The Informix DB can call a shell script from a stored procedure, but I don't know of a feature like this in MySQL (I'm not an expert on mysql).
The closest thing that works with all databases would be to have a thread and periodically poll the database for new records.
SELECT * FROM studentinfo WHERE id > last_seen_id
Or you could use a timestamp:
SELECT * FROM studentinfo WHERE create_date >= last_seen_create_date
In this case you would have to filter duplicated rows which have already loaded from the previous run.