I am trying to find a standard approach on the following problem I have.
I have a web application deployed in a container (specifically Tomcat) and it uses a database for its functionality (in my case it is an SQL database in file mode, so there is no back-end SQL server).
What I am interested in is what is the best way to handle the various changes of my database on newer versions of my web application as the database schema changes (new tables/ new columns, removal of columns etc).
I.e. how can I handle the case of someone upgrading to a newer version of my web application and still retain his old data from the old database in the best (automatic? seemless? less manual?) manner.
I think that this is not a rare case so I believe there some best practice I can follow here.
Can anyone help me on this?
Recently we discovered Flyway - it works pretty well and embraces versioning of database schema changes (plain SQL scripts).
Obviously this topic is much broader. For instance you need to be extra careful when both the old and the new version of the application should run flawlessly in updated schema. Also you should consider rollback strategy (when upgrade didn't work well or you want to downgrade your application) - sometimes it is as simple as removing added objects (tables, columns), but when your scripts removes something, rollback should restore them.
First of all, you'd want to keep changes to the database and especially to existing columns as low as possible.
Second, if you need to rename a column or change some constraints (be careful not to get more restrictive because there might be some data that would not match), use ALTER TABLE statements. This way the data in the columns is preserved unless you drop columns. :)
Additionally, provide default values for new columns that have constraints (like not null) because there might already be datasets in that table that need to be updated in order not to violate those constraints. (Alternatively add the column, run some code to fill the column and then add the constraint.)
Third, since there seem to be multiple users of your application and they might have different versions, the easiest way for providing updates is to provide for sequential updates to the next higher version. Thus if someone wants to update from version 2 to 5, you'd first do the 2->3 update, then 3->4 and finally 4->5.
This might take longer to run but should reduce complexity since you'd bot have to worry about all possible combinations (e.g. 2->4, 2->5, 3->5 etc.)
Related
I'm using ORMLite database in my android application, now I want to change the whole structure of the Database like (renaming tables, add/remove columns, change relations, ...etc).
The question here, is there are any conflicts would happen in the devices with my app previously installed? in another words, when updating the app, is ORMLite leave any trails from the previous install that would make conflicts with the new one?!! so if I have a table named parent and I changed its name to guardian will I have two tables now in the new release?!!
If the answer is No, so why there is something like database version?
and if the answer is Yes, so how would I drop a table that is not exist anymore in my application? and can I just use the same class name with a different table name annotation to override the previous table?
I have not used ORMLite specifically. But it's just an ORM which means, it will won't decide if the table would be dropped or not based on a certain condition. That is something client has to do specifically based on their business rules. Now in Android there are specific ways you can upgrade the current database schema without dropping existing tables - https://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html
But upgrading database schema on SQLIte has lots of limitations i.e. there are so many operations that are not supported unlike a full blown DBMS.But that's part of the reason why SQLite is so light. Generally during your development cycle, try to conclude to a stable database schema as early as possible which needs minor additions later (SQLIte specifically does not support removing columns and etc.). Once you are in production and you don't wanna play with user's data, implementing an upgrade logic is the best bet you got.
But if you still want to drop a table explicitly, i see there are API's for the same in ORMLite -
http://ormlite.com/javadoc/ormlite-core/com/j256/ormlite/table/TableUtils.html
I am at the almost ready stage of my JEE development. With a lot of recommendation NOT to use Hibernate's hbm2ddl.auto in production, I decided to remove it.
So now, I found out about Flyway, which seems great for future db changes and migrations, but I am stuck at first step: I have many entities, some entities inherit from base entities. This makes the CREATE statement very complex.
What is the best practice to create the first migration file?
Thanks!
If you've taken an "entities first" approach during development you'll need to generate the initial schema in the same way for the first live deployment: This will produce the first creation script used by Flyway and there may also need to be a second associated script for populating reference data.
In a nutshell, the reasons for no longer being able to use hbm2ddl.auto after the first deployment are that create will destroy existing data and update isn't reliable enough to cover all types of schema changes (as it sounds like you may already know from this SO question).
Flyway is a very useful tool but it does require a level of discipline that may not have existed during development. When going forward from the initial release, database update scripts need to be produced for Flyway that are equivalent to the changes made to the entities since the last release. There are tools (e.g. various commercial products from Redgate) that may help here: These attempt to "diff" two schemas and generate schema and/or data update scripts for getting from database A to database B. But in my experience, none of them are perfect and they don't quite reach the holy grail of enabling a completely automated approach.
Arguably, the best way is an "as you go" manual approach to ensure that non-destructive update scripts are committed to source control whenever an entity change is made that affects the schema or reference data - but as already mentioned, this will require some discipline and/or documented processes for all team members to follow.
For the first migration file, you just need the current ddl of your database. There are many tools which can get this for you (such as the "copy ddl" option in the IntelliJ IDEA Database tool or a GUI client from your database vendor).
I am not sure about Flyway but there is an alternate way, you can use ant tasks for hibernate to generate or update schema.
Hope it helps.
If you build your project with Maven, you could use Hibernate maven plugin.
I have an application with a huge code base which uses an Oracle database. I want to develop an hibernate app which can interact with incoming and outgoing request from the above said application without any dependencies of database.
Like if I want to change the database to mysql or postgresql it would not have any problem. Is this practical? Can it be done? Asking for help.
As to practicality, very seldom does an app ever change databases. While the idea sounds great it isn't often done and generally the benefits you can get from using built in database features sometimes outweighs the work of keeping it database independent.
As to it being done, it certainly can between SQL databases. To go from SQL to noSQL is a bit more tricky as they are in the process of supporting them in JPA. If interested in that take a look at Hibernate OGM. If you want to truly keep it so you can easily switch databases you need to stick to the JPA standard. See this on generating JPA compliant entities from the database. So long as you use ONLY JPA you can easily switch between the databases that provide a JPA implementation. Then you just include the correct implementation set the dialect and you are switched.
If you have access to change the current application it will probably be easier to just update each of the actions that contain the hard coded queries with your JPA code. If you have unit testing that would make this process much easier as well.
If you want to write something new, but not change the front end, you would need to handle whatever actions your forms on the front end are submitting. Making sure to make them available at the same path and with the same HTTP methods (GET, POST, PUT, etc.), that take the same parameters, and returning the same structure as what your actions due today.
Both approaches would allow you to go action by action replacing them. With writing something new though, replacing them one at a time is a little more difficult if both the new app and old app aren't in the same domain OR if authentication/authorization is involved.
Good luck and best wishes!
Our framework is Grails. Say domain.com contains an application and currently used by some client. If we want to allow another client with the same functionality but providing a separation for the data of two clients, so that they can't mix both, how to do this? And whenever we want to add n clients to this application, what is the best method to be followed, so that with less / no configuration we can share the common war file for these clients by separating the db.
How the real time web development handle these type of situations?
And, one more point is how to provide client1.domain.com works for client1 and client2.domain.com works for client2. How to make the war file (in Java / Grails) to work like this? Otherwise we have to programmatically control the clients with in the project for every feature to be allowed or unnecessarily maintain separate war file for each client, which will be a waste of resources.
You're describing multitenancy - create one table for N 'tenants' instead of N identical (or nearly) tables, but partition it with a tenant_id column, and use that to filter results in SQL WHERE clauses.
For example the generated code for findByUsername would be something like select * from person where username='foo' and tenant_id=3' - the same code as a regular call but with the tenant_id column to restrict within that tenant's data.
Note that previously simple things like unique constraints are now harder because you would want to restrict uniqueness within a tenant, but allow a value to be reused across tenants. In this case changing the unique constraint to be on the combo of username and tenant_id works and does the heavy lifting in the database.
For a while there were several related plugins, but they relied on tweaking internal APIs and some features broke in newer Hibernate versions. But I believe that http://grails.org/plugin/multi-tenant-single-db is active; it was updated over a year ago, but it is being used. Contact the authors if it looks like it'll be what you need to be sure it's active. Note that this can only work with Hibernate 3.x.
Hibernate 4 added support for multitenancy, but I haven't heard much about its use in Grails (which is expected, since it's not that common a requirement). It's not well documented, but this bug report highlights some of the potential pitfalls and should still be a working example (the test app is still on GitHub): https://jira.grails.org/browse/GPHIB-6.
I'd like to ensure that this is working and continues to work, so please let me know via email if you have issues later. It's a great feature and having it in Hibernate core makes things a lot easier for us. But we need to make it easy to use and well-documented, and that will happen a lot faster when it's being used in a real project.
I'm using Hibernate EntityManager and Hibernate Annotations for ORM in a very early stage project. The project needs to launch soon, but the specs are changing constantly and I am concerned that the system will be launched and live data will be collected, and then the specs will change again and I will be in a situation where I need to change the database schema.
How can I set things up in order to minimize the impact of this? Are there any open source projects that deal with this kind of migration? Can Hibernate do this automatically (without wiping the database)?
Your advice is much appreciated.
It's more a functional or organizational problem than a technical one. No tool will automatically guess how to migrate data from one schema to another one. You'd better learn how to write stored procedure in order to migrate your data.
You'll probably need to disable constraints, create temporary table and columns, copy lots of data, and then delete the temporary tables and columns and re-enable constraints to have migrate your data.
Once in maintenance mode, every new feature that modifies the schema should also come with the script allowing to migrate from the current schema and data in production to the new one.
No system can possibly create datamigration scripts automatically from just the original and the final schema. There just isn't enough information.
Consider for example a new column. Should it just contain the default value? Or a value calculated from other fields/tables.
There is a good book about refactoring databases: http://www.amazon.com/Refactoring-Databases-Evolutionary-Addison-Wesley-Signature/dp/0321774515/ref=sr_1_1?ie=UTF8&qid=1300140045&sr=8-1
But there is little to no tool support for this kind of stuff.
I think the best thing you can do in advance:
Don't let anybody access the database but your application
If something else absolutely must access the db directly, give it a separate set of view specially for that purpose. This allows you to change your table structure by keeping at least the structure of what other systems see.
Have tons of tests. I just posted an article wich (with the upcoming 2nd and 3rd part) might help a little with this: http://blog.schauderhaft.de/2011/03/13/testing-databases-with-junit-and-hibernate-part-1-one-to-rule-them/
Hibernate can update the database entity model with data in the database. So do that and write migration code in java which sets or removes data relationships.
This works, and we have done it multiple times. But of course, try to follow a flexible development process; make what you know for sure first, then reevaluate the requirements - scrum etc.
In your case, I would recommend a NoSQL database. I don't have much experience with such kind of databases so I can't recommend any current implementation so you may want to check this too.