We have a web-based application in dev phase where we use Spring 5, JPA(Hibernate) and Postgresql 9.4
Till this moment we were using one instance of the posgresql db for our work. Basically, we don't have any schema generation script and we simply were updating the db if we needed some new table, column etc. For the Hibernate we were generating classes from the db.
Now when we have some amount of test data and each change in the db brings a lot of trouble and confusion. We realized that we need to create and start maintaining some schema generation file along with some scripts which generate test data.
After some research, we see two options
Create two *.sql files. The first will contain the schema generation script the second one SQL to create test data. Then add a small module with a class which will execute the *.sql files using plain jdbc. Basically, we will continue developing and whenever we made some changes we quickly wipe->create->populate the db. This approach looks the most appealing to us at this point. It quick, simple, robust.
Second is to set up some tool which may help with that e.g. Liquibase
This approach also looks good in terms of versioning support and other capabilities. However, we are not in production yet, we are in an active development phase. We don't have much of the devs who do the db changes and we are not sure how frequently we will update the db schema in production, it could be rare.
The question is the following. Would the first approach be a bad practice and applying the second one will give the most benefits and it worth to use it?
Would appreciate any comments or any other suggestions!
First approach is NOT a bad practice, until this generation. But it will be considering the growth of tools like Liquibase.
If you are in the early or middle of the Development Phase, go ahead with LiquiBase, along with Spring Data. Contrarily, in the closing stages of the Development Phase, Think you real need for it.
I would suggest second approach as it will automatically find the new script as you add and execute the script on startup. Moreover, when you have tools available like liquibase and flyway why reinvent the wheel ?.
2nd approach will also reduce the un-necessary code for manually executing the *.sql files. Moreover this code also needs testing and if updated can be error prone.
Moreover 1st approach where you write manual code to execute script also has to check which scripts needs to be executed.. If you already has existing database and you are adding some new scripts you need to execute those new scripts only. These things are taken care of automatically with 2nd approach and you don't need to worry about already executed script being executed again
Hope this answers your concern. Happy coding
Related
I am at the almost ready stage of my JEE development. With a lot of recommendation NOT to use Hibernate's hbm2ddl.auto in production, I decided to remove it.
So now, I found out about Flyway, which seems great for future db changes and migrations, but I am stuck at first step: I have many entities, some entities inherit from base entities. This makes the CREATE statement very complex.
What is the best practice to create the first migration file?
Thanks!
If you've taken an "entities first" approach during development you'll need to generate the initial schema in the same way for the first live deployment: This will produce the first creation script used by Flyway and there may also need to be a second associated script for populating reference data.
In a nutshell, the reasons for no longer being able to use hbm2ddl.auto after the first deployment are that create will destroy existing data and update isn't reliable enough to cover all types of schema changes (as it sounds like you may already know from this SO question).
Flyway is a very useful tool but it does require a level of discipline that may not have existed during development. When going forward from the initial release, database update scripts need to be produced for Flyway that are equivalent to the changes made to the entities since the last release. There are tools (e.g. various commercial products from Redgate) that may help here: These attempt to "diff" two schemas and generate schema and/or data update scripts for getting from database A to database B. But in my experience, none of them are perfect and they don't quite reach the holy grail of enabling a completely automated approach.
Arguably, the best way is an "as you go" manual approach to ensure that non-destructive update scripts are committed to source control whenever an entity change is made that affects the schema or reference data - but as already mentioned, this will require some discipline and/or documented processes for all team members to follow.
For the first migration file, you just need the current ddl of your database. There are many tools which can get this for you (such as the "copy ddl" option in the IntelliJ IDEA Database tool or a GUI client from your database vendor).
I am not sure about Flyway but there is an alternate way, you can use ant tasks for hibernate to generate or update schema.
Hope it helps.
If you build your project with Maven, you could use Hibernate maven plugin.
I use Play 2.3 with Hibernate.
On starting up the application the first time, I want to have some data inserted into the database as default values.
In my case I have an entity class "Studycourse". All tables are created through JPA on first run.
I use DB evolution (1.sql) to insert the default data, e.g.:
INSERT INTO studycourse (id, title) VALUES (1, 'Computer Science');
This works when using the normal "activator run" command. But if I do "activator test" and start a simple integration test with inMemoryDatabase(), I get following error:
[error] play - Table "STUDYCOURSE" not found; SQL statement: INSERT INTO studycourse (id, title) VALUES (1, 'Computer Science')
I guess, that the initial JPA setup is not done in the in-memory DB.
Question: Is there a best practice on how to do this?
The integration test looks like:
public class IntegrationTest {
#Test
public void test() {
running(testServer(3333, fakeApplication(inMemoryDatabase())), HTMLUNIT, new Callback<TestBrowser>() {
public void invoke(TestBrowser browser) {
browser.goTo("http://localhost:3333");
assertThat(browser.pageSource()).contains("Your new application is ready.");
}
});
}
}
Thanks in advance.
Your original question was essentially asking "How can I execute my JPA initialization steps in a test environment so that my in-memory database is populated when I run integration tests involving my database?".
My answer will not directly address that but it will summarize how we solve the same underlying issues you're trying to solve.
My interpretation of your objectives is that you want to:
Establish a good practice for database schema migration
Establish a common practice for database integration testing
Database Schema Migration
As I mentioned in my comment above, we use http://flywaydb.org for database schema migrations and it has been an outstanding tool. Flyway has an SBT plugin so you can run flywayClean and flywayMigrate right from activator to delete and re-initialize your database instantly.
Flyway supports sophisticated file name versions so that you can execute SQL scripts like v1.1.0.sql, v1.1.1.sql, and v1.2.0.sql. Flyway will also complain if you try to execute migration scripts that are not a pure improvement on the existing state of the database. This means we're using flyway to push database schema migrations to production, resting confident that this will fail if we've done something silly. Of course, we always take a DB backup right before the migration just to be safe.
Finally, Flyway will even let you execute java programs to populate the database in case you want to use service methods instead of just raw SQL.
Anyway, your choices here are basically Play evolutions, Flyway, or Liquibase.
In-Memory Database vs. Dev-Database
On this issue, I've seen two primary positions:
Never test on an in-memory database because then your tests won't
reveal the subtle differences between your in-memory database and
your production database, or
Use an in-memory database for local testing, but at least have your
build server use a dev database.
You can see, for example, the comments at the end of http://blog.jooq.org/2014/06/26/stop-unit-testing-database-code/.
Option #1 gives you higher speed overall, but delays the feedback time between writing bad code and getting a failing integration test.
Option #2 gives you slightly lower speed overall, but gives you immediate feedback on the real database.
As with most things in engineering, there is no "best" solution, just a set of tradeoffs which make the most sense for your team.
Choosing an ORM Layer
We initially began with Hibernate but eventually switched to http://jooq.org. See http://www.vertabelo.com/blog/technical-articles/jooq-a-really-nice-alternative-to-hibernate for a jOOQ-positive overview, and http://blog.jooq.org/2012/04/21/jooq-and-hibernate-a-discussion/ for a good discussion on the two.
Hibernate seemed attractive to us because it was so mature and so popular, but when we began running into classic SQL vs. Object-Oriented impedance mismatches like how to handle inheritance, Hibernate required a learning curve and some setup overhead.
We reasoned that, if we're going to incur that overhead at all, why not just use SQL directly to do the mappings? So, we switched to JOOQ and have been able to write some very clean, elegant, and testable code. If you're not too far down the hibernate path, I would encourage you to take a look at jOOQ.
If you're already deep into Hibernate, and it's working well for you, there's probably little value to switching.
Best Practices for Database Integration Testing
I wondered this exact question and posted about it at https://groups.google.com/forum/#!topic/jooq-user/GkBW5ZGdgwQ. Lukas, the author of jOOQ responded with some general remarks.
At this point, we integration test most of our DAO's and service classes. Our tests are run after flywayClean and flywayMigrate have been run. Then, each test is written to clean up after itself. The biggest issue is performance, which so far is not a problem, but may be an issue later.
I also posted on that and received a helpful answer. See https://groups.google.com/d/msg/play-framework/BgOCIgz_9q0/jBy8zxejPEkJ.
Disclaimer: we are close to launching our app but not yet running it in production, so others may have additional best practices to add.
It seems that whenever I change my models, Play Framework asks me to run a script that deletes my entire schema and recreates it. Obviously this won't work for production, so what is the proper way to handle this in production?
Note, I'm using ebean and Postgres, and hosting on heroku.
Unfortunately Ebean can create only CREATE DDL (and not UPDATE DDL) (as answered on their group), therefore you need to switch to manual evolutions ASAP.
some rules:
Always backup your live DB before implementing any changes :)
ebean plugin recreates whole DDL if it has only 1.sql evolution created by it
You need to remove two first comments from 1.sql and start to writing own evolutions with next numbers 2.sql, 3.sql etc. Try to place as many models/fields as possible before switching to manual evolutions. The biggest part will be done automatically by plugin.
manual evolutions should contain ALTERS to existing tables/columns instead of DROP/CREATE, they should have both: Ups and Downs for each change.
try to place as many changes in each evolution as possible, it's easier to manage then writing separate evolution for each small change.
De facto sometimes it's just easier to modify DB structure with DB gui, anyway it works mainly for the single developer... when you need to share your code with other developers writing evolutions will be better option.
If after some time you'll add next 'big' portion of new models you can enable temporary auto DDL again and using local git just to copy new parts. Then revert to own revolution and paste new parts generated by Ebean plugin.
Biesior basically summed it up quite well. However, as a beginner in Play, I find that a bit more clarification with a concrete example might be helpful.
First, the following example is for Java.
Suppose you added a new field
public String dum_str;
in your model Dum. Then, you will need a 2.sql under conf/evolutions/ like this:
# --- !Ups
ALTER TABLE dum ADD COLUMN dum_str VARCHAR(255);
# --- !Downs
ALTER TABLE dum DROP dum_str;
I hope this would be helpful.
I'm using Hibernate EntityManager and Hibernate Annotations for ORM in a very early stage project. The project needs to launch soon, but the specs are changing constantly and I am concerned that the system will be launched and live data will be collected, and then the specs will change again and I will be in a situation where I need to change the database schema.
How can I set things up in order to minimize the impact of this? Are there any open source projects that deal with this kind of migration? Can Hibernate do this automatically (without wiping the database)?
Your advice is much appreciated.
It's more a functional or organizational problem than a technical one. No tool will automatically guess how to migrate data from one schema to another one. You'd better learn how to write stored procedure in order to migrate your data.
You'll probably need to disable constraints, create temporary table and columns, copy lots of data, and then delete the temporary tables and columns and re-enable constraints to have migrate your data.
Once in maintenance mode, every new feature that modifies the schema should also come with the script allowing to migrate from the current schema and data in production to the new one.
No system can possibly create datamigration scripts automatically from just the original and the final schema. There just isn't enough information.
Consider for example a new column. Should it just contain the default value? Or a value calculated from other fields/tables.
There is a good book about refactoring databases: http://www.amazon.com/Refactoring-Databases-Evolutionary-Addison-Wesley-Signature/dp/0321774515/ref=sr_1_1?ie=UTF8&qid=1300140045&sr=8-1
But there is little to no tool support for this kind of stuff.
I think the best thing you can do in advance:
Don't let anybody access the database but your application
If something else absolutely must access the db directly, give it a separate set of view specially for that purpose. This allows you to change your table structure by keeping at least the structure of what other systems see.
Have tons of tests. I just posted an article wich (with the upcoming 2nd and 3rd part) might help a little with this: http://blog.schauderhaft.de/2011/03/13/testing-databases-with-junit-and-hibernate-part-1-one-to-rule-them/
Hibernate can update the database entity model with data in the database. So do that and write migration code in java which sets or removes data relationships.
This works, and we have done it multiple times. But of course, try to follow a flexible development process; make what you know for sure first, then reevaluate the requirements - scrum etc.
In your case, I would recommend a NoSQL database. I don't have much experience with such kind of databases so I can't recommend any current implementation so you may want to check this too.
Im working on some database migration code in Java. Im also using a factory pattern so I can use different kinds of databases. And each kind of database im using implements a common interface.
What I would like to do is have a migration check that is internal to the class and runs some database schema update code automatically. The actual update is pretty straight forward (I check schema version in a table and compare against a constant in my app to decide whether to migrate or not and between which versions of schema).
To make this automatic I was thinking the test should live inside (or be called from) the constructor. OK, fair enough, that's simple enough. My problem is that I dont want the test to run every single time I instantiate a database object (it runs a query so having it run on every construction is not efficient). So maybe this should be a class static method? I guess my question is, what is a good design pattern for this type of problem? There ought to be a clean way to ensure the migration test runs only once OR is super-efficient.
Have a look at liquibase.
Here's an ibm developerworks article that has a nice walk-thru http://www.ibm.com/developerworks/java/library/j-ap08058/index.html
Flyway fits your needs perfectly. It supports multiple databases, compares the schema version with the available migrations on the classpath and upgrades the database accordingly.
You can embed it in your application and have it run once on startup as described in the Flyway docs.
Note: Flyway also comes with a Maven plugin and the ability to clean an existing schema in case you messed things up in development.
[Disclaimer: I'm one of Flyway's developers]
I've been using the iBatis SQL Mapper and really like it. The next version, iBatis 3.0, has schema migrations support. This is still in beta, but I'm planning on using it when it gets closer to a release candidate.