In Spring Boot, the documentation seems to encourage running migrations on app startup.
This is fine, but sometimes app startup may have side effects / dependencies that I don't want to bother with - I just want to run the migrations on their own. Think just setting up a local dev database, to poke around in it, without even running the app.
In Dropwizard by comparison, running migrations alone is straightforward with built in arguments for the app, like so
java -jar hello-world.jar db migrate helloworld.yml
Is there anything equivalent for Spring Boot? Or do I just have to drop down and run liquibase directly?
I'm interested in a direct answer, but also kind of interested in seeing if I'm misunderstanding something at a higher level - like perhaps this approach of running on startup is generally 'better' for some reasons I haven't discovered yet, so you're encouraged solely to do it this way by Spring Boot as a design choice.
You can use different Spring profiles: For instance, use a profile called 'init' which will activate the 'liquibase' profile.
application.yml: (disable Liquibase by default)
spring:
liquibase:
enabled: false
application-init.yml: (does not run a web container, so spring will close automatically after startup)
spring:
profiles:
include: liquibase
main:
web-application-type: none
spring-liquibase.yml: (enables liquibase)
spring:
liquibase:
enabled: true
change-log: classpath:/db/changelog/changelog.xml
This setup allows you to run Liquibase as an init container (spring.profiles.active=init), but if you want to you can still run Liquibase as part of your web-app (spring.profiles.active=liquibase).
I know this is an old question, but in case someone else stumbles upon it, this might be useful.
You can define a command line argument for your app, that you will use to only spin up a portion of the app context that will run migration.
Here's an example in Kotlin:
import org.springframework.boot.ApplicationContextFactory.ofContextClass
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration
import org.springframework.boot.autoconfigure.liquibase.LiquibaseAutoConfiguration
import org.springframework.boot.builder.SpringApplicationBuilder
import org.springframework.boot.runApplication
import org.springframework.context.annotation.AnnotationConfigApplicationContext
import org.springframework.context.annotation.Import
#SpringBootApplication
class Application
#Import(DataSourceAutoConfiguration::class, LiquibaseAutoConfiguration::class)
class LiquibaseInit
fun main(args: Array<String>) {
if (args.contains("dbinit")) {
SpringApplicationBuilder(LiquibaseInit::class.java)
.contextFactory(ofContextClass(AnnotationConfigApplicationContext::class.java))
.run(*args)
return
}
runApplication<Application>(*args)
}
Note: with Spring Boot 2.4.0 and earlier use SpringApplicationBuilder#contextClass instead of contextFactory.
We have 2 classes declared here: Application (the main app class having #SpringBootApplication annotation) and LiquiBaseInit (having #DataSourceAutoConfiguration and #LiquibaseAutoConfiguration), the first one will spin up the whole context, while the latter will only spin up the beans necessary for Liquibase to run migration.
Inside the main function we check if the arguments array has a string dbinit and if it is there we start an application context out of LiquiBaseInit class.
Now you can run migration with your jar file like this:
java -jar hello-world.jar dbinit
If you're going to run your app in Kubernetes, you might also want to check out this article in my blog: https://blog.monosoul.dev/2021/12/26/using-liquibase-with-kubernetes/ .
This answer mentions a hook that runs after Liquibase. In that question, it was used to populate the database, probably with test or default values.
#Bean
#DependsOn("liquibase")
public YourBean yourBean() {
return new YourBean();
}
static class YourBean {
#PostConstruct
public void shutdown() {
// Exit your application here.
);
}
}
That could work. I don't know, you'd probably even have access to the liquibase mode and only shut down when it was "create."
In general, it might be just less work to run migrations on app start, especially if you're using Spring Boot and its all set up for you. You can then just stop the app.
However if you really would like to run migrations without the Spring Boot app, you can use the Liquibase Gradle and Maven plugins:
https://github.com/liquibase/liquibase-gradle-plugin
https://docs.liquibase.com/tools-integrations/maven/home.html
This will require you to set up the database credentials in another config file, in addition to your application config, so that the plugins can connect to the database.
For this purpose we created our own script that allows not only to gen migrations but also to manually apply them, however it could be applied only from gradle, see:
https://github.com/Wissance/SpringUu
to apply update we use following command:
.\gradlew.bat update -PrunList='changesApply'
to specify db connection edit gradle section changesApply
For more details about our tool see article:
https://m-ushakov.medium.com/code-first-with-spring-boot-hibernate-and-liquibase-48f5c9998d95
U can set up and configure maven liquibase plugin:
https://docs.liquibase.com/tools-integrations/maven/commands/home.html
It allows to use migrations without starting spring boot app itself
can be set up as part of building artifact or like separate task
Related
I'm making an application for a school project, but I'm running into the issue that when I try to run the unit tests that it tries to connect to the database while starting up the application, which isn't required for the tests (because it will be mocked), and is not available in the CI/CD pipeline.
jdbc connection error
I'm building my project in Java Maven Springboot and would like to know how I can prevent it from trying to connect to the database when running my test.
here is a link to my repository: https://gitlab.com/kwetter_jack/Kwetter_posts/-/tree/ci_cd_setup
Your test classes have #SpringBootTest annotation which will start a Spring application context - as your application uses a database the tests will also try to setup and use a database connection.
The simplest solution is to remove the annotation so the tests no longer try to connect to a database. You'll probably need to mock some more dependencies as a result as Spring is no longer creating these for you. You could also have a look at https://www.baeldung.com/spring-boot-testing for some other ideas how you could alter your tests.
Alternatively if you do want / need the application context to run you can add a application.yaml for the tests that defines and uses a in memory DB so the tests have something to connect to - see https://www.baeldung.com/spring-boot-h2-database for details how to do this.
Just change value under spring.datasource to H2 database to prevent
The application connect the real database.
Test application.yml
FYI, You no need to copy all config from original application.yml, just only some config that you need to override.
while I was investigating the spring boot H2 in-memory database (as suggested by Chris Olive and Paranaaan) I also came across the option of using a test container. after looking into this I saw that this enables the project to create a temp docker container with a MySQL image that I can use during the testing of my project, considering I was planning on using docker anyway for the integration testing of my microservices project I attempted this and it worked as I had hoped it would.
if anyone is interested in the test container solution that I used, the information can be found here:
https://www.testcontainers.org/modules/databases/mysql/
CONTEXT:
I have a non Spring Boot app which I wrap in Spring Boot in order to test functionality. It happened that code had an in-memory database in tests, I want to test against a different database to reproduce the production environment. I decided to go with testcontainers to keep all existing tests untouched.
PROBLEM:
I need to load millions of rows into the test container. The question is how to populate the database when in testcontainers? I found a similar question -> How to populate testcontainers? but I still cannot get how to populate data in it.
How can I populate data in testcontainers?
DatabaseTestInitalizer.java I use for instantiating container:
import org.springframework.boot.test.util.EnvironmentTestUtils;
import org.springframework.context.ApplicationContextInitializer;
import org.springframework.context.ConfigurableApplicationContext;
import org.testcontainers.containers.MSSQLServerContainer;
public class DatabaseTestInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
private MSSQLServerContainer mssqlServerContainer;
#Override
public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
mssqlServerContainer = new MSSQLServerContainer();
mssqlServerContainer.start();
// This is solution for 1.x.x Spring Boot framework
// Article for migration from 1.x.x to 2.x.x Spring Boot https://stackoverflow.com/questions/54718995/appropriate-usage-of-testpropertyvalues-in-spring-boot-tests
EnvironmentTestUtils.addEnvironment(configurableApplicationContext.getEnvironment(),
"spring.datasource.url=" + mssqlServerContainer.getJdbcUrl(),
"spring.datasource.username=" + mssqlServerContainer.getUsername(),
"spring.datasource.password=" + mssqlServerContainer.getPassword()
);
}
}
As it came out after a more rigorous search I can use withInitScript and define the schema plus data into init.sql, make sure that db/init.sql resides into test/resources folder.
It works as below:
public MSSQLServerContainer mssqlServerContainer = (MSSQLServerContainer) new MSSQLServerContainer()
.withInitScript("db/init.sql")
.withExposedPorts(1433);
In this case, you don't need Flyway.
you have two options:
create container from dockerfile Links : testcontainer dockerfile Customize your MySQL Database in Docker
you can use a database migration tools like flyway and database will automatically populated
When I worked with a similar setup I've implemented the following approach:
I've managed the schema in Flyway which is integrated with spring boot anyway. All the migrations were Schema definition related migrations and did not do anything to the data.
I've placed them in src/main/sesources/... (not the test classpath) - because they were also relevant for the regular application startup.
I've created special migrations for tests that populate the test data. These migrations were under src/test/resources of my module.
These "test" migrations were written in a way that they've always started after the regular migrations.
I've configured Flyway to start before the test so that it has populated the test data.
After all, the main idea is to decouple the code of test from the data generation.
It worked for us - so I can recommend this approach.
Having said that, if you're using spring, there are other ways:
Add the data "manually" in a setup if you're using JPA. Place the call to this
code before tests kick in.
Use #Sql annotation to run a script with test data. All-in-all there are many options there, read this tutorial for more information
I'm trying to make Spring Boot application with Flyway (and Hikari pool) to start the server even when the DB is not available at that time.
I need to support cases when:
1. DB is not available when applicaition starts (it should run Flyway after DB starts, it can be up to 30 mins).
2. DB goes offline during the application lifetime and then goes back up.
I got a problem with the first case, Flyway always tries to do migrations even when DB is not available and application stops.
I tried adding spring.datasource.continue-on-error: true but Flyway ignores that, and I couldn't find any flyway configuration that would allow such operation.
Is it possible or should I wrap Flyway and do it myself?
Spring boot 2.1.4
A couple of points to consider
What is the desired behavior of the application when the DB is really not available when the instance of java application? Ok, so flyway won't start, but how the application will be able to handle requests that will have to reach the database?
Flyway itself relies on DataSource bean, maybe on hibernate if you use it, and these are much more complicated infrastructures than flyway itself?
Maybe if the database is not available the application won't need to start at all?
Instead it worth to rely on orchestrators (like kubernetes, ECS or whatever that will recognize that the application didn't start and will try to retrigger the start again, again, and again till the database will be ready)?
This is my recommendation in general.
Now, assuming find answers to all these questions and still, want to proceed with this path:
Spring Boot by itself works like this when it comes to flyway integration:
If the relevant classes (Flyway class) exist on classpath and spring.flyway.enabled=true then the bean of flyway starts and spring boot does its magic.
Technically the relevant auto configuration can be found in class org.springframework.boot.autoconfigure.flyway.FlywayAutoConfiguration (org.springframework.boot:spring-boot-autoconfigure module)
I think the way to go is to disable flyway, and given that fact that beans like DataSource are available (somehow) - create a Flyway Bean by yourself and trigger the migration in some kind of loop in the background that will exit only if the migration actually succeeds (or already applied)
I am having an issue where EC2MetaDataUtils.getItems is being invoked on application start up ( Spring boot app), we do not use EC2 and so the calls made to AWS to get Metadata always fail, the application attempts to get this data 3 times and so this is adding around 15 seconds to the start time of the application.
I have been searching high and low for solutions I found a promising solution would suggested the following #EnableAutoConfiguration(exclude = { ContextResourceLoaderAutoConfiguration.class, ContextResourceLoaderConfiguration.class, ContextInstanceDataAutoConfiguration.class })
However when I try to start up the application it complains that ContextResourceLoaderConfiguration.class cannot be excluded as it is not auto configuration; if I just exclude the other 2 the application still invokes the MetaDataUtils.
Has anyone experienced this in the past and managed to resolve it?
Thank you for your time.
Resolved with the following:
#EnableAutoConfiguration(exclude = {ContextInstanceDataAutoConfiguration.class, ContextStackAutoConfiguration.class, ContextResourceLoaderAutoConfiguration.class})
when running spring-boot-application with AWS dependencies ,
It invokes stack auto-configuration , you need to disable it ,
add following to application.yml
cloud.aws.stack.auto: false
SpringBoot application should not do any call to EC2. This mean your are using some AWS specific library/component/what ever and this library on startup do this call.
Please check your dependencies and context configuration. There are nothing about SpringBoot. There is something with your custom dependencies/components.
If you're not using EC2, you can try removing the spring-cloud-aws* libraries from your dependencies.
You can use Spring profiles to differentiate between cloud and default profiles. For cloud profile, you can use spring-cloud-aws artifact to get metadata about EC2 instance which you need EC2 read permission access from an attached IAM role whereas for default profile, you don't need to worry about cloud environment and disable the cloud configuration properties which should not cause an issue for the application startup.
I'm currently using an EnvironmentPostProcessor to add my external PropertySource, the code looks like this :
public class ExternalPropertySourceEnvironmentPostProcessor implements EnvironmentPostProcessor
{
private static final String EXTERNAL_PROPERTY_SOURCE_NAME = "ExternalPropertySource";
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application)
{
if (environment.acceptsProfiles(Profiles.EXTERNAL_PROPERTY_SOURCE_ENABLED_PROFILE)) {
environment.getPropertySources()
.addLast(new ExternalPropertySource(EXTERNAL_PROPERTY_SOURCE_NAME, new ExternalSource()));
}
}
}
A spring.factories is also used to register this EnvironmentPostProcessor.
This code actually works if the profile is set in the vm environment variables, but if it is added in src/main/resources/application.yml, the profile doesn't seem to be injected yet in the environment and is not returned by the environment.getActiveProfiles(). I've tried using the interface Ordered with the lowest precedence, but it doesn't help.
To add a bit of context around what I'm trying to achieve, this class is in a small library that adds an external property source like a database. Then we can use it in some other spring boot applications. Something like Spring Cloud Config does.
I'd like a clean way to enable or disable this property source depending on the environment where the code runs. I don't mind using something else then profiles or even another way to inject the property source, I just want something clean that doesn't depend on several factors to work.
The main problem in my code right now is that I'm using spring boot's property sources to make my own property source configurable.
UPDATE : I used a Spring Cloud app to debug this, and was confusing the bootstrap context with the normal spring boot context. See my answer below.
Further investigation made me figure out the problem appeared only with a Spring Cloud application.
In fact the breakpoint I had in this code was triggered twice, once after the bootstrap context initialization and once after the spring boot context initialization. I was only debugging the profiles in the first occurence. At that point, only the bootstrap.yml properties are loaded and not the ones from the application.yml file. The second occurence obviously had the profiles from my application.yml file.
My code worked as expected with a vanilla Spring Boot application. As the documentation states :
The Environment has already been prepared with all the usual property
sources that Spring Boot loads by default.
I was confused by the behaviour of my app which seemed to be different from that statement, but it was Spring Cloud's bootstrap that was messing with my debugging.
Since I need a PropertySource that has the highest precedence, I need to add it in the post bootstrap initialization for Spring Cloud apps. I used an init flag on my EnvironmentPostProcessor so it doesn't get executed twice and used the bootstrap.yml on Spring Cloud apps to set the profile.
TL;DR :
With Spring Cloud, an EnvironmentPostProcessor gets called twice: once after the bootstrap init and once after the normal Spring Boot context init. If you need injected properties and are targeting the Spring Cloud's post bootstrap initialization, use the bootstrap.yml instead of application.yml.