I'm using both r2dbc and Liquibase in the same application. However, Liquibase is not able to run migrations using r2dbc so I need to use a separate jdbc driver just for it.
I followed the solution here, and used testcontainers for testing, so my application-test.yaml looks exactly like this:
spring:
liquibase:
url: jdbc:tc:postgresql:14.1:///testdb
r2dbc:
url: r2dbc:tc:postgresql:///testdb?TC_IMAGE_TAG=14.1
This works perfectly, migrations are launched and then the queries are able to run. The problem is, that this starts two different containers! So the migrations are run against one of them, and the queries against the other, and thus they find the database empty.
Is there any way I can tell testcontainers to use the same container for both connections.
When you use Testcontainers' JDBC support, which you configure by adding tc in the jdbc url, the lifecycle of the container is managed automatically.
Since you have two different urls instrumented like that, you get 2 containers.
Instead you can choose a different way to manage the lifecycle that gives you more control.
You can either do it yourself by creating a containers instances and calling start()/stop() or for example use JUnit integration which will correspond containers lifecycle with the tests lifecycle.
For example for JUnit5, you mark you class with #Testcontainers and the fields with #Container, something like:
#Testcontainers
class MixedLifecycleTests {
#Container
private static PostgreSQLContainer postgresqlContainer = new PostgreSQLContainer();
}
Since you're working on a Spring application you want to configure it to use the container, for that use #DynamicPropertySource: https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/test/context/DynamicPropertySource.html
In a nutshell, you mark a method with it and inside it configure Spring to use the databases in the container:
#DynamicPropertySource
static void redisProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", postgresqlContainer:: getJdbcUrl);
registry.add("spring.datasource.username", postgresqlContainer::getUsername);
registry.add("spring.datasource.password", postgresqlContainer::getPassword);
registry.add("spring.r2dbc.url", () -> "r2dbc:postgresql://"
+ postgreSQLContainer.getHost() + ":" + postgreSQLContainer.getFirstMappedPort()
+ "/" + postgreSQLContainer.getDatabaseName());
registry.add("spring.r2dbc.username", postgreSQLContainer::getUsername);
registry.add("spring.r2dbc.password", postgreSQLContainer::getPassword);
}
Note that since your app uses r2dbc and Liquibase works with non-reactive jdbc you should configure both.
Related
I am working on a bigger Spring Boot application that has a lot of tests (more like integration tests). My job now is to speed the process of the tests up. I have found, that testData, we need to test the application are set up multiple times in one testrun, if i run multiple test classes. We use something like this to setup the Data in the classes (edit: the Repository and testDataBuilder are #Autowired):
#BeforeEach
public void setup() {
if (Repository.findByShortId("someId") == null) {
testDataBuilder.createTestData();
}
}
Within the testclass, this works fine. But if my testrun gets to the next class, it seems to drop the data (the data are normally stored in a database, I think within the test the data gets stored in an in - memory database, not sure about that.
I tried multiple things to make this work but nothing did work out in the end:
building data in an abstract class that every test extends
working with #commit on some tests
using a testSuite and trying to create data before all tests like this:
#RunWith(Suite.class)
#SuiteClasses({testClass1.class ...})
#SpringBootTest
#ActiveProfiles({ "unit-test" })
public class testSuite {
#ClassRule
public static setupTestData setup = new setupTestData();
}
this didn't work, because spring does not run at the time, the #ClassRule is run.
What is the best way to setup the testData so not every testClass has to set them up ?
Why doesnt your Test #Configuration map the proper database beans by having H2 inmemory database as a dependency?
Spring/Springboot caches context automatically without you having to do anything special.
If your #Repository uses the H2 memory, then it will be cached across all Tests cases.
H2 can also be configured to write to a file (Instead of memory),if you want it instead of memory.
By default, closing the last connection to a database closes the database. For an in-memory database, this means the content is lost.
To keep the database open, add ;DB_CLOSE_DELAY=-1 to the database URL.
To keep the content of an in-memory database as long as the virtual
machine is alive, use jdbc:h2:mem:test;DB_CLOSE_DELAY=-1.
jdbc:h2:~/test;DB_CLOSE_DELAY=-1; <-- Writes to a file, and disable closing as long as the VM is alive
http://www.h2database.com/html/features.html#database_only_if_exists
Main question - this may seem like a basic flyway question and I might have (somehow) missed this during my research but - is it possible to access an applications services (spring configured) when attempting to migrate data using flyway? Few details below -
Additional details -
I know we cannot inject spring data services etc. (learnt from this
SO question). And I understand this from a data access point of
view.
But can we not use (by injection) any other application services
either while using flyway (I searched for examples - but without
luck, and no details given on flyway documentation either)
Let us say we cannot use any Spring services (and I find some way to
work around that), can we access properties declared in
application.properties / .yml (this does not appear possible either).
Putting the above in context of our requirement - we have added a couple of new fields to few tables and as part of the release we want to populate those columns with data. This requires us (or flyway) to execute the following algorithm -
Get data from first table.
Using some of the data from each row, lookup more data with an API
call.
The URL of the API is environment specific (hence the third point
above).
Update the data returned from the API into the newly added columns.
Repeat steps above for next table.
P.S. - I know, adding columns that depend on other columns in the same table is not in accordance with 3rd normal etc. but for reasons outside this post, it is required.
Tech Stack -
Spring boot 1.3.x
Flyway 4.0.3
Using Java migration
Few examples I tried as below -
My flyway migration class is as below.
public class R__MigrationYeah implements SpringJdbcMigration {
#Value("${mypath.subpath}") // this does not work !
private String someStringIwannaUse;
#Inject // this does not work either (even with Autowired or Const. injection)!
private MyService myService;
}
I have seen some posts / blogs that have complicated details on how to configure flyways MigrationResolver or ConfigurationAware etc. - and not sure if they solve this problem (even if they do - it is a LOT of work just to write a quick migration script - is this the only way?).
Finally - I know I'm missing something because if we have to write flyway Java code without being able to use ANY existing application classes thru Spring, then it is no different than writing an independent migration project (therefore no value added by flyway other than making a DB connection available) - I'm sure this cannot be the case.
Any help would be great on this !
It is not possible to use dependency injection in a flyway migration.
The next version from flyway will support dependency injection from spring beans. See the Github issue for more details. On Stack Overflow is a workaround for the current version available.
I think you want to use flyway more dynamically than it's designed for.
Basically it's just for DB-Schema, you can do anything sql can do, but since
it does it job in a repeatable, reliable, step-by-step way you wouldn't want
any real business data in it. Flyway uses static scripts you provide, you
can't have them dynamically change over time (it would protest by checksum not matching) or with "API-Calls".
For that kind of stuff you can create your own Spring Boot app and use Flyway through it's Java API. Something along this lines.
#SpringBootApplication
#Import(ServiceConfig.class)
public class FlyWayApp implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(FlyWayApp.class, args);
}
#Value("${mypath.subpath}")
private String someStringIwannaUse;
#Autowired
private MyService myService;
#Override
public void run(String... args) throws Exception {
// Create the Flyway instance
Flyway flyway = new Flyway();
// Point it to the database
flyway.setDataSource("jdbc:h2:file:./target/foobar", "sa", null);
//Fetch data and create migration scripts needed by Flyway
myService.createMigrationScripts();
// Start the migration
flyway.migrate();
}
}
So, I'm working on a Spring Rest API using JPA and based on an Oracle database.
I have some unit tests, mostly very specific (like regex checks).
I also have some bigger integration tests and at the moment they would interact with the real database.
What I want is to use Derby for the tests and Oracle for the real app.
In an application.properties file, I have the properties for the two datasource scenarios.
I have no other configuration file, no XML file in the entire project
#spring.datasource.url=jdbc:oracle:thin:#1.2.3.4:1521/orcl
#spring.datasource.driverClassName=oracle.jdbc.driver.OracleDriver
#spring.datasource.username=user
#spring.datasource.password=password
spring.datasource.driverClassName=org.apache.derby.jdbc.EmbeddedDriver
spring.datasource.urljdbc:derby:target/database/message;create=true
spring.datasource.username=app
spring.datasource.password=app
also:
#Configuration
#PropertySource(value = { "classpath:application.properties"}, ignoreResourceNotFound = true)
#EnableAutoConfiguration
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
by manually commenting, uncommenting the right block in the properties file, i can either use derby or the real oracle DB
I would like to make that process automatic by having two "named datasources" that i can use. I'm no Spring expert :-)
What's the easiest/recommended way to achieve this?
Please if you mention some XML, tell me where they should go or how they should be referenced as i don't know about them and there seems to be very conflicting advice online depending on wether you use Spring, EE, different versions of JPA, etc.
Many Thanks!
You can annotate your Tests with the annotation #TestPropertySource which:
can be used to selectively override properties defined in system and
application property sources
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/integration-testing.html#integration-testing-annotations-spring
So, you can remove the test database properties from application.properties and either put them into a separate file and specify that as a value for #TestPropertySource or you can use the inline mechanism e.g.
#ContextConfiguration
#TestPropertySource(
locations = "/test.db.properties"
)
public class MyDatabaseTest{
}
or
#ContextConfiguration
#TestPropertySource(
properties = {
"spring.datasource.driverClassName=org.apache.derby.jdbc.EmbeddedDriver",
"spring.datasource.urljdbc:derby:target/database/message;create=true",
"spring.datasource.username=app",
"spring.datasource.password=app"
}
)
public class MyDatabaseTest {
}
Spring JpaRepository can generate SQL query from interface method name:
#Repository
#Transactional
public interface SomeEntityRepository extends JpaRepository<SomeEntity, Long> {
List<SomeEntity> findAllByEntityId(Long id);
List<SomeEntity> findAllByEntityIdAndDateOfCreationBetweenAnd/*...*/(Long id, /*...*/);
}
As you can see - it can be complicated. So it's possible that method name won't be correct and compiler errors occurs.
Question:
Is it possible to check/validate method name before using it in real project (which require some compile time, trial and error)?
Or maybe it is possible to convert such method name to SQL before usage - to see if this method name is valid and if it do all operation right?
Of course the obvious method is to make smaller project for test only. But I feel that it must be better way.
Thanks for any suggestions.
You can write a test using a test database and Spring test runner. There are multiple ways to do that, so I'll just give a rough sketch to get you started:
Instead of your real DataSource Spring bean, create a test DataSource bean using an in-memory (eg. H2) database
Create a Spring test context, that will:
discover your entities
set up the DB schema (eg. using hibernate.hbm2ddl.auto=create)
optionally pre-fill the DB with some data (eg. using hibernate.hbm2ddl.import_files)
enable and discover your Spring Data repositories
Use #RunWith(SpringJUnit4ClassRunner.class) to run your tests with JUnit.
Use #ContextConfiguration to point to your test Spring context
Use #Transactional to make the tests transactional and have any changes done in test methods rolled back automatically
#Autowire the EntityManager and the Spring Data repositories and use them in tests.
See Spring docs on Integration Testing.
A bit late to this, but....
If you want to test the name of the method without compiling the interface you can find a class called PartTree in org.springframework.data.repository.query.parser in the spring-data-commons:1.12.6 jar.
new PartTree(stringVersionOfMethodName,classOfEntityRepoIsFor)
It will throw an exception if it can't parse the name properly.
For maven I'm using spring-boot-starter-parent of 1.4.3.RELEASE and including the spring-boot-starter-data-jpa depedency
I want to setup my database with initial data programmatically. I want to populate my database for development runs, not for testing runs (it's easy). The product is built on top of Spring and JPA/Hibernate.
Developer checks out the project
Developer runs command/script to setup database with initial data
Developer starts application (server) and begins developing/testing
then:
Developer runs command/script to flush the database and set it up with new initial data because database structures or the initial data bundle were changed
What I want is to setup my environment by required parts in order to call my DAOs and insert new objects into database. I do not want to create initial data sets in raw SQL, XML, take dumps of database or whatever. I want to programmatically create objects and persist them in database as I would in normal application logic.
One way to accomplish this would be to start up my application normally and run a special servlet that does the initialization. But is that really the way to go? I would love to execute the initial data setup as Maven task and I don't know how to do that if I take the servlet approach.
There is somewhat similar question. I took a quick glance at the suggested DBUnit and Unitils. But they seem to be heavily focused in setting up testing environments, which is not what I want here. DBUnit does initial data population, but only using xml/csv fixtures, which is not what I'm after here. Then, Maven has SQL plugin, but I don't want to handle raw SQL. Maven also has Hibernate plugin, but it seems to help only in Hibernate configuration and table schema creation (not in populating db with data).
How to do this?
Partial solution 2010-03-19
Suggested alternatives are:
Using unit tests to populate the database #2423663
Using ServletContextListener to gain control on web context startup #2424943 and #2423874
Using Spring ApplicationListener and Spring's Standard and Custom Events #2423874
I implemented this using Spring's ApplicationListener:
Class:
public class ApplicationContextListener implements ApplicationListener {
public void onApplicationEvent(ApplicationEvent event) {
if (event instanceof ContextRefreshedEvent) {
...check if database is already populated, if not, populate it...
}
}
}
applicationContext.xml:
<bean id="applicationContextListener" class="my.namespaces.ApplicationContextListener" />
For some reason I couldn't get ContextStartedEvent launched, so I chose ContextRefreshedEvent which is launched in startup as well (haven't bumped into other situations, yet).
How do I flush the database? Currently, I simply remove HSQLDB artifacts and a new schema gets generated on startup by Hibernate. As the DB is then also empty.
You can write a unit test to populate the database, using JPA and plain Java. This test would be called by Maven as part of the standard build lifecycle.
As a result, you would get an fully initialized database, using Maven, JPA and Java as requested.
The usual way to do this is to use a SQL script. Then you run a specific bash file that populate the db using your .sql
If you want to be able to programmatically set your DB during the WebApp StartUp you can use a Web Context Listener. During the initialization of your webContext you can use a Servlet Context Listener to get access to your DAO (Service Layer.. whatever) create your entities and persist them as you use to do in your java code
p.s. as a reference Servlet Life Cycle
If you use Spring you should have a look at the Standard and Custom Events section of the Reference. That's a better way to implement a 'Spring Listener' that is aware of Spring's Context (in the case you need to retrieve your Services form it)
You could create JPA entities in a pure Java class and persist them. This class could be invoked by a servlet but also have a main method and be invoked on the command line, by maven (with the Exec Maven Plugin) or even wrapped as a Maven plugin.
But you're final workflow is not clear (do you want the init to be part of the application startup or done during the build?) and requires some clarification.
I would us a Singleton bean for that:
import javax.annotation.PostConstruct;
import javax.ejb.Startup;
import javax.ejb.Singleton;
#Singleton
#Startup
public class InitData {
#PostConstruct
public void load() {
// Load your data here.
}
}
Depend on your db. It is better to have script to set up db
In the aforementioned ServletContextListener or in a common startup place put all the forthcoming code
Define your data in an agreeable format - XML, JSON or even java serialization.
Check whether the initial data exists (or a flag indicating a successful initial import)
If it exists, skip. If it does not exist, get a new DAO (using WebApplicationContextUtils.getRequiredWebApplicationContext().getBean(..)) , iterate all predefined objects and persist them via the EntityManager in the database.
I'm having the same problem. I've tried using an init-method on the bean, but that runs on the raw bean w/o AOP and thus cannot use #Transactional. The same seems to go for #PostConstruct and the other bean lifecycle mechanism.
Given that, I switched to ApplicationListener w/ ContextRefreshedEvent; however, in this case, #PersistenceContext is failing to get an entity manager
javax.persistence.PersistenceException: org.hibernate.SessionException: Session is closed!
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:630)
at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:108)
Using spring 2.0.8, jpa1, hibernate 3.0.5.
I'm tempted to create a non-spring managed entitymanagerfactory and do everything directly but fear that would interfere w/ the rest of the Spring managed entity and transaction manager.
I'm not sure if you can get away from using some SQL. This would depend if your develoeprs are staring with an empty database with no schema defined or if the tables are there but they are empty.
If you starting with empty tables then you could use a Java approach to generating the data. I'm not that familiar with Maven but I assume you can create some task that would use your DAO classes to generate the data. You could probably even write it using a JVM based scripting language like Groovy that would be able to use your DAO classes directly. You would have a similar task that would clear the data from the tables. Then your developers would just run these tasks on the command line or through their IDE as a manual step after checkout.
If you have a fresh database instance that I think you will need to execute some SQL just to create the schema. You could technically do that with executing SQL calls with hibernate but that really doesn't seem worth it.
Found this ServletContextListener example by mkyong. Quoting the article:
You want to initialize the database connection pool before the web
application is start, is there a “main()” method in whole web
application?
This sounds to me like the right place where to have code to insert initial DB data.
I tested this approach to insert some initial data for my webapp; it works.
I found some interesting code in this repository: https://github.com/resilient-data-systems/fast-stateless-api-authentication
This works pretty neat in my project:
#Component
#DependsOn({ "dataSource" })
public class SampleDataPopulator {
private final static Logger log = LoggerFactory.getLogger(SampleDataPopulator.class);
#Inject
MyRepository myRepository
#PostConstruct
public void populateSampleData() {
MyItem item = new ResourceDatabasePopulator();
myRepository.save(item);
log.info("Populated DB with sample data");
}
}
You can put a file called data.sql in src/main/resources, it will be read and executed automatically on startup. See this tutorial.
The other answers did not work for me.