I am trying configuring multiple couchbase data source using springboot-data-couchbase.
This is a way I tried to attach two couchbase sources with 2 repositories.
#Configuration
#EnableCouchbaseRepositories("com.xyz.abc")
public class AbcDatasource extends AbstractCouchbaseConfiguration {
#Override
protected List<String> getBootstrapHosts() {
return Collections.singletonList("ip_address_of_couchbase");
}
//bucket_name
#Override
protected String getBucketName() {
return "bucket_name";
}
//password
#Override
protected String getBucketPassword() {
return "user_password";
}
#Override
#Bean(destroyMethod = "disconnect", name = "COUCHBASE_CLUSTER_2")
public Cluster couchbaseCluster() throws Exception {
return CouchbaseCluster.create(couchbaseEnvironment(), "ip_address_of_couchbase");
}
#Bean( name = "BUCKET2")
public Bucket bucket2() throws Exception {
return this.couchbaseCluster().openBucket("bucket2", "somepassword");
}
#Bean( name = "BUCKET2_TEMPLATE")
public CouchbaseTemplate newTemplateForBucket2() throws Exception {
CouchbaseTemplate template = new CouchbaseTemplate(
couchbaseClusterInfo(), //reuse the default bean
bucket2(), //the bucket is non-default
mappingCouchbaseConverter(), translationService()
);
template.setDefaultConsistency(getDefaultConsistency());
return template;
}
#Override
public void configureRepositoryOperationsMapping(RepositoryOperationsMapping baseMapping) {
baseMapping
.mapEntity(SomeDAOUsedInSomeRepository.class, newTemplateForBucket2());
}
}
similarly:
#Configuration
#EnableCouchbaseRepositories("com.xyz.mln")
public class MlnDatasource extends AbstractCouchbaseConfiguration {...}
Now the problem is there is no straight forward way to specify namespace based datasource by attaching different beans to these configurations like in springdata-jpa as springdata-jpa support this feature do using entity-manager-factory-ref and transaction-manager-ref.
Due to which only one configuration is being picked whoever comes first.
Any suggestion is greatly appreciated.
Related question: Use Spring Data Couchbase to connect to different Couchbase clusters
#anshul you are almost there.
Make one of the Data Source as #primary which will be used as by default bucket.
Wherever you want to use the other bucket .Just use specific bean in your service class with the qualifier below is the example:
#Qualifier(value = "BUCKET1_TEMPLATE")
#Autowired
CouchbaseTemplate couchbaseTemplate;
Now you can use this template to perform all couch related operations on the desired bucket.
I have two different test classes, one testing a module that I wrote and the other testing a user-defined function that I developed. These two tests instantiate a Neo4j for testing purposes differently. Module test does it like this:
class ModuleTest
{
GraphDatabaseService database;
#Before
public void setUp()
{
String confFile = this.getClass().getClassLoader().getResource("neo4j-module.conf").getPath();
database = new TestGraphDatabaseFactory()
.newImpermanentDatabaseBuilder()
.loadPropertiesFromFile(confFile)
.newGraphDatabase();
}
}
While the UDF test class instantiates its embedded database this way:
public class UdfTest
{
#Rule
public Neo4jRule neo4j = new Neo4jRule()
.withFunction(Udf.class);
#Test
public void someTest() throws Throwable
{
try (Driver driver = GraphDatabase.driver(neo4j.boltURI() , Config.build().withEncryptionLevel(Config.EncryptionLevel.NONE).toConfig())) {
Session session = driver.session();
//...
}
}
}
The problem here is that in the first form the UDFs are not registered and in the second the module. My question is; how can I start an embedded Neo4j database for my tests in which both my module and UDF is loaded?
Take a look at how APOC Procedures loads procedures and functions within their test classes. They call a utility method during setUp():
public static void registerProcedure(GraphDatabaseService db, Class<?>...procedures) throws KernelException {
Procedures proceduresService = ((GraphDatabaseAPI) db).getDependencyResolver().resolveDependency(Procedures.class);
for (Class<?> procedure : procedures) {
proceduresService.registerProcedure(procedure);
proceduresService.registerFunction(procedure);
}
}
Just pass the GraphDatabaseService and the Class with the procedures/functions to register, and this should set everything up for your test class.
My Spring-Boot-Mvc-Web application has the following database configuration in application.properties file:
spring.datasource.url=jdbc:h2:tcp://localhost/~/pdk
spring.datasource.username=sa
spring.datasource.password=
spring.datasource.driver-class-name=org.h2.Driver
this is the only config I made. No any other configurations made by me anywhere. Nevertheless the Spring and subsystems are automatically recreate database on each web application run. Database is recreated namely on system run while it contains data after application ends.
I was not understanding this defaults and was expecting this is suitable for tests.
But when I started to run tests I found that database is recreated only once. Since tests are executed at no predefined order, this is senseless at all.
So, the question is: how to make any sense? I.e. how to make database recreate before each test as it happens at application first start?
My test class header is follows:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = myapp.class)
//#WebAppConfiguration
#WebIntegrationTest
#DirtiesContext
public class WebControllersTest {
As you see, I tried #DirtiesContext at class level and it didn't help.
UPDATE
I have a bean
#Service
public class DatabaseService implements InitializingBean {
which has a method
#Override
#Transactional()
public void afterPropertiesSet() throws Exception {
log.info("Bootstrapping data...");
User user = createRootUser();
if(populateDemo) {
populateDemos();
}
log.info("...Bootstrapping completed");
}
Now I made it's populateDemos() method to clear all data from database. Unfortunately, it does not called before each test despite #DirtiesContext. Why?
Actually, I think you want this:
#DirtiesContext(classMode = ClassMode.BEFORE_EACH_TEST_METHOD)
http://docs.spring.io/autorepo/docs/spring-framework/4.2.6.RELEASE/javadoc-api/org/springframework/test/annotation/DirtiesContext.html
#DirtiesContext may be used as a class-level and method-level
annotation within the same class. In such scenarios, the
ApplicationContext will be marked as dirty after any such annotated
method as well as after the entire class. If the
DirtiesContext.ClassMode is set to AFTER_EACH_TEST_METHOD, the context
will be marked dirty after each test method in the class.
You put it on your Test class.
Using the accepted answer in Spring-Boot 2.2.0, I was seeing JDBC syntax errors related to constraints:
Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Constraint "FKEFFD698EA2E75FXEERWBO8IUT" already exists; SQL statement:
alter table foo add constraint FKeffd698ea2e75fxeerwbo8iut foreign key (bar) references bar [90045-200]
To fix this, I added #AutoConfigureTestDatabase to my unit test (part of spring-boot-test-autoconfigure):
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase.Replace;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.annotation.DirtiesContext.ClassMode;
import org.springframework.boot.test.context.SpringBootTest;
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringRunner;
#RunWith(SpringRunner.class)
#SpringBootTest
#DirtiesContext(classMode = ClassMode.BEFORE_EACH_TEST_METHOD)
#AutoConfigureTestDatabase(replace = Replace.ANY)
public class FooRepositoryTest { ... }
To create the database you have to do what the other answers say with the spring.jpa.hibernate.ddl-auto=create-drop, now if your intent is to pupulate the database on each test then spring provides a very usefull anotation
#Transactional(value=JpaConfiguration.TRANSACTION_MANAGER_NAME)
#Sql(executionPhase=ExecutionPhase.BEFORE_TEST_METHOD,scripts="classpath:/test-sql/group2.sql")
public class GroupServiceTest extends TimeoffApplicationTests {
that is from this package org.springframework.test.context.jdbc.Sql; and you can run a before test method and a after test method. To populate the database.
Regarding creating the database each time, Say you only want your Test to have the create-drop option you can configure your tests with a custom properties with this annotation
#TestPropertySource(locations="classpath:application-test.properties")
public class TimeoffApplicationTests extends AbstractTransactionalJUnit4SpringContextTests{
Hope it helps
If you are looking for an alternative for the #DirtiesContext, this code below will help you. I used some code from this answer.
First, setup the H2 database on the application.yml file on your test resources folder:
spring:
datasource:
platform: h2
url: jdbc:h2:mem:test
driver-class-name: org.h2.Driver
username: sa
password:
After that, create a class called ResetDatabaseTestExecutionListener:
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.TestContext;
import org.springframework.test.context.support.AbstractTestExecutionListener;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.HashSet;
import java.util.Set;
public class ResetDatabaseTestExecutionListener extends AbstractTestExecutionListener {
#Autowired
private DataSource dataSource;
public final int getOrder() {
return 2001;
}
private boolean alreadyCleared = false;
#Override
public void beforeTestClass(TestContext testContext) {
testContext.getApplicationContext()
.getAutowireCapableBeanFactory()
.autowireBean(this);
}
#Override
public void prepareTestInstance(TestContext testContext) throws Exception {
if (!alreadyCleared) {
cleanupDatabase();
alreadyCleared = true;
}
}
#Override
public void afterTestClass(TestContext testContext) throws Exception {
cleanupDatabase();
}
private void cleanupDatabase() throws SQLException {
Connection c = dataSource.getConnection();
Statement s = c.createStatement();
// Disable FK
s.execute("SET REFERENTIAL_INTEGRITY FALSE");
// Find all tables and truncate them
Set<String> tables = new HashSet<>();
ResultSet rs = s.executeQuery("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA='PUBLIC'");
while (rs.next()) {
tables.add(rs.getString(1));
}
rs.close();
for (String table : tables) {
s.executeUpdate("TRUNCATE TABLE " + table);
}
// Idem for sequences
Set<String> sequences = new HashSet<>();
rs = s.executeQuery("SELECT SEQUENCE_NAME FROM INFORMATION_SCHEMA.SEQUENCES WHERE SEQUENCE_SCHEMA='PUBLIC'");
while (rs.next()) {
sequences.add(rs.getString(1));
}
rs.close();
for (String seq : sequences) {
s.executeUpdate("ALTER SEQUENCE " + seq + " RESTART WITH 1");
}
// Enable FK
s.execute("SET REFERENTIAL_INTEGRITY TRUE");
s.close();
c.close();
}
}
The code above will reset the database (truncate tables, reset sequences, etc) and is prepared to work with H2 database. If you are using another memory database (like HsqlDB) you need to make the necessary changes on the SQLs queries to accomplish the same thing.
After that, go to your test class and add the #TestExecutionListeners annotation, like:
#TestExecutionListeners(mergeMode =
TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS,
listeners = {ResetDatabaseTestExecutionListener.class}
)
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class CreateOrderIT {
This should work.
If you not see any performance difference between this approach and #DirtiesContext, probably you are using #MockBean in your tests, what marks the Spring context as dirty and automatically reload the entire context.
With spring boot the h2 database can be defined uniquely for each test. Just override the data source URL for each test
#SpringBootTest(properties = {"spring.config.name=myapp-test-h2","myapp.trx.datasource.url=jdbc:h2:mem:trxServiceStatus"})
The tests can run in parallel.
Within the test the data can be reset by
#DirtiesContext(classMode = DirtiesContext.ClassMode.BEFORE_EACH_TEST_METHOD)
There is library that covers "reset H2 database" feature in JUnit 5 tests:
https://github.com/cronn/test-utils#h2util
Sample usage:
#ExtendWith(SpringExtension.class)
#Import(H2Util.class)
class MyTest {
#BeforeEach
void resetDatabase(#Autowired H2Util h2Util) {
h2Util.resetDatabase();
}
// tests...
}
Maven coords:
<dependency>
<groupId>de.cronn</groupId>
<artifactId>test-utils</artifactId>
<version>0.2.0</version>
<scope>test</scope>
</dependency>
Disclaimer: I’m the author of suggested library.
If you use spring.jpa.hibernate.ddl-auto=create-drop should be enough to create/drop database?
Unless you're using some kind of Spring-Data integration (which I don't know at all), this seems like custom logic you'll need to implement yourself. Spring doesn't know about your databases, its schemas, and tables.
Assuming JUnit, write appropriate #Before and #After methods to set up and clean up your database, its tables, and data. Your tests can themselves write the data they need, and potentially clean up after themselves if appropriate.
A solution using try/resources and a configurable schema based on this answer. Our trouble was that our H2 database leaked data between test cases. So this Listener fires before each test method.
The Listener:
public class ResetDatabaseTestExecutionListener extends AbstractTestExecutionListener {
private static final List<String> IGNORED_TABLES = List.of(
"TABLE_A",
"TABLE_B"
);
private static final String SQL_DISABLE_REFERENTIAL_INTEGRITY = "SET REFERENTIAL_INTEGRITY FALSE";
private static final String SQL_ENABLE_REFERENTIAL_INTEGRITY = "SET REFERENTIAL_INTEGRITY TRUE";
private static final String SQL_FIND_TABLE_NAMES = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA='%s'";
private static final String SQL_TRUNCATE_TABLE = "TRUNCATE TABLE %s.%s";
private static final String SQL_FIND_SEQUENCE_NAMES = "SELECT SEQUENCE_NAME FROM INFORMATION_SCHEMA.SEQUENCES WHERE SEQUENCE_SCHEMA='%s'";
private static final String SQL_RESTART_SEQUENCE = "ALTER SEQUENCE %s.%s RESTART WITH 1";
#Autowired
private DataSource dataSource;
#Value("${schema.property}")
private String schema;
#Override
public void beforeTestClass(TestContext testContext) {
testContext.getApplicationContext()
.getAutowireCapableBeanFactory()
.autowireBean(this);
}
#Override
public void beforeTestMethod(TestContext testContext) throws Exception {
cleanupDatabase();
}
private void cleanupDatabase() throws SQLException {
try (
Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()
) {
statement.execute(SQL_DISABLE_REFERENTIAL_INTEGRITY);
Set<String> tables = new HashSet<>();
try (ResultSet resultSet = statement.executeQuery(String.format(SQL_FIND_TABLE_NAMES, schema))) {
while (resultSet.next()) {
tables.add(resultSet.getString(1));
}
}
for (String table : tables) {
if (!IGNORED_TABLES.contains(table)) {
statement.executeUpdate(String.format(SQL_TRUNCATE_TABLE, schema, table));
}
}
Set<String> sequences = new HashSet<>();
try (ResultSet resultSet = statement.executeQuery(String.format(SQL_FIND_SEQUENCE_NAMES, schema))) {
while (resultSet.next()) {
sequences.add(resultSet.getString(1));
}
}
for (String sequence : sequences) {
statement.executeUpdate(String.format(SQL_RESTART_SEQUENCE, schema, sequence));
}
statement.execute(SQL_ENABLE_REFERENTIAL_INTEGRITY);
}
}
}
Using a custom annotation:
#Target(ElementType.TYPE)
#Retention(RetentionPolicy.RUNTIME)
#TestExecutionListeners(mergeMode =
TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS,
listeners = { ResetDatabaseTestExecutionListener.class }
)
public #interface ResetDatabase {
}
You can easily mark each test in which you want to reset the database:
#SpringBootTest(
webEnvironment = RANDOM_PORT,
classes = { Application.class }
)
#ResetDatabase
public class SomeClassIT {
You can annotate your test class with #Transactional:
import org.springframework.transaction.annotation.Transactional;
...
...
#RunWith(SpringRunner.class)
#Transactional
public class MyClassTest {
#Autowired
private SomeRepository repository;
#Before
public void init() {
// add some test data, that data would be rolled back, and recreated for each separate test
repository.save(...);
}
#Test
public void testSomething() {
// add some more data
repository.save(...);
// update some base data
repository.delete(...);
// all the changes on database done in that test would be rolled back after test finish
}
}
All tests are wrapped inside a transaction, that is rolled back at the end of each test. There are unfortunately some problems with that annotation of course, and you need to pay special attention, when for example your production code uses transactions with different score.
You could also try out https://www.testcontainers.org/ which helps you to run databases inside containers and you can create a fresh database for each test run too. It will be very slow though, since each time a container has to be created and the database server has to be started, configured and then migrations have to be run, then the test can be executed.
Nothing worked for me, but the following:
For every test class you can put the following annotations:
#TestMethodOrder(MethodOrderer.OrderAnnotation.class) //in case you need tests to be in specific order
#DataJpaTest // will disable full auto-configuration and instead apply only configuration relevant to JPA tests
#AutoConfigureTestDatabase(replace = NONE) //configures a test database to use instead of the application-defined or auto-configured DataSource
To order specific tests within the class you have to put also #Order annotation:
#Test
#Order(1) //first test
#Test
#Order(2) //second test, etc.
Rerunning the tests will not fail because of previous manipulations with db.
I'm currently having the issue that the #Transactional annotation doesn't seem to start a transaction for Neo4j, yet (it doesn't work with any of my #Transactional annotated methods, not just with the following example).
Example:
I have this method (UserService.createUser), which creates a user node in the Neo4j graph first and then creates the user (with additional information) in a MongoDB. (MongoDB doesn't support transactions, thus create the user-node first, then insert the entity into MongoDB and commit the Neo4j-transaction afterwards).
The method is annotated with #Transactional yet a org.neo4j.graphdb.NotInTransactionException is thrown when it comes to creating the user in Neo4j.
Here is about my configuration and coding, respectively:
Code based SDN-Neo4j configuration:
#Configuration
#EnableTransactionManagement // mode = proxy
#EnableNeo4jRepositories(basePackages = "graph.repository")
public class Neo4jConfig extends Neo4jConfiguration {
private static final String DB_PATH = "path_to.db";
private static final String CONFIG_PATH = "path_to.properties";
#Bean(destroyMethod = "shutdown")
public GraphDatabaseService graphDatabaseService() {
return new GraphDatabaseFactory().newEmbeddedDatabaseBuilder(DB_PATH)
.loadPropertiesFromFile(CONFIG_PATH).newGraphDatabase();
}
}
Service for creating the user in Neo4j and the MongoDB:
#Service
public class UserService {
#Inject
private UserMdbRepository mdbUserRepository; // MongoRepository
#Inject
private Neo4jTemplate neo4jTemplate;
#Transactional
public User createUser(User user) {
// Create the graph-node first, because if this fails the user
// shall not be created in the MongoDB
this.neo4jTemplate.save(user); // NotInTransactionException is thrown here
// Then create the MongoDB-user. This can't be rolled back, but
// if this fails, the Neo4j-modification shall be rolled back too
return this.mdbUserRepository.save(user);
}
...
}
Side-notes:
I'm using spring version 3.2.3.RELEASE and spring-data-neo4j version 2.3.0.M1
UserService and Neo4jConfig are in separate Maven artifacts
Starting the server and SDN reading operations work so far, I'm just having troubles with writing operations
I'm currently migrating our project from the tinkerpop-framework to SDN-Neo4j. This user creation-process has worked before (with tinkerpop), I just have to make it work again with SDN-Neo4j.
I'm running the application in Jetty
Does anyone have any clue why this is not working (yet)?
I hope, this information is sufficient. If anything is missing, please let me know and I'll add it.
Edit:
I forgot to mention that manual transaction-handling works, but of course I'd like to implement it the way "as it's meant to be".
public User createUser(User user) throws ServiceException {
Transaction tx = this.graphDatabaseService.beginTx();
try {
this.neo4jTemplate.save(user);
User persistantUser = this.mdbUserRepository.save(user);
tx.success();
return persistantUser;
} catch (Exception e) {
tx.failure();
throw new ServiceException(e);
} finally {
tx.finish();
}
}
Thanks to m-deinum I finally found the issue. The problem was that I scanned for those components / services in a different spring-configuration-file, than where I configured SDN-Neo4j. I moved the component-scan for those packages which might require transactions to my Neo4jConfig and now it works
#Configuration
#EnableTransactionManagement // mode = proxy
#EnableNeo4jRepositories(basePackages = "graph.repository")
#ComponentScan({
"graph.component",
"graph.service",
"core.service"
})
public class Neo4jConfig extends Neo4jConfiguration {
private static final String DB_PATH = "path_to.db";
private static final String CONFIG_PATH = "path_to.properties";
#Bean(destroyMethod = "shutdown")
public GraphDatabaseService graphDatabaseService() {
return new GraphDatabaseFactory().newEmbeddedDatabaseBuilder(DB_PATH)
.loadPropertiesFromFile(CONFIG_PATH).newGraphDatabase();
}
}
I still will have to separate those components / services which require transactions from those which don't, though. However, this works for now.
I assume that the issue was that the other spring-configuration-file (which included the component-scan) was loaded before Neo4jConfig, since neo4j:repositories has to be put before context:component-scan. (See Note in Example 20.26. Composing repositories http://static.springsource.org/spring-data/data-neo4j/docs/current/reference/html/programming-model.html#d0e2948)
I am performing integration tests by using embedded Glassfish 3.1.2. The first thing I do in the test is to reset the database so each test have a completely fresh database to play with.
However, the problem is that the objects are persisted in the shared cache and not stored in the database. So when the next test starts it will get the old records from the cache instead of the database.
I can easily get rid of the problem by define
<property name="eclipselink.cache.shared.default" value="false"/>
in my persistence.xml file.
#BeforeClass
public static void startup() throws Exception {
container = EJBContainer.createEJBContainer();
context = container.getContext();
}
#Before
public void setUp() throws Exception {
//Clean database before every test using dbunit
}
#Test // This is the first test, works well since the test is first in order
public final void testCreateUser() throws Exception {
UserService userService = (UserService) context.lookup("java:global/galleria/galleria-ejb/UserService");
User user = new User(TEST_USER_ID, TEST_PASSWORD);
User actualUser = userService.signupUser(user);
assertTrue(actualUser != null);
assertEquals(TEST_USER_ID, actualUser.getUserId());
assertFalse(Arrays.equals(TEST_PASSWORD, actualUser.getPassword()));
logger.info("Finished executing test method {}", testMethod.getMethodName());
}
#Test // This is the second test, fails since the database not is clean
public final void testCreateUser() throws Exception {
UserService userService = (UserService) context.lookup("java:global/galleria/galleria-ejb/UserService");
User user = new User(TEST_USER_ID, TEST_PASSWORD);
User actualUser = userService.signupUser(user); // FAILS since TEST_USER_ID already in cache!!
//..
}
#Stateless
#EJB(name = "java:global/galleria/galleria-ejb/UserService", beanInterface = UserService.class)
#TransactionAttribute(TransactionAttributeType.REQUIRED)
public class UserServiceImpl implements UserService
{
#EJB
private UserRepository userRepository;
#Override
#PermitAll
public User signupUser(User user) throws UserException {
User existingUser = userRepository.findById(user.getUserId());
if (existingUser != null)
{
logger.error("Attempted to create a duplicate user.");
throw new UserException(DUPLICATE_USER);
}
try {
user = userRepository.create(user);
} catch (EntityExistsException entityExistsEx) {
logger.error("Attempted to create a duplicate user.");
throw new UserException(DUPLICATE_USER, entityExistsEx);
}
return user;
}
//..
}
However, I do not want to disable caching in persistence.xml file, since I will get performance loss later on. I only want to do it while testing. Note that I am using JTA data source here.
Any ideas?
Off topic, I am trying to learn java ee, and following the Galleria EE project and try to modify it for my needs.
Best regards
Check out http://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching
as both JPA 2.0 and EclipseLink native api allow clearing the shared cache. You could call this api at the start or end of your tests.