Develop Reference

flapdoodle.embed.mongo gets always started with Spring Boot Main application in Eclipse, how to remove

I have got a problem. A simple spring boot application works fine with existing MongoDB configuration.
For integration test, I added required configuration for embeded mongodb with flapdoodle configuration. All the unit tests are getting executed properly. When I run the main Spring Boot application, it by default considers the flapdoodle embeded mongodb configuration. As a result, the embeded mongodb never exits and while running the junit test cases, it still runs. I provide below the code snippet.
Whenever I start Spring Boot main application, it still runs the embeded mongodb. I see always the following lines in the console.
Download PRODUCTION:Windows:B64 START
Download PRODUCTION:Windows:B64 DownloadSize: 231162327
Download PRODUCTION:Windows:B64 0% 1% 2% 3% 4% 5% 6% 7% 8%
I provide the code for Mongodb configuration which should be picked up when running the main Spring Boot application.
#Slf4j
#Configuration
public class NoSQLAutoConfiguration {
#Autowired
private NoSQLEnvConfigProperties configProperties;
/**
* Morphia.
*
* #return the morphia
*/
private Morphia morphia() {
final Morphia morphia = new Morphia();
morphia.mapPackage(DS_ENTITY_PKG_NAME);
return morphia;
}
#Bean
public Datastore datastore(#Autowired #Qualifier("dev") MongoClient mongoClient) {
String dbName = configProperties.getDatabase();
final Datastore datastore = morphia().createDatastore(mongoClient, dbName);
datastore.ensureIndexes();
return datastore;
}
/**
* Mongo client.
*
* #return the mongo client
*/
#Primary
#Bean(name = "dev")
public MongoClient mongoClient() {
MongoClient mongoClient = null;
String dbHost = configProperties.getHost();
int dbPort = configProperties.getPort();
String database = configProperties.getDatabase();
log.debug("MongDB Host: {} - MongoDB Port: {}", dbHost, dbPort);
List<ServerAddress> serverAddresses = new ArrayList<>();
serverAddresses.add(new ServerAddress(dbHost, dbPort));
MongoClientOptions options = getMongoOptions();
String dbUserName = configProperties.getMongodbUsername();
String encRawPwd = configProperties.getMongodbPassword();
char[] dbPwd = null;
try {
dbPwd = Util.decode(encRawPwd).toCharArray();
} catch (Exception ex) {
// Ignore exception
dbPwd = null;
}
Optional<String> userName = Optional.ofNullable(dbUserName);
Optional<char[]> password = Optional.ofNullable(dbPwd);
if (userName.isPresent() && password.isPresent()) {
MongoCredential credential = MongoCredential.createCredential(dbUserName, database, dbPwd);
List<MongoCredential> credentialList = new ArrayList<>();
credentialList.add(credential);
mongoClient = new MongoClient(serverAddresses, credentialList, options);
} else {
log.debug("Connecting to local Mongo DB");
mongoClient = new MongoClient(dbHost, dbPort);
}
return mongoClient;
}
private MongoClientOptions getMongoOptions() {
MongoClientOptions.Builder builder = MongoClientOptions.builder();
builder.maxConnectionIdleTime(configProperties.getMongodbIdleConnection());
builder.minConnectionsPerHost(configProperties.getMongodbMinConnection());
builder.connectTimeout(configProperties.getMongodbConnectionTimeout());
return builder.build();
}
}
For integration testing, I have the configuration for embeded mongodb which is part of src/test.
#TestConfiguration
public class MongoConfiguration implements InitializingBean, DisposableBean {
MongodExecutable executable;
private static final String DBNAME = "embeded";
private static final String DBHOST = "localhost";
private static final int DBPORT = 27019;
#Override
public void afterPropertiesSet() throws Exception {
IMongodConfig mongodConfig = new MongodConfigBuilder().version(Version.Main.PRODUCTION)
.net(new Net(DBHOST, DBPORT, Network.localhostIsIPv6())).build();
MongodStarter starter = MongodStarter.getDefaultInstance();
executable = starter.prepare(mongodConfig);
executable.start();
}
private Morphia morphia() {
final Morphia morphia = new Morphia();
morphia.mapPackage(DS_ENTITY_PKG_NAME);
return morphia;
}
#Bean
public Datastore datastore(#Autowired #Qualifier("test") MongoClient mongoClient) {
final Datastore datastore = morphia().createDatastore(mongoClient, DBNAME);
datastore.ensureIndexes();
return datastore;
}
#Bean(name = "test")
public MongoClient mongoClient() {
return new MongoClient(DBHOST, DBPORT);
}
#Override
public void destroy() throws Exception {
executable.stop();
}
}
Please help me how to remove this embeded mongo configuration while running Spring Boot main application in eclipse.
I also provide below my main application below.
#EnableAspectJAutoProxy
#EnableSwagger2
#SpringBootApplication(scanBasePackages = { "com.blr.app" })
public class ValidationApplication {
/**
* The main method. f
*
* #param args the arguments
*/
public static void main(String[] args) {
SpringApplication.run(ValidationApplication.class, args);
}
}

I see the code that you have not added any profile to MongoConfiguration class. During eclipse build, this class is also picked up by Spring framework. Add the below lines to this class so that while running Spring Boot test this class will be picked up and while running main Spring Boot app, the actual Mongo Configuration file will be picked up. That is why Spring comes up with concept Profile. Add the profile appropriately for different environment.
#Profile("test")
#ActiveProfiles("test")
So final code will look like this.
#Profile("test")
#ActiveProfiles("test")
#TestConfiguration
public class MongoConfiguration implements InitializingBean, DisposableBean {
...
...
}

Spring Boot Rest + ClearDB mySQL "exceeded 'max_user_connections'" error

I keep randomly getting this error every once in a while: "java.sql.SQLSyntaxErrorException: User '{key}' has exceeded the 'max_user_connections' resource (current value: 10)".
I have tried googling help for this, but all I can find is:
"increase the max connections limit" (which can't be done in free clearDB)
"adjust maxActive amount" or "release old connections" (both of which I can't find how to do it in Spring Boot)
Here's what my code looks like:
// application.properties
# Connect to heroku ClearDB MySql database
spring.datasource.url=jdbc:mysql://{heroku_url}?reconnect=true
spring.datasource.username={user}
spring.datasource.password={password}
# Hibernate ddl auto (create, create-drop, update)
spring.jpa.hibernate.ddl-auto=update
#MySQL DIALECT
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
spring.jpa.open-in-view=false
server.port=8080
#Configuration
public class DatabaseConfig {
#Value("${spring.datasource.url}")
private String dbUrl;
#Bean
public DataSource dataSource() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(dbUrl);
return new HikariDataSource(config);
}
}
EDIT 1: I was following PauMAVA's instructions as best as I could and I came up with this code, which for some reason fails:
#Configuration
public class DatabaseConfig {
#Value("${spring.datasource.url}")
private String dbUrl;
public static DataSource ds;
#Bean
public DataSource dataSource() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(dbUrl);
DataSource ds = new HikariDataSource(config);
DatabaseConfig.ds = ds;
return ds;
}
}
// Main class
public static void main(String[] args) {
SpringApplication.run(BloggerApplication.class, args);
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
public void run() {
DataSource ds = DatabaseConfig.ds;
if (ds != null) {
try {
ds.getConnection().close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}, "Shutdown-thread"));
}

Whenever you create a connection object in you code, it is advisable to close the same in finally block. This way the number of connections do not get exhausted.
Hope this helps!

You should close the DataSource on application termination so that no unused connections remain open.
public void close(DataSource ds) {
if(ds != null) {
ds.close();
}
}
But do this only on program termination as stated here.
To use the data source later (on closing) you can register the DataSource as a Field in your class:
private DataSource ds;
#Bean
public DataSource dataSource() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(dbUrl);
DataSource ds = new HikariDataSource(config);
this.ds = ds;
return ds;
}
If you are going to have more than one data source you can make a List based approach:
private List<DataSource> activeDataSources = new ArrayList<>();
public DataSource dataSource() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(dbUrl);
DataSource ds = new HikariDataSource(config);
this.activeDataSources.add(ds);
return ds;
}
public void closeAllDataSources() {
for(DataSource ds: this.activeDataSources) {
if(ds != null) {
ds.close();
}
}
this.activeDataSources = new ArrayList<>();
}
To execute a function on program close refer to this.

Spring batch NoClassDefFoundError: oracle/xdb/XMLType

I have a Spring batch project which connect to an Oracle SQL Database, and allow to export/import some data with xls files.
In my job, I do first a delete in the table, before import the data.
Sometimes, the job failed because there is problems in the xls to import.
For example : If I have duplicate lines, I'm gonna have a SQLException for duplicate when the job will insert the lines in database.
I want to simply no commit anything (especially the delete part).
If the job is successful -> commit
If the job failed -> rollback
So I find that I have to put "setAutocommit" to false.
I have my datasource loaded at the beginning of my job, so I do a :
dataSource.getConnection().setAutoCommit(false);
The instructions works, but when I launch the job, I have this error :
ERROR o.s.batch.core.step.AbstractStep -
Encountered an error executing step step_excel_sheet_1551274910254 in job importExcelJob
org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'scopedTarget.xlsListener'
defined in class path resource [com/adeo/config/ImportExcelConfig.class]:
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [org.springframework.batch.core.StepExecutionListener]:
Factory method 'xlsListener' threw exception; nested exception is
java.lang.NoClassDefFoundError: oracle/xdb/XMLType
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
The Job config is :
#Configuration
public class ImportExcelConfig {
private static final Logger LOG = LoggerFactory.getLogger("ImportExcelConfig");
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Resource(name = "dataSource")
private DataSource dataSource;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean(name = "importExcelJob")
public Job importExcel(#Qualifier("xlsPartitionerStep") Step xlsPartitionerStep) throws Exception {
return jobBuilderFactory.get("importExcelJob").start(xlsPartitionerStep).build();
}
#Bean(name = "xlsPartitionerStep")
public Step xlsPartitionerStep(#Qualifier("xlsParserSlaveStep") Step xlsParserSlaveStep, XlsPartitioner xlsPartitioner){
return stepBuilderFactory.get("xls_partitioner_step_builder")
.partitioner(xlsParserSlaveStep)
.partitioner("xls_partitioner_step_builder",XlsPartitioner)
.gridSize(3)
.build();
}
#Bean(name = "xlsParserSlaveStep")
#StepScope
public Step xlsParserSlaveStep(#Qualifier("step") Step step,XlsSheetPartitioner xlsPartitioner) throws Exception {
return stepBuilderFactory.get("sheet_partitioner_"+System.currentTimeMillis())
.partitioner(step)
.partitioner("sheet_partitioner_"+System.currentTimeMillis(),XlsPartitioner)
.gridSize(3)
.build();
}
#Bean(name = "step")
#StepScope
public Step step(#Qualifier("xlsReader") PoiItemReader xlsReader,
#Qualifier("jdbcWriter") ItemWriter jdbcWriter,
#Qualifier("xlsListener") StepExecutionListener xlsListener
) throws Exception {
return ((SimpleStepBuilder)stepBuilderFactory
.get("step_excel_sheet_"+System.currentTimeMillis())
.<Object, Map>chunk(1000)
.reader(xlsReader)
.writer(jdbcWriter)
.listener(xlsListener)
).build();
}
#Bean(name = "xlsListener")
#StepScope
#DependsOn
public StepExecutionListener xlsListener() {
XlsStepExecutionListener listener = new xlsStepExecutionListener();
listener.setDataSource(dataSource);
listener.afterPropertiesSet();
return listener;
}
#Bean(name = "jdbcWriter")
#StepScope
#DependsOn
public ItemWriter<Map> jdbcWriter(#Value("#{stepExecutionContext[sheetConfig]}") SheetConfig sheetConfig) throws IOException, ClassNotFoundException {
JdbcBatchItemWriter<Map> writer = new JdbcBatchItemWriter<>();
writer.setItemPreparedStatementSetter(preparedStatementSetter());
String sql = sheetConfig.getSqlInsert().replaceAll("#TABLE#", sheetConfig.getTable());
LOG.info(sql);
writer.setSql(sql);
writer.setDataSource(dataSource);
writer.afterPropertiesSet();
return writer;
}
#Bean
#StepScope
public ItemPreparedStatementSetter preparedStatementSetter(){
return new ItemPreparedStatementSetter();
}
#Bean
public ItemProcessor testProcessor() {
return new TestProcessor();
}
#Bean(name = "xlsReader")
#StepScope
#DependsOn
public PoiItemReader xlsReader(#Value("#{stepExecutionContext[sheetConfig]}") SheetConfig sheetConfig,
#Value("#{stepExecutionContext[xls]}") File xlsFile) throws IOException {
PoiItemReader reader = new PoiItemReader();
reader.setResource(new InputStreamResource(new PushbackInputStream(new FileInputStream(xlsFile))));
reader.setRowMapper(mapRowMapper());
reader.setSheet(sheetConfig.getSheetIndex());
reader.setLinesToSkip(sheetConfig.getLinesToSkip());
return reader;
}
#Bean
#StepScope
#DependsOn
public RowMapper mapRowMapper() throws IOException {
return new MapRowMapper();
}
}
The listener is :
public class XlsStepExecutionListener implements StepExecutionListener, InitializingBean {
private final static Logger LOGGER = LoggerFactory.getLogger(XlsStepExecutionListener.class);
#Value("#{stepExecutionContext[sheetConfig]}")
private SheetConfig config;
#Value("#{jobParameters['isFull']}")
private boolean isFull;
#Value("#{stepExecutionContext[supp]}")
private String supp;
private DataSource dataSource;
#Override
public void afterPropertiesSet() {
Assert.notNull(dataSource, "dataSource must be provided");
}
#Override
public void beforeStep(StepExecution stepExecution) {
LOGGER.info("Start - Import sheet {}", config.sheetName);
dataSource.getConnection().setAutoCommit(false);
JdbcTemplate jt = new JdbcTemplate(dataSource);
if(config.sqlDelete != null){
//DELETE DATA
LOGGER.info("beforeStep - PURGE DATA"+config.getSqlDelete().replaceAll("#TABLE#", config.getTable()));
jt.update(config.getSqlDelete().replaceAll("#TABLE#", config.getTable()),supp);
}
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
LOGGER.info ("End - Import sheet {}",config.sheetName);
//TODO :
//If status failed -> rollback, if status success : commit
return ExitStatus.COMPLETED;
}
public DataSource getDataSource() {
return dataSource;
}
public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
}
}
In the pom.xml, I have the oracle jar :
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0.3</version>
</dependency>
I see that the class XMLType is in another jar of Oracle, but I don't know why I need to add this jar when I simply do a modification of auto commit mode ?
Also, I see that, for ALL the method I can call from getConnection().XXXX, the same exception happen. So it's not specific to the auto commit.
Thank you

How can I start flyway migration before hibernate validation?

I use flyway + hibernate validate. I have flyway bean:
#Component
public class DbMigration {
private static final Logger LOG = LoggerFactory.getLogger(DbMigration.class);
private final Config config;
#Autowired
public DbMigration(Config config) {
this.config = config;
}
public void runMigration() {
try {
Flyway flyway = new Flyway();
flyway.configure(properties());
int migrationApplied = flyway.migrate();
LOG.info("[" + migrationApplied + "] migrations are applied");
} catch (FlywayException ex) {
throw new DatabaseException("Exception during database migrations: ", ex);
}
}
public Properties properties() {
//my prop
}
}
And in Apllication class I do it:
public static void main(String[] args) {
try {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);
context.getBean(DbMigration.class).runMigration();
But my hibernate start before runMigration(); And validate throw exeption. How can I start next?
run Migration
start hibernate validation
EDIT:
#Bean
#Autowired
public LocalContainerEntityManagerFactoryBean entityManagerFactory(DataSource datasource) {
log.info("entityManagerFactory start");
dbMigration.runMigration();
But I think it is bad

In your spring application configuration, if you have an entity manager factory bean configuration you can make it depend on the flyway bean so that it gets initialized after it. Something like:
#Bean
#DependsOn("flyway")
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
// Initialize EntityManagerFactory here
}
The flyway bean configuration can be something like:
#Bean(initMethod = "migrate")
public Flyway flyway() {
Flyway flyway = new Flyway();
// configure bean here
return flyway;
}

Quartz Scheduler create schedulerFactoryBean Beans without quartz.properties

I have running quartz scheduler with inside my spring app.
right now, i'm using quartz.properties to contain any properties value and use it to create schedulerFactoryBean Bean and it works fine.
this is my QuartzConfiguration..
#Configuration
public class QuartzConfiguration {
public static final String CONTEXT_KEY = "applicationContext";
//#Autowired
//private DataSource dataSource;
#Bean
public SchedulerFactoryBean schedulerFactoryBean() {
SchedulerFactoryBean scheduler = new SchedulerFactoryBean();
scheduler.setApplicationContextSchedulerContextKey(CONTEXT_KEY);
scheduler.setConfigLocation(new ClassPathResource("config/quartz.properties"));
//scheduler.setDataSource(dataSource);
//scheduler.setAutoStartup(true);
scheduler.setWaitForJobsToCompleteOnShutdown(true);
return scheduler;
}
}
My quartz.properties :
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.dataSource=myDS
org.quartz.dataSource.myDS.driver =oracle.jdbc.OracleDriver
org.quartz.dataSource.myDS.URL = jdbc:oracle:thin:#example:1521:db
org.quartz.dataSource.myDS.user = user
org.quartz.dataSource.myDS.password = password
org.quartz.dataSource.myDS.maxConnections = 5
org.quartz.dataSource.myDS.validationQuery = select 1 from dual
org.quartz.jobStore.isClustered=false
org.quartz.jobStore.tablePrefix = DPPA.QUARTZ_
org.quartz.threadPool.threadCount=1
org.quartz.scheduler.skipUpdateCheck=true
org.quartz.plugin.jobHistory.class=id.co.fifgroup.dpa.batch.BatchHistoryListener
i want to create schedulerFactoryBean without any quartz.properties, because my client dont want to change any database connection inside the war archieve.
is it possible to create schedulerFactoryBean without any quartz.properties ?

You can configure it without properties file in this way;
Properties p = new Properties();
p.put("org.quartz.scheduler.instanceName", "Scheduler_test");
p.put("org.quartz.threadPool.threadCount", 2);
...
StdSchedulerFactory factory = new StdSchedulerFactory(p);