Develop Reference

Why spring boot boot not indicates that connection string removed from yml file?

I work on a java spring boot project.
I have an application.yml file that contains the connection string:
spring:
data:
mongodb:
uri: mongodb://localhost:27017/development
I comment on connection string rows in the application.yml file:
#spring:
# data:
# mongodb:
# uri: mongodb://localhost:27017/development
And then I rebuild the project and trigger repository functions.
To my surprise, I do not get any exceptions or errors while the repositories are executed and application.yml has commented connection string.
The result that I get from repository functions is an empty result.
So my question is very simple, why i do not get any indications on a runtime that the connection string is removed does spring create some default connection in case it cannot find the conn string in yml file?

I can't say if there is anything specific with Mongo, but for Spring in general you can create typesafe config and validate it on startup.
application.yml
myapp:
mongo-uri: mongodb://localhost:27017/development
spring:
data:
mongodb:
uri: ${myapp.mongo-uri}
MyAppProperties.java
#Validated
#ConfigurationProperties(prefix = "myapp")
public class MyAppProperties {
#NotEmpty
private String mongoUri;
// accessors
}
In #Configuration's or #SpringBootApplication
#EnableConfigurationProperties(MyAppProperties.class)
The key things are #Validated and #NotEmpty. You will get startup error if ${myapp.mongo-uri} is not provided.

Configuring hikari connection pool

I have following settings for my database (I have multiple databases, so they are configured in spring.datasource hierarchy.
spring:
datasource:
db-write:
url: jdbc:sqlserver://whatever.database.windows.net:1433;database=dbname;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;
username: 'myusername'
password: 'mynotsosecretpassword'
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
Then I am configuring my datasource here
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackages = "com.datasources.dbwrite.repository",
entityManagerFactoryRef = "entityManagerFactoryDbWrite",
transactionManagerRef= "transactionManagerDbWrite"
)
public class DataSourceConfigurationDbWrite {
#Bean
#Primary
#ConfigurationProperties("spring.datasource.db-write")
public DataSourceProperties dataSourcePropertiesDbWrite() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource.db-write.configuration")
public DataSource dataSourceDbWrite() {
return dataSourcePropertiesDbWrite().initializeDataSourceBuilder()
.type(HikariDataSource.class).build();
}
#Primary
#Bean(name = "entityManagerFactoryDbWrite")
public LocalContainerEntityManagerFactoryBean entityManagerFactoryDbWrite(EntityManagerFactoryBuilder builder) {
return builder
.dataSource(dataSourceDbWrite())
.packages("com.datasources.dbwrite.models")
.build();
}
#Primary
#Bean
public PlatformTransactionManager transactionManagerDbWrite(
final #Qualifier("entityManagerFactoryDbWrite") LocalContainerEntityManagerFactoryBean entityManagerFactoryDbWrite) {
return new JpaTransactionManager(Objects.requireNonNull(entityManagerFactoryDbWrite.getObject()));
}
}
I am configuring my hikari datasource in dataSourceDbWrite method based on the properties i read in dataSourcePropertiesDbWrite method. I believe i need to configure properties in specific hierarchy so that dataSourceDbWrite method can easily detect which properties are needed for hikari. Is that correct?
What that hierarchy would be?
Moreover, how can and where can i find what properties i can configure for hikari? connection-timeout? connection pool size etc?

Me personally prefer application.yml than code to configurate Hikari:
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: {JDBC URL}
username: {USERNAME}
password: {PASSWORD}
type: com.zaxxer.hikari.HikariDataSource
hikari:
minimum-idle: 5
idle-timeout: 600000
maximum-pool-size: 10
auto-commit: true
pool-name: HikariCorePool
max-lifetime: 1800000
connection-timeout: 30000
connection-test-query: select * from information_schema.tables limit 1
(BTW, that piece of configuration was originally writen by a colleague years ago. We didn't change it and just copy-and-paste into any new projects those years.😆)
If you want to check out all configurable fields, those spring.datasource.hikari.* keys inorg.springframework.boot:spring-boot-autoconfigure:{VERSION}/META-INF/spring/spring-configuration-metadata.json may could help.
And javadoc in com.zaxxer.hikari.HikariConfigMXBean could help too.

See example in article, the properties hierarchy are according to #ConfigurationProperties's value
If we want to configure Hikari, we just need to add a #ConfigurationProperties to the data source definition:
#Bean
#ConfigurationProperties("spring.datasource.todos.hikari")
public DataSource todosDataSource() {
return todosDataSourceProperties()
.initializeDataSourceBuilder()
.build();
}
Then, we can insert the following lines into the application.properties file:
spring.datasource.todos.hikari.connectionTimeout=30000
spring.datasource.todos.hikari.idleTimeout=600000
spring.datasource.todos.hikari.maxLifetime=1800000
See relevant hikari's spring properties
spring.datasource.hikari.connection-timeout
spring.datasource.hikari.data-source-class-name
spring.datasource.hikari.data-source-properties
spring.datasource.hikari.driver-class-name
spring.datasource.hikari.idle-timeout
spring.datasource.hikari.initialization-fail-timeout
spring.datasource.hikari.jdbc-url
spring.datasource.hikari.leak-detection-threshold
spring.datasource.hikari.login-timeout
spring.datasource.hikari.max-lifetime
spring.datasource.hikari.maximum-pool-size
spring.datasource.hikari.minimum-idle
spring.datasource.hikari.validation-timeout
And explanation on each property in HikariCP, for example
connectionTimeout
This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown. Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
Notice that camelCase hikari properties (connectionTimeout) is shown as snake-case in spring (connection-timeout)

spring boot application.properties value from bean data

I am working with spring boot. I have properties defined in application.yml.
spring:
datasource:
username: username
password: password
username and password values are stored externally which program fetches during startup. let's say the bean which fetches them during startup is dbConfig
How can I inject values from dbConfgig to application.yml?
I am using spring-data-jpa autoconfigure which automatically connects to database at startup. I want these values to be loaded to application.yml before spring connects to database.

There is no need to inject the user/password in application.yml. You can set them programmatically like this:
#Configuration
public class DataSourceConfig {
#Bean
public DataSource getDataSource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.driverClassName("org.h2.Driver");
dataSourceBuilder.url("jdbc:h2:mem:test");
// Take the values from external source, then set them
dataSourceBuilder.username("username");
dataSourceBuilder.password("password");
return dataSourceBuilder.build();
}
}

I think that first, you must create a thread to detect the change at your db Config file and then may you must re-init your bean (data source) to make your change effect.
See:
how-to-reinitialize-a-spring-bean

You may also try spring cloud to store properties. And you can then use with the help of placeholders.
https://cloud.spring.io/spring-cloud-config/reference/html/

Spring DataSource password in separate properties file causes "Failed to determine a suitable driver class"

I have the following Spring configuration
myprefix.systems[0].name=Some System
myprefix.systems[0].datasource.driverclassname=oracle.jdbc.OracleDriver
myprefix.systems[0].datasource.url=jdbc:oracle:thin:#foo:1521/bar
myprefix.systems[0].datasource.username=username
myprefix.systems[0].datasource.password=password
Which is configured into the following class (annotations are lombok and Spring).
#Configuration
#ConfigurationProperties("myprefix")
public class SystemConfig {
#Getter
#Setter
public static class ConfiguredSystem {
private final DataSourceProperties datasource = new DataSourceProperties();
private String name;
public JdbcTemplate getTemplate() {
return new JdbcTemplate(datasource.initializeDataSourceBuilder().build());
}
}
#Getter
private final List<ConfiguredSystem> systems = new ArrayList<>();
#Bean
public List<ConfiguredSystem> allSystems() {
return Collections.unmodifiableList(tradingSystems);
}
}
This works just fine when all the properties are in one application.properties file. The application starts up properly.
I am trying to move the password line into it's own application-secret.properties file, and keep the other properties in the main application.properties file.
myprefix.systems[0].datasource.password=password
I run with
-Dspring.config.location="C:/my/app/dir/conf/"
-Dspring.profiles.active=secret
However, when I do this I get the following error:
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class
Action:
Consider the following:
If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
If you have database settings to be loaded from a particular profile you may need to activate it (the profiles secret are currently active).
| o.s.b.d.LoggingFailureAnalysisReporter [main]
After putting a breakpoint in getTemplate, it seems the DataSourceProperties only contains the password and none of the other properties. I presume Spring cannot do list comprehension (for lack of a better term), myprefix.systems[0]... across different files?

This won't answer why the initial solution (merging myprefix.systems[0].datasource.password) did not work, but you may solve your problem by:
Creating a db.password property in the application-secret.properties
Use ${db.password} in application.properties.

Can you initialize Spring Batch metadata tables with Liquibase?

Currently I have a setup like below. On running the batch job locally the job will create the necessary metadata tables automatically using the data-source property values since initialize-schema is set to always. Liquibase will also run and create any tables listed in its changelog.
Here is my application.yml file
spring:
batch:
initialize-schema: always
job:
enabled: true
liquibase:
url: db_url
user: deploy_user
password: deploy_pass
change-log: classpath:db/changelog/db.changelog-master.yaml
enabled: true
data-source:
mysql:
user: r_user
password: r_pass
jdbc-url: db_url
Here is my db.changelog-master.yaml file.
databaseChangeLog:
- changeSet:
dbms: mysql
id: create-sample-table
author: me
sql: CREATE TABLE sample_table (
sample_id VARCHAR(255) NOT NULL,
sample_text TEXT,
PRIMARY KEY (samoke_id)
) ENGINE=InnoDB DEFAULT
CHARSET=utf8 COLLATE=utf8_bin;
Mysql datasource config:
#Configuration
public class DataSourceConfiguration {
#Primary
#Bean(name = "mySQLDataSource")
#ConfigurationProperties("data-source.mysql")
public DataSource mySQLDataSource() {
return DataSourceBuilder.create().type(HikariDataSource.class).build();
}
}
Liquibase Configuration (probably posting more than what's needed):
#Configuration
#EnableConfigurationProperties(LiquibaseProperties.class)
public class LiquibaseConfiguration {
private static final Logger LOG = LoggerFactory.getLogger(LiquibaseConfiguration.class);
#Autowired
private LiquibaseProperties liquibaseProperties;
public DataSource liquibaseDataSource() {
DataSourceBuilder factory = DataSourceBuilder
.create()
.url(liquibaseProperties.getUrl())
.username(liquibaseProperties.getUser())
.password(liquibaseProperties.getPassword());
return factory.build();
}
public void testLiquibaseConnection() throws SQLException {
LOG.info("Testing connection to Liquibase (in case PCF restarts and we have stale dynamic secrets)...");
liquibaseDataSource().getConnection();
LOG.info("Testing connection to Liquibase (in case PCF restarts and we have stale dynamic secrets)... Succeeded");
}
#Bean
public SpringLiquibase liquibase() {
try {
testLiquibaseConnection();
} catch (Exception ex) {
LOG.warn("WARNING: Could not connect to the database using " + liquibaseProperties.getUser() + ", so we will be skipping the Liquibase Migration for now. ", ex);
return null;
}
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setChangeLog(this.liquibaseProperties.getChangeLog());
liquibase.setContexts(this.liquibaseProperties.getContexts());
liquibase.setDataSource(liquibaseDataSource());
liquibase.setDefaultSchema(this.liquibaseProperties.getDefaultSchema());
liquibase.setDropFirst(this.liquibaseProperties.isDropFirst());
liquibase.setShouldRun(this.liquibaseProperties.isEnabled());
liquibase.setLabels(this.liquibaseProperties.getLabels());
liquibase.setChangeLogParameters(this.liquibaseProperties.getParameters());
return liquibase;
}
}
The issue is we have different credentials for creating/deploying tables and reading/writing to tables in our deployed environments. So the below setup will work to create tables via Liquibase, but fail creating the metadata tables due to having the incorrect credentials upon deployment. Our current work-around to get the metadata tables created is to deploy with the data-source properties having deploy credentials, run the job to initialize the tables and then redeploy with read/write credentials. (We can't just leave the deploy credentials for reads because they have very short TTL).
Is it possible to create the metadata tables for Spring Batch via Liquibase automatically? Specifically, without adding the creation SQL manually to the changelog files?
UPDATE:
Using veljkost's answer below having a changelog file that looks like this works:
databaseChangeLog:
- changeSet:
dbms: mysql
id: create-spring-batch-metadata
author: dev.me
changes:
- sqlFile:
encoding: UTF-8
path: classpath:/org/springframework/batch/core/schema-mysql.sql
relativeToChangelogFile: false
splitStatements: true
stripComments: true

Yes, you can reference the schema files that already exist in Spring Batch project. In org.springframework.batch.core package you can find schema-*.sql files where * is the name of the targeted db. Since you are running on mysql, your change set would look something like this:
- changeSet:
id: 1234
author: adam.sandler
changes:
- sqlFile:
encoding: utf8
path: classpath:/org/springframework/batch/core/schema-mysql.sql
relativeToChangelogFile: false
splitStatements: true
stripComments: true

To auto-migrate to your database without the use of liquabase add
spring.batch.initialize-schema=always
to your application.properties file, it will auto migrate to the embedded data-source