I'm trying to have a H2 database setup on spring boot application startup. I have configured the database in application.properties:
spring.datasource.url = jdbc:h2:file:~/testdb
spring.datasource.username = sa
spring.datasource.password = sa
spring.datasource.driverClassName = org.h2.Driver
The Application.java file:
#Configuration
#ComponentScan
#EnableAutoConfiguration
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
CreateH2Database createH2Database = new CreateH2Database();
createH2Database.create();
}
}
CreateH2Database.java:
public class CreateH2Database {
private Logger log = Logger.getLogger(CreateH2Database.class);
#Autowired
protected JdbcTemplate jdbcTemplate;
public void create() {
log.info("Creating H2 Database");
createUsers();
}
private void createUsers() {
log.info("Creating users table");
jdbcTemplate.execute("create table if not exists users (id serial, first_name varchar(255), last_name varchar(255))");
String[] names = "John Woo;Jeff Dean;Josh Bloch;Josh Long".split(";");
for (String fullname : names) {
String[] name = fullname.split(" ");
log.info("Inserting user record for " + name[0] + " " + name[1] + "\n");
jdbcTemplate.update(
"INSERT INTO users(first_name,last_name) values(?,?)",
name[0], name[1]);
}
}
}
Once the application is started it should create the Users table if it doesn't already exist, and insert the users into the table. If the table does already exist, I don't want it to be modified.
I get a NullPointerException on jdbcTemplate.execute. How can I get the jdbcTemplate injected? All the example I've seen require the datasource to be created manually, and then the JdbcTemplate is created. However, the datasource in this example seems to be created based on the application.properties values.
Is this the correct approach to setup the database (i.e. calling a CreateH2Database after starting the SpringApplication)? Will this approach work if I want to run the application as a WAR on another application server?
Since you are using Spring Boot, you should take advantage of it's database initialization features. There is no need to roll out your own implementation.
All you have to do is have the files schema.sql and data.sql on the root of the classpath (most likely under /resources). Spring Boot will auto-detect these and run the first one in order to create the database and the second one to populate it.
Check out this part of the Spring Boot documentation
If you need to perform the initialization conditionally (perhaps only when running integration tests), you can advantage of Spring profiles. What you would do in that case is have the properties file for the test profile contain
spring.datasource.initialize=true
while the properties file for the other profiles would contain
spring.datasource.initialize=false
Related
Context
I am trying to start my spring app without a database(so when no database is available at initialization the app won't be stopped), i managed to do this with the following commands in app.prop:
#DB should not kill the app
spring.sql.init.continue-on-error=true //app should continue if a sql init error arrises
spring.liquibase.enabled=false // liquibase bean shouldn't be initialized at start up, without this command the app crashes anyway
spring.jpa.hibernate.ddl-auto=none
Now the only thing that i need to do is figure a way so when the app does make a successful connection with the db the liquibase migration files will get executed. For this task I understood I need to customize the liquibase bean, the following code shows my progress so far:
#Configuration
public class Config {
#Value("${postgres.host}")
private String host;
#Value("${postgres.port}")
private Integer port;
#Value("${postgres.database}")
private String database;
#Value("${postgres.user}")
private String user;
#Value("${postgres.password}")
private String password;
#Value("${spring.liquibase.change-log}")
private String changelog;
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("org.postgresql.Driver");
dataSource.setUrl(String.format("jdbc:postgresql://%s:%d/%s", host, port, database));
dataSource.setUsername(user);
dataSource.setPassword(password);
return dataSource;
}
#Bean
public SpringLiquibase liquibase() {
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setDataSource(dataSource());
liquibase.setChangeLog(changelog);
return liquibase;
}
}
Preferably if the database is down the bean should not be created and if the database is running/ the server established connection with the db at some point the bean will be brought in the context and execute the migration files, i don't know if that is possible as I am a newbie,but let me know if you have any suggestions.
I am not able to figure out how to implement this. Any help and/or pointers will be greatly appreciated.
Currently, my Java/Spring application backend is deployed on EC2 and accessing MySQL on RDS successfully using the regular Spring JDBC setup. That is, storing database info in application.properties and configuring DataSource and JdbcTemplate in #Configuration class. Everything works fine.
Now, I need to access MySQL on RDS securely. RDS instance has IAM Authentication enabled. I have also successfully created IAM role and applied inline policy. Then, following the AWS RDS documentation and Java example on this link, I am able to access the database from a standalone Java class successfully using Authentication Token and the user I created instead of regular db username and password. This standalone Java class is dealing with "Connection" object directly.
The place I am stuck is how I translate this to Spring JDBC configuration. That is, setting up DataSource and JdbcTemplate beans for this in my #Configuration class.
What would be a correct/right approach to implement this?
----- EDIT - Start -----
I am trying to implement this as a library that can be used for multiple projects. That is, it will be used as a JAR and declared as a dependency in a project's POM file. This library is going to include configurable AWS Services like this RDS access using general DB username and password, RDS access using IAM Authentication, KMS (CMK/data keys) for data encryption, etc.
Idea is to use this library on any web/app server depending on the project.
Hope this clarifies my need more.
----- EDIT - End -----
DataSource internally has getConnection() so I can basically create my own DataSource implementation to achieve what I want. But is this a good approach?
Something like:
public class MyDataSource implements DataSource {
#Override
public Connection getConnection() throws SQLException {
Connection conn = null;
// get a connection using IAM Authentication Token for accessing AWS RDS, etc. as in the AWS docs
return conn;
}
#Override
public Connection getConnection(String username, String password) throws SQLException {
return getConnection();
}
//other methods
}
You can use the following snippet as a replacement for the default connection-pool provided by SpringBoot/Tomcat. It will refresh the token password every 10 minutes, since the token is valid for 15 minutes. Also, it assumes the region can be extracted from the DNS hostname. If this is not the case, you'll need to specify the region to use.
public class RdsIamAuthDataSource extends org.apache.tomcat.jdbc.pool.DataSource {
private static final Logger LOG = LoggerFactory.getLogger(RdsIamAuthDataSource.class);
/**
* The Java KeyStore (JKS) file that contains the Amazon root CAs
*/
public static final String RDS_CACERTS = "/rds-cacerts";
/**
* Password for the ca-certs file.
*/
public static final String PASSWORD = "changeit";
public static final int DEFAULT_PORT = 3306;
#Override
public ConnectionPool createPool() throws SQLException {
return pool != null ? pool : createPoolImpl();
}
protected synchronized ConnectionPool createPoolImpl() throws SQLException {
return pool = new RdsIamAuthConnectionPool(poolProperties);
}
public static class RdsIamAuthConnectionPool extends ConnectionPool implements Runnable {
private RdsIamAuthTokenGenerator rdsIamAuthTokenGenerator;
private String host;
private String region;
private int port;
private String username;
private Thread tokenThread;
public RdsIamAuthConnectionPool(PoolConfiguration prop) throws SQLException {
super(prop);
}
#Override
protected void init(PoolConfiguration prop) throws SQLException {
try {
URI uri = new URI(prop.getUrl().substring(5));
this.host = uri.getHost();
this.port = uri.getPort();
if (this.port < 0) {
this.port = DEFAULT_PORT;
}
this.region = StringUtils.split(this.host,'.')[2]; // extract region from rds hostname
this.username = prop.getUsername();
this.rdsIamAuthTokenGenerator = RdsIamAuthTokenGenerator.builder().credentials(new DefaultAWSCredentialsProviderChain()).region(this.region).build();
updatePassword(prop);
final Properties props = prop.getDbProperties();
props.setProperty("useSSL","true");
props.setProperty("requireSSL","true");
props.setProperty("trustCertificateKeyStoreUrl",getClass().getResource(RDS_CACERTS).toString());
props.setProperty("trustCertificateKeyStorePassword", PASSWORD);
super.init(prop);
this.tokenThread = new Thread(this, "RdsIamAuthDataSourceTokenThread");
this.tokenThread.setDaemon(true);
this.tokenThread.start();
} catch (URISyntaxException e) {
throw new RuntimeException(e.getMessage());
}
}
#Override
public void run() {
try {
while (this.tokenThread != null) {
Thread.sleep(10 * 60 * 1000); // wait for 10 minutes, then recreate the token
updatePassword(getPoolProperties());
}
} catch (InterruptedException e) {
LOG.debug("Background token thread interrupted");
}
}
#Override
protected void close(boolean force) {
super.close(force);
Thread t = tokenThread;
tokenThread = null;
if (t != null) {
t.interrupt();
}
}
private void updatePassword(PoolConfiguration props) {
String token = rdsIamAuthTokenGenerator.getAuthToken(GetIamAuthTokenRequest.builder().hostname(host).port(port).userName(this.username).build());
LOG.debug("Updated IAM token for connection pool");
props.setPassword(token);
}
}
}
Please note that you'll need to import Amazon's root/intermediate certificates to establish a trusted connection. The example code above assumes that the certificates have been imported into a file called 'rds-cacert' and is available on the classpath. Alternatively, you can also import them into the JVM 'cacerts' file.
To use this data-source, you can use the following properties for Spring:
datasource:
url: jdbc:mysql://dbhost.xyz123abc.us-east-1.rds.amazonaws.com/dbname
username: iam_app_user
driver-class-name: com.mysql.cj.jdbc.Driver
type: com.mydomain.jdbc.RdsIamAuthDataSource
Using Spring Java config:
#Bean public DataSource dataSource() {
PoolConfiguration props = new PoolProperties();
props.setUrl("jdbc:mysql://dbname.abc123xyz.us-east-1.rds.amazonaws.com/dbschema");
props.setUsername("iam_dbuser_app");
props.setDriverClassName("com.mysql.jdbc.Driver");
return new RdsIamAuthDataSource(props);
}
UPDATE: When using MySQL, you can also decide to use the MariaDB JDBC driver, which has builtin support for IAM authentication:
spring:
datasource:
host: dbhost.cluster-xxx.eu-west-1.rds.amazonaws.com
url: jdbc:mariadb:aurora//${spring.datasource.host}/db?user=xxx&credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
type: org.mariadb.jdbc.MariaDbPoolDataSource
The above requires MariaDB and AWS SDK libraries, and needs the CA-bundle in the classpath
I know this is an older question, but after a some searching I found a pretty easy way you can now do this using the MariaDB driver. In version 2.5 they added an AWS IAM credential plugin to the driver. It will handle generating, caching and refreshing the token automatically.
I've tested using Spring Boot 2.3 with the default HikariCP connection pool and it is working fine for me with these settings:
spring.datasource.url=jdbc:mariadb://host/db?credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.username=iam_username
#spring.datasource.password=dont-need-this
spring.datasource.hikari.maxLifetime=600000
Download rds-combined-ca-bundle.pem and put it in src/main/resources so you can connect via SSL.
You will need these dependencies on the classpath as well:
runtime 'org.mariadb.jdbc:mariadb-java-client'
runtime 'com.amazonaws:aws-java-sdk-rds:1.11.880'
The driver uses the standard DefaultAWSCredentialsProviderChain so make sure you have credentials with policy allowing IAM DB access available wherever you are running your app.
Hope this helps someone else - most examples I found online involved custom code, background threads, etc - but using the new driver feature is much easier!
There is a library that can make this easy. Effectively you just override the getPassword() method in the HikariDataSource. You use STS to assume the role and send a "password" for that role.
<dependency>
<groupId>io.volcanolabs</groupId>
<artifactId>rds-iam-hikari-datasource</artifactId>
<version>1.0.4</version>
</dependency>
How to use JMX MBean for HikariCP in Spring boot application? I have a code like this:
#SpringBootApplication
public class App() { ... }
And other class:
#Configuration
public class DatabaseCfg() {
#Bean
#ManagedOperation
public DataSource ds (#Value("${hikari.proprerties}") String config) {
HikariConfig hikariConfig = new HikariConfig(config);
return new HikariDataSource(hikariConfig);
}
In Java Mission Control (or JMX Console) a saw only Datasource managed bean, not JMX MBean for HikariCP (link). Is it possible to add it too?
In Spring Boot 2.0+ you can set the register-mbeans property in your application.properties file
spring.datasource.hikari.register-mbeans = true
If you are using an earlier version of Spring Boot you will also have to set the datasource
spring.datasource.type = com.zaxxer.hikari.HikariDataSource
I believe on your hikariConfig you need to set a few additional settings. You need to register the MBeans and set a pool name on the configuration.
HikariConfig hiakriConfig = new HikariConfig(config);
hikariConfig.setRegisterMbeans(true);
kikariConfig.setPoolName("my-pool-1");
Yes you obviously could drive these through the properties as well. I'm not sure if you are including these in your properties file as they are not listed. Also please note you are spelling properties wrong (#Value("${ds.proprerties}") should probably should be (#Value("${ds.properties}") but I'm not sure how you actually have named variables and property files. You may want to double check if that is where you want to set all of the properties.
Try this. Exclude your Hiakri DataSource Bean from being registered by Spring.
#Resource
private ObjectProvider<MBeanExporter> mBeanExporter;
#Bean("dataSource")
public DataSource createDataSource() {
String url = hikariDataSourceConfig.getUrl();
String username = hikariDataSourceConfig.getUsername();
String password = hikariDataSourceConfig.getPassword();
long idleTimeoutInMilliSeconds =
hikariDataSourceConfig.getIdleTimeOutInMilliseconds();
long maxLifetimeInMilliseconds =
hikariDataSourceConfig.getMaxLifetimeInMilliseconds();
int maximumPoolSize = hikariDataSourceConfig.getMaximumPoolSize();
int minimumIdle = hikariDataSourceConfig.getMinimumIdle();
String poolName = "HikariDataSource";
HikariConfig hikariConfig = new HikariConfig();
hikariConfig.setRegisterMbeans(true);
hikariConfig.setJdbcUrl(url);
hikariConfig.setUsername(username);
hikariConfig.setPassword(password);
hikariConfig.setIdleTimeout(idleTimeoutInMilliSeconds);
hikariConfig.setMaxLifetime(maxLifetimeInMilliseconds);
hikariConfig.setMaximumPoolSize(maximumPoolSize);
hikariConfig.setMinimumIdle(minimumIdle);
hikariConfig.setPoolName(poolName);
HikariDataSource dataSource = new HikariDataSource(hikariConfig);
mBeanExporter
.ifUnique((exporter) -> exporter.addExcludedBean("dataSource"));
return dataSource;
}
I need to create a mongodb database user in a Spring boot application using spring data mongodb. I will be creating this user as part of application startup.
I could not find any reference for doing this using spring data mongodb.
Is that possible by using Spring data mongodb?
I had the same issue in the past and I end up by creating the user before the context load, like this:
#Configuration
#EnableAutoConfiguration
#ComponentScan
public class Application extends SpringBootServletInitializer {
#SuppressWarnings("resource")
public static void main(final String[] args) {
createMongoDbUser();
ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
}
private void createMongoDbUser() {
MongoClient mongo = new MongoClient(HOST, PORT);
MongoDatabase db = mongo.getDatabase(DB);
Map<String, Object> commandArguments = new BasicDBObject();
commandArguments.put("createUser", USER_NAME);
commandArguments.put("pwd", USER_PWD);
String[] roles = { "readWrite" };
commandArguments.put("roles", roles);
BasicDBObject command = new BasicDBObject(commandArguments);
db.runCommand(command);
}
}
Spring-data-mongodb will create the db all by itself if it can't find it, when declaring your mongo-db factory.
For instance, I declare my db-factory in xml using the following:
<mongo:db-factory id="mongofactory" dbname="dbNameHere" mongo-ref="mongo" />
I did not have to create it myself, it was created by spring-data-mongodb upon firing may app the first time.
I'm new in MongoDB and Spring Data, usually the connection between an ordinary relational DB configuration is done in the .proprietes file such as :
# EMBEDDED SERVER CONFIGURATION
server.contextPath=/api
# JPA
spring.datasource.platform=org.hibernate.dialect.PostgreSQLDialect
spring.jpa.show-sql=true
spring.jpa.hibernate.ddl-auto=update
spring.database.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localhost:5432/test
spring.datasource.username=postgres
spring.datasource.password=root
But now i've migrated to MongoDB and i suceeded to insert an object in it however it is classically configured (in the main.java), and this is how it is :
public class Application {
public static final String DB_NAME = "TestDB";
public static final String COMPTE_COLLECTION = "Compte";
public static final String MONGO_HOST = "localhost";
public static final int MONGO_PORT = 27017;
public static void main(String[] args) throws UnknownHostException {
try {
MongoClient mongo = new MongoClient(MONGO_HOST, MONGO_PORT);
MongoOperations mongoOps = new MongoTemplate(mongo, DB_NAME);
Compte p = new Compte("jon", "jon");
mongoOps.insert(p, COMPTE_COLLECTION);
System.out.println(p1);
mongo.close();
} catch (UnknownHostException e) {
e.printStackTrace();
}
}
}
And I want to know how to move the above MongoDB configuration to the .proprieties file ? I tried to put them as they are and it doesnt work.
Thank You in advance.
As you can see it documentation:
# MONGODB (MongoProperties)
spring.data.mongodb.authentication-database= # Authentication database name.
spring.data.mongodb.database=test # Database name.
spring.data.mongodb.field-naming-strategy= # Fully qualified name of the FieldNamingStrategy to use.
spring.data.mongodb.grid-fs-database= # GridFS database name.
spring.data.mongodb.host=localhost # Mongo server host.
spring.data.mongodb.password= # Login password of the mongo server.
spring.data.mongodb.port=27017 # Mongo server port.
spring.data.mongodb.repositories.enabled=true # Enable Mongo repositories.
spring.data.mongodb.uri=mongodb://localhost/test # Mongo database URI. When set, host and port are ignored.
spring.data.mongodb.username= # Login user of the mongo server.
Also from this link connecting to mongo note that:
if you’re using the Mongo 3.0 Java driver. In such cases,
spring.data.mongodb.uri should be used to provide all of the
configuration.