I'm creating a spring-batch application, and I'm having trouble creating a unit test class with junit that tests my reader that uses JdbcPaginItemReaderBuilder.
Reader Code:
#Configuration
public class RelatorioReader {
#Bean("relatorioreader")
#StepScope
public ItemReader<Relatorio> relatorioItemReader(
#Qualifier("dataSource") DataSource dataSource,
#Value("#{jobParameters[dateParam]}") String dateParam) {
return new JdbcPagingItemReaderBuilder<Relatorio>()
.name("relatorioDiario")
.dataSource(dataSource)
.selectClause("SELET * ")
.fromClause("FROM myTable ")
.whereClause(" WHERE date = :dateParam")
.parameterValues(Collections.singletonMap("dateParam", dateParam))
.sortKeys(Collections.singletonMap("ID", Order.ASCENDING))
.rowMapper(new RelatorioMapper())
.build();
}
}
Junit Code
#ExtendWith(MockitoExtension.class)
public class RelatorioReaderTest {
#InjectMocks
RelatorioReader reader;
#Mock
DataSource dataSource;
#Test
public void test_itemReader() {
ItemReader<Relatorio> itemReader = reader.relatorioItemReader(dataSource, "2023-02-16");
assertNotNull(itemReader);
}
}
Exception when running Junit:
java.lang.IllegalArgumentException: Unable to determine PagingQueryProvider type
at org.springframework.batch.item.database.builder.JdbcPagingItemReaderBuilder.determineQueryProvider(JdbcPagingItemReaderBuilder.java:383)
at org.springframework.batch.item.database.builder.JdbcPagingItemReaderBuilder.build(JdbcPagingItemReaderBuilder.java:335)
at com.erico.relatorio.item.reader.RelatorioReader.relatorioItemReader(RelatorioReader.java:34)
at com.erico.relatorio.item.reader.RelatorioReaderTest.test_itemReader(RelatorioReaderTest.java:27)
...
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection: DataSource returned null from getConnection(): dataSource
at ...
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection: DataSource returned null from getConnection(): dataSource
at ...
When you do not specify a paging query provider, the builder will try to determine a suitable one from the meta-data of your data source. Since you are using a mocked database, you need to mock the call to getConnection(). Otherwise, you have to use a stub database for tests (like an embedded H2 or HSQL).
If you know what datasource you will be using, the best way is to specify its paging query provider implementation in your builder. Here is an example if you use H2:
#Configuration
public class RelatorioReader {
#Bean("relatorioreader")
#StepScope
public ItemReader<Relatorio> relatorioItemReader(
#Qualifier("dataSource") DataSource dataSource,
#Value("#{jobParameters[dateParam]}") String dateParam) {
return new JdbcPagingItemReaderBuilder<Relatorio>()
.name("relatorioDiario")
.dataSource(dataSource)
.selectClause("SELET * ")
.fromClause("FROM myTable ")
.whereClause(" WHERE date = :dateParam")
.parameterValues(Collections.singletonMap("dateParam", dateParam))
.sortKeys(Collections.singletonMap("ID", Order.ASCENDING))
.rowMapper(new RelatorioMapper())
.queryProvider(new H2PagingQueryProvider())
.build();
}
}
Related
I'm setting up an instance of Spring Cloud Data Flow. I've run the following commands:
java -jar spring-cloud-dataflow-server-2.9.2.jar \
--spring.cloud.dataflow.features.streams-enabled=false \
--spring.cloud.dataflow.features.schedules-enabled=true \
--spring.datasource.url=jdbc:postgresql://localhost:5432/batch \
--spring.datasource.username=postgres \
--spring.datasource.password=postgres \
--spring.datasource.driver-class-name=org.postgresql.Driver \
--spring.datasource.initialization_mode=always
I've developed a batch job using spring batch to be deployed in this platform. The job uses two data sources: batch for Spring and task Metadata and app_db for my business logic. When I run the app locally, it persists metadata in batch and my business data in app_db, as expected. The problem is when I try to execute de job inside the Spring Cloud Dataflow. The platform overrides my configured business logic database and uses only the batch database, which is supposed to store metadata only.
application.yaml
spring:
batch:
datasource:
url: jdbc:postgresql://localhost:5432/batch
username: postgres
password: postgres
datasource:
url: jdbc:postgresql://localhost:5432/app_db
username: postgres
password: postgres
DatasourceConfiguration
public class DatasourceConfiguration {
#Bean
#ConfigurationProperties("spring.datasource")
#Primary
public DataSourceProperties dataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#Primary
public DataSource dataSource(DataSourceProperties dataSourceProperties) {
return dataSourceProperties.initializeDataSourceBuilder().build();
}
#Bean(name = "batchDataSourceProperties")
#ConfigurationProperties("spring.batch.datasource")
public DataSourceProperties batchDataSourceProperties() {
return new BatchDataSourceProperties();
}
#Bean(name = "batchDataSource")
public DataSource batchDataSource() {
return batchDataSourceProperties.initializeDataSourceBuilder().build();
}
}
#SpringBootApplication
#EnableTask
#EnableBatchProcessing
public class BatchApplication {
#Bean
public TaskConfigurer taskConfigurer(#Qualifier("batchDataSource") DataSource dataSource) {
return new DefaultTaskConfigurer(dataSource);
}
#Bean
public BatchConfigurer batchConfigurer(#Qualifier("batchDataSource") DataSource dataSource) {
return new DefaultBatchConfigurer(dataSource);
}
public static void main(String[] args) {
SpringApplication.run(BatchApplication.class, args);
}
}
Job
#Bean
public Job startJob(JobBuilderFactory jobBuilderFactory, DataSource dataSource) {
try {
System.out.println(dataSource.getConnection().getMetaData().getURL().toString());
} catch (Exception e) {
//TODO: handle exception
}
}
When I look at the data source,jdbc:postgresql://localhost:5432/app_db will be printed when the batch is executed from local and jdbc:postgresql://localhost:5432/batch will be printed when the batch (task) is executed from SCDF.
I want to know how dataflow is overriding application the spring.datasource even though I am not passing any arguments while executing the task. Please suggest a solution to avoid the overriding of datasource.
One solution I am thinking of is creating AppDatasourceConfiguration(app.datasource) use it. But is there a possibility to use spring.datasource without getting overiddien by SCDF.
I am migrating our application to openjdk11 and with this setup my application is throwing below error.
PLease help on this
Note : With Jdk 1.8 the same code and configurations are working fine .
Java version: openjdk 11
Springboot-hadoop : 2.4.0 RELEASE
application properties
spring.hadoop.fsshell.enabled=false
#hadoop security properties
hadoop.config.key=hadoop.security.authentication
hadoop.config.value=Kerberos
#Hive connection properties
hive.datasource.keytab=/config/security/sit.001.keytab
hive.datasource.drivername=org.apache.hive.jdbc.HiveDriver
hive.datasource.username=ssit.001
#hive.datasource.password=password
hive.truststore.file=/config/security/hivetrust.jks
hive.krb5.conf=/config/security/krb5.conf
hive.datasource.url=url
hive.krb5.conf.debug.prop=sun.security.krb5.debug
hive.krb5.conf.isdebug=true
Java changes
#Value("${hive.datasource.drivername}")
private String driverName;
#Value("${hive.datasource.url}")
private String jdbcUrl;
#Value("${hive.datasource.username}")
private String userId;
#Value("${hive.datasource.keytab}")
private String keytab;
#Value("${hive.krb5.conf}")
private String kerberosConf;
#Value("${hadoop.config.key}")
public String hadoopConfigKey;
#Value("${hadoop.config.value}")
public String hadoopConfigValue;
#Bean(name = "hiveDS")
public DataSource configureHiveDataSource() throws IOException, ClassNotFoundException, SQLException {
Connection con = null;
// System.setProperty("hadoop.home.dir", hadoopHome);
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
System.setProperty("java.security.krb5.conf", kerberosConf);
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.set(hadoopConfigKey, hadoopConfigValue);
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(userId, keytab);
Class.forName(driverName);
con = DriverManager.getConnection(jdbcUrl);
LOGGER.info("Hive Db Connected");
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverName);
dataSource.setUrl(jdbcUrl);
return dataSource;
}
#Bean(name = "hiveJdbc")
public JdbcTemplate getHiveJdbcTemplate(#Qualifier("hiveDS") DataSource hiveDS) {
return new JdbcTemplate(hiveDS);
}
#Bean(name = "hiveNamedJdbc")
public NamedParameterJdbcTemplate getHiveNamedJdbcTemplate(#Qualifier("hiveDS") DataSource hiveNamedDS) {
return new NamedParameterJdbcTemplate(hiveNamedDS);
}
}
2021-04-28T21:18:18.829+0530 [main] ERROR o.s.d.h.c.c.a.AbstractConfiguredAnnotationBuilder - Failed to perform build. Returning null
java.lang.IllegalArgumentException: Bean name must not be null
at org.springframework.util.Assert.notNull(Assert.java:201)
Error creating bean with name 'hadoopConfiguration' defined in class path resource [org/springframework/data/hadoop/config/annotation/configuration/SpringHadoopConfiguration.class]: Bean instantiation via factory method failed; nested exception is **org.springframework.beans.BeanInstantiationException: Failed to instantiate **[org.apache.hadoop.conf.Configuration]: Factory method 'configuration' threw exception; nested exception is java.lang.NullPointerException
I'm trying to write a simple function which will connect to postgres and execute a select statement.
PostgresqlConnectionFactory connectionFactory = new PostgresqlConnectionFactory(
PostgresqlConnectionConfiguration.builder()
.host("localhost")
.port(5432)
.database("MyDB")
.username("username")
.password("password").build());
DatabaseClient client = DatabaseClient.create(connectionFactory);
Flux<Map<String, Object>> result = client.execute("select * from table").fetch().all();
result.map(s -> {
System.out.println(s);
return null;
});
The above piece of code isn't printing anything. There is no error as well. I can connect to DB using the same credentials. What is missing in the code to stream data from DB?
Create the configuration class which is similar to the below code to connect to the PostgreSQL database
#Configuration
#EnableR2dbcRepositories
public class DatabaseConfig extends AbstractR2dbcConfiguration {
#Override
public ConnectionFactory connectionFactory() {
return ConnectionFactories.get("r2dbc:postgresql://localhost:5432/DATABASE_NAME");
}
}
Is it possible to retrieve a OracleDataSource from the default SpringBoot 2 Hikari connection pool using a NamedParameterJdbcTemplate object?
Using Java 8, Oracle 11g (ojdbc6-11.2.0.1.jar) and gradle
This is what i've tried.
#Repository
public class MyClass{
#Autowired
NamedParameterJdbcTemplate jdbcTemplate;
public void myMethod(){
try{
//OracleDataSource ods = new OracleDataSource(); // This works but is obviously not Spring
OracleDataSource ods = (OracleDataSource) jdbcTemplate.getJdbcTemplate().getDataSource(); // This fails
ods.setURL(url);
ods.setUser(user);
ods.setPassword(pass);
...
catch(Exception e){
System.out.println("In Exception");
e.printStackTrace();
}
}
}
Application.properties:
spring.datasource.url=jdbc:oracle:thin:#//${ORA_HOST}:${ORA_PORT}/${ORA_SID}
spring.datasource.username=${USER}
spring.datasource.password=${PASS}
Error message:
In Exception
java.lang.ClassCastException: com.zaxxer.hikari.HikariDataSource cannot be cast to oracle.jdbc.pool.OracleDataSource
I don't think this is possible (or neccessary). The easiest way is to unwrap() a connection object, which has already connected to the dB:
Connection conn = this.jdbcTemplate.getJdbcTemplate().getDataSource().getConnection().unwrap(OracleConnection.class);
Just a query like, how are you able to get OracleConnection.class, because in my case I get squiggly line underneath OracleConnection.class and there are no packages available to import.
I am writing a Spring Boot (Batch) application, that should exit with a specific exit code. A requirement is to return an exit code, when the database cannot be connected.
My approach is to handle this exception as early as possible by explicitly creating a DataSource bean, calling getConnection() and catch and throw a custom exception that implements ExitCodeGenerator. The configuration is as follows:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
...
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSourceProperties dataSourceProps() {
return new DataSourceProperties();
}
#Bean
#ConfigurationProperties("spring.datasource")
public DataSource customDataSource(DataSourceProperties props) {
DataSource ds = props.initializeDataSourceBuilder().create().build();
try {
ds.getConnection();
} catch (SQLException e) {
throw new DBConnectionException(e); // implements ExitCodeGenerator interface
}
return ds;
}
...
}
I want to reuse as much of the Spring Boot Autoconfiguration as possible, thats why I use the #ConfigurationProperties. I do not know if this is the way to go.
A call on DataSourceProperties.getUrl() returns the configured url (from my properties file):
spring.datasource.url=jdbc:oracle:....
But why does Spring Boot throw this exception when I call DataSource.getConnection():
java.sql.SQLException: The url cannot be null
at java.sql.DriverManager.getConnection(DriverManager.java:649) ~[?:1.8.0_141]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_141]
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:308) ~[tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:203) ~[tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:735) ~[tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:667) ~[tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:482) [tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154) [tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) [tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) [tomcat-jdbc-8.5.15.jar:?]
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) [tomcat-jdbc-8.5.15.jar:?]
at com.foo.bar.BatchConfiguration.customDataSource(BatchConfiguration.java:xxx) [main/:?]
...
Or do you know some cleaner way of handling this situation?
Thanks
Edit: Spring Boot version is 1.5.4
The error is subtle and lies in the line
DataSource ds = props.initializeDataSourceBuilder().create().build();
The create() creates a new DataSourceBuilder and erases the preconfigured properties.
props.initializeDataSourceBuilder() already returns a DataSourceBuilder with all the properties (url, username etc.) set. So you only have to add new properties or directly build() it. So the solution is removing create():
DataSource ds = props.initializeDataSourceBuilder().build();
In this context the dataSourceProps() method bean can be removed too.
It looks like you dont set any value to your Datasource.
props.initializeDataSourceBuilder().create().build(); does not set the values of your properties to your datasource. It just creates and builds one.
Try to set your values manually by using the static DataSourceBuilder. You will get the values from your dataSourceProps bean like that:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
...
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSourceProperties dataSourceProps() {
return new DataSourceProperties();
}
#Bean
#ConfigurationProperties("spring.datasource")
public DataSource customDataSource(DataSourceProperties props) {
DataSource ds = DataSourceBuilder.create()
.driverClassName(dataSourceProps().getDriverClassName())
.url(dataSourceProps().getUrl())
.username(dataSourceProps().getUsername())
.password(dataSourceProps().getPassword())
.build();
try {
ds.getConnection();
} catch (SQLException e) {
throw new DBConnectionException(e); // implements ExitCodeGenerator interface
}
return ds;
}
...
}