Develop Reference

How to test JdbcPagingItemReaderBuilder with Junit

I'm creating a spring-batch application, and I'm having trouble creating a unit test class with junit that tests my reader that uses JdbcPaginItemReaderBuilder.
Reader Code:
#Configuration
public class RelatorioReader {
#Bean("relatorioreader")
#StepScope
public ItemReader<Relatorio> relatorioItemReader(
#Qualifier("dataSource") DataSource dataSource,
#Value("#{jobParameters[dateParam]}") String dateParam) {
return new JdbcPagingItemReaderBuilder<Relatorio>()
.name("relatorioDiario")
.dataSource(dataSource)
.selectClause("SELET * ")
.fromClause("FROM myTable ")
.whereClause(" WHERE date = :dateParam")
.parameterValues(Collections.singletonMap("dateParam", dateParam))
.sortKeys(Collections.singletonMap("ID", Order.ASCENDING))
.rowMapper(new RelatorioMapper())
.build();
}
}
Junit Code
#ExtendWith(MockitoExtension.class)
public class RelatorioReaderTest {
#InjectMocks
RelatorioReader reader;
#Mock
DataSource dataSource;
#Test
public void test_itemReader() {
ItemReader<Relatorio> itemReader = reader.relatorioItemReader(dataSource, "2023-02-16");
assertNotNull(itemReader);
}
}
Exception when running Junit:
java.lang.IllegalArgumentException: Unable to determine PagingQueryProvider type
at org.springframework.batch.item.database.builder.JdbcPagingItemReaderBuilder.determineQueryProvider(JdbcPagingItemReaderBuilder.java:383)
at org.springframework.batch.item.database.builder.JdbcPagingItemReaderBuilder.build(JdbcPagingItemReaderBuilder.java:335)
at com.erico.relatorio.item.reader.RelatorioReader.relatorioItemReader(RelatorioReader.java:34)
at com.erico.relatorio.item.reader.RelatorioReaderTest.test_itemReader(RelatorioReaderTest.java:27)
...
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection: DataSource returned null from getConnection(): dataSource
at ...
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection: DataSource returned null from getConnection(): dataSource
at ...

When you do not specify a paging query provider, the builder will try to determine a suitable one from the meta-data of your data source. Since you are using a mocked database, you need to mock the call to getConnection(). Otherwise, you have to use a stub database for tests (like an embedded H2 or HSQL).
If you know what datasource you will be using, the best way is to specify its paging query provider implementation in your builder. Here is an example if you use H2:
#Configuration
public class RelatorioReader {
#Bean("relatorioreader")
#StepScope
public ItemReader<Relatorio> relatorioItemReader(
#Qualifier("dataSource") DataSource dataSource,
#Value("#{jobParameters[dateParam]}") String dateParam) {
return new JdbcPagingItemReaderBuilder<Relatorio>()
.name("relatorioDiario")
.dataSource(dataSource)
.selectClause("SELET * ")
.fromClause("FROM myTable ")
.whereClause(" WHERE date = :dateParam")
.parameterValues(Collections.singletonMap("dateParam", dateParam))
.sortKeys(Collections.singletonMap("ID", Order.ASCENDING))
.rowMapper(new RelatorioMapper())
.queryProvider(new H2PagingQueryProvider())
.build();
}
}

How to force Spring boot embedded tomcat to use SNI (Server Name Indication)

I need to make sure that my client supports SNI and it works. To do that, I want to start some dummy server (Spring boot application with it's embedded Tomcat 9) that requires hostname from client. The question is how to force Spring boot embedded Tomcat to require SNI from client?
I've read that to do that, we need to provide to server 2 (or more) certificates.
I tried to follow example from this topic. But it does not work for me.
I generated 2 keystores mycompany1.keystore and mycompany2.keystore with 1 certificate in each of them. And 1 truststore mycompany.truststore that stores both these certificates. Certificates have CN my.hostname.com and my.another.hostname.com.
This is the way how I configure embedded tomcat:
#Component
public class MultipleHostsTomcatFactory {
#Value("${abc.com.key-store}")
String abcComKeyStore;
#Value("${xyz.com.key-store}")
String xyzComKeyStore;
#Value("${server.port}")
int serverPort;
#Value("${server.http.port:8080}")
private int httpPort;
#Bean
public ServletWebServerFactory servletContainer() throws Exception {
TomcatServletWebServerFactory factor = new TomcatServletWebServerFactory();
factor.addAdditionalTomcatConnectors(createSSLConnectorForMultipleHosts());
return factor;
}
private Connector createSSLConnectorForMultipleHosts() {
Connector connector = null;
try {
connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
connector.setScheme("https");
connector.setSecure(true);
connector.setProperty("SSLEnabled", "true");
connector.setProperty("defaultSSLHostConfigName", /*"*.abc.com"*/ "*.hostname.com");
connector.setPort(8444);
// *.abc.com
SSLHostConfig sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("*.abc.com");
SSLHostConfigCertificate sslHostConfigCertificate = new SSLHostConfigCertificate(sslHostConfig, SSLHostConfigCertificate.Type.EC);
sslHostConfigCertificate.setCertificateKeystoreFile(abcComKeyStore);
sslHostConfig.addCertificate(sslHostConfigCertificate);
connector.addSslHostConfig(sslHostConfig);
// *.xyz.com
sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("*.xyz.com");
sslHostConfigCertificate = new SSLHostConfigCertificate(sslHostConfig, SSLHostConfigCertificate.Type.EC);
sslHostConfigCertificate.setCertificateKeystoreFile(xyzComKeyStore);
sslHostConfig.addCertificate(sslHostConfigCertificate);
connector.addSslHostConfig(sslHostConfig);
} catch (Exception e) {
System.out.println("Catched exception!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
e.printStackTrace();
}
return connector;
}
}
This code is based on this topic.
But I have an error:
2022-01-13 16:10:21.301 ERROR 22364 --- [ main] org.apache.catalina.util.LifecycleBase : Failed to initialize component [Connector[HTTP/1.1-8444]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
...
Caused by: java.lang.IllegalArgumentException: Keystore was tampered with, or password was incorrect
...
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
...
Caused by: java.security.UnrecoverableKeyException: Password verification failed
Looks like I miss somethimg.
So, how can I force embedded Tomcat to enable SNI?

I solved the problem. Maybe my explanation will help somebody. I just followed the errors that I saw and configured all data that server missed. I don't configure certificate, because I have keystore that stores this certificate. My final server has 2 hosts that support only TLSv1.3 (for my needs):
private Connector createSSLConnectorForMultipleHosts() {
Connector connector = null;
try {
Http2Protocol http2Protocol = new Http2Protocol();
Http11NioProtocol http11Protocol = new Http11NioProtocol();
http2Protocol.setHttp11Protocol(http11Protocol);
connector = new Connector(http11Protocol);
connector.addUpgradeProtocol(http2Protocol);
connector.setScheme("https");
connector.setSecure(true);
connector.setProperty("SSLEnabled", "true");
connector.setProperty("defaultSSLHostConfigName", "my.hostname.com");
connector.setPort(8445);
// first
SSLHostConfig sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("my.hostname.com");
sslHostConfig.setCertificateKeystorePassword("mypassword");
sslHostConfig.setCertificateKeystoreFile(firstKeyStore);
sslHostConfig.setProtocols("TLSv1.3");
connector.addSslHostConfig(sslHostConfig);
// second
sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("my.another.hostname.com");
sslHostConfig.setCertificateKeystorePassword("mypassword");
sslHostConfig.setCertificateKeystoreFile(secondKeyStore);
sslHostConfig.setProtocols("TLSv1.3");
connector.addSslHostConfig(sslHostConfig);
} catch (Exception e) {
e.printStackTrace();
}
return connector;
}

hadoop configuration error in runtime with openjdk11

I am migrating our application to openjdk11 and with this setup my application is throwing below error.
PLease help on this
Note : With Jdk 1.8 the same code and configurations are working fine .
Java version: openjdk 11
Springboot-hadoop : 2.4.0 RELEASE
application properties
spring.hadoop.fsshell.enabled=false
#hadoop security properties
hadoop.config.key=hadoop.security.authentication
hadoop.config.value=Kerberos
#Hive connection properties
hive.datasource.keytab=/config/security/sit.001.keytab
hive.datasource.drivername=org.apache.hive.jdbc.HiveDriver
hive.datasource.username=ssit.001
#hive.datasource.password=password
hive.truststore.file=/config/security/hivetrust.jks
hive.krb5.conf=/config/security/krb5.conf
hive.datasource.url=url
hive.krb5.conf.debug.prop=sun.security.krb5.debug
hive.krb5.conf.isdebug=true
Java changes
#Value("${hive.datasource.drivername}")
private String driverName;
#Value("${hive.datasource.url}")
private String jdbcUrl;
#Value("${hive.datasource.username}")
private String userId;
#Value("${hive.datasource.keytab}")
private String keytab;
#Value("${hive.krb5.conf}")
private String kerberosConf;
#Value("${hadoop.config.key}")
public String hadoopConfigKey;
#Value("${hadoop.config.value}")
public String hadoopConfigValue;
#Bean(name = "hiveDS")
public DataSource configureHiveDataSource() throws IOException, ClassNotFoundException, SQLException {
Connection con = null;
// System.setProperty("hadoop.home.dir", hadoopHome);
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
System.setProperty("java.security.krb5.conf", kerberosConf);
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.set(hadoopConfigKey, hadoopConfigValue);
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(userId, keytab);
Class.forName(driverName);
con = DriverManager.getConnection(jdbcUrl);
LOGGER.info("Hive Db Connected");
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverName);
dataSource.setUrl(jdbcUrl);
return dataSource;
}
#Bean(name = "hiveJdbc")
public JdbcTemplate getHiveJdbcTemplate(#Qualifier("hiveDS") DataSource hiveDS) {
return new JdbcTemplate(hiveDS);
}
#Bean(name = "hiveNamedJdbc")
public NamedParameterJdbcTemplate getHiveNamedJdbcTemplate(#Qualifier("hiveDS") DataSource hiveNamedDS) {
return new NamedParameterJdbcTemplate(hiveNamedDS);
}
}
2021-04-28T21:18:18.829+0530 [main] ERROR o.s.d.h.c.c.a.AbstractConfiguredAnnotationBuilder - Failed to perform build. Returning null
java.lang.IllegalArgumentException: Bean name must not be null
at org.springframework.util.Assert.notNull(Assert.java:201)
Error creating bean with name 'hadoopConfiguration' defined in class path resource [org/springframework/data/hadoop/config/annotation/configuration/SpringHadoopConfiguration.class]: Bean instantiation via factory method failed; nested exception is **org.springframework.beans.BeanInstantiationException: Failed to instantiate **[org.apache.hadoop.conf.Configuration]: Factory method 'configuration' threw exception; nested exception is java.lang.NullPointerException

Cannot get Jedis connection, Could not get a resource from the pool

I am creating a Web Application and using Redis for data storage.
I have used JedisPoolConfig (redis client-jedis version 2.9.0) .
So when I test my application with help of Apache Jmeter with following configurations :
Number of threads(Users): 30
Ramp-Up period(in seconds) : 0
Loop-Count : 1
and JedisPoolConfig with following configurations :
redis.sentinel.enable=true
redis.sentinel.ip=localhost
redis.sentinel.port=6379
redis.instance.account.masterName=master
redis.pool.max-active = 20
redis.pool.max-idle = 10
with this code (which is created as Bean on startup of Spring Application) :
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setPort(port);
factory.setHostName(hostName);
factory.afterPropertiesSet();
factory.setUsePool(true);
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(redisPoolMaxActive);
jedisPoolConfig.setMaxIdle(redisPoolMaxIdle);
factory.setPoolConfig(jedisPoolConfig);
return factory;
with this code for fetching new Connection everytime a Request comes :
#Autowired
private JedisConnectionFactory redisConnectionFactory;
private StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
private RedisTemplate getNewConnection(){
RedisTemplate redisTemplate = new RedisTemplate();
redisTemplate.setConnectionFactory(redisConnectionFactory);
redisTemplate.setKeySerializer(stringRedisSerializer);
redisTemplate.setHashKeySerializer(stringRedisSerializer);
redisTemplate.setValueSerializer(stringRedisSerializer);
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
Edit : my usage for redisTemplate for multi/exec inside Redis :
redisTemplate.setEnableTransactionSupport(true);
redisTemplate.multi();
for (ArrayList<Pair<String, String>> pairs : keys){
for (Pair<String,String> pair: pairs)
redisTemplate.opsForHash().get(makeMap(pair.getFirst(), version) , pair.getSecond());
}
return redisTemplate.exec();
I successfully get the response for 20 requests but could not get the response for rest 10 requests. The issue I am facing is after 20 connections have been used by first 20 requests , the rest 10 requests are not able to get the jedis connection from the pool and thus I get the following exception :
2017-10-06 17:38:11.753 ERROR 3539 --- [io-8080-exec-12] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool] with root cause
java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) ~[jedis-2.9.0.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:129) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisConnectionUtils.bindConnection(RedisConnectionUtils.java:67) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:192) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisTemplate.multi(RedisTemplate.java:868) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at net.media.mml.redis.repository.RedisRepository.makeTransaction(RedisRepository.java:50) ~[max-data-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at net.media.mml.redis.repository.RedisRepository.getFromRedis(RedisRepository.java:66) ~[max-data-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at net.media.mml.service.DimensionGetter.getDimensions(DimensionGetter.java:64) ~[ml-api.jar:0.0.1-SNAPSHOT]
at net.media.mml.service.MLQueriesGetter.makeQueries(MLQueriesGetter.java:32) ~[ml-api.jar:0.0.1-SNAPSHOT]
at net.media.mml.controller.MaxQueryController.query(MaxQueryController.java:61) ~[ml-api.jar:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.10.RELEASE.jar:4.3.10.RELEASE]
I have searched over the net but not able to find the solution to this. Can anybody help me?

Using Redis transactions requires invocations in a callback. setEnableTransactionSupport(true) is intended for transaction-manager/#Transactional use. See the reference documentation for further details.
Do not use setEnableTransactionSupport(true) unless you have a transaction manager in place.
You're code should look like:
List<Object> txResults = redisTemplate.execute(new SessionCallback<List<Object>>() {
public List<Object> execute(RedisOperations operations) throws DataAccessException {
operations.multi();
operations.opsForSet().add("key", "value1");
// This will contain the results of all ops in the transaction
return operations.exec();
}
});

JedisConnectionException: java.net.ConnectException: Address already in use

i have meet a problem when i use redis in multi-threads environment.
some redis config codes below.
#Bean
public RedisTemplate<String, String> redisTemplate() {
StringRedisTemplate redisTemplate = new StringRedisTemplate(connectionFactory());
redisTemplate.setDefaultSerializer(serializer());
return redisTemplate;
}
private RedisConnectionFactory connectionFactory() {
JedisShardInfo shardInfo = new JedisShardInfo(host, port);
JedisConnectionFactory factory = new JedisConnectionFactory(shardInfo);
factory.setPassword(password);
factory.setDatabase(db);
factory.setTimeout(timeOut);
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxTotal(20);
factory.setPoolConfig(config);
return factory;
}
i execute a brpoplpush operation in multi-threads. some codes below
RedisConnection redisConnection = redisTemplate.getConnectionFactory().getConnection();
try {
value = redisConnection.bRPopLPush(5, RWeixinDao.WEIXIN_MSG_KEY.getBytes(), key.getBytes());
} finally {
redisConnection.close();
}
and i got some exceptions
23:35:56.859 [pool-1-thread-4] ERROR c.j.n.i.s.impl.MessageProcessorImpl - process weixin message fail.
org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Address already in use
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:162)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:251)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:58)
at com.jiyu.nowitzki.im.service.impl.MessageProcessorImpl.process(MessageProcessorImpl.java:82)
at com.jiyu.nowitzki.im.service.impl.MessageProcessorImpl.lambda$run$0(MessageProcessorImpl.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Address already in use
at redis.clients.jedis.Connection.connect(Connection.java:164)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:82)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1641)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:159)
... 7 common frames omitted
it says this code is wrong
RedisConnection redisConnection = redisTemplate.getConnectionFactory().getConnection();
does it cause by i call the close() method?
anyone could tell me plz... thxs