i have meet a problem when i use redis in multi-threads environment.
some redis config codes below.
#Bean
public RedisTemplate<String, String> redisTemplate() {
StringRedisTemplate redisTemplate = new StringRedisTemplate(connectionFactory());
redisTemplate.setDefaultSerializer(serializer());
return redisTemplate;
}
private RedisConnectionFactory connectionFactory() {
JedisShardInfo shardInfo = new JedisShardInfo(host, port);
JedisConnectionFactory factory = new JedisConnectionFactory(shardInfo);
factory.setPassword(password);
factory.setDatabase(db);
factory.setTimeout(timeOut);
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxTotal(20);
factory.setPoolConfig(config);
return factory;
}
i execute a brpoplpush operation in multi-threads. some codes below
RedisConnection redisConnection = redisTemplate.getConnectionFactory().getConnection();
try {
value = redisConnection.bRPopLPush(5, RWeixinDao.WEIXIN_MSG_KEY.getBytes(), key.getBytes());
} finally {
redisConnection.close();
}
and i got some exceptions
23:35:56.859 [pool-1-thread-4] ERROR c.j.n.i.s.impl.MessageProcessorImpl - process weixin message fail.
org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Address already in use
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:162)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:251)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:58)
at com.jiyu.nowitzki.im.service.impl.MessageProcessorImpl.process(MessageProcessorImpl.java:82)
at com.jiyu.nowitzki.im.service.impl.MessageProcessorImpl.lambda$run$0(MessageProcessorImpl.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Address already in use
at redis.clients.jedis.Connection.connect(Connection.java:164)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:82)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1641)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:159)
... 7 common frames omitted
it says this code is wrong
RedisConnection redisConnection = redisTemplate.getConnectionFactory().getConnection();
does it cause by i call the close() method?
anyone could tell me plz... thxs
Related
I'm creating a spring-batch application, and I'm having trouble creating a unit test class with junit that tests my reader that uses JdbcPaginItemReaderBuilder.
Reader Code:
#Configuration
public class RelatorioReader {
#Bean("relatorioreader")
#StepScope
public ItemReader<Relatorio> relatorioItemReader(
#Qualifier("dataSource") DataSource dataSource,
#Value("#{jobParameters[dateParam]}") String dateParam) {
return new JdbcPagingItemReaderBuilder<Relatorio>()
.name("relatorioDiario")
.dataSource(dataSource)
.selectClause("SELET * ")
.fromClause("FROM myTable ")
.whereClause(" WHERE date = :dateParam")
.parameterValues(Collections.singletonMap("dateParam", dateParam))
.sortKeys(Collections.singletonMap("ID", Order.ASCENDING))
.rowMapper(new RelatorioMapper())
.build();
}
}
Junit Code
#ExtendWith(MockitoExtension.class)
public class RelatorioReaderTest {
#InjectMocks
RelatorioReader reader;
#Mock
DataSource dataSource;
#Test
public void test_itemReader() {
ItemReader<Relatorio> itemReader = reader.relatorioItemReader(dataSource, "2023-02-16");
assertNotNull(itemReader);
}
}
Exception when running Junit:
java.lang.IllegalArgumentException: Unable to determine PagingQueryProvider type
at org.springframework.batch.item.database.builder.JdbcPagingItemReaderBuilder.determineQueryProvider(JdbcPagingItemReaderBuilder.java:383)
at org.springframework.batch.item.database.builder.JdbcPagingItemReaderBuilder.build(JdbcPagingItemReaderBuilder.java:335)
at com.erico.relatorio.item.reader.RelatorioReader.relatorioItemReader(RelatorioReader.java:34)
at com.erico.relatorio.item.reader.RelatorioReaderTest.test_itemReader(RelatorioReaderTest.java:27)
...
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection: DataSource returned null from getConnection(): dataSource
at ...
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection: DataSource returned null from getConnection(): dataSource
at ...
When you do not specify a paging query provider, the builder will try to determine a suitable one from the meta-data of your data source. Since you are using a mocked database, you need to mock the call to getConnection(). Otherwise, you have to use a stub database for tests (like an embedded H2 or HSQL).
If you know what datasource you will be using, the best way is to specify its paging query provider implementation in your builder. Here is an example if you use H2:
#Configuration
public class RelatorioReader {
#Bean("relatorioreader")
#StepScope
public ItemReader<Relatorio> relatorioItemReader(
#Qualifier("dataSource") DataSource dataSource,
#Value("#{jobParameters[dateParam]}") String dateParam) {
return new JdbcPagingItemReaderBuilder<Relatorio>()
.name("relatorioDiario")
.dataSource(dataSource)
.selectClause("SELET * ")
.fromClause("FROM myTable ")
.whereClause(" WHERE date = :dateParam")
.parameterValues(Collections.singletonMap("dateParam", dateParam))
.sortKeys(Collections.singletonMap("ID", Order.ASCENDING))
.rowMapper(new RelatorioMapper())
.queryProvider(new H2PagingQueryProvider())
.build();
}
}
Hello I am using file poller with Spring Integration and we have recently migrated to Java 17. Spring Integration moved from 5.5.15 to 6.0.1
My old bean was:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory(){
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(serverhost);
sf.setPort(portname);
sf.setUser(username);
Resource resource = resourceLoader.getResource(sftpKeyPrivateKey);
sf.setPrivateKey(resource);
sf.setPrivateKeyPassphrase(sftpKeyPrivateKeyPassword);
sf.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry> (sf);
}
My new code for Session Factory looked like:
#Bean
public SessionFactory<org.apache.sshd.sftp.client.SftpClient.DirEntry> sftpSessionFactory()
{
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(serverhost);
sf.setPort(portname);
sf.setUser(username);
Resource resource = resourceLoader.getResource(sftpKeyPrivateKey);
sf.setPrivateKey(resource);
sf.setPrivateKeyPassphrase(sftpKeyPrivateKeyPassword);
sf.setAllowUnknownKeys(true);
return new CachingSessionFactory<org.apache.sshd.sftp.client.SftpClient.DirEntry(sf);
}
Everything else remains same. Old code was working as expected but in new code when i start my service i get below exception:
Caused by: java.lang.IllegalStateException: failed to create SFTP Session
at org.springframework.integration.sftp.session.DefaultSftpSessionFactory.getSession(DefaultSftpSessionFactory.java:291)
at org.springframework.integration.sftp.session.DefaultSftpSessionFactory.getSession(DefaultSftpSessionFactory.java:67)
at org.springframework.integration.file.remote.session.CachingSessionFactory$1.createForPool(CachingSessionFactory.java:85)
at org.springframework.integration.file.remote.session.CachingSessionFactory$1.createForPool(CachingSessionFactory.java:82)
at org.springframework.integration.util.SimplePool.doGetItem(SimplePool.java:206)
at org.springframework.integration.util.SimplePool.getItem(SimplePool.java:187) ... 23 more Caused by: org.apache.sshd.common.SshException: No more authentication methods available
at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:127)
at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:39)
at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:32)
at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43)
at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:68)
at org.springframework.integration.sftp.session.DefaultSftpSessionFactory.initClientSession(DefaultSftpSessionFactory.java:318)
at org.springframework.integration.sftp.session.DefaultSftpSessionFactory.getSession(DefaultSftpSessionFactory.java:281) ... 28 more
Caused by: org.apache.sshd.common.SshException: No more authentication methods available
I am migrating our application to openjdk11 and with this setup my application is throwing below error.
PLease help on this
Note : With Jdk 1.8 the same code and configurations are working fine .
Java version: openjdk 11
Springboot-hadoop : 2.4.0 RELEASE
application properties
spring.hadoop.fsshell.enabled=false
#hadoop security properties
hadoop.config.key=hadoop.security.authentication
hadoop.config.value=Kerberos
#Hive connection properties
hive.datasource.keytab=/config/security/sit.001.keytab
hive.datasource.drivername=org.apache.hive.jdbc.HiveDriver
hive.datasource.username=ssit.001
#hive.datasource.password=password
hive.truststore.file=/config/security/hivetrust.jks
hive.krb5.conf=/config/security/krb5.conf
hive.datasource.url=url
hive.krb5.conf.debug.prop=sun.security.krb5.debug
hive.krb5.conf.isdebug=true
Java changes
#Value("${hive.datasource.drivername}")
private String driverName;
#Value("${hive.datasource.url}")
private String jdbcUrl;
#Value("${hive.datasource.username}")
private String userId;
#Value("${hive.datasource.keytab}")
private String keytab;
#Value("${hive.krb5.conf}")
private String kerberosConf;
#Value("${hadoop.config.key}")
public String hadoopConfigKey;
#Value("${hadoop.config.value}")
public String hadoopConfigValue;
#Bean(name = "hiveDS")
public DataSource configureHiveDataSource() throws IOException, ClassNotFoundException, SQLException {
Connection con = null;
// System.setProperty("hadoop.home.dir", hadoopHome);
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
System.setProperty("java.security.krb5.conf", kerberosConf);
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.set(hadoopConfigKey, hadoopConfigValue);
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(userId, keytab);
Class.forName(driverName);
con = DriverManager.getConnection(jdbcUrl);
LOGGER.info("Hive Db Connected");
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverName);
dataSource.setUrl(jdbcUrl);
return dataSource;
}
#Bean(name = "hiveJdbc")
public JdbcTemplate getHiveJdbcTemplate(#Qualifier("hiveDS") DataSource hiveDS) {
return new JdbcTemplate(hiveDS);
}
#Bean(name = "hiveNamedJdbc")
public NamedParameterJdbcTemplate getHiveNamedJdbcTemplate(#Qualifier("hiveDS") DataSource hiveNamedDS) {
return new NamedParameterJdbcTemplate(hiveNamedDS);
}
}
2021-04-28T21:18:18.829+0530 [main] ERROR o.s.d.h.c.c.a.AbstractConfiguredAnnotationBuilder - Failed to perform build. Returning null
java.lang.IllegalArgumentException: Bean name must not be null
at org.springframework.util.Assert.notNull(Assert.java:201)
Error creating bean with name 'hadoopConfiguration' defined in class path resource [org/springframework/data/hadoop/config/annotation/configuration/SpringHadoopConfiguration.class]: Bean instantiation via factory method failed; nested exception is **org.springframework.beans.BeanInstantiationException: Failed to instantiate **[org.apache.hadoop.conf.Configuration]: Factory method 'configuration' threw exception; nested exception is java.lang.NullPointerException
I am trying to make a extra class that would take care of connection and status of the user, but I am still getting some wierd error, that I don´t understand.
This is my ´FTPConnection class
I created private ftpClient using DI (I have #Bean annotation in main class) and I am trying to connect to the server using this method from another class
#Autowired
private FTPClient ftpClient;
public void connect() {
try {
ftpClient.connect("host");
ftpClient.login("username", "password");
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftpClient.disconnect();
System.err.println("FTP server refused connection.");
System.exit(1);
}
}
catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
And this is the class with the RequestMapping methods
#PostMapping("/uploadFile")
public void uploadFile(#RequestParam("file") MultipartFile file) {
var connection = new FTPConnection();
try {
connection.connect();
System.out.println(connection.isConnected());
FTPClient ftpClient = connection.getFtpClient();
ftpClient.changeWorkingDirectory("/usb");
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
[...]
And the error that I am getting is
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root
java.lang.NullPointerException: null
at com.tenpetr.FTPFileUploader.FTPConnection.connect(FTPConnection.java:16) ~[main/:na]
at com.tenpetr.FTPFileUploader.FileTransfer.uploadFile(FileTransfer.java:29) ~[main/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ...
Dou you have any idea, why I am stuck here? Thanks for your advices.
My guess is (will update after having more data) that the FtpClient is not correctly initialized and thus the FtpConnection is not correctly initialized as well.
TO BE UPDATED
Defining a bean with singleton scope means the container creates a single instance of that bean
edited:
#Bean
#Scope("singleton")
public FTPClient ftpClient(){
return new FTPClient();
}
This won't work if you're trying to autowire an interface
I am creating a Web Application and using Redis for data storage.
I have used JedisPoolConfig (redis client-jedis version 2.9.0) .
So when I test my application with help of Apache Jmeter with following configurations :
Number of threads(Users): 30
Ramp-Up period(in seconds) : 0
Loop-Count : 1
and JedisPoolConfig with following configurations :
redis.sentinel.enable=true
redis.sentinel.ip=localhost
redis.sentinel.port=6379
redis.instance.account.masterName=master
redis.pool.max-active = 20
redis.pool.max-idle = 10
with this code (which is created as Bean on startup of Spring Application) :
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setPort(port);
factory.setHostName(hostName);
factory.afterPropertiesSet();
factory.setUsePool(true);
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(redisPoolMaxActive);
jedisPoolConfig.setMaxIdle(redisPoolMaxIdle);
factory.setPoolConfig(jedisPoolConfig);
return factory;
with this code for fetching new Connection everytime a Request comes :
#Autowired
private JedisConnectionFactory redisConnectionFactory;
private StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
private RedisTemplate getNewConnection(){
RedisTemplate redisTemplate = new RedisTemplate();
redisTemplate.setConnectionFactory(redisConnectionFactory);
redisTemplate.setKeySerializer(stringRedisSerializer);
redisTemplate.setHashKeySerializer(stringRedisSerializer);
redisTemplate.setValueSerializer(stringRedisSerializer);
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
Edit : my usage for redisTemplate for multi/exec inside Redis :
redisTemplate.setEnableTransactionSupport(true);
redisTemplate.multi();
for (ArrayList<Pair<String, String>> pairs : keys){
for (Pair<String,String> pair: pairs)
redisTemplate.opsForHash().get(makeMap(pair.getFirst(), version) , pair.getSecond());
}
return redisTemplate.exec();
I successfully get the response for 20 requests but could not get the response for rest 10 requests. The issue I am facing is after 20 connections have been used by first 20 requests , the rest 10 requests are not able to get the jedis connection from the pool and thus I get the following exception :
2017-10-06 17:38:11.753 ERROR 3539 --- [io-8080-exec-12] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool] with root cause
java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) ~[jedis-2.9.0.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:129) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisConnectionUtils.bindConnection(RedisConnectionUtils.java:67) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:192) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at org.springframework.data.redis.core.RedisTemplate.multi(RedisTemplate.java:868) ~[spring-data-redis-1.8.6.RELEASE.jar:na]
at net.media.mml.redis.repository.RedisRepository.makeTransaction(RedisRepository.java:50) ~[max-data-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at net.media.mml.redis.repository.RedisRepository.getFromRedis(RedisRepository.java:66) ~[max-data-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at net.media.mml.service.DimensionGetter.getDimensions(DimensionGetter.java:64) ~[ml-api.jar:0.0.1-SNAPSHOT]
at net.media.mml.service.MLQueriesGetter.makeQueries(MLQueriesGetter.java:32) ~[ml-api.jar:0.0.1-SNAPSHOT]
at net.media.mml.controller.MaxQueryController.query(MaxQueryController.java:61) ~[ml-api.jar:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.10.RELEASE.jar:4.3.10.RELEASE]
I have searched over the net but not able to find the solution to this. Can anybody help me?
Using Redis transactions requires invocations in a callback. setEnableTransactionSupport(true) is intended for transaction-manager/#Transactional use. See the reference documentation for further details.
Do not use setEnableTransactionSupport(true) unless you have a transaction manager in place.
You're code should look like:
List<Object> txResults = redisTemplate.execute(new SessionCallback<List<Object>>() {
public List<Object> execute(RedisOperations operations) throws DataAccessException {
operations.multi();
operations.opsForSet().add("key", "value1");
// This will contain the results of all ops in the transaction
return operations.exec();
}
});