I'm working on a project whereby my spring service needs to connect to multiple Mongo DB's.
In order to achieve the ability to connect to multiple mongo db's I've created a MongoDatabaseFactory config class to represent each connection. I'am however getting random mongo SSL timeout errors. Normally when I would connect to a single mongo db I would create the client like so:
#Value("${spring.data.mongodb.host}")
private String connectionString;
#Bean
public MongoClient mongoClient() {
CodecRegistry pojoCodecRegistry = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry codecRegistry = fromRegistries(MongoClientSettings.getDefaultCodecRegistry(), pojoCodecRegistry);
return MongoClients.create(MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(connectionString))
.uuidRepresentation(UuidRepresentation.STANDARD)
.codecRegistry(codecRegistry)
.build());
}
#Bean
public MongoClientSettings mongoClientSettings() {
final MongoClientSettings clientSettings = MongoClientSettings.builder()
.retryWrites(true)
.applyToConnectionPoolSettings((ConnectionPoolSettings.Builder builder) -> {
builder.maxSize(300) //connections count
.minSize(100)
.maxConnectionLifeTime(300, TimeUnit.MILLISECONDS)
.maintenanceFrequency(5000, TimeUnit.MILLISECONDS)
.maxConnectionIdleTime(200, TimeUnit.MILLISECONDS)
.maxWaitTime(150, TimeUnit.MILLISECONDS);
})
.applyToSocketSettings(builder -> {
builder.connectTimeout(2000, TimeUnit.MILLISECONDS)
.readTimeout(5500, TimeUnit.MILLISECONDS);
})
.applicationName("TestApplication")
.retryWrites(true)
.build();
return clientSettings;
}
#Bean
public MongoCustomConversions mongoCustomConversions() {
return new MongoCustomConversions(Arrays.asList(
new BigDecimalDecimal128Converter(),
new Decimal128BigDecimalConverter()
));
}
#WritingConverter
private static class BigDecimalDecimal128Converter implements Converter<BigDecimal, Decimal128> {
#Override
public Decimal128 convert(#NonNull BigDecimal source) {
return new Decimal128(source);
}
}
#ReadingConverter
private static class Decimal128BigDecimalConverter implements Converter<Decimal128, BigDecimal> {
#Override
public BigDecimal convert(#NonNull Decimal128 source) {
return source.bigDecimalValue();
}
}
And when connecting to multiple mongo db's here's how I do it:
#Configuration
public class UserProfileInformationMongoConfig {
#Autowired
private Environment env;
#Bean
public MongoDatabaseFactory userProfileInformationFactory() {
return new SimpleMongoClientDatabaseFactory(new ConnectionString(env.getProperty("spring.data.mongo.userprofileinformationdb.uri")));
}
#Bean
public MongoTemplate userProfileInformationMongoTemplate() {
return new MongoTemplate(userProfileInformationFactory());
}
}
So, my question again is how do I set my mongo client options like in the first code example when using MongoDatabaseFactory.
Many thanks
Except the maintenanceFrequency configuration, the rest can be provided in the connection string e.g.
mongodb://db1.example.net:27017,db2.example.net:2500/?replicaSet=test&maxPoolSize=300&minPoolSize=100&maxLifeTimeMS=300&maxIdleTimeMS=200&waitQueueTimeoutMS=150&connectTimeoutMS=2000&socketTimeoutMS=5500
Related
I'm configuring mongoDB in my spring boot application like this:
#Configuration
#SpringBootApplication
public class ConfigDbApp extends SpringBootServletInitializer {
#Value("${myapp.mongodb.uri}")
private String mongoDbUri;
[...]
#Bean
public MongoClient mongoClient() {
MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(mongoDbUri.trim()))
.build();
return MongoClients.create(mongoClientSettings);
}
[...]
}
The database uri is:
myapp.mongodb.uri=mongodb://localhost:27017/myApp?sslInvalidHostNameAllowed=true&streamType=netty&ssl=false
The connection is fine but instead of connecting to "myApp" database it connect to "test" database (the default).
If I use the spring property:
spring.data.mongodb.uri=mongodb://localhost:27017/myApp?sslInvalidHostNameAllowed=true&streamType=netty&ssl=false
it works well, but I don't want to use it because I can have diferent apps in the same server reading the same configuration file.
Any suggestion is appreciated, thanks.
This worked with me : override the database name
#Configuration
public class MongoConfig extends AbstractMongoClientConfiguration {
#Value("${spring.data.mongodb.uri}")
private String uri;
#Value("${spring.data.mongodb.database}")
private String database;
#Override
protected String getDatabaseName() {
return database;
}
#Bean
#Override
public MongoClient mongoClient() {
final ConnectionString connectionString = new ConnectionString(uri);
final MongoClientSettings.Builder mongoClientSettings = MongoClientSettings.builder().applyConnectionString(connectionString)
.applyToConnectionPoolSettings(builder -> builder.applySettings(connectionPoolSettings()));
return MongoClients.create(mongoClientSettings.build());
}
private ConnectionPoolSettings connectionPoolSettings() {
return ConnectionPoolSettings.builder()
.maxSize(50)
.maxWaitTime(20, TimeUnit.SECONDS)
.maxConnectionIdleTime(20, TimeUnit.SECONDS)
.maxConnectionLifeTime(60, TimeUnit.SECONDS).build();
}
}
I'm getting this exception when adding Transactional to Spring mongo code.
Caused by: com.mongodb.MongoClientException: Transactions are not supported by the MongoDB cluster to which this client is connected
This is my code: #Configuration
#EnableMongoRepositories(basePackages = { "com.repository" })
public class MongoConfig extends AbstractMongoClientConfiguration {
#Value("${spring.data.mongodb.uri}")
private String connection;
#Value("${spring.data.mongodb.database}")
private String database;
#Autowired
MongoProperties mongoProperties;
#Bean
MongoTransactionManager transactionManager(MongoDatabaseFactory dbFactory) {
return new MongoTransactionManager(dbFactory);
}
#Override
protected String getDatabaseName() {
return "hendrix";
}
#Override
public MongoClient mongoClient() {
final ConnectionString connectionString = new ConnectionString(connection);
final MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applyConnectionString(connectionString)
.build();
return MongoClients.create(mongoClientSettings);
}
}
How do I add replicas to a barebones Mongo setup I have access to? If the cluster is on AWS what setting would I have to change?
I've got two repositories:
#Repository
public interface SpringLRepository extends MongoRepository<L, String> {
}
#Repository
public interface SpringSRepository extends MongoRepository<S, String> {
}
In properties file i've got:
spring.data.mongodb.authentication-database=admin
spring.data.mongodb.l.host=localhost
spring.data.mongodb.l.port=27017
spring.data.mongodb.l.username=root
spring.data.mongodb.l.password=example
spring.data.mongodb.l.authentication-database=admin
spring.data.mongodb.l.database=l
spring.data.mongodb.s.host=localhost
spring.data.mongodb.s.port=27017
spring.data.mongodb.s.username=root
spring.data.mongodb.s.password=example
spring.data.mongodb.s.authentication-database=admin
spring.data.mongodb.s.database=s
I want to have seperate properties for SpringLRepository and SpringSRepository or only change part of them like spring.data.mongodb.database parameter.
I tried to follow this: Configure Multiple MongoDB repositories with Spring Data Mongo
First if i don't exclude MongoDataAutoConfiguration.class:
#SpringBootApplication(
exclude = {
//MongoDataAutoConfiguration.class
})
I've got:
Parameter 1 of method gridFsTemplate in org.springframework.boot.autoconfigure.data.mongo.MongoDbFactoryDependentConfiguration required a single bean, but 2 were found:
- mongoLTemplate: defined by method 'mongoLTemplate' in class path resource [...l/core/application/LMongoConfig.class]
- mongoSTemplate: defined by method 'mongoSTemplate' in class path resource [...s/core/application/SMongoConfig.class]
What can I do about it?
Now I don't use #Repository anymore and I've added two classes.
First config:
#Configuration
#EnableMongoRepositories(
basePackages = "...s.infrastructure.secondary.persistence",
mongoTemplateRef = "mongoSTemplate"
)
public class SMongoConfig {
#Value("${spring.data.mongodb.lead.host}")
private String mongoHost;
#Value("${spring.data.mongodb.lead.port}")
private Integer mongoPort;
#Bean
#Qualifier("mongoSTemplate")
MongoTemplate mongoSTemplate(#Qualifier("mongoSDbFactory") MongoDbFactory mongoDbFactory, MongoConverter converter) {
MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory, converter);
mongoTemplate.setWriteConcern(WriteConcern.MAJORITY);
mongoTemplate.setWriteResultChecking(WriteResultChecking.EXCEPTION);
return mongoTemplate;
}
#Bean
#Qualifier("mongoSimulationsDbFactory")
public MongoDbFactory mongoSimulationsDbFactory() {
MongoClientOptions.Builder mongoOperations = MongoClientOptions.builder();
mongoOperations.socketTimeout(1000 * 2);
mongoOperations.connectTimeout(1000 * 2);
MongoCredential mongoCredential = MongoCredential.createCredential("root", "admin", "example".toCharArray());
return new SimpleMongoDbFactory(
new MongoClient(
new ServerAddress(mongoHost, mongoPort),
Collections.singletonList(mongoCredential)
),
"s");
}
}
Second config:
#Configuration
#EnableMongoRepositories(
basePackages = "...l.infrastructure.secondary.persistence",
mongoTemplateRef = "mongoLTemplate"
)
public class LMongoConfig{
#Value("${spring.data.mongodb.l.host}")
private String mongoHost;
#Value("${spring.data.mongodb.l.port}")
private Integer mongoPort;
#Value("${spring.data.mongodb.l.database}")
private String mongoDB;
#Value("${spring.data.mongodb.l.password}")
private char[] mongoPassword;
#Bean
#Qualifier("mongoLTemplate")
MongoTemplate mongoLTemplate(#Qualifier("mongoLDbFactory") MongoDbFactory mongoDbFactory, MongoConverter converter) {
MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory, converter);
mongoTemplate.setWriteConcern(WriteConcern.MAJORITY);
mongoTemplate.setWriteResultChecking(WriteResultChecking.EXCEPTION);
return mongoTemplate;
}
#Bean
#Qualifier("mongoLDbFactory")
public MongoDbFactory mongoLDbFactory() {
MongoClientOptions.Builder mongoOperations = MongoClientOptions.builder();
mongoOperations.socketTimeout(1000 * 2);
mongoOperations.connectTimeout(1000 * 2);
MongoCredential mongoCredential = MongoCredential.createCredential("root", "admin", "example".toCharArray());
return new SimpleMongoDbFactory(
new MongoClient(
new ServerAddress(mongoHost, mongoPort),
Collections.singletonList(mongoCredential)
),
"s");
}
}
Now this code save s and l to seperate repositories but
I have a few concerns.
SimpleMongoDbFactory and MongoClient are deprecated and it's hard to find newer example of Mongo doing that.
#Bean
public SimpleMongoClientDbFactory metadataFactory(final MongoProperties mongo) throws Exception {
return new SimpleMongoClientDbFactory(mongo.getUri());
}
#Bean
public MongoConfigurationProperties mongoProperties() {
return new MongoConfigurationProperties();
}
I use this. AFAIK it's not deprecated. In this case mongoProperties is a #ConfigurationProperties bean which points to the uri - need multiple URIs since I connect to multiple databases in this app.
I'm unable to make works queue listener with Spring Boot and SQS
(the message is sent and appear in SQS ui)
The #MessageMapping or #SqsListener not works
Java: 11
Spring Boot: 2.1.7
Dependencie: spring-cloud-aws-messaging
This is my config
#Configuration
#EnableSqs
public class SqsConfig {
#Value("#{'${env.name:DEV}'}")
private String envName;
#Value("${cloud.aws.region.static}")
private String region;
#Value("${cloud.aws.credentials.access-key}")
private String awsAccessKey;
#Value("${cloud.aws.credentials.secret-key}")
private String awsSecretKey;
#Bean
public Headers headers() {
return new Headers();
}
#Bean
public MessageQueue queueMessagingSqs(Headers headers,
QueueMessagingTemplate queueMessagingTemplate) {
Sqs queue = new Sqs();
queue.setQueueMessagingTemplate(queueMessagingTemplate);
queue.setHeaders(headers);
return queue;
}
private ResourceIdResolver getResourceIdResolver() {
return queueName -> envName + "-" + queueName;
}
#Bean
public DestinationResolver destinationResolver(AmazonSQSAsync amazonSQSAsync) {
DynamicQueueUrlDestinationResolver destinationResolver = new DynamicQueueUrlDestinationResolver(
amazonSQSAsync,
getResourceIdResolver());
destinationResolver.setAutoCreate(true);
return destinationResolver;
}
#Bean
public QueueMessagingTemplate queueMessagingTemplate(AmazonSQSAsync amazonSQSAsync,
DestinationResolver destinationResolver) {
return new QueueMessagingTemplate(amazonSQSAsync, destinationResolver, null);
}
#Bean
public QueueMessageHandlerFactory queueMessageHandlerFactory() {
QueueMessageHandlerFactory factory = new QueueMessageHandlerFactory();
MappingJackson2MessageConverter messageConverter = new MappingJackson2MessageConverter();
messageConverter.setStrictContentTypeMatch(false);
factory.setArgumentResolvers(Collections.singletonList(new PayloadArgumentResolver(messageConverter)));
return factory;
}
#Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(AmazonSQSAsync amazonSqs) {
SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
factory.setAmazonSqs(amazonSqs);
factory.setMaxNumberOfMessages(10);
factory.setWaitTimeOut(2);
return factory;
}
}
I notice also that org.springframework.cloud.aws.messaging.config.SimpleMessageListenerContainerFactory and org.springframework.cloud.aws.messaging.config.annotation.SqsConfiguration run on startup
And my test
#RunWith(SpringJUnit4ClassRunner.class)
public class ListenTest {
#Autowired
private MessageQueue queue;
private final String queueName = "test-queue-receive";
private String result = null;
#Test
public void test_listen() {
// given
String data = "abc";
// when
queue.send(queueName, data).join();
// then
Awaitility.await()
.atMost(10, TimeUnit.SECONDS)
.until(() -> Objects.nonNull(result));
Assertions.assertThat(result).equals(data);
}
#MessageMapping(value = queueName)
public void receive(String data) {
this.result = data;
}
}
Do you think something is wrong ?
I create a repo for exemple : (https://github.com/mmaryo/java-sqs-test)
In test folder, change aws credentials in 'application.yml'
Then run tests
I had the same issue when using the spring-cloud-aws-messaging package, but then I used the queue URL in the #SqsListener annotation instead of the queue name and it worked.
#SqsListener(value = { "https://full-queue-URL" }, deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void receive(String message) {
// do something
}
It seems you can use the queue name when using the spring-cloud-starter-aws-messaging package. I believe there is some configuration that allows usage of the queue name instead of URL if you don't want to use the starter package.
EDIT: I noticed the region was being defaulted to us-west-2 despite me listing us-east-1 in my properties file. Then I created a RegionProvider bean and set the region to us-east-1 in there and now when I use the queue name in the #SqsMessaging it is found and correctly resolved to the URL in the framework code.
you'll need to leverage the #Primary annotation, this is what worked for me:
#Autowired(required = false)
private AWSCredentialsProvider awsCredentialsProvider;
#Autowired
private AppConfig appConfig;
#Bean
public QueueMessagingTemplate getQueueMessagingTemplate() {
return new QueueMessagingTemplate(sqsClient());
}
#Primary
#Bean
public AmazonSQSAsync sqsClient() {
AmazonSQSAsyncClientBuilder builder = AmazonSQSAsyncClientBuilder.standard();
if (this.awsCredentialsProvider != null) {
builder.withCredentials(this.awsCredentialsProvider);
}
if (appConfig.getSqsRegion() != null) {
builder.withRegion(appConfig.getSqsRegion());
} else {
builder.withRegion(Regions.DEFAULT_REGION);
}
return builder.build();
}
build.gradle needs these deps:
implementation("org.springframework.cloud:spring-cloud-starter-aws:2.2.0.RELEASE")
implementation("org.springframework.cloud:spring-cloud-aws-messaging:2.2.0.RELEASE")
How can I configure the inbound channel adapter via annotations instead of the regular configuration file? I was able to define the bean for the session factory though as under:
#Bean
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory ftpSessionFactory = new
DefaultFtpSessionFactory();
ftpSessionFactory.setHost(host);
ftpSessionFactory.setPort(port);
ftpSessionFactory.setUsername(username);
ftpSessionFactory.setPassword(password);
return ftpSessionFactory;
}
How can I configure the inbound channel adapter given as under via annotations?
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpSessionFactory"
filename-pattern="*.xml"
auto-create-local-directory="true"
delete-remote-files="false"
remote-directory="/"
local-directory="ftp-inbound"
local-filter="acceptOnceFilter">
<int:poller fixed-delay="60000" max-messages-per-poll="-1">
<int:transactional synchronization-factory="syncFactory" />
</int:poller>
</int-ftp:inbound-channel-adapter>
#Artem Bilan
The modified code is as under
#EnableIntegration
#Configuration
public class FtpConfiguration {
#Value("${ftp.host}")
private String host;
#Value("${ftp.port}")
private Integer port;
#Value("${ftp.username}")
private String username;
#Value("${ftp.password}")
private String password;
#Value("${ftp.fixed.delay}")
private Integer fixedDelay;
#Value("${ftp.local.directory}")
private String localDirectory;
private final static Logger LOGGER = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
#Bean
public SessionFactory<FTPFile> ftpSessionFactory() {
DefaultFtpSessionFactory sessionFactory = new DefaultFtpSessionFactory();
sessionFactory.setHost(host);
sessionFactory.setPort(port);
sessionFactory.setUsername(username);
sessionFactory.setPassword(password);
return new CachingSessionFactory<FTPFile>(sessionFactory);
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(ftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("/");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(value = "ftpChannel",
poller = #Poller(fixedDelay = "60000", maxMessagesPerPoll = "-1"))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source =
new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File(localDirectory));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
}
While running this,I get an exception as under
No bean named 'ftpChannel' is defined
Please note that 'channel' keyword in not available while wiring the Inbound channel adapter,its 'value' instead.
I tried wiring the channel with PollableChannel,that also went in vain though. It is as under:
#Bean
public MessageChannel ftpChannel() {
return new PollableChannel() {
#Override
public Message<?> receive() {
return this.receive();
}
#Override
public Message<?> receive(long l) {
return null;
}
#Override
public boolean send(Message<?> message) {
return false;
}
#Override
public boolean send(Message<?> message, long l) {
return false;
}
};
}
I got an error "failed to send message within timeout: -1".Am I doing something wrong still?
What I'm looking for is to wire up all the beans on application start-up, and then expose some method to start polling the server,process them and then delete them from local,something like this
public void startPollingTheServer() {
getPollableChannel().receive();
}
where getPollableChannel() gives me the bean I had wired for Polling.
There is an #InboundChannelAdapter for you.
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(ftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("/");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel")
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source =
new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
Plus take a look into the Reference Manual.
Also pay attention, please, for Java DSL for Spring Integration, where the same might look like:
#Bean
public IntegrationFlow ftpInboundFlow() {
return IntegrationFlows
.from(s -> s.ftp(this.ftpSessionFactory)
.preserveTimestamp(true)
.remoteDirectory("ftpSource")
.regexFilter(".*\\.txt$")
.localFilename(f -> f.toUpperCase() + ".a")
.localDirectory(this.ftpServer.getTargetLocalDirectory()),
e -> e.id("ftpInboundAdapter").autoStartup(false))
.channel(MessageChannels.queue("ftpInboundResultChannel"))
.get();
}