Develop Reference

Question regarding FtpInboundFileSynchronizer running with multiple instances/applications

I've recently been trying to configure and set up a spring boot application that will later be run in kubernetes and have multiple pods running of it. The application is meant to download files from a FTP server. I've found some existing code for doing this in Springboot, particularly FtpInboundFileSynchronizer and so I tried set it up and make sure it works. I have a working solution with a ConcurrentMetaDataStore. So my only real question is if it will be fine running it with multiple instances or if I require something additional for it to be run with multiple pods?
My configuration looks something like this:
#Getter
#Setter
#Configuration
#ConfigurationProperties(prefix = "ftp")
public class FtpConfiguration
{
private final static int PASSIVE_LOCAL_DATA_CONNECTION_MODE = 2;
private final static int DEFAULT_FTP_PORT = 21;
String host;
String username;
String password;
String localDirectory;
String remoteDirectory;
FtpRemoteFileTemplate template;
FtpInboundFileSynchronizer synchronizer;
DataSource templateSource;
#Bean
public ConcurrentMetadataStore metadataStore(DataSource dataSource)
{
var jbdcMetaDatastore = new JdbcMetadataStore(dataSource);
jbdcMetaDatastore.setTablePrefix("INT_");
jbdcMetaDatastore.setRegion("TEMPORARY");
jbdcMetaDatastore.afterPropertiesSet();
return jbdcMetaDatastore;
}
#Bean
public DefaultFtpSessionFactory defaultFtpSessionFactory()
{
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(host);
sf.setUsername(username);
sf.setPassword(password);
sf.setPort(DEFAULT_FTP_PORT);
sf.setConnectTimeout(5000);
sf.setClientMode(PASSIVE_LOCAL_DATA_CONNECTION_MODE);
return sf;
}
#Bean
FtpRemoteFileTemplate ftpRemoteFileTemplate(DefaultFtpSessionFactory dsf)
{
return new FtpRemoteFileTemplate(dsf);
}
#Bean
FtpInboundFileSynchronizer ftpInboundFileSynchronizer(DefaultFtpSessionFactory dsf)
{
FtpInboundFileSynchronizer ftpInSync = new FtpInboundFileSynchronizer(dsf);
ftpInSync.setRemoteDirectory(remoteDirectory);
ftpInSync.setFilter(ftpFileListFilter());
return ftpInSync;
}
public FileListFilter<FTPFile> ftpFileListFilter()
{
try (ChainFileListFilter<FTPFile> chain = new ChainFileListFilter<>())
{
chain.addFilter(new FtpPersistentAcceptOnceFileListFilter(metadataStore(templateSource), "TEST"));
return chain;
}
catch (IOException e)
{
throw new RuntimeException("Failed to create FtpPersistentAcceptOnceFileListFilter", e);
}
}
}
and then I just call the the SynchronizeToLocalDirectory method.
FtpClient(
FtpRemoteFileTemplate template, FtpInboundFileSynchronizer synchronizer,
#Value("${ftp.remote-directory}") String remoteDirectory,
#Value("${ftp.local-directory}") String localDirectory)
{
this.template = template;
this.synchronizer = synchronizer;
this.remoteDirectory = remoteDirectory;
this.localDirectory = localDirectory;
}
synchronizer.setRemoteDirectory(remoteDirectory);
synchronizer.synchronizeToLocalDirectory(new File(localDirectory));
Would this solution handle multiple applications without problems? Or what else would I need? Does the ConcurrentMetaData store alone make sure this works? (so for example there wouldn't be a conflict/crash if two instances at the same time try to synchronise same directory as they'd both be fine thanks to the metastore being #Transactional).

Your assumption is correct: as long as all your pods are connecting to the same data base, that JdbcMetadataStore will ensure that no concurrent read for the same file are going to happen.
It is not clear, though, why would one use an FtpInboundFileSynchronizer manually, but not via an FtpInboundFileSynchronizingMessageSource and subsequent integration flow, but that's I guess fully different story and question.
On the other hand: why do you ask this question at all? Didn't you try your solution? Isn't docs enough to be sure where and how to go: https://docs.spring.io/spring-integration/docs/current/reference/html/file.html#remote-persistent-flf ?

Factory Method vs Constructor with ServletContext as Parameter

To test the health of all our applications we are including a Healthcheck servlet in each application. These healthchecks simply test the dependencies of each application. One of these dependency types are Sql server connections. To test the connections we have a method called HealthCheckRunner.Result run(). (Shown in the code below). The method will take a url, username, and password and attempt to connect to the server.
This method works fine but I have found that across all our apps I am still repeating a lot of code to retrieve the url, username, password, and driver from the context.xml. To save time and repetition I would like to refactor with either another constructor or a factory method, shown below in Options 1 and 2.
Neither method seems very appealing to me though. First the constructor is pretty messy and doesn't seem very user friendly. Second, the static method may be difficult to test. And lastly, they both take a ServletContext as a parameter.
Will unit testing the static method be difficult? For simplicity I'd rather stay away from PowerMock and only use Mockito. And also, will copies of ServletContext be created for every SqlHealthCheck I create? Or will they all use the same reference? And, since I'm only using a few values from the context would it be better to create another class and pass only the values I need? The solutions I have come up with are not great and I know there must be a better way.
public class SqlHealthCheck extends HealthCheck {
private String url;
private String username;
private String password;
private String driver;
// Option 1: Constructor with ServletContext as parameter.
public SqlHealthCheck (ServletContext context, String prefix) {
this.url = context.getInitParameter(prefix + ".db-url");
this.username = context.getInitParameter(prefix + ".db-user");
this.password = context.getInitParameter(prefix + ".db-passwd");
setType("sqlcheck");
setDescription("SQL database check: " + this.url);
this.decodePassword();
this.setDriver(context.getInitParameter(prefix + ".db-driver"));
}
// Option 2: Static factory method with ServletContext as parameter
public static HealthCheck createHealthCheck(ServletContext context, String prefix) {
String dbUrl = context.getInitParameter(prefix + ".db-url");
String username = context.getInitParameter(prefix + ".db-user");
String password = context.getInitParameter(prefix + ".db-passwd");
String sqlDriver = context.getInitParameter(prefix + ".db-driver");
SqlHealthCheck healthCheck = new SqlHealthCheck("SQL database check: " + dbUrl, dbUrl, username, password);
healthCheck.decodePassword();
healthCheck.setDriver(sqlDriver);
return healthCheck;
}
public HealthCheckRunner.Result run() {
Connection connection = null;
Statement statement = null;
try {
if (driver != null) { Class.forName(driver); }
connection = DriverManager.getConnection(this.url, this.username, this.password);
statement = connection.createStatement();
statement.executeQuery("SELECT 1");
return HealthCheckRunner.Result.Pass;
} catch (SQLException | ClassNotFoundException ex) {
setMessage(ex.getMessage());
return getFailureResult();
}
finally {
try {
if (statement != null) {statement.close();}
if (connection != null) {connection.close();}
} catch (SQLException ex) {
setMessage(ex.getMessage());
}
}
}
public void decodePassword() {
// Decode password
try {
if (password != null && !"".equals(password)) {
password = new String(Base64.decode(password.getBytes()));
}
} catch (Exception e) {
if (e.getMessage()!=null) {
this.setMessage(e.getMessage());}
}
}
}

I have found that across all our apps I am still repeating a lot of code to retrieve the url, username, password, and driver from the context.xml
4 lines of code is far, far, far from being a lot of code. But if you're actually copying and pasting this class in all your apps, then you simply shouldn't. Create a separate project containing reusable health checks like this one, producing a jar, and use this jar in each app that needs the health checks.
the constructor is pretty messy and doesn't seem very user friendly
Frankly, it's not that messy. But it could be less messy if you didn't repeat yourself, initialized private fields all the same way, and if you grouped comparable code together:
public SqlHealthCheck (ServletContext context, String prefix) {
this.url = getParam(context, prefix, "db-url");
this.username = getParam(context, prefix, "db-user");
this.password = getParam(context, prefix, "db-password");
this.driver = getParam(context, prefix, "db-driver");
this.decodePassword();
setType("sqlcheck");
setDescription("SQL database check: " + this.url);
}
Will unit testing the static method be difficult?
No. ServletContext is an interface. So you can create your own fake implementation or use a mocking framework to mock it. Then you can just call the constructor of the factory method, run the health check, and see if it returns the correct value.
will copies of ServletContext be created for every SqlHealthCheck I create?
Of course not. Java passes references to objects by value.
would it be better to create another class and pass only the values I need?
You could do that, but then the logic of getting the values from the servlet context will just be elsewhere, and you'll have to test that too, basically in the same way as you would test this class.

Mongo Connection is created multiple times in RESTful API and never released

I have written a RESTful API using Apache Jersey. I am using MongoDB as my backend. I used Morphia (v.1.3.4) to map and persist POJO to database. I tried to follow "1 application 1 connection" in my API as recommended everywhere but I am not sure I am successful. I run my API in Tomcat 8. I also ran Mongostat to see the details and connection. At start, Mongostat showed 1 connection to MongoDB server. I tested my API using Postman and it was working fine. I then created a load test in SoapUI where I simulated 100 users per second. I saw the update in Mongostat. I saw there were 103 connections. Here is the gif which shows this behaviour.
I am not sure why there are so many connections. The interesting fact is that number of mongo connection are directly proportional to number of users I create on SoapUI. Why is that? I found other similar questions but I think I have implemented there suggestions.
Mongo connection leak with morphia
Spring data mongodb not closing mongodb connections
My code looks like this.
DatabaseConnection.java
// Some imports
public class DatabaseConnection {
private static volatile MongoClient instance;
private static String cloudhost="localhost";
private DatabaseConnection() { }
public synchronized static MongoClient getMongoClient() {
if (instance == null ) {
synchronized (DatabaseConnection.class) {
if (instance == null) {
ServerAddress addr = new ServerAddress(cloudhost, 27017);
List<MongoCredential> credentialsList = new ArrayList<MongoCredential>();
MongoCredential credentia = MongoCredential.createCredential(
"test", "test", "test".toCharArray());
credentialsList.add(credentia);
instance = new MongoClient(addr, credentialsList);
}
}
}
return instance;
}
}
PourService.java
#Secured
#Path("pours")
public class PourService {
final static Logger logger = Logger.getLogger(Pour.class);
private static final int POUR_SIZE = 30;
#POST
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response createPour(String request)
{
WebApiResponse response = new WebApiResponse();
Gson gson = new GsonBuilder().setDateFormat("dd/MM/yyyy HH:mm:ss").create();
String message = "Pour was not created.";
HashMap<String, Object> data = null;
try
{
Pour pour = gson.fromJson(request, Pour.class);
// Storing the pour to
PourRepository pourRepository = new PourRepository();
String id = pourRepository.createPour(pour);
data = new HashMap<String, Object>();
if ("" != id && null != id)
{
data.put("id", id);
message = "Pour was created successfully.";
logger.debug(message);
return response.build(true, message, data, 200);
}
logger.debug(message);
return response.build(false, message, data, 500);
}
catch (Exception e)
{
message = "Error while creating Pour.";
logger.error(message, e);
return response.build(false, message, new Object(),500);
}
}
PourDao.java
public class PourDao extends BasicDAO<Pour, String>{
public PourDao(Class<Pour> entityClass, Datastore ds) {
super(entityClass, ds);
}
}
PourRepository.java
public class PourRepository {
private PourDao pourDao;
final static Logger logger = Logger.getLogger(PourRepository.class);
public PourRepository ()
{
try
{
MongoClient mongoClient = DatabaseConnection.getMongoClient();
Datastore ds = new Morphia().map(Pour.class)
.createDatastore(mongoClient, "tilt45");
pourDao = new PourDao(Pour.class,ds);
}
catch (Exception e)
{
logger.error("Error while creating PourDao", e);
}
}
public String createPour (Pour pour)
{
try
{
return pourDao.save(pour).getId().toString();
}
catch (Exception e)
{
logger.error("Error while creating Pour.", e);
return null;
}
}
}

When I work with Mongo+Morphia I get better results using a Factory pattern for the Datastore and not for the MongoClient, for instance, check the following class:
public DatastoreFactory(String dbHost, int dbPort, String dbName) {
final Morphia morphia = new Morphia();
MongoClientOptions.Builder options = MongoClientOptions.builder().socketKeepAlive(true);
morphia.getMapper().getOptions().setStoreEmpties(true);
final Datastore store = morphia.createDatastore(new MongoClient(new ServerAddress(dbHost, dbPort), options.build()), dbName);
store.ensureIndexes();
this.datastore = store;
}
With that approach, everytime you need a datastore you can use the one provided by the factory. Of course, this can implemented better if you use a framework/library that support factory pattern (e.g.: HK2 with org.glassfish.hk2.api.Factory), and also singleton binding.
Besides, you can check the documentation of MongoClientOptions's builder method, perhaps you can find a better connection control there.

Override default Spring-Boot application.properties settings in Junit Test with dynamic value

I want to override properties defined in application.properties in tests, but #TestPropertySource only allows to provide predefined values.
What I need is to start a server on a random port N, then pass this port to spring-boot application. The port has to be ephemeral to allow running multiple tests on the same host at the same time.
I don't mean the embedded http server (jetty), but some different server that is started at the beginning of the test (e.g. zookeeper) and the application being tested has to connect to it.
What's the best way to achieve this?
(here's a similar question, but answers do not mention a solution for ephemeral ports - Override default Spring-Boot application.properties settings in Junit Test)

As of Spring Framework 5.2.5 and Spring Boot 2.2.6 you can use Dynamic Properties in tests:
#DynamicPropertySource
static void dynamicProperties(DynamicPropertyRegistry registry) {
registry.add("property.name", "value");
}

Thanks to the changes made in Spring Framework 5.2.5, the use of #ContextConfiguration and the ApplicationContextInitializer can be replaced with a static #DynamicPropertySource method that serves the same purpose.
#SpringBootTest
#Testcontainers
class SomeSprintTest {
#Container
static LocalStackContainer localStack =
new LocalStackContainer().withServices(LocalStackContainer.Service.S3);
#DynamicPropertySource
static void initialize(DynamicPropertyRegistry registry) {
AwsClientBuilder.EndpointConfiguration endpointConfiguration =
localStack.getEndpointConfiguration(LocalStackContainer.Service.S3);
registry.add("cloud.aws.s3.default-endpoint", endpointConfiguration::getServiceEndpoint);
}
}

You could override the value of the port property in the #BeforeClass like this:
#BeforeClass
public static void beforeClass() {
System.setProperty("zookeeper.port", getRandomPort());
}

The "clean" solution is to use an ApplicationContextInitializer.
See this answer to a similar question.
See also this github issue asking a similar question.
To summarize the above mentioned posts using a real-world example that's been sanitized to protect copyright holders (I have a REST endpoint which uses an #Autowired DataSource which needs to use the dynamic properties to know which port the in-memory MySQL database is using):
Your test must declare the initializer (see the #ContextConfiguration line below).
// standard spring-boot test stuff
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#ActiveProfiles("local")
#ContextConfiguration(
classes = Application.class,
// declare the initializer to use
initializers = SpringTestDatabaseInitializer.class)
// use random management port as well so we don't conflict with other running tests
#TestPropertySource(properties = {"management.port=0"})
public class SomeSprintTest {
#LocalServerPort
private int randomLocalPort;
#Value("${local.management.port}")
private int randomManagementPort;
#Test
public void testThatDoesSomethingUseful() {
// now ping your service that talks to the dynamic resource
}
}
Your initializer needs to add the dynamic properties to your environment. Don't forget to add a shutdown hook for any cleanup that needs to run. Following is an example that sets up an in-memory database using a custom DatabaseObject class.
public class SpringTestDatabaseInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
private static final int INITIAL_PORT = 0; // bind to an ephemeral port
private static final String DB_USERNAME = "username";
private static final String DB_PASSWORD = "password-to-use";
private static final String DB_SCHEMA_NAME = "default-schema";
#Override
public void initialize(ConfigurableApplicationContext applicationContext) {
DatabaseObject databaseObject = new InMemoryDatabaseObject(INITIAL_PORT, DB_USERNAME, DB_PASSWORD, DB_SCHEMA_NAME);
registerShutdownHook(databaseObject);
int databasePort = startDatabase(databaseObject);
addDatabasePropertiesToEnvironment(applicationContext, databasePort);
}
private static void addDatabasePropertiesToEnvironment(ConfigurableApplicationContext applicationContext, int databasePort) {
String url = String.format("jdbc:mysql://localhost:%s/%s", databasePort, DB_SCHEMA_NAME);
System.out.println("Adding db props to environment for url: " + url);
TestPropertySourceUtils.addInlinedPropertiesToEnvironment(
applicationContext,
"db.port=" + databasePort,
"db.schema=" + DB_SCHEMA_NAME,
"db.url=" + url,
"db.username=" + DB_USERNAME,
"db.password=" + DB_PASSWORD);
}
private static int startDatabase(DatabaseObject database) {
try {
database.start();
return database.getBoundPort();
} catch (Exception e) {
throw new IllegalStateException("Failed to start database", e);
}
}
private static void registerShutdownHook(DatabaseObject databaseObject) {
Runnable shutdownTask = () -> {
try {
int boundPort = databaseObject.getBoundPort();
System.out.println("Shutting down database at port: " + boundPort);
databaseObject.stop();
} catch (Exception e) {
// nothing to do here
}
};
Thread shutdownThread = new Thread(shutdownTask, "Database Shutdown Thread");
Runtime.getRuntime().addShutdownHook(shutdownThread);
}
}
When I look at the logs, it shows that for both of my tests that use this initializer class, they use the same object (the initialize method only gets called once, as does the shutdown hook). So it starts up a database, and leaves it running until both tests finish, then shuts the database down.

How do I change my java code to use replset? with Spring Data and MongoDB

I would like to know how to change my java code to support replset with spring-data and MongoDB.
I have 3 MongoDB servers running.. example:
./mongod --dbpath=/home/jsmith/tmp/db1 --replSet=spring --port=27017
./mongod --dbpath=/home/jsmith/tmp/db2 --replSet=spring --port=27027
./mongod --dbpath=/home/jsmith/tmp/db3 --replSet=spring --port=27037
if I do rs.status() I can see that if the db on 27017 goes down then one of the others become primary so I know that mongoDB is working right but in my java code if I try to run it I get the following error:
Exception in thread "main" org.springframework.dao.DataAccessResourceFailureException: can't call something : /127.0.0.1:27017/demo
Its looking only on port 27017
here is my mongodbconfig:
#Configuration
#EnableMongoRepositories
#ComponentScan(basePackageClasses = {MongoDBApp.class})
#PropertySource("classpath:application.properties")
public class MongoConfiguration extends AbstractMongoConfiguration {
#Override
protected String getDatabaseName() {
return "demo";
}
#Override
public Mongo mongo() throws Exception {
return new Mongo(new ArrayList<ServerAddress>() {{ add(new ServerAddress("127.0.0.1", 27017)); add(new ServerAddress("127.0.0.1", 27027)); add(new ServerAddress("127.0.0.1", 27037)); }});
}
#Override
protected String getMappingBasePackage() {
return "com.xxxx.mongodb.example.domain";
}
}
how do I change it to support replset? but if its reading and one of the servers goes down I get a error.. anyway to make in reconnect?

The URI method should work, or there's a clearer way to initialise the replica set using a list of servers:
final List<ServerAddress> seeds = Arrays.asList(new ServerAddress("127.0.0.1", 27017),
new ServerAddress("127.0.0.1", 27027),
new ServerAddress("127.0.0.1", 27037));
final Mongo mongo = new Mongo(seeds);

This is how I do it:
String mongoURI="mongodb://myUsrName:pass#mongoServer-001.company.com:27017,mongoServer-002.company.com:27017,mongoServer-003.company.com:27017/myDBname?waitqueuemultiple=1500&w=1&maxpoolsize=40&safe=true";
MongoURI uri = new MongoURI(mongoURI);
Mongo mongo = new Mongo(uri);
I specify the 3 servers in the URI (along with extra parameters like max pool size).
The third server (mongoServer-003) is the arbiter and it doesn't store any info. The arbiter helps in the election of the primary server when the current primary goes down. Take a look at this article.
With this configuration, the app can keep working even if the primary server goes down.

You can also do it the CustomEditorConfigurer and a class that implements PropertyEditorRegistrar.
So you your configuration class needs this:
#Bean
public static CustomEditorConfigurer customEditorConfigurer(){
CustomEditorConfigurer configurer = new CustomEditorConfigurer();
configurer.setPropertyEditorRegistrars(
new PropertyEditorRegistrar[]{new ServerAddressPropertyEditorRegistrar()});
return configurer;
}
#Override
protected String getDatabaseName() {
return authenticationDb;
}
#Override
#Bean
public MongoClient mongoClient() {
MongoClient mongoClient = new MongoClient(Arrays.asList(new ServerAddress(host, port)), mongoCredentials(), mongoClientOptions());
return mongoClient;
}
public final class ServerAddressPropertyEditorRegistrar implements PropertyEditorRegistrar {
#Override
public void registerCustomEditors(PropertyEditorRegistry registry) {
registry.registerCustomEditor(ServerAddress[].class, new ServerAddressPropertyEditor());
}
}