Spring Integration RecursiveDirectoryScanner gives too many files open exception - java

I am using Spring Integration RecursiveDirectoryScanner to scan a directory recursively to process the incoming file that will be placed under the configured directory (/home/test).
I am frequently getting the below error:
ERROR org.springframework.integration.handler.LoggingHandler - java.lang.IllegalArgumentException: java.nio.file.FileSystemException: /home/test: Too many open files
at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:89)
at org.springframework.integration.file.FileReadingMessageSource.scanInputDirectory(FileReadingMessageSource.java:387)
at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:361)
at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:90)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:134)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:224)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /home/test: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
at java.nio.file.Files.walk(Files.java:3574)
at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:73)
My Spring Integration flow is as below:
Configuration in XML
<task:executor id="pollerPool"
pool-size="${pollerThreadPoolSize}"
queue-capacity="${pollerThreadQueueCapacity}" rejection-policy="ABORT" />
<task:executor id="fileHandlerPool"
pool-size="${fileHandlerPoolSize}"
queue-capacity="${fileHandlerPoolThreadQueueCapacity}" rejection-policy="CALLER_RUNS" />
<bean id="iFilter" class="org.springframework.integration.file.filters.ChainFileListFilter">
<constructor-arg>
<list>
<bean id="lastModifiedFileListFilter" class="org.springframework.integration.file.filters.LastModifiedFileListFilter">
<property name="age" value="120" />
</bean>
<ref bean="acceptOnceFileListFilter"/>
<bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
<constructor-arg value="^.*\.(txt|csv|xls|xlsx|asc)$"/>
</bean>
</list>
</constructor-arg>
</bean>
<bean id="acceptOnceFileListFilter" name="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" primary="true" />
<bean id="recursiveDirectoryScanner" class="org.springframework.integration.file.RecursiveDirectoryScanner">
<property name="filter" ref="iFilter" />
<property name="locker" ref="nioFileLocker" />
</bean>
<bean id="nioFileLocker" class="org.springframework.integration.file.locking.NioFileLocker" />
<int-file:inbound-channel-adapter
id="fileSource" channel="fileReceivedChannel" auto-startup="true"
directory="file:${polling.directory}"
scanner="recursiveDirectoryScanner" >
<int:poller task-executor="pollerPool"
fixed-rate="${pollerFixedRate}"
receive-timeout="${pollerReceiveTimeout}">
</int:poller>
</int-file:inbound-channel-adapter>
Dynamic parameters are as below:
polling.directory=/home/test
pollerThreadPoolSize=1
pollerThreadQueueCapacity=10
pollerFixedRate=5000
pollerReceiveTimeout=5000
fileHandlerPoolSize=2
fileHandlerPoolThreadQueueCapacity=100
EDIT:
I do unlock file in a service activator that comes in to picture when a file is picked. I get some information from file and unlock it.
#Autowired
NioFileLocker nioFileLocker;
protected void doTransform(Message<?> message) throws Exception {
MessageBuilder<File> payload = (MessageBuilder<File>) message.getPayload();
File inFile = payload.getPayload();
try {
nioFileLocker.unlock(inFile);
} catch (Exception e) {
LOGGER.error("file not unlock");
}
}
Is there any issue with the configuration ? How do I make sure this exception never appear again ?
Thank you in advance.

I would suggest to test your solution without NioFileLocker. Doesn't look like you are using it for unlocking files, but the lock(File fileToLock) really keeps some file marker in OS.
On the other hand the file locker doesn't work reliable on UNIX systems. It still allows an access to files. At least for reading.
For better exclusive file access I would recommend to use a FileSystemPersistentAcceptOnceFileListFilter with the external MetadataStore instead of in memory AcceptOnceFileListFilter. This way only one instance of your application will get access to the file and it won't be processed again at all.

Related

Spring Data Context - lookup placeholder variables from Java map, not properties file

Currently in my data context XML file, it's looking up values to substitute from a application.properties file:
<bean id="applicationProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" lazy-init="default">
<property name="location" value="classpath:application.properties" />
</bean>
<bean id="appleDataSource" class="org.apache.commons.dbcp2.BasicDataSource">
<property name="url" value="${apple.url}" />
</bean>
I'd like to change this from being looked up from the application.properties file, to being read out of a Properties/Map object.
This is because the configuration could come from more than one place (i.e. not just the classpath), and this logic is already implemented in my application.
I've seen MapPropertySource but I think while that can be done when the bean is configured in Java, I'm not sure this logic can be implemented when working with the XML file alone?

How to get the file from service-activator Message object in listener class

I need to pass the file to service layer which i am receiving in SFTP path.
below is configuration and i am seeing the message receiving in my service-activator like
GenericMessage [payload=com.jcraft.jsch.ChannelSftp$2#322906a2,
headers={closableResource=org.springframework.integration.sftp.session.SftpSession#379662a7,
id=704c58e7-1d93-3bef-0330-233c0f9af55c, file_remoteDirectory=/tmp/charge/,
file_remoteFile=Charge.txt, timestamp=1594158522576}]
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="hostname"/>
<property name="port" value="22"/>
<property name="user" value="vkp"/>
<property name="password" value="1234"/>
<property name="allowUnknownKeys" value="true"/>
</bean>
<int-sftp:inbound-streaming-channel-adapter id="sftpAdapterAutoCreate"
session-factory="sftpSessionFactory"
filename-pattern="*.txt"
channel="receiveChannel"
remote-directory="/tmp/charge/">
</int-sftp:inbound-streaming-channel-adapter>
<int:poller fixed-rate="25000" max-messages-per-poll="1" id="shippingChargePoller" default="true"/>
<int:channel id="receiveChannel">
<int:queue/>
</int:channel>
<int:stream-transformer id="withCharset" charset="UTF-8" input-
channel="receiveChannel" />
<int:service-activator id="FeedListener" input-channel="receiveChannel" method="onMessage">
<bean class="com.listener.ChargeFeedListener"/>
</int:service-activator>
public void onMessage(Message<?> message){
System.out.println(message.toString());
System.out.println( " Received File is "+message.getPayload());
}
But i am not receiving the file in my java class . What i need to do to get the file ?
Please, read documentation: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-streaming. The <int-sftp:inbound-streaming-channel-adapter> is not about files. It does open an InputStream for a remote entry (probably file on SFTP) and let you to do with this stream whatever you want. For example (also according that docs), there is a StreamTransformer which let's you to read that stream into a byte[] or string if you provide a charset. If you really want to deal with files, then you need to consider to switch to the <int-sftp:inbound-channel-adapter>. That one pull the remote entry and store its content into a local file. Then that java.io.File is sent to the channel for your consideration.
I think we had a chat with you on the matter in other your question: Spring SFTP Integration is not polling the file.
Please, let us know what is wrong with our docs that confuses you so you have to raise questions like this over here.

SFTP inbound channel adapter hangs at PollableChannel receive method

My context xml reads as follows:
<bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${host}"/>
<property name="password" value="${password}"/>
<property name="port" value="${port}"/>
<property name="user" value="${username}"/>
</bean>
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
session-factory="sftpSessionFactory"
channel="receiveChannel"
filename-pattern="*.txt"
remote-directory="/loblawln"
local-directory="/local-dir"
auto-create-local-directory="true"
temporary-file-suffix=".writing"
delete-remote-files="false">
<int:poller fixed-rate="1000" max-messages-per-poll="1"/>
</int-sftp:inbound-channel-adapter>
<int:channel id="receiveChannel">
<int:queue/>
</int:channel>
The java file reads like this:
PollableChannel localFileChannel = context.getBean("receiveChannel", PollableChannel.class);
SourcePollingChannelAdapter adapter = context.getBean(SourcePollingChannelAdapter.class);
adapter.start();
System.out.println("Adapter started...");
Message<?> received = localFileChannel.receive();
System.out.println("Received first file message: " + received);
I have referred to the implementation provided by Gary Russell at Github and also followed spring docs for sftp integration. I think i am missing something important.
Moreover logs are also not showing anything at all.
The problem got solved by changing the key to read username.
This seemed a bit weird to me when I debugged the code and read values in defaultSessionFactory instance, I found it was not reading the username from the properties file, rather it read the value correspnding to System.getProperty("user.name"). When I changed the key from {username} to some thing else like {sftp.username} it read it correctly.

Spring - how to use PropertyPlaceholderConfigurer to dynamically load files

i have the next properties files with Spring Framework
config.properties
with content
environment=devel //posible values: devel, testing, prod
and with the previous environment property, choose some of the following files to load dynamically
config-service1-devel.properties
config-service1-testing.properties
config-service1-prod.properties
config-serviceN-devel.properties
config-serviceN-testing.properties
config-serviceN-prod.properties
and then, with spring i want load the properties, i'm solve to load the first properties file but i dont understand how to use expression language to complete the values of the dependent properties.
<bean id="MainApplicationProperties"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location"
value="file://#{systemProperties['jboss.server.home.dir']}/conf/services.properties" />
<property name="placeholderPrefix" value="$mainProperty{" />
<property name="placeholderSuffix" value="}" />
</bean>
<bean id="SecondApplicationProperties"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
depends-on="MainApplicationProperties">
<property name="locations">
<list>
<value>file://#{systemProperties['jboss.server.home.dir']}/conf/serviceOne/service1-$mainProperty{environment}.properties</value>
<value>file://#{systemProperties['jboss.server.home.dir']}/conf/serviceTwo/service2-$mainProperty{environment}.properties</value>
<value>file://#{systemProperties['jboss.server.home.dir']}/conf/serviceN/serviceN-$mainProperty{environment}.properties</value>
</list>
</property>
</bean>
the error output is the next,
java.io.FileNotFoundException: /..../conf/serviceOne/service1-$mainProperty{environment}.properties (No such file or directory)
my opinion is, the value has not replaced
helpme, thanks
The problem is that when BeanFactoryPostProcessors are starting to be invoked, they are already instantiated. So even thou the first PropertyPlaceholderConfigurer modifies the bean definition of the second PropertyPlaceholderConfigurer, it has no effect as both beans have been already instantiated.

externalizing terracottaconfig property in ehcache.xml file inside Spring framework

There is this tag <terracottaConfig url="host1:9510,host2:9510,host3:9510"/> in ehcache.xml file inside spring web application. I want to externalize url attribute of this tag. Value of URL should be replaced by a property from external file. It will be very helpful if you suggest any solution to this problem.
You can put something like this -
<terracottaConfig url="${terracotta.config.location}" /> , however the big catch is that this will be loaded only from the system properties. It is not resolved from PropertyPlaceHolder as it is not a Spring configuration file.
So if you want to use an external config file, you will basically have to programatically set this system property before the Spring Application starts loading up the ehcache.xml file - one way to do that will be write your custom ServletContextListener to load up your properties file and set the system property based on that, this way when the ehcache.xml is loaded up it would be able to resolve the place holder properly.
Your answer helped me to solve my problem. I just want to add that instead of setting system property through program, I am using util:properties as follows
<bean id="sysProperties" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetObject" value="#{#systemProperties}"/>
<property name="targetMethod" value="putAll"/>
<property name="arguments">
<util:properties>
<prop key="propertyname_used_in_ecache_xml">#{proerties_defined_using_property_factory['propertyname_defined_in_external_properties_file']}</prop>
</util:properties>
</property>
</bean>
<bean id="cacheManager"
class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean" depends-on="sysProperties">
<property name="configLocation">
<value>classpath:ehcache.xml</value>
</property>
</bean>

Categories

Resources