Develop Reference

Spring Integration Java DSL: Remote filter is not applied on handle method

I want to read specific files from a SFTP server and get only compressed files only once.
I encounter a problem when handling message because the defined filter on the remote server seems to not be applied in handle method.
Dependencies:
SpringBoot: 2.2.1
spring-integration: 5.2.1
spring-integration-jdbc: 5.2.1
spring-integration-sftp: 5.2.1
public IntegrationFlow buildSftpInboundIntegrationFlow() {
return IntegrationFlows
.from(
Sftp
.inboundStreamingAdapter(buildSftpRemoteFileTemplate())
.remoteDirectory(getRemoteDirectoryPath())
.filter(buildRemoteFileFilter())
.remoteFileSeparator(
Optional
.ofNullable(getRemoteFileSeparator())
.orElse(DEFAULT_REMOTE_PATH_SEPARATOR))
.maxFetchSize(
Optional.ofNullable(getMaxFetchSize()).orElse(DEFAULT_MAX_FETCH_SIZE)),
sourcePollingChannelAdapterSpec -> sourcePollingChannelAdapterSpec
.id(getSftpInboundStreamingAdapterIdentifier())
.autoStartup(true)
.poller(buildPollerSpec()))
.handle(handleMessage())
.get();
}
/**
* Allows to build a regex to filter files.
*
* #return a regex as a {#link String}.
*/
private String buildRegexFileFilter() {
return String.format(".*\\.%s", getFileExtensionToFilter());
}
/**
* Allows to build an instance of {#link SftpRemoteFileTemplate}.
*
* #return an instance of {#link SftpRemoteFileTemplate}.
*/
private SftpRemoteFileTemplate buildSftpRemoteFileTemplate() {
final SftpRemoteFileTemplate sftpRemoteFileTemplate = new SftpRemoteFileTemplate(getSftpSessionFactory());
sftpRemoteFileTemplate.setAutoCreateDirectory(true);
return sftpRemoteFileTemplate;
}
/**
* Allows to build the filters to apply to the remote files.
*
* #return an instance of {#link CompositeFileListFilter}.
*/
#SuppressWarnings("resource")
private CompositeFileListFilter<LsEntry> buildRemoteFileFilter() {
return new ChainFileListFilter<LsEntry>() // NOSONAR
.addFilters(
new SftpRegexPatternFileListFilter(buildRegexFileFilter()),
getSftpPersistentAcceptOnceFileListFilter());
}
/**
* Allows to build the poller specifications.
*
* #return an instance of {#link PollerSpec}.
*/
private PollerSpec buildPollerSpec() {
return Pollers
.fixedDelay(
Optional.ofNullable(getPollerDelayInSeconds()).orElse(DEFAULT_POLLER_DELAY_IN_SECONDS),
TimeUnit.SECONDS)
.transactional()
.transactionSynchronizationFactory(getTransactionSynchronizationFactory());
}
...
Do you have any ideas to suggest to me ?
why in the handle method I receive files which should be excluded by the remote filter ?
It is a bug ? How get filtered messages ?

It's a bug in the modules spring-integration and spring-integration-sftp in version 5.2.1.
It works by upgrading these dependencies in version 5.2.2. (December 6, 2019)

How to find the path of properties file dynamically from the code?

I have maven project in Java in which I have a property file (quartz.properties) under this directory:
/src/main/resources
Now I can use this property file in two ways from this class as shown below:
/**
* Create a StdSchedulerFactory that has been initialized via
* <code>{#link #initialize(Properties)}</code>.
*
* #see #initialize(Properties)
*/
public StdSchedulerFactory(Properties props) throws SchedulerException {
initialize(props);
}
/**
* Create a StdSchedulerFactory that has been initialized via
* <code>{#link #initialize(String)}</code>.
*
* #see #initialize(String)
*/
public StdSchedulerFactory(String fileName) throws SchedulerException {
initialize(fileName);
}
I am not sure how can I use StdSchedulerFactory class to provide the path of my quartz.properties file.
As of now I am providing hardcoded path like this but this is not the right way to provide the path since if anyone else is running this code in their desktop or laptop then it will not work. I will be running this application from my desktop and also I will be making a runnable jar as well so I want that my program should load my properties file dynamically without any hardcoded path.
public class TestingQuartz {
public static void main(String[] args) throws SchedulerException {
SchedulerFactory factory = new StdSchedulerFactory(
"C:\\workspace\\tester_quartz\\quartzmain\\src\\main\\resources\\quartz.properties");
Scheduler scheduler = factory.getScheduler();
scheduler.start();
}
}

As your configuration file is in src/main/resources of a mavenized project, it will be embedded in the resulting artifact (jar, war...) you build with maven. Thus you should load the file "from the classpath" like this :
StdSchedulerFactory factory = new StdSchedulerFactory();
factory.initialize(this.getClass().getClassLoader().getResourceAsStream("quartz.properties"));

Since it appears to be on your class path you could do this:
getClass().getClassLoader().getResource("quartz.properties").toExternalForm()

Java - Loading External Properties File - FileNotFoundException

I've written a class that loads external properties for use in my program, it is modeled off of this link: Load Properties File in Singleton Class. I've read here (FileNotFoundException with properties file) and here (Java Properties File not loading) of how to load an external property file, however I continue to get a FileNotFoundException. My properties file is external and is located in the same directory as my executable jar.
public class ExternalProperties extends Properties{
private static ExternalProperties instance = null;
private Properties properties;
private static String location;
protected ExternalProperties() throws IOException {
properties = new Properties();
properties.load(getClass().getResourceAsStream("test.properties"));
}
public static ExternalProperties getInstance(){
if(instance == null){
try{
instance = new ExternalProperties();
} catch (IOException e) {
e.printStackTrace();
}
}
return instance;
}
public static void setLocation(String path){
location = path;
}
}
I am passing the location of the property file through the command line such as:
java -jar chef-deploy-tests-0.0.0009-SNAPSHOT-jar-with-dependencies.jar test.properties
Any suggestions on what I am doing wrong?

It depends on your location of the test.properties file.
* <li> If the name begins with a {#code '/'}
* then the absolute name of the resource is the
* portion of the name following the {#code '/'}.
*
* <li> Otherwise, the absolute name is of the following form:
*
* <blockquote>
* {#code modified_package_name/name}
* </blockquote>
*
* <p> Where the {#code modified_package_name} is the package name of this
* object with {#code '/'} substituted for {#code '.'}.

I think your problem is that test.properties is NOT on your classpath. The way you are running it now is simply passing the file as an argument to the application. I think what you mean to do is add it to the application's classpath:
java -jar chef-deploy-tests-0.0.0009-SNAPSHOT-jar-with-dependencies.jar -classpath .
This will include all the files in the the current directory in the java classpath. Another option may be to put that test.properties file into a separate directory and specify that one instead of the '.'

Building effective model with DefaultModelBuilder . build()

I'm trying to get a effective model for a project and currently under the Maven Core API 3.0.3 (or 3.0.4) there's this method called build() which is neat BUT..
it requires too much like ModelNormalizer, ProfileInjector etc for it not to throw null pointer exception while building. There's so much things that need initialization before I can build the effective model and all the information I have is a filepath to the pom.xml
Anyone has work this out?

Here is sample code:
DefaultModelBuilder modelBuilder = new DefaultModelBuilderFactory().newInstance();
ModelBuildingRequest req = new DefaultModelBuildingRequest();
req.setProcessPlugins(false);
req.setPomFile(file);
req.setValidationLevel(ModelBuildingRequest.VALIDATION_LEVEL_MINIMAL);
req.setModelResolver(new MyLocalModelResolver());
Model model = modelBuilder.build(req).getEffectiveModel();

Maybe this library can help: https://github.com/jenkinsci/lib-jenkins-maven-embedder
Or at least as a sample.
hth
Olivier

Yes it works for me. I have to set all data with default values. Very annoying...
I have created a MavenModelBuilder builder class for that
/**
* Default MavenModelBuilder.
*/
public class MavenModelBuilder extends DefaultModelBuilder {
/**
* Constructor
*/
public MavenModelBuilder() {
super();
ModelProcessor modelProcessor = new DefaultModelProcessor()
.setModelLocator(new DefaultModelLocator())
.setModelReader(new DefaultModelReader());
ModelInterpolator modelInterpolator = new StringSearchModelInterpolator()
.setPathTranslator(new DefaultPathTranslator())
.setUrlNormalizer(new DefaultUrlNormalizer());
setProfileSelector(new DefaultProfileSelector());
setModelProcessor(modelProcessor);
setModelValidator(new DefaultModelValidator());
setSuperPomProvider(new DefaultSuperPomProvider().setModelProcessor(modelProcessor));
setModelNormalizer(new DefaultModelNormalizer());
setInheritanceAssembler(new DefaultInheritanceAssembler());
setModelInterpolator(modelInterpolator);
setModelUrlNormalizer(new DefaultModelUrlNormalizer().setUrlNormalizer(new DefaultUrlNormalizer()));
setModelPathTranslator(new DefaultModelPathTranslator().setPathTranslator(new DefaultPathTranslator()));
setPluginManagementInjector(new DefaultPluginManagementInjector());
setLifecycleBindingsInjector(new DefaultLifecycleBindingsInjector());
setDependencyManagementInjector(new DefaultDependencyManagementInjector());
setReportConfigurationExpander(new DefaultReportConfigurationExpander());
setReportingConverter(new DefaultReportingConverter());
setPluginConfigurationExpander(new DefaultPluginConfigurationExpander());
setDependencyManagementImporter(new DefaultDependencyManagementImporter());
setProfileInjector(new DefaultProfileInjector());
}
}
And i call it like this
/**
* Parse the xml file to get the model of the xml file.
*
* #param path of the project
* #throws IllegalArgumentException : exception
*/
public static Model parseFile(String path) throws IllegalArgumentException {
DefaultModelBuilder builder = new MavenModelBuilder();
ModelBuildingRequest req = new DefaultModelBuildingRequest();
req.setProcessPlugins(false);
req.setPomFile(new File(path));
req.setValidationLevel(ModelBuildingRequest.VALIDATION_LEVEL_MINIMAL);
Model model = null;
try {
model = builder.build(req).getEffectiveModel();
} catch (ModelBuildingException e) {
e.printStackTrace();
}
return model;
}

How to programatically invoke a maven dependency plugin

I am trying to invoke maven-dependency-plugin programatically. i am using maven 3 version. the problem is that when i invoke it through pluginManager.executeMojo(session, execution), i receive the following error message:
[ERROR] **The parameters 'project', 'local', 'remoteRepos',
'reactorProjects' for goal
org.apache.maven.plugins:maven-dependency-plugin:2.1:unpack are
missing or invalid**
**org.apache.maven.plugin.PluginParameterException: The parameters 'project',
'local', 'remoteRepos', 'reactorProjects' for goal
org.apache.maven.plugins:maven-dependency-plugin:2.1:unpack are missing or
invalid**
at org.apache.maven.plugin.internal.DefaultMavenPluginManager
.populatePluginFields(DefaultMavenPluginManager.java:518)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager
.getConfiguredMojo(DefaultMavenPluginManager.java:471)
at org.apache.maven.plugin.DefaultBuildPluginManager
.executeMojo(DefaultBuildPluginManager.java:99)
at com.sap.ldi.qi.osgi.OSGiManifesrMfHandlerMojo
.invokeMavenDependencyPlugin(OSGiManifesrMfHandlerMojo.java:139)
at com.sap.ldi.qi.osgi.OSGiManifesrMfHandlerMojo
.execute(OSGiManifesrMfHandlerMojo.java:100)
at org.apache.maven.plugin.DefaultBuildPluginManager
.executeMojo(DefaultBuildPluginManager.java:110)
at org.apache.maven.lifecycle.internal.MojoExecutor
.execute(MojoExecutor.java:144)
at org.apache.maven.lifecycle.internal.MojoExecutor
.execute(MojoExecutor.java:87)
at org.apache.maven.lifecycle.internal.MojoExecutor
.execute(MojoExecutor.java:79)
-- many lines stripped from stack trace --
[INFO] ----------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ----------------------------------------------------------------------
[INFO] Total time: 17.938s
[INFO] Finished at: Mon Nov 22 10:27:42 EET 2010
[INFO] Final Memory: 12M/23M
[INFO] ----------------------------------------------------------------------
[ERROR] Failed to execute goal
com.sap.ldi.qi:osgi-manifest-handler-plugin:0.0.1-SNAPSHOT:handle
(osgi-manifest-handler plugin) on project com.sap.ldi.demo.calc
.cmd.tests: The parameters 'project', 'local', 'remoteRepos',
'reactorProjects' for goal
org.apache.maven.plugins:maven-dependency-plugin:2.1:unpack are missing
or invalid -> [Help 1]
-- stripped rest --
As I know, the only required parameter for the unpack goal of maven dependency plugin is artifactItems. I set the plugin configuration by using PluginExecution.setConfiguration() method. It seems that this plugin configuration is not correctly set.
Do you have any idea why this exception is thrown?
Here is the configuration that I am using:
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.sap.ldi.demo.calc</groupId>
<artifactId>com.sap.ldi.demo.calc.cmd</artifactId>
<version>0.1.2-SNAPSHOT</version>
<type>jar</type>
<overWrite>true</overWrite>
<outputDirectory>target/demo-calc-stuff</outputDirectory>
<includes>**/*.*</includes>
</artifactItem>
</artifactItems>
</configuration>
Thanks
One correction from my side. The used Maven version is not Maven 3.0 but Maven 3.0-beta-1. I see that BuildPluginManager.loadPlugin() in version 3.0-beta-1 has two args, and the same method in version 3.0 has three.
I am wondering, does anyone tried to invoke a maven plugin programatically with maven 3.0 or maven 3.0-beta-1. I am still trying to invoke it with maven 3.0-beta-1, but it still returns the same exception as pasted above.

Here is an updated version of Mojo Executor designed for Maven 3:
package com.googlecode.boostmavenproject;
import java.util.Collections;
import org.apache.maven.execution.MavenSession;
import org.apache.maven.model.Plugin;
import org.apache.maven.plugin.MojoExecution;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.descriptor.PluginDescriptor;
import org.apache.maven.project.MavenProject;
import org.codehaus.plexus.util.xml.Xpp3Dom;
import org.apache.maven.plugin.BuildPluginManager;
import org.apache.maven.plugin.descriptor.MojoDescriptor;
import org.codehaus.plexus.configuration.PlexusConfiguration;
import org.codehaus.plexus.util.xml.Xpp3DomUtils;
import org.sonatype.aether.repository.RemoteRepository;
/**
* Executes an arbitrary mojo using a fluent interface. This is meant to be executed within the context of a Maven 2
* mojo.
*
* Here is an execution that invokes the dependency plugin:
* <pre>
* executeMojo(
* plugin(
* groupId("org.apache.maven.plugins"),
* artifactId("maven-dependency-plugin"),
* version("2.0")
* ),
* goal("copy-dependencies"),
* configuration(
* element(name("outputDirectory"), "${project.build.directory}/foo")
* ),
* executionEnvironment(
* project,
* session,
* pluginManager
* )
* );
* </pre>
* #see http://code.google.com/p/mojo-executor/
*/
public class MojoExecutor
{
/**
* Entry point for executing a mojo
*
* #param plugin The plugin to execute
* #param goal The goal to execute
* #param configuration The execution configuration
* #param env The execution environment
* #throws MojoExecutionException If there are any exceptions locating or executing the mojo
*/
public static void executeMojo(Plugin plugin, String goal, Xpp3Dom configuration,
ExecutionEnvironment env) throws MojoExecutionException
{
if (configuration == null)
throw new NullPointerException("configuration may not be null");
try
{
MavenSession session = env.getMavenSession();
PluginDescriptor pluginDescriptor = env.getPluginManager().loadPlugin(plugin,
Collections.<RemoteRepository>emptyList(), session.getRepositorySession());
MojoDescriptor mojo = pluginDescriptor.getMojo(goal);
if (mojo == null)
{
throw new MojoExecutionException("Could not find goal '" + goal + "' in plugin "
+ plugin.getGroupId() + ":"
+ plugin.getArtifactId() + ":"
+ plugin.getVersion());
}
configuration = Xpp3DomUtils.mergeXpp3Dom(configuration,
toXpp3Dom(mojo.getMojoConfiguration()));
MojoExecution exec = new MojoExecution(mojo, configuration);
env.getPluginManager().executeMojo(session, exec);
}
catch (Exception e)
{
throw new MojoExecutionException("Unable to execute mojo", e);
}
}
/**
* Converts PlexusConfiguration to a Xpp3Dom.
*
* #param config the PlexusConfiguration
* #return the Xpp3Dom representation of the PlexusConfiguration
*/
private static Xpp3Dom toXpp3Dom(PlexusConfiguration config)
{
Xpp3Dom result = new Xpp3Dom(config.getName());
result.setValue(config.getValue(null));
for (String name: config.getAttributeNames())
result.setAttribute(name, config.getAttribute(name));
for (PlexusConfiguration child: config.getChildren())
result.addChild(toXpp3Dom(child));
return result;
}
/**
* Constructs the {#link ExecutionEnvironment} instance fluently
* #param mavenProject The current Maven project
* #param mavenSession The current Maven session
* #param pluginManager The Build plugin manager
* #return The execution environment
* #throws NullPointerException if mavenProject, mavenSession or pluginManager
* are null
*/
public static ExecutionEnvironment executionEnvironment(MavenProject mavenProject,
MavenSession mavenSession,
BuildPluginManager pluginManager)
{
return new ExecutionEnvironment(mavenProject, mavenSession, pluginManager);
}
/**
* Builds the configuration for the goal using Elements
* #param elements A list of elements for the configuration section
* #return The elements transformed into the Maven-native XML format
*/
public static Xpp3Dom configuration(Element... elements)
{
Xpp3Dom dom = new Xpp3Dom("configuration");
for (Element e: elements)
dom.addChild(e.toDom());
return dom;
}
/**
* Defines the plugin without its version
* #param groupId The group id
* #param artifactId The artifact id
* #return The plugin instance
*/
public static Plugin plugin(String groupId, String artifactId)
{
return plugin(groupId, artifactId, null);
}
/**
* Defines a plugin
* #param groupId The group id
* #param artifactId The artifact id
* #param version The plugin version
* #return The plugin instance
*/
public static Plugin plugin(String groupId, String artifactId, String version)
{
Plugin plugin = new Plugin();
plugin.setArtifactId(artifactId);
plugin.setGroupId(groupId);
plugin.setVersion(version);
return plugin;
}
/**
* Wraps the group id string in a more readable format
* #param groupId The value
* #return The value
*/
public static String groupId(String groupId)
{
return groupId;
}
/**
* Wraps the artifact id string in a more readable format
* #param artifactId The value
* #return The value
*/
public static String artifactId(String artifactId)
{
return artifactId;
}
/**
* Wraps the version string in a more readable format
* #param version The value
* #return The value
*/
public static String version(String version)
{
return version;
}
/**
* Wraps the goal string in a more readable format
* #param goal The value
* #return The value
*/
public static String goal(String goal)
{
return goal;
}
/**
* Wraps the element name string in a more readable format
* #param name The value
* #return The value
*/
public static String name(String name)
{
return name;
}
/**
* Constructs the element with a textual body
* #param name The element name
* #param value The element text value
* #return The element object
*/
public static Element element(String name, String value)
{
return new Element(name, value);
}
/**
* Constructs the element containg child elements
* #param name The element name
* #param elements The child elements
* #return The Element object
*/
public static Element element(String name, Element... elements)
{
return new Element(name, elements);
}
/**
* Element wrapper class for configuration elements
*/
public static class Element
{
private final Element[] children;
private final String name;
private final String text;
public Element(String name, Element... children)
{
this(name, null, children);
}
public Element(String name, String text, Element... children)
{
this.name = name;
this.text = text;
this.children = children;
}
public Xpp3Dom toDom()
{
Xpp3Dom dom = new Xpp3Dom(name);
if (text != null)
{
dom.setValue(text);
}
for (Element e: children)
{
dom.addChild(e.toDom());
}
return dom;
}
}
/**
* Collects Maven execution information
*/
public static class ExecutionEnvironment
{
private final MavenProject mavenProject;
private final MavenSession mavenSession;
private final BuildPluginManager pluginManager;
public ExecutionEnvironment(MavenProject mavenProject, MavenSession mavenSession,
BuildPluginManager pluginManager)
{
if (mavenProject == null)
throw new NullPointerException("mavenProject may not be null");
if (mavenSession == null)
throw new NullPointerException("mavenSession may not be null");
if (pluginManager == null)
throw new NullPointerException("pluginManager may not be null");
this.mavenProject = mavenProject;
this.mavenSession = mavenSession;
this.pluginManager = pluginManager;
}
public MavenProject getMavenProject()
{
return mavenProject;
}
public MavenSession getMavenSession()
{
return mavenSession;
}
public BuildPluginManager getPluginManager()
{
return pluginManager;
}
}
}
I will attempt to contribute my changes back into the official Mojo Executor plugin.

Folks, I think I get it.
The problem is not in the version of Maven that I am using. It is in the configuration that I am using for invoking maven-dependency-plugin. The unpack goal of maven-dependency-plugin requires the following parameters: artifactItems, local, project, reactorProjects and remoteRepos. Here is the correct version of the configuration used for invoking the plugin:
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.sap.ldi.demo.calc</groupId>
<artifactId>com.sap.ldi.demo.calc.cmd</artifactId>
<version>0.1.3-SNAPSHOT</version>
<type>jar</type>
<overWrite>true</overWrite>
<outputDirectory>target/demo-calc-stuff</outputDirectory>
<includes>**/*.*</includes>
</artifactItem>
</artifactItems>
<local>${localRepository}</local>
<project>${project}</project>
<reactorProjects>${reactorProjects}</reactorProjects>
<remoteRepos>${project.remoteArtifactRepositories}</remoteRepos>
</configuration>`

Maven Plugins are not meant to be invoked programmatically.
They rely on values that are injected by the underlying plexus container.
So either you will have to find out how to inject those values or you will have to rely on the default mechanism.
One thing you can use is the Maven Invoker. With that, you can programmatically launch maven lifecycles, but they will execute in a separate VM. So if you need to change the model dynamically beforehand, you will need to serialize the model out to a temporary pom.xml and use that with maven invoker. This is heavy stuff, but I have done it successfully some two years ago.