Develop Reference

Spring Integration is not detecting files on FTP Folder

Currently I'm facing a problem where I don't know what to do to resolve it. I'm developing a simple application that transfer files through different FTP and SFTP servers to be processed. Well, at the beginnig those servers weren't ready, so I used Apache Mina and RebexTinySftpServer to set a FTP server and a SFTP server on my computer to test my development.
With those applications I completed and tested my application locally, so it was time to test it using the servers that will be used in production, but something is wrong with the FTP server and Spring Integration is not detecting the files that are put in the folder to be transferred to the SFTP server.
I have two folders on each server: one for Input files and the another one for Output files. So when Spring Integration detects that there's a new file in the Input folder on the SFTP Server, it transfers the file to the Input folder on FTP Server to be processed for another service. That part works fine.
After that service processes the file, it generates an output file and stores it in the Output folder on the FTP Server to be transferred to the Output folder on the SFTP server, but for some reason Spring Integration is not detecting those files and doesn't transfer none of them. This part is what I don't know how to solve because none Exception is being thrown, so I don't know what part of my code I have to modify.
Here is my code where I define the DefaultFtpSessionFactory:
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory session = new DefaultFtpSessionFactory();
session.setUsername(username);
session.setPassword(password);
session.setPort(port);
session.setHost(host);
session.setFileType(FTP.ASCII_FILE_TYPE);
session.setClientMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
return session;
}
And here is the code where I define the FTP Inbound Adapter:
#Bean
FtpInboundChannelAdapterSpec salidaAS400InboundAdapter() {
return Ftp.inboundAdapter(as400Session.ftpSessionFactory())
.preserveTimestamp(true)
.remoteDirectory(as400Session.getPathSalida())
.deleteRemoteFiles(true)
.localDirectory(new File("tmp/as400/salida"))
.temporaryFileSuffix(".tmp");
}
I should mention that the FTP Server is running on an AS/400 system, so maybe that has something to do with the situation I'm facing.

I found the solution of my problem. I'm posting this in case something similar happens to someone else.
My project is using spring-integration-ftp 5.5.16 that has commons-net 3.8.0 as a dependency. For some reason, that version of commons-net wasn't retrieving the files inside the directory of the AS400, so I excluded that dependency from spring-integration-ftp and added commons-net 3.9.0 in my project. Now everything works fine.

How can I produce messages with Kafka 8.2 API in Java?

I'm trying to work with the kafka API in java. I'm using the following maven dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.8.2.0</version>
</dependency>
I'm having trouble connecting to a remote kafka server.
I changed the kafka 'server.properties' file port attribute to be port 8080.
I can start both the zookeeper and the kafka server no problem.
I can also use the console producer and consumer applications that came with the kafka download. (Scala 2.10 version)
I'm using the following client code to create a remote KafkaProducer
Properties propsProducer = new Properties();
propsProducer.put("bootstrap.servers", "172.xx.xx.xxx:8080");
propsProducer.put("key.serializer", org.apache.kafka.common.serialization.ByteArraySerializer.class);
propsProducer.put("value.serializer", org.apache.kafka.common.serialization.ByteArraySerializer.class);
propsProducer.put("topic.metadata.refresh.interval.ms", "0");
KafkaProducer<byte[], byte[]> m_kafkaProducer = new KafkaProducer<byte[], byte[]>(propsProducer);
Once I've created the producer, I can run the following line and get valid topic info returned, granted strTopic is an existing topic name.
List<PartitionInfo> partitionInfo = m_kafkaProducer.partitionsFor(strTopic);
When I try to send a message, I do the following:
ProducerRecord<byte[], byte[]> prMessage = new ProducerRecord<byte[],byte[]>(strTopic, strMessage.getBytes());
RecordMetadata futureData = m_kafkaProducer.send(prMessage).get();
The call to send() blocks indefinitely and when I manually terminate the process, I see that the ERROR Closing socket because of error on kafka server(IOException, Connection Reset by Peer) error.
Also, it's worth nothing that the host.name, advertised.host.name, and advertised.port properties are all still commented out on the 'server.properties' file. Oh, and if I change the line:
propsProducer.put("bootstrap.servers", "172.xx.xx.xxx:8080");
to
propsProducer.put("bootstrap.servers", "127.0.0.1:8080");
and run it on the same server as where the kafka server is installed, it works but I'm trying to work with it remotely.
Appreciate any help and if I can clarify at all let me know.

After lots of digging, I decided to implement the example found here: Kafka Producer Example. I shortened the code and didn't implement a partitioner class. I updated my pom with the dependency listed and I was still having the same issue. Ultimately, I made some configuration changes and everything worked.
The final piece of the puzzle was defining the Kafka server in /etc/hosts of both the server and the client machines. I added the following to both files.
172.xx.xx.xxx serverHost1
Again, the x's are just masks. Then, I set the advertised.host.name in the server.properties file to serverHost1. NOTE: I got that IP after running an ifconfig on the server machine.
I changed the line
propsProducer.put("metadata.broker.list", "172.xx.xx.xxx:8080");
to
propsProducer.put("metadata.broker.list", "serverHost1:8080");
The Kafka API didn't like the fact that I was defining an IP as a string. Instead it was looking up the IP from within the etc/hosts file although the documentation says:
"Hostname the broker will advertise to producers and consumers. If not set, it uses the value for "host.name" if configured. Otherwise, it will use the value returned from java.net.InetAddress.getCanonicalHostName()."
Which will just return the IP, in the string form, I was previously using if not defined in etc/hosts of client machine, otherwise it returns the name paired with the IP (serverHost1 in my case). Also, I never did set the value of host.name either.

Not able to Configure Queue Connection Factories in WebSphere Application Server

I'm using RAD 9.0 and trying to configure Queue connection factories in WebSphere Application Server 8.5. I have IBM MQ 7.0 (32-bit) installed on the same machine (Win7 64-bit).
After configuring Queue connection factories when I click on Test Connection it give an error :
A connection could not be made to WebSphere MQ for the following reason: CC=2;RC=2495;AMQ8568: The native JNI library 'mqjbnd' was not found. For a client installation this is expected. [3=mqjbnd]
Native library path (under JMS>WebSphere MQ messeging provider) is set to C:/Program Files (x86)/IBM/WebSphere MQ\java\lib.
I also tried to set it to C:/Program Files (x86)/IBM/WebSphere MQ\java\lib64 but still I'm getting the same error.
Also is it necessary to configure Queue Connection Factory for configuring Listener Port for MDB?

The error message means you have configured the connection factory to have a transport type of Bindings and so the WMQ Resource Adapter within WAS needs to load the native libraries located in the MQ installation (note the MQ client installation does not come with these libraries).
Assuming you want to connect in Bindings mode AND you have a full local MQ Server installation on the same box as the WAS server then you will need to configure the 'Native Library Path' on the WebSphere MQ messaging provider panel in WAS (Resources > JMS > JMS Providers). You should alter the provider that is at the same scope as the defined queue connection factory.
The MQ_INSTALL_ROOT property is an old property used in WAS 6.0 and WAS 6.1 and is only intended to be used for migration reasons in WAS 7 and onwards.
Note: If you have an ND environment then the 'Test Connection' operation could potentially run on the dMgr process rather than the server. If your dMgr is on a different host then the libraries will not be found. In this case you should ensure that the application server is running before selecting the 'Test Connection' button.

WAS uses environment variable MQ_INSTALL_ROOT to point to (embedded) WebSphereMQ Client (Environment->WebSphere Variables). Default value is ${WAS_INSTALL_ROOT}/lib/WMQ. I think you don't need a separate installation of WebSphereMQ client - it comes with WAS (I'm working with WAS8, but I guess they did not change it in 8.5).
As for your question, it might be the issue with the path: it uses spaces. Try to install WebSphereMQ client libs in a directory without spaces (e.g. C:\IBM\WMQClient). But I think you do not need it, check directory ${WAS_INSTALL_ROOT}/lib/WMQ - it should be there.
And about Activation Specification - you don't need Queue Connection Factory, you need only queue definition where Activation Specification will connect to.

java.lang.UnsatisfiedLinkError

i am trying to access MQ queues using JMS. i am getting the below
java.lang.UnsatisfiedLinkError: no mqjbnd05 in java.library.path
i am passing
-Djava.library.path="C:\Program Files\IBM\WebSphere MQ\java\lib"
as the VM argument while running the program in eclipse. This issue is discussed quite a lot on the net but with out any conclusion. Has anyone resolved this? TIA.

As I had to deal with this error myself; and it took me a lot of time to find the right answer, I'd like to share it with the next one, who comes along this thread...
Actually the solution to the problem was very simple (at least in my case). It was not related to any CLASSPATH, java.library.path or installation issues.
I simply forgot to switch the MQConnectionFactory into the Client mode.
This has to be done, by simply calling
cf.setTransportType(WMQConstants.WMQ_CM_CLIENT);
or
cf.setTransportType(WMQConstants.WMQ_CM_BINDINGS_THEN_CLIENT);
or any other connection type, that fits your needs.
By default, the ConnectionFactory is in "Binding" mode (WMQ_CM_BINDINGS), which is intended for local server installations, as it is is stated in the IBM Documentation:
To connect to a queue manager in bindings mode, a WebSphere MQ classes for JMS application must run on the same system on which the queue manager is running.
This transport type is the same as the XMSC_WMQ_CONNECTION_MODE (WMQConstants.WMQ_CONNECTION_MODE) property, when using JNDI or the JmsFactoryFactory.
The same should apply to the other ConnectionFactory types: MQQueueConnectionFactory, MQTopicConnectionFactory, MQXAConnectionFactory, MQXAQueueConnectionFactory and MQXATopicConnectionFactory
Check the IMB Knowledge Center for more information about the different connection/binding options:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.dev.doc/q031720_.htm
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.dev.doc/q030560_.htm

You probably have some older MQ jar files either in your CLASSPATH, in the lib or in the EAR.
Remove them and you should be fine.
You should not put MQ files in your EAR or in the WEB-INF/lib folders. They should be in the classpath of your appserver.

I came across this while connecting using IBM MQ api.
I didn't find this issue to be related to classpath either.
This happened to me when I instantiated MQQueueManager before setting MQEnvironment's hostname and channel.
Just ensure that your code does not do that and that it instantiates the manager after the environment is set. Something like..
MQEnvironment.hostname = "mq hostname";
MQEnvironment.channel = "mq channel";
..more code..
this._queueManager = new MQQueueManager(qManager);
(Observed that it's OK to set MQEnvironment.port after MQQueueManager is initialized, but one would probably initialize everything related to MQEnvironment together)

This can happen if you actually installed MQ Client instead of MQ Server.
IBM has even written a whole help page about it:
WebSphere MQ Client installation missing mqjb*.dll files
Problem(Abstract)
You install the WebSphere MQ Client and notice three dll's are missing from the \Program Files\IBM\WebSphere MQ\Java\lib\ directory.
Symptom
The following dlls appear in the directory on a server install, but are not part of the Java™ client:
03/17/2003 10:59a 19,456 mqjbdf02.dll
03/17/2003 10:59a 57,856 mqjbnd05.dll
03/17/2003 10:59a 36,864 MQXAi02.dll
The subdirectory \jdbc\ appears on the server, but not on the client machine.
03/17/2003 10:59a 61,440 jdbcdb2.dll
03/17/2003 10:59a 61,440 jdbcora.dll
Cause
The files are missing because they are not provided nor needed in a client install.
Resolving the problem
The files are only included in the WebSphere MQ Server product.

Here is an easy recipe: Tell the Java VM to Load the DLL. Is your code similiar, e.g. do you use System.loadLibrary to load mqjbnd05.dll?
If yes - does it work outside eclipse, like starting the application from the command line? If this is the case, you could try starting the whole eclipse IDE with that library path.
And sometimes we have trouble with pathnames that contain spaces. Copy the dll to C:\, put that on the lib path and try again.
Ah, that's the problem, the specified dll is missing. This blog has a solution. Good luck!

In my case when I set the transport type , the error goes away. I was using MQConnectionFactory
mQQueueConnectionFactory.setTransportType( JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);

What are the minimum requirements for writing a Java client for MQSeries?

I need to write a simple MQSeries client in Java.
The client just has to connect to the queue and pull off the next message.
I have done this before a number of years ago and have all the sample code etc.
All I remember needing are the three jar files:
com.ibm.mq.iiop.jar
com.ibm.mq.jar
connector.jar
I have been doing some reading and a lot of people talk about a properties file, but I have no recollection of this from my past experience.
And so on to my question:
What is the absolute minimum I need on my system to develop, test and ultimately deploy a simple MQSeries client?
And where can I find (download) these things?
NOTE: This question is related to but not the same as this one.

The answer to the question depends on whether you want to put the resulting code into Production. You can grab a few jar files and include them on the CLASSPATH and get it to work. In fact with the repackaging done in v7, the number of jars has been reduced. However, it is not supported by IBM without the full client install.
Why is that? The full install includes diagnostic facilities, supporting code, NLS support and the ability to collect all the Must Gather information to open a trouble ticket. If all you grab is a few jar files, you do not get the additional functionality and therefore no vendor support if something goes wrong.
The WMQ v7 client install provided as SupportPac MQC7 which is a free download.
The jar files and CLASSPATH settings for both base Java and JMS are provided in the Using Java manual.
Use the v7 client, even on a v6 server. It's backward compatible. In v7.0 the Java/JMS has all been rewritten and you just need the jars listed in the manual.
Also, someone mentioned the etclient jar. That's the Extended Transactional Client and please be aware that it incurs the full license cost of a QMgr. You only need it if you are doing 2-phase commit (XA with WMQ and another resource manager in the same unit of work) and not using WebSphere App Server or Message Broker. For example, you want to do TPC using JBoss, MQ and a database, you need etclient jar file and pay for a full WMQ license. If you are not doing XA transactions you don't need it.

Here is another way...
Using the three jar files:
com.ibm.mq.jar
com.ibm.mqetclient.jar
com.ibm.mqjms.jar
Here is a code sample that will read an MQ message -
import com.ibm.mq.*; // Include the WebSphere MQ classes for Java package
public class MQSample
{
private String qManager = "your_Q_manager"; // define name of queue
// manager to connect to.
private MQQueueManager qMgr; // define a queue manager
// object
public static void main(String args[]) {
new MQSample();
}
public MQSample() {
try {
// Create a connection to the queue manager
qMgr = new MQQueueManager(qManager);
// Set up the options on the queue we wish to open...
// Note. All WebSphere MQ Options are prefixed with MQC in Java.
int openOptions = MQC.MQOO_INPUT_AS_Q_DEF |
MQC.MQOO_OUTPUT ;
// Now specify the queue that we wish to open,
// and the open options...
MQQueue system_default_local_queue =
qMgr.accessQueue("SYSTEM.DEFAULT.LOCAL.QUEUE",
openOptions);
// Define a WebSphere MQ message buffer to receive the message into..
MQMessage retrievedMessage = new MQMessage();
// Set the get message options...
MQGetMessageOptions gmo = new MQGetMessageOptions(); // accept the defaults
// same as MQGMO_DEFAULT
// get the message off the queue...
system_default_local_queue.get(retrievedMessage, gmo);
// And prove we have the message by displaying the UTF message text
String msgText = retrievedMessage.readUTF();
System.out.println("The message is: " + msgText);
// Close the queue...
system_default_local_queue.close();
// Disconnect from the queue manager
qMgr.disconnect();
}
// If an error has occurred in the above, try to identify what went wrong
// Was it a WebSphere MQ error?
catch (MQException ex)
{
System.out.println("A WebSphere MQ error occurred : Completion code " +
ex.completionCode + " Reason code " + ex.reasonCode);
}
// Was it a Java buffer space error?
catch (java.io.IOException ex)
{
System.out.println("An error occurred whilst writing to the message buffer: " + ex);
}
}
} // end of sample
Code taken from this SO answer.

Ok it looks like you need the three jars I mentioned in the question as well as a properties file.
com.ibm.mq.iiop.jar
com.ibm.mq.jar
connector.jar
mqji.properties
Unless you have access to these things already the only way I could figure out to get them was to download and install the FULL trial version for MQSeries from IBM:
http://www.ibm.com/developerworks/downloads/ws/wmq/
If that link dies over time you can also find it by just going to www.ibm.com and then following the menu from "Support & Downloads" -> "Download" -> "Trials and demos" and then choosing "WebSphere MQ" from the list.
Once the install is done, you have all the jars you need in the java/lib folder below where the installation happened. The Jars in this version are different from the jars I mention above, I suspect because of version differences.
The properties file was not installed with the install (perhaps the new versions does not need this file), but it can be found here.

Its been a while since I did this too, IBM provides java api for directly using mqseries and JMS api too that is a wrapper around it.
Go here, they have many examples in java http://www304.ibm.com/jct01005c/isv/tech/sampmq.html
For testing, you will need to download and setup a MQseries Server, or maybe they will be kind enough to set you up with a test queue on the real server.
In anycase, when you download the MQseries server from the IBM's website (trial lic) that should contain the jar's you need.
I would strongly research into the JMS api for MQSeries, so that you can stay away from actually using the mqseries api.