Develop Reference

how to connect to multiple ftp sites based on the ftphost name using one talend job

I am trying to connect to multiple ftp sites ,the thing is only the ftp host name (ip address) varies between different ftp sites and the username,password,port ,directory are all the same where i would to download and read /retrieve the latest files.
I have an existing flow,where it connects to one particular ftp site and does the operation,now i want to add multiple ftp sites using the context variable and invoke multiple flows for different ftp sites as part of the same flow considering only the HOSTNAME will vary,everything else is exactly the same.
Should i use a trunjob component or which is the easiest way to handle this?

Have you thought of using SFTP?
Here's a snippet using JSch library, you can modify it for multiple connections:
private static final String SFTP_SERVER_URL = "server.com";
private static final String SFTP_SERVER_USERNAME = "user";
private static final String SFTP_SERVER_PASSWORD = "pass";
private static final int SFTP_SERVER_PORT = port;
private final static String LOCAL_DOWNLOAD_PATH = "downloads/";
private static ChannelSftp setupSFTP() {
ChannelSftp channel = null;
try {
JSch jsch = new JSch();
JSch.setConfig("StrictHostKeyChecking", "no");
Session jschSession = jsch.getSession(SFTP_SERVER_USERNAME, SFTP_SERVER_URL);
jschSession.setPassword(SFTP_SERVER_PASSWORD);
jschSession.setPort(SFTP_SERVER_PORT);
jschSession.connect();
channel = (ChannelSftp) jschSession.openChannel("sftp");
} catch (JSchException e) {
e.printStackTrace();
}
return channel;
}
public static void downloadFile(String path, String fileName) {
try {
ChannelSftp channelSftp = setupSFTP();
channelSftp.connect();
// Download file and close connection
channelSftp.get(path, LOCAL_DOWNLOAD_PATH + fileName);
channelSftp.exit();
} catch (SftpException e) {
e.printStackTrace();
} catch (JSchException e) {
e.printStackTrace();
}
}
public static void uploadFile(String localPath, String remotePath) {
try {
ChannelSftp channelSftp = setupSFTP();
channelSftp.connect();
// Upload file and close connection
channelSftp.put(localPath, remotePath);
channelSftp.exit();
} catch (SftpException e) {
e.printStackTrace();
} catch (JSchException e) {
e.printStackTrace();
}
}
You can have multiple setup functions, one for each of the servers.
Link: http://www.jcraft.com/jsch/

SSLSocket.getInputStream() hangs when called Java 11

I am writing a test class for a networking module which establishes a SSL connection used for sending messages. The Junit 4 test class sets up a client side keystore and truststore along with a server side keystore. These variables are used in setting up client side and server side SSLContexts from which I get SSLServerSocket and SSLSocket necessary for setting up a connection through their respective factories.
The SSLServerSocket successfully accepts the connection of my SSLSocket on localhost at the same port. However when I call the SSLSocket.getInputStream() method on the server side socket it hangs whereas calling the SSLSocket.getOutputStream() mehtod on the client side is successful. I am aware that this stage is responsible for initiating the SSL handshake but through my search I have found little on what could be causing a single side to hand. Someone elses post on a separate site mentioned that is could be a reverse dns lookup hanging how would I prevent this? I also tried explicitly starting the handshake in the first of the two Callable threads which hung in a similar fashion. This is my test class:
public class ReceiverClientThreadTest {
// ADD REG AND A SINGLE NETWORK
// ESTABLISH A TLS CONNECTION BETWEEN TWO POINTS WITH
private final static String KEY_MANAGER = "SunX509";
private final static String TLS_VERSION = "TLSv1.2";
private final static String RNG_ALGORITHM = "DEFAULT";
private final static String RNG_PROVIDER = "BC";
private static final String PROVIDER = "BC";
private static final String KEYSTORE_TYPE = "PKCS12";
private static KeyStore keyStore1, keyStore2, trustStore2;
private SSLSocket serverSocket;
private SSLSocket clientSocket;
#BeforeClass
public static void setUp() throws SQLException, GeneralSecurityException, OperatorCreationException, IOException {
String name1 = "localhost", name2 = "client";
KeyPair kp1 = SecurityUtilities.generateKeyPair();
KeyPair kp2 = SecurityUtilities.generateKeyPair();
X509Certificate cert1 = SecurityUtilities.makeV1Certificate(kp1.getPrivate(), kp1.getPublic(), name1);
X509Certificate cert2 = SecurityUtilities.makeV1Certificate(kp2.getPrivate(), kp2.getPublic(), name2);
keyStore1 = KeyStore.getInstance(KEYSTORE_TYPE, PROVIDER);
keyStore2 = KeyStore.getInstance(KEYSTORE_TYPE, PROVIDER);
trustStore2 = KeyStore.getInstance(KEYSTORE_TYPE, PROVIDER);
keyStore1.load(null, null);
keyStore1.setKeyEntry(name1, kp1.getPrivate(), "relaypass".toCharArray(), new X509Certificate[]{cert1});
// keyStore2.load(null, null);
// keyStore2.setKeyEntry(name2, kp2.getPrivate(), null, new X509Certificate[]{cert2});
trustStore2.load(null, null);
trustStore2.setCertificateEntry(name2, cert1);
// secureSocketManager = new SecureSocketManager(keyStore1, password);
}
#Before
public void init() throws IOException, GeneralSecurityException, InterruptedException, ExecutionException {
SSLServerSocket sslServerSocket = getSSLServerSocket();
SSLSocketFactory sslSocketFactory = getSSLSocketFactory();
ExecutorService pool = Executors.newFixedThreadPool(2);
Callable<SSLSocket> c1 = () -> {
return (SSLSocket) sslServerSocket.accept();
};
Callable<SSLSocket> c2 = () -> {
return (SSLSocket) sslSocketFactory.createSocket("localhost", 2048);
};
Future<SSLSocket> server = pool.submit(c1);
Thread.sleep(1000);
Future<SSLSocket> client = pool.submit(c2);
Thread.sleep(1000);
serverSocket = server.get();
clientSocket = client.get();
}
#After
public void tearDown(){
serverSocket = null;
clientSocket = null;
}
#org.junit.Test
public void endSession(){
Thread test = new Thread(new ReceiverClientThread(serverSocket));
test.start();
try (ObjectOutputStream output = new ObjectOutputStream(new BufferedOutputStream(clientSocket.getOutputStream()))) {
System.out.println("here");
}catch (IOException e){
fail();
}
}
private SSLServerSocket getSSLServerSocket() throws GeneralSecurityException, IOException {
char[] entryPassword = "relaypass".toCharArray();
// COULD ADD PROVIDER IN THESE FOR CONSISTENCY
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
keyManagerFactory.init(keyStore1, entryPassword);
// specify TLS version e.g. TLSv1.3
SSLContext sslContext = SSLContext.getInstance(TLS_VERSION, "BCJSSE");
sslContext.init(keyManagerFactory.getKeyManagers(),null, null);
SSLServerSocketFactory fact = sslContext.getServerSocketFactory();
return (SSLServerSocket) fact.createServerSocket(2048 );
}
private SSLSocketFactory getSSLSocketFactory() throws GeneralSecurityException{
char[] entryPassword = "relaypass".toCharArray();
// COULD ADD PROVIDER IN THESE FOR CONSISTENCY
// KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KEY_MANAGER, "BCJSSE");
// keyManagerFactory.init(keyStore1, entryPassword);
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance("PKIX", "BCJSSE");
trustManagerFactory.init(trustStore2);
// specify TLS version e.g. TLSv1.3
SSLContext sslContext = SSLContext.getInstance(TLS_VERSION, "BCJSSE");
sslContext.init(null,trustManagerFactory.getTrustManagers(), null);
return sslContext.getSocketFactory();
}
This is the class which it is testing and the only relevant line, the one where the code hangs is commented as such:
public class ReceiverClientThread implements Runnable {
private final SSLSocket sslSocket;
public ReceiverClientThread(SSLSocket sslSocket) {
this.sslSocket = sslSocket;
}
public void run() {
try (ObjectInputStream input = new ObjectInputStream(new BufferedInputStream(sslSocket.getInputStream()))) {
System.out.println("here");
} catch (IOException e) {
}
}
}
Thanks

You could set a timeout on your sslSocket, so that if it hangs during stream read, it will only hang for a set period of time and then will throw an exception. This way the thread will not just hang indefinitely.
sslSocket.setSoTimeout(120000); // timeout of 2 min

Spring Webflux: Webclient receive data null when call API Http redirected to Https

I configure Https server Netty for spring 5:
#Component
public class NettyWebServerFactorySslCustomizer implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
#Override
public void customize(NettyReactiveWebServerFactory serverFactory) {
Ssl ssl = new Ssl();
ssl.setEnabled(true);
ssl.setKeyStore("classpath:sample.jks");
ssl.setKeyAlias("alias");
ssl.setKeyPassword("password");
ssl.setKeyStorePassword("password");
Http2 http2 = new Http2();
http2.setEnabled(false);
serverFactory.addServerCustomizers(new SslServerCustomizer(ssl, http2, null));
serverFactory.setPort(8888);
}
}
Redirect http to https:
#Configuration
public class HttpToHttpsRedirectConfig {
#PostConstruct
public void startRedirectServer() {
NettyReactiveWebServerFactory httpNettyReactiveWebServerFactory = new NettyReactiveWebServerFactory(8080);
httpNettyReactiveWebServerFactory.getWebServer((request, response) -> {
URI uri = request.getURI();
URI httpsUri;
try {
httpsUri = new URI("https", uri.getUserInfo(), uri.getHost(), 8888, uri.getPath(), uri.getQuery(), uri.getFragment());
} catch (URISyntaxException e) {
return Mono.error(e);
}
response.setStatusCode(HttpStatus.MOVED_PERMANENTLY);
response.getHeaders().setLocation(httpsUri);
return response.setComplete();
}).start();
}
}
When I try access api using Browser:
https://localhost:8888/api -> Receive Data
http://localhost:8080/api -> Redirect: https://localhost:8888/api -> Receive Data
But I using Webclient:
#Bean
public Webclient getWebclient() {
try {
KeyStore ks = KeyStore.getInstance("JKS");
try (InputStream is = getClass().getResourceAsStream("/sample.jks")) {
ks.load(is, "password".toCharArray());
}
X509Certificate[] trusted = Collections.list(ks.aliases()).stream().map(alias -> {
try {
return (X509Certificate) ks.getCertificate(alias);
} catch (KeyStoreException e) {
throw new RuntimeException(e);
}
}).toArray(X509Certificate[]::new);
SslContext sslContext = SslContextBuilder.forClient().trustManager(trusted).build();
HttpClient httpClient = httpclient.create().secure(t -> t.sslContext(sslContext));
return WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient));
} catch (Exception e) {
e.prinStackTrace();
return null;
}
}
When I call Api:
https://localhost:8888/api -> Receive Data
http://localhost:8080/api -> Null
Can you help me?

How to authenticate with user and password using Custom JMX server using TLS and JMXMP

I am using the following code to create a custom JMX server with TLS and JMXMP following the Oracle documentation. It works well and I can connect to the server with no problem, however I would like to add "USER" and "PASSWORD" to the authentication, however specifying the "password.properties" and the "access.properties" is not working, JMX seems to be ignoring these two options. Can somebody shed some light on the correct way to configure USER and PASSWORD and correct this problem? Thanks
private JMXServiceURL url() {
final String url = String.format( "service:jmx:jmxmp://%s:%s", host(), port() );
try {
return new JMXServiceURL( url );
} catch( Throwable exception ) {
throw new RuntimeException( String.format( "Failed to create JMX Service URL: %s", url ), exception );
}
}
private Map<String, Object> env() {
final Map<String, Object> env = new LinkedHashMap<String, Object>();
try {
String keystore = "jmx.keystore";
char keystorepass[] = "12345678".toCharArray();
char keypassword[] = "12345678".toCharArray();
KeyStore ks = KeyStore.getInstance("JKS");
ks.load(new FileInputStream(keystore), keystorepass);
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(ks, keypassword);
SSLContext ctx = SSLContext.getInstance("TLSv1");
ctx.init(kmf.getKeyManagers(), null, null);
SSLSocketFactory ssf = ctx.getSocketFactory();
env.put("jmx.remote.profiles", "TLS");
env.put("jmx.remote.tls.socket.factory", ssf);
env.put("jmx.remote.tls.enabled.protocols", "TLSv1");
env.put("jmx.remote.tls.enabled.cipher.suites","SSL_RSA_WITH_NULL_MD5");
env.put("jmx.remote.x.password.file", "password.properties");
env.put("jmx.remote.x.access.file","access.properties");
} catch (Exception e) {
e.printStackTrace();
}
return env;
}
private MBeanServer server() {
return ManagementFactory.getPlatformMBeanServer();
}
private JMXConnectorServer connector() {
try {
ServerProvider.class.getName();
return JMXConnectorServerFactory.newJMXConnectorServer( url(), env(), server() );
}catch( Throwable exception ) {
throw new RuntimeException( "Failed to create JMX connector server factory", exception );
}
}

I was finally able to configure the additional user and password for the JMXMP connection with the following code from the Oracle documentation
MBeanServer mbs = MBeanServerFactory.createMBeanServer();
Security.addProvider(new com.sun.jdmk.security.sasl.Provider());
HashMap env = new HashMap();
env.put("jmx.remote.profiles", "TLS SASL/PLAIN");
env.put("jmx.remote.sasl.callback.handler",
new PropertiesFileCallbackHandler("password.properties"));
env.put("jmx.remote.x.access.file",access.properties");
JMXServiceURL url = new JMXServiceURL("jmxmp", null, 5555);
JMXConnectorServer cs =
JMXConnectorServerFactory.newJMXConnectorServer(url,
env,
mbs);
cs.start();
I implemented a simple callBackHandler for the password validation
public final class PropertiesFileCallbackHandler
implements CallbackHandler {
private Properties pwDb;
/**
* Contents of files are in the Properties file format.
*
* #param pwFile name of file containing name/password
*/
public PropertiesFileCallbackHandler(String pwFile) throws IOException {
if (pwFile != null) {
File file = new File(pwFile);
if(file.exists()) {
pwDb = new Properties();
pwDb.load(new FileInputStream(file));
} else {
throw new IOException("File " + pwFile + " not found");
}
}
}
public void handle(Callback[] callbacks)
throws UnsupportedCallbackException {
AuthorizeCallback acb = null;
AuthenticateCallback aucb = null;
for (int i = 0; i < callbacks.length; i++) {
if (callbacks[i] instanceof AuthorizeCallback) {
acb = (AuthorizeCallback) callbacks[i];
} else if (callbacks[i] instanceof AuthenticateCallback) {
aucb = (AuthenticateCallback)callbacks[i];
} else {
throw new UnsupportedCallbackException(callbacks[i]);
}
}
// Process retrieval of password; can get password if
// username is available
if (aucb != null) {
String username = aucb.getAuthenticationID();
String password = new String(aucb.getPassword());
String pw = pwDb.getProperty(username);
if (pw != null) {
if(pw.equals(password)){
aucb.setAuthenticated(true);
}
}
}
// Check for authorization
if (acb != null) {
String authid = acb.getAuthenticationID();
String authzid = acb.getAuthorizationID();
if (authid.equals(authzid)) {
// Self is always authorized
acb.setAuthorized(true);
}
}
}
}

HTTPS POST using HttpClient API opening 2 sockets

After a lot of R&D and googling, not able to troubleshoot my problem.
Environment Setup
Web Server (Tomcat 6.0.20) --> Proxy Server (Windows Server 2007) --> Thirdy part host
We have application, which does online payment transaction, after completion of this transaction, we want to send status of transaction to third party server. So posting data to third part server from our web server is opening 2 sockets for one transaction at proxy server, but when we check at web server it has created only one socket. SO why 2 socket at proxy server.
Below is my sample code
import javax.net.ssl.*;
import javax.net.SocketFactory;
import java.net.*;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.IOException;
import java.security.MessageDigest;
import java.util.Hashtable;
import java.math.BigInteger;
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.protocol.*;
public class HTTPPostDemo {
private String privateKey;
private String host;
private int port;
private String userName;
private Header[] headers = null;
public class MySSLSocketFactory implements SecureProtocolSocketFactory {
private TrustManager[] getTrustManager() {
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
}
};
return trustAllCerts;
}
public Socket createSocket(String host, int port) throws IOException, UnknownHostException {
TrustManager[] trustAllCerts = getTrustManager();
try {
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
SocketFactory socketFactory = HttpsURLConnection.getDefaultSSLSocketFactory();
return socketFactory.createSocket(host, port);
} catch (Exception ex) {
throw new UnknownHostException("Problems to connect " + host + ex.toString());
}
}
public Socket createSocket(Socket socket, String host, int port, boolean flag) throws IOException, UnknownHostException {
TrustManager[] trustAllCerts = getTrustManager();
try {
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
SocketFactory socketFactory = HttpsURLConnection.getDefaultSSLSocketFactory();
return socketFactory.createSocket(host, port);
} catch (Exception ex) {
throw new UnknownHostException("Problems to connect " + host + ex.toString());
}
}
public Socket createSocket(String host, int port, InetAddress clientHost, int clientPort) throws IOException, UnknownHostException {
TrustManager[] trustAllCerts = getTrustManager();
try {
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
SocketFactory socketFactory = HttpsURLConnection.getDefaultSSLSocketFactory();
return socketFactory.createSocket(host, port, clientHost, clientPort);
} catch (Exception ex) {
throw new UnknownHostException("Problems to connect " + host + ex.toString());
}
}
}
public SslClient(String host, int port, String userName, String privateKey) {
this.host = host;
this.port = port;
this.userName = userName;
this.privateKey = privateKey;
}
protected String md5Sum(String str) {
String sum = new String();
try {
MessageDigest md5 = MessageDigest.getInstance("MD5");
sum = String.format("%032x", new BigInteger(1, md5.digest(str.getBytes())));
} catch (Exception ex) {
}
return sum;
}
public String getSignature(String xml) {
return md5Sum(md5Sum(xml + privateKey) + privateKey);
}
public String sendRequest(String xml) throws Exception {
HttpClient client = new HttpClient();
client.setConnectionTimeout(60000);
client.setTimeout(60000);
String response = new String();
String portStr = String.valueOf(port);
Protocol.registerProtocol("https", new Protocol("https", new MySSLSocketFactory(), port));
String signature = getSignature(xml);
String uri = "https://" + host + ":" + portStr + "/";
PostMethod postRequest = new PostMethod(uri);
postRequest.addRequestHeader("Content-Length", String.valueOf(xml.length()));
postRequest.addRequestHeader("Content-Type", "text/xml");
postRequest.addRequestHeader("X-Signature", signature);
postRequest.addRequestHeader("X-Username", userName);
postRequest.setRequestBody(xml);
System.out.println("Sending https request....." + postRequest.toString());
try {
client.executeMethod(postRequest);
} catch (Exception ex) {
throw new TaskExecuteException("Sending post got exception ", ex);
}
response = postRequest.getResponseBodyAsString();
headers = postRequest.getRequestHeaders();
return response;
}
public String getPrivateKey() {
return privateKey;
}
public void setPrivateKey(String privateKey) {
this.privateKey = privateKey;
}
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public int getPort() {
return port;
}
public void setPort(int port) {
this.port = port;
}
public String getUserName() {
return userName;
}
public void setUserName(String userName) {
this.userName = userName;
}
public Header[] getHeaders() {
return headers;
}
public void setHeaders(Header[] headers) {
this.headers = headers;
}
public static void main(String[] args) {
String privateKey = "your_private_key";
String userName = "your_user_name";
String host = "demo.site.net";
int port = 55443;
String xml =
"<?xml version='1.0' encoding='UTF-8' standalone='no' ?>"
+ "<!DOCTYPE OPS_envelope SYSTEM 'ops.dtd'>"
+ "<OPS_envelope>"
+ "<header>"
+ "<version>0.9</version>"
+ "<msg_id>2.21765911726198</msg_id>"
+ "<msg_type>standard</msg_type>"
+ "</header>"
+ "<body>"
+ "<data_block>"
+ "<dt_assoc>"
+ "<item key='attributes'>"
+ "<dt_assoc>"
+ "<item key='domain'>test-1061911771844.com</item>"
+ "<item key='pre-reg'>0</item>"
+ "</dt_assoc>"
+ "</item>"
+ "<item key='object'>DOMAIN</item>"
+ "<item key='action'>LOOKUP</item>"
+ "<item key='protocol'>XCP</item>"
+ "</dt_assoc>"
+ "</data_block>"
+ "</body>"
+ "</OPS_envelope>";
SslClient sslclient = new SslClient(host, port, userName, privateKey);
try {
String response = sslclient.sendRequest(xml);
System.out.println("\nResponse is:\n" + response);
} catch (Exception e) {
e.printStackTrace();
}
}
}
As in a day we are processing 10,000 + transactions, so number of socket at proxy are getting increased, so after 2-3 days, we need to do hard reboot of web server to free all the open sockets with proxy server.
Does HTTPClient, opens one socket for SSL Handshake and another for actual data post ? I don't think so. Then it should be at Web server and not at Proxy Server
For checking sockets and open ports at web server we are using netstat command.
For checking sockets and open ports at proxy server we are using proxy tool

when we check at web server it has created only one socket.
Because there is only one inbound connection to it.
SO why 2 socket at proxy server.
Because you are connecting to two different servers via the proxy server?
number of socket at proxy are getting increased, so after 2-3 days, we need to do hard reboot of web server to free all the open sockets with proxy server.
That doesn't make sense. It's the proxy server that has the dual connections, not the web server. You said that above. If the web server is running out of sockets, somebody isn't closing their connections correctly: the client, the proxy server, or the web server. Possibly your socket factory needs to override equals() and maybe hashCode() too, to enable whatever connection pooling HttpClient may do, I'm not an expert on that.
BUT your TrustManager is radically insecure. If you have this deployed in production, you have already committed a major security breach. This is currently a much bigger problem that running out of sockets every few days.

When socket ports run out, transaction timeouts occur. The solution to this problem is to tune the TIMEWAIT-related Windows registry parameters:
TcpTimedWaitDelay
MaxUserPort
StrictTimeWaitSeqCheck
The TIMEWAIT-related Windows registry parameters control how long a socket port remains unavailable after it is closed and how many ports are available for use.
By setting these windows registry parameters, I have solved this problem, but don't know, weather it is a correct solution to implement or not.