I am having issues running multiple SSH jobs (using JSch) with ExecutorService.
When running a single job, they execute as expected. However, when I attempt to run multiple jobs, only the first job executes fully. I have changed the thread pool to use only 1 thread, thinking that somehow its a port issue but it still doesn't work correctly. Any help will be appreciated.
ExecutorService ex = Executors.newFixedThreadPool(4);
Set<Future<Task>> set = new HashSet<Future<Task>>();
for (int i = 0; i < tasks.size(); i++) {
Task task = tasks.get(i);
creds.setHostName(task.getFirewall().getAddress());
System.out.println("Task(" + i + ")=" + task);
Callable<Task> worker = new SSH.SSHWorker(creds, task, i);
ex.submit(worker);
}
Here is the SSH
public class SSH {
LoginCredentials creds;
//ChannelExec channelExec;
ChannelExec channelExec;
BufferedReader consoleOutput;
Session session;
InputStream is;
UI ui;
String output = "";
// String command;
public boolean debug = false;
public SSH(LoginCredentials creds, String command) throws JSchException, IOException {
// System.out.println("NEW SSH");
this.creds = creds;
consoleOutput = new BufferedReader(new InputStreamReader(System.in));
ui = new UI();
this.connect();
Channel channel = session.openChannel("exec");
this.sendCommand(channel, command);
}
public String getOutput() {
return output;
}
public class UI extends MyUserInfo {
String message;
#Override
public void showMessage(String message) {
this.message = message;
super.showMessage(message); //To change body of generated methods, choose Tools | Templates.
}
#Override
public String getMessage() {
return this.message;
}
}
private void sendCommand(Channel channel, String command) throws IOException, JSchException {
this.channelExec = (ChannelExec) channel;
this.channelExec.setCommand(command);
//channel.setInputStream(null);
channel.setOutputStream(System.out);
this.is = channel.getInputStream();
channel.connect();
byte[] buffer = new byte[1024];
while (channel.getExitStatus() == -1) {
//System.out.println("Here 1.1");
while (is.available() > 0) {
//System.out.println("Here 1.2");
int i = is.read(buffer, 0, 1024);
//System.out.println("i= " + i);
if (i < 0) {
System.out.println("breaking");
break;
}
String string = new String(buffer, 0, i);
output = output.concat(string);
//System.out.println("output= "+output);
if (string.contains("Command authorization failed")) {
channel.disconnect();
break;
}
}
if (channel.isClosed()) {
System.out.println("exit-status: " + channel.getExitStatus());
break;
}
}
is.close();
channel.disconnect();
this.session.disconnect();
}
private void connect() throws JSchException {
JSch jsch = new JSch();
session = jsch.getSession(creds.getUser(), creds.getHostName(), 22);
session.setTimeout(0);
session.setPassword(creds.getPassword());
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.setUserInfo(ui);
session.connect();
System.out.println("Connected in SSH");
}
public static class SSHWorker implements Callable {
private LoginCredentials lc;
private Task t;
private int thread;
public SSHWorker(LoginCredentials creds, Task task, int thread) {
this.t = task;
lc = creds;
this.thread = thread;
System.out.println("Creds= " + lc.getHostName() + " Task= " + t + " Thread-" + thread);
}
#Override
public Task call() throws JSchException, IOException {
System.out.println("Doing Call For: Creds= " + lc.getHostName() + " Task= " + t + " Thread-" + thread);
String enablepassword1 = (String) lc.getEnablePassword().get(0);
SSH ssh = new SSH(lc, t.getSSHCommand(enablepassword1));
this.t.setTaskResult(ssh.getOutput());
return t;
}
}
}
The output is Here (IP addresses have been changed)
In your loop that creates the tasks, you use only one instance of LoginCredentials. So all tasks share the instance. In the loop, you overwrite the hostname in each iteration. So in the end the LoginCredentials refers to the hostname of the last task. So all tasks connect to that host.
Just to do not pass the hostname via LoginCredentials and use task.getFirewall().getAddress() directly when creating the JSch session:
SSH ssh = new SSH(lc, t.getSSHCommand(enablepassword1), task.getFirewall().getAddress());
...
public SSH(LoginCredentials creds, String command, String hostname)
{
...
this.connect(hostname);
}
...
private void connect(String hostname) throws JSchException {
JSch jsch = new JSch();
session = jsch.getSession(creds.getUser(), hostname, 22);
...
}
Related
How do I discover the Queue depth, and other metrics about queues, from IBM MQ using java with standard IBM MQ libraries?
Try out the below sample code snippet for fetching the queue depth.
String mqQMgr = "";
String mqQueue = "";
MQEnvironment.hostname = "";
MQEnvironment.port = "";
MQEnvironment.channel = "";
MQEnvironment.properties.put(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES);
MQQueueManager qMgr = new MQQueueManager(mqQMgr);
MQQueue destQueue = qMgr.accessQueue(mqQueue, openOptions);
int depth = destQueue.getCurrentDepth();
destQueue.close();
qMgr.disconnect();
A full code version (Change your parameters accordingly, like Bindings or client mode, options etc ) :
import com.ibm.mq.*;
public class QueueManager {
private final String host;
private final int port;
private final String channel;
private final String manager;
private final MQQueueManager qmgr;
public QueueManager(String host, int port, String channel, String manager) throws MQException {
this.host = host;
this.port = port;
this.channel = channel;
this.manager = manager;
this.qmgr = createQueueManager();
}
private MQQueueManager createQueueManager() throws MQException {
MQEnvironment.channel = channel;
MQEnvironment.port = port;
MQEnvironment.hostname = host;
MQEnvironment.properties.put(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES);
return new MQQueueManager(manager);
}
// This method will return the Queue Depth
public int queueDepth(String queueName) {
int depth = -1;
int openOptions = MQC.MQOO_INQUIRE + MQC.MQOO_FAIL_IF_QUIESCING;
MQQueue queue = null;
try {
queue = qmgr.accessQueue(queueName, openOptions);
depth = queue.getCurrentDepth();
}
catch (MQException e)
{
System.out.println("CC=" +e.completionCode + " : RC=" + e.reasonCode);
}
finally {
try {
if (queue != null)
queue.close();
}
catch (MQException e) {
System.out.println("CC=" +e.completionCode + " : RC=" + e.reasonCode);
}
}
return depth;
}
........ Other Methods .......
}
I write my own Java FTP server. Until recently I used PUttY to debug my control telnet connection and everything seemed fine - I had successful two-way communication. Now I try to debug my server with FileZilla, but it does not seem to read my text, nor to send some to server, so it just hangs and wait for something.
Control connection class
public class ControlConnection extends Thread {
private enum OperationMode {
ACTIVE, PASSIVE
}
private final Map<String, Supplier<String>> COMMANDS;
private String[] userTokens;
private User user;
private String userLogin;
private boolean authenticated;
private boolean dataConnected;
private boolean userExists;
private final Socket socket;
private DataInputStream inputStream;
private DataOutputStream outputStream;
private DataConnection ftpSession;
private OperationMode operationMode;
private String errorMessage;
public ControlConnection(Socket socket) {
super(ControlConnection.class.toString());
this.socket = socket;
// constants initialization
authenticated = false;
dataConnected = false;
// commands initialization
COMMANDS = new HashMap<>();
// commands init
}
#Override
public void run() {
try {
inputStream = new DataInputStream(socket.getInputStream());
outputStream = new DataOutputStream(socket.getOutputStream());
sendGreetings();
IOProcessing.writeBytes(outputStream, pasvCommand());;
boolean running = true;
while (running) {
sendGreetings();
String input = IOProcessing.readBytes(inputStream);
if (!(input.equals("")))
System.out.println(input);
if (!checkInput(input))
continue;
userTokens = input.split(" ");
String command = userTokens[0].toUpperCase();
String answer = COMMANDS.get(command).get();
outputStream.writeBytes(answer);
}
}
catch (IOException e) {
System.err.println(e);
System.exit(-1);
}
}
private boolean commonCheck() {
// some checks
return true;
}
private String getErrorMessage() {
return errorMessage;
}
public void sendGreetings() {
String greetings = String.format("220 Control connection established: %s", getConnectionInfo());
IOProcessing.writeBytes(outputStream, greetings);
}
public String getConnectionInfo() {
String info = String.format("%s: %d %s",
socket.getInetAddress().toString(), socket.getPort(), user != null ? user.getUsername(): "");
return info;
}
// input/output proccessing functions
public boolean checkInput(String input) {
// checks
return true;
}
// commands functions
private String pasvCommand() {
if (operationMode == OperationMode.PASSIVE) {
errorMessage = "Already in passive mode.%n";
return errorMessage;
}
String answer;
new ListenToSocket().start();
answer = String.format("227 Entering Passive Mode (%s, %d)",
"127.0.0.1", DataConnection.PORT);
operationMode = OperationMode.PASSIVE;
return answer;
}
private class ListenToSocket extends Thread {
public ListenToSocket() {
}
#Override
public void run() {
try {
ServerSocket ftpSocket =
new ServerSocket(DataConnection.PORT);
ftpSession =
DataConnection.getDataConnection(ftpSocket.accept());
if (ftpSession != null) {
ftpSession.start();
dataConnected = true;
String greetings = "Data connection established: " + ftpSession.getConnectionInfo();
IOProcessing.writeBytes(outputStream, greetings);
} else {
dataConnected = false;
}
} catch (IOException e) {
System.out.print(e);
}
}
}
also, server does not get user credentials, entered in FileZilla - input from server is always empty
IOProcessing class
public class IOProcessing {
private static final Charset UTF8_CHARSET;
static {
UTF8_CHARSET = Charset.forName("UTF-8");
}
public static String readBytes(DataInputStream inputStream) {
String result = "";
try {
int len = inputStream.available();
if (len == 0) {
return result;
}
byte[] byteInput = new byte[len];
inputStream.readFully(byteInput, 0, len);
result = new String(byteInput, "UTF-8").trim();
} catch (IOException e) {
System.err.println(e);
}
return result;
}
output FileZlla log
Status: Resolving address of localhost
Status: Connecting to [::1]:21...
Status: Connection established, waiting for welcome message.
You didn't show us the writeBytes. So I can only guess that you are not sending \r\n after the messages sent to the client. Particularly after the welcome message. So FileZilla keeps waiting forever for it, as any FTP client would do.
When download file from ftp server via FTPClient class files comes corrupted check for initial file (img1) and downloaded file (img2)
Source:
public class FtpClient extends FTPClient {
private String host;
private int port;
private String username;
private String password;
private boolean connected;
public FtpClient(String host, int port, String username, String password) {
this.host = host;
this.port = port;
this.username = username;
this.password = password;
connected = connect();
}
private void verifyConnection() throws ConnectionException {
if(!connected){
ConnectionException ex = new ConnectionException();
log.error(ex.getMessage());
throw ex;
}
}
private boolean connect(){
try {
//try to connect
connect(host, port);
if(!login(username, password)){
logout();
return false;
}
int reply = getReplyCode();
if(!FTPReply.isPositiveCompletion(reply)){
disconnect();
return false;
}
enterRemotePassiveMode();
setFileTransferMode(BINARY_FILE_TYPE);
setFileType(BINARY_FILE_TYPE);
log.debug("Remote system is " + getSystemType());
log.debug("Current directory is " + printWorkingDirectory());
return true;
} catch (IOException e) {
log.error(e.getMessage());
e.printStackTrace();
return false;
}
}
public void get(String batchName, String fileName) throws IOException, ConnectionException {
verifyConnection();
File file = new File("/tmp/" + batchName);
if (!file.exists()){
boolean mkdir = file.mkdir();
log.info("Create batch directory '{}' result: {}",batchName, mkdir);
}
OutputStream outputStream = new FileOutputStream("/tmp/" + batchName + "/" + fileName);
boolean result = retrieveFile(batchName + "/" + fileName, outputStream);
log.debug("Retrieve file {}, result: {}", batchName + "/" + fileName, result);
outputStream.close();
}
}
img1
img2
What is going wrong?
You are using an incorrect CONSTANT for FTPClient.setFileTransferMode.
setFileTransferMode(BINARY_FILE_TYPE);
Like the doc says :
mode - The new transfer mode to use (one of the FTP class _TRANSFER_MODE constants).
You can use one of those :
FTP.BLOCK_TRANSFER_MODE
FTP.COMPRESSED_TRANSFER_MODE
FTP.STREAM_TRANSFER_MODE
This will probably correct your problem of transfer, since your are using a value that could be anything ...
You are passing a value of 2 where it is expecting one of [10|11|12]. So It probably keep the default STREAM_TRANSFER_MODE value.
In case of file upload fails then how to do multiple attempts while doing SFTP by using JSCH API?
How to ensure file is uploaded successfully?
How to create thread-safe file upload utility?
Create a common static utility method which can be invoked from the external class. This method has a map argument to persist the values of sFTPUser, sFTPHost, sFTPPort, sFTPPwd, destinationLocation and uploadedFileName :
public static void doSFTP(Map<String, String> ftpParameters) {
if (ftpParameters.get("ID_NAME").equals(
NAPSCommonConstants.MFT_NAPSGPCS_INTF)) {
// do sftp for NAPS GPCS Interface.
uploadUsingSFTP(ftpParameters);
}
}
Use synchronized method to ensure thread safety:
private static synchronized void uploadUsingSFTP(
Map<String, String> ftpPrameterList) {
new SFTPUtility().uploadFileMFT(ftpPrameterList.get("sFTPUser"),
ftpPrameterList.get("sFTPHost"), new Integer(ftpPrameterList
.get("sFTPPort")), ftpPrameterList.get("sFTPPwd"),
ftpPrameterList.get("sourceLocation"), ftpPrameterList
.get("destinationLocation"), ftpPrameterList
.get("uploadedFileName"));
}
Responsible method to upload files using SFTP with 5 attempts:
private void uploadFileMFT(String sFTPUser, String sFTPHost, int sFTPPort,
String sFTPPwd, String sourceLocation, String destinationLocation,
String uploadedFileName) {
LOG.info("Inside uploadFileMFT to upload and verify the file.");
JSch jsch = new JSch();
Vector<String> fileList = null;
/** 5 re-attempt logic to get session */
int attempts = 0;
boolean successfulConnect;
do {
try {
successfulConnect = true;
session = jsch.getSession(sFTPUser, sFTPHost, sFTPPort);
LOG.debug("session connected ...");
session.setPassword(sFTPPwd);
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect();
LOG.debug("Sftp Session connected ...");
channel = session.openChannel("sftp");
LOG.debug("Sftp Channel opened ...");
channelSftp = (ChannelSftp) channel;
channelSftp.connect();
LOG.info(" Sftp channel opened and connected ...");
channelSftp.put(sourceLocation, destinationLocation);
fileList = channelSftp.ls(destinationLocation);
} catch (JSchException e) {
++attempts;
successfulConnect = false;
LOG.error(e);
} catch (SftpException e) {
++attempts;
successfulConnect = false;
LOG.error(e);
} finally {
if (null != channelSftp) {
channelSftp.exit();
LOG.debug(" sftp Channel exited.");
}
if (null != channel) {
channel.disconnect();
LOG.debug(" Channel disconnected.");
}
if (null != session) {
session.disconnect();
LOG.debug(" Host Session disconnected.");
}
}
} while (attempts < 5 && successfulConnect == false);
fileUploadValidation(fileList, uploadedFileName);
LOG.info("Exiting from method - uploadFileMFT ...");
}
Finally uploaded file can be validated:
private void fileUploadValidation (Vector<String> fileList, String uploadedFileName){
boolean isFileExistis = false;
Object[] objArr = fileList.toArray();
for (int i = 0; i < objArr.length; i++) {
String fileName = objArr[i].toString();
isFileExistis = fileName.contains(uploadedFileName);
if (isFileExistis) {
LOG.info("Uploaded file '" + uploadedFileName + "' was transferred successfull ...");
break;
}else if(i >= objArr.length){
LOG.info("Uploaded file '" + uploadedFileName + "' was failed to transfer ...");
}
}
}
I want to get a file from unix system to my local system which is on windows using java. I'm very much new to this concept. Any ideas on how it could be done? Which is the best java API for this task?
If the Unix machine supports SFTP, JSch is an option. You could adapt the following code to meet your needs:
private static final String USER_PROMPT = "Enter username#hostname:port";
private static final boolean USE_GUI = true;
public static void main(final String[] arg) {
Session session = null;
ChannelSftp channelSftp = null;
try {
final JSch jsch = new JSch();
final String defaultInput = System.getProperty("user.name") + "#localhost:22";
String input = (USE_GUI) ? JOptionPane.showInputDialog(USER_PROMPT, defaultInput) : System.console().readLine("%s (%s): ", USER_PROMPT, defaultInput);
if (input == null || input.trim().length() == 0) {
input = defaultInput;
}
final int indexOfAt = input.indexOf('#');
final int indexOfColon = input.indexOf(':');
final String user = input.substring(0, indexOfAt);
final String host = input.substring(indexOfAt + 1, indexOfColon);
final int port = Integer.parseInt(input.substring(indexOfColon + 1));
jsch.setKnownHosts("/path/to/known_hosts");
// if you have set up authorized_keys on the server, using that identitiy
// with the code on the next line allows for password-free, trusted connections
// jsch.addIdentity("/path/to/id_rsa", "id_rsa_password");
session = jsch.getSession(user, host, 22);
final UserInfo ui = new MyUserInfo();
session.setUserInfo(ui);
session.connect();
channelSftp = (ChannelSftp) session.openChannel("sftp");
channelSftp.connect();
channelSftp.get("/remotepath/remotefile.txt", "/localpath/localfile.txt");
} finally {
if (channelSftp != null) {
channelSftp.exit();
}
if (session != null) {
session.disconnect();
}
}
}
public static class MyUserInfo implements UserInfo {
private String password;
#Override
public String getPassword() {
return password;
}
#Override
public boolean promptYesNo(final String str) {
final Object[] options = {"yes", "no"};
final boolean yesNo = (USE_GUI) ? JOptionPane.showOptionDialog(null, str, "Warning", JOptionPane.DEFAULT_OPTION, JOptionPane.WARNING_MESSAGE, null, options, options[0]) == 0 : System.console().readLine("Enter y or n: ").equals("y");
return yesNo;
}
#Override
public String getPassphrase() {
return null;
}
#Override
public boolean promptPassphrase(final String message) {
return true;
}
#Override
public boolean promptPassword(final String message) {
if (!USE_GUI) {
password = new String(System.console().readPassword("Password: "));
return true;
} else {
final JTextField passwordField = new JPasswordField(20);
final Object[] ob = {passwordField};
final int result = JOptionPane.showConfirmDialog(null, ob, message, JOptionPane.OK_CANCEL_OPTION);
if (result == JOptionPane.OK_OPTION) {
password = passwordField.getText();
return true;
} else {
return false;
}
}
}
#Override
public void showMessage(final String message) {
if (!USE_GUI) {
System.console().printf(message);
} else {
JOptionPane.showMessageDialog(null, message);
}
}
}
I have found JSch to be very useful and straight foreword. Below is a snippet of code written to put all .txt files in a given folder on the sftp server.
public static void sftpConnection() {
// Object Declaration.
JSch jsch = new JSch();
Session session = null;
Channel channel = null;
// Variable Declaration.
String user = "foo";
String host = "10.9.8.7";
Integer port = 22;
String password = "test123";
String watchFolder = "\\localhost\textfiles";
String outputDir = "/remote/textFolder/";
String filemask = "*.txt";
try {
session = jsch.getSession(user, host, port);
/*
* StrictHostKeyChecking Indicates what to do if the server's host
* key changed or the server is unknown. One of yes (refuse connection),
* ask (ask the user whether to add/change the key) and no
* (always insert the new key).
*/
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword(password);
session.connect();
channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp)channel;
// Go through watch folder looking for files.
File[] files = findFile(watchFolder, filemask);
for(File file : files) {
// Upload file.
putFile(file, sftpChannel, outputDir);
}
} finally {
sftpChannel.exit();
session.disconnect();
}
}
public static void putFile(File file, ChannelSftp sftpChannel, String outputDir) {
FileInputStream fis = null;
try {
// Change to output directory.
sftpChannel.cd(outputDir);
// Upload file.
fis = new FileInputStream(file);
sftpChannel.put(fis, file.getName());
fis.close();
} catch{}
}
public static File[] findFile(String dirName, final String mask) {
File dir = new File(dirName);
return dir.listFiles(new FilenameFilter() {
public boolean accept(File dir, String filename)
{ return filename.endsWith(mask); }
} );
}
First thing that goes into my mind is FTP.
There are multiple choices to do that. First one simple socket communication between a java client and a server. If you want to go with this approach then follow this:
http://mrbool.com/file-transfer-between-2-computers-with-java/24516
Then there are other high level protocols implementations that can be used such as FTP, HTTP, etc
Follow a related SO post for java FTP client server: FTP client server model for file transfer in Java