In case of file upload fails then how to do multiple attempts while doing SFTP by using JSCH API?
How to ensure file is uploaded successfully?
How to create thread-safe file upload utility?
Create a common static utility method which can be invoked from the external class. This method has a map argument to persist the values of sFTPUser, sFTPHost, sFTPPort, sFTPPwd, destinationLocation and uploadedFileName :
public static void doSFTP(Map<String, String> ftpParameters) {
if (ftpParameters.get("ID_NAME").equals(
NAPSCommonConstants.MFT_NAPSGPCS_INTF)) {
// do sftp for NAPS GPCS Interface.
uploadUsingSFTP(ftpParameters);
}
}
Use synchronized method to ensure thread safety:
private static synchronized void uploadUsingSFTP(
Map<String, String> ftpPrameterList) {
new SFTPUtility().uploadFileMFT(ftpPrameterList.get("sFTPUser"),
ftpPrameterList.get("sFTPHost"), new Integer(ftpPrameterList
.get("sFTPPort")), ftpPrameterList.get("sFTPPwd"),
ftpPrameterList.get("sourceLocation"), ftpPrameterList
.get("destinationLocation"), ftpPrameterList
.get("uploadedFileName"));
}
Responsible method to upload files using SFTP with 5 attempts:
private void uploadFileMFT(String sFTPUser, String sFTPHost, int sFTPPort,
String sFTPPwd, String sourceLocation, String destinationLocation,
String uploadedFileName) {
LOG.info("Inside uploadFileMFT to upload and verify the file.");
JSch jsch = new JSch();
Vector<String> fileList = null;
/** 5 re-attempt logic to get session */
int attempts = 0;
boolean successfulConnect;
do {
try {
successfulConnect = true;
session = jsch.getSession(sFTPUser, sFTPHost, sFTPPort);
LOG.debug("session connected ...");
session.setPassword(sFTPPwd);
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect();
LOG.debug("Sftp Session connected ...");
channel = session.openChannel("sftp");
LOG.debug("Sftp Channel opened ...");
channelSftp = (ChannelSftp) channel;
channelSftp.connect();
LOG.info(" Sftp channel opened and connected ...");
channelSftp.put(sourceLocation, destinationLocation);
fileList = channelSftp.ls(destinationLocation);
} catch (JSchException e) {
++attempts;
successfulConnect = false;
LOG.error(e);
} catch (SftpException e) {
++attempts;
successfulConnect = false;
LOG.error(e);
} finally {
if (null != channelSftp) {
channelSftp.exit();
LOG.debug(" sftp Channel exited.");
}
if (null != channel) {
channel.disconnect();
LOG.debug(" Channel disconnected.");
}
if (null != session) {
session.disconnect();
LOG.debug(" Host Session disconnected.");
}
}
} while (attempts < 5 && successfulConnect == false);
fileUploadValidation(fileList, uploadedFileName);
LOG.info("Exiting from method - uploadFileMFT ...");
}
Finally uploaded file can be validated:
private void fileUploadValidation (Vector<String> fileList, String uploadedFileName){
boolean isFileExistis = false;
Object[] objArr = fileList.toArray();
for (int i = 0; i < objArr.length; i++) {
String fileName = objArr[i].toString();
isFileExistis = fileName.contains(uploadedFileName);
if (isFileExistis) {
LOG.info("Uploaded file '" + uploadedFileName + "' was transferred successfull ...");
break;
}else if(i >= objArr.length){
LOG.info("Uploaded file '" + uploadedFileName + "' was failed to transfer ...");
}
}
}
Related
I am having issues running multiple SSH jobs (using JSch) with ExecutorService.
When running a single job, they execute as expected. However, when I attempt to run multiple jobs, only the first job executes fully. I have changed the thread pool to use only 1 thread, thinking that somehow its a port issue but it still doesn't work correctly. Any help will be appreciated.
ExecutorService ex = Executors.newFixedThreadPool(4);
Set<Future<Task>> set = new HashSet<Future<Task>>();
for (int i = 0; i < tasks.size(); i++) {
Task task = tasks.get(i);
creds.setHostName(task.getFirewall().getAddress());
System.out.println("Task(" + i + ")=" + task);
Callable<Task> worker = new SSH.SSHWorker(creds, task, i);
ex.submit(worker);
}
Here is the SSH
public class SSH {
LoginCredentials creds;
//ChannelExec channelExec;
ChannelExec channelExec;
BufferedReader consoleOutput;
Session session;
InputStream is;
UI ui;
String output = "";
// String command;
public boolean debug = false;
public SSH(LoginCredentials creds, String command) throws JSchException, IOException {
// System.out.println("NEW SSH");
this.creds = creds;
consoleOutput = new BufferedReader(new InputStreamReader(System.in));
ui = new UI();
this.connect();
Channel channel = session.openChannel("exec");
this.sendCommand(channel, command);
}
public String getOutput() {
return output;
}
public class UI extends MyUserInfo {
String message;
#Override
public void showMessage(String message) {
this.message = message;
super.showMessage(message); //To change body of generated methods, choose Tools | Templates.
}
#Override
public String getMessage() {
return this.message;
}
}
private void sendCommand(Channel channel, String command) throws IOException, JSchException {
this.channelExec = (ChannelExec) channel;
this.channelExec.setCommand(command);
//channel.setInputStream(null);
channel.setOutputStream(System.out);
this.is = channel.getInputStream();
channel.connect();
byte[] buffer = new byte[1024];
while (channel.getExitStatus() == -1) {
//System.out.println("Here 1.1");
while (is.available() > 0) {
//System.out.println("Here 1.2");
int i = is.read(buffer, 0, 1024);
//System.out.println("i= " + i);
if (i < 0) {
System.out.println("breaking");
break;
}
String string = new String(buffer, 0, i);
output = output.concat(string);
//System.out.println("output= "+output);
if (string.contains("Command authorization failed")) {
channel.disconnect();
break;
}
}
if (channel.isClosed()) {
System.out.println("exit-status: " + channel.getExitStatus());
break;
}
}
is.close();
channel.disconnect();
this.session.disconnect();
}
private void connect() throws JSchException {
JSch jsch = new JSch();
session = jsch.getSession(creds.getUser(), creds.getHostName(), 22);
session.setTimeout(0);
session.setPassword(creds.getPassword());
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.setUserInfo(ui);
session.connect();
System.out.println("Connected in SSH");
}
public static class SSHWorker implements Callable {
private LoginCredentials lc;
private Task t;
private int thread;
public SSHWorker(LoginCredentials creds, Task task, int thread) {
this.t = task;
lc = creds;
this.thread = thread;
System.out.println("Creds= " + lc.getHostName() + " Task= " + t + " Thread-" + thread);
}
#Override
public Task call() throws JSchException, IOException {
System.out.println("Doing Call For: Creds= " + lc.getHostName() + " Task= " + t + " Thread-" + thread);
String enablepassword1 = (String) lc.getEnablePassword().get(0);
SSH ssh = new SSH(lc, t.getSSHCommand(enablepassword1));
this.t.setTaskResult(ssh.getOutput());
return t;
}
}
}
The output is Here (IP addresses have been changed)
In your loop that creates the tasks, you use only one instance of LoginCredentials. So all tasks share the instance. In the loop, you overwrite the hostname in each iteration. So in the end the LoginCredentials refers to the hostname of the last task. So all tasks connect to that host.
Just to do not pass the hostname via LoginCredentials and use task.getFirewall().getAddress() directly when creating the JSch session:
SSH ssh = new SSH(lc, t.getSSHCommand(enablepassword1), task.getFirewall().getAddress());
...
public SSH(LoginCredentials creds, String command, String hostname)
{
...
this.connect(hostname);
}
...
private void connect(String hostname) throws JSchException {
JSch jsch = new JSch();
session = jsch.getSession(creds.getUser(), hostname, 22);
...
}
private final String host;
private final String userAccount;
private final String keyDir;
private ChannelSftp sftpChannel;
private Session session;
private Channel channel
public void send(List<Path> filesToSend, String destination) throws SftpException, IOException {
if (sftpChannel == null) {
logger.error("Failed to create SFTP channel");
}
for (Path file : filesToSend) {
send(file, destination);
}
//summary of sent files over sftpchannel
}
public void send(Path file, String destination) throws SftpException, IOException {
if (sftpChannel == null) {
logger.error("Failed to create SFTP channel");
}
sftpChannel.put(Files.newInputStream(file), destination + File.separator + file.getFileName());
} //end send
}//end class
After all the files have been sent, can somebody show me how I can get a count of how many number of files have successfully been sent. Not important but also if any failed or any kind of monitoring. How can I do this with Jsch library.
I would like something in my log such as :
Preparing to send [14] files
Number of files sent is [14]...
public void send(List<Path> filesToSend, String destination) {
logger.debug("Preparing to send ["+filesToSend.size()+"] files");
if (sftpChannel == null) {
logger.error("Failed to create SFTP channel");
}
int successCount = 0;
int failedCount = 0;
for (Path file : filesToSend) {
try {
send(file, destination);
successCount++;
} catch (Exception e) {
failedCount++;
}
}
//summary of sent files over sftpchannel
logger.debug("Successfully sent " + successCount);
logger.debug("Failed to sent " + failedCount);
}
Given that jsch throws exceptions when an error occurs, why don't you just implement your own counter (just an int would work)?
I want to get a file from unix system to my local system which is on windows using java. I'm very much new to this concept. Any ideas on how it could be done? Which is the best java API for this task?
If the Unix machine supports SFTP, JSch is an option. You could adapt the following code to meet your needs:
private static final String USER_PROMPT = "Enter username#hostname:port";
private static final boolean USE_GUI = true;
public static void main(final String[] arg) {
Session session = null;
ChannelSftp channelSftp = null;
try {
final JSch jsch = new JSch();
final String defaultInput = System.getProperty("user.name") + "#localhost:22";
String input = (USE_GUI) ? JOptionPane.showInputDialog(USER_PROMPT, defaultInput) : System.console().readLine("%s (%s): ", USER_PROMPT, defaultInput);
if (input == null || input.trim().length() == 0) {
input = defaultInput;
}
final int indexOfAt = input.indexOf('#');
final int indexOfColon = input.indexOf(':');
final String user = input.substring(0, indexOfAt);
final String host = input.substring(indexOfAt + 1, indexOfColon);
final int port = Integer.parseInt(input.substring(indexOfColon + 1));
jsch.setKnownHosts("/path/to/known_hosts");
// if you have set up authorized_keys on the server, using that identitiy
// with the code on the next line allows for password-free, trusted connections
// jsch.addIdentity("/path/to/id_rsa", "id_rsa_password");
session = jsch.getSession(user, host, 22);
final UserInfo ui = new MyUserInfo();
session.setUserInfo(ui);
session.connect();
channelSftp = (ChannelSftp) session.openChannel("sftp");
channelSftp.connect();
channelSftp.get("/remotepath/remotefile.txt", "/localpath/localfile.txt");
} finally {
if (channelSftp != null) {
channelSftp.exit();
}
if (session != null) {
session.disconnect();
}
}
}
public static class MyUserInfo implements UserInfo {
private String password;
#Override
public String getPassword() {
return password;
}
#Override
public boolean promptYesNo(final String str) {
final Object[] options = {"yes", "no"};
final boolean yesNo = (USE_GUI) ? JOptionPane.showOptionDialog(null, str, "Warning", JOptionPane.DEFAULT_OPTION, JOptionPane.WARNING_MESSAGE, null, options, options[0]) == 0 : System.console().readLine("Enter y or n: ").equals("y");
return yesNo;
}
#Override
public String getPassphrase() {
return null;
}
#Override
public boolean promptPassphrase(final String message) {
return true;
}
#Override
public boolean promptPassword(final String message) {
if (!USE_GUI) {
password = new String(System.console().readPassword("Password: "));
return true;
} else {
final JTextField passwordField = new JPasswordField(20);
final Object[] ob = {passwordField};
final int result = JOptionPane.showConfirmDialog(null, ob, message, JOptionPane.OK_CANCEL_OPTION);
if (result == JOptionPane.OK_OPTION) {
password = passwordField.getText();
return true;
} else {
return false;
}
}
}
#Override
public void showMessage(final String message) {
if (!USE_GUI) {
System.console().printf(message);
} else {
JOptionPane.showMessageDialog(null, message);
}
}
}
I have found JSch to be very useful and straight foreword. Below is a snippet of code written to put all .txt files in a given folder on the sftp server.
public static void sftpConnection() {
// Object Declaration.
JSch jsch = new JSch();
Session session = null;
Channel channel = null;
// Variable Declaration.
String user = "foo";
String host = "10.9.8.7";
Integer port = 22;
String password = "test123";
String watchFolder = "\\localhost\textfiles";
String outputDir = "/remote/textFolder/";
String filemask = "*.txt";
try {
session = jsch.getSession(user, host, port);
/*
* StrictHostKeyChecking Indicates what to do if the server's host
* key changed or the server is unknown. One of yes (refuse connection),
* ask (ask the user whether to add/change the key) and no
* (always insert the new key).
*/
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword(password);
session.connect();
channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp)channel;
// Go through watch folder looking for files.
File[] files = findFile(watchFolder, filemask);
for(File file : files) {
// Upload file.
putFile(file, sftpChannel, outputDir);
}
} finally {
sftpChannel.exit();
session.disconnect();
}
}
public static void putFile(File file, ChannelSftp sftpChannel, String outputDir) {
FileInputStream fis = null;
try {
// Change to output directory.
sftpChannel.cd(outputDir);
// Upload file.
fis = new FileInputStream(file);
sftpChannel.put(fis, file.getName());
fis.close();
} catch{}
}
public static File[] findFile(String dirName, final String mask) {
File dir = new File(dirName);
return dir.listFiles(new FilenameFilter() {
public boolean accept(File dir, String filename)
{ return filename.endsWith(mask); }
} );
}
First thing that goes into my mind is FTP.
There are multiple choices to do that. First one simple socket communication between a java client and a server. If you want to go with this approach then follow this:
http://mrbool.com/file-transfer-between-2-computers-with-java/24516
Then there are other high level protocols implementations that can be used such as FTP, HTTP, etc
Follow a related SO post for java FTP client server: FTP client server model for file transfer in Java
I use the JSch library to develop a SFTP client.
The problem is both get and put methods' status is -1.
Here is my code:
class SftpClient {
private static final Logger LOG = Logger.getLogger(SftpClient.class);
/** Connection port number */
public static final int PORT = 22;
/** SECURED protocol name */
private static final String PROTOCOL = "sftp";
/** Connection time out in milliseconds */
public static final int TIME_OUT = 5000;
/** This class serves as a central configuration point, and as a factory for Session objects configured with these settings */
private JSch _client;
/** A session represents a connection to a SSH server */
private Session _session;
/** Channel connected to a SECURED server (as a subsystem of the SSH server) */
private ChannelSftp _channelSftp;
/**
* Value returned by the last executed command.
*/
private int _exitValue;
/**
* Computer contains the url, the login and the password to connect.
*/
private Computer _computer;
/**
* Initialize a SECURED client
* #param target - Machine we want to connect to
*/
public SftpClient(Computer target) {
_client = new JSch();
_computer = target;
}
protected void connect() throws Exception {
try {
if (_client == null) {
_client = new JSch();
}
if (_session == null) {
_session = _client.getSession(_computer.getLogin(), _computer.getUrl(), PORT);
Properties props = new Properties();
props.put("StrictHostKeyChecking", "no");
props.put("compression.s2c", "zlib,none");
props.put("compression.c2s", "zlib,none");
_session.setConfig(props);
_session.setPassword(_computer.getPassword());
if (LOG.isDebugEnabled()) {
LOG.debug("Connecting to "+_computer.getUrl()+" with login "+_computer.getLogin()+"...");
}
}
if (!_session.isConnected()) {
_session.connect(TIME_OUT);
}
// disconnect previous channel if it has not been killed properly
if (_channelSftp != null && _channelSftp.isConnected()) {
_channelSftp.disconnect();
}
_channelSftp = (ChannelSftp) _session.openChannel(PROTOCOL);
_channelSftp.connect();
if (LOG.isInfoEnabled()) {
LOG.info("Connected to "+_computer.getUrl()+" with login "+_computer.getLogin());
}
} catch(JSchException e) {
LOG.error("Auth failed", e);
throw e;
}
}
protected void connect(String path) throws Exception {
connect();
if (_channelSftp != null && _channelSftp.isConnected()) {
_channelSftp.cd(path);
}
}
public boolean get(final String remoteDirectory, final String remoteFile, final String localDirectory) throws Exception {
boolean res = false;
if (LOG.isInfoEnabled()) {
LOG.info("Download file "+remoteDirectory+"/"+remoteFile+" from "+_computer+" in "+localDirectory);
}
if (remoteDirectory != null && remoteFile != null && !remoteFile.isEmpty() && localDirectory != null) {
try {
// connect to the server and change directory
connect(remoteDirectory);
// change local directory
_channelSftp.lcd(localDirectory);
// download the file, keeping the same name
_channelSftp.get(remoteFile, remoteFile);
// update exit value
_exitValue = _channelSftp.getExitStatus();
if (_exitValue == 0) {
res = true;
}
if (LOG.isDebugEnabled()) {
LOG.debug("Exit status is: "+_exitValue);
}
} catch(SftpException e){
LOG.error("Auth failed", e);
throw e;
} finally {
if (_channelSftp != null && _channelSftp.isConnected()) {
_channelSftp.disconnect();
_channelSftp.exit();
}
}
} else {
LOG.warn("Check remoteDirectory ('"+remoteDirectory+"') or remoteFile ('"+remoteFile+"') or localDirectory ('"+localDirectory+"').");
}
return res;
}
public void put(final File localFile, final String destPath) throws Exception {
if (LOG.isInfoEnabled()) {
LOG.info("Send file "+localFile+" to "+_computer+" in "+destPath);
}
if (localFile == null) {
_exitValue = -1;
LOG.error("The given local file is null. Aborting tranfer.");
return;
}
if (!localFile.exists()) {
_exitValue = -1;
LOG.error("The given local file '"+localFile+"' does not exist. Aborting tranfer.");
return;
}
final InputStream input = new FileInputStream(localFile);
if (input == null || input.available() <= 0) {
_exitValue = -1;
LOG.error("Cannot read file "+localFile);
return;
}
try {
connect(destPath);
_channelSftp.put(input, localFile.getName());
_exitValue = _channelSftp.getExitStatus();
if (LOG.isDebugEnabled()) {
LOG.debug("Exit status is: "+_exitValue);
}
} catch(SftpException e){
LOG.error("Auth failed", e);
throw e;
} finally {
if (_channelSftp != null && _channelSftp.isConnected()) {
_channelSftp.disconnect();
_channelSftp.exit();
}
IOUtils.closeQuietly(input);
}
}
public void disconnect() {
if (_channelSftp != null && _channelSftp.isConnected()) {
_channelSftp.disconnect();
_channelSftp.exit();
}
if (_session != null && _session.isConnected()) {
_session.disconnect();
if (LOG.isInfoEnabled()) {
LOG.info("SECURED FTP disconnected");
}
}
}
}
No exception is thrown.
By the way, the files I want to upload and download are tar files. Does Jsch have a FTP binary mode?
Files are well transferred. I can use them properly but I am worried about the exit status.
I had a look to the Jsch API and it says:
The exit status is only available for certain types of channels, and only after the channel was closed (more exactly, just before the channel is closed).
How can I know if the exit status is available for ChannelSftp?
Looking at the source code it looks ExitStatus is not implemented for SFTP
channel.setExitStatus(reason_code);
is implemented for only below two cases in Session class
case SSH_MSG_CHANNEL_OPEN_FAILURE:
case SSH_MSG_CHANNEL_REQUEST:
and both the cases are not called atleast for successful scenarios that I tested.
I have this simple code to perdiodically access a pop3 server and check if there are new messages in the inbox folder
public static void main(String[] args) {
try {
String storeType = "pop3";
String host = "...";
int port = ...;
String user = "...";
String psw = "...";
//
int delay = 1;
long mins = 200 * 60 * delay;
firstAccess = true;
Properties properties = new Properties();
properties.put("mail.pop3.host", host);
properties.put("mail.pop3.user", user);
properties.put("mail.pop3.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
properties.put("mail.pop3.port", port);
RomasAuthenticator r = new RomasAuthenticator(user, psw);
Session session = Session.getDefaultInstance(properties, r);
POP3Store emailStore = (POP3Store) session.getStore(storeType);
emailStore.connect(host, port, user, psw);
Folder emailFolder = emailStore.getFolder("INBOX");
emailFolder.open(Folder.READ_ONLY);
while (true) {
if (checkInbox(emailFolder)) {
//prepara la notifica per il voice_service
System.out.println("mes!");
}
firstAccess = false;
Thread.sleep(mins);
}
} catch (Exception ex) {
//return null;
}
}
private static boolean checkInbox(Folder inbox_folder) throws MessagingException, IOException {
System.out.println("checking");
boolean res = false;
try {
int email_actual_count = inbox_folder.getMessageCount();
if (email_actual_count > email_prev_count /*or hasNewMessages()*/) {
if (!firstAccess) {
res = true;
}
}
//bisogna aggiornare comunque email_count in caso siano stati cancellati messaggi
email_prev_count = email_actual_count;
}
catch (Exception e) {
e.printStackTrace();
}
return res;
}
Both getMessageCount() and hasNewMessages() do not work, because the first always returns the same number of messages, and the second always returns false. What's that I'm doing wrong?
I don't want to use a messagelistener because this code will run on a robot with really low performances..
thanks everyone
The JavaMail FAQ explains why these features don't work with the POP3 protocol.