Develop Reference

JSch not uploading complete file to remote SFTP server, only partial

I am trying to use the Jsch library to transfer a locally created XML file (marshalled from a Java object using JAXB) to a remote server. However, the file only gets partially uploaded. It is missing the end tag and an arbitrary amount of characters at the end.
My code looks like this (TradeLimits is a JAXB annotated Java class)
TradeLimits limits = getTradeLimits(); //complex object with many fields
JSch jsch = new JSch();
jschSession = jsch.getSession(username, remoteHost);
//to avoid unknown host issues
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
jschSession.setConfig(config);
jschSession.setPassword(password);
jschSession.setPort(22);
jschSession.connect();
ChannelSftp channelSftp = (ChannelSftp) jschSession.openChannel("sftp");
channelSftp.connect();
jaxbContext = JAXBContext.newInstance(TradeLimits.class);
Marshaller marshaller = jaxbContext.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); //for pretty print
marshaller.marshal(limits, channelSftp.put(limitUploadPathString)); //this uploads only partial xml file to sftp server
marshaller.marshal(limits, System.err)); //THIS WORKS CORRECTLY AND THE FULL XML IS PRINTED!
channelSftp.disconnect();
channelSftp.exit();
Note how this cannot be a JAXB issue because it will print the complete XML elsewhere, but only partial is uploaded to remote server. What can possibly be the issue? Thanks in advance!

Always ensure you flush/close an OutputStream when you are done writing to it.
try(OutputSteam fileStream = channelSftp.put(limitUploadPathString)) {
marshaller.marshal(limits, fileStream);
}

Is there a simple method to check if there are changes in a SFTP server?

My objective is to poll the SFTP server for changes. My first thought is to check if the number of files in the dir changed. Then maybe some additional checks for changes in the dir.
Currently I'm using the following:
try {
FileSystemOptions opts = new FileSystemOptions();
SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(opts, "no");
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);
SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 60000);
FileSystemManager manager = VFS.getManager();
FileObject remoteFile = manager.resolveFile(SFTP_URL, opts);
FileObject[] fileObjects = remoteFile.getChildren();
System.out.println(DateTime.now() + " --> total number of files: " + Objects.length);
for (FileObject fileObject : fileObjects) {
if (fileObject.getName().getBaseName().startsWith("zzzz")) {
System.out.println("found one: " + Object.getName().getBaseName());
}
}
} catch (Exception e) {
e.printStackTrace();
}
This is using apache commons vfs2 2.2.0. It works "fine", but when the server has too many files, it takes over minutes just to get the count(currently, it takes over 2 minutes to get the count for a server that has ~10k files). Any way to get the count or other changes on the server faster?

Unfortunately there's no simple way in the SFTP protocol to get the changes. If you can have some daemon running on the server OR if the source of the new files can create/update a helper file, creation of such file with the last modification time in its name or contents can be an option.

I know the SFTP protocol fairly well, having developed commercial SFTP clients and an SFTP server (CompleteFTP), and as far as I know there's no way within the protocol to get a count of files in a directory without listing it. Some servers, such as ours, provide ways of adding custom commands to servers that you can invoke from the client, so it would be possible to add a custom command that returns the number of files in a directory. CompleteFTP also allows you to write custom file-systems so you could potentially write one that only shows files that have changed after a given timestamp when you do a listing, which might be another approach. Our server only runs on Windows though, so that might be show-stopper for you.

Java code for connect alfresco through ftps connection

When I try to connect our alfresco through SFTP it is not able to connect alfresco. It hangs the explorer and no error goes in the logger file also.
public void FTPTest()throws SocketException, IOException, NoSuchAlgorithmException
{
FTPSClient ftp = new FTPSClient("SSL");
System.out.println("1");
ftp.connect("172.17.178.144",2121); // or "localhost" in your case
System.out.println("2"+ftp.getReplyString());
System.out.println("login: "+ftp.login("admin", "admin"));
System.out.println("3"+ ftp.getReplyString());
ftp.changeWorkingDirectory("/alfresco");
// list the files of the current directory
FTPFile[] files = ftp.listFiles();
System.out.println("Listed "+files.length+" files.");
for(FTPFile file : files) {
System.out.println(file.getName());
}
// lets pretend there is a JPEG image in the present folder that we want to copy to the desktop (on a windows machine)
ftp.setFileType(FTPClient.BINARY_FILE_TYPE); // don't forget to change to binary mode! or you will have a scrambled image!
FileOutputStream br = new FileOutputStream("C:\\Documents and Settings\\casonkl\\Desktop\\my_downloaded_image_new_name.jpg");
ftp.retrieveFile("name_of_image_on_server.jpg", br);
ftp.disconnect();
}
I got output in our console only
1
at the execution of ftp.connect("172.17.178.144",2121); this code system will be hang no error got in our console
I am able to connect to my Alfresco through SFTP with the Filezila FTP client software. Can any one help me resolve this issue?

If I'm not mistaken then Alfresco chose for FTPS.
So try it with the following code here: http://alvinalexander.com/java/jwarehouse/commons-net-2.2/src/main/java/examples/ftp/FTPSExample.java.shtml

Secure FTP using private key authentication in java

import com.jcraft.jsch.*;
import com.jcraft.jsch.JSchException;
import oracle.jdbc.driver.OracleDriver;
import java.io.*;
import java.util.*;
import java.sql.*;
import java.net.*;
public class SecureFTP {
public static void main(String[] args) throws IOException , ClassNotFoundException, JSchException, SftpException{
JSch jsch = new JSch();
File file = new File("/home/xxxxx/.ssh/id_rsa");
Session session = null;
URL keyFileURL = null;
URI keyFileURI = null;
if (file.exists())
{
keyFileURL = file.toURL();
if (keyFileURL == null)
{
System.out.println("what");
throw new RuntimeException("Key file not found in classpath");
}
}
else System.out.println("FIle not found");
try{
keyFileURI = keyFileURL.toURI();
}
catch(Exception URISyntaxException)
{
System.out.println("Wrong URL");
}
String privateKey = ".ssh/id_rsa";
//jsch.addIdentity(privateKey);
jsch.addIdentity(new File(keyFileURI).getAbsolutePath());
System.out.println(new File(keyFileURI).getAbsolutePath() + " LOL");
session = jsch.getSession("username", "servername");
//session.setPassword("password");
java.util.Properties config = new java.util.Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
// connect
session.connect();
// get SFTP channel
Channel channel = session.openChannel("sftp");
channel.connect();
ChannelSftp schannel = (ChannelSftp) channel;
FileInputStream fis = new FileInputStream(sourcefile);
schannel.put(fis, destinationfile );
fis.close();
}
schannel.exit();
session.disconnect();
}
}
As you can see from the code I have commented out, I have tried everything possible to get this working and the only thing that works is if I set the password directly. I am trying to use the RSA private key generated, but I keep getting an auth fail.
I have added the public key to the list of authorized keys on the target server. And there is no passphrase.
Is there something else I am supposed to do? Like say, while generating the keys? Is there a step I am missing?
Is there another library I can use to implement the same function?

Make sure the necessary files exist (id_rsa and id_rsa.pub on the client, authorized_keys on the server). Make sure you can use public key authentication with another tool, like ssh, using these files.
If that looks alright, the problem may be with your Java security provider. Read on if you think you have the right files in place.
There are different formats for RSA private key storage, and SSH uses one that is not standard. Most providers expect something called a CRT RSA key, and when JSch doesn't give them a key in that format, they raise an exception which JSch silently eats and goes on to the next authentication method.
What is your provider? The following snippet will help you find out:
import java.security.KeyFactory;
…
KeyFactory f = KeyFactory.getInstance("RSA");
System.out.println(f.getProvider().getName());
Update: I did some checking around, and as of Java 5, the SunPKCS11 provider is installed with the highest precedence on Solaris systems, for performance. Since I don't run Solaris, I can't test it, but I believe this may be causing the problem.
JSch doesn't allow you to specify the provider to use for this operation through its API, so you will have to change the precedence of the installed providers. In fact, I'd suggest trying to remove the SunPKCS11 from this application; run this code once when your application starts up:
Security.removeProvider("SunPKCS11-Solaris");

Have you have copied the key into the file $HOME/.ssh/authorized_keys on the target server? If so, you should probably mention that. If not, that is required for this to work. Also, are you generating the key without a password? If the private key is password protected, you will need to provide that password to addIdentity.
After verifying those things, I'd recommend trying to connect via the command line using OpenSSH, as the Java code you have here looks correct. If the command line does not work, invoke it with -vvv to get verbose output about what it is doing. It is possible that the server is configured with PubkeyAuthentication set to no.

Apache FTPClient - incomplete file retrieval on Linux, works on Windows

I have a java application on Websphere that is using Apache Commons FTPClient to retrieve files from a Windows server via FTP. When I deploy the application to Websphere running in a Windows environment, I am able to retrieve all of the files cleanly. However, when I deploy the same application to Webpshere on Linux, there are cases where I am getting an incomplete or corrupt files. These cases are consistent though, such that the same files will fail every time and give back the same number of bytes (usually just a few bytes less than what I should be getting). I would say that I can read approximately 95% of the files successfully on Linux.
Here's the relevant code...
ftpc = new FTPClient();
// set the timeout to 30 seconds
ftpc.enterLocalPassiveMode();
ftpc.setDefaultTimeout(30000);
ftpc.setDataTimeout(30000);
try
{
String ftpServer = CoreApplication.getProperty("ftp.server");
String ftpUserID = CoreApplication.getProperty("ftp.userid");
String ftpPassword = CoreApplication.getProperty("ftp.password");
log.debug("attempting to connect to ftp server = "+ftpServer);
log.debug("credentials = "+ftpUserID+"/"+ftpPassword);
ftpc.connect(ftpServer);
boolean login = ftpc.login(ftpUserID, ftpPassword);
if (login)
{
log.debug("Login success..."); }
else
{
log.error("Login failed - connecting to FTP server = "+ftpServer+", with credentials "+ftpUserID+"/"+ftpPassword);
throw new Exception("Login failed - connecting to FTP server = "+ftpServer+", with credentials "+ftpUserID+"/"+ftpPassword);
}
is = ftpc.retrieveFileStream(fileName);
ByteArrayOutputStream out = null;
try {
out = new ByteArrayOutputStream();
IOUtils.copy(is, out);
} finally {
IOUtils.closeQuietly(is);
IOUtils.closeQuietly(out);
}
byte[] bytes = out.toByteArray();
log.info("got bytes from input stream - byte[] size is "+ bytes.length);
Any assistance with this would be greatly appreciated.
Thanks.

I have a suspicion that the FTP might be using ASCII rather than binary transfer mode, and mapping what it thinks are Window end-of-line sequences in the files to Unix end-of-lines. For files that are really text, this will work. For files that are really binary, the result will be corruption and a slightly shorter file if the file contains certain sequences of bytes.
See FTPClient.setFileType(...).
FOLLOWUP
... so why this would work on Windows and not Linux remains a mystery for another day.
The mystery is easy to explain. You were FTP'ing files from a Windows machine to a Windows machine, so there was no need to change the end-of-line markers.