I implemented java code to upload files to server with org.apache.commons.net.ftp.FTPClient
For multiple files the ftp upload speed is very slow.
How can I improve the speed.
-Change library?
What is the powerful FTP client class library for uploading multiple files?
-Use multiple threads?
How can I implement ftp upload function with multiple thread?
Could someone show me an example?
I am a new for multiple threading programming.
After I read all answer, I try to change my code and test it.
Following is a sample FTPClient code:
// create instance of FTPClient
FTPClient ftp = new FTPClient();
ftp.setControlEncoding("UTF-8");
ftp.setDefaultTimeout(30000);
// connect to server
try
{
ftp.connect("10.1.1.1", 990);
}
catch(Exception e)
{
System.out.println("Cannot connect to server");
return;
}
// login to server
if (!ftp.login("username", "password"))
{
ftp.logout();
System.out.println("Cannot login to server");
return;
}
try
{
ftp.setFileTransferMode(FTP.BINARY_FILE_TYPE);
ftp.enterLocalPassiveMode();
// ftp.setBufferSize(0); <-- someone suggest me to set buffer size to 0, but it throw error sometime.
}
catch(Exception e)
{
}
// create directory on server
// dirs is list of required directories on server
for (String dir : dirs)
{
try
{
ftp.makeDirectory(dir);
}
catch(IOException e)
{
}
}
// files is a map of local file and string of remote file
// such as
// file on client is "C://test/a.txt"
// location on server is "/test/a.txt"
for (Map.Entry<File, String> entry : files.entrySet())
{
File localFile = entry.getKey();
String remoteFile = entry.getValue();
FileInputStream input = null;
try
{
input= new FileInputStream(localFile);
ftp.storeFile(remoteFile, input);
}
catch (Exception e)
{
try
{
ftp.deleteFile(remoteFile);
}
catch (IOException e1)
{
}
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
}
}
}
}
// disconnect
if (ftp != null && ftp.isConnected())
{
try
{
ftp.disconnect();
}
catch (IOException f)
{
// do nothing
}
}
When I uploaded 1050 files (each file is about 1-20 KB), it took about 49406 - 51000 millisec (This is upload time only).
I would like to improve the speed.
Some people suggest me to use ftp4j, but when I test the library with 1050 files, the upload speed of ftp4j is slowly than FTPClient about 10000 millisec.
it took about 60000 millisec.
Following is sample ftp4j code:
// create instance of FTPClient
FTPClient ftp = new FTPClient();
ftp.setCharset("UTF-8");
// connect to server
try
{
ftp.connect("10.1.1.1", 990);
}
catch(Exception e)
{
System.out.println("Cannot connect to server")
return;
}
// login to server
try
{
ftp.login("username", "password");
}
catch (Exception e)
{
try
{
ftp.logout();
}
catch (Exception e1)
{
}
System.out.println("Cannot login to server")
return;
}
try
{
ftp.setType(FTPClient.TYPE_BINARY);
ftp.setPassive(true);
}
catch(Exception e)
{
}
// create directory on server
// dirs is list of required directories on server
for (String dir : dirs)
{
try
{
ftp.createDirectory(dir);
}
catch (Exception e)
{
}
}
// files is a map of local file and string of remote file
// such as
// file on client is "C://test/a.txt"
// location on server is "/test/a.txt"
for (Map.Entry<File, String> entry : files.entrySet())
{
final File localFile = entry.getKey();
final String remoteFile = entry.getValue();
BufferedInputStream input = null;
boolean success = false;
try
{
input = new BufferedInputStream(new FileInputStream(localFile));
// ftp.upload(localFile); <-- if I use ftp.upload(File), it will took more time.
ftp.upload(remoteFile, input, 0, 2048, new MyTransferListener());
success = true;
}
catch (Exception e)
{
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
}
}
if (!success)
{
try
{
ftp.deleteFile(remoteFile);
}
catch (Exception e)
{
}
}
}
}
// disconnect
if (ftp != null && ftp.isConnected())
{
try
{
ftp.disconnect();
}
catch (IOException f)
{
// do nothing
}
}
I try to use multiple threads.
Following is multiple threading code:
final CountDownLatch latch = new CountDownLatch(files.size());
ExecutorService pool = Executors.newFixedThreadPool(10);
for (Map.Entry<File, String> entry : files.entrySet())
{
final File localFile = entry.getKey();
final String remoteFile = entry.getValue();
pool.execute(new Runnable() {
public void run()
{
FileInputStream input = null;
try
{
input= new FileInputStream(localFile);
ftp.storeFile(remoteFile, input);
}
catch (Exception e)
{
try
{
ftp.deleteFile(remoteFile);
}
catch (IOException e1)
{
}
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
}
}
latch.countDown();
}
}
});
}
try
{
// waiting for all threads finish
// see: http://stackoverflow.com/questions/1250643/how-to-wait-for-all-threads-to-finish-using-executorservice
latch.await();
}
catch(Exception e)
{
}
Is it correct? It work correctly but it cannot improve speed.
it took about 49000 - 51000 millisec same as the code without thread.
I test the speed with intranet. It will take more time for internet.
How should I do for improve upload speed?
I don't know why, but Apache Commons FTP is pretty slow in uploading, i had the same problem and i couldn't solve it.
Now i use FTP4j, it's pretty similar to apache commons ftp but uploads are really fast.
This is an example:
FTPClient client = new FTPClient();
client.connect("www.yoursite.com");
client.login("login", "password");
client.setPassive(true);
client.setType(FTPClient.TYPE_BINARY);
client.changeDirectory("a");
File f = new File("path/to/your/file");
client.upload(f);
client.disconnect(true);
With this library i uplodaded a 340KB file in less than one second, while with Apache Commons FTP it took about 1 minute.
If you want to transfer different files with threads, try to put every client.upload(f) into a different thread, but i'm not sure it will boost up the transfer.
Quoting #fge previous answer:
Basically, chances are, you can't.
Don't forget that FTP has two types of channels: the command channel and the data channels. One upload is initiated by sending the instructions over the command channel to open a data channel for the upload proper.
Now:
most FTP servers are configured so that one command channel can only open one data channel at anyone time;
there are bandwidth limits: your upstream bandwidth, and the server's downstream bandwidth.
Were it possible to upload several files in parallel, ie, opening more than one data channel, you'd have the problem that the overhead of TCP itself would in fact slow down the upload process in general.
Basically: keep one data channel open at any time. Trying and opening more than one is just not worth it. It may work in ~1% of cases in general. This is just not worth the hassle.
This Q&A some possible explanations of what is going on: why is ftp upload slow in java 7
And furthermore, it offers a couple of workarounds:
Upgrade to the 3.3 snapshot which can (currently) be found here
Call FTPClient.setBufferSize(0).
Apparently there is also a regression in Java 7 for Windows where the firewall application filter for the FTP client is blocking the client from using PASV mode FTP. It is not clear what the best solution to that is, but you could try the following:
Change your Windows firewall to disable the firewall application filter (as described in the Microsoft KB page.
Change your FTP application to use "active" mode ... though this requires that the FTP server can initiate connections to the machine running your clients.
Note: that there seem to be more than one explanation for the problem ... or maybe more than one possible problems.
Related
I want to write a code which will transfer a file from one machine to another in Linux and Windows platforms.
I used ssh libraries (sftp connections) to transfer file to Linux machine.
Now, I wanted to do same for Windows machine. Can someone please help me with this?
Description: To transfer a file from one windows machine(Local) to another windows machine(Server).
Also, I checked with FTP libraries in java, but I wasn't able to create a directory outside the folder created/shared for ftp.
Below is my code I am using currently for ftp.
FTPClient ftpClient = new FTPClient();
FileInputStream inputStream = null;
try {
// pass directory path on server to connect
ftpClient.connect("172.30.17.17");
// pass username and password, returned true if authentication is
// successful
boolean login = ftpClient.login("Administrator", "Password1!");
if (login) {
System.out.println("Connection established...");
inputStream = new FileInputStream("C:/Demo/abcd.txt");
boolean uploaded = ftpClient.storeFile("uploadedFile3.txt",inputStream);
if (uploaded) {
System.out.println("File uploaded successfully !");
} else {
System.out.println("Error in uploading file !");
}
ftpClient.makeDirectory("C:/Demo1"); //Unable to create this here
System.out.println("Folder Created successfully !");
// logout the user, returned true if logout successfully
boolean logout = ftpClient.logout();
if (logout) {
System.out.println("Connection close...");
}
} else {
System.out.println("Connection fail...");
}
} catch (SocketException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
ftpClient.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I am using Apache Commons net FTPClient to login and read files from an FTP Server. I manage to login and I can see it logs in successfully, because it shows the working directory path in the header string. However it shows no files when I use listFiles(). (I have also tried using listDirectories() and listNames() but with no success)
Below is a snippet:
try {
client.connect(ftpHost);
} /*catch (SocketException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}*/ catch (IOException ex) {
// TODO Auto-generated catch block
ex.printStackTrace();
}
String header ="";
InputStream stream=null;
BufferedReader reader=null;
try{
client.login(ftpUser, ftpPass);
client.changeWorkingDirectory(targetWorkingDir);
header = client.printWorkingDirectory();
//client.setFileType(FTP.BINARY_FILE_TYPE);
for(int i=0; i<client.listFiles().length;i++){
header+=client.listFiles()[i].getName() +"\n";
}
}
catch (IOException ex){
ex.printStackTrace();
header="ERROR 1: " + ex.getMessage();
for(int i=0;i<ex.getStackTrace().length;i++){
header += "\n" + ex.getStackTrace()[i];
}
}
catch(NullPointerException e){
header = "ERROR 2: "+ e.getMessage()+"\n";
for(int i=0;i<e.getStackTrace().length;i++){
header+= e.getStackTrace()[i] + "\n";
}
}
finally{
if(reader!=null){
try{reader.close();}catch(IOException e){e.printStackTrace();}
try{stream.close();}catch(IOException e){e.printStackTrace();}
}
}
I have also tried using something like this to read a file:
try {
stream = ftpClient.retrieveFileStream("klasa.csv");
reader = new BufferedReader(new InputStreamReader(stream));
header = reader.readLine();
} finally {
if (reader != null) try { reader.close(); } catch (IOException logOrIgnore){}
}
In both cases I'm pretty sure I'm at the right directory, and I make sure that my file is there via FileZilla, but the client can't seem to read any file.
Try to avoid calling listFiles() in the loop. Every call will perform the entire FTP LIST command sequence, so eventually, you will add unnecessary traffic to every call.
You can try to simplify your program first like this:
private static void ftpTest() {
FTPClient f = new FTPClient();
try {
f.connect("{UOUR FTP SERVER}");
f.login("{USER}", "{PASSWORD}");
FTPFile[] files = f.listFiles(".");
for (FTPFile fi: files) {
System.out.printf("f%s\n", fi.getName());
}
} catch (IOException e) {
e.printStackTrace();
}
}
If this program will not give you the list of files in the root directory of your server, you can try to compare your FileZilla FTP options (PASSIVE/ACTIVE mode in particular): https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#enterLocalPassiveMode()
If it won't help you can try to sniff the network traffic using WireShark or tcpdump and compare the commands set to the FTP server.
I'm programming a network software with Java, but I have a real problem using my application through a "true" network.
Let a software be a host, and listening for client connexions.
Here is my Server loop :
public void run() {
while (mServerSocket != null) {
try {
Socket wClient = mServerSocket.accept();
System.out.println("Client connecté");
wClient.setSoTimeout(50);
wClient.setTcpNoDelay(false);
Client c = new Client(wClient);
synchronized(this) {
mWaitingClients.add(c);
c.start();
}
} catch(Exception ex) {
System.out.println("Server error : " + ex.getMessage());
}
}
}
When a client tried to connect to the server, I use this function :
public Client connect(InetAddress addr, int port) throws Exception {
Socket socket = new Socket(addr, port);
socket.setSoTimeout(50);
socket.setTcpNoDelay(false);
Client c = new Client(socket);
c.start();
return c;
}
And here is the client loop :
public void run() {
try {
ObjectOutputStream out = new ObjectOutputStream(mSocket.getOutputStream());
ObjectInputStream in = new ObjectInputStream(mSocket.getInputStream());
while(mSocket.isConnected() && !mSocket.isClosed()) {
for (int i = 0; i < mOutMessages.size(); i++) {
Message msg = mOutMessages.get(i);
out.writeObject(msg);
}
out.flush();
mOutMessages.clear();
Thread.sleep(50);
out.reset();
while(true) {
try {
Message m = (Message) in.readObject();
mInMessages.add(m);
} catch (Exception e) {
break;
}
}
Thread.sleep(50);
}
} catch(Exception ex) {
try {
mSocket.close();
} catch(Exception exx) {
exx.printStackTrace();
}
ex.printStackTrace();
}
}
Some other parts of the program do Message and put them in the Output list of the Client (mOutMessages).
Some other parts of the program read Message from the mInMessages of the Client.
But something is wrong with this. It works fine locally (server and client on the same computer), but fail or is hazardous (some messages are sent but never received) using two computers (with LAN or through the Internet).
Server ever detect connexions from the clients, send "handshake" messages to the client, but the client never receives them.
I'm more a C programmer than a Java one, and I never had this kind of problem using libc Sockets, so, why my way of doing is wrong ?
Thank you !
Edit :
My Server is created using this function :
public void open(int port) throws Exception {
mServerSocket = new ServerSocket(port);
start(); // Call the run mentionned above.
}
Edit :
Here is my solution, maybe it's not perfect but it works !
public void run() {
try {
BufferedOutputStream buf_out = new BufferedOutputStream(
mSocket.getOutputStream()
);
BufferedInputStream buf_in = new BufferedInputStream(
mSocket.getInputStream()
);
ObjectOutputStream out = new ObjectOutputStream(buf_out);
out.flush();
ObjectInputStream in = new ObjectInputStream(buf_in);
while(mSocket.isConnected() && !mSocket.isClosed()) {
for (int i = 0; i < mOutMessages.size(); i++) {
Message msg = mOutMessages.get(i);
out.writeObject(msg);
out.flush();
}
mOutMessages.clear();
out.reset();
while(true) {
try {
Message m = (Message) in.readObject();
mInMessages.add(m);
} catch (Exception e) {
break;
}
}
}
} catch(Exception ex) {
try {
mSocket.close();
} catch(Exception exx) {
exx.printStackTrace();
}
ex.printStackTrace();
}
If I understand right, both client and server use the run method. If both client and server happen to write sufficiently large messages (not fitting in involved buffers) at the same time then you get a deadlock because neither partner advances to reading (which would drain full buffers). Due to network delays, this might only happen in the non-local scenario, i.e. there may be enough time to pile up enough messages in the mOutMessages buffer.
Note that documentation of Socket.setSoTimeout (which you used) only says that it affects read()s. (For example, in my JDK, ObjectOutputStream seems to use a BlockDataOutputStream with a buffer size of 1024 bytes).
I recommend to either use a separate thread for reading/writing or (if you know the maximum messages size) use a sufficiently large buffer (by wrapping the SocketOutputStream in a BufferedOutputStream). If you opt for larger buffers, you may also want to write one message at a time (and try to read messages after each).
I want to implement a FTP Client with Apache Commons Net only for uploading data.
The Connection and Login to FTP-Server works fine.
But the upload does not work right.
The files are a little to big as the originals.
And the files are damaged.
I tried an image, a video and a textfile. Only the textfile is alright.
Now I see while debugging
boolean tmp=client.setFileTransferMode(FTPClient.BINARY_FILE_TYPE);
gives me false. So it can not be set. Why?
(Maybe this is not the problem?)
Here a the rest of my code
client=new FTPClient();
try {
int reply;
client.connect(url, port);
reply = client.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply))
{
client.disconnect();
System.err.println("FTP server refused connection.");
System.exit(1);
}
client.login(user, pw);
boolean xxx=client.setFileTransferMode(FTPClient.BINARY_FILE_TYPE);
client.setControlKeepAliveTimeout(300);
client.enterLocalPassiveMode();
if (client.isConnected())
{
try {
File file=new File(<FILE>);
FileInputStream inputStream = new FileInputStream(file);
OutputStream outputStream = client.storeFileStream(file.getName());
byte[] buffer = new byte[4096];
int l;
while((l = inputStream.read(buffer))!=-1)
{
outputStream.write(buffer, 0, l);
}
inputStream.close();
outputStream.flush();
outputStream.close();}
Change the following:
boolean xxx=client.setFileTransferMode(FTPClient.BINARY_FILE_TYPE);
Should be:
boolean xxx=client.setFileType(FTP.BINARY_FILE_TYPE);
You have confused FileTransferModes with FileTypes.
The available FileTypes are:
FTP.ASCII_FILE_TYPE (default)
FTP.BINARY_FILE_TYPE
FTP.EBCDIC_FILE_TYPE
FTP.LOCAL_FILE_TYPE
The available FileTransferModes are:
FTP.STREAM_TRANSFER_MODE (default)
FTP.BLOCK_TRANSFER_MODE
FTP.COMPRESSED_TRANSFER_MODE
I suppose if apache introduced enums for these constant types, then this kind of problem could be avoided, but then the library would not be available to pre-java-5 runtimes.
I wonder how much of an issue java 1.4 compatibility really is.
If only the text file was transferred successfully, I suspect you need to set the binary transfer file type.
See the setFileType method to see how to do this.
The commons-net wiki mentions this is the cause of most file corruption issues.
This work for me, Uploading Image and download after It´s Ok: Using
FTP.LOCAL_FILE_TYPE
this code using logger, replace for you logger or use System.out.println("");
private void cargarData(File filelocal) {
FTPClient client = new FTPClient();
try {
client.connect("URLHOSTFTP", "PORT: DEFAULT 21");
if (!FTPReply.isPositiveCompletion(client.getReplyCode())) {
client.disconnect();
logger.error("FTP server refused connection.");
System.exit(1);
}
client.login("USER FTP", "PASS FTP");
boolean type = client.setFileType(FTP.LOCAL_FILE_TYPE);
logger.info("Tipo Aceptado:" + type);
client.setControlKeepAliveTimeout(300);
client.enterLocalPassiveMode();
if (client.isConnected()) {
FileInputStream fis = null;
fis = new FileInputStream(filelocal);
client.storeFile(filelocal.getName(), fis);
client.logout();
if (fis != null) {
fis.close();
}
}
logger.info(client.getReplyString());
} catch (IOException e) {
logger.error("error" + e.getMessage());
e.printStackTrace();
} catch (Exception e) {
logger.error("error" + e.getMessage());
e.printStackTrace();
}
}
I have the following method to upload_files to an FTP server, I am not receiving any errors yet the file is not appearing on the server after its run. What could be the problem?
public static void upload_files(String un, String pw, String ip, String dir, String fn){
FTPClient client = new FTPClient();
FileInputStream fis = null;
try {
client.connect(ip);
client.login(un, pw);
String filename = dir+"/"+fn;
fis = new FileInputStream(filename);
client.storeFile(filename, fis);
client.logout();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fis != null) {
fis.close();
}
client.disconnect();
System.out.println("uploaded");
} catch (IOException e) {
e.printStackTrace();
}
}
}
There are a number of possible issues. Assumption is that you are using FTPClient 3.x from Apache commons-net. If using something else, you should probably indicate that in your question. Ideas:
Check the reply status of the connection to make sure you are connecting as expected. There's an example on how to do this in the JavaDoc.
Your filename variable is the path to the local file you want to send. Is that really the same path you want to use for storing the file on the server (relative to the FTP login root)? It might be, but usually isn't. If not, your first parameter to client.storeFile(...) needs to be changed.
Most FTP servers provide ability to log all actions. Are you able to access yours? If so, that usually quickly makes clear what is going wrong.