I am trying to connect to FTP that requires 'Explicit FTP over TLS' and upload a file.
I am trying to do it from my local machine which uses Java version 8 and commons net FTP version 3.6
Below is the code which I use
try {
FTPSClient ftpClient = new FTPSClient();
ftpClient.setDataTimeout(300);
ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
ftpClient.setAuthValue("TLS");
ftpClient.connect(server, port);
int reply = ftpClient.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
ftpClient.execAUTH("TLS"); //SSL
// Login
if (ftpClient.login(user, pass)) {
// Set protection buffer size
ftpClient.execPBSZ(0);
// Set data channel protection to private
ftpClient.execPROT("P");
// Enter local passive mode
ftpClient.enterLocalPassiveMode();
ftpClient.changeWorkingDirectory("/");
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
File firstLocalFile = new File(outputFileNameCSV);
String firstRemoteFile = csvFileNameOut;
InputStream inputStream = new FileInputStream(firstLocalFile);
// Store file on host
boolean done = ftpClient.storeFile(firstRemoteFile, inputStream);
inputStream.close();
if (done)
{
System.out.println("The file is uploaded successfully.");
}
// Logout
ftpClient.logout();
} else {
System.out.println("FTP login failed");
}
// Disconnect
ftpClient.disconnect();
} else {
System.out.println("FTP connect to host failed");
}
} catch (IOException ioe) {
System.out.println("FTP client received network error "+ioe);
}
Below is the log I get from Java CommandListener
220-FileZilla Server 0.9.60 beta
220-written by Tim Kosse (tim.kosse#filezilla-project.org)
220 Please visit https://filezilla-project.org/
AUTH TLS
234 Using authentication type TLS
AUTH TLS
534 Authentication type already set to TLS
FTP login screen
USER test
331 Password required for test
PASS pass
230 Logged on
PBSZ 0
200 PBSZ=0
PROT P
200 Protection level set to P
CWD /
250 CWD successful. "/" is current directory.
TYPE I
200 Type set to I
PASV
227 Entering Passive Mode ()
STOR file.csv
150 Opening data channel for file upload to server of "/file.csv"
FTP client received network error javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Below is the log when I try to connect using FileZilla. It is connecting fine and I can transfer the files.
But when I try using Java, it isn't transferring
Response: 220-FileZilla Server 0.9.60 beta
Response: 220-written by Tim Kosse (tim.kosse#filezilla-project.org)
Response: 220 Please visit https://filezilla-project.org/
Command: AUTH TLS
Response: 234 Using authentication type TLS
Status: Initializing TLS...
Status: Verifying certificate...
Command: USER test
Status: TLS/SSL connection established.
Response: 331 Password required for test
Command: PASS pass
Response: 230 Logged on
Command: PBSZ 0
Response: 200 PBSZ=0
Command: PROT P
Response: 200 Protection level set to P
Status: Connected
Status: Starting upload of file.csv
Command: CWD /
Response: 250 CWD successful. "/" is current directory.
Command: TYPE I
Response: 200 Type set to I
Command: PASV
Response: 227 Entering Passive Mode ()
Command: STOR file.csv
Response: 150 Opening data channel for file upload to server of "/file.csv"
Response: 226 Successfully transferred "/file.csv"
Status: File transfer successful, transferred 12,252 bytes in 1 second
Could you all please help?
Kind regards
Jon
Modern FTP servers expect the data channel to be opened by using TLS session resume. Java is capable of doing this but only for connections to the same IP and port. Because the data channel of an FTP session uses a different port the mechanism fails, so you need to force the JVM to do that and includes quite heavy fiddling with the interns of java classes using reflection.
You're using the Apache FTPSClient, so you might check if a more recent version of it already does that for you but if it's still not done, you might do it yourself as it is described in a Blog post on Wealthfront.
The hack (I still fail it to call it differently) works well with clients but there is some significant additional work to be done if you want to do this on a server side (i.e. if you want to implement you own FTP server). Your question doesn't look like it, so I spare this part in this answer.
So I am using HAProxy in front of Jetty servlets.
The goal at the moment is just proof of concept and load and stress testing once everything's configured.
However I have a problem configuring haproxy. I know that it's not a problem with my application cause I have nginx(tengine) running and everything works properly. So it has to be something with the haproxy configuration or just the way haproxy works is not suitable for my needs.
So what my client tries to do is connect to haproxy using two different connections and keep them open:
Connect with a chunked streaming mode for upload.
Connect with a normal mode and establish a download channel.
Here's how my haproxy.conf file looks like:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
# ca-base /etc/ssl/certs
# crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
maxconn 2048
defaults
log global
mode http
option forwardfor
option http-server-close
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
stats enable
stats uri /stats
stats realm Haproxy\ Statistics
stats auth user:password
frontend www-http
bind *:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend
frontend www-https
bind *:443 ssl crt /etc/haproxy/server.pem
reqadd X-Forwarded-Proto:\ https
default_backend www-backend
backend www-backend
redirect scheme https if !{ ssl_fc }
server www-1 localhost:8080 check maxconn 2048
And here's what my logs say when I try to access port 443:
Sep 17 11:10:18 xxxxx-pc haproxy[15993]: 127.0.0.1:32875 [17/Sep/2014:11:10:18.464] www- https~ www-backend/www-1 0/0/0/-1/1 502 212 - - PH-- 0/0/0/0/0 0/0 "GET /test HTTP/1.1"
Any ideas what the problem might be?
An issue with the configuration or ?
Thanks.
PH means that haproxy rejected the header from the backend because it was malformed.
http://www.haproxy.org/download/1.4/doc/configuration.txt
PH - The proxy blocked the server's response, because it was invalid,
incomplete, dangerous (cache control), or matched a security filter.
In any case, an HTTP 502 error is sent to the client. One possible
cause for this error is an invalid syntax in an HTTP header name
containing unauthorized characters. It is also possible but quite
rare, that the proxy blocked a chunked-encoding request from the
client due to an invalid syntax, before the server responded. In this
case, an HTTP 400 error is sent to the client and reported in the
logs.
I am creating a simple HTTPS connection and returning a response
URL google = new URL("https://www.google.com/");
HttpsURLConnection connection = (HttpsURLConnection)google.openConnection();
System.out.println( "Response: " + connection.getResponseCode());
This works fine and I receive a 200 response (I have all the keystores and truststores etc. defined)
I then proxy the request through a Jetty server by using https.proxyPort=8443 and https.proxyHost-localhost
The server has connectors that look like this:
Server server = new Server();
SelectChannelConnector connector0 = new SelectChannelConnector();
connector0.setPort(8080);
SslSelectChannelConnector sslConnector = new SslSelectChannelConnector();
sslConnector.setPort(8443);
SslContextFactory cf = sslConnector.getSslContextFactory();
cf.setKeyStorePath(keyStoreHome);
cf.setKeyStorePassword(keyStorePassword);
cf.setTrustStore(trustStoreHome);
cf.setTrustStorePassword(trustStorePassword);
cf.setCertAlias("jetty");
server.setConnectors(new Connector[] {connector0, sslConnector});
server.setHandler(new HelloHandler());
server.start();
server.join();
Which gives me the error below:
DEBUG: org.eclipse.jetty.io.nio - created SCEP#33589e56{l(/127.0.0.1:62348)<->r(/127.0.0.1:8443),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0}-{SslConnection#7f033a6f SSL NOT_HANDSHAKING i/o/u=-1/-1/-1 ishut=false oshut=false {AsyncHttpConnection#26c623af,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
DEBUG: org.eclipse.jetty.io.nio.ssl - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection#7f033a6f SSL NOT_HANDSHAKING i/o/u=180/0/0 ishut=false oshut=false {AsyncHttpConnection#26c623af,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} NOT_HANDSHAKING filled=180/180 flushed=0/0
DEBUG: org.eclipse.jetty.io.nio.ssl - SCEP#33589e56{l(/127.0.0.1:62348)<->r(/127.0.0.1:8443),d=true,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0r}-{SslConnection#7f033a6f SSL NEED_WRAP i/o/u=180/0/0 ishut=false oshut=false {AsyncHttpConnection#26c623af,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:519)
at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:354)
at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Thread.java:662)
DEBUG: org.eclipse.jetty.io.nio.ChannelEndPoint - close SCEP#33589e56{l(/127.0.0.1:62348)<->r(/127.0.0.1:8443),d=true,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0!}-{SslConnection#7f033a6f SSL NEED_WRAP i/o/u=180/0/0 ishut=false oshut=false {AsyncHttpConnection#26c623af,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
DEBUG: org.eclipse.jetty.http.HttpParser -
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
Can anybody please tell me what might be wrong with this?
By the looks of it the exception is occurring when the proxy is connecting the HTTP - 8080 and HTTPS - 8443. Does JETTY do decryption between those ports because if not the exception is about the 8080 being plain text.
This is more or less the same problem as in this question.
When you use https.proxyPort=8443 and https.proxyHost=localhost, the connection between your client and the proxy host/port isn't done using SSL/TLS, but in plain text HTTP. The client then sends the proxy a CONNECT request with the target host name (and this is way you get this "plain text" exception).
Then, the proxy makes a request to that target server and establishes a tunnel that relays all the raw traffic back and forth. From the client's point of view (HttpsUrlConnection), the socket that is connected to the proxy is upgraded to SSL/TLS (i.e. it initiate an handshake, which is relayed to the target server via that tunnel.
There aren't many clients that also support having the connection between the client and the proxy itself using HTTPS. This isn't generally very useful, since an attacker could only see the host name and port anyway.
The initial problem we encountered was that a regular FTPs download started failing due to an untrusted server certificate. This prompted us to wonder whether the certificate had been updated without the counterparty notifying us so we wanted to download the current certificate and compare it to the one we have in our keystore.
This seems to be a trickier problem than we had anticipated. The usual suspects (firefox, filezilla, ...) did not seem up to the task of connecting to an FTPs server through an FTP proxy so out of curiosity I started playing around with a more low level java approach. I can not for the life of me get it to work though.
First (overly simplistic) java attempt:
// create proxy connection
SocketFactory plainFactory = SocketFactory.getDefault();
Socket proxy = plainFactory.createSocket(proxyServer, proxyPort);
// create ssl connection on top of it?
SSLSocketFactory sslFactory = getSocketFactory();
SSLSocket socket = (SSLSocket) sslFactory.createSocket(proxy, server, port, true);
This approach obviously does not work.
Next I started playing around with ftp4j (http://www.sauronsoftware.it/projects/ftp4j/) it seems to have a clean and accessible codebase:
FTPClient client = new FTPClient();
client.setConnector(new FTPProxyConnector(proxyHost, proxyPort));
client.getConnector().setConnectionTimeout(0);
client.getConnector().setReadTimeout(0);
client.setSSLSocketFactory(getSocketFactory());
// also tried SECURITY_FTPS
client.setSecurity(FTPClient.SECURITY_FTPES);
client.connect(server, port);
This outputs:
REPLY: 220 Blue Coat FTP Service
SEND: USER anonymous
REPLY: 530-User Access denied.
REPLY: 530-
REPLY: 530-Usage: USER username#hostname
REPLY: 331 PASS userpassword
Exception in thread "main" java.io.IOException: Invalid proxy response
The proxy server has optional authentication and on our development servers we generally use "user#host" without proxy authentication. As such I assume the username, hostname and password are those of the remote server?
So I tried adding the remote parameters, this does not work:
REPLY: 220 Blue Coat FTP Service
SEND: USER test#ftps.example.com
Exception in thread "main" java.io.IOException: FTPConnection closed
Adding the proxy user to match the bluecoat format does not seem to work either:
USER %u#%h %s
PASS %p
ACCT %w
Any help with either of these two problems would be most welcome:
how to retrieve the server certificate from an ftps server through an ftp proxy
how to connect to an ftps server through an ftp proxy in java
You might want to try Apache Net commons libs.
Here is a similar thread that uses that Net Commons library
Net commons also has a fully functional FTP Client Example so you can test with something you know works.
I'm working on a project to auto-configure a user's email server settings in Java. I am extracting the mail server from his email address and looking up the MX records of that mail server using the DirContext class with com.sun.jndi.dns.DnsContextFactory.
Then I'm opening a Socket to each server and testing them using a HELO command and checking the responses.
My problem is that this works only when I test it with the unsecure SMTP port 25. How can I use it with the secure port 465?
I tried using Secure Sockets by using
SSLSocketFactory sslsocketfactory = (SSLSocketFactory)SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket)sslsocketfactory.createSocket(mailserver, STANDARD_SMTP_PORT);
But all connections get a timeout exception as follows
alt1.gmail-smtp-in.l.google.com. java.net.ConnectException: Operation timed out
Please help.
Further Info: I am not creating a mail client. This is to simplify the mail server settings for an already existing mail client.
SMTP-SSL (Port 465) afaik is deprecated, so instead of trying to connect to a ssl socket you'd actually connect "plaintext" on port 25 (or 587 for smtp auth connections), but send EHLO instead of HELO and see if the server supports STARTTLS which replaces the deprecated SMTP-SSL. after the client sends the STARTTLS command, the connection is encrypted, so, port 25 is not per se "unsecure"
telnet alt1.gmail-smtp-in.l.google.com 25
Trying 173.194.71.26...
Connected to alt1.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP i7si4620651lbb.76
EHLO myserver.com
250-mx.google.com at your service, [x.x.x.x]
250-SIZE 35882577
250-8BITMIME
250-STARTTLS <<--------- this is what you're looking for
250 ENHANCEDSTATUSCODES
STARTTLS <<--- client command to actually start encrypting the traffic
220 2.0.0 Ready to start TLS
Have you authenticated the program before establishing a connection with the server . Every remote mail server will need to authenticate the machine trying to establish a socket connection . If your using javaMail library for example , you will be given options to add the username and password . You will need to find a workaround while using simple sockets in java