Develop Reference

How to establish a FTPS data connection to a FileZilla Server 1.2.0

It is a known problem to use the Java FTPSClient of Apache commons-net with session resumption. Session resumption is a security feature which a FTPS server can require for data connections. The Apache FTPSClient does not support session resumption, and the JDK APIs make it hard to build a custom implementation. There are a couple of workarounds using reflection, see e.g. this answer and this commons-net bug entry.
I use such a workaround (see snipped below) in JDK 11 and tested it against a local FileZilla Server. It works with FileZilla Server 0.9.6, but it doesn't with FileZilla Server 1.2.0, which is the latest version at the time of writing. With that version, when trying to establish a data connection, the server responds with:
425 Unable to build data connection: TLS session of data connection not resumed.
As I said, FileZilla Server 0.9.6 is fine with how I do session resumption, and I made sure that the setting for requiring session resumption is activated.
In FileZilla Server 1.2.0, such settings are now set implicitly and cannot be changed via the GUI, maybe not at all. Are there some server settings that I can tweak for this to work? Or is it an issue with how I implemented the workaround? Does anyone experience similar issues?
This is the workaround I am using:
public class FTPSClientWithSessionResumption extends FTPSClient {
static {
System.setProperty("jdk.tls.useExtendedMasterSecret", "false");
System.setProperty("jdk.tls.client.enableSessionTicketExtension", "false");
}
#Override
protected void _connectAction_() throws IOException {
super._connectAction_();
execPBSZ(0);
execPROT("P");
}
#Override
protected void _prepareDataSocket_(Socket socket) throws IOException {
if (useSessionResumption && socket instanceof SSLSocket) {
// Control socket is SSL
final SSLSession session = ((SSLSocket)_socket_).getSession();
if (session.isValid()) {
final SSLSessionContext context = session.getSessionContext();
try {
final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
sessionHostPortCache.setAccessible(true);
final Object cache = sessionHostPortCache.get(context);
final Method putMethod = cache.getClass().getDeclaredMethod("put", Object.class, Object.class);
putMethod.setAccessible(true);
Method getHostMethod;
try {
getHostMethod = socket.getClass().getMethod("getPeerHost");
}
catch (NoSuchMethodException e) {
// Running in IKVM
getHostMethod = socket.getClass().getDeclaredMethod("getHost");
}
getHostMethod.setAccessible(true);
Object peerHost = getHostMethod.invoke(socket);
InetAddress iAddr = socket.getInetAddress();
int port = socket.getPort();
putMethod.invoke(cache, String.format("%s:%s", peerHost, port).toLowerCase(Locale.ROOT), session);
putMethod.invoke(cache, String.format("%s:%s", iAddr.getHostName(), port).toLowerCase(Locale.ROOT), session);
putMethod.invoke(cache, String.format("%s:%s", iAddr.getHostAddress(), port).toLowerCase(Locale.ROOT), session);
}
catch (Exception e) {
throw new IOException(e);
}
}
else {
throw new IOException("Invalid SSL Session");
}
}
}
}
The address under which the socket is cached is determined using getPeerHost, getInetAddress().getHostName(), and getInetAddress().getHostAddress(). I tried several combinations of doing or not doing these three, but I always get the same result.
Edit:
Here is a screenshot of the server logs of the full session:

As stated in this StackOverflow post it is possible to tell the JVM that only TLS 1.2 should be used.
Here is the link to the original answer which worked for me: command for java to use TLS1.2 only
You have to add a command line parameter at the start of the JVM in this case this is: java -Djdk.tls.client.protocols=TLSv1.2 -jar ... <rest of command line here>
This simple parameter worked for me, now I can connect and transfer data from a FTP-Server wich runs FileZilla FTP-Server 1.3.0

Why does Apache Commons VFS consider Http Proxy and Socks5 Proxy but simply ignore Socks4 Proxy?

I am working on a application that connects to an SFTP server and downloads files using Apache Commons VFS, it works just fine, with the exception that the system needs to allow the user to specify a proxy, as needed.
Now, I know Apache Commons VFS is built on top of Jsch and I know Jsch contains the classes: com.jcraft.jsch.ProxyHTTP, com.jcraft.jsch.ProxySOCKS4 and com.jcraft.jsch.ProxySOCKS5.
The code below is an extract of VFS class org.apache.commons.vfs2.provider.sftp.SftpClientFactory:
public static Session createConnection(
...
final SftpFileSystemConfigBuilder.ProxyType proxyType = builder.getProxyType(fileSystemOptions);
...
final String proxyUser = builder.getProxyUser(fileSystemOptions);
final String proxyPassword = builder.getProxyPassword(fileSystemOptions);
Proxy proxy = null;
if (SftpFileSystemConfigBuilder.PROXY_HTTP.equals(proxyType)) {
proxy = createProxyHTTP(proxyHost, proxyPort);
((ProxyHTTP)proxy).setUserPasswd(proxyUser, proxyPassword);
} else if (SftpFileSystemConfigBuilder.PROXY_SOCKS5.equals(proxyType)) {
proxy = createProxySOCKS5(proxyHost, proxyPort);
((ProxySOCKS5)proxy).setUserPasswd(proxyUser, proxyPassword);
} else if (SftpFileSystemConfigBuilder.PROXY_STREAM.equals(proxyType)) {
proxy = createStreamProxy(proxyHost, proxyPort, fileSystemOptions, builder);
}
...
As you can you see, there's no "if" statement to instantiate ProxySOCKS4!
I have duplicated the SftpClientFactory class, set my version to load before the original class on the classpath and changed the code as follow:
public static Session createConnection(
...
final SftpFileSystemConfigBuilder.ProxyType proxyType = builder.getProxyType(fileSystemOptions);
...
final String proxyUser = builder.getProxyUser(fileSystemOptions);
final String proxyPassword = builder.getProxyPassword(fileSystemOptions);
Proxy proxy = null;
if (SftpFileSystemConfigBuilder.PROXY_HTTP.equals(proxyType)) {
proxy = createProxyHTTP(proxyHost, proxyPort);
((ProxyHTTP)proxy).setUserPasswd(proxyUser, proxyPassword);
/// change start (I also created the PROXY_SOCKS4 constant)
} else if (SftpFileSystemConfigBuilder.PROXY_SOCKS4.equals(proxyType)) {
proxy = createProxySOCKS4(proxyHost, proxyPort);
((ProxySOCKS4)proxy).setUserPasswd(proxyUser, proxyPassword);
/// change end
} else if (SftpFileSystemConfigBuilder.PROXY_SOCKS5.equals(proxyType)) {
proxy = createProxySOCKS5(proxyHost, proxyPort);
((ProxySOCKS5)proxy).setUserPasswd(proxyUser, proxyPassword);
} else if (SftpFileSystemConfigBuilder.PROXY_STREAM.equals(proxyType)) {
proxy = createStreamProxy(proxyHost, proxyPort, fileSystemOptions, builder);
}
...
.. and guess what, when I set my application to use a Socks 4 Proxy it works alright with the change above. It is important to say that setting the application to work with Socks 5 does not work if the proxy server is a Socks 4 type, and that's true not only for my application with VFS, but also any other client I tested, like Fillezila or WinSCP.
So, the main question is:
Why does VFS predicts the usage of ProxyHTTP, ProxySOCKS5 but completely ignores the JSch ProxySOCKS4 class? Am I missing some SFTP or Proxy concept here or should I consider VFS bugged? That's the first time I work with VFS.
Please consider the question in bold as the main question not to make it too broad.

I wasn't able to get or find a better answer in time, so what I did to solve my problem was exactly what I described in the question.
I duplicated the classes SftpClientFactory e SftpFileSystemConfigBuilder, made the necessary adjustments and used them instead of the original classes, it's ugly and now I am stuck with a specific VFS version, I know, but the problem was solved.
Lesson for next time: use Jsch instead of VFS.
I'll leave the question open though, in case someone else have a proper solution or answer.

Vertx HttpClient getNow not working

I have problem with vertx HttpClient.
Here's code which shows that tests GET using vertx and plain java.
Vertx vertx = Vertx.vertx();
HttpClientOptions options = new HttpClientOptions()
.setTrustAll(true)
.setSsl(false)
.setDefaultPort(80)
.setProtocolVersion(HttpVersion.HTTP_1_1)
.setLogActivity(true);
HttpClient client = vertx.createHttpClient(options);
client.getNow("google.com", "/", response -> {
System.out.println("Received response with status code " + response.statusCode());
});
System.out.println(getHTML("http://google.com"));
Where getHTML() is from here: How do I do a HTTP GET in Java?
This is my output:
<!doctype html><html... etc <- correct output from plain java
Feb 08, 2017 11:31:21 AM io.vertx.core.http.impl.HttpClientRequestImpl
SEVERE: java.net.UnknownHostException: failed to resolve 'google.com'. Exceeded max queries per resolve 3
But vertx can't connect. What's wrong here? I'm not using any proxy.

For reference: a solution, as described in this question and in tsegismont's comment here, is to set the flag vertx.disableDnsResolver to true:
-Dvertx.disableDnsResolver=true
in order to fall back to the JVM DNS resolver as explained here:
sometimes it can be desirable to use the JVM built-in resolver, the JVM system property -Dvertx.disableDnsResolver=true activates this behavior
I observed this DNS resolution issue with a redis client in a kubernetes environment.

I had this issue, what caused it for me was stale DNS servers being picked up by the Java runtime, i.e. servers registered for a network the machine was no longer connected to. The issue is first in the Sun JNDI implementation, it also exists in Netty which uses JNDI to bootstrap its list of name servers on most platforms, then finally shows up in VertX.
I think a good place to fix this would be in the Netty layer where the set of default DNS servers is bootstrapped. I have raised a ticket with the Netty project so we'll see if they agree with me! Here is the Netty ticket
In the mean time a fairly basic workaround is to filter the default DNS servers detected by Netty, based on whether they are reachable or not. Here is a code Sample in Kotlin to apply before constructing the main VertX instance.
// The default set of name servers provided by JNDI can contain stale entries
// This default set is picked up by Netty and in turn by VertX
// To work around this, we filter for only reachable name servers on startup
val nameServers = DefaultDnsServerAddressStreamProvider.defaultAddressList()
val reachableNameServers = nameServers.stream()
.filter {ns -> ns.address.isReachable(NS_REACHABLE_TIMEOUT)}
.map {ns -> ns.address.hostAddress}
.collect(Collectors.toList())
if (reachableNameServers.size == 0)
throw StartupException("There are no reachable name servers available")
val opts = VertxOptions()
opts.addressResolverOptions.servers = reachableNameServers
// The primary Vertx instance
val vertx = Vertx.vertx(opts)
A little more detail in case it is helpful. I have a company machine, which at some point was connected to the company network by a physical cable. Details of the company's internal name servers were set up by DHCP on the physical interface. Using the wireless interface at home, DNS for the wireless interface gets set to my home DNS while the config for the physical interface is not updated. This is fine since that device is not active, ipconfig /all does not show the internal company DNS servers. However, looking in the registry they are still there:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
They get picked up by the JNDI mechanism, which feeds Netty and in turn VertX. Since they are not reachable from my home location, DNS resolution fails. I can imagine this home/office situation is not unique to me! I don't know whether something similar could occur with multiple virtual interfaces on containers or VMs, it could be worth looking at if you are having problems.

Here is the sample code which works for me.
public class TemplVerticle extends HttpVerticle {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
// Create the web client and enable SSL/TLS with a trust store
WebClient client = WebClient.create(vertx,
new WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
.setDefaultHost("www.w3schools.com")
);
client.get("www.w3schools.com")
.as(BodyCodec.string())
.send(ar -> {
if (ar.succeeded()) {
HttpResponse<String> response = ar.result();
System.out.println("Got HTTP response body");
System.out.println(response.body().toString());
} else {
ar.cause().printStackTrace();
}
});
}
}

Try using web client instead of httpclient, here you have an example (with rx):
private val client: WebClient = WebClient.create(vertx, WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
)
open fun <T> get(uri: String, marshaller: Class<T>): Single<T> {
return client.getAbs(host + uri).rxSend()
.map { extractJson(it, uri, marshaller) }
}
Another option is to use getAbs.

SNI client-side mystery using Java8

I have an Apache web server that runs several TLS virtualhosts with different certs and SNI.
I can access the various virtual hosts just fine using curl (presumably SNI makes it work). I can also access them fine with a little command-line Java program that basically just openConnection()s on a URL.
In my Tomcat application, the basic same client-side code accesses the same Apache server as a client, but always ends up with the default cert (defaulthost.defaultdomain) instead of the cert of the virtual host that was specified in the URL that it attempts to access. (This produces a SunCertPathBuilderException -- basically it can't verify the certificate path to the cert, which of course is true as it is a non-official cert. But then the default cert should not be used anyway.)
It's just as if SNI had been deactivated client-side in my application / Tomcat. I am at a loss why it should behave differently between my app and the command-line; same JDK, same host etc.
I found property jsse.enableSNIExtension, but I verified that it is set to true for both cases. Questions:
Any ideas, even wild ones, why these two programs behave differently?
Any ideas how I would debug this?
This is Arch Linux on 86_64, JDK 8u77, Tomcat 8.0.32.

This answer comes late, but we just have hit the problem (I can't believe it, it seems a very big bug).
All what it said seems true, but it's not default HostnameVerifier the culprit but the troubleshooter. When HttpsClient do afterConnect first try to establish setHost (only when socket is SSLSocketImpl):
SSLSocketFactory factory = sslSocketFactory;
try {
if (!(serverSocket instanceof SSLSocket)) {
s = (SSLSocket)factory.createSocket(serverSocket,
host, port, true);
} else {
s = (SSLSocket)serverSocket;
if (s instanceof SSLSocketImpl) {
((SSLSocketImpl)s).setHost(host);
}
}
} catch (IOException ex) {
// If we fail to connect through the tunnel, try it
// locally, as a last resort. If this doesn't work,
// throw the original exception.
try {
s = (SSLSocket)factory.createSocket(host, port);
} catch (IOException ignored) {
throw ex;
}
}
If you use a custom SSLSocketFactory without override createSocket() (the method without parameters), the createSocket well parametrized is used and all works as expected (with client sni extension). But when second way it's used (try to setHost en SSLSocketImpl) the code executed is:
// ONLY used by HttpsClient to setup the URI specified hostname
//
// Please NOTE that this method MUST be called before calling to
// SSLSocket.setSSLParameters(). Otherwise, the {#code host} parameter
// may override SNIHostName in the customized server name indication.
synchronized public void setHost(String host) {
this.host = host;
this.serverNames =
Utilities.addToSNIServerNameList(this.serverNames, this.host);
}
The comments say all. You need to call setSSLParameters before client handshake. If you use default HostnameVerifier, HttpsClient will call setSSLParameters. But there is no setSSLParameters execution in the opposite way. The fix should be very easy for Oracle:
SSLParameters paramaters = s.getSSLParameters();
if (isDefaultHostnameVerifier) {
// If the HNV is the default from HttpsURLConnection, we
// will do the spoof checks in SSLSocket.
paramaters.setEndpointIdentificationAlgorithm("HTTPS");
needToCheckSpoofing = false;
}
s.setSSLParameters(paramaters);
Java 9 is working as expected in SNI. But they (Oracle) seem not to want fix this:
https://bugs.openjdk.java.net/browse/JDK-8072464
https://bugs.openjdk.java.net/browse/JDK-8144566
and many more

After some hours of debugging the JDK, here is the unfortunate result. This works:
URLConnection c = new URL("https://example.com/").openConnection();
InputStream i = c.getInputStream();
...
This fails:
URLConnection c = new URL("https://example.com/").openConnection();
((HttpsURLConnection)c).setHostnameVerifier( new HostnameVerifier() {
public boolean verify( String s, SSLSession sess ) {
return false; // or true, won't matter for this
}
});
InputStream i = c.getInputStream(); // Exception thrown here
...
Adding the setHostnameVerifier call has the consequence of disabling SNI, although the custom HostnameVerifier is never invoked.
The culprit seems to be this code in sun.net.www.protocol.https.HttpsClient:
if (hv != null) {
String canonicalName = hv.getClass().getCanonicalName();
if (canonicalName != null &&
canonicalName.equalsIgnoreCase(defaultHVCanonicalName)) {
isDefaultHostnameVerifier = true;
}
} else {
// Unlikely to happen! As the behavior is the same as the
// default hostname verifier, so we prefer to let the
// SSLSocket do the spoof checks.
isDefaultHostnameVerifier = true;
}
if (isDefaultHostnameVerifier) {
// If the HNV is the default from HttpsURLConnection, we
// will do the spoof checks in SSLSocket.
SSLParameters paramaters = s.getSSLParameters();
paramaters.setEndpointIdentificationAlgorithm("HTTPS");
s.setSSLParameters(paramaters);
needToCheckSpoofing = false;
}
where some bright mind checks whether the configured HostnameVerifier's class is the default JDK class (which, when invoked, just returns false, like my code above) and based on that, changes the parameters for the SSL connection -- which, as a side effect, turns off SNI.
How checking the name of a class and making some logic depend on it is ever a good idea escapes me. ("Mom! We don't need virtual methods, we can just check the class name and dispatch on that!") But worse, what in the world does SNI have to do with the HostnameVerifier in the first place?
Perhaps the workaround is to use a custom HostnameVerifier with the same name, but different capitalization, because that same bright mind also decided to do case-insensitive name comparison.
'nuff said.

This is a Java 8 bug (JDK-8144566) fixed by 8u141. See Extended server_name (SNI Extension) not sent with jdk1.8.0 but send with jdk1.7.0 for more.

Java websocket with proxy

I have been trying all day and night for couple of days trying to make websocket to work using proxy in Java. I tried different library like
https://github.com/TooTallNate/Java-WebSocket
https://github.com/AsyncHttpClient/async-http-client
But sadly these library doesn't support proxy with credentials. If you guys have known any other library that supports proxy then I would be appreciated.
Thanks in advance

Try nv-websocket-client library. It supports authentication at a proxy server. Note that, however, the current implementation supports Basic Authentication only.
// 1. Create a WebSocketFactory instance.
WebSocketFactory factory = new WebSocketFactory();
// 2. Set up information about a proxy server.
// Credentials can be set here.
ProxySettings settings = factory.getProxySettings();
settings.setServer("http://proxy.example.com");
settings.setCredentials("id", "password");
// 3. Connect to a WebSocket endpoint via the proxy.
WebSocket ws = factory.createSocket("ws://websocket.example.com");
// 4. Add a listener to receive WebSocket events.
ws.addListener(new WebSocketAdapter() {
#Override
public void onTextMessage(WebSocket ws, String message) {
// Received a text message.
......
}
});
// 5. Perform a WebSocket opening handshake.
ws.connect();
// 6. Send frames.
// 6-1. Text
ws.sendText("Hello.");
// 6-2. Binary
byte[] binary = ......;
ws.sendBinary(binary);
// 6-3. Ping
ws.sendPing("Are you there?");
// 6-4. Pong (unsolicited pong; RFC 6455, 5.5.3. Pong)
ws.sendPong("I'm OK.");
// 6-5. Fragmented Frames
ws.sendText("How ", false)
.sendContinuation("are ")
.sendContinuation("you?", true);
// 6-6. Periodical Ping
ws.setPingInterval(60 * 1000);
// 6-7. Periodical Pong (unsolicited pong; RFC 6455, 5.5.3. Pong)
ws.setPongInterval(60 * 1000);
// 6-8. Close (if you want to send one manually).
ws.sendClose(WebSocketCloseCode.NORMAL, "Bye.");
// 7. Disconnect
ws.disconnect();
Blog
WebSocket client library (Java SE 1.5+, Android)
http://darutk-oboegaki.blogspot.jp/2015/05/websocket-client-library-java-se-15.html
GitHub
https://github.com/TakahikoKawasaki/nv-websocket-client
JavaDoc
http://takahikokawasaki.github.io/nv-websocket-client/
Maven
<dependency>
<groupId>com.neovisionaries</groupId>
<artifactId>nv-websocket-client</artifactId>
<version>1.3</version>
</dependency>
The size of nv-websocket-client-1.3.jar is 62,854 bytes and it does not require any external dependencies.

You can try Tyrus (reference implementation of WebSocket API in Java EE); client side does not require any Java EE server to be running and if you are using Java 7, the client could be minimized to ~500kb.
Client behing proxy and Dependencies should provide enough info to try.