Develop Reference

Implementing Spring + Apache Flink project with Postgres

I have a SpringBoot gradle project using apache flink to process datastream signals. When a new signal comes through the datastream, I would like to query look up (i.e. findById() ) it's details using an ID in a postgres database table which is already created in order to get additional information about the signal and enrich the data. I would like to avoid using spring dependencies to perform the lookup (i.e Autowire repository) and want to stick with flink implementation for the lookup.
Where can i specify how to add the postgres connection config information such as port, database, url, username, password etc... (for simplicity purposes can assume the postgres db is local in my machine). Is it as simple as adding the configuration to the application.properties file? if so how can i write the query method to look up the record in the postgres table when searching by non primary key value?
Some online sources are suggesting using this skeleton code but I am not sure how/id it fits my use case. (I have a EventEntity model created which contains all the params/columns from the table which i'm looking up).
like so
public class DatabaseMapper extends RichFlatMapFunction<String, EventEntity> {
// Declare DB connection & query statements
public void open(Configuration parameters) throws Exception {
//Initialize DB connection
//prepare query statements
}
#Override
public void flatMap(String value, Collector<EventEntity> out) throws Exception {
}
}

Your sample code is correct. You can set all your custom initialization and preparation code for PostgreSQL in open() method. Then you can use your pre-configured fields in your flatMap() function.
Here is one sample for Redis operations
I have used RichAsyncFunction here and I suggest you do the same as it is suggested as best practice. Read here for more: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/asyncio.html)
You can pass configuration parameteres in your constructor method and use it in your initialization process
public static class AsyncRedisOperations extends RichAsyncFunction<Object,Object> {
private JedisPool jedisPool;
private Configuration redisConf;
public AsyncRedisOperations(Configuration redisConf) {
this.redisConf = redisConf;
}
#Override
public void open(Configuration parameters) {
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(this.redisConf.getInteger("pool", 8));
jedisPoolConfig.setMaxIdle(this.redisConf.getInteger("pool", 8));
jedisPoolConfig.setMaxWaitMillis(this.redisConf.getInteger("maxWait", 0));
JedisPool jedisPool = new JedisPool(jedisPoolConfig,
this.redisConf.getString("host", "192.168.10.10"),
this.redisConf.getInteger("port", 6379), 5000);
try {
this.jedisPool = jedisPool;
this.logger.info("Redis connected: " + jedisPool.getResource().isConnected());
} catch (Exception e) {
this.logger.error(BaseUtil.append("Exception while connecting Redis"));
}
}
#Override
public void asyncInvoke(Object in, ResultFuture<Object> out) {
try (Jedis jedis = this.jedisPool.getResource()) {
String key = jedis.get(key);
this.logger.info("Redis Key: " + key);
}
}
}

How to block SFTP remove operations with Apache MINA SSHD

I am trying to create a custom sftp server using Apache Mina SSHD. My code so far:
SshServer sshd = SshServer.setUpDefaultServer();
sshd.setPort(PORT_NUMBER);
sshd.setKeyPairProvider(new SimpleGeneratorHostKeyProvider(Paths.get("keys/private_key.ppk")));
SftpSubsystemFactory factory = new SftpSubsystemFactory.Builder()
.build();
factory.addSftpEventListener(new BasicSftpEventListener());
sshd.setSubsystemFactories(Collections.singletonList(factory));
sshd.setShellFactory(new ProcessShellFactory("/bin/sh", "-i", "-l"));
sshd.start();
As you can see, I implemented my own SftpEventListener:
public class BasicSftpEventListener implements SftpEventListener {
#Override
public void removing(ServerSession session, Path path) throws IOException {
System.out.println("Removin");
}
#Override
public void removed(ServerSession session, Path path, Throwable thrown) throws IOException {
System.out.println("removed");
}
When I want to remove file, it executes my removing and removed listeners, BUT the remove operation proceeds and the file is deleted.
Is there a way how to stop this from happening?
Thanks for help!

If you want to block delete actions, you will need to interrupt the flow of the removing method with an exception. This will tell Mina to stop and not remove the file. I would recommend using java.lang.UnsupportedOperationException for this:
#Override
public void removing(ServerSession session, Path path) throws UnsupportedOperationException{
throw new UnsupportedActionException("Removing files is not permitted.");
}

Jetty WebSocket Client Doesn't work for Binary Listener

I have a Java app that uses the Jetty WebSocket Client, version 9.x. It works fine for text messages sent from the server, but the binary listener is never invoked. I have a Javascript client implementation which I'm basically duplicating. I'm doing the same exact thing in Javascript that I do in Java, calling the same server. The Javascript works, and the Java doesn't. So I'm thinking that something is not configured properly in Jetty for binary listeners.
For example, the server is sending blob data. I know that in the Javascript client, I can set the binarytype to either arraybuffer or blob. I figured there may be a similar setting required in Jetty, but I've looked all through the API and searched many examples online. There are precious few examples of binary listeners online, and no mention anywhere of setting the binarytype, or any other special setting required to make binary llisteners work.
Here's a consolidated representation of my code. The code is spread throughout various classes, so this is not a stand-alone app, but I think it shows what I'm doing. The server is implemented with libwebsockets.
Client implementation
import org.eclipse.jetty.websocket.client.WebSocketClient;
import org.eclipse.jetty.websocket.client.ClientUpgradeRequest;
client = new WebSocketClient();
client.start();
client.setMaxBinaryMessageBufferSize((int) 500e6);//just to be sure
ClientUpgradeRequest request = new ClientUpgradeRequest();
request.setSubProtocols("pipe-data");
client = new SimpleSocket();
client.connect(socket, uri, request);
Socket implementation
#WebSocket
public class SimpleSocket {
#SuppressWarnings("unused")
private Session session;
private SocketHandlerBase handler;
private boolean connected = false;
public SimpleSocket(SocketHandlerBase listener) {
this.handler = listener;
}
#OnWebSocketClose
public void onClose(int statusCode, String reason) {
this.handler.onClose(statusCode, reason);
this.connected = false;
}
#OnWebSocketConnect
public void onConnect(Session session) {
this.handler.onConnect(session);
this.connected = true;
}
//invoked when text messages are sent
#OnWebSocketMessage
public void onMessage(String msg) {
this.handler.onMessage(msg);
}
//does not get invoked when binary data is sent
#OnWebSocketMessage
public void onMessage(byte buf[], int offset, int length) {
this.handler.onMessage(buf, offset, length);
}
public boolean isConnected() {
return this.connected;
}
public SocketHandlerBase getHandler() {
return this.handler;
}
}

There was a hard to find problem with the server I was calling. A very specific configuration of invocation arguments was causing the binary listener to not be called. Nothing about the Jetty client or WebSockets in general involved here.

Java Integration Testing of proxy related code

I am developping a Swing application that needs to communicate with a distant HTTP server. That application can be potentially used behind proxies.
I must then :
- detect automatically network proxy (potentially several on the same network)
- let the user manually enter a proxy configuration.
I want to write an integration test to validate thoses aspects, without having to install proxies on every CI machines and every developper machine.
How I see things :
integration test (with junit) start an "embedded" proxy (#BeforeClass) and a somewhat dummy http server
my tests (#Test)
test that this proxy can be detected automatically and open a connection to my dummy http server and successfully retrieve datas from it
manually set the proxy and perform the same test as above
I have heard about the "littleProxy" component but didn"t tried it yet.
Can anyone shed some advice / help / guidance regarding the best approach to solve my problem ?

I would consider whether you are testing the right thing. You don't need to test proxy servers or Java's network classes.
Consider this utility type for reading data from a URL:
public final class Network {
public interface UrlOpener {
public InputStream open(URL url) throws IOException;
}
private static UrlOpener urlOpener = new UrlOpener() {
public InputStream open(URL url) throws IOException {
return url.openStream();
}
};
public static InputStream openUrl(URL url) throws IOException {
return urlOpener.open(url);
}
public static void setUrlOpener(UrlOpener urlOpener) {
Network.urlOpener = urlOpener;
}
}
This can be used as an abstraction layer between your code and Java's network I/O:
public class SomeType {
public void processData(URL url) throws IOException {
InputStream input = Network.openUrl(url);
// process the data
}
}
Your tests use this layer to mock out the data:
#Before public void setup() throws IOException {
final URL mockUrl = this.getClass().getResource("/foo/bar.txt");
Network.UrlOpener opener = Mockito.mock(Network.UrlOpener.class);
Answer<InputStream> substituteUrl = new Answer<InputStream>() {
public InputStream answer(InvocationOnMock invocation) throws Throwable {
return mockUrl.openStream();
}
};
Mockito.when(opener.open(Mockito.any(URL.class))).then(substituteUrl);
Network.setUrlOpener(opener);
}
#Test public void testSomething() throws IOException {
SomeType something = new SomeType();
something.processData(new URL("http://example.com"));
}
This saves any mucking around with firewalls etc.
In order for this approach to work you would want to have confidence in a set of recorded transactions from real servers to use in your tests.
This approach can be complemented with a dedicated machine running more comprehensive integration tests.

How do you set the configuration for jschconfigsessionfactory for jgit so that pull and push work?

I am trying to do a git pull/push using jgit's api with the following code
org.eclipse.jgit.api.Git.open(theRepoFile).pull().call()
but I am getting exceptions
JSchException Auth fail
com.jcraft.jsch.Session.connect (Session.java:461)
org.eclipse.jgit.transport.JschConfigSessionFactory.getSession (JschConfigSessionFactory.java:116)
org.eclipse.jgit.transport.SshTransport.getSession (SshTransport.java:121)
org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init> (TransportGitSsh.java:306)
org.eclipse.jgit.transport.TransportGitSsh.openPush (TransportGitSsh.java:152)
org.eclipse.jgit.transport.PushProcess.execute (PushProcess.java:130)
org.eclipse.jgit.transport.Transport.push (Transport.java:1127)
org.eclipse.jgit.api.PushCommand.call (PushCommand.java:153)
Even though using cgit pull and pushing works.
I tried checking SO for example code
Java git client using jgit
but the above question does not provide a complete coded example of what is necessary to do a git pull with a remote repo that is normally authenticated via ssh keys. There should be a way to get the credential information from ~/.ssh/ or the windows equivalent.

Jsch will automatically detect your SSH keys but will fail if these are protected by a password. You need to specify the passphrase through a CredentialsProvider like this:
JschConfigSessionFactory sessionFactory = new JschConfigSessionFactory() {
#Override
protected void configure(OpenSshConfig.Host hc, Session session) {
CredentialsProvider provider = new CredentialsProvider() {
#Override
public boolean isInteractive() {
return false;
}
#Override
public boolean supports(CredentialItem... items) {
return true;
}
#Override
public boolean get(URIish uri, CredentialItem... items) throws UnsupportedCredentialItem {
for (CredentialItem item : items) {
((CredentialItem.StringType) item).setValue("yourpassphrase");
}
return true;
}
};
UserInfo userInfo = new CredentialsProviderUserInfo(session, provider);
session.setUserInfo(userInfo);
}
};
SshSessionFactory.setInstance(sessionFactory);

The problem is Jsch does not support ssh-agents out of the box. One will need to configure https://github.com/ymnk/jsch-agent-proxy to get it to work.
An alternative is to make your own org.eclipse.jgit.transport.CredentialsProvider and set the org.eclipse.jgit.transport.CredentialItem to the correct values (by requesting them from the user or looking up a file). You can change the default CredentialsProvider with org.eclipse.jgit.transport.CredentialsProvider/setDefault
See my clojure library dj for details: https://github.com/bmillare/dj/blob/library/src/dj/git.clj

I vaguely remember getting an error with JSch that blocked me for a while because the log was not very explicit. I can't tell for sure it's the same problem but I followed this page to solve my problem:
https://help.github.com/articles/generating-ssh-keys
(it was due to a wrong network configuration)