Develop Reference

Is there a way to get the Apache Commons FileAlterationMonitor to only alert once for a batch of incoming files?

I am monitoring several (about 15) paths for incoming files using the Apache Commons FileAlterationMonitor. These incoming files can come in batches of anywhere between 1 and 500 files at a time. I have everything set up and the application monitors the folders as expected, I have it set to poll the folders every minute. My issue is that, as expected, the listener that I have set up alerts for each incoming file when all I really need, and want, is to know when a new batch of files come in. So I would like to receive a single alert as opposed to up to 500 at a time.
Does anyone have any ideas for how to control the number of alerts or only pick up the first or last notification or something to that effect? I would like to stick with the FileAlterationMonitor if at all possible because it will be running for long periods and so far from what I can tell in testing is that it doesn't seem to put a heavy load on the system or slow the rest of the application down. But I am definitely open to other ideas if what I'm looking for isn't possible with the FileAlterationMonitor.
public class FileMonitor{
private final String newDirectory;
private FileAlterationMonitor monitor;
private final Alerts gui;
private final String provider;
public FileMonitor (String d, Alerts g, String pro) throws Exception{
newDirectory = d;
gui = g;
provider = pro;
}
public void startMonitor() throws Exception{
// Directory to monitor
final File directory = new File(newDirectory);
// create new observer
FileAlterationObserver fao = new FileAlterationObserver(directory);
// add listener to observer
fao.addListener(new FileAlterationListenerImpl(gui, provider));
// wait 1 minute between folder polls.
monitor = new FileAlterationMonitor(60000);
monitor.addObserver(fao);
monitor.start();
}
}
public class FileAlterationListenerImpl implements FileAlterationListener{
private final Alerts gui;
private final String provider;
private final LogFiles monitorLogs;
public FileAlterationListenerImpl(Alerts g, String pro){
gui = g;
provider = pro;
monitorLogs = new LogFiles();
}
#Override
public void onStart(final FileAlterationObserver observer){
System.out.println("The FileListener has started on: " + observer.getDirectory().getAbsolutePath());
}
#Override
public void onDirectoryCreate(File file) {
}
#Override
public void onDirectoryChange(File file) {
}
#Override
public void onDirectoryDelete(File file) {
}
#Override
public void onFileCreate(File file) {
try{
switch (provider){
case "Spectrum": gui.alertsAreaAppend("New/Updated schedules available for Spectrum zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for Spectrum zones!\r\n");
break;
case "DirecTV ZTA": gui.alertsAreaAppend("New/Updated schedules available for DirecTV ZTA zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for DirecTV ZTA zones!\r\n");
break;
case "DirecTV RSN": gui.alertsAreaAppend("New/Updated schedules available for DirecTV RSN zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for DirecTV RSN zones!\r\n");
break;
case "Suddenlink": gui.alertsAreaAppend("New/Updated schedules available for Suddenlink zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for Suddenlink zones!\r\n");
break;
}
}catch (IOException e){}
}
#Override
public void onFileChange(File file) {
}
Above is the FileMonitor class and overridden FileAlterationListener I have so far.
Any suggestions would be greatly appreciated.

Here's a quick and crude implementation:
public class FileAlterationListenerAlterThrottler {
private static final int DEFAULT_THRESHOLD_MS = 5000;
private final int thresholdMs;
private final Map<String, Long> providerLastFileProcessedAt = new HashMap<>();
public FileAlterationListenerAlterThrottler() {
this(DEFAULT_THRESHOLD_MS);
}
public FileAlterationListenerAlterThrottler(int thresholdMs) {
this.thresholdMs = thresholdMs;
}
public synchronized boolean shouldAlertFor(String provider) {
long now = System.currentTimeMillis();
long last = providerLastFileProcessedAt.computeIfAbsent(provider, x -> 0l);
if (now - last < thresholdMs) {
return false;
}
providerLastFileProcessedAt.put(provider, now);
return true;
}
}
And a quicker and cruder driver:
public class Test {
public static void main(String[] args) throws Exception {
int myThreshold = 1000;
FileAlterationListenerAlterThrottler throttler = new FileAlterationListenerAlterThrottler(myThreshold);
for (int i = 0; i < 3; i++) {
doIt(throttler);
}
Thread.sleep(1500);
doIt(throttler);
}
private static void doIt(FileAlterationListenerAlterThrottler throttler) {
boolean shouldAlert = throttler.shouldAlertFor("Some Provider");
System.out.println("Time now: " + System.currentTimeMillis());
System.out.println("Should alert? " + shouldAlert);
System.out.println();
}
}
Yields:
Time now: 1553739126557
Should alert? true
Time now: 1553739126557
Should alert? false
Time now: 1553739126557
Should alert? false
Time now: 1553739128058
Should alert? true

RxNetty reuse the connection

I want to use Netflix-Ribbon as TCP client load balancer without Spring Cloud,and i write test code.
public class App implements Runnable
{
public static String msg = "hello world";
public BaseLoadBalancer lb;
public RxClient<ByteBuf, ByteBuf > client;
public Server echo;
App(){
lb = new BaseLoadBalancer();
echo = new Server("localhost", 8000);
lb.setServersList(Lists.newArrayList(echo));
DefaultClientConfigImpl impl = DefaultClientConfigImpl.getClientConfigWithDefaultValues();
client = RibbonTransport.newTcpClient(lb, impl);
}
public static void main( String[] args ) throws Exception
{
for( int i = 40; i > 0; i--)
{
Thread t = new Thread(new App());
t.start();
t.join();
}
System.out.println("Main thread is finished");
}
public String sendAndRecvByRibbon(final String data)
{
String response = "";
try {
response = client.connect().flatMap(new Func1<ObservableConnection<ByteBuf, ByteBuf>,
Observable<ByteBuf>>() {
public Observable<ByteBuf> call(ObservableConnection<ByteBuf, ByteBuf> connection) {
connection.writeStringAndFlush(data);
return connection.getInput();
}
}).timeout(1, TimeUnit.SECONDS).retry(1).take(1)
.map(new Func1<ByteBuf, String>() {
public String call(ByteBuf ByteBuf) {
return ByteBuf.toString(Charset.defaultCharset());
}
})
.toBlocking()
.first();
}
catch (Exception e) {
System.out.println(((LoadBalancingRxClientWithPoolOptions) client).getMaxConcurrentRequests());
System.out.println(lb.getLoadBalancerStats());
}
return response;
}
public void run() {
for (int i = 0; i < 200; i++) {
sendAndRecvByRibbon(msg);
}
}
}
i find it will create a new socket everytime i callsendAndRecvByRibbon even though the poolEnabled is setting to true. So,it confuse me,i miss something?
and there are no option to configure the size of the pool,but hava a PoolMaxThreads and MaxConnectionsPerHost.
My question is how to use a connection pool in my simple code, and what's wrong with my sendAndRecvByRibbon,it open a socket then use it only once,how can i reuse the connection?thanks for your time.
the server is just a simple echo server writing in pyhton3,i comment outconn.close() because i want to use long connection.
import socket
import threading
import time
import socketserver
class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
def handle(self):
conn = self.request
while True:
client_data = conn.recv(1024)
if not client_data:
time.sleep(5)
conn.sendall(client_data)
# conn.close()
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
if __name__ == "__main__":
HOST, PORT = "localhost", 8000
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
ip, port = server.server_address
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
server.serve_forever()
and the pom of mevan,i just add two dependency in IED's auto generated POM.
<dependency>
<groupId>commons-configuration</groupId>
<artifactId>commons-configuration</artifactId>
<version>1.6</version>
</dependency>
<dependency>
<groupId>com.netflix.ribbon</groupId>
<artifactId>ribbon</artifactId>
<version>2.2.2</version>
</dependency>
the code for printing src_port
#Sharable
public class InHandle extends ChannelInboundHandlerAdapter {
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
System.out.println(ctx.channel().localAddress());
super.channelRead(ctx, msg);
}
}
public class Pipeline implements PipelineConfigurator<ByteBuf, ByteBuf> {
public InHandle handler;
Pipeline() {
handler = new InHandle();
}
public void configureNewPipeline(ChannelPipeline pipeline) {
pipeline.addFirst(handler);
}
}
and change the client = RibbonTransport.newTcpClient(lb, impl);to Pipeline pipe = new Pipeline();client = RibbonTransport.newTcpClient(lb, pipe, impl, new DefaultLoadBalancerRetryHandler(impl));

So, your App() constructor does the initialization of lb/client/etc.
Then you're starting 40 different threads with 40 different RxClient instances (each instance has own pool by default) by calling new App() in the first for loop. To make things clear - the way you spawn multiple RxClient instances here does not allow them to share any common pool. Try to use one RxClient instance instead.
What if you change your main method like below, does it stop creating extra sockets?
public static void main( String[] args ) throws Exception
{
App app = new App() // Create things just once
for( int i = 40; i > 0; i--)
{
Thread t = new Thread(()->app.run()); // pass the run()
t.start();
t.join();
}
System.out.println("Main thread is finished");
}
If above does not help fully (at least it will reduce created sockets count in 40 times) - can you please clarify how exactly do you determine that:
i find it will create a new socket everytime i call sendAndRecvByRibbon
and what are your measurements after you update constructor with this line:
DefaultClientConfigImpl impl = DefaultClientConfigImpl.getClientConfigWithDefaultValues();
impl.set(CommonClientConfigKey.PoolMaxThreads,1); //Add this one and test
Update
Yes, looking at the sendAndRecvByRibbon it seems that it lacks marking the PooledConnection as no longer acquired by calling close once you don't expect any further reads from it.
As long as you expect the only single read event, just change this line
connection.getInput()
to the
return connection.getInput().zipWith(Observable.just(connection), new Func2<ByteBuf, ObservableConnection<ByteBuf, ByteBuf>, ByteBuf>() {
#Override
public ByteBuf call(ByteBuf byteBuf, ObservableConnection<ByteBuf, ByteBuf> conn) {
conn.close();
return byteBuf;
}
});
Note, that if you'd design more complex protocol over TCP, then input bytebuf can be analyzed for your specific 'end of communication' sign which indicates the connection can be returned to the pool.

Using Javafx TimeLine In a Thread

I'm trying to create an internet checker class, that will check the connection to a certain url and update the status property accordingly. To avoid ui freeze i want to use a thread, and a timer to recheck after a certain interval. Problem is the CHECK method call in the timeline keyframe is called from the FX thread still. How can i use a timeline inside a thread?
CODE:
public class InternetChecker {
private String baseUrl;
/***Properties***/
private ObjectProperty<Status> status = new SimpleObjectProperty<>(Status.ACTIVE);
/****************************************************************
********** CONSTRUCTORS ************
****************************************************************/
public InternetChecker(String baseUrl) {
this(baseUrl, 1000);
}
public InternetChecker(String baseUrl, int millisCheckInterval) {
this.baseUrl = baseUrl;
new Thread(() -> {
Timeline timelineCheck = new Timeline();
timelineCheck.getKeyFrames().add(
new KeyFrame(Duration.millis(millisCheckInterval), e -> {
check();
}));
timelineCheck.setCycleCount(Animation.INDEFINITE);
timelineCheck.play();
}).start();
}
/*******************************
* Will check if there is an internet connection present
* and update the status accordingly
*******************************/
public void check() {
// Check if base internet connection
// is working, if it is we continue
// to see if domain connection is working
try {
if ("127.0.0.1".equals(InetAddress.getLocalHost().getHostAddress())) {
setStatus(Status.INTERNET_DISCONNECTED);
return;
}
} catch (UnknownHostException e) {
throw new RuntimeException(e);
}
// Check if base domain connection is working
try {
final URL url = new URL(baseUrl);
final URLConnection conn = url.openConnection();
conn.connect();
conn.getInputStream().close();
setStatus(Status.ACTIVE);
} catch (MalformedURLException e) {
throw new RuntimeException(e);
} catch (IOException e) {
setStatus(Status.BASE_URL_UNREACHABLE);
}
}
/****************************************************************
********** ACCESSORS ************
****************************************************************/
public Status getStatus() {
return status.get();
}
public ObjectProperty<Status> statusProperty() {
return status;
}
private void setStatus(Status status) {
this.status.set(status);
}
/*******************************
* ACTIVE (Base url reachable)
* BASE_URL_UNREACHABLE (Internet available, but base url is unreachable)
* INTERNET_DISCONNECTED (Internet is not available)
********************************/
public enum Status {
ACTIVE,
BASE_URL_UNREACHABLE,
INTERNET_DISCONNECTED;
}
}

Since you need to do a periodic background task that communicates with the JavaFX Application Thread it would be better to use a ScheduledService. This class executes a (new) Task periodically using an Executor that can be defined by the developer. Note that ScheduledService extends javafx.concurrent.Service.
Here is a skeleton example of what you'd need to do to implement this:
import javafx.beans.property.SimpleStringProperty;
import javafx.beans.property.StringProperty;
import javafx.concurrent.ScheduledService;
import javafx.concurrent.Task;
public class ConnectionStatusService extends ScheduledService<Status> {
// Property allows you to change the "baseUrl" between executions
private final StringProperty baseUrl = new SimpleStringProperty(this, "baseUrl");
// property getter and setters omitted...
#Override
protected Task<Status> createTask() {
// creates a new Task and gives the current "baseUrl"
// as an argument. This is called every cycle
return new ConnectionStatusTask(getBaseUrl());
}
private static class ConnectionStatusTask extends Task<Status> {
// A Task is a one-shot thing and its initial state should be
// immutable (or at least encapsulated from external modification).
private final String baseUrl;
private ConnectionStatusTask(String baseUrl) {
this.baseUrl = baseUrl;
}
#Override
protected Status call() throws Exception {
// Do what you need to determine connection status
return computedStatus;
}
}
}
Then you'd listen/bind to the lastValue property.
public void initService() {
ConnectionStatusService service = new ConnectionStatusService();
service.setBaseUrl(/* your URL */);
service.setPeriod(Duration.seconds(1)); // run every 1 seconds
service.lastValueProperty().addListener(/* your listener */); // or bind to this property
// you may also want to add EventHandlers/Listeners to handle when the
// service fails and such.
service.start();
}
It's important you observe the lastValue property and not the value property. The reason is given in the Javadoc of lastValue:
The last successfully computed value. During each iteration, the
"value" of the ScheduledService will be reset to null, as with any
other Service. The "lastValue" however will be set to the most
recently successfully computed value, even across iterations. It is
reset however whenever you manually call reset or restart.
I recommend reading the Javadoc of Task, Service, and ScheduledService for more information. All three of these classes implement the javafx.concurrent.Worker interface.

You only want a single statement to be executed on the JavaFX application thread and that is status.set(status);. Since you're planing to run this statement with some delay in between, you can simply use Platform.runLater to do this.
As for repeatedly executing the check: ScheduledExecutorService is designed for this purpose.
public class InternetChecker implements Runnable {
private final String baseUrl;
/***Properties***/
// use readonly wrapper here to restrict outside access to the property
private final ReadOnlyObjectWrapper<Status> status = new ReadOnlyObjectWrapper<>(Status.ACTIVE);
/****************************************************************
********** CONSTRUCTORS ************
****************************************************************/
public InternetChecker(String baseUrl) {
this.baseUrl = baseUrl;
}
/*******************************
* Will check if there is an internet connection present
* and update the status accordingly
*******************************/
#Override
public void run() {
// Check if base internet connection
// is working, if it is we continue
// to see if domain connection is working
try {
if ("127.0.0.1".equals(InetAddress.getLocalHost().getHostAddress())) {
setStatus(Status.INTERNET_DISCONNECTED);
return;
}
} catch (UnknownHostException e) {
throw new RuntimeException(e);
}
// Check if base domain connection is working
try {
final URL url = new URL(baseUrl);
final URLConnection conn = url.openConnection();
conn.connect();
conn.getInputStream().close();
setStatus(Status.ACTIVE);
} catch (MalformedURLException e) {
throw new RuntimeException(e);
} catch (IOException e) {
setStatus(Status.BASE_URL_UNREACHABLE);
}
}
/****************************************************************
********** ACCESSORS ************
****************************************************************/
public Status getStatus() {
return status.get();
}
public ReadOnlyObjectProperty<Status> statusProperty() {
return status.getReadOnlyProperty​();
}
private void setStatus(final Status status) {
Platform.runLater(() -> this.status.set(status));
}
/*******************************
* ACTIVE (Base url reachable)
* BASE_URL_UNREACHABLE (Internet available, but base url is unreachable)
* INTERNET_DISCONNECTED (Internet is not available)
********************************/
public enum Status {
ACTIVE,
BASE_URL_UNREACHABLE,
INTERNET_DISCONNECTED;
}
}
InternetChecker checker = new InternetChecker(url);
ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor​();
// use delay here to avoid race condition
executorService.scheduleAtFixedDelay(checker, 0, millisCheckInterval, TimeUnit.MILLISECONDS);
Note that you need to shut down the service "manually" or use a ThreadFactory returning daemon threads:
ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor​(r -> {
Thread t = new Thread(r);
t.setDaemon(true);
return t;
});

Java - Runnable, lambda function, and methods of class

I'm quite new with Java (studied on University but was version 2).
Now I've developed an application that downloads files from s3 in parallel. I've used ExecutorService and Runnable to download multiple files in parallel in this way:
public class DownloaderController {
private AmazonS3 s3Client;
private ExecutorService fixedPool;
private TransferManager dlManager;
private List<MultipleFileDownload> downloads = new ArrayList<>();
public DownloaderController() {
checkForNewWork();
}
public void checkForNewWork(){
Provider1 provider = new Provider1();
fixedPool = Executors.newFixedThreadPool(4);
List<Download> providedDownloadList = provider.toBeDownloaded();
for (Download temp : providedDownloadList) {
if (!downloadData.contains(temp)) {
fixedPool.submit(download.downloadCompletedHandler(s3Client));
}
}
}
}
public void printToTextArea(String msg){
Date now = new Date();
if ( !DateUtils.isSameDay(this.lastLogged, now)){
this._doLogRotate();
}
this.lastLogged = now;
SimpleDateFormat ft = new SimpleDateFormat("dd/MM/yyyy H:mm:ss");
String output = "[ " + ft.format(now) + " ] " + msg + System.getProperty("line.separator");
Platform.runLater(() -> {
//this is a FXML object
statusTextArea.appendText(output);
});
}
}
public class Provider1 implements downloadProvider {
}
public class Download {
abstract Runnable downloadCompletedHandler(AmazonS3 s3Client);
}
public class DownloadProvider1 extends Download {
#Override
public Runnable downloadCompletedHandler(AmazonS3 s3Client){
Runnable downloadwork = () -> {
ObjectListing list = s3Client.listObjects(this.bucket,this.getFolder());
List<S3ObjectSummary> objects = list.getObjectSummaries();
AtomicLong workSize = new AtomicLong(0);
List<DeleteObjectsRequest.KeyVersion> keys = new ArrayList<>();
objects.forEach(obj -> {
workSize.getAndAdd(obj.getSize());
keys.add((new DeleteObjectsRequest.KeyVersion(obj.getKey())));
});
MultipleFileDownload fileDownload = dlManager.downloadDirectory("myBucket","folder","outputDirectory");
try {
fileDownload.waitForCompletion();
} catch (Exception e){
printToTextArea("Exception while download from AmazonS3");
}
};
return downloadwork;
}
}
In the downloadController i call every minute a function that adds some Download objects to a List that contains folders that has to be downloaded from s3. when a new Download is added it's also added to ExecutorService pool. The Download object returns the code that has to be executed to download the folder from s3 and what to do when it's download is finished.
My problem is, what is the best way to communicate between the Runnable and the DownloadController ?

Your code does not make entirely clear what the goal is. From what I understand, I would have done it something like this:
public class Download {
private AmazonS3 s3Client;
public Download(AmazonS2 client) { s3Client = client; }
public void run() { // perform download }
}
That class does nothing but download the file (cfg Separation of Concern) and is a Runnable. You can do executorService.submit(new Download(client)) and the download will be finished eventually; also, you can test it without being called concurrently.
Now, you want a callback method for logging it being finished.
public class LoggingCallback {
public void log() {
System.out.println("finished");
}
}
Also a Runnable (the method doesn't have to be run()).
And, to make sure it's triggered one after the other, maybe
class OneAfterTheOther {
private Runnable first;
private Runnable second;
public OneAfterTheOther(Runnable r1, Runnable r2) {
first = r1; second = r2;
}
public void run() { first.run(); second.run(); }
}
which if submitted like this
Download dl = new Download(client);
Logger l = new LoggingCallback();
executorService.submit(new OneAfterTheOther(dl::run, l::log));
will do what I think you're trying to do.

Thread's run does not see its array size increase

I am currently working on a Java homework. I am asked to create a basic DNS server.
There is an UDPSender class which is a thread listening on port 53.
There is also another thread which is called UDPManager.
UDPManager starts a thread with a nested runnable class which holds an ArrayList of DatagramPacket. The UDPSender aggregates the UDPManager and whenever it receives an UDP packet, it sends it to the manager for him to add it to the arrayList.
import java.net.DatagramPacket;
import java.util.ArrayList;
import java.util.HashMap;
public class UDPManager {
private UDPManagerRunnable manager;
public UDPManager(String hostsFile, String remoteDNS, boolean localResolution) {
manager = new UDPManagerRunnable(hostsFile, remoteDNS, localResolution);
new Thread(manager).start();
}
public void managePacket(DatagramPacket p) {
manager.managePacket(p);
}
public void close() {
manager.close();
}
private class UDPManagerRunnable implements Runnable {
private ArrayList<DatagramPacket> packets;
private HashMap<Integer, String> clients;
private boolean localResolution;
private boolean running;
private String hostsFile;
private String remoteDNS;
public UDPManagerRunnable(String hostsFile, String remoteDNS, boolean localResolution) {
packets = new ArrayList<DatagramPacket>();
clients = new HashMap<Integer, String>();
this.localResolution = localResolution;
this.running = true;
this.hostsFile = hostsFile;
this.remoteDNS = remoteDNS;
}
public void managePacket(DatagramPacket p) {
packets.add(p);
System.out.println("Received packet. "+packets.size());
}
public void close() {
running = false;
}
public void run() {
DatagramPacket currentPacket = null;
while(running) {
if(!packets.isEmpty()) {
currentPacket = packets.remove(0);
byte[] data = currentPacket.getData();
int anCountValue = data[Constant.ANCOUNT_BYTE_INDEX];
if(anCountValue == Constant.ANCOUNT_REQUEST)
this.processRequest(currentPacket);
else if(anCountValue == Constant.ANCOUNT_ONE_ANSWER)
this.processResponse(currentPacket);
}
}
}
private void processRequest(DatagramPacket packet) {
System.out.println("it's a request!");
}
private void processResponse(DatagramPacket packet) {
System.out.println("it's a response!");
}
}
}
This is the UDPManager. The packets are transmitted to the manager correctly as the System.out.println correctly displays "Received packet." and the size of the array does increase. The problem I'm running into is that inside the "run()" it never see the size increasing. The weird thing is that it works perfectly fine in debug.
Any idea why it's acting this way?
Thanks a lot for your help.

The problem is, that your first thread is putting the new data into the packets variable, but for the second thread this is not visible. You should synchronize the access to the array.
When you start a second thread all variables are copied. The second thread is only working on the copies. You need to synchronize access to this variables, so changes are made visible to the other threads.

you should synchronize packets when you access or modify it