Develop Reference

How to check if a folder exists at a regular interval in java

My goal is to check if a Hadoop path exists and if yes, then it should download some file. Now in every five minutes, I want to check if the file exists in the Hadoop path.
currently, my code is checking if the file exists but not on an interval basis.
public boolean checkPathExistance(final org.apache.hadoop.fs.Path hdfsAbsolutePath)
{
final Configuration configuration = getHdfsConfiguration();
try
{
final FileSystem fileSystems = hdfsAbsolutePath.getFileSystem(configuration);
if (fileSystems.exists(hdfsAbsolutePath))
{
return true;
}
}
catch (final IOException e)
{
e.printStackTrace();
}
return false;
}
The criteria are this checkPathExistance method should be called in every five minutes to check if the file exists. And when it will return true, the file should be downloaded.
public void download(final String hdfs, final Path outputPath)
{
final org.apache.hadoop.fs.Path hdfsAbsolutePath = getHdfsFile(hdfsLocalPath).getPath();
logger.info("path check {}", hdfsAbsolutePath.getName());
final boolean isPathExist = checkPathExistance(hdfsAbsolutePath);
downloadFromHDFS(hdfsAbsolutePath, outputPath);
}
Can I please get some help here ?

For the file copying (and not folder copying, if I understood correctly within your question's context) you can just use the copyToLocalFile method from FileSystem as seen here by specifying the boolean that checks if you want to delete the source file, and the input (HDFS)/output (local) paths.
As for the periodic checking of the existence of the file in HDFS, you can use a ScheduledExecutorService object (Java 8 docs here) by specifying that you want your functions' execution to run every 5 minutes.
The following program takes two arguments, the path of the input file in the HDFS and the path of the output file locally.
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.IOException;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class RegularFileCheck
{
public static boolean checkPathExistence(Path inputPath, Configuration conf) throws IOException
{
boolean flag = false;
FileSystem fs = FileSystem.get(conf);
if(fs.exists(inputPath))
flag = true;
return flag;
}
public static void download(Path inputPath, Path outputPath, Configuration conf) throws IOException
{
FileSystem fs = FileSystem.get(conf);
fs.copyToLocalFile(false, inputPath, outputPath); // don't delete the source input file
System.out.println("File copied!");
}
public static void main(String[] args)
{
Path inputPath = new Path(args[0]);
Path outputPath = new Path(args[1]);
Configuration conf = new Configuration();
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
Runnable task = () ->
{
System.out.println("New Timer!");
try
{
if(checkPathExistence(inputPath, conf))
download(inputPath, outputPath, conf);
}
catch (IOException e)
{
e.printStackTrace();
}
};
executor.scheduleWithFixedDelay(task, 0, 5, TimeUnit.MINUTES);
}
}
The console output of course is continuous and looks like the one on the screenshot below (test.txt is the file stored in HDFS, test1.txt is the file to be copied locally). You can additionally modify the code above if you want to stop the re-executions after the file has been found and copied already, or if you want to stop checking for the file after a while.
To stop the search and copying, simply replace to the code above with the following snippet:
Runnable task = () ->
{
System.out.println("New Timer!");
try
{
if(new File(String.valueOf(outputPath)).exists())
System.exit(0);
else if(checkPathExistence(inputPath, conf))
download(inputPath, outputPath, conf);
}
catch (IOException e)
{
e.printStackTrace();
}
};
And the program will stop after the file has been copied, as seen from the console output:

Reading multiple files in directory and printing specific content

What I am trying to achieve is basically a Java file which looks through a specific directory on the users computer, search all the files in the directory for specific word (in this case an email) and then at the end print them out.
The current script of which I have now, looks for all the files in a certain directory, prints out those file names. As well as that I have also figured out how to have that script search through one file for a specific word and then print it out. The only problem is that although it searches through that one file and gets that word/phrase it has to be given the full directory and file to work. I just want it to have a specific directory and then search all the files in it. I have tried doing this using the directory variable of which I have created to find all files, but it does not work when using that as the directory for the files to search through to find the word(s).
Here underneath is the part of my code which is used for the function I want. The actual function is called in my real script so don't worry about that as it is working. I have also just commented in the script what variable I want to work where.
package aProject;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
public class aScanner {
static String usernameMac = System.getProperty("user.name");
final static File foldersMac = new File("/Users/" + usernameMac + "/Library/Mail/V2"); // this is the right directory I want to look through
public static void listFilesForFolder(final File foldersMac) {
for (final File fileEntry : foldersMac.listFiles()) {
if (fileEntry.isDirectory()) {
listFilesForFolder(fileEntry);
try {
BufferedReader bReaderM = new BufferedReader(new FileReader("/Users/username/Library/Mail/V2/AosIMAP-/INBOX.mbox/longnumber-folder/Data/Messages/1.emlx")); //this is where I would like the foldersMac variable to work in, instead of this full directory
String lineMe;
while((lineMe = bReaderM.readLine()) != null)
{
if(lineMe.contains(".com"))
System.out.println(lineMe);
}
bReaderM.close();
}
catch (IOException e) {
}
} else {
System.out.println(fileEntry.getName());
}
}
}
}

I think this is what you're trying to achieve:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
public class aScanner {
static String usernameMac = System.getProperty("user.name");
final static File foldersMac = new File("/Users/" + usernameMac + "/Library/Mail/V2");
public static void main(String[] args) throws IOException {
listFilesForFolder(foldersMac);
}
public static void listFilesForFolder(final File foldersMac) throws IOException {
for (final File fileEntry : foldersMac.listFiles()) {
if (fileEntry.isDirectory()) {
listFilesForFolder(fileEntry);
} else {
ArrayList<String> lines = new ArrayList<>();
try (BufferedReader bReaderM = new BufferedReader(new FileReader(fileEntry))) {
String lineMe;
while ((lineMe = bReaderM.readLine()) != null) {
if (lineMe.contains(".com")) {
lines.add(lineMe);
}
}
}
if (!lines.isEmpty()) {
System.out.println(fileEntry.getAbsolutePath() + ":");
for (String line : lines) {
System.out.println(" " + line.trim());
}
}
}
}
}
}

I think your problem lies around your recursion logic,
You go down recursively in the directory structure, you walk through you tree, but write out nothing cause of this if statement:
if (fileEntry.isDirectory()) {
listFilesForFolder(fileEntry);
...
}
Close that If statement earlier, then it should work.

How to detect disk full during write operation with Java nio?

I would like to write a file that comes from over a network, so I don't know the size of the file that's comming in. Sometimes the disk on the file server might get filled up and I would like return a message to my client notifying them of this error. I couldn't find any documentation on being able to catch this type of i/o error. FileChannel streams bytes from memory to disk, so it may not be trivial to detect this. Is the saving happening asynchronously? Is it possible to detect disk full?
// Create a new file to write to
RandomAccessFile mFile = new RandomAccessFile(this.mFilePath, "rw");
FileChannel mFileChannel = this.mFile.getChannel();
// wrappedBuffer has my file in it
ByteBuffer wrappedBuffer = ByteBuffer.wrap(fileBuffer);
while(wrappedBuffer.hasRemaining()) {
bytesWritten += this.mFileChannel.write(wrappedBuffer, this.mBytesProcessed);
}
I figured in the File class, we can do something like this:
// if there is less than 1 mb left on disk
new File(this.mFilePath, "r").getUsableSpace() < 1024;
But if there a way to throw an except if this.mFileChannel.write() fails because the disk is full?

Even if it's not recommended to parse the error message you could do something like this :
import java.io.IOException;
import java.nio.file.FileStore;
import java.nio.file.FileSystemException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.regex.Pattern;
public class SmallDisk {
final static String SMALL_DISK_PATH = "/Volumes/smallDisk";
final static Pattern NO_SPACE_LEFT = Pattern.compile(": No space left on device$");
public static void main(String[] args) throws NoSpaceException {
Path p = Paths.get(SMALL_DISK_PATH);
FileStore fs = null;
try {
fs = Files.getFileStore(p);
System.out.println(fs.getUsableSpace());
Path newFile = Paths.get(SMALL_DISK_PATH + "/newFile");
Files.createFile(newFile);
} catch (FileSystemException e) {
//We catch the "No space left on device" from the FileSystemException and propagate it
if(NO_SPACE_LEFT.matcher(e.getMessage()).find()){
throw new NoSpaceException("Not enough space");
}
//Propagate exception or deal with it here
} catch (IOException e) {
//Propagate exception or deal with it here
}
}
public static class NoSpaceException extends IOException{
public NoSpaceException(String message) {
super(message);
}
}
}
Another way but it doesn't guaranty that you won't have exception is to use the FileStore to check that you enough space before you write (not enough if you are working with a shared folder or multi threads software)

Java Clipboard availability

When I research things on the internet I like to copy and paste certain paragraphs so I could review them later on.
I'm trying to write a program that would continuously check the clipboard for text content and write it to a text file any time it is renewed.
In the following test of the program I had "public class Clipboard" in my clipboard before running the program and the exception happened when I copied text from netbeans (The IDE I was using to run the program) while the program was running:
import java.awt.Toolkit;
import java.awt.datatransfer.Clipboard;
import java.awt.datatransfer.DataFlavor;
import java.awt.datatransfer.UnsupportedFlavorException;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class TestClipboard {
public static void main(String[] args) {
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String initial = "";
while(true) {
try {
String paste = c.getContents(null).getTransferData(DataFlavor.stringFlavor).toString();
if(!paste.equals(initial)) {
System.out.println(paste);
initial = paste;
}
} catch (UnsupportedFlavorException | IOException ex) {
Logger.getLogger(TestClipboard.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
The output:
public class TestClipboard
Exception in thread "main" java.lang.IllegalStateException: cannot open system clipboard
at sun.awt.windows.WClipboard.openClipboard(Native Method)
at sun.awt.datatransfer.ClipboardTransferable.<init>(ClipboardTransferable.java:78)
at sun.awt.datatransfer.SunClipboard.getContents(SunClipboard.java:144)
at delete.TestClipboard.main(TestClipboard.java:21)
Java Result: 1
BUILD SUCCESSFUL (total time: 34 seconds)
Why can't it open the system clipboard?
Does the getSystemClipboard() method not have global scope? - In other words, can I not get the clipboard's contents if the copy operation was performed in an internet browser?

You appear to be trying to read from the clipboard while another process is updating to it (or some such).
I fixed by:
Requesting an instance of the Clipboard within the loop
Adding a Thread.sleep into the while-loop
For example...
public class TestClipboard {
public static void main(String[] args) {
String initial = "";
while (true) {
try {
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String paste = c.getContents(null).getTransferData(DataFlavor.stringFlavor).toString();
if (!paste.equals(initial)) {
System.out.println(paste);
initial = paste;
}
} catch (UnsupportedFlavorException | IOException ex) {
Logger.getLogger(TestClipboard.class.getName()).log(Level.SEVERE, null, ex);
}
try {
Thread.sleep(40);
} catch (InterruptedException ex) {
}
}
}
}
It should be noted that it won't stop it from happening, it will only reduce the number of occurrences. When it is thrown, you could (just about) ignore and try again...

Run a task based on a trigger java

I want to run a task if there is a trigger (i.e. Some event like new file added to directory) in Java. Does Java have inbuilt support for this?
If not, what third party library I can use to facilitate this?

In Java 7 there is the Watch Service that allows a task to happen when a change or event is detected on a file or directory.
Tutorial: http://docs.oracle.com/javase/tutorial/essential/io/notification.html#overview
API documentation: http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
Here is a quick example I've cooked up:
package watcher;
import java.io.IOException;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
public class Watcher {
private final FileCreatedAction action;
private final String pathToWatchString;
public Watcher(FileCreatedAction action, String pathToWatchString) {
this.action = action;
this.pathToWatchString = pathToWatchString;
}
public void start() throws IOException {
FileSystem defaultFileSystem = FileSystems.getDefault();
WatchService watchService = defaultFileSystem.newWatchService();
Path pathToWatch = defaultFileSystem.getPath(pathToWatchString);
pathToWatch.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
while(true) {
try {
WatchKey key = watchService.take();
if (key != null) {
for (WatchEvent<?> event: key.pollEvents()) {
if (event.kind().equals(StandardWatchEventKinds.ENTRY_CREATE))
{
WatchEvent<Path> ev = (WatchEvent<Path>)event;
Path filename = ev.context();
Path fullFilename = pathToWatch.resolve(filename);
action.performAction(fullFilename);
}
}
}
} catch (InterruptedException error) {
return;
}
}
}
public static void main(String[] args) throws IOException {
FileCreatedAction action = new FileCreatedAction() {
#Override
public void performAction(Path fullPath) {
System.out.printf("Found file %s", fullPath);
}
};
Watcher watcher = new Watcher(action, "/foo");
watcher.start();
}
}
interface FileCreatedAction {
void performAction(Path fullPath);
}

You could easily implement your own file system tracker.
There's a nice, working example in here:
How to watch the file system for changes in Java 7 (JDK 7)

Generally, what you need is a design pattern called 'Observer Pattern'. You can implement your own, without needing any inbuilt support or external frameworks.
For inbuilt support, check Java's 'util' package for 'Observer' and EventListener (since Java 7) interfaces.
Also, check the following links:
1) Generic, annotation-driven event notification frameworks
2) Alternative to Java's Observable class?