Scan a set of directories continuously for a set of file name filters.
For each file name filter arrived, process the file and repeat the steps for all
What can be the recommended design for this in jdk 1.5 , possibly using java.concurrent.Executor and Future
I have done a similar task with the web crawler.Just a few changes had to be made... It is a concurrent implementation with newly found directories getting scanned by the thread pool in the Executor Framework.It uses concurrent collections for Queue and List to hold the indexed files. The indexer picks up the files from the queue and does something with them.
here is the FileFilter implementation
public class ImageFileFilter implements FileFilter
{
private final String[] okFileExtensions =
new String[] {"jpg", "png", "gif"};
public boolean accept(File file)
{
for (String extension : okFileExtensions)
{
if (file.getName().toLowerCase().endsWith(extension))
{
return true;
}
}
return false;
}
}
here is the Class with the main method...
public class FileFilterTest {
public static void main(String[] args) {
File dir = new File("D:\\dev\\css-templates\\cms-admin");
BlockingQueue blockingQueue = new ArrayBlockingQueue(5);
FileCrawler fileCrawler = new FileCrawler(blockingQueue,
new ImageFileFilter(), dir);
new Thread(fileCrawler).start();
FileIndexer indexer = new FileIndexer(blockingQueue);
new Thread(indexer).start();
}
}
Here is the file crawler thread
public class FileCrawler implements Runnable {
private final BlockingQueue fileQueue;
private ConcurrentSkipListSet indexedFiles = new ConcurrentSkipListSet();
private final FileFilter fileFilter;
private final File root;
private final ExecutorService exec = Executors.newCachedThreadPool();
public FileCrawler(BlockingQueue fileQueue,
final FileFilter fileFilter,
File root) {
this.fileQueue = fileQueue;
this.root = root;
this.fileFilter = new FileFilter() {
public boolean accept(File f) {
return f.isDirectory() || fileFilter.accept(f);
}
};
}
public void run() {
submitCrawlTask(root);
}
private void submitCrawlTask(File f) {
CrawlTask crawlTask = new CrawlTask(f);
exec.execute(crawlTask);
}
private class CrawlTask implements Runnable {
private final File file;
CrawlTask(File file ) {
this.file= file;
}
public void run() {
if(Thread.currentThread().isInterrupted())
return;
File[] entries = file.listFiles(fileFilter);
if (entries != null) {
for (File entry : entries)
if (entry.isDirectory())
submitCrawlTask(entry);
else if (entry !=null && !indexedFiles.contains(entry)){
indexedFiles.add(entry);
try {
fileQueue.put(entry);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
}
}
Here is the file indexer thread
public class FileIndexer implements Runnable {
private final BlockingQueue queue;
public FileIndexer(BlockingQueue queue) {
this.queue = queue;
}
public void run() {
try {
while (true) {
indexFile(queue.take());
}
} catch (InterruptedException e) {
System.out.println("Indexer Interrupted");
Thread.currentThread().interrupt();
}
}
public void indexFile(File file) {
// do something with the file...
System.out.println("Indexing File : " + file.getAbsolutePath() + " " + file.getName());
};
}
I guess this is what you are trying to do:
You have a set of dirs:
dir1
dir2
dir3
And you need to place a "watch" on these 3 directories for a specific file name pattern.
Example: If a new file is added with name: watchme_9192.log, then you java logic should kick in and process that file.
So, based on that assumption, you can try: jnotify
JNotify is a java library that allow java application to listen to
file system events, such as:
File created
File modified
File renamed
File deleted
Also, related: best practice for directory polling
Related
I have the following design where I need to interact with client's API. And the WatchService (or Change Detection Handler method) had to be in the Testing Class in this caseand not in the FoodOrder class or ClientAPI class.
The ClientAPI class receives food orders from the various food machines via a callback method. All food orders have a unique reference number and are stored in a hashmap. Now when there is an update in instructions for the same food order (identified by the same reference number), a "change" is detected and new instruction is written to a file. The creation / modification of file will be picked up by the WatchService in the Testing class and trigger the proceesInstruction method to read the updated instructions.
I am looking for a better way to detect the change and pass the updated instructions to the Testing class without reading and writing to file, replacing the WatchService entirely, as this is very clumsy?
Class Testing{
public static void main(String[] args){
// connect to API...
ClientAPI clientAPI = new ClientAPI();
clientAPI.connect();
/*** Starting WatchService ***/
WatchService watchService = null;
try {
watchService = FileSystems.getDefault().newWatchService();
} catch (IOException e) {
e.printStackTrace();
}
scheduler.schedule(new WatchFile(watchService), 5, TimeUnit.SECONDS);
}
private static class WatchFile implements Runnable{
WatchService watchService;
public WatchFile(WatchService watchService) {
this.watchService = watchService;
}
public void run() {
try {
System.out.println("2. WatchService Started Running # " + LocalDateTime.now() + " !!!");
Path path = Paths.get("C:\\Testing\\csv");
path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE,StandardWatchEventKinds.ENTRY_MODIFY,StandardWatchEventKinds.ENTRY_DELETE, StandardWatchEventKinds.OVERFLOW);
WatchKey key;
while((key = watchService.take()) != null) {
for(WatchEvent<?> event: key.pollEvents()) {
final Path changed = (Path) event.context();
if(changed.endsWith("newInstructions.csv") && event.kind() == StandardWatchEventKinds.ENTRY_MODIFY) {
System.out.println("Instruction File modified!!!");
processInstruction("C:\\Testing\\csv\\newInstructions.csv");
}
}
key.reset();
}
} catch (IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public static void proceesInstruction(string filePath){
// read the csv file path and process instruction
}
}
Class FoodOrder{
int machineNumber;
String item;
String instruction;
String referenceNumber;
public FoodOrder(int machineNumber, String item, String instruction, String referenceNumber){
this.machineNumber = machineNumber;
this.item = item;
this.instruction = instruction;
this.referenceNumber = referenceNumber;
}
public void updateFoodOrder(int machineNumber, String item, String instruction, String referenceNumber){
this.machineNumber = machineNumber;
this.item = item;
this.referenceNumber = referenceNumber;
// Detect change in instruction
if(!instruction.equals(this.instruction)){
// my original code writed the new instruction to the
// C:\\Testing\\csv\\newInstructions.csv such that watchservice can detect file has been modified or created
// and process the instructions in the csv
}
this.instruction = instruction;
}
}
Class ClientAPI{
private HashMap<String,FoodOrder> foodOrderHashMap;
// Constructor
public clientAPI(){
foodOrderHashMap = new HashSet<String,FoodOrder>();
}
// API callback function
public void receiveMachineOrder(int machineNumber, String item, String instruction, String referenceNumber){
// my original is to write the instruction
if(foodOrderHashMap.containsKey(referenceNumber)){
foodOrderHashMap.get(referenceNumber).updateFoodOrder(machineNumber,item,instruction,referenceNumber);
} else{
foodOrderHashMap.put(referenceNumber, new FoodOrder(machineNumber, item, instruction, referenceNumber));
}
}
}
Is there an easy way to show the directory listing of my SPRING BOOT (v 2.1) resources/static folder?
The files are located under resources/static and I can access them separately, but I want to have a listing of all files and open them by clicking on the title like shown in the picture.
I want to "expose" the Log Files under resources/static/logs. If possible answer the question in Kotlin.
I found a similar question on SO but it didn't help:
Spring boot Tomcat – Enable/disable directory listing
Try this. It detects new files and folders (register new folder watcher) and performs some logic.
Somewhere in config class...
#Bean(name = "storageWatchService")
public WatchService createWatchService() throws IOException {
return FileSystems.getDefault().newWatchService();
}
#Component
public class StorageWatcher implements ApplicationRunner {
private static final Logger LOG = LoggerFactory.getLogger(StorageWatcher.class);
private static final WatchEvent.Kind<Path>[] WATCH_EVENTS_KINDS = new WatchEvent.Kind[] {StandardWatchEventKinds.ENTRY_CREATE};
private static final Map<WatchKey, Path> KEY_PATH_MAP = new HashMap<>();
#Resource
private PPAFacade ppaFacade;
#Resource
private WatchService storageWatchService;
#Resource
private Environment environment;
#Override
public void run(ApplicationArguments args) {
try {
registerDir(Paths.get(environment.getProperty(RINEX_FOLDER)), storageWatchService);
while (true) {
final WatchKey key = storageWatchService.take();
for (WatchEvent<?> event : key.pollEvents()) {
if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE && event.context() instanceof Path) {
final String fullPath = KEY_PATH_MAP.get(key) + "\\" + event.context().toString();
final File file = new File(fullPath);
if (file.isDirectory()) {
registerDir(file.toPath(), storageWatchService);
} else {
ppaFacade.process(file);
}
}
}
if (!key.reset()) {
KEY_PATH_MAP.remove(key);
}
if (KEY_PATH_MAP.isEmpty()) {
break;
}
}
} catch (InterruptedException e) {
LOG.error("StorageWatcher has been interrupted. No new files will be detected and processed.");
}
}
private static void registerDir(Path path, WatchService watchService) {
if (!Files.isDirectory(path, LinkOption.NOFOLLOW_LINKS)) {
return;
}
try {
LOG.info("registering: " + path);
final WatchKey key = path.register(watchService, WATCH_EVENTS_KINDS);
KEY_PATH_MAP.putIfAbsent(key, path);
Arrays.stream(path.toFile().listFiles()).forEach(f -> registerDir(f.toPath(), watchService));
} catch (IOException e) {
LOG.error(MessageFormat.format("Can not register file watcher for {0}", path), e);
}
}
}
I would like to create a synchronized file writing mechanism for Spring application. I have about 10 000 000 jsons which should be saved in separate files e.g:
text "abc" from json: { "id": "1", "text": "abc" } should be saved into the "1.json" file
text "pofd" from json: { "id": "2", "text": "pofd" } should be saved into the "2.json" file
Other requirements:
it is possible to write into multiple files at the same time
one file can be updated by multiple threads at the same time (many jsons with the same id but different text)
I've created FileWriterProxy (singleton bean) which is the main component for saving files. It loads lazy FileWriter component (prototype bean) which is responsible for writing into a file (it has synchronized write method). Each FileWriter object represents a separate file.
I'm suspecting that my solution is not thread safety. Let's consider the following scenario:
there are 3 threads (Thread1, Thread2, Thread3) which want to write into the same file (1.json), all of them hit the write method from FileWriterProxy component
Thread1 is getting the correct FileWriter
Thread1 is locking FileWriter for 1.json file
Thread1 is writing into 1.json file
Thread1 is finishing writing into the file and going to remove FileWriter from ConcurrentHashMap
in meanwhile Thread2 is getting FileWriter for the 1.json file and waiting for Thread1 to release lock
Thread1 is releasing the lock and removing FileWriter from ConcurrentHashMap
now Thread2 can write into the 1.json file (it has FileWriter which has been removed from ConcurrentHashMap)
Thread3 is getting FileWriter for 1.json (a new one! old FileWriter has been removed by Thread1)
Thread2 and Thread3 are writing into the same file at the same time because they have lock on different FileWriters objects
Please correct me if I'm wrong. How can I fix my implementation?
FileWriterProxy:
#Component
public class FileWriterProxy {
private final BeanFactory beanFactory;
private final Map<String, FileWriter> filePathsMappedToFileWriters = new ConcurrentHashMap<>();
public FileWriterProxy(BeanFactory beanFactory) {
this.beanFactory = beanFactory;
}
public void write(Path path, String data) {
FileWriter fileWriter = getFileWriter(path);
fileWriter.write(data);
removeFileWrite(path);
}
private FileWriter getFileWriter(Path path) {
return filePathsMappedToFileWriters.computeIfAbsent(path.toString(), e -> beanFactory.getBean(FileWriter.class, path));
}
private void removeFileWrite(Path path) {
filePathsMappedToFileWriters.remove(path.toString());
}
}
FileWriterProxyTest:
#RunWith(SpringRunner.class)
#SpringBootTest
public class FileWriterProxyTest {
#Rule
public TemporaryFolder temporaryFolder = new TemporaryFolder();
private static final String FILE_NAME = "filename.txt";
private File baseDirectory;
private Path path;
#Autowired
private FileWriterProxy fileWriterProxy;
#Before
public void setUp() {
baseDirectory = temporaryFolder.getRoot();
path = Paths.get(baseDirectory.getAbsolutePath(), FILE_NAME);
}
#Test
public void writeToFile() throws IOException {
String data = "test";
fileWriterProxy.write(path, data);
String fileContent = new String(Files.readAllBytes(path));
assertEquals(data, fileContent);
}
#Test
public void concurrentWritesToFile() throws InterruptedException {
Path path = Paths.get(baseDirectory.getAbsolutePath(), FILE_NAME);
List<Task> tasks = Arrays.asList(
new Task(path, "test1"),
new Task(path, "test2"),
new Task(path, "test3"),
new Task(path, "test4"),
new Task(path, "test5"));
ExecutorService executorService = Executors.newFixedThreadPool(5);
List<Future<Boolean>> futures = executorService.invokeAll(tasks);
wait(futures);
}
#Test
public void manyRandomWritesToFiles() throws InterruptedException {
List<Task> tasks = createTasks(1000);
ExecutorService executorService = Executors.newFixedThreadPool(5);
List<Future<Boolean>> futures = executorService.invokeAll(tasks);
wait(futures);
}
private void wait(List<Future<Boolean>> tasksFutures) {
tasksFutures.forEach(e -> {
try {
e.get(10, TimeUnit.SECONDS);
} catch (Exception e1) {
e1.printStackTrace();
}
});
}
private List<Task> createTasks(int number) {
List<Task> tasks = new ArrayList<>();
IntStream.range(0, number).forEach(e -> {
String fileName = generateFileName();
Path path = Paths.get(baseDirectory.getAbsolutePath(), fileName);
tasks.add(new Task(path, "test"));
});
return tasks;
}
private String generateFileName() {
int length = 10;
boolean useLetters = true;
boolean useNumbers = false;
return RandomStringUtils.random(length, useLetters, useNumbers) + ".txt";
}
private class Task implements Callable<Boolean> {
private final Path path;
private final String data;
Task(Path path, String data) {
this.path = path;
this.data = data;
}
#Override
public Boolean call() {
fileWriterProxy.write(path, data);
return true;
}
}
}
Config:
#Configuration
public class Config {
#Bean
#Lazy
#Scope("prototype")
public FileWriter fileWriter(Path path) {
return new FileWriter(path);
}
}
FileWriter:
public class FileWriter {
private static final Logger logger = LoggerFactory.getLogger(FileWriter.class);
private final Path path;
public FileWriter(Path path) {
this.path = path;
}
public synchronized void write(String data) {
String filePath = path.toString();
try {
Files.write(path, data.getBytes());
logger.info("File has been saved: {}", filePath);
} catch (IOException e) {
logger.error("Error occurred while writing to file: {}", filePath);
}
}
}
I'm not sure if that's the right way to ask this, but I'm gonna try to explain my case and what I need.
I have a big java project, that upload files in many different java classes, like too many, and I have around 7 different main folders where the files are uploaded. The files at the moment are saved inside the webapp context, and I need to save them outside of context.
If there were only a few classes that upload these files I could spend a few days changing every class and direct it to a path outisde of context, but there are way too many classes, so I have to figure out a way to do it without changing every class, or any class at all, which would be ideal.
Every upload is done in the following way:
I get real path of one of my main folders:
String realpath = httpServletRequest.getSession()
.getServletContext()
.getRealPath("/mainfolder1/mainsubfolder1/");
Then I get the file and set custom file name:
FormFile file = myForm.getFile();
String contentType = file.getContentType();
String fileName = file.getFileName();
int fileSize = file.getFileSize();
customFileName = "anyName" + fileName.substring(fileName.lastIndexOf("."));
Then I validate and save the file:
if (fileSize > 0 && contentType != null && fileName.length() > 0){
InputStream in = file.getInputStream();
OutputStream bos = new FileOutputStream(realpath + "/" + customFileName);
int byteRead = 0;
byte[] buffer = new byte[8192];
while ((byteRead = in.read(buffer, 0, 8192)) != -1){
bos.write(buffer, 0, byteRead);
}
bos.close();
in.close();
}
Very simple way to save my files, and as you can see, they are saved inside context.
So if I could somehow override java.io.FileOutputStream, to not only save it inside context, but to make a copy outside of context too, that would be great, like save it in the specified path and also on some other path outside of context.
But I don't know if this is possible, or how to reproduce this behaviour.
What I need is to keep the class code exactly as it is but write the file 2 times:
First here: "/insideContext/mainfolder1/mainsubfolder1/"
Then here: "/outsideContext/mainfolder1/mainsubfolder1/"
Is this possible? If not, what would be the best way to accomplish this?
I'd refactor and use Decorator or Wrapper pattern. More about it here
Below some simple idea you could use.
public class ContextAwareDuplicatorOutputStream extends OutputStream {
FileOutputStream insideContext;
FileOutputStream outsideContext;
public ContextAwareDuplicatorOutputStream(String insideContextPath,
String outsideContextPath, String fileName)
throws FileNotFoundException {
insideContext = new FileOutputStream(insideContextPath
+ File.pathSeparator + fileName);
outsideContext = new FileOutputStream(outsideContextPath
+ File.pathSeparator + fileName);
}
#Override
public void close() throws IOException {
insideContext.close();
outsideContext.close();
}
#Override
public void flush() throws IOException {
insideContext.flush();
outsideContext.flush();
}
#Override
public void write(byte[] b) throws IOException {
insideContext.write(b);
outsideContext.write(b);
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
insideContext.write(b, off, len);
outsideContext.write(b, off, len);
}
#Override
public void write(int b) throws IOException {
insideContext.write(b);
outsideContext.write(b);
}
}
Since you don't want to edit anything on your code, create a ServletContextListener that monitor the folder where you upload, and on the new file event, you copy it to the proper directory. Here is awnsered how to monitor a directory. Directory listener in Java
Below here is a small code, not really perfect, but the idea is there
public class FileMonitorServletContextListener implements
ServletContextListener {
public interface FileMonitor {
void start(String fromFolder, String toFolder);
void stop();
}
public class SimpleThreadedWatcher implements FileMonitor {
private class SimpleThread extends Thread {
private boolean running = true;
private String fromFolder;
private String toFolder;
public SimpleThread(String fromFolder, String toFolder) {
this.fromFolder = fromFolder;
this.toFolder = toFolder;
}
private void copy(Path child, String toFolder) {
// Copy the file to the folder
}
#Override
public void run() {
try {
WatchService watcher = FileSystems.getDefault()
.newWatchService();
Path fromPath = Paths.get(fromFolder);
watcher = FileSystems.getDefault().newWatchService();
WatchKey key = fromPath.register(watcher,
StandardWatchEventKinds.ENTRY_CREATE);
while (running) {
for (WatchEvent<?> event : key.pollEvents()) {
// Context for directory entry event is the file
// name of
// entry
#SuppressWarnings("unchecked")
WatchEvent<Path> ev = (WatchEvent<Path>) event;
Path name = ev.context();
Path child = fromPath.resolve(name);
// print out event
System.out.format("%s: %s\n", event.kind().name(),
child);
copy(child, toFolder);
boolean valid = key.reset();
if (!valid) {
break;
}
}
Thread.sleep(1000);
}
} catch (Exception e) {
throw new RuntimeException("Error: ", e);
}
}
public void stopWorking() {
running = false;
}
}
private SimpleThread worker;
#Override
public void start(String fromFolder, String toFolder) {
worker = new SimpleThread(fromFolder, toFolder);
worker.start();
}
#Override
public void stop() {
worker.stopWorking();
}
}
FileMonitor fileMonitor = new SimpleThreadedWatcher();
#Override
public void contextDestroyed(ServletContextEvent arg0) {
fileMonitor.stop();
}
#Override
public void contextInitialized(ServletContextEvent arg0) {
fileMonitor.start("FROM", "TO");
}
}
So I found some code earlier that looks like it would work but it doesn't call to delete the files just to list them. What do I need to add so that it deletes the files?
import java.io.File;
import java.util.regex.Pattern;
public class cleardir {
static String userprofile = System.getenv("USERPROFILE");
private static void walkDir(final File dir, final Pattern pattern) {
final File[] files = dir.listFiles();
if (files != null) {
for (final File file : files) {
if (file.isDirectory()) {
walkDir(file, pattern);
} else if (pattern.matcher(file.getName()).matches()) {
System.out.println("file to delete: " + file.getAbsolutePath());
} } } }
public static void main(String[] args) {
walkDir(new File(userprofile+"/Downloads/Software_Tokens"),
Pattern.compile(".*\\.sdtid"));
}
}
Once you have the path to the file, delete him:
File physicalFile = new File(path); // This is one of your file objects inside your for loop, since you already have them just delete them.
try {
physicalFile.delete(); //Returns true if the file was deleted or false otherwise.
//You might want to know this just in case you need to do some additional operations based on the outcome of the deletion.
} catch(SecurityException securityException) {
//TODO Handle.
//If you haven't got enough rights to access the file, this exception is thrown.
}
To delete a file you can call the delete function
file.delete();
You can invoke the delete() method on an instance of File. Be sure to check the returncode to make sure your file was actually deleted.
Use file.delete(); to delete a file.
You need to learn Java basics properly before attempting to write programs. Good resource: http://docs.oracle.com/javase/tutorial/index.html
Call File.delete() for each file you want to delete. So your code would be:
import java.io.File;
import java.util.regex.Pattern;
public class cleardir {
static String userprofile = System.getenv("USERPROFILE");
private static void walkDir(final File dir, final Pattern pattern) {
final File[] files = dir.listFiles();
if (files != null) {
for (final File file : files) {
if (file.isDirectory()) {
walkDir(file, pattern);
} else if (pattern.matcher(file.getName()).matches()) {
System.out.println("file to delete: " + file.getAbsolutePath());
boolean deleteSuccess=file.delete();
if(!deleteSuccess)System.err.println("[warning]: "+file.getAbsolutePath()+" was not deleted...");
}
}
}
}
public static void main(String[] args) {
walkDir(new File(userprofile+"/Downloads/Software_Tokens"),
Pattern.compile(".*\\.sdtid"));
}
}
final File folder = new File("C:/Temp");
FileFilter ff = new FileFilter() {
#Override
public boolean accept(File pathname) {
String ext = FilenameUtils.getExtension(pathname.getName());
return ext.equalsIgnoreCase("EXT"); //Your extension
}
};
final File[] files = folder.listFiles(ff);
for (final File file : files) {
file.delete();
}
public class cleardir {
static String userprofile = System.getenv("USERPROFILE");
private static final String FILE_DIR = userprofile+"\\Downloads\\Software_Tokens";
private static final String FILE_TEXT_EXT = ".sdtid";
public static void run(String args[]) {
new cleardir().deleteFile(FILE_DIR,FILE_TEXT_EXT);
}
public void deleteFile(String folder, String ext){
GenericExtFilter filter = new GenericExtFilter(ext);
File dir = new File(folder);
if (dir.exists()) {
//list out all the file name with .txt extension
String[] list = dir.list(filter);
if (list.length == 0) return;
File fileDelete;
for (String file : list){
String temp = new StringBuffer(FILE_DIR)
.append(File.separator)
.append(file).toString();
fileDelete = new File(temp);
boolean isdeleted = fileDelete.delete();
System.out.println("file : " + temp + " is deleted : " + isdeleted);
}
}
}
//inner class, generic extension filter
public class GenericExtFilter implements FilenameFilter {
private String ext;
public GenericExtFilter(String ext) {
this.ext = ext;
}
public boolean accept(File dir, String name) {
return (name.endsWith(ext));
}
}
}