I have a file with name foo.txt. This file contains some text. I want to achieve following functionality:
I launch program
write something to the file (for example add one row: new string in foo.txt)
I want to get ONLY NEW content of this file.
Can you clarify the best solution of this problem? Also I want resolve related issues: in case if I modify foo.txt I want to see diff.
The closest tool which I found in Java is WatchService but if I understood right this tool can only detect type of event happened on filesystem (create file or delete or modify).
Java Diff Utils is designed for that purpose.
final List<String> originalFileContents = new ArrayList<String>();
final String filePath = "C:/Users/BackSlash/Desktop/asd.txt";
FileListener fileListener = new FileListener() {
#Override
public void fileDeleted(FileChangeEvent paramFileChangeEvent)
throws Exception {
// use this to handle file deletion event
}
#Override
public void fileCreated(FileChangeEvent paramFileChangeEvent)
throws Exception {
// use this to handle file creation event
}
#Override
public void fileChanged(FileChangeEvent paramFileChangeEvent)
throws Exception {
System.out.println("File Changed");
//get new contents
List<String> newFileContents = new ArrayList<String> ();
getFileContents(filePath, newFileContents);
//get the diff between the two files
Patch patch = DiffUtils.diff(originalFileContents, newFileContents);
//get single changes in a list
List<Delta> deltas = patch.getDeltas();
//print the changes
for (Delta delta : deltas) {
System.out.println(delta);
}
}
};
DefaultFileMonitor monitor = new DefaultFileMonitor(fileListener);
try {
FileObject fileObject = VFS.getManager().resolveFile(filePath);
getFileContents(filePath, originalFileContents);
monitor.addFile(fileObject);
monitor.start();
} catch (InterruptedException ex) {
ex.printStackTrace();
} catch (FileNotFoundException e) {
//handle
e.printStackTrace();
} catch (IOException e) {
//handle
e.printStackTrace();
}
Where getFileContents is :
void getFileContents(String path, List<String> contents) throws FileNotFoundException, IOException {
contents.clear();
BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(path), "UTF-8"));
String line = null;
while ((line = reader.readLine()) != null) {
contents.add(line);
}
}
What I did:
I loaded the original file contents in a List<String>.
I used Apache Commons VFS to listen for file changes, using FileMonitor. You may ask, why? Because WatchService is only available starting from Java 7, while FileMonitor works with at least Java 5 (personal preference, if you prefer WatchService you can use it). note: Apache Commons VFS depends on Apache Commons Logging, you'll have to add both to your build path in order to make it work.
I created a FileListener, then I implemented the fileChanged method.
That method load new contents form the file, and uses Patch.diff to retrieve all differences, then prints them
I created a DefaultFileMonitor, which basically listens for changes to a file, and I added my file to it.
I started the monitor.
After the monitor is started, it will begin listening for file changes.
Related
I want to process files with a flink stream in which two lines belong together. In the first line there is a header and in the second line a corresponding text.
The files are located on my local file system. I am using the readFile(fileInputFormat, path, watchType, interval, pathFilter, typeInfo) method with a custom FileInputFormat.
My streaming job class looks like this:
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Read> inputStream = env.readFile(new ReadInputFormatTest("path/to/monitored/folder"), "path/to/monitored/folder", FileProcessingMode.PROCESS_CONTINUOUSLY, 100);
inputStream.print();
env.execute("Flink Streaming Java API Skeleton");
and my ReadInputFormatTest like this:
public class ReadInputFormatTest extends FileInputFormat<Read> {
private transient FileSystem fileSystem;
private transient BufferedReader reader;
private final String inputPath;
private String headerLine;
private String readLine;
public ReadInputFormatTest(String inputPath) {
this.inputPath = inputPath;
}
#Override
public void open(FileInputSplit inputSplit) throws IOException {
FileSystem fileSystem = getFileSystem();
this.reader = new BufferedReader(new InputStreamReader(fileSystem.open(inputSplit.getPath())));
this.headerLine = reader.readLine();
this.readLine = reader.readLine();
}
private FileSystem getFileSystem() {
if (fileSystem == null) {
try {
fileSystem = FileSystem.get(new URI(inputPath));
} catch (URISyntaxException | IOException e) {
throw new RuntimeException(e);
}
}
return fileSystem;
}
#Override
public boolean reachedEnd() throws IOException {
return headerLine == null;
}
#Override
public Read nextRecord(Read r) throws IOException {
r.setHeader(headerLine);
r.setSequence(readLine);
headerLine = reader.readLine();
readLine = reader.readLine();
return r;
}
}
As expected, the headers and the text are stored together in one object. However, the file is read eight times. So the problem is the parallelization. Where and how can I specify that a file is processed only once, but several files in parallel?
Or do I have to change my custom FileInputFormat even further?
I would modify your source to emit the available filenames (instead of the actual file contents) and then add a new processor to read a name from the input stream and then emit pairs of lines. In other words, split the current source into a source followed by a processor. The processor can be made to run at any degree of parallelism and the source would be a single instance.
I want to save data with jackson to existing file (update it) but it won't work when I run my project from jar.
I need to use json as "database" (I know it's pretty stupid but that's for a school project) and to do it I load and save all the data when I do any of CRUD operations. It's working fine when I run it with IDE but when I tried as a jar it had a problem with reading file from ClassPathResource.
So I have this method to save changes to file:
private List<Item> items;
private ObjectMapper mapper;
private ObjectWriter writer;
public void saveData() {
mapper = new ObjectMapper();
writer = mapper.writer(new DefaultPrettyPrinter());
try {
writer.writeValue(new ClassPathResource("items.json").getFile(), items);
} catch (IOException e) {
e.printStackTrace();
}
}
And it works just fine when i run this through IntelliJ but it won't work when I run it as a jar.
I found a solution to loading the data by using InputStream from this question and method looks like this:
public void loadData() {
mapper = new ObjectMapper();
try {
ClassPathResource classPathResource = new ClassPathResource("items.json");
InputStream inputStream = classPathResource.getInputStream();
File tempFile = File.createTempFile("test", ".json");
FileUtils.copyInputStreamToFile(inputStream, tempFile);
System.out.println(tempFile);
System.out.println(ItemDao.class.getProtectionDomain().getCodeSource().getLocation().getPath().toString());
items = mapper.readValue(tempFile, new TypeReference<List<Item>>() {
});
} catch (IOException e) {
items = null;
e.printStackTrace();
}
}
But I still have no idea how to actually save changes. I was thinking about making use of FileOutputStreambut I achieved nothing.
So I want to get this working in jar file and be able to save changes to the same file, thanks for help in advance!
when you want to do read/write operations, it is better keep the file outside of the project. when running the jar, pass file name with path as an argument. like -DfileName=/Users/chappa/Documents/items.json etc. This way, you have absolute path, and you can perform read/write operations on it
if you are using java 1.7 or above, use below approach to write data.
To read data, you can use jackson api to load the json file as is.
Path wipPath = Paths.get("/Users/chappa/Documents/items.json");
try (BufferedWriter writer = Files.newBufferedWriter(wipPath)) {
for (String record : nosRecords) {
writer.write(record);
}
}
Just in case if you want to read json using IO streams, you can use below code
Path wipPath = Paths.get("/Users/chappa/Documents/items.json");
try (BufferedReader reader = Files.newBufferedReader(wipPath)) {
String line=null;
while((line = reader.readLine()) != null) {
System.out.println(line);
}
}
I have implemented a simple file listener using Apache commons io monitor.I have also implemented a selenium script that simply do is download file into the pre hard coded path folder.That is totally working fine.My listener monitoring downloaded files and collect the necessary information.According to my requirement i should be able to stop the file listener once my selenium script is finished execution.To do that i must know that the incoming file transferring status to handle it better way.because i can not stop listener middle of the file transferring.(sometimes download file can be heavy so it will take some time to download).So how can i know that incoming file status in Apache commons io monitor before stop the file listener.If any one knows please let me know.
Sample code snippet
public class SimpleTestMonitor {
// A hardcoded path to a folder you are monitoring .
public static final String FOLDER =
"/home/skywalker/Desktop/simple-test-monitor/watchdir";
public static void main(String[] args) throws Exception {
// The monitor will perform polling on the folder every 5 seconds
final long pollingInterval = 5 * 1000;
File folder = new File(FOLDER);
if (!folder.exists()) {
// Test to see if monitored folder exists
throw new RuntimeException("Directory not found: " + FOLDER);
}
FileAlterationObserver observer = new FileAlterationObserver(folder);
FileAlterationMonitor monitor =
new FileAlterationMonitor(pollingInterval);
FileAlterationListener listener = new FileAlterationListenerAdaptor() {
// Is triggered when a file is created in the monitored folder
#Override
public void onFileCreate(File file) {
try {
// "file" is the reference to the newly created file
System.out.println("File created: "
+ file.getCanonicalPath());
} catch (IOException e) {
e.printStackTrace(System.err);
}
}
// Is triggered when a file is deleted from the monitored folder
#Override
public void onFileDelete(File file) {
try {
// "file" is the reference to the removed file
System.out.println("File removed: "
+ file.getCanonicalPath());
// "file" does not exists anymore in the location
System.out.println("File still exists in location: "
+ file.exists());
} catch (IOException e) {
e.printStackTrace(System.err);
}
}
};
observer.addListener(listener);
monitor.addObserver(observer);
monitor.start();
}
}
Reference
I try make implementation for comparing the files before they are uploaded.
If file whith name is exist in system ask about create new version or just override it.
Here is the problem, how to get file name?
I can't use receiveUpload(), because after this method file is remove from upload component ?
The problem is that once you start an upload using the Upload component, it can only be interrupted by calling the interruptUpload() method, and you cannot resume anytime later.
The interruption is permanent.
This means you cannot pause in the middle of the upload to see if you already have the file in your system. You have to upload the file all the way.
Considering this drawback, you can sill check in your system if you have the file, after the upload finishes. If you have the file, you can show a confirmation dialog in which you decide wether to keep the file or overwrite.
The following is an example in which I check in the "system" (I just keep a String list with the filenames) if the file has already been uploaded:
public class RestrictingUpload extends Upload implements Upload.SucceededListener, Upload.Receiver {
private List<String> uploadedFilenames;
private ByteArrayOutputStream latestUploadedOutputStream;
public RestrictingUpload() {
setCaption("Upload");
setButtonCaption("Upload file");
addSucceededListener(this);
setReceiver(this);
uploadedFilenames = new ArrayList<String>();
}
#Override
public OutputStream receiveUpload(String filename, String mimeType) {
latestUploadedOutputStream = new ByteArrayOutputStream();
return latestUploadedOutputStream;
}
#Override
public void uploadSucceeded(SucceededEvent event) {
if (fileExistsInSystem(event.getFilename())) {
confirmOverwrite(event.getFilename());
} else {
uploadedFilenames.add(event.getFilename());
}
}
private void confirmOverwrite(final String filename) {
ConfirmDialog confirmDialog = new ConfirmDialog();
String message = String.format("The file %s already exists in the system. Overwrite?", filename);
confirmDialog.show(getUI(), "Overwrite?", message, "Overwrite", "Cancel", new ConfirmDialog.Listener() {
#Override
public void onClose(ConfirmDialog dialog) {
if (dialog.isConfirmed()) {
copyFileToSystem(filename);
}
}
});
}
private void copyFileToSystem(String filename) {
try {
IOUtils.write(latestUploadedOutputStream.toByteArray(), new FileOutputStream(filename));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e2) {
e2.printStackTrace();
}
}
private boolean fileExistsInSystem(String filename) {
return uploadedFilenames.contains(filename);
}
}
Note that I have used 2 external libraries:
Apache Commons IO 2.4 (http://mvnrepository.com/artifact/commons-io/commons-io/2.4) for writing to streams
ConfirmDialog from Vaadin Directory (https://vaadin.com/directory#addon/confirmdialog)
You can get the code snippet for this class from Gist: https://gist.github.com/gabrielruiu/9960772 which you can paste into your UI and test it out.
I am working on java Application where I am dealing with read and
write file. But I want to do this process simultaneously. As my file
is log file from which I am reading contents by some time interval for
this I used java.util.Timer. And during this time interval I am trying
to writing some contents in the file, but this is not updating in my
log file.
My Timer class is
public class Timer extends java.util.Timer {
public void startTimer() {
Timer t = new Timer();
Task task = new Task();
t.schedule(task, 0, 10000);
}
}
My class form which file read write class is called
public class Task extends TimerTask {
#Override
public void run() {
System.out.println("In task ....");
try {
SMSQuestions smsQuestions = new SMSQuestions();
smsQuestions.sendSMSAnswer();
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
My file read write class is
public void sendSMSAnswer()
{
long sender;
String question;
try {
BufferedReader br = new BufferedReader(new FileReader("C:/temp/NowSMS/TEST.log"));
String line = br.readLine();
while (line != null) {
sender = 0;
question = "";
System.out.println(line);
line = br.readLine();
}
//To delete all contents in file
BufferedWriter bw = new BufferedWriter(new FileWriter("C:/temp/NowSMS/TEST.log"));
bw.write("");
getDetails();
}catch(Exception exp)
{
exp.printStackTrace();
}
}
My problem is when i updated my log file by writing new contents it
can't be updated so my question is: is it possible in java to simultaneously read write file or not?
I would suggest you have a look at random access files (see http://docs.oracle.com/javase/tutorial/essential/io/rafs.html). This API is a little more complicated to work with since you'll be reading bytes, not lines, but it should give you what you need.
If you don't want to deal with NIO you can use http://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html instead.
Try this topic. But the only answer you will get is, its only working with a few OSs.
I have two things to say about your code.
First:
I think that you are missing the finally block where you are supposed to close br and bw.
Try:
finally {
br.close();
bw.close();
}
after the catch block.
Second: It is not a good idea to write content to a file while it is opened. I mean, if you open a file with br maybe it is better to gives another file path to bw