I have a pretty big number of files that need to be converted to a different format. The converting is done via a Java-JAR-File that gets takes the filename as a parameter. I now have a Windows batchfile that uses a for loop to loop through all the files (there is a file that contains a list of all files that need to be converted)
for /F %%i in (all_files.txt) do call java -cp %Classpath% de.xyz.Convert -xml %%i .\xml
Now the machine I want to do this on has eight cores. The number of files is about 360.000 and I would like it to take as little time as possible, so I'd like to use as many cores as possible. How would I go about using multiple cores as easy as possible? Is Windows going to be doing that on its own?
Ok, because I hadn't actually done it before, I knocked this up. It's not great, and the lib I used was a jar I created to crash after 2 mins.. Hopefully you'll be able to reverse engineer this for your needs.
package test;
import java.io.IOException;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class Test {
public static void main(String[] args) throws InterruptedException, IOException {
BlockingQueue<Runnable> runnableQueue = new LinkedBlockingQueue<>();
ExecutorService executorServ = new ThreadPoolExecutor(8, 8, 1, TimeUnit.MINUTES, runnableQueue);
runnableQueue.add(new RunCrash("Example")); // Add one for each file...
executorServ.shutdown();
while(!executorServ.isTerminated()) {
// running
}
}
}
class RunCrash implements Runnable {
private String fileName;
RunCrash(String fileName) {
this.fileName = fileName;
}
#Override
public void run() {
System.out.println(fileName);
try {
crash.CrashMe.main(new String[]{fileName});
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Oh, you can let the main thread die before the others finish, I believe the JVM will keep the executor and associated queue. :)
Related
Pretty much, I'm trying to write a simple program that lets the user choose a file. Unfortunately, JFileChooser through Swing is a little outdated, so I am trying to use JavaFX FileChooser for this. The goal is to run FileGetter as a thread, transfer the file data to the Main Class, and continue from there.
Main Class:
package application;
import java.io.File;
import javafx.application.Application;
import javafx.stage.Stage;
import javafx.scene.Scene;
import javafx.scene.layout.BorderPane;
public class Main {
public static void main(String[] args) {
Thread t1 = new Thread(new FileGetter());
FileGetter fg = new FileGetter();
t1.start();
boolean isReady = false;
while(isReady == false){
isReady = FileGetter.getIsReady();
}
File file = FileGetter.getFile();
System.out.println(file.getAbsolutePath());
...
}
}
FileGetter Class:
package application;
import java.io.File;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.stage.FileChooser;
import javafx.stage.Stage;
import javafx.scene.Scene;
import javafx.scene.layout.BorderPane;
public class FileGetter extends Application implements Runnable {
static File file;
static boolean isReady = false;
#Override
public void start(Stage primaryStage) {
try {
FileChooser fc = new FileChooser();
while(file == null){
file = fc.showOpenDialog(primaryStage);
}
isReady = true;
Platform.exit();
} catch(Exception e) {
e.printStackTrace();
}
}
#Override
public void run() {
launch();
}
public static boolean getIsReady(){
return isReady;
}
public static File getFile(){
return file;
}
}
Problem is that the value of isReady in the while loop doesn't update to true when the user picked a file (the reason I have it is to prevent the code in Main from continuing with a File set to null).
Any help, alternative suggestions, or explanations as of why this happens is very much appreciated!
The java memory model does not require variable values to be the same in different threads except under specific conditions.
What is happening here is that the FileGetter thread is updating the value in the own memory that is only accessed from this thread, but your main thread doesn't see the updated value, since it only sees the version of the variable stored in it's own memory that is different from the one of the FileGetter thread. Each of the threads has it's own copy of the field in memory, which is perfectly fine according to the java specification.
To fix this, you can simply add the volatile modifier to isReady:
static volatile boolean isReady = false;
which makes sure the updated value will be visible from your main thread.
Furthermore I recommend reducing the number of FileGetter instances you create. In your code 3 instances are created, but only 1 is used.
Thread t1 = new Thread(() -> Application.launch(FileGetter.class));
t1.start();
...
The easiest way to implement this
Instead of trying to drive the horse with the cart, why not just follow the standard JavaFX lifecycle? In other words, make your Main class a subclass of Application, get the file in the start() method, and then proceed (in a background thread) with the rest of the application?
public class Main extends Application {
#Override
public void init() {
// make sure we don't exit when file chooser is closed...
Platform.setImplicitExit(false);
}
#Override
public void start(Stage primaryStage) {
File file = null ;
FileChooser fc = new FileChooser();
while(file == null){
file = fc.showOpenDialog(primaryStage);
}
final File theFile = file ;
new Thread(() -> runApplication(theFile)).start();
}
private void runApplication(File file) {
// run your application here...
}
}
What is wrong with your code
If you really want the Main class to be separate from the JavaFX Application class (which doesn't really make sense: once you have decided to use a JavaFX FileChooser, you have decided you are writing a JavaFX application, so the startup class should be a subclass of Application), then it gets a bit tricky. There are several issues with your code as it stands, some of which are addressed in other answers. The main issue, as shown in Fabian's answer, is that you are referencing FileGetter.isReady from multiple threads without ensuring liveness. This is exactly the issue addressed in Josh Bloch's Effective Java (Item 66 in the 2nd edition).
Another issue with your code is that you won't be able to use the FileGetter more than once (you can't call launch() more than once), which might not be an issue in your code now, but almost certainly will be at some point with this application as development progresses. The problem is that you have mixed two issues: starting the FX toolkit, and retrieving a File from a FileChooser. The first thing must only be done once; the second should be written to be reusable.
And finally your loop
while(isReady == false){
isReady = FileGetter.getIsReady();
}
is very bad practice: it checks the isReady flag as fast as it possibly can. Under some (fairly unusual) circumstances, it could even prevent the FX Application thread from having any resources to run. This should just block until the file is ready.
How to fix without making Main a JavaFX Application
So, again only if you have a really pressing need to do so, I would first create a class that just has the responsibility of starting the FX toolkit. Something like:
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.stage.Stage;
public class FXStarter extends Application {
private static final AtomicBoolean startRequested = new AtomicBoolean(false);
private static final CountDownLatch latch = new CountDownLatch(1);
#Override
public void init() {
Platform.setImplicitExit(false);
}
#Override
public void start(Stage primaryStage) {
latch.countDown();
}
/** Starts the FX toolkit, if not already started via this method,
** and blocks execution until it is running.
**/
public static void startFXIfNeeded() throws InterruptedException {
if (! startRequested.getAndSet(true)) {
new Thread(Application::launch).start();
}
latch.await();
}
}
Now create a class that gets a file for you. This should ensure the FX toolkit is running, using the previous class. This implementation allows you to call getFile() from any thread:
import java.io.File;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;
import javafx.application.Platform;
import javafx.stage.FileChooser;
public class FileGetter {
/**
** Retrieves a file from a JavaFX File chooser. This method can
** be called from any thread, and will block until the user chooses
** a file.
**/
public File getFile() throws InterruptedException {
FXStarter.startFXIfNeeded() ;
if (Platform.isFxApplicationThread()) {
return doGetFile();
} else {
FutureTask<File> task = new FutureTask<File>(this::doGetFile);
Platform.runLater(task);
try {
return task.get();
} catch (ExecutionException exc) {
throw new RuntimeException(exc);
}
}
}
private File doGetFile() {
File file = null ;
FileChooser chooser = new FileChooser() ;
while (file == null) {
file = chooser.showOpenDialog(null) ;
}
return file ;
}
}
and finally your Main is just
import java.io.File;
public class Main {
public static void main(String[] args) throws InterruptedException {
File file = new FileGetter().getFile();
// proceed...
}
}
Again, this is pretty complex; I see no reason not to simply use the standard FX Application lifecycle for this, as in the very first code block in the answer.
In this code
while(isReady == false){
isReady = FileGetter.getIsReady();
}
there is nothing that is going to change the state of FileGetter's isReady to true
I would like to write a file that comes from over a network, so I don't know the size of the file that's comming in. Sometimes the disk on the file server might get filled up and I would like return a message to my client notifying them of this error. I couldn't find any documentation on being able to catch this type of i/o error. FileChannel streams bytes from memory to disk, so it may not be trivial to detect this. Is the saving happening asynchronously? Is it possible to detect disk full?
// Create a new file to write to
RandomAccessFile mFile = new RandomAccessFile(this.mFilePath, "rw");
FileChannel mFileChannel = this.mFile.getChannel();
// wrappedBuffer has my file in it
ByteBuffer wrappedBuffer = ByteBuffer.wrap(fileBuffer);
while(wrappedBuffer.hasRemaining()) {
bytesWritten += this.mFileChannel.write(wrappedBuffer, this.mBytesProcessed);
}
I figured in the File class, we can do something like this:
// if there is less than 1 mb left on disk
new File(this.mFilePath, "r").getUsableSpace() < 1024;
But if there a way to throw an except if this.mFileChannel.write() fails because the disk is full?
Even if it's not recommended to parse the error message you could do something like this :
import java.io.IOException;
import java.nio.file.FileStore;
import java.nio.file.FileSystemException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.regex.Pattern;
public class SmallDisk {
final static String SMALL_DISK_PATH = "/Volumes/smallDisk";
final static Pattern NO_SPACE_LEFT = Pattern.compile(": No space left on device$");
public static void main(String[] args) throws NoSpaceException {
Path p = Paths.get(SMALL_DISK_PATH);
FileStore fs = null;
try {
fs = Files.getFileStore(p);
System.out.println(fs.getUsableSpace());
Path newFile = Paths.get(SMALL_DISK_PATH + "/newFile");
Files.createFile(newFile);
} catch (FileSystemException e) {
//We catch the "No space left on device" from the FileSystemException and propagate it
if(NO_SPACE_LEFT.matcher(e.getMessage()).find()){
throw new NoSpaceException("Not enough space");
}
//Propagate exception or deal with it here
} catch (IOException e) {
//Propagate exception or deal with it here
}
}
public static class NoSpaceException extends IOException{
public NoSpaceException(String message) {
super(message);
}
}
}
Another way but it doesn't guaranty that you won't have exception is to use the FileStore to check that you enough space before you write (not enough if you are working with a shared folder or multi threads software)
When I research things on the internet I like to copy and paste certain paragraphs so I could review them later on.
I'm trying to write a program that would continuously check the clipboard for text content and write it to a text file any time it is renewed.
In the following test of the program I had "public class Clipboard" in my clipboard before running the program and the exception happened when I copied text from netbeans (The IDE I was using to run the program) while the program was running:
import java.awt.Toolkit;
import java.awt.datatransfer.Clipboard;
import java.awt.datatransfer.DataFlavor;
import java.awt.datatransfer.UnsupportedFlavorException;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class TestClipboard {
public static void main(String[] args) {
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String initial = "";
while(true) {
try {
String paste = c.getContents(null).getTransferData(DataFlavor.stringFlavor).toString();
if(!paste.equals(initial)) {
System.out.println(paste);
initial = paste;
}
} catch (UnsupportedFlavorException | IOException ex) {
Logger.getLogger(TestClipboard.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
The output:
public class TestClipboard
Exception in thread "main" java.lang.IllegalStateException: cannot open system clipboard
at sun.awt.windows.WClipboard.openClipboard(Native Method)
at sun.awt.datatransfer.ClipboardTransferable.<init>(ClipboardTransferable.java:78)
at sun.awt.datatransfer.SunClipboard.getContents(SunClipboard.java:144)
at delete.TestClipboard.main(TestClipboard.java:21)
Java Result: 1
BUILD SUCCESSFUL (total time: 34 seconds)
Why can't it open the system clipboard?
Does the getSystemClipboard() method not have global scope? - In other words, can I not get the clipboard's contents if the copy operation was performed in an internet browser?
You appear to be trying to read from the clipboard while another process is updating to it (or some such).
I fixed by:
Requesting an instance of the Clipboard within the loop
Adding a Thread.sleep into the while-loop
For example...
public class TestClipboard {
public static void main(String[] args) {
String initial = "";
while (true) {
try {
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String paste = c.getContents(null).getTransferData(DataFlavor.stringFlavor).toString();
if (!paste.equals(initial)) {
System.out.println(paste);
initial = paste;
}
} catch (UnsupportedFlavorException | IOException ex) {
Logger.getLogger(TestClipboard.class.getName()).log(Level.SEVERE, null, ex);
}
try {
Thread.sleep(40);
} catch (InterruptedException ex) {
}
}
}
}
It should be noted that it won't stop it from happening, it will only reduce the number of occurrences. When it is thrown, you could (just about) ignore and try again...
I'm using Watcher in JDK7 which relies on inotify events. If the file is on a NFS, I want my program to fallback and use polling instead. Is there a way to detect if a file is on a remote drive (other than using Runtime.exec and parsing the mount table)? I'm only concerned with Linux compatibility for now.
I suppose one option is to use both inotify and polling when the program starts, but then disable the polling thread if an inotify event for my file is created.
You should be able to get relatively reliable info about the underlying file system type with FileStore.type().
It will definitely tell you if it's an NFS, or CIFS, not sure about other network mount types.
However I have no info about how reliable it is, #hoaz's suggestion to check if events are coming through might be a good idea.
I had the same problem. I have solved it by creating a new thread in de main class and touching the files periodically so a new change event gets fired.
The sample polls the dir for every 10 seconds does a touch.
Here a sample of the code:
package com.ardevco.files;
import java.io.IOException;
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.FileTime;
import java.util.ArrayList;
import java.util.List;
public class Touch implements Runnable {
private Path touchPath;
public Touch(Path touchPath) {
this.touchPath = touchPath;
this.checkPath = checkPath;
}
public static void touch(Path file) throws IOException {
long timestamp = System.currentTimeMillis();
touch(file, timestamp);
}
public static void touch(Path file, long timestamp) throws IOException {
if (Files.exists(file)) {
FileTime ft = FileTime.fromMillis(timestamp);
Files.setLastModifiedTime(file, ft);
}
}
List<Path> listFiles(Path path) throws IOException {
final List<Path> files = new ArrayList<>();
try (DirectoryStream<Path> stream = Files.newDirectoryStream(path)) {
for (Path entry : stream) {
if (Files.isDirectory(entry)) {
files.addAll(listFiles(entry));
}
files.add(entry);
}
}
return files;
}
#Override
public void run() {
while (true) {
try {
for (Path path : listFiles(touchPath)) {
touch(path);
}
} catch (IOException e) {
System.out.println("Exception: " + e);
}
try {
Thread.sleep(10000L);
} catch (InterruptedException e) {
System.out.println("Exception: " + e);
}
}
}
}
I am running java 7 applications on unix machines. Is there a way to get the current umask value in pure java ?
In C I would use a combination of umask system calls, but I don't think I can call that in Java without resorting to JNI. Is there another approach ?
Edit: Here is a C example (from GUN libc docs):
mode_t
read_umask (void)
{
mode_t mask = umask (0);
umask (mask);
return mask;
}
A simple solution, if there is no Class/Method to get the umask, why don't you get it before call java and pass as a property?
Can you clarify? Do you want to read the umask of the application(the current java process)? Or do you want to read the umask value of some files on the file system?
You can use NIO (the used code is from the javadocs) to get some file attributes, or you can execute a shell command, since the process created with Runtime.execute inherits the umask of it's creator process.
So you should be able to solve your problem without the use of JNI.
package test;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.attribute.PosixFileAttributeView;
import java.nio.file.attribute.PosixFileAttributes;
import java.nio.file.attribute.PosixFilePermissions;
public class Test {
private static final String COMMAND = "/bin/bash -c umask -S";
public static String getUmask() {
final Runtime runtime = Runtime.getRuntime();
Process process = null;
try {
process = runtime.exec(COMMAND);
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String umask = reader.readLine();
if (process.waitFor() == 0)
return umask;
} catch (final IOException e) {
e.printStackTrace();
} catch (final InterruptedException e) {
e.printStackTrace();
Thread.currentThread().interrupt();
}
return "";
}
public static void main(String[] args) throws IOException {
/*
* NIO
*/
PosixFileAttributes attrs = Files.getFileAttributeView(Paths.get("testFile"), PosixFileAttributeView.class)
.readAttributes();
System.out.format("%s %s%n", attrs.owner().getName(), PosixFilePermissions.toString(attrs.permissions()));
/*
* execute shell command to get umask of current process
*/
System.out.println(getUmask());
}
}