Develop Reference

Converting serially reading multiple files to reading them in parallel? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
Following code in Java reads multiple files one after another serially and it works well till here. (The files are JSON and at this step they are stored in a String/Buffer without parsing.)
for (int fileIndex = 0; fileIndex < numberOfFiles; fileIndex++) {
BufferedReader br = new BufferedReader(new FileReader("Files/file" + fileIndex + ".json"));
try {
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(System.lineSeparator());
line = br.readLine();
}
String contentJSON = sb.toString();
} finally {
br.close();
}
}
How to read those files in parallel by using Threads ?
I could not match Multithreading to above code and every time got errors.

I've not tested this code directly (as I don't have a bunch of files to read), but the basic idea would be to do something like...
import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
public static void main(String[] args) {
new Test();
}
public Test() {
try {
int numberOfFiles = 10;
ExecutorService service = Executors.newFixedThreadPool(10);
List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
for (int fileIndex = 0; fileIndex < numberOfFiles; fileIndex++) {
workers.add(new ReadWorker(fileIndex));
}
List<Future<String>> results = service.invokeAll(workers);
for (Future<String> result : results) {
try {
String value = result.get();
} catch (ExecutionException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
} catch (InterruptedException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
public class ReadWorker implements Callable<String> {
private int fileIndex;
public ReadWorker(int fileIndex) {
this.fileIndex = fileIndex;
}
#Override
public String call() throws Exception {
try (BufferedReader br = new BufferedReader(new FileReader("Files/file" + fileIndex + ".json"))) {
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(System.lineSeparator());
line = br.readLine();
}
return sb.toString();
}
}
}
}
This will basically execute a series of Callables and wait for them all to complete, at which time, you can then read the results (or errors)
See the Executors trail for more details
Tested and verified version...
So, I dumped a series of files into a the Files folder at the root of my working directory, modified the above example to list all the files in that directory and read them....
import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
public static void main(String[] args) {
new Test();
}
public Test() {
File files[] = new File("Files").listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.getName().toLowerCase().endsWith(".svg");
}
});
try {
int numberOfFiles = files.length;
ExecutorService service = Executors.newFixedThreadPool(20);
List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
for (File file : files) {
workers.add(new ReadWorker(file));
}
System.out.println("Execute...");
List<Future<String>> results = service.invokeAll(workers);
System.out.println("Results...");
for (Future<String> result : results) {
try {
String value = result.get();
System.out.println(value);
} catch (ExecutionException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
service.shutdownNow();
} catch (InterruptedException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
public class ReadWorker implements Callable<String> {
private File file;
public ReadWorker(File file) {
this.file = file;
}
#Override
public String call() throws Exception {
System.out.println("Reading " + file);
try (BufferedReader br = new BufferedReader(new FileReader(file))) {
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(System.lineSeparator());
line = br.readLine();
}
return sb.toString();
}
}
}
}
And this works just fine and I have no issue.
java.io.FileNotFoundException: Files\file0.json is a localised issue you are going to have to solve. Does file0.json actually exist? Does it exist in the Files directory? Is the Files directory in the root of the working directory when the program is executed?
None of these issues can be solved by us, as we don't have access to your environment
Test #3
I then renamed all the files in my Files directory to file{x}.json using...
File files[] = new File("Files").listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.getName().toLowerCase().endsWith(".svg");
}
});
for (int index = 0; index < files.length; index++) {
File source = files[index];
File target = new File(source.getParent(), "file" + index + ".json");
source.renameTo(target);
}
And the modified the example slightly to include a File#exists report...
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
public static void main(String[] args) {
new Test();
}
public Test() {
try {
int numberOfFiles = 10;
ExecutorService service = Executors.newFixedThreadPool(20);
List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
for (int index = 0; index < numberOfFiles; index++) {
workers.add(new ReadWorker(index));
}
System.out.println("Execute...");
List<Future<String>> results = service.invokeAll(workers);
System.out.println("Results...");
for (Future<String> result : results) {
try {
String value = result.get();
System.out.println(value);
} catch (ExecutionException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
service.shutdownNow();
} catch (InterruptedException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
public class ReadWorker implements Callable<String> {
private int fileIndex;
public ReadWorker(int fileIndex) {
this.fileIndex = fileIndex;
}
#Override
public String call() throws Exception {
System.out.println("Reading " + fileIndex);
File file = new File("Files/file" + fileIndex + ".json");
System.out.println("File " + fileIndex + " exists = " + file.exists());
try (BufferedReader br = new BufferedReader(new FileReader(file))) {
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(System.lineSeparator());
line = br.readLine();
}
return sb.toString();
} finally {
System.out.println("All done here");
}
}
}
}
Which prints
Execute...
Reading 8
Reading 1
Reading 2
Reading 4
Reading 6
Reading 9
Reading 3
Reading 7
Reading 0
Reading 5
File 8 exists = true
File 1 exists = true
File 5 exists = true
File 4 exists = true
File 9 exists = true
File 2 exists = true
File 0 exists = true
File 3 exists = true
File 7 exists = true
File 6 exists = true
All done here
All done here
All done here
All done here
All done here
All done here
All done here
All done here
All done here
All done here
Results...
// I won't bore you with the results, as it's a lot of pointless text
which all worked without issues

Copy files from the list using threads

I'm working on a project that copies files with four threads.
I create List and store in there name of files to copy.
I want to use this 4 thread to work together, but I don't really get it how make it happend.
public class CopyingFiles implements Runnable
{
static File source = new File("C:\\test\\1\\");
static File dest = new File("C:\\test\\2\\");
#Override
public void run()
{
try
{
CopyingFromList(source, dest);
}
catch (IOException e)
{
e.printStackTrace();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
public static void CopyFile(File sourceFile, File destination) throws IOException
{
InputStream inputStream = null;
OutputStream outputStream = null;
try {
inputStream = new FileInputStream(sourceFile);
outputStream = new FileOutputStream(destination);
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) > 0)
{
outputStream.write(buffer, 0 ,length);
}
} finally {
if(inputStream != null)
{
inputStream.close();
}
if(inputStream != null)
{
outputStream.close();
}
}
}
public static void CopyingFromList(File source, File dest) throws IOException, InterruptedException
{
List<String> fileList = FilesList.CreateFilesList(source);
for(String file : fileList)
{
System.out.println(Thread.currentThread().getName() + " > " + FilesList.DestinationOfFile(source) + file + " > " + FilesList.DestinationOfFile(dest) + file );
CopyFile(new File(FilesList.DestinationOfFile(source) + file), new File(FilesList.DestinationOfFile(dest) + file));
}
}
}
AND SECOND CLASS
public class FilesList
{
static File source = new File("C:\\test\\1\\");
static File source1 = new File("C:\\test\\3\\");
static File dest = new File("C:\\test\\2\\");
static File dest1 = new File("C:\\test\\4\\");
public static List<String> CreateFilesList(File source) throws InterruptedException, IOException
{
List<String> fileList = new ArrayList<>(Arrays.asList(source.list()));
return fileList;
}
public static String DestinationOfFile(File source)
{
return new String(source + "\\");
}
public static void PrintWholeList(File source) throws IOException, InterruptedException
{
List<String> fileList = CreateFilesList(source);
for(String file : fileList)
{
System.out.println(DestinationOfFile(source) + file);
}
}
public static void main(String []args) throws IOException, InterruptedException
{ /*
//CopyingFiles.CopyFile(new File(source+"\\file1.txt"), new File(dest+"\\file1.txt"));
//CopyingFiles.CopyingFromList(source,dest);
CopyingFiles t1 = new CopyingFiles();
CopyingFiles t2 = new CopyingFiles();
CopyingFiles t3 = new CopyingFiles();
CopyingFiles t4 = new CopyingFiles();
t1.start();
t2.start();
t3.start();
t4.start();
*/
ExecutorService executorService = Executors.newFixedThreadPool(5);
System.out.println(Thread.activeCount());
executorService.submit(() -> {
try
{
CopyingFiles.CopyingFromList(source,dest);
}
catch (IOException e)
{
e.printStackTrace();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
finally
{
executorService.shutdown();
}
});
}
}
Could anyone help me,or show some other way to solve my problem.

I don't actually get what the problem is. This one works for me (and it is almost your program, just cleaned up a bit):
package multicp;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import static java.nio.file.StandardCopyOption.COPY_ATTRIBUTES;
/**
* Copy files using threads.
*/
public class Multicp {
public static void main(String[] args) {
// List of source/dest pairs ("tasks")
List<CopierCallable<Void>> opsList = new ArrayList<>();
opsList.add(new CopierCallable<>(Paths.get("f1src.dat"), Paths.get("f1dest.dat")));
opsList.add(new CopierCallable<>(Paths.get("f2src.dat"), Paths.get("f2dest.dat")));
opsList.add(new CopierCallable<>(Paths.get("f3src.dat"), Paths.get("f3dest.dat")));
opsList.add(new CopierCallable<>(Paths.get("f4src.dat"), Paths.get("f4dest.dat")));
ExecutorService execSvc = Executors.newFixedThreadPool(2); // 4 in your case. 2 is just for testing
try {
execSvc.invokeAll(opsList);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
execSvc.shutdown();
}
}
}
/**
* Performs actual copying from one source to one destination.
*/
class CopierCallable<Void> implements Callable<Void> {
private Path pathFrom;
private Path pathTo;
public CopierCallable(Path pathFrom, Path pathTo) {
this.pathFrom = pathFrom;
this.pathTo = pathTo;
}
#Override
public Void call() {
try {
// REPLACE_EXISTING is destructive, uncomment at your own risk
Files.copy(pathFrom, pathTo, COPY_ATTRIBUTES /*, REPLACE_EXISTING*/);
System.out.println(pathFrom + " copied");
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}
I can see files being copied simultaneously in groups (by 2 for testing; replace with 4, but it would make initialization code larger for no profit).

Java application not completing

Was trying to get a JFrame added to see if it would help with using launch4j to convert a small jar file to an .exe. I wrote a short program to help sort HPLC data at work and want to make it just a simple point and click.
It works when I run it from the command line java KFile and the JFileChooser lets me choose directories for the script to work on. When I converted it to the .exe, the JFileChooser never rendered and the .exe closes.
I read that I might need a JFrame parent and so I created a JFrame, but now the script hangs before completion as if waiting for the frame to close. I'm pretty new to java, so I'm not sure how I to resolve this issue.
import java.io.*;
import java.nio.channels.FileChannel;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.Files;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.file.FileVisitResult;
import java.nio.MappedByteBuffer;
import javax.swing.JFileChooser;
import javax.swing.JFrame;
import java.util.Collection;
import java.util.ArrayList;
import java.nio.file.SimpleFileVisitor;
public class KFile extends SimpleFileVisitor<Path> {
public static void main(String[] args) {
Path currPath = Paths.get("");
String currDir = currPath.toAbsolutePath().toString();
System.out.println(currDir);
File dataDir = chooseDir("open");
File destDir = chooseDir("save");
if(!destDir.exists()) {
try {
destDir.mkdir();
}
catch (SecurityException se) {
System.out.println("Couldn't make directory!");
}
}
int n = 0;
if(dataDir.exists()) {
Collection<Path> allDir = new ArrayList<Path>();
try {
addTree(dataDir.toPath(),allDir);
}
catch (IOException e) {
System.out.println("Error with scanning");
}
for( Path thisPath : allDir ) {
if(thisPath.toString().contains("Report.pdf")) {
Path thisDir = thisPath.getParent();
File f = new File(thisDir.toString(), "\\Report.txt");
n = n + 1;
String fileName = "Report " + n + ".pdf";
try {
fileName = parseName(f);
System.out.println(fileName);
} catch (IOException e) {
e.printStackTrace();
}
File thisFile = new File(destDir + "\\" + fileName);
try {
copyFile(thisPath.toFile(),thisFile);
} catch ( IOException e) {
e.printStackTrace();
}
}
}
}
}
public static boolean copyFile(File sourceFile, File destFile) throws IOException {
//create file if it doesn't exist.
if(!destFile.exists()) {
destFile.createNewFile();
}
FileChannel source = null;
FileChannel destination = null;
try {
source = new FileInputStream(sourceFile).getChannel();
destination = new FileOutputStream(destFile).getChannel();
destination.transferFrom(source, 0, source.size());
}
finally {
if(source != null) {
source.close();
}
if(destination != null) {
destination.close();
return true;
}
return false;
}
}
public static File chooseDir(String s) {
JFrame myFrame = new JFrame("HPLC Data Transfer");
myFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
myFrame.pack();
myFrame.setVisible(true);
JFileChooser chooser = new JFileChooser();
File currDir = new File(System.getProperty("user.home") + "\\Documents");
chooser.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
chooser.setCurrentDirectory(currDir);
int choice = 0;
if (s.equals("save")) {
choice = chooser.showSaveDialog(myFrame);
} else {
choice = chooser.showOpenDialog(myFrame);
}
myFrame.setVisible(false);
myFrame.removeAll();
if(choice == JFileChooser.APPROVE_OPTION) {
System.out.println("You chose to open: " + chooser.getSelectedFile().getName());
return chooser.getSelectedFile();
}
return new File("");
}
static String parseName(File f) throws IOException {
BufferedReader textReader = new BufferedReader(new InputStreamReader(new FileInputStream(f), "UTF-16"));
int lnCnt = 32;
String[] fileData = new String[lnCnt];
for (int i = 0; i < lnCnt; i++) {
fileData[i] = textReader.readLine();
}
fileData[1] = fileData[1].replace("\uFEFF","");
String name = fileData[1].substring(13) + ".pdf";
textReader.close();
return name;
}
static void addTree(Path directory, final Collection<Path> all)
throws IOException {
Files.walkFileTree(directory, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs)
throws IOException {
all.add(file);
return FileVisitResult.CONTINUE;
}
});
}
}

You could try changing
myFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
to
myFrame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
and then call
myFrame.dispose();
to terminate the JFrame.
Since javadocs says EXIT_ON_CLOSE terminates the whole program using System.exit(); I'm not sure if that's the problem that is stopping your application but I hope it helps :)

It looks like you just called setVisible(false) when dealing with your JFrame. That just hides your JFrame, it doesn't get rid of it. If you want to get rid of your frame entirely (and all of its resources), call myFrame.dispose();

ArrayList not persisting between methods

I'm trying to load a csv file into an arrayList to later break it up and store it. Between my methods the arrayList is being reset to null. I'm confused as to the cause and would be grateful for any advice
package TestInput;
import java.io.BufferedInputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
public class Bean {
private String fileContent;
private String fileContent2;
private ArrayList<String> fileContentArray;
private int counter = 0;
public int getCounter() {
return counter;
}
public void setCounter(int counter) {
this.counter = counter;
}
public ArrayList<String> getFileContentArray() {
return fileContentArray;
}
public void setFileContentArray(ArrayList<String> fileContentArray) {
this.fileContentArray = fileContentArray;
}
public String getFileContent() {
return fileContent;
}
public void setFileContent(String fileContent) {
this.fileContent = fileContent;
}
public String getFileContent2() {
return fileContent2;
}
public void setFileContent2(String fileContent2) {
this.fileContent2 = fileContent2;
}
public void upload() {
File file = new File("/Users/t_sedgman/Desktop/FinalProject/test_output_data.rtf");
FileInputStream fis = null;
BufferedInputStream bis = null;
DataInputStream dis = null;
ArrayList<String> tempArray = new ArrayList<>();
try {
fis = new FileInputStream(file);
// Here BufferedInputStream is added for fast reading.
bis = new BufferedInputStream(fis);
dis = new DataInputStream(bis);
// dis.available() returns 0 if the file does not have more lines.
while (dis.available() != 0) {
// this statement reads the line from the file and print it to
// the console.
tempArray.add(dis.readLine());
}
setFileContentArray(tempArray);
// dispose all the resources after using them.
fis.close();
bis.close();
dis.close();
fileContent = fileContentArray.get((fileContentArray.size() - 2));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
public void next() {
ArrayList<String> tempArray = getFileContentArray();
int size = fileContentArray.size();
if (counter <= size) {
counter++;
fileContent2 = tempArray.get(counter);
} else {
counter = 0;
}
}
}
Many Thanks
Tom

You can try By marking your bean with #ViewScoped/#SessionScoped

FileReader Error when I read a text file

I am trying to store string of array in a text file and read it. But I can't get it working. I am getting NullPointerError.
Exception in thread "main" java.lang.NullPointerException
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileReader.<init>(Unknown Source)
at in.isuru.justconverter.FileDbTool.readFile(FileDbTool.java:41)
at in.isuru.justconverter.Test.main(Test.java:10)
Here's two classes.
package in.isuru.justconverter;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.StringTokenizer;
import javax.swing.JOptionPane;
public class FileDbTool {
File dataFile;
ArrayList<String> filePath;
public void checkFile(){
dataFile = new File("db.txt");
if(dataFile.exists()){
readFile();
}else{
try {
dataFile.createNewFile();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
JOptionPane.showMessageDialog(null, "Coudn't Create New File!");
System.exit(1);
}
}
}
public void readFile(){
int len;
try{
char[] chr = new char[4096];
StringBuffer buffer = new StringBuffer();
FileReader reader = new FileReader(dataFile);
try {
while ((len = reader.read(chr)) > 0) {
buffer.append(chr, 0, len);
}
}finally {
reader.close();
}
System.out.println(buffer.toString());
StringTokenizer st = new StringTokenizer(buffer.toString(), ",");
while (st.hasMoreTokens()) {
String value = st.nextToken();
filePath = null;
filePath = new ArrayList<String>();
filePath.add(value);
}
}catch(IOException e){
JOptionPane.showMessageDialog(null, "Read Error");
}
}
public String[] getFilePathArray(){
readFile();
return filePath.toArray(new String[filePath.size()]);
}
public File[] getFiles(){
String[] paths = getFilePathArray();
ArrayList<File> files = new ArrayList<File>();
for(int i = 0; i < paths.length; i++){
File file = new File(paths[i]);
files.add(file);
}
return files.toArray(new File[files.size()]);
}
public void eraseFile(){
dataFile.delete();
}
public void writeFile(String[] stuff){
try{
BufferedWriter out = new BufferedWriter(new FileWriter(dataFile, true));
out.append(stuff + ",");
}catch(IOException e){
}
}
public void writeToDb(String[] array){
writeFile(array);
}
}
And main class
package in.isuru.justconverter;
public class Test {
/**
* #param args
*/
public static void main(String[] args) {
FileDbTool app = new FileDbTool();
app.checkFile();
}
}
Well this is a portion of a swing program. I am trying to use text file as a small database.

Line 41 is this:
FileReader reader = new FileReader(dataFile);
so I'd wager that dataFile is null here.
However, you do seem to initialize it before calling this method, otherwise the exception would be thrown inside checkFile.
Are you sure you are not calling readFile directly somewhere without calling checkFile first? In any case, this pattern is not a recommended approach, because you are requiring the users of your class to call methods in a specific order.

From the stack trace , it seems like you called readfile() directly from main rather than through checkfile() . So dataFile is null since it is not initialized by checkfile . Also the stack trace and the given code doesn't match . When FileReader constructor is called with null argument , it will throw NullPointerException when it reaches FileInputstream constructor .
Here is the code from jdk source :
public FileInputStream(File file) throws FileNotFoundException {
String name = (file != null ? file.getPath() : null);
SecurityManager security = System.getSecurityManager();
if (security != null) {
security.checkRead(name);
}
if (name == null) {
throw new NullPointerException();
}
fd = new FileDescriptor();
fd.incrementAndGetUseCount();
open(name);
}