Develop Reference

Limit file size while writing in java

I need to limit the file size to 1 GB while writing preferably using BufferedWriter.
Is it possible using BufferedWriter or I have to use other libraries ?
like
try (BufferedWriter writer = Files.newBufferedWriter(path)) {
//...
writer.write(lines.stream());
}

You can always write your own OutputStream to limit the number of bytes written.
The following assumes you want to throw exception if size is exceeded.
public final class LimitedOutputStream extends FilterOutputStream {
private final long maxBytes;
private long bytesWritten;
public LimitedOutputStream(OutputStream out, long maxBytes) {
super(out);
this.maxBytes = maxBytes;
}
#Override
public void write(int b) throws IOException {
ensureCapacity(1);
super.write(b);
}
#Override
public void write(byte[] b) throws IOException {
ensureCapacity(b.length);
super.write(b);
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
ensureCapacity(len);
super.write(b, off, len);
}
private void ensureCapacity(int len) throws IOException {
long newBytesWritten = this.bytesWritten + len;
if (newBytesWritten > this.maxBytes)
throw new IOException("File size exceeded: " + newBytesWritten + " > " + this.maxBytes);
this.bytesWritten = newBytesWritten;
}
}
You will of course now have to set up the Writer/OutputStream chain manually.
final long SIZE_1GB = 1073741824L;
try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
new LimitedOutputStream(Files.newOutputStream(path), SIZE_1GB),
StandardCharsets.UTF_8))) {
//
}

Exact bytes to 1 GB is very difficult in cases where you are writing lines. Each line may contain unknown number of bytes in it. I am assuming you want to write data line by line in file.
However, you can check how many bytes does line has before writing it to the file and another approach is to check file size after writing each line.
Following basic example writes one same line each time. Here This is just a test ! text takes 21 bytes on file in UTF-8 encoding. Ultimately after 49 writes it reaches to 1029 Bytes and stops writing.
public class Test {
private static final int ONE_KB = 1024;
public static void main(String[] args) {
File file = new File("D:/test.txt");
try (BufferedWriter writer = Files.newBufferedWriter(file.toPath())) {
while (file.length() < ONE_KB) {
writer.write("This is just a test !");
writer.flush();
}
System.out.println("1 KB Data is written to the file.!");
} catch (IOException e) {
e.printStackTrace();
}
}
}
As you can see we have already written out of the limit of 1KB as above program writes 1029 Bytes and not less than 1024 Bytes.
Second approach is checking the bytes according to specific encoding before writing it to file.
public class Test {
private static final int ONE_KB = 1024;
public static void main(String[] args) throws UnsupportedEncodingException {
File file = new File("D:/test.txt");
String data = "This is just a test !";
int dataLength = data.getBytes("UTF-8").length;
try (BufferedWriter writer = Files.newBufferedWriter(file.toPath())) {
while (file.length() + dataLength < ONE_KB) {
writer.write(data);
writer.flush();
}
System.out.println("1 KB Data written to the file.!");
} catch (IOException e) {
e.printStackTrace();
}
}
}
In this approach we check length of bytes prior to writing it to the file. So, it will write 1008 Bytes and it will stop writing.
Problems with both the approaches,
Write and Check : You may end up with some extra bytes and file size may cross the limit
Check and Write : You may have less bytes than the limit if next line has lot of data in it. You should be careful about the encoding.
However, there are other ways to do this validations with some third party library like apache io and I find it more cumbersome then conventional java ways.

int maxSize = 1_000_000_000;
Charset charset = StandardCharsets.UTF_F);
int size = 0;
int lineCount = 0;
while (lineCount < lines.length) {
long size2 = size + (lines[lineCount] + "\r\n").getBytes(charset).length;
if (size2 > maxSize) {
break;
}
size = size2;
++lineCount;
}
List<String> linesToWrite = lines.substring(0, lineCount);
Path path = Paths.get("D:/test.txt");
Files.write(path, linesToWrite , charset);
Or a bit faster while decoding only once:
int lineCount = 0;
try (FileChannel channel = new RandomAccessFile("D:/test.txt", "w").getChannel()) {
ByteBuffer buf = channel.map(FileChannel.MapMode.WRITE, 0, maxSize);
lineCount = lines.length;
for (int i = 0; i < lines.length; i++) {
bytes[] line = (lines.get(i) + "\r\n").getBytes(charset);
if (line.length > buffer.remaining()) {
lineCount = i;
break;
}
buffer.put(line);
}
}

IIUC, there are various ways to do it.
Keep writing data in chucks and flushing it and keep checking the file size after every flush.
Use log4j (or some logging framework) which can let us rollover to new file after certain size or time or some other trigger point.
While BufferedReader is great, there are some new APIs in java which could make it faster. Fastest way to write huge data in text file Java

Replacing text from a file in Java

I've been trying to create a simple program, in java, that replaces some words into a file. To change the text into the file I created, I create a String and set it to the file text:
Path path = Paths.get("somePath/someFile.someExtension");
Charset charset = StandardCharsets.UTF_8;
String s = new String(Files.readAllBytes(path), charset);
EDIT: To save to the file with s, I used Files.write(path, s.getBytes(charset));.
Then I change the String with commands like s.replaceAll("A", "B"). But now, I'm stuck. I want to make a more complicated then just replacing "A" with "B". I'll try to explain has best I can:
I need to find at the file if wall someNumber someNumer someNumber is in there, and if there are three arguments (someNumber someNumber someNumber), then get the value of "someNumber" at the center. For example:
If the commands are:
wall 200 500 100
wall 200 500 100
Then I want to get the argument from the center (in the first case 500 and in the second 500), and store it into a variable, then delete it from the String. After that, on the top of these commands (in the example wall 200 500 100 wall 200 500 100), I want to write:
usemtl texture
ceil (someNumber that we stored, in the case, 500)
Note that if the arguments wall wall don't have any kind of separation (for example #other wall), then the someNumber at the center will be equal (500 and 500 are equal). So, the command below will just appear per group (if wall wall wall... isn't separed with #other wall, for example).
Other example, this would be the file before / after:
Before:
wall 100 300 50
wall 100 300 100
wall 100 300 400
After:
usemtl texture
ceil 300
wall 100 50
wall 100 100
wall 100 400
So, how can I make this replace?
Please answer! I have no idea how!
EDIT: question to #Roan, owner of most of this code:
Now, after the answers #Roan code transformed into:
package com.fileConverter.main;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import javax.swing.JFileChooser;
public class FileReplace extends JFileChooser {
private static final long serialVersionUID = -254322941935132675L;
private static FileReplace chooser = new FileReplace();
public static void main(String[] args) {
chooser.showDialog(chooser, "Open");
}
public void cancelSelection() {
System.exit(0);
}
public void approveSelection() {
super.approveSelection();
System.out.println("starting...");
// The path were your file is
String path = chooser.getSelectedFile().getAbsolutePath();
File file = new File(path);
// try to create an inputstream from the file
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
} catch (FileNotFoundException e) {
// If we are here the file is not found
e.printStackTrace();
}
// make it a buffered reader
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(fis));
// to store the current line
String line;
// array to store the different words
String[] words;
// create a second temporally file that will replace the original file
File file2 = new File(chooser.getSelectedFile().getParentFile()
+ "$$$$$$$$$$$$$$$.tmp");
try {
file.createNewFile();
} catch (IOException e1) {
e1.printStackTrace();
}
// and create the streams
FileOutputStream file2Os = null;
try {
file2Os = new FileOutputStream(file2);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
PrintWriter writer = new PrintWriter(file2Os);
try {
System.out.println("replacing code...");
writer.println("mtllib textures.mtl");
// loop through all lines and
while ((line = bufferedReader.readLine()) != null) {
line = line
.replace("//", "#")
.replace("(", "wall")
.replace(")", "\n")
.replace("{", "")
.replace("}", "")
.replace("# brush from cube",
"room cube" + countWords(line, "cube"))
.replace(" NULL 0 0 0 1 1 0 0 0", "")
.replace("\"classname\"", "")
.replace("\"worldspawn\"", "");
// get all the diffent terms
words = line.split(" ");
// see if there are 4 terms in there: wall x x x
// and if the first term equals wall28
// and if the middle number is the number you want to delete
// if not just copy the line over
if (words.length == 4 && words[0].contains("wall")) {
double doubleVal = Double.parseDouble(words[2]);
int val = (int) doubleVal;
// now modify the line by removing the middel number
String newLine = words[0] + " " + words[1] + " " + words[3];
String valInsert = null;
if (val >= 0)
valInsert = "\n" + "usemtl texture" + "\n" + "ceil "
+ val;
else if (val < 0)
valInsert = "\n" + "usemtl texture" + "\n" + "floor "
+ val;
// write this to the new file
writer.println(valInsert);
writer.println(newLine);
} else {
// copy the old line
writer.println(line);
}
}
} catch (IOException e) {
e.printStackTrace();
}
// close our resources
writer.close();
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
// now we rename the temp file and replace the old file
// with the new file with the new content
file.delete();
file2.renameTo(file);
System.out.println("done!");
}
public int countWords(String string, String word) {
int lastIndex = 0;
int count = 0;
while (lastIndex != -1) {
lastIndex = string.indexOf(word, lastIndex);
if (lastIndex != -1) {
count++;
lastIndex += word.length();
}
}
return count;
}
}
The problem is that this part doesn't make any replaces:
if (words.length == 4 && words[0].contains("wall")) {
double doubleVal = Double.parseDouble(words[2]);
int val = (int) doubleVal;
// now modify the line by removing the middel number
String newLine = words[0] + " " + words[1] + " " + words[3];
String valInsert = null;
if (val >= 0)
valInsert = "\n" + "usemtl texture" + "\n" + "ceil "
+ val;
else if (val < 0)
valInsert = "\n" + "usemtl texture" + "\n" + "floor "
+ val;
// write this to the new file
writer.println(valInsert);
writer.println(newLine);
}
How can I fix it? Other thing, this part is suppose to create a number that grows after checking how many times cube is wrote, but it doesn't works too :(
.replace("# brush from cube", "room cube" + countWords(line, "cube"))
The countWords method:
public int countWords(String string, String word) {
int lastIndex = 0;
int count = 0;
while (lastIndex != -1) {
lastIndex = string.indexOf(word, lastIndex);
if (lastIndex != -1) {
count++;
lastIndex += word.length();
}
}
return count;
}
Many Thanks

Ok I'm very unsure if I understood this correctly.
This is my interpretation of wat your question is:
You have a file with lines that say: wall [number] [number] [number]
Now you want to check if there there are 3 numbers and then delete the middle number if it's the number you're searching for.
So I would do this like this:
Befor you run the programm you'll need a folder on your C: drive with the name "text" and inside that folder you'll need a file called text.txt with you format in it so: for example:
wall 123 300 320
If you change the value of number you can specify wich number the middle number has to be in oder for it to be deleted.
public class FileReplace {
public static void main(String[] args){
//The path were your file is
String path = "C:\\text\\text.txt";
File file = new File(path);
//The number you want to delete
int number = 300;
//try to create an inputstream from the file
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
} catch (FileNotFoundException e) {
//If we are here the file is not found
e.printStackTrace();
}
//make it a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(fis));
//to store the current line
String line;
//array to store the different words
String[] words;
//create a second temporally file that will replace the original file
File file2 = new File("C:\\text\\$$$$$$$$$$.tmp");
try {
file.createNewFile();
} catch (IOException e1) {
e1.printStackTrace();
}
//and create the streams
FileOutputStream fos = null;
try {
fos = new FileOutputStream(file2);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
PrintWriter writer = new PrintWriter(fos);
try {
//loop through all lines and
while((line = bufferedReader.readLine()) != null){
//get all the diffent terms
words = line.split(" ");
//see if there are 4 terms in there: wall x x x
//and if the first term equals wall
//and if the middle number is the number you want to delete
//if not just copy the line over
if(words.length == 4 && words[0].equals("wall") && words[2].equals(String.valueOf(number))){
//now modify the line by removing the middel number
String newLine = words[0] + " " + words[1] + " " + words[3];
//write this to the new file
writer.println(newLine);
}else{
//copy the old line
writer.println(line);
}
}
} catch (IOException e) {
e.printStackTrace();
}
//close our resources
writer.close();
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
//now we rename the temp file and replace the old file
//with the new file with the new content
file.delete();
file2.renameTo(file);
}
}
If you have ay questions about this code feel free to ask them.
Oh and also you might need to run this with administrator right as it uses files.
Hope this helps.

To analyse a string and see if it matches ("wall" number number number), you can use a REGEX expression: see the doc here.
To use the regex expression, just apply .matches() on your String variable and it'll return true or false depending on if the format is verified.
If the format is verified, then just use the SubString function, specify the start and end index so you get the middle number.
To take it out, you could do the opposite. SubString the start (everything until middle number), then SubString the end (everything after the middle number), and then create a new string using those 2.

A simple solution without using (explicitly) regex is to split the String using token (in your case it's a white space.
line = "wall 100 300 50";
String[] words = line.split("\\s+");
You can then get the words[2] convert to an int etc. Then you can write back to a new file (or the same if you have read all file contents).
Regex are more powerful but to me a bit more intimidating so you can pick whatever matches your needs.

You could use this to count the number of occurances of a word in a string:Try 1:
public static int countWords(String string, String word) {
//get all individual words
String[] terms = string.split(" ");
//this variable counts how many times word occurs
int count = 0;
//a count variable for the loop
int counter = 0;
//loop through all the words and if we see a word that equals word we add one to the count variable
while(counter < terms.length){
//check if the term equals the word
if(terms[counter].equals(word)){
//the term matched add one to the count variable
count++;
}
counter++;
}
//return the number of occurrences
return count;
}
Try 2:
public static String countWords(String string, String word) {
//get all individual words
String[] terms = string.split(" ");
//this variable counts how many times word occurs
int count = 0;
//a count variable for the loop
int counter = 0;
StringBuffer sb = new StringBuffer();
sb.append("1");
//loop trough all the words and if we see a word that equals word we add one to the count variable
while(counter < terms.length){
//check if the term equals the word
if(terms[counter].equals(word)){
//the term matched add one to the count variable
count++;
sb.append(" " + word + (count + 1));
}
counter++;
}
//return the number of occurrences
return sb.toString();
}<br><br>
Try 3: you need to have a static variable in your code called lastVar:
static int lastVar = 0;
public static String countWords(String string, String word) {
//get all individual words
String[] terms = string.split(" ");
//this variable counts how many times word occurs
int count = 0;
//a count variable for the loop
int counter = 0;
StringBuffer sb = new StringBuffer();
sb.append("1");
//loop trough all the words and if we see a word that equals word we add one to the count variable
while(counter < terms.length){
//check if the term equals the word
if(terms[counter].equals(word)){
//the term matched add one to the count variable
count++;
sb.append(" " + word + (count + 1 + lastVar));
}
counter++;
}
lastVar += count + 1;
//return the number of occurrences
return sb.toString();
}
That should work.
Hope this helps :D.

To reformat your cube lines you could use:
Try 1:
// loop through all lines and
while ((line = bufferedReader.readLine()) != null) {
if(line.contains("// brush from cube")){
line = line.replace("// brush from cube ", "").replace("(", "").replace(")", "");
String[] arguments = line.split("\\s+");
line = "cube" + Cube + " usemtl texture ceil " + arguments[2] + " wall " + arguments[1] + " " + arguments[3] + " usemtl texture floor " + arguments[5] + " wall " + arguments[4] + " " + arguments[6];
Cube++;
}
line = line
.replace("//", "#")
.replace("(", "wall")
.replace(")", "\n")
.replace("{", "")
.replace("}", "")
.replace(" NULL 0 0 0 1 1 0 0 0", "")
.replace("\"classname\"", "")
.replace("\"worldspawn\"", "");
try 2:
// loop through all lines and
while ((line = bufferedReader.readLine()) != null) {
if(line.contains("// brush from cube")){
line = line + bufferedReader.readLine() + " " + bufferedReader.readLine();
line = line.replace("// brush from cube ", "").replace("(", "").replace(")", "");
String[] arguments = line.split("\\s+");
line = "cube" + Cube + " usemtl texture ceil " + arguments[2] + " wall " + arguments[1] + " " + arguments[3] + " usemtl texture floor " + arguments[5] + " wall " + arguments[4] + " " + arguments[6];
Cube++;
}
line = line
.replace("//", "#")
.replace("(", "wall")
.replace(")", "\n")
.replace("{", "")
.replace("}", "")
.replace(" NULL 0 0 0 1 1 0 0 0", "")
.replace("\"classname\"", "")
.replace("\"worldspawn\"", "");
P.S. I've only posted the important part. You should be able to see where in the code this goes. Also you need to have a static int called cube somewhere in you code like:
static int Cube = 1;
That should be it if it doesn't work let me know! :D

java: need to increase performance of checksum calculation

I'm using the following function to calculate checksums on files:
public static void generateChecksums(String strInputFile, String strCSVFile) {
ArrayList<String[]> outputList = new ArrayList<String[]>();
try {
MessageDigest m = MessageDigest.getInstance("MD5");
File aFile = new File(strInputFile);
InputStream is = new FileInputStream(aFile);
System.out.println(Calendar.getInstance().getTime().toString() +
" Processing Checksum: " + strInputFile);
double dLength = aFile.length();
try {
is = new DigestInputStream(is, m);
// read stream to EOF as normal...
int nTmp;
double dCount = 0;
String returned_content="";
while ((nTmp = is.read()) != -1) {
dCount++;
if (dCount % 600000000 == 0) {
System.out.println(". ");
} else if (dCount % 20000000 == 0) {
System.out.print(". ");
}
}
System.out.println();
} finally {
is.close();
}
byte[] digest = m.digest();
m.reset();
BigInteger bigInt = new BigInteger(1,digest);
String hashtext = bigInt.toString(16);
// Now we need to zero pad it if you actually / want the full 32 chars.
while(hashtext.length() < 32 ){
hashtext = "0" + hashtext;
}
String[] arrayTmp = new String[2];
arrayTmp[0] = aFile.getName();
arrayTmp[1] = hashtext;
outputList.add(arrayTmp);
System.out.println("Hash Code: " + hashtext);
UtilityFunctions.createCSV(outputList, strCSVFile, true);
} catch (NoSuchAlgorithmException nsae) {
System.out.println(nsae.getMessage());
} catch (FileNotFoundException fnfe) {
System.out.println(fnfe.getMessage());
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
}
}
The problem is that the loop to read in the file is really slow:
while ((nTmp = is.read()) != -1) {
dCount++;
if (dCount % 600000000 == 0) {
System.out.println(". ");
} else if (dCount % 20000000 == 0) {
System.out.print(". ");
}
}
A 3 GB file that takes less than a minute to copy from one location to another, takes over an hour to calculate. Is there something I can do to speed this up or should I try to go in a different direction like using a shell command?
Update: Thanks to ratchet freak's suggestion I changed the code to this which is ridiculously faster (I would guess 2048X faster...):
byte[] buff = new byte[2048];
while ((nTmp = is.read(buff)) != -1) {
dCount += 2048;
if (dCount % 614400000 == 0) {
System.out.println(". ");
} else if (dCount % 20480000 == 0) {
System.out.print(". ");
}
}

use a buffer
byte[] buff = new byte[2048];
while ((nTmp = is.read(buff)) != -1)
{
dCount+=ntmp;
//this logic won't work anymore though
/*
if (dCount % 600000000 == 0)
{
System.out.println(". ");
}
else if (dCount % 20000000 == 0)
{
System.out.print(". ");
}
*/
}
edit: or if you don't need the values do
while(is.read(buff)!=-1)is.skip(600000000);
nvm apparently the implementers of DigestInputStream were stupid and didn't test everything properly before release

Have you tried removing the println's? I imagine all that string manipulation could be consuming most of the processing!
Edit: I didn't read it clearly, I now realise how infrequently they'd be output, I'd retract my answer but I guess it wasn't totally invaluable :-p (Sorry!)

The problem is that System.out.print is used too often. Every time it is called new String objects have to be created and it is expensive.
Use StringBuilder class instead or its thread safe analog StringBuffer.
StringBuilder sb = new StringBuilder();
And every time you need to add something call this:
sb.append("text to be added");
Later, when you are ready to print it:
system.out.println(sb.toString());

Frankly there are several problems with your code that makes it slow:
Like ratchet freak said, disk reads must be buffered because Java read()'s are probably translated to operating system IOs calls without automatically buffering, so one read() is 1 system call!!!
The operating system will normally perform much better if you use an array as buffer or the BufferedInputStream. Better yet, you can use nio to map the file into memory and read it as fast as the OS can handle it.
You may not believe it, but the dCount++; counter may have used a lot of cycles. I believe even for the latest Intel Core processor, it takes several clock cycles to complete a 64-bit floating point add. You will be much better of to use a long for this counter.
If the sole purpose of this counter is to display progress, you can make use of the fact that Java integers overflow without causing an Error and just advance your progress display when a char type wraps to 0 (that's per 65536 reads).
The following string padding is also inefficient. You should use a StringBuilder or a Formatter.
while(hashtext.length() < 32 ){
hashtext = "0"+hashtext;
}
Try using a profiler to find further efficiency problems in your code

Multiple 'if' statements into one outputStream

import java.io.*;
import java.io.File;
import java.io.FilenameFilter;
public class YDSearch{
public void listFiles(String dir) throws IOException{
File directory = new File(dir);
if (!directory.isDirectory()) {
System.out.println("No directory provided");
return;
}
//create a FilenameFilter and override its accept-method
FilenameFilter filefilter = new FilenameFilter() {
public boolean accept(File dir, String name) {
//if the file extension is .mp3 return true, else false
return name.endsWith(".mp3")||name.endsWith(".mp4")||name.endsWith(".3gp")
||name.endsWith(".mov")||name.endsWith(".avi")||name.endsWith(".wmv");
}
};
String[] filenames = directory.list(filefilter);
DataOutputStream output = new DataOutputStream(new FileOutputStream("C:/Users/Jonathan/Desktop/YouDetect/SearchByFileType/AllMediaFiles.dat"));
for (String name : filenames) {
output.writeUTF(dir + name);
}
output.close();
DataInputStream input = new DataInputStream(new FileInputStream("C:/Users/Jonathan/Desktop/YouDetect/SearchByFileType/AllMediaFiles.dat"));
DataOutputStream output2 = new DataOutputStream(new FileOutputStream("C:/Users/Jonathan/Desktop/ReadyForAnalysis.dat"));
for (String name : filenames) {
FileInputStream in = new FileInputStream(input.readUTF());
int byteCounter = 0;
int rowCounter = 0;
long bufferCounter = 0;
if(name.endsWith(".mp3")){
byte[] b = new byte[36];
int read = in.read(b, 0, 36);
if (byteCounter != 1000){
if (rowCounter == 1){
System.out.println("\n");
rowCounter = 0;
}
output2.writeUTF(org.apache.commons.codec.binary.Hex.encodeHexString(b)+ " " + dir + name);
bufferCounter ++;
rowCounter ++;
}else{
byteCounter = 0;
try{
Thread.sleep(200);
}catch(InterruptedException e) {
}
}
}
else if(name.endsWith(".mp4")){
byte[] b = new byte[29];
int read = in.read(b, 0, 29);
if (byteCounter != 1000){
if (rowCounter == 1){
System.out.println("\n");
rowCounter = 0;
}
output2.writeUTF(org.apache.commons.codec.binary.Hex.encodeHexString(b)+ " " + dir + name);
bufferCounter ++;
rowCounter ++;
}else{
byteCounter = 0;
try{
Thread.sleep(200);
}catch(InterruptedException e) {
}
}
}
//System.out.println("====================");
}
output2.close();
input.close();
DataInputStream input2 = new DataInputStream(new FileInputStream("C:/Users/Jonathan/Desktop/ReadyForAnalysis.dat"));
for (String name : filenames) {
System.out.println(input2.readUTF()+"\n");
}
}
public void checkHeaderSC(String allFiles)throws IOException{
}
public static void main(String[] args) throws IOException {
YDSearch YDSearch = new YDSearch();
YDSearch.listFiles("C:/Users/Jonathan/Desktop/YD Tests/1) High Quality/");
YDSearch.listFiles("C:/Users/Jonathan/Desktop/YD Tests/2) Medium Quality/");
YDSearch.listFiles("C:/Users/Jonathan/Desktop/YD Tests/3) Low Quality/");
YDSearch.checkHeaderSC("C:/Users/Jonathan/Desktop/YouDetect/SearchByFileType/ReadyForAnalysis.dat");
}
}
Hey there, having a little issue with the above coding and hoped someone here might be able to help. This is sort of a partial version of the code as the real one has 4 more if/else if statements involved.
The program compiles and begins to run fine. It produces several results back from the file that is being read into/then out of again in input2 but then stops, produces no more results and gives the error:
Exception in thread "main" java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
at java.io.DataInputStream.readUTF(DataInputStream.java:572)
at java.io.DataInputStream.readUTF(DataInputStream.java:547)
at YDSearch.listFiles(YDSearch.java:85)
at YDSearch.main(YDSearch.java:93)
Anybody know why this might be happening and have a solution they could share?
I've also tried making the variable 'b' to be inside of an if statement but that doesn't work because of scope. If b was defined by if's then there would only need to be one if statement to output to the file
Please let me know if you've got any ideas, I'd really appreciate it :)

As far as I can see, you don't always put out an output record for every name, only for when the name matches one of your patterns. However, you do try to read an input record for every name.
Ergo, if you have any filenames that don't match the patterns you try to read more than you write, and you will get the EOF.
EDIT:
In more detail, the problem is that you get a list of all the files that end with "mp3", "mp4", "3gp", "mov", "avi or "wmv". You then process that list, and write out something into C:/Users/Jonathan/Desktop/ReadyForAnalysis.dat for each "mp3" and "mp4" file. You then assume that for each entry in your list of files, that you will have an entry in ReadyForAnalysis.dat. However, if there are any files ending in "3gp", "mov", "avi or "wmv" then this will not hold true.

Java File I/O Performance Decreases Over Time

I'm trying to perform a once-through read of a large file (~4GB) using Java 5.0 x64 (on Windows XP).
Initially the file read rate is very fast, but gradually the throughput slows down substantially, and my machine seems very unresponsive as time goes on.
I've used ProcessExplorer to monitor the File I/O statistics, and it looks like the process initially reads 500MB/sec, but this rate gradually drops to around 20MB/sec.
Any ideas on the the best way to maintain File I/O rates, especially with reading large files using Java?
Here's some test code that shows the "interval time" continuing to increase. Just pass Main a file that's at least 500MB.
import java.io.File;
import java.io.RandomAccessFile;
public class MultiFileReader {
public static void main(String[] args) throws Exception {
MultiFileReader mfr = new MultiFileReader();
mfr.go(new File(args[0]));
}
public void go(final File file) throws Exception {
RandomAccessFile raf = new RandomAccessFile(file, "r");
long fileLength = raf.length();
System.out.println("fileLen: " + fileLength);
raf.close();
long startTime = System.currentTimeMillis();
doChunk(0, file, 0, fileLength);
System.out.println((System.currentTimeMillis() - startTime) + " ms");
}
public void doChunk(int threadNum, File file, long start, long end) throws Exception {
System.out.println("Starting partition " + start + " to " + end);
RandomAccessFile raf = new RandomAccessFile(file, "r");
raf.seek(start);
long cur = start;
byte buf[] = new byte[1000];
int lastPercentPrinted = 0;
long intervalStartTime = System.currentTimeMillis();
while (true) {
int numRead = raf.read(buf);
if (numRead == -1) {
break;
}
cur += numRead;
if (cur >= end) {
break;
}
int percentDone = (int)(100.0 * (cur - start) / (end - start));
if (percentDone % 5 == 0) {
if (lastPercentPrinted != percentDone) {
lastPercentPrinted = percentDone;
System.out.println("Thread" + threadNum + " Percent done: " + percentDone + " Interval time: " + (System.currentTimeMillis() - intervalStartTime));
intervalStartTime = System.currentTimeMillis();
}
}
}
raf.close();
}
}
Thanks!

I very much doubt that you're really getting 500MB per second from your disk. Chances are the data is cached by the operating system - and that the 20MB per second is what happens when it really hits the disk.
This will quite possibly be visible in the disk section of the Vista Resource Manager - and a low-tech way to tell is to listen to the disk drive :)

Depending on your specific hardware and what else is going on, you might need to work reasonably hard to do much more than 20MB/sec.
I think perhaps you don't really how completely off-the-scale the 500MB/sec is...
What are you hoping for, and have you checked that your specific drive is even theoretically capable of it?

The Java Garbage Collector could be a bottleneck here.
I would make the buffer larger and private to the class so it is reused instead of allocated by each call to doChunk().
public class MultiFileReader {
private byte buf[] = new byte[256*1024];
...
}

You could use JConsole to monitor your app, including memory usage. The 500 MB/sec sounds to good to be true.
Some more information about the implementation and VM arguments used would be helpful.

Check
static void read3() throws IOException {
// read from the file with buffering
// and with direct access to the buffer
MyTimer mt = new MyTimer();
FileInputStream fis =
new FileInputStream(TESTFILE);
cnt3 = 0;
final int BUFSIZE = 1024;
byte buf[] = new byte[BUFSIZE];
int len;
while ((len = fis.read(buf)) != -1) {
for (int i = 0; i < len; i++) {
if (buf[i] == 'A') {
cnt3++;
}
}
}
fis.close();
System.out.println("read3 time = "
+ mt.getElapsed());
}
from http://java.sun.com/developer/JDCTechTips/2002/tt0305.html
The best buffer size might depend on the operating system.
Yours is maybe to0 small.