I am trying to index documents using Lucene... However I am getting a StreamClosed exception..
I think it is related more to Java then Lucene....
Can someone please guide..
The code snippet is as follows:
static void indexDocs(File file,boolean flag,Directory dir,IndexWriterConfig iwc)
throws IOException {
// do not try to index files that cannot be read
FileInputStream fis = null;
if (file.canRead()) {
if (file.isDirectory())
{
String[] files = file.list();
System.out.println("list " + files.length);
if (files != null) {
for (int i = 0; i < files.length; i++) {
System.out.println("Invoked for " + i + "and " + files[i]);
indexDocs(new File(file, files[i]),flag,dir,iwc);
}
}
}
else {
boolean flags=true;
try {
fis = new FileInputStream(file);
} catch (FileNotFoundException fnfe) {
fnfe.printStackTrace();
}
try {
Document doc = new Document();
LineNumberReader lnr=new LineNumberReader(new FileReader(file));
Field pathField = new StringField("path", file.getPath(), Field.Store.YES);
doc.add(pathField);
String line=null;
int i=0;
doc.add(new StringField("TT",file.getName(),Field.Store.YES));
BufferedReader br=new BufferedReader(new InputStreamReader(fis));
doc.add(new TextField("DD", br));
System.out.println("Looping Again");
while(flags)
{
IndexWriter iwcTemp1=new IndexWriter(dir,iwc);
while( null != (line = lnr.readLine()) ){
i++;
StringField sf=new StringField("EEE",line.trim(),Field.Store.YES);
doc.add(sf);
if(i%10000==0)
{
System.out.println("Breaking " + i);
lnr.mark(i);
break;
}
sf=null;
}
if(line==null)
{
System.out.println("FALSE " );
flags=false;
}
System.out.println("FILE NAME IS FTP " + file.getName());
if (iwcTemp1.getConfig().getOpenMode() == OpenMode.CREATE_OR_APPEND) {
try
{
iwcTemp1.addDocument(doc);
iwcTemp1.commit();
iwcTemp1.close();
}catch(Throwable t)
{
lnr.close();
br.close();
fis.close();
t.printStackTrace();
}
} else {
try
{
System.out.println("updating " + file);
iwcTemp1.updateDocument(new Term("path", file.getPath()), doc);
}catch(Exception e)
{
e.printStackTrace();
}
}
System.out.println("END OF WHILE");
lnr.reset();
}//end of While
}catch (Exception e) {
e.printStackTrace();
}finally {
fis.close();
}
}
}
}
Exception I am getting is on the line where I am adding Document to Writer...
Exception trace:
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:114)
at java.io.BufferedReader.read(BufferedReader.java:270)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:923)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1133)
at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:171)
at org.apache.lucene.analysis.standard.StandardFilter.incrementToken(StandardFilter.java:49)
at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:102)
at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:245)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:265)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:432)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1513)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1188)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1169)
at com.rancore.MainClass2.indexDocs(MainClass2.java:236)
Can someone please guide...Where am I going wrong...Kindly guide...
Your exception handling is incorrectly structured. It shouldn't be possible to continue with the read code if new FileInputStream() throws an exception.
Related
I have written an downloader which should be used to download text files, as well as images. So I download the files as binaries. Many of the downloads work very well, but some parts of the text files and many image files are corrupted. The errors occur always at the same files and at the same places (as long as I can tell when analysing the text files). I used this code for downloading:
public File downloadFile(HttpURLConnection connection) {
return writeFileDataToFile(getFileData(connection));
}
//downloads the data of the file and returns the content as string
private List<Byte> getFileData(HttpURLConnection connection) {
List<Byte> fileData = new ArrayList<>();
try (InputStream input = connection.getInputStream()) {
byte[] fileChunk = new byte[8*1024];
int bytesRead;
do {
bytesRead = input.read(fileChunk);
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(fileChunk));
fileChunk = new byte[8*1024];
}
} while (bytesRead != -1);
return fileData;
} catch (IOException e) {
System.out.println("Receiving file at " + url.toString() + " failed");
System.exit(1);
return null; //shouldn't be reached
}
}
//writes data to the file
private File writeFileDataToFile(List<Byte> fileData) {
if (!this.file.exists()) {
try {
this.file.getParentFile().mkdirs();
this.file.createNewFile();
} catch (IOException e) {
System.out.println("Error while creating file at " + file.getPath());
System.exit(1);
}
}
try (OutputStream output = new FileOutputStream(file)) {
output.write(Bytes.toArray(fileData));
return file;
} catch (IOException e) {
System.out.println("Error while accessing file at " + file.getPath());
System.exit(1);
return null;
}
}
I could suggest you to not pass through List of Byte, since you create a list of Byte from an array, to get it back to an array of Byte, which is not really efficient.
Moreover you wrongly assume the chunk size (not necesseraly 8192 bytes).
Why don't you just do something as:
private File writeFileDataToFile(HttpURLConnection connection) {
if (!this.file.exists()) {
try {
this.file.getParentFile().mkdirs();
//this.file.createNewFile(); // not needed, will be created at FileOutputStream
} catch (IOException e) {
System.out.println("Error while creating file at " + file.getPath());
//System.exit(1);
// instead do a throw of error or return null
throw new YourException(message);
}
}
OutputStream output = null;
InputStream input = null;
try {
output = new FileOutputStream(file):
input = connection.getInputStream();
byte[] fileChunk = new byte[8*1024];
int bytesRead;
while ((bytesRead = input.read(fileChunk )) != -1) {
output.write(fileChunk , 0, bytesRead);
}
return file;
} catch (IOException e) {
System.out.println("Receiving file at " + url.toString() + " failed");
// System.exit(1); // you should avoid such exit
// instead do a throw of error or return null
throw new YourException(message);
} finally {
if (input != null) {
try {
input.close();
} catch (Execption e2) {} // ignore
}
if (output != null) {
try {
output.close();
} catch (Execption e2) {} // ignore
}
}
}
The failure was adding the whole fileChunk Array to file data, even if it wasn't completely filled by the read operation.
Fix:
//downloads the data of the file and returns the content as string
private List<Byte> getFileData(HttpURLConnection connection) {
List<Byte> fileData = new ArrayList<>();
try (InputStream input = connection.getInputStream()) {
byte[] fileChunk = new byte[8*1024];
int bytesRead;
do {
bytesRead = input.read(fileChunk);
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(Arrays.copyOf(fileChunk, bytesRead)));
}
} while (bytesRead != -1);
return fileData;
} catch (IOException e) {
System.out.println("Receiving file at " + url.toString() + " failed");
System.exit(1);
return null; //shouldn't be reached
}
}
Where the relevant change is changing
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(fileChunk));
fileChunk = new byte[8*1024];
}
into
if (bytesRead != -1) {
fileData.addAll(Bytes.asList(Arrays.copyOf(fileChunk, bytesRead)));
}
So Im working of reading a file containing appointments that I wrote to earlier in my code. I want to sift through the text file and find appointments on a certain date and add them to an ArrayList but when the BufferedReader goes through it, it skips ever other line... Heres my code
public ArrayList<String> read(int checkDay, int checkMonth, int checkYear) {
ArrayList<String> events = new ArrayList<String>();
BufferedReader in = null;
String read;
try {
in = new BufferedReader(new FileReader("calendar.txt"));
while ((read = in.readLine()) != null) {
read = in.readLine();
String[] split = read.split(",");
System.out.println(read);
if (split[1].equals(Integer.toString(checkDay)) && split[2].equals(Integer.toString(checkMonth)) && split[3].equals(Integer.toString(checkYear))) {
events.add(split[0] + " : " + split[1] + "/" + split[2] + "/" + split[3]);
}
}
} catch (IOException e) {
System.out.println("There was a problem: " + e);
e.printStackTrace();
} finally {
try {
in.close();
} catch (Exception e) {
}
}
return events;
}
You are reading the line twice..
while ((read = in.readLine()) != null) { // here
read = in.readLine(); // and here
You have error here:
while ((read = in.readLine()) != null)
read = in.readLine();
you should keep the read = in.readLine() in the while. and remove the other line.
pl try this
you r using "read = in.readLine())" two times in while loop that why it is skiping the lomes
public ArrayList<String> read(int checkDay, int checkMonth, int checkYear) {
ArrayList<String> events = new ArrayList<String>();
BufferedReader in = null;
String read;
try {
in = new BufferedReader(new FileReader("calendar.txt"));
while ((read = in.readLine()) != null) {
String[] split = read.split(",");
System.out.println(read);
if (split[0].equals(Integer.toString(checkDay)) && split[1].equals(Integer.toString(checkMonth)) && split[2].equals(Integer.toString(checkYear))) {
events.add(split[0] + " : " + split[1] + "/" + split[2] + "/" + split[3]);
}
}
} catch (IOException e) {
System.out.println("There was a problem: " + e);
e.printStackTrace();
} finally {
try {
in.close();
} catch (Exception e) {
}
}
return events;
I'm having a problem with reading and writing arraylist to a text file. Specifically with reading. What I'm trying to do is read from a text file and transfer it to an array list. After which i would edit the list and write it back to the text file. I think I go the writing done but not the reading. I've tried reading several similar questions here but cant seem to inject it into my code.
Reading code
public void read(List<AddressBook> addToList){
BufferedReader br = null;
try {
String currentLine= "";
br = new BufferedReader(new FileReader("bank_account.csv"));//file na gusto mo basahin
while ((currentLine = br.readLine()) != null) {
System.out.println(currentLine); // print per line
for (AddressBook read : addToList) {
br.read(read.getName() + read.getAddress() + read.getTelNum() + read.getEmailAdd());
addToList.add(read);
} }
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)
{
br.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
Here's what I've done with the write
public void write(List<AddressBook> addToList) {
try {
File file = new File("bank_account.csv"); //file
// if file doesnt exists, then create it
if (!file.exists()) {
file.createNewFile();
}
//FileWriter fw = new FileWriter(file.getAbsoluteFile());
FileWriter fw = new FileWriter(file.getAbsoluteFile(), true);
BufferedWriter bw = new BufferedWriter(fw);
for (AddressBook write : addToList) {
bw.write(write.getName() + "," + write.getAddress() + "," + write.getTelNum() + "," + write.getEmailAdd());
bw.newLine();
}
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
while ((currentLine = br.readLine()) != null) {
System.out.println(currentLine); // print per line
for (AddressBook read : addToList) {
br.read(read.getName() + read.getAddress() + read.getTelNum() + read.getEmailAdd());
addToList.add(read);
}
}
I bet in there you will need to do something like:
reading each line
parsing it (each line is a CSV)
creating a new AddressBook object with all that info
add it to the collection
The code for that will look like:
while ((currentLine = br.readLine()) != null) {
System.out.println(currentLine); // print per line
String[] splitted = currentLine.split(",");
AddressBook address = new AddressBook(splitted[0], splitted[1], splitted[2], splitted[3]);
addToList.add(address);
}
Of course there are things you will need to check and validate, but that is roughtly it.
Maybe you need read method like this.
public void read() {
List<AddressBook> addToList =new ArrayList<AddressBook>();
BufferedReader br = null;
try {
String currentLine= "";
br = new BufferedReader(new FileReader("bank_account.csv"));//file na gusto mo basahin
while ((currentLine = br.readLine()) != null) {
System.out.println(currentLine); // print per line
// for (AddressBook read : addToList) {
String[] split =currentLine.split(",");
AddressBook read = new AddressBook();
read.setName(split[0]);
read.setAddress(split[1]);
read.setTelNum(split[2]);
read.setEmailAdd(split[3]);
// br.read(read.getName() + read.getAddress() + read.getTelNum() + read.getEmailAdd());
addToList.add(read);
// }
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)
{
br.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
Here's example:
public static void main (String[] args){
String path = "C:\\Users\\Charbel\\Desktop\\Dictionary.txt";
String temppath = "C:\\Users\\Charbel\\Desktop\\temp.txt";
File file = new File(path);
File tempfile = new File(temppath);
int numl = search("x");
int countL = 0;
String line;
try {
BufferedReader bf = new BufferedReader(new FileReader(path));
BufferedWriter bw = new BufferedWriter(new FileWriter(temppath));
while (( line = bf.readLine()) != null)
{
if(countL != numl){
bw.write(line);
bw.newLine();
}
countL++;
}
bf.close();
bw.close();
file.delete();
boolean successful = tempfile.renameTo(file);
System.out.println(successful);
}
catch (IOException e) {
System.out.println("IO Error Occurred: " + e.toString());
}
}
public static int search(String name)
{
String path = "C:\\Users\\Charbel\\Desktop\\Dictionary.txt";
int countL = 0;
String line;
try {
BufferedReader bf = new BufferedReader(new FileReader(path));
while (( line = bf.readLine()) != null)
{
int indexfound = line.indexOf(name);
if (indexfound == 0) {
return countL;
}
countL++;
}
bf.close();
}
catch (IOException e) {
System.out.println("IO Error Occurred: " + e.toString());
}
return -1;
}
}
Hello there .. i am trying to read the line of a specific string in a text file , get the number of its line , then copy all the data in the file to another text file except the line of the string
the code is sometimes working 100% and sometimes no ; I go to my desktop I see both files the temp and the original one without deleting and renaming it
i think i have a problem in deleting the file what do you think coders ?
Because when the search method find name(actually "x"),don't reach the line bf.close(), so bf is still opened and file.delete() fails.
So, you need to modify the search method to the below:
public static int search(String name) {
String path = "C:\\Users\\Charbel\\Desktop\\Dictionary.txt";
int countL = 0;
String line;
BufferedReader bf = null;
try {
bf = new BufferedReader(new FileReader(path));
while (( line = bf.readLine()) != null)
{
int indexfound = line.indexOf(name);
if (indexfound == 0) {
return countL;
}
countL++;
}
}
catch (IOException e) {
System.out.println("IO Error Occurred: " + e.toString());
}
finally {
if(bf != null) {
try {
bf.close();
}
catch(IOException ignored) {}
}
}
return -1;
}
On this code I get an java.util.ConcurrentModificationException the method is in a webservice and first reads the file and checks if the vakNaam is in the file. Then it will be removed and the file will be rewritten. The exception is thrown by Exception2 (in the println)
#WebMethod
public boolean removeVak(String naam){
ArrayList<String> tempFile = new ArrayList<String>();
//Read the lines
boolean found = false;
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader("C:/vak.txt"));
String strLine;
while ((strLine = br.readLine()) != null){
tempFile.add(strLine);
}
}catch(Exception e){
System.out.println("Exception " + e);
}finally {
try {
if (br != null)
br.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
//Write the lines
BufferedWriter out= null;
try{
for(String s : tempFile){
String [] splitted = s.split(" ");
if(splitted[0].equals(naam)){
tempFile.remove(s);
found = true;
}
}
out = new BufferedWriter(new FileWriter("C:/vak.txt", false));
for(String s: tempFile){
out.newLine();
out.write(s);
}
out.close();
} catch (Exception e) {
System.out.println("Exception2 " + e);
return false;
}finally {
try {
if (out != null)
out.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
return found;
}
The error is in this part:
for (String s : tempFile){
String [] splitted = s.split(" ");
if (splitted[0].equals(naam)){
tempFile.remove(s);
found = true;
}
}
Don't modify the list you are iterating over. You could solve this by using the Iterator explicitely:
for (Iterator<String> it = tempFile.iterator(); it.hasNext();) {
String s = it.next();
String [] splitted = s.split(" ");
if (splitted[0].equals(naam)){
it.remove();
found = true;
}
}
The Java 5 enhanced for loop uses an Iterator underneath. So When you remove from tempFile the fail fast nature kicks in and throws the Concurrent exception. Use an iterator and call its remove method, which will remove from the underlying Collection.