Develop Reference

Java - How to read a big file word by word instead of line by line?

I'd like to read the "text8" corpus in Java and reformat some words. The problem is, in this 100MB corpus all words are on one line. So if I try to load it with BufferedReader and readLine, it takes away too much space at once and can't handle it to separate all the words in one list/array.
So my question: Is it possible in Java to read instead of line by line a corpus, to read it word by word? So for example because all words are on one line, to read for example 100 words per iteration?

you can try using Scanner and set the delimiter to whatever suits you:
Scanner input=new Scanner(myFile);
input.useDelimiter(" +"); //delimitor is one or more spaces
while(input.hasNext()){
System.out.println(input.next());
}

I would suggest you to use the "Character stream" with FileReader
Here is the example code from http://www.tutorialspoint.com/java/java_files_io.htm
import java.io.*;
public class CopyFile {
public static void main(String args[]) throws IOException
{
FileReader in = null;
FileWriter out = null;
try {
in = new FileReader("input.txt");
out = new FileWriter("output.txt");
int c;
while ((c = in.read()) != -1) {
out.write(c);
}
}finally {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
}
}
It reads 16 bit Unicode characters. This way it doesnt matter if your text is in one whole line.
Since you're trying to search word by word, you can easy read till you stumble upon a space and there's your word.

Use the next method of java.util.Scanner
The next method finds and returns the next complete token from this scanner. A
complete token is preceded and followed by input that matches the
delimiter pattern. This method may block while waiting for input to
scan, even if a previous invocation of Scanner.hasNext returned true.
Example:
public static void main(String[] args) {
Scanner sc = new Scanner (System.in);
String a = sc.next();
String b = sc.next();
System.out.println("First Word: "+a);
System.out.println("Second Word: "+b);
sc.close();
}
Input :
Hello Stackoverflow
Output :
First Word: Hello
Second Word: Stackoverflow
In your case use Scanner for reading the file and then use scannerobject.next() method for reading each token(word)

try(FileInputStream fis = new FileInputStream("Example.docx")) {
ZipSecureFile.setMinInflateRatio(0.009);
XWPFDocument file = new XWPFDocument(OPCPackage.open(fis));
ext = new XWPFWordExtractor(file);
Scanner scanner = new Scanner(ext.getText());
while(scanner.hasNextLine()) {
String[] value = scanner.nextLine().split(" ");
for(String v:value) {
System.out.println(v);
}
}
}catch(Exception e) {
System.out.println(e);
}

Searching through a text file java

So I am trying to search through a text file and if the user input is found, it returns the entire sentence including white spaces.But apparently I only get the first string and nothing pass the first string in the sentence. For example if i have a text file called "data.txt" and the contents in the first line is " I am a legend". after user enters "I am a legend" the output after the file is searched is "I". Any help would be appreciated.
public static void Findstr() { // This function searches the text for the string
File file = new File("data.txt");
Scanner kb = new Scanner(System.in);
System.out.println(" enter the content you looking for");
String name = kb.next();
Scanner scanner;
try {
scanner = new Scanner(file).useDelimiter( ",");
while (scanner.hasNext()) {
final String lineFromFile = scanner.nextLine();
if (lineFromFile.contains(name)) {
// a match!
System.out.println("I found " + name);
break;
}
}
} catch (IOException e) {
System.out.println(" cannot write to file " + file.toString());
}

package com.example;
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class FileSearch {
public void parseFile(String fileName,String searchStr) throws FileNotFoundException{
Scanner scan = new Scanner(new File(fileName));
while(scan.hasNext()){
String line = scan.nextLine().toLowerCase().toString();
if(line.contains(searchStr)){
System.out.println(line);
}
}
}
public static void main(String[] args) throws FileNotFoundException{
FileSearch fileSearch = new FileSearch();
fileSearch.parseFile("src/main/resources/test.txt", "am");
}
}
test.txt contains:
I am a legend
Hello World
I am Ironman
Output:
i am a legend
i am ironman
The above code does case insensitive search. You should use nextLine() to get the complete line. next() breaks on whitespaces.
Reference:
http://docs.oracle.com/javase/7/docs/api/java/util/Scanner.html#next()

Scanner.next(); returns the next caracter instead use Scanner.readLine();
Edit:
Belive Scanners use .nextLine(); not .readLine();

When you are scanning your input..
Scanner kb = new Scanner(System.in);
System.out.println(" enter the content you looking for");
String name = kb.next();
You are accepting only one token. You should accept whole line to be searched as your token using kb.nextLine()

String tokenizer and Scanner java

I have a large text file which I want to separate into different strings using the delimiter !- (Each string is multiple lines).
I then want to discard all the other strings that do not contain:
=========== ALL OBJECTS IN CLASS: FENESTRATIONSURFACE:DETAILED ===========
So far I've got this and its not outputting anything (it complies but no output to console).I'm new to programming and I'm not making much progress after researching this for sometime so any suggestions or pointers would be most appreciated thanks!
public class Main {
public static void main(String[] args) throws FileNotFoundException {
File file = new File("C:/Users/Anton/Pictures/1 x geotransform0.5m shading.txt");
Scanner scan = new Scanner(file);
while(scan.hasNextLine()){
StringTokenizer st = new StringTokenizer(scan.nextLine(),"!-");
if(st.equals(" =========== ALL OBJECTS IN CLASS: FENESTRATIONSURFACE:DETAILED ===========")) {
System.out.print(st);
}
}
scan.close();
}
}

You should consider reading the java doc page to StringTokenizer: http://docs.oracle.com/javase/7/docs/api/java/util/StringTokenizer.html
There two different problems in your code:
Because your desired strings are more than one line out of a file, you first have to add them up (into a String) and then work with it via StringTokenizer to separate again.
You have to compare the StringTokenizer.nextToken() to your check String not the whole StringTokenizer. StringTokenizer.nextToken() then gives you the next String separated through !-.
The following code should work:
public class Main {
public static void main(String[] args) throws FileNotFoundException {
File file = new File("C:/Users/Anton/Pictures/1 x geotransform0.5m shading.txt");
Scanner scan = new Scanner(file);
//Scanning
//first scan the whole file into a string (because a sting can have more than one line)
String temp = "";
while(scan.hasNextLine()){
temp = scan.nextLine();
}
//now add the string to tokeinzer
StringTokenizer st = new StringTokenizer(temp,"!-");
//now give output
while(st.hasMoreTokens()){
String temp2 = st.nextToken();
if(temp2.equals(" =========== ALL OBJECTS IN CLASS: FENESTRATIONSURFACE:DETAILED ===========")) {
System.out.print(temp2);
}
}
scan.close();
}
}
}

remove the if condition and add the below while condition
while (st.hasMoreElements()) {
System.out.println(st.nextElement());
}
because stringtokenizer splits the text based on token. so your if never becomes true.

try this...........
while(scan.hasNextLine()){
StringTokenizer st = new StringTokenizer(scan.nextLine(),"!-");
while(st.hasMoreTokens()){
String stt = st.nextElement().toString();
if(stt.equals("=========== ALL OBJECTS IN CLASS: FENESTRATIONSURFACE:DETAILED ===========")) {
System.out.print(stt);
}
}
}
you comparing string with stringtokenizer object not value....

Regarding scanner class

I am currently using StringTokennizer class to split a String into different token as by defined delimiter..
public class App {
public static void main(String[] args) {
String str = "This is String , split by StringTokenizer, created by Neera";
StringTokenizer st = new StringTokenizer(str);
System.out.println("---- Split by comma ',' ------");
StringTokenizer st2 = new StringTokenizer(str, ",");
while (st2.hasMoreElements()) {
System.out.println(st2.nextElement());
}
}
}
My query is that can same thing can also be achieved through scanner class also ...!! Is it the right approach to use the scanner class since I was reading The Scanner class allows you to tokenize data from within a loop, which allows you to stop whenever you want to... I have tried the following thing, but it doesn't work ...please advise me ..!!!
public class App1 {
public static void main(String[] args)
{
Scanner scanner = new Scanner("This is String , split by StringTokenizer, created by Neera").useDelimiter(", ");
while (scanner.hasNextLine()) {
System.out.println(scanner.nextLine());
}
}
}

Use scanner.next() to return the next token instead of scanner.nextLine(), which returns the next full line (and your input only has one). And of course you'll want to use hasNext() in your while instead of hasNextLine().
In response to your comment:
Your code has a syntax mistake, which I originally took for a typo and corrected in the question. You're writing:
while (scanner.hasNext()) ;
System.out.println(scanner.next());
Which properly formatted should tell you what's really happening:
while (scanner.hasNext())
; // empty statement
System.out.println(scanner.next());
It should be:
while (scanner.hasNext()) {
System.out.println(scanner.next());
}

Read one line from input file per iteration

Task: read a line from an input file. If the first word of the line is PRINT, then print the contents of the rest of the line.
Code:
else if(Data.compareTo("PRINT") == 0){
while(inFile.hasNext()){
Data = inFile.next();
System.out.print( Data + " ");
}
}
Question: How to code the scanner so that the scanner only reads one line of information at a time?

public static void ReadAndProcessPrint(File fileToRead) throws FileNotFoundException {
java.util.Scanner scanner = new Scanner(fileToRead);
while(scanner.hasNextLine()){
String line = scanner.nextLine();
if(line.startsWith("PRINT")){
String restOfLine = line.substring(5);
System.out.println(restOfLine);
}else{
//do other things
}
}
}

Hint: http://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html
Create a InputStreamReader and using it create a BufferedReader, use readLine method.