Develop Reference

How to split a file into several tokens

I was trying to tokenize an input file from sentences into tokens(words).
For example,
"This is a test file." into five words "this" "is" "a" "test" "file", omitting the punctuations and the white spaces. And store them into an arraylist.
I tried to write some codes like this:
public static ArrayList<String> tokenizeFile(File in) throws IOException {
String strLine;
String[] tokens;
//create a new ArrayList to store tokens
ArrayList<String> tokenList = new ArrayList<String>();
if (null == in) {
return tokenList;
} else {
FileInputStream fStream = new FileInputStream(in);
DataInputStream dataIn = new DataInputStream(fStream);
BufferedReader br = new BufferedReader(new InputStreamReader(dataIn));
while (null != (strLine = br.readLine())) {
if (strLine.trim().length() != 0) {
//make sure strings are independent of capitalization and then tokenize them
strLine = strLine.toLowerCase();
//create regular expression pattern to split
//first letter to be alphabetic and the remaining characters to be alphanumeric or '
String pattern = "^[A-Za-z][A-Za-z0-9'-]*$";
tokens = strLine.split(pattern);
int tokenLen = tokens.length;
for (int i = 1; i <= tokenLen; i++) {
tokenList.add(tokens[i - 1]);
}
}
}
br.close();
dataIn.close();
}
return tokenList;
}
This code works fine except I found out that instead of make a whole file into several words(tokens), it made a whole line into a token. "area area" becomes a token, instead of "area" appeared twice. I don't see the error in my codes. I believe maybe it's something wrong with my trim().
Any valuable advices is appreciated. Thank you so much.
Maybe I should use scanner instead?? I'm confused.

I think Scanner is more approprate for this task. As to this code, you should fix regex, try "\\s+";

Try pattern as String pattern = "[^\\w]"; in the same code

How parse char character in java

My file
ABABCCC
My java code:
BufferedReader br = new BufferedReader(new FileReader("My file"));
StringTokenizer sr = new StringTokenizer(br.readLine());
char[] problem = null;
int i = 0;
while(sr.hasMoreTokens())
{
problem[i] = sr.nextToken();
i++;
}
desired output:
problem[0] = 'A'
problem[1] = 'B'
and so on
Please help me in this and provide me a good method for this.

You don't need a StringTokenizer for this data. Just read the data from the file into a String and convert it to a char array.
String line = br.readLine();
char[] problem = line.toCharArray();
You'd only need a loop to read this data if you had multiple lines in your file, or if you had multiple tokens to parse.

StingTokenizer doesn't split up a line by chars it will do it by tokens which is equivalent to words in a sentence. Here is an example usage of that class.
You can store the entire line into a string and then convert that into the char array you have setup.
String fileInput = br.readLine();
char[] problem = fileInput.toCharArray();

You shouldn't be using StringTokenizer, from the Javadoc -
StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.
If I understand what you're trying to do, here's one way to do it with a Scanner and String.toCharArray() -
Scanner sc;
char[] problem = null;
try {
sc = new Scanner(new File("My file"));
if (sc.hasNext()) {
problem = sc.next().toCharArray();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
System.out.println(Arrays.toString(problem));

converting one line string into individual integers

if i have this line in a file: 2 18 4 3
and i want to read it as individual integers, how could i?
i'm using bufferreader:
BufferedReader(new FileReader("mp1.data.txt"));
i have tried to use:
BufferedReader(new RandomAccessFile("mp1.data.txt"));
so i can use the method
.readCahr();
but i got an error
if i use
int w = in.read();
it will read the ASCII, and i want it as it is(in dec.)
i was thinking to read it as a string first, but then could i separate each number?
also i was thinking to let each number in a line, but the file i have is long with numbers

Consider using a Scanner:
Scanner scan = new Scanner(new File("mp1.data.txt"));
You can then use scan.nextInt() (which returns an int, not a String) so long as scan.hasNextInt().
No need for that ugly splitting and parsing :)
However, note that this approach will continue reading integers past the first line (if that's not what you want, you should probably follow the suggestions outlined in the other answers for reading and handling only a single line).
Furthermore, hasNextInt() will return false as soon as a non-integer is encountered in the file. If you require a way to detect and handle invalid data, you should again consider the other answers.

It's important to approach larger problems in software engineering by breaking them into smaller ones. In this case, you've got three tasks:
Read a line from the file
Break it into individual parts (still strings)
Convert each part into an integer
Java makes each of these simple:
Use BufferedReader.readLine() to read the line as a string first
It looks like the splitting is as simple as splitting by a space with String.split():
String[] bits = line.split(" ");
If that's not good enough, you can use a more complicated regular expression in the split call.
Parse each part using Integer.parseInt().
Another option for the splitting part is to use the Splitter class from Guava. Personally I prefer that, but it's a matter of taste.

You can split() the String and then use the Integer.parseInt() method in order to convert all the elements to Integer objects.
try {
BufferedReader br = new BufferedReader(new FileReader("mp1.data.txt"));
String line = null;
while ((line = br.readLine()) != null) {
String[] split = line.split("\\s");
for (String element : split) {
Integer parsedInteger = Integer.parseInt(element);
System.out.println(parsedInteger);
}
}
}
catch (IOException e) {
System.err.println("Error: " + e);
}

Once you read the line using BufferedReader, you can use String.split(regex) method to split the string by space ("\\s").
for(String s : "2 18 4 3".split("\\s")) {
int i = Integer.parseInt(s);
System.out.println(i);
}

If you use Java 7+, you can use this utility method:
List<String> lines = Files.readAllLines(file, Charset.forName("UTF-8"));
for (String line: lines) {
String[] numbers = line.split("\\s+");
int firstNumber = Integer.parseInt(numbers[0]);
//etc
}

Try this;
try{
// Open the file that is the first
FileInputStream fstream = new FileInputStream("textfile.txt");
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
String strLine;
//Read File Line By Line
while ((strLine = br.readLine()) != null) {
//split line by whitespace
String[] ints = strLine.split(" ");
int[] integers = new int[ints.length];
// to convert from string to integers - Integer.parseInt ("123")
for ( int i = 0; i < ints.length; i++) {
integers[i] = Integer.parseInt(ints[i]);
}
// now do what you want with your integer
// ...
}
in.close();
} catch (Exception e) {//Catch exception if any
System.err.println("Error: " + e.getMessage());
}

Print data from file to array

I need to have this file print to an array, not to screen.And yes, I MUST use an array - School Project - I'm very new to java so any help is appreciated. Any ideas? thanks
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.Scanner;
public class HangmanProject
{
public static void main(String[] args) throws FileNotFoundException
{
String scoreKeeper; // to keep track of score
int guessesLeft; // to keep track of guesses remaining
String wordList[]; // array to store words
Scanner keyboard = new Scanner(System.in); // to read user's input
System.out.println("Welcome to Hangman Project!");
// Create a scanner to read the secret words file
Scanner wordScan = null;
try {
wordScan = new Scanner(new BufferedReader(new FileReader("words.txt")));
while (wordScan.hasNext()) {
System.out.println(wordScan.next());
}
} finally {
if (wordScan != null) {
wordScan.close();
}
}
}
}

Nick, you just gave us the final piece of the puzzle. If you know the number of lines you will be reading, you can simply define an array of that length before you read the file
Something like...
String[] wordArray = new String[10];
int index = 0;
String word = null; // word to be read from file...
// Use buffered reader to read each line...
wordArray[index] = word;
index++;
Now that example's not going to mean much to be honest, so I did these two examples
The first one uses the concept suggested by Alex, which allows you to read an unknown number of lines from the file.
The only trip up is if the lines are separated by more the one line feed (ie there is a extra line between words)
public static void readUnknownWords() {
// Reference to the words file
File words = new File("Words.txt");
// Use a StringBuilder to buffer the content as it's read from the file
StringBuilder sb = new StringBuilder(128);
BufferedReader reader = null;
try {
// Create the reader. A File reader would be just as fine in this
// example, but hay ;)
reader = new BufferedReader(new FileReader(words));
// The read buffer to use to read data into
char[] buffer = new char[1024];
int bytesRead = -1;
// Read the file to we get to the end
while ((bytesRead = reader.read(buffer)) != -1) {
// Append the results to the string builder
sb.append(buffer, 0, bytesRead);
}
// Split the string builder into individal words by the line break
String[] wordArray = sb.toString().split("\n");
System.out.println("Read " + wordArray.length + " words");
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (Exception e) {
}
}
}
The second demonstrates how to read the words into an array of known length. This is probably closer to the what you actually want
public static void readKnownWords()
// This is just the same as the previous example, except we
// know in advance the number of lines we will be reading
File words = new File("Words.txt");
BufferedReader reader = null;
try {
// Create the word array of a known quantity
// The quantity value could be defined as a constant
// ie public static final int WORD_COUNT = 10;
String[] wordArray = new String[10];
reader = new BufferedReader(new FileReader(words));
// Instead of reading to a char buffer, we are
// going to take the easy route and read each line
// straight into a String
String text = null;
// The current array index
int index = 0;
// Read the file till we reach the end
// ps- my file had lots more words, so I put a limit
// in the loop to prevent index out of bounds exceptions
while ((text = reader.readLine()) != null && index < 10) {
wordArray[index] = text;
index++;
}
System.out.println("Read " + wordArray.length + " words");
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (Exception e) {
}
}
}
If you find either of these useful, I would appropriate it you would give me a small up-vote and check Alex's answer as correct, as it's his idea that I've adapted.
Now, if you're really paranoid about which line break to use, you can find the values used by the system via the System.getProperties().getProperty("line.separator") value.

Do you need more help with the reading the file, or getting the String to a parsed array? If you can read the file into a String, simply do:
String[] words = readString.split("\n");
That will split the string at each line break, so assuming this is your text file:
Word1
Word2
Word3
words will be: {word1, word2, word3}

If the words you are reading are stored in each line of the file, you can use the hasNextLine() and nextLine() to read the text one line at a time. Using the next() will also work, since you just need to throw one word in the array, but nextLine() is usually always preferred.
As for only using an array, you have two options:
You either declare a large array, the size of whom you are sure will never be less than the total amount of words;
You go through the file twice, the first time you read the amount of elements, then you initialize the array depending on that value and then, go through it a second time while adding the string as you go by.
It is usually recommended to use a dynamic collection such as an ArrayList(). You can then use the toArray() method to turnt he list into an array.

Read multiline text with values separated by whitespaces

I have a following test file :
Jon Smith 1980-01-01
Matt Walker 1990-05-12
What is the best way to parse through each line of this file, creating object with (name, surname, birthdate) ? Of course this is just a sample, the real file has many records.

import java.io.*;
class Record
{
String first;
String last;
String date;
public Record(String first, String last, String date){
this.first = first;
this.last = last;
this.date = date;
}
public static void main(String args[]){
try{
FileInputStream fstream = new FileInputStream("textfile.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
while ((strLine = br.readLine()) != null) {
String[] tokens = strLine.split(" ");
Record record = new Record(tokens[0],tokens[1],tokens[2]);//process record , etc
}
in.close();
} catch (Exception e){
System.err.println("Error: " + e.getMessage());
}
}
}

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class ScannerReadFile {
public static void main(String[] args) {
//
// Create an instance of File for data.txt file.
//
File file = new File("tsetfile.txt");
try {
//
// Create a new Scanner object which will read the data from the
// file passed in. To check if there are more line to read from it
// we check by calling the scanner.hasNextLine() method. We then
// read line one by one till all line is read.
//
Scanner scanner = new Scanner(file);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
System.out.println(line);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
This:
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
Could also be changed to
while (scanner.hasNext()) {
String line = scanner.next();
Which will read whitespace.
You could do
Scanner scanner = new Scanner(file).useDelimiter(",");
To do a custom delimiter
At the time of the post, now you have three different ways to do this. Here you just need to parse the data you need. You could read the the line, then split or read one by one and everything 3 would a new line or a new person.

At first glance, I would suggest the StringTokenizer would be your friend here, but having some experience doing this for real, in business applications, what you probably cannot guarantee is that the Surname is a single name (i.e. someone with a double barrelled surname, not hyphenated would cause you problems.
If you can guarantee the integrity of the data then, you code would be
BufferedReader read = new BufferedReader(new FileReader("yourfile.txt"));
String line = null;
while( (line = read.readLine()) != null) {
StringTokenizer tokens = new StringTokenizer(line);
String firstname = tokens.nextToken();
...etc etc
}
If you cannot guarantee the integrity of your data, then you would need to find the first space, and choose all characters before that as the last name, find the last space and all characters after that as the DOB, and everything inbetween is the surname.

Use a FileReader for reading characters from a file, use a BufferedReader for buffering these characters so you can read them as lines. Then you have a choice.. Personally I'd use String.split() to split on the whitespace giving you a nice String Array, you could also tokenize this string.
Of course you'd have to think about what would happen if someone has a middle name and such.

Look at BufferedReader class. It has readLine method. Then you may want to split each line with space separators to construct get each individual field.