Develop Reference

How to speed up reading in from a massive file (Java)

So basically, for this assignment I'm working on, we have to read in from a huge file of about a million lines, store the keys and values in a data structure of our choice (I'm using hash tables), offer functionality to change values for keys, and then save the key value stores back into a file. I'm using the cuckoo hashing method along with a method I found from a Harvard paper called "stashing" to accomplish this, and I'm fine with all of it. My only concern is the amount of time it is taking the program just to read in the data from the file.
The file is formatted so that each line has a key (integer) and a value (String) written like this:
12345 'abcdef'
23456 'bcdefg'
and so on. The method I have come up with to read this in is this:
private static void readData() throws IOException {
try {
BufferedReader inStream = new BufferedReader(new FileReader("input/data.db"));
StreamTokenizer st = new StreamTokenizer(inStream);
String line = inStream.readLine();
do{
String[] arr = line.split(" ");
line = inStream.readLine();
Long n = Long.parseLong(arr[0]);
String s = arr[1];
//HashNode<Long, String> node = HashNode.create(n, s);
//table = HashTable.empty();
//table.add(n, s);
}while(line != null);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
The method works fine for actually getting the data, however I tested it with our test file of a million lines and it took about 20 minutes for it to get all the way through reading this all in. Surely, this isn't a fast time for reading in data from a file, and I am positive there must be a better way of doing it.
I have tried several different methods for input (BufferedInputStream with FileInputStream, using Scanner however the file extension is .db so Scanner didn't work, I initially didn't have the tokenizer but added it in hopes it would help). I don't know if the computer I'm running it on makes much of a difference. I have a MacBook Air that I am currently doing the run on; however, I am having a mate run it on his laptop in a bit to see if that might help it along. Any input on how to help this or what I might be doing to slow things SO much would be sincerely and greatly appreciated.
P.S. please don't hate me for programming on a Mac :-)

You can use "java.nio.file.*", the following code is written in Java 8 style but can be easily modified to earlier versions on Java if needed:
Map<Long, String> map = new HashMap<>();
Files.lines(Paths.get("full-path-to-your-file")).forEach(line -> {
String[] arr = line.split(" ");
Long number = Long.parseLong(arr[0]);
String string = arr[1];
map.put(number, string);
});
There is an additional performance gain since Files.lines(..).forEach(...) is executed in parallel. Which means that the lines will not be in-order (and in our case - you don't need it to), in case you needed it to be in order you could call: forEachOrdered().
On my MacBook it took less than 5 seconds to write 2 million such records to a file and then read it and populate the map.

Get rid of the StreamTokenizer. You can read millions of lines per second with BufferedReader.readLine(), and that's all you're really doing: no tokenization.
But I strongly suspect the time isn't being spent in I/O but in processing each line.
NB Your do/while loop is normally written as a while loop:
while ((line = in.readLine()) != null)
Much clearer that way, and no risk of NPEs.

Android: How Reliable is InputStream.read() and its "-1" return?

My question is related to a method InputStream.read() - socket programming.
Every source i have found states that when the server or client closes the connection, "-1" is returned.
But, "what if", just saying, "what if" the connection is closed but read() does not return "-1"? We rely on someone else's code. The reason i am worried is because in order to read the remote end's input, one will have to create an infinite loop and I have always been thought to stay away from infinite loops. With Java however, it looks like I do not have a choice! Here is a sample:
int b = 0;
while (true)
{
b = inputStream.read()
if (b == -1) break; // connection closed
}
or
while (b > -1)
b = inputStream.read()
What if something happens, and -1 never becomes true? One will end up in an infinite loop increasing the temperature of someone's device and wasting CPU cycles! How can one be certain?
References: [http://developer.android.com/reference/java/io/InputStream.html#read%28%29][1] and [http://docs.oracle.com/javase/6/docs/api/java/io/InputStream.html#read%28byte[],%20int,%20int%29][2]
I want to have a fail-safe check. What I have done in the loop is also do a check whether socket has become null and if so, break the loop.
while (b > -1)
{
b = inputStream.read()
if (socket == null) break;
if (outputStream == null) break;
}
What else can I do to ensure the loop exists in case "-1" never becomes true?

One will end up in an infinite loop increasing the temperature of
someone's device and wasting CPU cycles!
This is not true and I think a major part of why you ask this question. read() does not busy wait (i.e ask for the CPU constantly) if it doesn't have anything to read. So in practice there wouldn't be a major problem if -1 isn't returned.
read() also throws an IOException in case something goes wrong with the reading (for example if the InputStream is closed). Making sure you have have the loop inside the try-block together with checking for -1 is more than enough for making sure it'll get out of the loop, as such:
try {
int b;
while ((b=inputStream.read() > -1)
{
//Do something
}
} catch (IOException e) {
e.printStackTrace();
}

what if", just saying, "what if" the connection is closed but read() does not return "-1
There is no such condition. It is a contradiction in terms.
What if something happens, and -1 never becomes true?
It can't.
One will end up in an infinite loop increasing the temperature of someone's device and wasting CPU cycles!
Untrue. If this impossible situation ever happened the read() would block indefinitely without consuming any CPU at all.
How can one be certain?
You could believe the specification, and not ask self-contradictory questions.
NB Testing for the socket or the input stream being null after you've attempted a read and haven't had an exception or an EOS is futile.

Need a better method of reading and storing values from text file

Goal: to get values from a text file and store it into values to load into my sqlite database.
Problem: My method is not efficient, and I need help comming up with an easier way.
As of right now I am parsing my textfile that looks like this.
agency_id,agency_name,agency_url,agency_timezone,agency_lang,agency_phone
1,"NJ TRANSIT BUS","http://www.njtransit.com/",America/New_York,en,""
2,"NJ TRANSIT RAIL","http://www.njtransit.com/",America/New_York,en,""
I am parsing everytime i read a comma, then storing that value into a variable, then I will use that variable as my database value.
This method works and is time consuming, The next text file I have to read in has over 200 lines of code, and i need to find an easier way.
AgencyString = readText();
tv = (TextView) findViewById(R.id.letter);
tv.setText(readText());
StringTokenizer st = new StringTokenizer(AgencyString, ",");
for (int i = 0; i < AgencyArray.length; i++) {
size = i; // which value i am targeting in the textfile
//ex. 1 would be agency_id, 2 would be agency_name
AgencyArray[i] = st.nextToken();
}
tv.setText(AgencyArray[size]); //the value im going to store into database value
}
private String readText() {
InputStream inputStream = getResources().openRawResource(R.raw.agency);
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
int i;
try {
i = inputStream.read();
while (i != -1) {
byteArrayOutputStream.write(i);
i = inputStream.read();
}
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
return byteArrayOutputStream.toString();
}

First, why is this a problem? I don't mean to answer your question with a question so to speak, but more context is required to understand in what way you need to improve the efficiency of what you're doing. Is there a perceived delay in the application due to the parsing of the file, or do you have a more serious ANR problem due to you running on the UI thread? Unless there is some bottleneck in other code not shown, I honestly doubt you'd read and tokenise it faster that you're presently doing. Well, actually, no doubt you probably could; however, I believe it's more a case of designing your application so that delays involved in fetching and parsing of large data aren't perceived by or cause irritation to the user. My own application parses massive files like this and it does take a fraction of a second, but it doesn't present a problem due to the design of the overall application and UI. Also, have you used the profiler to see what's taking time? And also, have you run this on a real device, without debugger attached? Having the debugger attached to the real device, or using the simulator greatly increases execution time by several orders.
I am making the assumption that you need to parse this file type after receiving it over a network, as opposed to being something that is bundled with the application and only needs parsing once.

You could just bundle the SQLite database with your application instead of representing it in a text file. Look at the answer to this question

Faster implementation of more than one input in a single line (Java)

Well, this might be a silly problem.
I just want a faster implementation of following problem
I want to take three integer input in a single line eg:
10 34 54
One way is to make a BufferedReader and then use readLine()
which will read the whole line as a string
then we can use StringTokenizer to separate three integer. (Slow implemetation)
Another way is to use 'Scanner' and take input by nextInt() method. (Slower than previous method)
I want a fast implementation to take such type of inputs since I have to read more than 2,000,000 lines and these implementations are very slow.
My implementation:
BufferedReader br=new BufferedReader(new InputStreamReader(System.in));
for(i=0;i<n;i++) {
str=br.readLine();
st = new StringTokenizer(str);
t1=Integer.parseInt(st.nextElement().toString());
t2=Integer.parseInt(st.nextElement().toString());
z=Long.parseLong(st.nextElement().toString());
}
This one is looped for n times. ( n is number of entries)
Since I know each line will contain only three integer there is no need to check for hasMoreElements()

I just want a faster implementation of following problem.
The chances are that you DON'T NEED a faster implementation. Seriously. Not even with a 2 million line input file.
The chances are that:
more time is spent processing the file than reading it, and
most of the "read time" is spent doing things at the operating system level, or simply waiting for the next disk block to be read.
My advice is to not bother optimizing this unless the application as a whole takes too long to run. And when you find that this is the case, profile the application, and use the profile stats to tell you where it could be worthwhile spending effort on optimization.
(My gut feeling is that there is not much to be gained by optimizing this part of your application. But don't rely on that. Profile it!)

Here's a basic example that will be pretty fast:
public static void main(String[] args) throws IOException {
BufferedReader reader = new BufferedReader(new FileReader("myfile.txt"));
String line;
while ((line = reader.readLine()) != null) {
for (String s : line.split(" ")) {
final int i = Integer.parseInt(s);
// do something with i...
}
}
reader.close();
}
However your task is fundamentally going to take time.
If you are doing this on a website and reaching a timeout, you should consider doing it in a background thread, and send a response to the user saying that the data is being processed. You'll probably need to add a way for the user to check on the progress.

Here is what I mean when I say "specialized scanner". Depending upon parser's (or split's) efficiency, this might be a bit faster (it probably is not):
BufferedReader br=new BufferedReader(...);
for(i=0;i<n;i++)
{
String str=br.readLine();
long[] resultLongs = {-1,-1,-1};
int startPos=0;
int nextLongIndex=0;
for (int p=0;p<str.length();p++)
{
if (str.charAt(p)== ' ')
{
String intAsStr=str.substring(startPos, p-1);
resultLongs[nextLongIndex++]=Integer.parseInt(intAsStr);
startpos=p+1;
}
}
// t1, t2 and z are in resultLongs[0] through resultLongs[2]
}
Hths.
And of course this fails miserably if the input file contains garbage, i.e. anything else but longs separated by blanks.
And in addition, to minimize the "roundtrips" to the OS, it is a good idea to supply the buffered reader with a nonstandard (bigger-than-standard) buffer.
The other hint I gave in the comment refined: If you have to read such a huge text file more than once, i.e. more than once after it has been updated, you could read all longs into a data structure (maybe a List of elements that hold three longs), and stream that into a "cache" file. Next time, compare the text file's timestamp to the "cache" file's. If it is older, read the cache file. Since stream I/O does not serialize longs into its string representation, you will see much, much better reading times.
EDIT: Missed the startPos reassignment.
EDIT2: Added the cache idea explanation.

Which would have the greatest processor cost?

I have a class, as such:
private class verifierListener implements SerialPortEventListener {
String outBuffer;
char charBuffer;
public void serialEvent(SerialPortEvent event) {
if (event.isRXCHAR()) {//If data is available
timeOut = 1000;
lastReadTimer = System.currentTimeMillis();
if (event.getEventValue() > 0) {//Check bytes count in the input buffer
try {
byte[] buffer = verifierPort.readBytes(event.getEventValue());
//INSERT CODE HERE
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
With two possible methods of implementation(in the INSERT CODE HERE area)
Case A:
outBuffer = new String(buffer);
bfrFile.print(outBuffer);
sysOutWriter.append(outBuffer);
Case B:
for(byte bt : buffer) {
charBuffer = (char) bt;
bfrFile.print(charBuffer);
sysOutWriter.append(charBuffer);
}
Both compile and run, and do what they are supposed to do. But I'm trying to make this code perform as seamlessly as possible, so I don't possibly risk a loss of transmitted data on a lower end PC.
I'm assuming that case A will have more overhead due to the String initialization, but I wanted to make certain before I remove it.
Can y'all tell which one would be cleaner, and/or how to determine the processing cost for each?

You shouldn't be loosing any data, even on a lower-end PC. That's because there are (several) buffers in between your code and the actual data that's coming over the serial port(operating system buffers, Java buffers, etc..). Speed-wise, unless you're running this code a lot(as in, several thousand times per second) you shouldn't notice a difference.
Assuming this is a standard serial connection, and you were running at 115200 bits per second, that means that you're getting at most 14,400 characters per second. Even if you were to read those one character at a time, you shouldn't see a big speed hit.

if your using windows ctrl-alt-delete and view the thread when its running and see the memory that it uses. As far as your code goes, may I suggest you use a StringBuilder instead of String. I'm not sure if there really is a difference in overhead, but from a good code / programming perspective it should be A.