i am trying to read a huge file ( > 1GB) , i am thinking that reading it as a random access file with a buffered reader would be efficient.
i need to read the file line by line and parse it
However being new to JAVA IO Api , i'm not sure how can i do this..
i appreciate your help.
You can use Java's BufferedReader for this:
BufferedReader reader = new BufferedReader(new FileReader(fileName));
String line;
while ((line = reader.readLine()) != null) {
// Do some stuff with the line
}
fileName is the path to the file you want to read.
Do you need to read all of it and from the beginning? You can use a RandomAccessFile to jump to different parts of the file if you know what byte you can start at. I think it is the seek function that does this.
While it is perfectly doable in java, I wanted to suggest based on my experience:
If you're on Unix platform, you may use external shell script for searching through the GBs of log. sed is very optimum for this purpose. Specific usage here: http://www.grymoire.com/Unix/Sed.html
Call shell script through java file whenever you need to read/grep through the log file.
How?
1) In your java code, use ProcessBuilder class. It can take shell script as arg to constructor
ProcessBuilder obj = new ProcessBuilder("FastLogRead.sh");
2) Create object for Process
Process process = obj.start();
3) You can read the output of this shell, directly in your BufferedRead through this
BufferedReader br=new BufferedReader(new InputStreamReader(process.getInputStream()));
Pros:
Speeds up execution by avg. 10 times (I searched through around 4GB log file)
Cons:
Some developers don't like bringing in light-weight shell script in realms of java, hence want to go for java's RandomAccessFile. This is justified.
For your case, you may choose between standardization and performance.
Related
Hi so im using a api to run a search for books.
http://openlibrary.org/search.json?q=prolog
i then run a buffered search (below) to read each line in and select the lines that i want using a IF statement
"title_suggest": "Prolog",
URL search = new URL("http://openlibrary.org/search.json?q="+searchingTerm);//search string
BufferedReader in = new BufferedReader(
new InputStreamReader(search.openStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) //read file
for instance: if inputLine contains title_suggest add inputLine to an arrayList.
however this is quite slow and i was wondering if there is a more effecient way to read in the data?
I cannot imagine that the parsing is a giant time suck compared to retrieving the data over the internet. But, whatever you're doing, you're way better off using a genuine JSON parser rather than rolling up your own parser, especially if you're relying on if statements to do so.
Also, make damned sure your query sent to the API is as restrictive as you can make it, after all the more exact data they can give you the better off it is for all parties involved.
this is quite slow
BufferedReader isn't 'quite slow'. It is extremely fast. You can read millions of lines per second with BufferedReader.readLine().
The slow part is your code that processes each line. Or possibly the server is slow executiong the query or delivering the data.
You're barking up the wrong tree here.
Is it possible to download just the first 50 lines of a .txt file in java?
If possible, I'd need a solution without external libraries, compatible with Java 5 and as simple as possible (involving lines of text rather than streams... one can dream!)
Certainly it's possible, just read the first 50 lines and then stop reading.
You can't do it without streams, since that's what will happen underneath anyways, but a regular new BufferedReader(new InputStreamReader(inputStream, "UTF-8"))); (select the proper encoding) will work just fine.
I have a Java GUI and a C program. C program calculates given value(args). I want to call C calculator in Java, and change label to returned value in GUI (label changes almost every second).
I thought some ways,
Socket programming (probably not efficient in same computer)
File operations (Java writes input, c calculates then java reads calculated value)
JNICALL stuff, but i dont know how to get value from c file.
SharedMemory (i have no idea about windows shm)
What is the best way to solve this problem?
You could take a look at JNA, if your C program is a DLL with export functions.
Remember from a past project it works very nicely, without having to do the JNI bits yourself.
I had to do something similar at my job, I wrote a helper program in C++ to access some custom made database and my main application was in Java. The database API was only available to C++ programs.
What I did was to start the C++ process from Java and communicate via command line args and stdout.
List<String> cmdPathAndArgs = Arrays.asList("/path/to/binary", "--arg1", "--arg2");
ProcessBuilder pb = new ProcessBuilder(cmdPathAndArgs);
Processpp = pb.start();
InputStream stream = pp.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(stream));
InputStream errorStream = pp.getErrorStream();
BufferedReader errorBr = new BufferedReader(new InputStreamReader(errorStream));
If your progran generates output in both stdout and stderr you may get into a deadlock. You should empty both streams regularly.
If you can build your C code into a shared library, then JNI is probably your best bet. Sockets will be more than fast enough to handle 1 update a second. Using files will probably be slower than sockets. Shared memory is fine too, however I think sockets is probably a simpler interface for what you want to do.
so I'm assuming I'm not the only one who's asked this but I have a specific matter I want resolved. Before I start let me just say this is NOT a homework assignment, this is for my convenience and strictly personal use. It isn't so much the coding I'm having an issue with, it's mainly I don't know how to get file access.
I'm developing a program that runs off the command prompt and it is storing all my passwords in the program itself. What I want to know is how can i have the bat file that is running the java file, access another file (like passwords.whatever extension it can read) and store the information there so i don't have to store it in the program. I want to store the data in an array but the primary issue I'm having is how do I get java to access a file, write to it, then able to read it whenever i want to access certain pieces of it.
If something didn't make sense I'll glady elaborate, I just want to make a program for my self being because I have too many passwords to remember in my head.
Learning the entire Java API just to reinvent the functionality available in many high quality , freely available programs seems like overkill.
for writing a File in Java
try{
FileWriter fw=new FileWriter("C://myfile.txt",true);
BufferedWriter bw = new BufferedWriter(fw);
bw.write(mystringanddata);
bw.close();
}
catch(Exception e){
System.out.print("ERROR"+e);
}
for reading a File in Java
try{
FileReader fr=new FileReader("C://myfile.txt",true);
BufferedReader br = new BufferedReader(fr);
String mystring;
while((mystring=br.readLine())!=null){
System.out.print("DATA IS"+mystring);
}
br.close();
}
catch(Exception e){
System.out.print("ERROR"+e);
}
I want to execute this command:
/ceplinux_work3/myName/opt/myCompany/ourProduct/bin/EXECUTE_THIS -p cepamd64linux.myCompany.com:19021/ws1/project_name < /ceplinux_work3/myName/stressting/Publisher/uploadable/00000.bin >> /ceplinux_work3/myName/stressting/Publisher/stats/ws1.project_name.19021/2011-07-22T12-45-20_PID-2237/out.up
But it doesn't work because EXECUTE_THIS requires an input file via redirect, and simply passing this command to Runtime.exec doesn't work.
Side note: I searched all over on how to solve this before coming here to ask. There are many questions/articles on the web regarding Runtime.exec and Input/Output redirect. However, I cannot find any that deal with passing a file to a command and outputting the result to another file. Plus, I am totally unfamiliar with Input/Output streams, so I have a hard time putting all the info out there together for my specific situation.
That said, any help is much appreciated.
P.S. If there are multiple ways to do this, I prefer whatever is fastest in terms of throughput.
Edit: As discussed in my last question, I CANNOT change this to a bash call because the program must wait for this process to finish before proceeding.
Unless you are sending a file name to the standard input of the process, there is no distinction of whether the data came from a file or from any other data source.
You need to write to the OutputStream given by Process.getOutputStream(). The data you write to it you can read in from a file using a FileInputStream.
Putting that together might look something like this:
Process proc = Runtime.getRuntime().exec("...");
OutputStream standardInputOfChildProcess = proc.getOutputStream();
InputStream dataFromFile = new FileInputStream("theFileWithTheData.dat");
byte[] buff = new byte[1024];
for ( int count = -1; (count = dataFromFile.read(buff)) != -1; ) {
standardInputOfChildProcess.write(buff, 0, count);
}
I've left out a lot of details, this is just to get the gist of it. You'll want to safely close things, might want to consider buffering and you need to worry about the pitfalls of Runtime.exec().
Edit
Writing the output to a file is similar. Obtain a FileOutputStream pointing to the output file and write the data you read from Process.getInputStream() to that OutputStream. The major caveat here is that you must do this operation in a second thread, since accessing two blocking streams from the same thread will lead to deadlock (see the article above).