Develop Reference

Java - How to read a file twice using buffer reader or use stream twice

How to read file twice eihher using buffer reader or using stream twice ???
That I need manipulate large amounts of data in the code, so the performance needs to be considered.
Sample code 1 below, gives exception "stream closed" -
Url url = 'www.google.com'
InputStream in = url.openStream();
BufferReader br = new BufferReader(in);
Stream<String> ss = br.lines; // read all the lines
List ll = ss.collect();
br.close();
BufferReader br = new BufferReader(in); //exception occurs
Sample code 2 below, gives exception "stream closed/being used" -
Url url = 'www.google.com'
InputStream in = url.openStream();
BufferReader br = new BufferReader(in);
Supplier<Stream<String>> ss = br.lines; // read all the lines
List ll = ss.collect();
List xx = ss.collect();. // Exception occurs
Please ignore the syntax, it's just a draft code.
Kindly suggest.

Here have an example below. You could use it to read as many times as you wish.
BufferedReader br = new BufferedReader(new FileReader( "users/desktop/xxx.txt" ));
String strLine;
List<String> ans= new ArrayList<String>();
// Read rows
while ((strLine = br.readLine()) != null) {
System.out.println(strLine);
ans.add(strLine);
}
// Read again
for (String result: ans) {
System.out.println(result);
}
reference
https://www.dreamincode.net/forums/topic/272652-reading-from-same-file-twice/

You cannot. A stream is just like its real-life watery counterpart. You can observe the water going under the bridge you're standing on, but you can't instruct the water to go back to the top of the hill so that you can observe it again.
Either have each consumers process each line before moving on to the next line, or if that is not possible then you will need to create your own "buffer" of the entire thing: i.e. store each line to Collection<String>, which the second (and third, and fourth...) consumer can iterate over. The potential problem with this is that it's a bigger memory overhead. The HTML of most websites is not likely to prove to be much of a problem in this regard.
Your last example can be trivially fixed by copying the list.
List ll = ss.collect();
List xx = new ArrayList(ll);

In terms of use a stream is somewhat analogous to an iterator in that it can only be used once.
If you want to use the contents of the same stream again you need to create a new stream as you did the first.
As of Java 12, you can pass values of the same stream into two branches by using the Collectors.teeing() method.
List.stream().collect(Collectors.teeing(
Collector1, // do something with the stream
Collector2, // do something else with the stream
BiFunction, use to merge results)
You can also do this.
Supplier<Stream<String>> ss1 = br.lines; // read all the lines
Supplier<Stream<String>> ss2 = br.lines; // read all the lines
Now you can use ss1 and ss2 as two separate streams.

ProcessBuilder BufferedReader read() blocking

I'm using ProcessBuilder to run a command line tool we are using. During its run the tool asks 2 yes/no questions, so usually I answer 'y' twice and then press enter after each time. My problem is that the tool always finishes its run when running from cmd, but when I run it through my java code, it sometimes work and sometimes gets stuck on while ((n = op.read(buffer)) != -1) (with the same input).
Here is my code. Am I doing something wrong? What am I missing? Thanks.
List<String> processArgs = new ArrayList<>();
processArgs.add(0, "java");
processArgs.add(1, "-jar");
processArgs.add(2, JAR_PATH);
processArgs.add(3, "-put");
processArgs.addAll(args);
try
{
// run tool with put
ProcessBuilder pb = new ProcessBuilder(processArgs);
pb.directory(new File("src\\temp"));
pb.redirectErrorStream(true);
Process p = pb.start();
// write 'y' to the tool's stdin.
String answer = "y" + System.getProperty("line.separator");
// yes to first question
p.getOutputStream().write(answer.getBytes());
p.getOutputStream().flush();
// read tool's process stdout
this.op = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringWriter sw = new StringWriter();
int n = 0;
boolean answered = false;
char[] buffer = new char[BUFFER_SIZE];
while ((n = op.read(buffer)) != -1)
{
sw.write(buffer, 0, n);
if (sw.toString().contains("second question") && !answered)
{
// yes to second question
p.getOutputStream().write(answer.getBytes());
p.getOutputStream().flush();
answered = true;
}
}
stdout = sw.toString();
exitCode = p.waitFor();
}
catch (IOException | InterruptedException e)
{
throw new ToolException("process had an exception:\n" + e.getMessage());
}
UPDATE:
I changed my code and added pb.redirectErrorStream(true), but now the process still gets blocked on op.read(buffer). When I debug it seems that it's stuck on the second question, even though I wrote 'y' twice to the output stream. Am I using getOutputStream() incorrectly?
SECOND UPDATE:
The second question didn't get the second 'y' as an answer, and it caused the process to wait for input. I changed the code so i will show the proper way to insert input to the subprocess outputstream.

The output and error stream are buffered. When the buffer fills up the program stops waiting for you to read it. However you only read the output first so if the error stream fills you have a deadlock.
A simple solution is to redirect the error to the output so you have only one stream to read. i.e.
pb.redirectErrorStream(true);
As per the documentation https://docs.oracle.com/javase/8/docs/api/java/lang/ProcessBuilder.html
Alternatives include; writing error to a file, or reading it in another thread.

Reading group of lines in HUGE files

I have no idea how to do the following: I want to process a really huge textfile (almost 5 gigabytes). Since I cannot copy the file into temporarily memory, I thought of reading the first 500 lines (or as many as fit into the memory, I am not sure about that yet), do something with them, then go on to the next 500 until I am done with the whole file.
Could you post an example of the "loop" or command that you need for that? Because all the ways I tried resulted in starting from the beginning again but I want to go on after finishing the previous 500 lines.
Help appreciated.

BufferedReader br = new BufferedReader(new FileReader(file));
String line = null;
ArrayList<String> allLines = new ArrayList<String>();
while((line = br.readLine()) != null) {
allLines.add(line);
if (allLines.size() > 500) {
processLines(allLines);
allLines.clear();
}
}
processLines(allLines);

Ok so you indicated in a comment above that you only want to keep certain lines, writing them to a new file based on certain logic. You can read in one line at a time, decide whether to keep it and if so write it to the new file. This approach will use very little memory since you are only holding one line at a time in memory. Here is one way to do that:
BufferedReader br = new BufferedReader(new FileReader(file));
String lineRead = null;
FileWriter fw = new FileWriter(new File("newfile.txt"), false);
while((lineRead = br.readLine()) != null)
{
if (true) // put your test conditions here
{
fw.write(lineRead);
fw.flush();
}
}
fw.close();
br.close();

Creating a login page in java console application using a txt file

I am creating a java console application and i need to use a notepad (txt) file to store PIN, AccountNo and Balance.
When the user enters the PIN (from BufferedReader) it should get validated from the notepad file and login to thge system. However, the code fragment which I used below only displays Enter PIN, and when i click on Enter, the program exits.
Could you please tell me what is the mistake which I have done here?
BufferedReader getIt = new BufferedReader(new InputStreamReader(System.in));
String userPIN = "";
try {
// Open the file that is the first command line parameter
FileInputStream fstream = new FileInputStream(
"D:\\Studies\\BCAS\\HND\\Semester 1\\Programming Concepts\\Assignment\\AccountInfo.txt");
// Get the object of DataInputStream
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
//Read File Line By Line
System.out.println("Enter PIN");
userPIN = getIt.readLine();
while ((strLine = br.readLine()) != null) {
// Print the content on the console#
if (userPIN.equals(strLine)) {
System.out.println("You have login!");
}
}
//Close the input stream
in.close();
} catch (Exception e) {//Catch exception if any
System.err.println("Error: " + e.getMessage());
}

There are quite a few reasons why the program can simply terminate, or appear to be successful in program execution (and hence terminate successfully without doing any useful work):
The contents of the file are read, but if the end of the stream has been reached, then the code simply does not handle the case gracefully (or even consider handling it). Specifically, the line while ((strLine = br.readLine()) != null) can result in a scenario where the body of the loop is not executed.
The else condition is absent inside the body of the while loop. If none of the entries provided by the user input match the contents of the file, the code does not handle the condition, and the result is ambiguous. For all practical purposes, it appears that login is successful if the provided PIN is not present in the file.
Avoid unnecessary decoration of Readers and Streams. BufferedReader in
= new BufferedReader(new FileReader("foo.in")); is sufficient to read character data from files, and it does provide the read and readline methods to use.
PS: It is recommended that you learn to program defensively, especially when dealing with files and streams. That would help isolating cases in your code that would result in ambiguious/vague conditions.

File.ReadAllLines would be more convenient in your case.. Wouldn't be wrong to introduce a user class either if your going the good ol' oop way.
If you want to do this by yourself, StreamReader (which takes FileInfo) is a good way to go. Try to create it by using the using statement (:)) which makes cleans up after your done.

Java Process with Input/Output Stream

I have the following code example below. Whereby you can enter a command to the bash shell i.e. echo test and have the result echo'd back. However, after the first read. Other output streams don't work?
Why is this or am I doing something wrong? My end goal is to created a Threaded scheduled task that executes a command periodically to /bash so the OutputStream and InputStream would have to work in tandem and not stop working. I have also been experiencing the error java.io.IOException: Broken pipe any ideas?
Thanks.
String line;
Scanner scan = new Scanner(System.in);
Process process = Runtime.getRuntime ().exec ("/bin/bash");
OutputStream stdin = process.getOutputStream ();
InputStream stderr = process.getErrorStream ();
InputStream stdout = process.getInputStream ();
BufferedReader reader = new BufferedReader (new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
String input = scan.nextLine();
input += "\n";
writer.write(input);
writer.flush();
input = scan.nextLine();
input += "\n";
writer.write(input);
writer.flush();
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
input = scan.nextLine();
input += "\n";
writer.write(input);
writer.close();
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}

Firstly, I would recommend replacing the line
Process process = Runtime.getRuntime ().exec ("/bin/bash");
with the lines
ProcessBuilder builder = new ProcessBuilder("/bin/bash");
builder.redirectErrorStream(true);
Process process = builder.start();
ProcessBuilder is new in Java 5 and makes running external processes easier. In my opinion, its most significant improvement over Runtime.getRuntime().exec() is that it allows you to redirect the standard error of the child process into its standard output. This means you only have one InputStream to read from. Before this, you needed to have two separate Threads, one reading from stdout and one reading from stderr, to avoid the standard error buffer filling while the standard output buffer was empty (causing the child process to hang), or vice versa.
Next, the loops (of which you have two)
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
only exit when the reader, which reads from the process's standard output, returns end-of-file. This only happens when the bash process exits. It will not return end-of-file if there happens at present to be no more output from the process. Instead, it will wait for the next line of output from the process and not return until it has this next line.
Since you're sending two lines of input to the process before reaching this loop, the first of these two loops will hang if the process hasn't exited after these two lines of input. It will sit there waiting for another line to be read, but there will never be another line for it to read.
I compiled your source code (I'm on Windows at the moment, so I replaced /bin/bash with cmd.exe, but the principles should be the same), and I found that:
after typing in two lines, the output from the first two commands appears, but then the program hangs,
if I type in, say, echo test, and then exit, the program makes it out of the first loop since the cmd.exe process has exited. The program then asks for another line of input (which gets ignored), skips straight over the second loop since the child process has already exited, and then exits itself.
if I type in exit and then echo test, I get an IOException complaining about a pipe being closed. This is to be expected - the first line of input caused the process to exit, and there's nowhere to send the second line.
I have seen a trick that does something similar to what you seem to want, in a program I used to work on. This program kept around a number of shells, ran commands in them and read the output from these commands. The trick used was to always write out a 'magic' line that marks the end of the shell command's output, and use that to determine when the output from the command sent to the shell had finished.
I took your code and I replaced everything after the line that assigns to writer with the following loop:
while (scan.hasNext()) {
String input = scan.nextLine();
if (input.trim().equals("exit")) {
// Putting 'exit' amongst the echo --EOF--s below doesn't work.
writer.write("exit\n");
} else {
writer.write("((" + input + ") && echo --EOF--) || echo --EOF--\n");
}
writer.flush();
line = reader.readLine();
while (line != null && ! line.trim().equals("--EOF--")) {
System.out.println ("Stdout: " + line);
line = reader.readLine();
}
if (line == null) {
break;
}
}
After doing this, I could reliably run a few commands and have the output from each come back to me individually.
The two echo --EOF-- commands in the line sent to the shell are there to ensure that output from the command is terminated with --EOF-- even in the result of an error from the command.
Of course, this approach has its limitations. These limitations include:
if I enter a command that waits for user input (e.g. another shell), the program appears to hang,
it assumes that each process run by the shell ends its output with a newline,
it gets a bit confused if the command being run by the shell happens to write out a line --EOF--.
bash reports a syntax error and exits if you enter some text with an unmatched ).
These points might not matter to you if whatever it is you're thinking of running as a scheduled task is going to be restricted to a command or a small set of commands which will never behave in such pathological ways.
EDIT: improve exit handling and other minor changes following running this on Linux.

I think you can use thread like demon-thread for reading your input and your output reader will already be in while loop in main thread so you can read and write at same time.You can modify your program like this:
Thread T=new Thread(new Runnable() {
#Override
public void run() {
while(true)
{
String input = scan.nextLine();
input += "\n";
try {
writer.write(input);
writer.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
} );
T.start();
and you can reader will be same as above i.e.
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
make your writer as final otherwise it wont be able to accessible by inner class.

You have writer.close(); in your code. So bash receives EOF on its stdin and exits. Then you get Broken pipe when trying to read from the stdoutof the defunct bash.