Can I share an InputStream or OutputStream?
For example, let's say I first have:
DataInputStream incoming = new DataInputStream(socket.getInputStream()));
...incoming being an object variable. Later on I temporarily do:
BufferedReader dataReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
I understand that the stream is concrete and reading from it will consume its input, no matter from where it's done... But after doing the above, can I still access both incoming and dataReader simultaneously or is the InputStream just connected to ONE object and therefore incoming loses its input once I declare dataReader? I understand that if I close the dataReader then I will close the socket as well and I will refrain from this but I'm wondering whether I need to "reclaim" the InputStream somehow to incoming after having "transferred" it to dataReader? Do I have to do:
incoming = new DataInputStream(socket.getInputStream());
again after this whole operation?
You are using a teaspoon and a shovel to move dirt from a hole.
I understand that the stream is concrete and reading from it will
consume its input, no matter from where it's done
Correct. The teaspoon and shovel both move dirt from the hole. If you are removing dirt asynchronously (i.e. concurrently) you could get into fights about who has what dirt - so use concurrent construct to provide mutually exclusive access. If access is not concurrent, in other words ...
1) move one or more teaspoons of dirt from the hole
2) move one or more shovels of dirt from the hole
3) move one or more teaspoons of dirt from the hole
...
No problem. Teaspoon and shovel both remove dirt. But once dirt gets removed, it's removed, they do not get the same dirt. Hope this helps. Let's start shovelling, I'll use the teaspoon. :)
As fast-reflexes found, be very careful about sharing streams, particularly buffered readers since they can gobble up a lot more bytes off the stream than they need, so when you go back to your other input stream (or reader) it may look like a whole bunch of bytes have been skipped.
Proof you can read from same input stream:
import java.io.*;
public class w {
public static void main(String[] args) throws Exception {
InputStream input = new FileInputStream("myfile.txt");
DataInputStream b = new DataInputStream(input);
int data, count = 0;
// read first 20 characters with DataInputStream
while ((data = b.read()) != -1 && ++count < 20) {
System.out.print((char) data);
}
// if prematurely interrupted because of count
// then spit out last char grabbed
if (data != -1)
System.out.print((char) data);
// read remainder of file with underlying InputStream
while ((data = input.read()) != -1) {
System.out.print((char) data);
}
b.close();
}
}
Input file:
hello OP
this is
a file
with some basic text
to see how this
works when moving dirt
from a hole with a teaspoon
and a shovel
Output:
hello OP
this is
a file
with some basic text
to see how this
works when moving dirt
from a hole with a teaspoon
and a shovel
Proof to show BufferedReader is NOT gauranteed to work as it gobbles up lots of chars from the stream:
import java.io.*;
public class w {
public static void main(String[] args) throws Exception {
InputStream input = new FileInputStream("myfile.txt");
BufferedReader b = new BufferedReader(new InputStreamReader(input));
// read three lines with BufferedReader
String line;
for (int i = 0; (line = b.readLine()) != null && i < 3; ++i) {
System.out.println(line);
}
// read remainder of file with underlying InputStream
int data;
while ((data = input.read()) != -1) {
System.out.print((char) data);
}
b.close();
}
}
Input file (same as above):
hello OP
this is
a file
with some basic text
to see how this
works when moving dirt
from a hole with a teaspoon
and a shovel
Output:
hello OP
this is
a file
This will be disastrous. Both streams will have corrupted data. How could Java possibly know which data to send to which Stream?
If you need to do two different things with the same data, you're better off storing it somewhere (possibly copying it into two Queue<String>), and then reading it that way.
Ok, I solved this myself.. interesting links:
http://www.coderanch.com/t/276168//java/InputStream-multiple-Readers
Multiple readers for InputStream in Java
Basically... the InputStream can be connected to multiple objects reading from it and consuming it. However, a BufferedReader reads ahead, so when involving one of those, it might be a good idea to implement some sort of signal when you're switching from for example a BufferedReader to a DataInputStream (that is you want to use the DataInputStream to process the InputStream all of a sudden instead of the BufferedReader). Therefore I stop sending data to the InputStream once I know that all data has been sent that is for the BufferedReader to handle. After this, I wait for the other part to process what it should with the BufferedReader. It then sends a signal to show that it's ready for new input. The sending part should be blocking until it receives the signal input and then it can start sending data again. If I don't use the BufferedReader after this point, it won't have a chance to buffer up all the input and "steal" it from the DataInputStream and everything works very well :) But be careful, one read operation from the BufferedReader and you will be back in the same situation... Good to know!
Related
How to read file twice eihher using buffer reader or using stream twice ???
That I need manipulate large amounts of data in the code, so the performance needs to be considered.
Sample code 1 below, gives exception "stream closed" -
Url url = 'www.google.com'
InputStream in = url.openStream();
BufferReader br = new BufferReader(in);
Stream<String> ss = br.lines; // read all the lines
List ll = ss.collect();
br.close();
BufferReader br = new BufferReader(in); //exception occurs
Sample code 2 below, gives exception "stream closed/being used" -
Url url = 'www.google.com'
InputStream in = url.openStream();
BufferReader br = new BufferReader(in);
Supplier<Stream<String>> ss = br.lines; // read all the lines
List ll = ss.collect();
List xx = ss.collect();. // Exception occurs
Please ignore the syntax, it's just a draft code.
Kindly suggest.
Here have an example below. You could use it to read as many times as you wish.
BufferedReader br = new BufferedReader(new FileReader( "users/desktop/xxx.txt" ));
String strLine;
List<String> ans= new ArrayList<String>();
// Read rows
while ((strLine = br.readLine()) != null) {
System.out.println(strLine);
ans.add(strLine);
}
// Read again
for (String result: ans) {
System.out.println(result);
}
reference
https://www.dreamincode.net/forums/topic/272652-reading-from-same-file-twice/
You cannot. A stream is just like its real-life watery counterpart. You can observe the water going under the bridge you're standing on, but you can't instruct the water to go back to the top of the hill so that you can observe it again.
Either have each consumers process each line before moving on to the next line, or if that is not possible then you will need to create your own "buffer" of the entire thing: i.e. store each line to Collection<String>, which the second (and third, and fourth...) consumer can iterate over. The potential problem with this is that it's a bigger memory overhead. The HTML of most websites is not likely to prove to be much of a problem in this regard.
Your last example can be trivially fixed by copying the list.
List ll = ss.collect();
List xx = new ArrayList(ll);
In terms of use a stream is somewhat analogous to an iterator in that it can only be used once.
If you want to use the contents of the same stream again you need to create a new stream as you did the first.
As of Java 12, you can pass values of the same stream into two branches by using the Collectors.teeing() method.
List.stream().collect(Collectors.teeing(
Collector1, // do something with the stream
Collector2, // do something else with the stream
BiFunction, use to merge results)
You can also do this.
Supplier<Stream<String>> ss1 = br.lines; // read all the lines
Supplier<Stream<String>> ss2 = br.lines; // read all the lines
Now you can use ss1 and ss2 as two separate streams.
I used a regular expression to parse a text file to use the resulted group one and two as follows:
write group two in another file
make its name to be group one
Unfortunately, No data is written on the file!
I did not figure out where is the problem, here is my code:
package javaapplication5;
import java.io.*;
import java.util.regex.*;
public class JavaApplication5 {
public static void main(String[] args) {
// TODO code application logic here
try {
FileInputStream fstream = new FileInputStream("C:/Users/Welcome/Desktop/End-End-Delay.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
File newFile1= new File("C:/Users/Welcome/Desktop/AUV1.txt");
FileOutputStream fos1= new FileOutputStream(newFile1);
BufferedWriter bw1= new BufferedWriter(new OutputStreamWriter(fos1));
String strLine;
while ((strLine = br.readLine()) != null) {
Pattern p = Pattern.compile("sender\\sid:\\s(\\d+).*?End-End\\sDelay:(\\d+(?:\\.\\d+)?)");
Matcher m = p.matcher(strLine);
while (m.find()) {
String b = m.group(1);
String c = m.group(2);
int i = Integer.valueOf(b);
if(i==0){
System.out.println(b);
bw1.write(c);
bw1.newLine();
}
System.out.println(b);
// System.out.println(c);
}
}
}
catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
}
}
Can anyone here help me to solve this problem and Identify it?
You are using BufferedWriter, and never flush (flushing writer pushes the contents on disk) your writer or even close it at the end of your program.
Due to which, before your content gets written in actual file on disk from BufferedWriter, the program exits and the contents get lost.
To avoid this, either you can call flush just after writing contents in bw1,
bw1.write(c);
bw1.newLine();
bw1.flush();
OR
Before your program ends, you should call,
bw1.close(); // this ensures all content in buffered writer gets push to disk before jvm exists
Calling flush every time you write the data is not really recommended, as it defeats the purpose of buffered writing.
So best is to close the buffered writer object. You can do it in two ways,
Try-with-resources
Manually close the buffered writer object in the end, likely in the finally block so as to ensure it gets called.
Besides all this, you need to ensure that your regex matches and your condition,
if(i==0){
gets executed else code that is writing data in file won't get executed and of course in that case no write will happen in file.
Also, it is strongly recommended to close any of the resources you open like file resources, database (Connection, Statements, ResultSets) resources etc.
Hope that helps.
I need to do processes on a file ,first count the number of lines and compare with a value.
The next is one to read thru the file line by line and do validations.
if first one passes only i need to do second process.
I read the same file using FTP.
When i try to create a different input stream...ftp is busy reading the current file.
like this :
(is1 = ftp.getFile(feedFileName);)
below is the remaining :
InputStream is = null;
LineNumberReader lin = null;
LineNumberReader lin1 = null;
is = ftp.getFile(feedFileName);
lin = new LineNumberReader(new InputStreamReader(is));
so can i just use like below:
is1=is;
Will both streams be having the file contents from start to finish or the second object will become null as soon as the first stream object is read.
So is the only option left is to create a new ftp object to read a stream seperately ?
It can, but you would need to "rewind" the InputStream. First you need to call mark() method on it, and then reset. Here are docs: http://docs.oracle.com/javase/6/docs/api/java/io/InputStream.html#reset()
After you are done with the LineNumberReader, close the InputStream is. Then re-request the file from FTP, it will not be busy then anymore. You cannot 'just' read from the same InputStream, as that one is probably exhausted by the time the LineNumberReader is done. Furthermore, not all InputStreams support the mark() and reset() methods.
However I'd suggest that doing the second process only when the first one succeeds might not be the right way. As you're streaming the data anyways, why not stream it into a temporary data structure and then count the lines and then operate on the same data structure.
if you file is not big, you can save data to a String.
liek:
StringBuilder sb = new StringBuilder();
byte[] buffer = new byte[1024];
int len;
while((len = is.read(buffer))!=-1)
sb.append(buffer, 0, len);
String data = sb.toString();
then you can do further thing in the String
like:
int lineNumber = data.split("\n").length;
I've never had close experiences with Java IO API before and I'm really frustrated now. I find it hard to believe how strange and complex it is and how hard it could be to do a simple task.
My task: I have 2 positions (starting byte, ending byte), pos1 and pos2. I need to read lines between these two bytes (including the starting one, not including the ending one) and use them as UTF8 String objects.
For example, in most script languages it would be a very simple 1-2-3-liner like that (in Ruby, but it will be essentially the same for Python, Perl, etc):
f = File.open("file.txt").seek(pos1)
while f.pos < pos2 {
s = f.readline
# do something with "s" here
}
It quickly comes hell with Java IO APIs ;) In fact, I see two ways to read lines (ending with \n) from regular local files:
RandomAccessFile has getFilePointer() and seek(long pos), but it's readLine() reads non-UTF8 strings (and even not byte arrays), but very strange strings with broken encoding, and it has no buffering (which probably means that every read*() call would be translated into single undelying OS read() => fairly slow).
BufferedReader has great readLine() method, and it can even do some seeking with skip(long n), but it has no way to determine even number of bytes that has been already read, not mentioning the current position in a file.
I've tried to use something like:
FileInputStream fis = new FileInputStream(fileName);
FileChannel fc = fis.getChannel();
BufferedReader br = new BufferedReader(
new InputStreamReader(
fis,
CHARSET_UTF8
)
);
... and then using fc.position() to get current file reading position and fc.position(newPosition) to set one, but it doesn't seem to work in my case: looks like it returns position of a buffer pre-filling done by BufferedReader, or something like that - these counters seem to be rounded up in 16K increments.
Do I really have to implement it all by myself, i.e. a file readering interface which would:
allow me to get/set position in a file
buffer file reading operations
allow reading UTF8 strings (or at least allow operations like "read everything till the next \n")
Is there a quicker way than implementing it all myself? Am I overseeing something?
import org.apache.commons.io.input.BoundedInputStream
FileInputStream file = new FileInputStream(filename);
file.skip(pos1);
BufferedReader br = new BufferedReader(
new InputStreamReader(new BoundedInputStream(file,pos2-pos1))
);
If you didn't care about pos2, then you woundn't need Apache Commons IO.
I wrote this code to read utf-8 using randomaccessfiles
//File: CyclicBuffer.java
public class CyclicBuffer {
private static final int size = 3;
private FileChannel channel;
private ByteBuffer buffer = ByteBuffer.allocate(size);
public CyclicBuffer(FileChannel channel) {
this.channel = channel;
}
private int read() throws IOException {
return channel.read(buffer);
}
/**
* Returns the byte read
*
* #return byte read -1 - end of file reached
* #throws IOException
*/
public byte get() throws IOException {
if (buffer.hasRemaining()) {
return buffer.get();
} else {
buffer.clear();
int eof = read();
if (eof == -1) {
return (byte) eof;
}
buffer.flip();
return buffer.get();
}
}
}
//File: UTFRandomFileLineReader.java
public class UTFRandomFileLineReader {
private final Charset charset = Charset.forName("utf-8");
private CyclicBuffer buffer;
private ByteBuffer temp = ByteBuffer.allocate(4096);
private boolean eof = false;
public UTFRandomFileLineReader(FileChannel channel) {
this.buffer = new CyclicBuffer(channel);
}
public String readLine() throws IOException {
if (eof) {
return null;
}
byte x = 0;
temp.clear();
while ((byte) -1 != (x = (buffer.get())) && x != '\n') {
if (temp.position() == temp.capacity()) {
temp = addCapacity(temp);
}
temp.put(x);
}
if (x == -1) {
eof = true;
}
temp.flip();
if (temp.hasRemaining()) {
return charset.decode(temp).toString();
} else {
return null;
}
}
private ByteBuffer addCapacity(ByteBuffer temp) {
ByteBuffer t = ByteBuffer.allocate(temp.capacity() + 1024);
temp.flip();
t.put(temp);
return t;
}
public static void main(String[] args) throws IOException {
RandomAccessFile file = new RandomAccessFile("/Users/sachins/utf8.txt",
"r");
UTFRandomFileLineReader reader = new UTFRandomFileLineReader(file
.getChannel());
int i = 1;
while (true) {
String s = reader.readLine();
if (s == null)
break;
System.out.println("\n line " + i++);
s = s + "\n";
for (byte b : s.getBytes(Charset.forName("utf-8"))) {
System.out.printf("%x", b);
}
System.out.printf("\n");
}
}
}
For #Ken Bloom A very quick go at a Java 7 version. Note: I don't think this is the most efficient way, I'm still getting my head around NIO.2, Oracle has started their tutorial here
Also note that this isn't using Java 7's new ARM syntax (which takes care of the Exception handling for file based resources), it wasn't working in the latest openJDK build that I have. But if people want to see the syntax, let me know.
/*
* Paths uses the default file system, note no exception thrown at this stage if
* file is missing
*/
Path file = Paths.get("C:/Projects/timesheet.txt");
ByteBuffer readBuffer = ByteBuffer.allocate(readBufferSize);
FileChannel fc = null;
try
{
/*
* newByteChannel is a SeekableByteChannel - this is the fun new construct that
* supports asynch file based I/O, e.g. If you declared an AsynchronousFileChannel
* you could read and write to that channel simultaneously with multiple threads.
*/
fc = (FileChannel)file.newByteChannel(StandardOpenOption.READ);
fc.position(startPosition);
while (fc.read(readBuffer) != -1)
{
readBuffer.rewind();
System.out.println(Charset.forName(encoding).decode(readBuffer));
readBuffer.flip();
}
}
Start with a RandomAccessFile and use read or readFully to get a byte array between pos1 and pos2. Let's say that we've stored the data read in a variable named rawBytes.
Then create your BufferedReader using
new BufferedReader(new InputStreamReader(new ByteArrayInputStream(rawBytes)))
Then you can call readLine on the BufferedReader.
Caveat: this probably uses more memory than if you could make the BufferedReader seek to the right location itself, because it preloads everything into memory.
I think the confusion is caused by the UTF-8 encoding and the possibility of double byte characters.
UTF8 doesn't specify how many bytes are in a single character. I'm assuming from your post that you are using single byte characters. For example, 412 bytes would mean 411 characters. But if the string were using double byte characters, you would get the 206 character.
The original java.io package didn't deal well with this multi-byte confusion. So, they added more classes to deal specifically with strings. The package mixes two different types of file handlers (and they can be confusing until the nomenclature is sorted out). The stream classes provide for direct data I/O without any conversion. The reader classes convert files to strings with full support for multi-byte characters. That might help clarify part of the problem.
Since you state you are using UTF-8 characters, you want the reader classes. In this case, I suggest FileReader. The skip() method in FileReader allows you to pass by X characters and then start reading text. Alternatively, I prefer the overloaded read() method since it allows you to grab all the text at one time.
If you assume your "bytes" are individual characters, try something like this:
FileReader fr = new FileReader( new File("x.txt") );
char[] buffer = new char[ pos2 - pos ];
fr.read( buffer, pos, buffer.length );
...
I'm late to the party here, but I ran across this problem in my own project.
After much traversal of Javadocs and Stack Overflow, I think I found a simple solution.
After seeking to the appropriate place in your RandomAccessFile, which I am here calling raFile, do the following:
FileDescriptor fd = raFile.getFD();
FileReader fr = new FileReader(fd);
BufferedReader br = new BufferedReader(fr);
Then you should be able to call br.readLine() to your heart's content, which will be much faster than calling raFile.readLine().
The one thing I'm not sure about is whether UTF8 strings are handled correctly.
The java IO API is very flexible. Unfortunately sometimes the flexibility makes it verbose. The main idea here is that there are many streams, writers and readers that implement wrapper patter. For example BufferedInputStream wraps any other InputStream. The same is about output streams.
The difference between streams and readers/writers is that streams work with bytes while readers/writers work with characters.
Fortunately some streams, writers and readers have convenient constructors that simplify coding. If you want to read file you just have to say
InputStream in = new FileInputStream("/usr/home/me/myfile.txt");
if (in.markSupported()) {
in.skip(1024);
in.read();
}
It is not so complicated as you afraid.
Channels is something different. It is a part of so called "new IO" or nio. New IO is not blocked - it is its main advantage. You can search in internet for any "nio java tutorial" and read about it. But it is more complicated than regular IO and is not needed for most applications.
I currently use the following function to do a simple HTTP GET.
public static String download(String url) throws java.io.IOException {
java.io.InputStream s = null;
java.io.InputStreamReader r = null;
//java.io.BufferedReader b = null;
StringBuilder content = new StringBuilder();
try {
s = (java.io.InputStream)new URL(url).getContent();
r = new java.io.InputStreamReader(s);
//b = new java.io.BufferedReader(r);
char[] buffer = new char[4*1024];
int n = 0;
while (n >= 0) {
n = r.read(buffer, 0, buffer.length);
if (n > 0) {
content.append(buffer, 0, n);
}
}
}
finally {
//if (b != null) b.close();
if (r != null) r.close();
if (s != null) s.close();
}
return content.toString();
}
I see no reason to use the BufferedReader since I am just going to download everything in sequence. Am I right in thinking there is no use for the BufferedReader in this case?
In this case, I would do as you are doing (use a byte array for buffering and not one of the stream buffers).
There are exceptions, though. One place you see buffers (output this time) is in the servlet API. Data isn't written to the underlying stream until flush() is called, allowing you to buffer output but then dump the buffer if an error occurs and write an error page instead. You might buffer input if you needed to reset the stream for rereading using mark(int) and reset(). For example, maybe you'd inspect the file header before deciding on which content handler to pass the stream to.
Unrelated, but I think you should rewrite your stream handling. This pattern works best to avoid resource leaks:
InputStream stream = new FileInputStream("in");
try { //no operations between open stream and try block
//work
} finally { //do nothing but close this one stream in the finally
stream.close();
}
If you are opening multiple streams, nest try/finally blocks.
Another thing your code is doing is making the assumption that the returned content is encoded in your VM's default character set (though that might be adequate, depending on the use case).
You are correct, if you use BufferedReader for reading HTTP content and headers you will want InputStreamReader so you can read byte for byte.
BufferedReader in this scenario sometimes does weird things...escpecially when it comes to reading HTTP POST headers, sometimes you will be unable to read the POST data, if you use the InputStreamReader you can read the content length and read that many bytes...
Each invocation of one of an InputStreamReader's read() methods may cause one or more bytes to be read from the underlying byte-input stream. To enable the efficient conversion of bytes to characters, more bytes may be read ahead from the underlying stream than are necessary to satisfy the current read operation.
My gut tells me that since you're already performing buffering by using the byte array, it's redundant to use the BufferedReader.