I'm having an issue reading from a java input stream. I have a buffer of size 1024, and an input stream of size 29k-31k. I read the inputStream in a loop, but I only get 29 bytes for the first read, 39 for the second read, and nothing after that. The same behavior repeats for different InputStreams. (I'm writing the data to an output stream but I don't see how this can affect the first read)
int bytesRead = 0;
byte[] byteBuf = new byte[1024];
OutputStream fileStream = FileUtil.openFileForWrite(saveTo);
bytesRead = reader.read(byteBuf);
while(bytesRead!=-1){
fileStream.write(byteBuf, 0, bytesRead);
bytesRead = reader.read(byteBuf);
}
What am I missing?
Any help is appreciated :)
Where are you getting the input stream from? How do you know that it's 29K-31K?
Your code looks reasonable to me, although I generally structure the loop slightly different to avoid the duplication of the read call.
Have you tried using readline() instead of read()?
Path file = ...;
InputStream in = null;
try {
in = file.newInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (IOException x) {
System.err.println(x);
} finally {
if (in != null) in.close();
}
Related
I'm trying read a file(doesn't matter the extension) and write after this, but when I do it, the output file is different from the input.
my code is the next:
OutputStream outputStream = null;
FileReader fr = new FileReader("rute\\inputfile.PNG");
BufferedReader br = new BufferedReader(fr);
String line;
while ((line= br.readLine()) != null) {
content += line;
}
byte[] toBytes= content.getBytes();
InputStream inputStream = new ByteArrayInputStream(toBytes);
try {
outputStream = new FileOutputStream(new File("rute\\output.PNG"));
int read = 0;
byte[] bytes = new byte[1024];
while ((read = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
outputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
inputStream.close();
If you ask me why convert into bytes and write from this form, is because I need do something with the data, and I need this conversion.
If you tell me that i cant load an image on a String, yes I can do something like that:
File fil = ~~~~;
FileInputStream fis = null;
fis = new FileInputStream(fil);
byte[] bytess = IOUtils.toByteArray(fis);
But I dont want do it by this way because if I want load big files, the heap size is not enough an this could be solved by the "line per line" read.
Thanks for your answers
I will recommend read this question before. Since you are reading binary data into a String you are changing the encoding of that data. So the output will be different.
Best approach is read binary files as byte arrays. But I will depend which type of transformation/edition/changes you need to do with them.
UPDATE
And, of course, you are editing your content before writing
while ((line= br.readLine()) != null) {
content += line + "\n";
}
so the your output file will be different always.
UPDATE 2
Since the question/problem is how to read a big binary file, google is usually your friend.
Or you can check this other question
I have the following code:
BlobDomain blobDomain = null;
OutputStream out = null;
try {
blobDomain = new BlobDomain();
out = blobDomain.getBinaryOutputStream();
byte[] buffer = new byte[8192];
int bytesRead = 0;
while ((bytesRead = in.read(buffer, 0, 8192)) != -1) {
out.write(buffer, 0, bytesRead);
String line = (new String(buffer));
fullText += line;
}
} catch (Exception e) {
//do nothing
}finally{
if (out != null)
try {
out.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
when i print the fullText what i see for larger files is that end part of the text is added again to the fullText. So full text has some lines repeated in the end. any suggestions on what is wrong here?
The reason that you are getting this is that you are writing the entire buffer every time to your String. Thus, when you reach the end of the file you may not have read exactly the amount of bytes that your buffer is sized at. The old data is still in the buffer and will also be written to your String.
One option to solve this may be to write your data to a String first and then to write your String to the output stream. This should also be faster than adding to a String after each read.
Save inputStream to String:
java.util.Scanner s = new java.util.Scanner(in).useDelimiter("\\A");
fullText = s.hasNext() ? s.next() : "";
Write String to output stream:
out.write(fullText.getBytes());
If you want to keep you code as-is then perform a substring on the buffer and retrieve only the amount of bytes read. For example:
String line = (new String(buffer.substring(0,bytesRead));
I'm looking for a way to switch between reading bytes (as byte[]) and reading lines of Strings from a file. I know that a byte[] can be obtained form a file through a FileInputStream, and a String can be obtained through a BufferedReader, but using both of them at the same time is proving problematic. I know how long the section of bytes are. String encoding can be kept constant from when I write the file. The filetype is a custom one that is still in development, so I can change how I write data to it.
How can I read Strings and byte[]s from the same file in java?
Read as bytes. When you have read a sequence of bytes that you know should be a string, place those bytes in an array, put the array inside a ByteArrayInputStream and use that as the underlying InputStream for a Reader to get the bytes as characters, then read those characters to produce a String.
For the later parts of this process see the related SO question on how to create a String from an InputStream.
Read the file as Strings using a BufferedReader then use String.getBytes().
Why not try this:
BufferedReader bufferedReader = null;
try {
bufferedReader = new BufferedReader(new FileReader("testing.txt"));
String line = bufferedReader.readLine();
while(line != null){
byte[] b = line.getBytes();
}
} finally {
if(bufferedReader!=null){
bufferedReader.close();
}
}
or
FileInputStream in = null;
BufferedReader bufferedReader = null;
try {
bufferedReader = new BufferedReader(new FileReader("xanadu.txt"));
String line = bufferedReader.readLine();
while(line != null){
//read your line
}
in = new FileInputStream("xanadu.txt");
int c;
while ((c = in.read()) != -1) {
//read your bytes (c)
}
} finally {
if (in != null) {
in.close();
}
if(bufferedReader!=null){
bufferedReader.close();
}
}
Read everything as bytes from the buffered input stream, and convert string sections into String's using constructor that accepts the byte array:
String string = new String(bytes, offset, length, "US-ASCII");
Depending on how the data are actually encoded, you may need to use "UTF-8" or something else as the name of the charset.
I was doing some research on IO and I read the following article which talks about buffering techniques. To minimize disk accesses and work by the underlying operating system, buffering techniques use a temporary buffer that reads data in a chunk-wise manner, instead of reading data directly from the disk with every read operation.
Examples were given without and with buffering.
without buffering:
try
{
File f = new File("Test.txt");
FileInputStream fis = new FileInputStream(f);
int b; int ctr = 0;
while((b = fis.read()) != -1)
{
if((char)b== '\t')
{
ctr++;
}
}
fs.close();
// not the ideal way
} catch(Exception e)
{}
With buffering:
try
{
File f = new File("Test.txt");
FileInputStream fis = new FileInputStream(f);
BufferedInputStream bs = new BufferedInputStream(fis);
int b;
int ctr = 0;
while((b =bs.read()) != -1)
{
if((char)b== '\t')
{
ctr++;
}
}
fs.close(); // not the ideal way
}
catch(Exception e){}
The conclusion was:
Test.txt was a 3.5MB file
Scenario 1 executed between 5200 to 5950 milliseconds for 10 test runs
Scenario 2 executed between 40 to 62 milliseconds for 10 test runs.
Is there any other way to do this in Java that is better? Or any other method / technique to give better performance?Please advise..!
Is there any other way to do this in Java that is better? Or any other method / technique to give better performance?
In terms of IO performance, that probably is going to be the best without a lot of other code. You are going to be IO bound most likely anyway.
while((b =bs.read()) != -1)
This is very inefficient to read byte-by-byte. If you are reading a text file then you should be using a BufferedReader instead. This converts a byte array into String.
BufferedReader reader = new BufferedReader(new InputStreamReader(fis));
...
while ((String line = reader.readLine()) != null) {
...
}
Also, with any IO, you should always do it in a try/finally block to make sure you close it:
FileInputStream fis = new FileInputStream(f);
BufferedReader reader;
try {
reader = new BufferedReader(new InputStreamReader(fis));
// once we wrap the fis in a reader, we just close the reader
} finally {
if (reader != null) {
reader.close();
}
if (fis != null) {
fis.close();
}
}
the problem with your code is that you're reading file by bytes (one byte per request). Read it into array chunk by chunk - and performance will be equal to one with Buffer.
you may want to try out NIO and memory-mapped files as well, see http://www.linuxtopia.org/online_books/programming_books/thinking_in_java/TIJ314_029.htm
You can read blocks of data at a time which can still be faster than using a buffered input.
FileInputStream fis = new FileInputStream(new File("Test.txt"));
int len, ctr = 0;
byte[] bytes = new byte[8192];
while ((len = fis.read(bytes)) > 0)
for (int i = 0; i < len; i++)
if (bytes[len] == '\t')
ctr++;
fis.close();
You can also try memory mapping.
FileChannel fc = new FileInputStream(new File("Test.txt")).getChannel();
ByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
int ctr = 0;
for (int i = 0; i < bb.limit(); i++)
if (bb.get(i) == '\t')
ctr++;
fc.close();
I would expect both of these options being about twice as fast.
I'm attempting to output a text file to the console with Java. I was wondering what is the most efficient way of doing so?
I've researched several methods however, it's difficult to discern which is the least performance impacted solution.
Outputting a text file to the console would involve reading in each line in the file, then writing it to the console.
Is it better to use:
Buffered Reader with a FileReader, reading in lines and doing a bunch of system.out.println calls?
BufferedReader in = new BufferedReader(new FileReader("C:\\logs\\"));
while (in.readLine() != null) {
System.out.println(blah blah blah);
}
in.close();
Scanner reading each line in the file and doing system.print calls?
while (scanner.hasNextLine()) {
System.out.println(blah blah blah);
}
Thanks.
If all you want to do is print the contents of a file (and don't want to print the next int/double/etc.) to the console then a BufferedReader is fine.
Your code as it is won't produce the result you're after, though. Try this instead:
BufferedReader in = new BufferedReader(new FileReader("C:\\logs\\log001.txt"));
String line = in.readLine();
while(line != null)
{
System.out.println(line);
line = in.readLine();
}
in.close();
I wouldn't get too hung up about it, though because it's more likely that the main bottleneck will be the ability of your console to print the information that Java is sending it.
If you're not interested in the character based data the text file is containing, just stream it "raw" as bytes.
InputStream input = new BufferedInputStream(new FileInputStream("C:/logs.txt"));
byte[] buffer = new byte[8192];
try {
for (int length = 0; (length = input.read(buffer)) != -1;) {
System.out.write(buffer, 0, length);
}
} finally {
input.close();
}
This saves the cost of unnecessarily massaging between bytes and characters and also scanning and splitting on newlines and appending them once again.
As to the performance, you may find this article interesting. According the article, a FileChannel with a 256K byte array which is read through a wrapped ByteBuffer and written directly from the byte array is the fastest way.
FileInputStream input = new FileInputStream("C:/logs.txt");
FileChannel channel = input.getChannel();
byte[] buffer = new byte[256 * 1024];
ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
try {
for (int length = 0; (length = channel.read(byteBuffer)) != -1;) {
System.out.write(buffer, 0, length);
byteBuffer.clear();
}
} finally {
input.close();
}
If it's a relatively small file, a one-line Java 7+ way to do this is:
System.out.println(new String(Files.readAllBytes(Paths.get("logs.txt"))));
See https://docs.oracle.com/javase/7/docs/api/java/nio/file/package-summary.html for more details.
Cheers!
If all you want is most efficiently dump the file contents to the console with no processing in-between, converting the data into characters and finding line breaks is unnecessary overhead. Instead, you can just read blocks of bytes from the file and write then straight out to System.out:
package toconsole;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
public class Main {
public static void main(String[] args) {
BufferedInputStream bis = null;
byte[] buffer = new byte[8192];
int bytesRead = 0;
try {
bis = new BufferedInputStream(new FileInputStream(args[0]));
while ((bytesRead = bis.read(buffer)) != -1) {
System.out.write(buffer, /* start */ 0, /* length */ bytesRead);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try { bis.close(); } catch (Exception e) { /* meh */ }
}
}
}
In case you haven't come across this kind of idiom before, the statement in the while condition both assigns the result of bis.read to bytesRead and then compares it to -1. So we keep reading bytes into the buffer until we are told that we're at the end of the file. And we use bytesRead in System.out.write to make sure we write only the bytes we've just read, as we can't assume all files are a multiple of 8 kB long!
FileInputStream input = new FileInputStream("D:\\Java\\output.txt");
FileChannel channel = input.getChannel();
byte[] buffer = new byte[256 * 1024];
ByteBuffer byteBuffer = ByteBuffer.wrap(buffer);
try {
for (int length = 0; (length = channel.read(byteBuffer)) != -1;) {
System.out.write(buffer, 0, length);
byteBuffer.clear();
}
} finally {
input.close();
}
Path temp = Files.move
(Paths.get("D:\\\\Java\\\\output.txt"),
Paths.get("E:\\find\\output.txt"));
if(temp != null)
{
System.out.println("File renamed and moved successfully");
}
else
{
System.out.println("Failed to move the file");
}
}
For Java 11 you could use more convenient approach:
Files.copy(Path.of("file.txt"), System.out);
Or for more faster output:
var out = new BufferedOutputStream(System.out);
Files.copy(Path.of("file.txt"), out);
out.flush();