We're streaming a CSV file from a web service. It appears that we're losing the new line characters when streaming - the client gets the file all on a single line. Any idea what we're doing wrong?
Code:
public static void writeFile(OutputStream out, File file) throws IOException {
BufferedReader input = new BufferedReader(new FileReader(file)); //File input stream
String line;
while ((line = input.readLine()) != null) { //Read file
out.write(line.getBytes()); //Write to output stream
out.flush();
}
input.close();
}
Don't use BufferedReader. You already have an OutputStream at hands, so just get an InputStream of the file and pipe the bytes from input to output it the usual Java IO way. This way you also don't need to worry about newlines being eaten by BufferedReader:
public static void writeFile(OutputStream output, File file) throws IOException {
InputStream input = null;
byte[] buffer = new byte[10240]; // 10KB.
try {
input = new FileInputStream(file);
for (int length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
} finally {
if (input != null) try { input.close(); } catch (IOException logOrIgnore) {}
}
}
Using a Reader/Writer would involve character encoding problems if you don't know/specify the encoding beforehand. You actually also don't need to know about them here. So just leave it aside.
To improve performance a bit more, you can always wrap the InputStream and OutputStream in an BufferedInputStream and BufferedOutputStream respectively.
The readline method uses the newline chars to delimit what gets read, so the newlines themselves are not returned by readLine.
Don't use readline, you can use a BufferedInputStream and read the file one byte at a time if you want, or pass your own buffer into OutputStream.write.
Note that, like BalusC and Michael Borgwardt say, Readers and Writers are for text, if you just want to copy the file you should use InputStream and OutputStream, you are only concerned with bytes.
There are several things wrong with that code. It may also mutilate any NON-ASCII text since it converts via the platform default encoding twice - and for no good reason at all.
Don't use a Reader to read the file, use a FileInputStream and transfer bytes, avoiding the unnecessary and potentially destructive charset conversions. The line break problem will also be gone.
Any idea what we're doing wrong?
Yes. This line drops the "new line character"
while ((line = input.readLine()) != null) {
And then you write it without it:
out.write(line.getBytes());
This this related question.
BufferedReader.ReadLine() does not preserve the newline. Thus you'll have to add it when writing it out
You can use a PrintWriter which offers a prinln() method. This will also save you from converting the string into an array of chars.
public static void writeFile(OutputStream o, File file) throws IOException {
PrintWriter out = new PrintWriter(new OutputStreamWriter(o));
BufferedReader input = new BufferedReader(new FileReader(file)); //File input stream
String line;
while ((line = input.readLine()) != null) { //Read file
out.println(line); //Write to output stream
out.flush();
}
input.close();
}
Related
I use this code snippet to read text from a webpage aand save it to a string?
I would like the readline() function to start from the beggining. So it would read content of the webpage again. How Can I do that
if (response == httpURLConnection.HTTP_OK) {
in = httpURLConnection.getInputStream();
isr = new InputStreamReader(in);
br = new BufferedReader(isr);
while ((line = br.readLine()) != null) {
fullText += line;
}
// I want to go through a webpage source again, but
// I can't because br.readLine() = null. How can I put
// put a marker on the beginning of the page?
while ((line1 = br.readLine()) != null) {
fullText1 += line1;
// It will not go into this loop
}
You can only mark a position for a Reader (and return to it with reset()) if markSupported returns true, and I very much doubt that the stream returned by httpURLConnection.getInputStream() supports marks.
The best option, I think, is to read the response into a buffer and then you can create as many readers as you like over that buffer. You will need to include the line termination characters (which you are currently discarding) to preserve the line structure. (Alternatively, you can read the response into a List<String> rather than into a single String.)
From InputStream will not reset to beginning
your stream inside a BufferedInputStream object like:
with the markSupported() method if your InputStream actually support using mark. According to the API the InputStream class doesn't, but the java.io.BufferedInputStream class does. Maybe you should embed your stream inside a BufferedInputStream object like:
InputStream data = new BufferedInputStream(realResponse.getEntity().getContent());
// data.markSupported() should return "true" now
data.mark(some_size);
// work with "data" now
...
data.reset();
I found this code from another question
private void updateLine(String toUpdate, String updated) throws IOException {
BufferedReader file = new BufferedReader(new FileReader(data));
String line;
String input = "";
while ((line = file.readLine()) != null)
input += line + "\n";
input = input.replace(toUpdate, updated);
FileOutputStream os = new FileOutputStream(data);
os.write(input.getBytes());
file.close();
os.close();
}
This is my file before I replace some lines
example1
example2
example3
But when I replace a line, the file now looks like this
example1example2example3
Which makes it impossible to read the file when there are a lot of lines in it.
How would I go about editing the code above to make my file look what it looked like at the start?
Use System.lineSeparator() instead of \n.
while ((line = file.readLine()) != null)
input += line + System.lineSeparator();
The issue is that on Unix systems, the line separator is \n while on Windows systems, it's \r\n.
In Java versions older then Java 7, you would have to use System.getProperty("line.separator") instead.
As pointed out in the comments, if you have concerns about memory usage, it would be wise to not store the entire output in a variable, but write it out line-by-line in the loop that you're using to process the input.
If you read and modify line by line this has the advantage, that you dont need to fit the whole file in memory. Not sure if this is possible in your case, but it is generally a good thing to aim for streaming. In your case this would in addition remove the need for concatenate the string and you don't need to select a line terminator, because you can write each single transformed line with println(). It requires to write to a different file, which is generally a good thing as it is crash safe. You would lose data if you rewrite a file and get aborted.
private void updateLine(String toUpdate, String updated) throws IOException {
BufferedReader file = new BufferedReader(new FileReader(data));
PrintWriter writer = new PrintWriter(new File(data+".out"), "UTF-8");
String line;
while ((line = file.readLine()) != null)
{
line = line.replace(toUpdate, updated);
writer.println(line);
}
file.close();
if (writer.checkError())
throw new IOException("cannot write");
writer.close();
}
In this case, it assumes that you need to do the replace only on complete lines, not multiple lines. I also added an explicit encoding and use a writer, as you have a string to output.
This is because you use OutputStream which is better for handling binary data. Try using PrintWriter and don't add any line terminator at the end of the lines. Example is here
I am quite new to java, just started yesterday. Since I am a big fan of learning by doing, I am making a small project with it. But I am stucked in this part. I have written a file using this function:
public static boolean writeZippedFile(File destFile, byte[] input) {
try {
// create file if doesn't exist part was here
try (OutputStream out = new DeflaterOutputStream(new FileOutputStream(destFile))) {
out.write(input);
}
return true;
} catch (IOException e) {
// error handlind was here
}
}
Now that I have successully wrote a compressed file using above method, I want to read it back to console. First I need to be able to read the decompressed content and write string representaion of that content to console. However, I have a second problem that I don't want to write characters up to first \0 null character. Here is how I attempt to read the compressed file:
try (InputStream is = new InflaterInputStream(new FileInputStream(destFile))) {
}
and I am completely stuck here. Question is, how to discard first few character until '\0' and then write the rest of the decompressed file to console.
I understand that your data contain text since you want to print a string respresentation. I further assume that the text contains unicode characters. If this is true, then your console should also support unicode for the characters to be displayed correctly.
So you should first read the data byte by byte until you encounter the \0 character and then you can use a BufferedReader to print the rest of the data as lines of text.
try (InputStream is = new InflaterInputStream(new FileInputStream(destFile))) {
// read the stream a single byte each time until we encounter '\0'
int aByte = 0;
while ((aByte = is.read()) != -1) {
if (aByte == '\0') {
break;
}
}
// from now on we want to print the data
BufferedReader b = new BufferedReader(new InputStreamReader(is, "UTF8"));
String line = null;
while ((line = b.readLine()) != null) {
System.out.println(line);
}
b.close();
} catch(IOException e) { // handle }
Skip the first few characters using InputStream#read()
while (is.read() != '\0');
This question already has answers here:
Closed 12 years ago.
Possible Duplicates:
How do I convert an InputStream to a String in Java?
In Java how do a read an input stream in to a string?
I have an InputSteam and need to simply get a single simple String with the complete contents.
How is this done in Java?
Here is a modification of Gopi's answer that doesn't have the line ending problem and is also more effective as it doesn't need temporary String objects for every line and avoids the redundant copying in BufferedReader and the extra work in readLine().
public static String convertStreamToString( InputStream is, String ecoding ) throws IOException
{
StringBuilder sb = new StringBuilder( Math.max( 16, is.available() ) );
char[] tmp = new char[ 4096 ];
try {
InputStreamReader reader = new InputStreamReader( is, ecoding );
for( int cnt; ( cnt = reader.read( tmp ) ) > 0; )
sb.append( tmp, 0, cnt );
} finally {
is.close();
}
return sb.toString();
}
You need to construct an InputStreamReader to wrap the input stream, converting between binary data and text. Specify the appropriate encoding based on your input source.
Once you've got an InputStreamReader, you could create a BufferedReader and read the contents line by line, or just read buffer-by-buffer and append to a StringBuilder until the read() call returns -1.
The Guava library makes the second part of this easy - use CharStreams.toString(inputStreamReader).
Here is an example code adapted from here.
public String convertStreamToString(InputStream is) throws IOException {
/*
* To convert the InputStream to String we use the BufferedReader.readLine()
* method. We iterate until the BufferedReader return null which means
* there's no more data to read. Each line will appended to a StringBuilder
* and returned as String.
*/
if (is != null) {
StringBuilder sb = new StringBuilder();
String line;
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(is, "UTF-8"));
while ((line = reader.readLine()) != null) {
sb.append(line).append("\n");
}
} finally {
is.close();
}
return sb.toString();
} else {
return "";
}
}
You can also use Apache Commons IO library
Specifically, you can use IOUtils#toString(InputStream inputStream) method
You could also use a StringWriter as follows; each read from your InputStream is matched with a write (or append) to the StringWriter, and upon completion you can call getBuffer to get a StringBuffer which could be used directly or you could get call its toString method.
Wrap the Stream in a Reader to get locale conversion, and then keep reading while collecting in a StringBuffer. When done, do a toString() on the StringBuffer.
What ist most concise way to read the contents of a file or input stream in Java? Do I always have to create a buffer, read (at most) line by line and so on or is there a more concise way? I wish I could do just
String content = new File("test.txt").readFully();
Use the Apache Commons IOUtils package. In particular the IOUtils class provides a set of methods to read from streams, readers etc. and handle all the exceptions etc.
e.g.
InputStream is = ...
String contents = IOUtils.toString(is);
// or
List lines = IOUtils.readLines(is)
I think using a Scanner is quite OK with regards to conciseness of Java on-board tools:
Scanner s = new Scanner(new File("file"));
StringBuilder builder = new StringBuilder();
while(s.hasNextLine()) builder.append(s.nextLine());
Also, it's quite flexible, too (e.g. regular expressions support, number parsing).
Helper functions. I basically use a few of them, depending on the situation
cat method that pipes an InputStream to an OutputStream
method that calls cat to a ByteArrayOutputStream and extracts the byte array, enabling quick read of an entire file to a byte array
Implementation of Iterator<String> that is constructed using a Reader; it wraps it in a BufferedReader and readLine's on next()
...
Either roll your own or use something out of commons-io or your preferred utility library.
To give an example of such an helper function:
String[] lines = NioUtils.readInFile(componentxml);
The key is to try to close the BufferedReader even if an IOException is thrown.
/**
* Read lines in a file. <br />
* File must exist
* #param f file to be read
* #return array of lines, empty if file empty
* #throws IOException if prb during access or closing of the file
*/
public static String[] readInFile(final File f) throws IOException
{
final ArrayList lines = new ArrayList();
IOException anioe = null;
BufferedReader br = null;
try
{
br = new BufferedReader(new FileReader(f));
String line;
line = br.readLine();
while(line != null)
{
lines.add(line);
line = br.readLine();
}
br.close();
br = null;
}
catch (final IOException e)
{
anioe = e;
}
finally
{
if(br != null)
{
try {
br.close();
} catch (final IOException e) {
anioe = e;
}
}
if(anioe != null)
{
throw anioe;
}
}
final String[] myStrings = new String[lines.size()];
//myStrings = lines.toArray(myStrings);
System.arraycopy(lines.toArray(), 0, myStrings, 0, lines.size());
return myStrings;
}
(if you just want a String, change the function to append each lines to a StringBuffer (or StringBuilder in java5 or 6)
String content = (new RandomAccessFile(new File("test.txt"))).readUTF();
Unfortunately Java is very picky about the source file being valid UTF8 though, or you will get an EOFException or UTFDataFormatException.
You have to create your own function, I suppose. The problem is that Java's read routines (those I know, at least) usually take a buffer argument with a given length.
A solution I saw is to get the size of the file, create a buffer of this size and read the file at once. Hoping the file isn't a gigabyte log or XML file...
The usual way is to have a fixed size buffer or to use readLine and concatenate the results in a StringBuffer/StringBuilder.
I don't think reading using BufferedReader is a good idea because BufferedReader will return just the content of line without the delimeter. When the line contains nothing but newline character, BR will return a null although it still doesn't reach the end of the stream.
String org.apache.commons.io.FileUtils.readFileToString(File file)
Pick one from here.
How do I create a Java string from the contents of a file?
The favorite was:
private static String readFile(String path) throws IOException {
FileInputStream stream = new FileInputStream(new File(path));
try {
FileChannel fc = stream.getChannel();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
/* Instead of using default, pass in a decoder. */
return CharSet.defaultCharset().decode(bb).toString();
}
finally {
stream.close();
}
}
Posted by erickson
Or the Java 8 way:
try {
String str = new String(Files.readAllBytes(Paths.get("myfile.txt")));
...
} catch (IOException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
One may pass an appropriate Charset to the String constructor.