I want to use a BufferedReader to read a file uploaded to my server.
The file would by written as a CSV file, but I can't assume this, so I code some test where the file is an image or a binary file (supposing the client has sent me the wrong file or an attacker is trying to break my service), or even worse, the file is a valid CSV file but has a line of 100MB.
My application can deal with this problem, but it has to read the first line of the file:
...
String firstLine = bufferedReader.readLine();
//Perform some validations and reject the file if it's not a CSV file
...
But, when I code some tests, I've found a potential risk: BufferedReader doesn't perform any control over the amount of bytes it reads until it found a return line, so it can ended up throwing an OutOfMemoryError.
This is my test:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.Reader;
import org.junit.Test;
public class BufferedReaderTest {
#Test(expected=OutOfMemoryError.class)
public void testReadFileWithoutReturnLineCharacter() throws IOException {
BufferedReader bf = new BufferedReader(getInfiniteReader());
bf.readLine();
bf.close();
}
private Reader getInfiniteReader() {
return new Reader(){
#Override
public int read(char[] cbuf, int off, int len) throws IOException {
return 'A';
}
#Override
public void close() throws IOException {
}
};
}
}
I've been looking up some safe BufferedReader implementation on the internet, but I can't find anything. The only class I've found was BoundedInputStream from apache IO, that limits the amount of bytes read by an input stream.
I need an implementation of BufferedReader that knows how to limit the number of bytes/characters read in each line.
Something like this:
The app calls 'readLine()'
The BufferedReader reads bytes until it found a return line character or it reaches the maximum amount of bytes allowed
If it has found a return line character, then reset the bytes read (so it could read the next line) and return the content
If it has reached the maximum amount of bytes allowed, it throws an exception
Does anybody knows about an implementation of BufferedReader that has this behaviour?
This is not how you should proceed to detect whether a file is binary or not.
Here is how you can do to check whether a file is truly text or not; note that this requires that you know the encoding beforehand:
final Charset cs = StandardCharsets.UTF_8; // or another
final CharsetDecoder decoder = cs.newDecoder()
.onMalformedInput(CodingErrorAction.REPORT); // default is REPLACE!
// Here, "in" is the input stream from the file
try (
final Reader reader = new InputStreamReader(in, decoder);
) {
final char[] buf = new char[4096]; // or other size
while (reader.read(buf) != -1)
; // nothing
} catch (MalformedInputException e) {
// cannot decode; binary, or wrong encoding
}
Now, since you can initialize a BufferedReader over a Reader, you can use:
try (
final Reader r = new InputStreamReader(in, decoder);
final BufferedReader reader = new BufferedReader(r);
) {
// Read lines normally
} catch (CharacterCodingException e) {
// Not a CSV, it seems
}
// etc
Now, a little more explanation about how this works... While this is a fundamenal part of reading text in Java, it is a part which is equally fundamentally misunderstood!
When you read a file as text using a Reader, you have to specify a character coding; in Java, this is a Charset.
What happens internally is that Java will create a CharsetDecoder from that Charset, read the byte stream and output a char stream. And there are three ways to deal with errors:
CodingErrorAction.REPLACE (the default): unmappable byte sequences are replaced with the Unicode replacement character (it does ring a bell, right?);
CodingErrorAction.IGNORE: unmappable byte sequences do not trigger the emission of a char;
CodingErrorAction.REPORT: unmappable byte sequences trigger a CharacterCodingException to be thrown, which inherits IOException; in turn, the two subclasses of CharacterCodingException are MalformedInputException and UnmappableCharacterException.
Therefore, what you need to do in order to detect whether a file is truly text is to:
know the encoding beforehand!
use a CharsetDecoder configured with CodingErrorAction.REPORT;
use it in an InputStreamReader.
This is one way; there are others. All of them however will use a CharsetDecoder at some point.
Similarly, there is a CharsetEncoder for the reverse operation (char stream to byte stream), and this is what is used by the Writer family.
Thank you #fge for the answer. I ended up implementing a safe Readerthat can deal with files with too long lines (or without lines at all).
If anybody wants to see the code, the project (very small project even with many tests) is available here:
https://github.com/jfcorugedo/security-io
Related
I have file where every line represents vertice. (format for example- 1.0 0.0 vertice A)
My task is to create method
public void read(InputStream is) throws IOException
Which would save X and Y values of vertices and then label of it "vertice A". I have no idea how to parse it properly:
public void read(InputStream is) throws IOException {
try {
Reader r = new InputStreamReader(is);
BufferedReader br = new BufferedReader(r);
while(br.readLine()!=null){
//something
}
} catch(IOException ex){
ex.printStackTrace();
}
}
also I need to create method
public void read(File file) throws IOException
which makes exactly the same but with file instead of stream. Can you tell me difference between these two methods?
A File represents a node on the filesystem, a stream is a representation of a sequence of data with a read head. Opening a file for reading results in an input stream. System.In is an example of an input stream for which you did not provide a file, it is the stream for stdin.
public void read(File file) throws IOException
{
//using your input stream method, read the passed file
//Create an input stream from the given file, and then call what you've already implemented.
read(new FileInputStream(file));
//I assume your read function closes the stream when it's done
}
I'd do the following and explain it through the code :)
public void read(InputStream is) throws IOException {
//You create a reader hold the input stream (sequence of data)
//You create a BufferedReader which will wrap the input stream and give you methods to read your information
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
handleVerticeValues(reader);
reader.close();
}
public void read(File file) throws IOException {
//You create a buffered reader to manipulate the data obtained from the file representation
BufferedReader reader = new BufferedReader(new FileReader(file));
handleVerticeValues(reader);
reader.close();
}
private void handleVerticeValues(BufferedReader reader) throws IOException {
//Then you can read your file like this:
String lineCursor = null;//Will hold the value of the line being read
//Assuming your line has this format: 1.0 0.0 verticeA
//Your separator between values is a whitespace character
while ((lineCursor = reader.readLine()) != null) {
String[] lineElements = lineCursor.split(" ");//I use split and indicates that we will separate each element of your line based on a whitespace
double valX = Double.parseDouble(lineElements[0]);//You get the first element before an whitespace: 1.0
double valY = Double.parseDouble(lineElements[1]);//You get the second element before and after an whitespace: 0.0
String label = lineElements[2];//You get the third element after the last whitespace
//You do something with your data
}
}
You can avoid using split by using StringTokenizer as well, that is another approach :).
As mentioned in the other answer, a file is only a representation of a node in your file system, it just point to an element existing in your file system but it not hold any data or information at this point of your internal file, I mean, just information as a file (if is a file, directory or something like that) (if it does not exist, you receive a FileNotFoundException).
The InputStream is a sequence of data, at this point, you should have information here which needs to be handled or read by a BufferedReader, ObjectInputStream or another component, depending on what you need to do.
For more info, you can also ask to your friendly API docs:
https://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html
https://docs.oracle.com/javase/7/docs/api/java/io/File.html
Regards and... happy coding :)
I have been unable to find the reason for this. The only problem I am having in this code is that when the FileWriter tries to put the new value into the text file, it instead puts a ?. I have no clue why, or even what it means. Here is the code:
if (secMessage[1].equalsIgnoreCase("add")) {
if (secMessage.length==2) {
try {
String deaths = readFile("C:/Users/Samboni/Documents/Stuff For Streaming/deaths.txt", Charset.defaultCharset());
FileWriter write = new FileWriter("C:/Users/Samboni/Documents/Stuff For Streaming/deaths.txt");
int comb = Integer.parseInt(deaths) + 1;
write.write(comb);
write.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
And here is the readFile method:
static String readFile(String path, Charset encoding) throws IOException {
byte[] encoded = Files.readAllBytes(Paths.get(path));
return new String(encoded, encoding);
}
Also, the secMessage array is an array of strings containing the words of an IRC message split into individual words, that way the program can react to the commands on a word-by-word basis.
You're calling Writer.write(int). That writes a single UTF-16 code point to the file, taking just the bottom 16 bits. If your platform default encoding isn't able to represent the code point you're trying to write, it will write '?' as a replacement character.
I suspect you actually want to write out a text representation of the number, in which case you should use:
write.write(String.valueOf(comb));
In other words, turn the value into a string and then write it out. So if comb is 123, you'll get three characters ('1', '2', '3') written to the file.
Personally I'd avoid FileWriter though - I prefer using OutputStreamWriter wrapping FileOutputStream so you can control the encoding. Or in Java 7, you can use Files.newBufferedWriter to do it more simply.
write.write(new Integer(comb).toString());
You can convert the int into a string. Otherwise you will need the int to be a character. That will only work for a small subset of numbers, 0-9, so it is not recommended.
Why following code
import java.io.DataOutputStream;
import java.io.FileOutputStream;
import java.io.IOException;
public class Test {
public static void main(String[] args) throws IOException {
DataOutputStream out = new DataOutputStream( new FileOutputStream("plik.txt"));
String test = new String("test");
out.writeUTF(test);
out.close();
}
}
produces file with "null" and eot characters at start of the file, followed by "test"? I expected that it would produce file with only "test", without additional stuff.
From the Javadoc of DataOutputStream.writeUTF():
First, two bytes are written to the output stream as if by the
writeShort method giving the number of bytes to follow. This value is
the number of bytes actually written out, not the length of the
string. Following the length, each character of the string is output,
in sequence, using the modified UTF-8 encoding for the character.
[...]
So it's not 2 characters but just 2 bytes.
The purpose of DataOutputStream is:
A data output stream lets an application write primitive Java data
types to an output stream in a portable way. An application can then use a data input stream to read the data back in.
Also note, that's it's recommended to use the NIO2 API and the Automatic Resource Management. And don't wrap a String with a String. Also use a buffer for your streams when writing to the file system.
try (DataOutputStream out = new DataOutputStream(new BufferedOutputStream(Files.newOutputStream(Paths.get("plik.txt")))){
String test = "test";
out.writeUTF(test);
}
If you just want to write some text to a file in UTF-8, use the following code:
try (PrintWriter pw = new PrintWriter(Files.newBufferedWriter(Paths.get("plik.txt"), Charset.forName("UTF-8"))){
String test = "test";
pw.print(test);
}
I've never had close experiences with Java IO API before and I'm really frustrated now. I find it hard to believe how strange and complex it is and how hard it could be to do a simple task.
My task: I have 2 positions (starting byte, ending byte), pos1 and pos2. I need to read lines between these two bytes (including the starting one, not including the ending one) and use them as UTF8 String objects.
For example, in most script languages it would be a very simple 1-2-3-liner like that (in Ruby, but it will be essentially the same for Python, Perl, etc):
f = File.open("file.txt").seek(pos1)
while f.pos < pos2 {
s = f.readline
# do something with "s" here
}
It quickly comes hell with Java IO APIs ;) In fact, I see two ways to read lines (ending with \n) from regular local files:
RandomAccessFile has getFilePointer() and seek(long pos), but it's readLine() reads non-UTF8 strings (and even not byte arrays), but very strange strings with broken encoding, and it has no buffering (which probably means that every read*() call would be translated into single undelying OS read() => fairly slow).
BufferedReader has great readLine() method, and it can even do some seeking with skip(long n), but it has no way to determine even number of bytes that has been already read, not mentioning the current position in a file.
I've tried to use something like:
FileInputStream fis = new FileInputStream(fileName);
FileChannel fc = fis.getChannel();
BufferedReader br = new BufferedReader(
new InputStreamReader(
fis,
CHARSET_UTF8
)
);
... and then using fc.position() to get current file reading position and fc.position(newPosition) to set one, but it doesn't seem to work in my case: looks like it returns position of a buffer pre-filling done by BufferedReader, or something like that - these counters seem to be rounded up in 16K increments.
Do I really have to implement it all by myself, i.e. a file readering interface which would:
allow me to get/set position in a file
buffer file reading operations
allow reading UTF8 strings (or at least allow operations like "read everything till the next \n")
Is there a quicker way than implementing it all myself? Am I overseeing something?
import org.apache.commons.io.input.BoundedInputStream
FileInputStream file = new FileInputStream(filename);
file.skip(pos1);
BufferedReader br = new BufferedReader(
new InputStreamReader(new BoundedInputStream(file,pos2-pos1))
);
If you didn't care about pos2, then you woundn't need Apache Commons IO.
I wrote this code to read utf-8 using randomaccessfiles
//File: CyclicBuffer.java
public class CyclicBuffer {
private static final int size = 3;
private FileChannel channel;
private ByteBuffer buffer = ByteBuffer.allocate(size);
public CyclicBuffer(FileChannel channel) {
this.channel = channel;
}
private int read() throws IOException {
return channel.read(buffer);
}
/**
* Returns the byte read
*
* #return byte read -1 - end of file reached
* #throws IOException
*/
public byte get() throws IOException {
if (buffer.hasRemaining()) {
return buffer.get();
} else {
buffer.clear();
int eof = read();
if (eof == -1) {
return (byte) eof;
}
buffer.flip();
return buffer.get();
}
}
}
//File: UTFRandomFileLineReader.java
public class UTFRandomFileLineReader {
private final Charset charset = Charset.forName("utf-8");
private CyclicBuffer buffer;
private ByteBuffer temp = ByteBuffer.allocate(4096);
private boolean eof = false;
public UTFRandomFileLineReader(FileChannel channel) {
this.buffer = new CyclicBuffer(channel);
}
public String readLine() throws IOException {
if (eof) {
return null;
}
byte x = 0;
temp.clear();
while ((byte) -1 != (x = (buffer.get())) && x != '\n') {
if (temp.position() == temp.capacity()) {
temp = addCapacity(temp);
}
temp.put(x);
}
if (x == -1) {
eof = true;
}
temp.flip();
if (temp.hasRemaining()) {
return charset.decode(temp).toString();
} else {
return null;
}
}
private ByteBuffer addCapacity(ByteBuffer temp) {
ByteBuffer t = ByteBuffer.allocate(temp.capacity() + 1024);
temp.flip();
t.put(temp);
return t;
}
public static void main(String[] args) throws IOException {
RandomAccessFile file = new RandomAccessFile("/Users/sachins/utf8.txt",
"r");
UTFRandomFileLineReader reader = new UTFRandomFileLineReader(file
.getChannel());
int i = 1;
while (true) {
String s = reader.readLine();
if (s == null)
break;
System.out.println("\n line " + i++);
s = s + "\n";
for (byte b : s.getBytes(Charset.forName("utf-8"))) {
System.out.printf("%x", b);
}
System.out.printf("\n");
}
}
}
For #Ken Bloom A very quick go at a Java 7 version. Note: I don't think this is the most efficient way, I'm still getting my head around NIO.2, Oracle has started their tutorial here
Also note that this isn't using Java 7's new ARM syntax (which takes care of the Exception handling for file based resources), it wasn't working in the latest openJDK build that I have. But if people want to see the syntax, let me know.
/*
* Paths uses the default file system, note no exception thrown at this stage if
* file is missing
*/
Path file = Paths.get("C:/Projects/timesheet.txt");
ByteBuffer readBuffer = ByteBuffer.allocate(readBufferSize);
FileChannel fc = null;
try
{
/*
* newByteChannel is a SeekableByteChannel - this is the fun new construct that
* supports asynch file based I/O, e.g. If you declared an AsynchronousFileChannel
* you could read and write to that channel simultaneously with multiple threads.
*/
fc = (FileChannel)file.newByteChannel(StandardOpenOption.READ);
fc.position(startPosition);
while (fc.read(readBuffer) != -1)
{
readBuffer.rewind();
System.out.println(Charset.forName(encoding).decode(readBuffer));
readBuffer.flip();
}
}
Start with a RandomAccessFile and use read or readFully to get a byte array between pos1 and pos2. Let's say that we've stored the data read in a variable named rawBytes.
Then create your BufferedReader using
new BufferedReader(new InputStreamReader(new ByteArrayInputStream(rawBytes)))
Then you can call readLine on the BufferedReader.
Caveat: this probably uses more memory than if you could make the BufferedReader seek to the right location itself, because it preloads everything into memory.
I think the confusion is caused by the UTF-8 encoding and the possibility of double byte characters.
UTF8 doesn't specify how many bytes are in a single character. I'm assuming from your post that you are using single byte characters. For example, 412 bytes would mean 411 characters. But if the string were using double byte characters, you would get the 206 character.
The original java.io package didn't deal well with this multi-byte confusion. So, they added more classes to deal specifically with strings. The package mixes two different types of file handlers (and they can be confusing until the nomenclature is sorted out). The stream classes provide for direct data I/O without any conversion. The reader classes convert files to strings with full support for multi-byte characters. That might help clarify part of the problem.
Since you state you are using UTF-8 characters, you want the reader classes. In this case, I suggest FileReader. The skip() method in FileReader allows you to pass by X characters and then start reading text. Alternatively, I prefer the overloaded read() method since it allows you to grab all the text at one time.
If you assume your "bytes" are individual characters, try something like this:
FileReader fr = new FileReader( new File("x.txt") );
char[] buffer = new char[ pos2 - pos ];
fr.read( buffer, pos, buffer.length );
...
I'm late to the party here, but I ran across this problem in my own project.
After much traversal of Javadocs and Stack Overflow, I think I found a simple solution.
After seeking to the appropriate place in your RandomAccessFile, which I am here calling raFile, do the following:
FileDescriptor fd = raFile.getFD();
FileReader fr = new FileReader(fd);
BufferedReader br = new BufferedReader(fr);
Then you should be able to call br.readLine() to your heart's content, which will be much faster than calling raFile.readLine().
The one thing I'm not sure about is whether UTF8 strings are handled correctly.
The java IO API is very flexible. Unfortunately sometimes the flexibility makes it verbose. The main idea here is that there are many streams, writers and readers that implement wrapper patter. For example BufferedInputStream wraps any other InputStream. The same is about output streams.
The difference between streams and readers/writers is that streams work with bytes while readers/writers work with characters.
Fortunately some streams, writers and readers have convenient constructors that simplify coding. If you want to read file you just have to say
InputStream in = new FileInputStream("/usr/home/me/myfile.txt");
if (in.markSupported()) {
in.skip(1024);
in.read();
}
It is not so complicated as you afraid.
Channels is something different. It is a part of so called "new IO" or nio. New IO is not blocked - it is its main advantage. You can search in internet for any "nio java tutorial" and read about it. But it is more complicated than regular IO and is not needed for most applications.
I currently use the following function to do a simple HTTP GET.
public static String download(String url) throws java.io.IOException {
java.io.InputStream s = null;
java.io.InputStreamReader r = null;
//java.io.BufferedReader b = null;
StringBuilder content = new StringBuilder();
try {
s = (java.io.InputStream)new URL(url).getContent();
r = new java.io.InputStreamReader(s);
//b = new java.io.BufferedReader(r);
char[] buffer = new char[4*1024];
int n = 0;
while (n >= 0) {
n = r.read(buffer, 0, buffer.length);
if (n > 0) {
content.append(buffer, 0, n);
}
}
}
finally {
//if (b != null) b.close();
if (r != null) r.close();
if (s != null) s.close();
}
return content.toString();
}
I see no reason to use the BufferedReader since I am just going to download everything in sequence. Am I right in thinking there is no use for the BufferedReader in this case?
In this case, I would do as you are doing (use a byte array for buffering and not one of the stream buffers).
There are exceptions, though. One place you see buffers (output this time) is in the servlet API. Data isn't written to the underlying stream until flush() is called, allowing you to buffer output but then dump the buffer if an error occurs and write an error page instead. You might buffer input if you needed to reset the stream for rereading using mark(int) and reset(). For example, maybe you'd inspect the file header before deciding on which content handler to pass the stream to.
Unrelated, but I think you should rewrite your stream handling. This pattern works best to avoid resource leaks:
InputStream stream = new FileInputStream("in");
try { //no operations between open stream and try block
//work
} finally { //do nothing but close this one stream in the finally
stream.close();
}
If you are opening multiple streams, nest try/finally blocks.
Another thing your code is doing is making the assumption that the returned content is encoded in your VM's default character set (though that might be adequate, depending on the use case).
You are correct, if you use BufferedReader for reading HTTP content and headers you will want InputStreamReader so you can read byte for byte.
BufferedReader in this scenario sometimes does weird things...escpecially when it comes to reading HTTP POST headers, sometimes you will be unable to read the POST data, if you use the InputStreamReader you can read the content length and read that many bytes...
Each invocation of one of an InputStreamReader's read() methods may cause one or more bytes to be read from the underlying byte-input stream. To enable the efficient conversion of bytes to characters, more bytes may be read ahead from the underlying stream than are necessary to satisfy the current read operation.
My gut tells me that since you're already performing buffering by using the byte array, it's redundant to use the BufferedReader.