Develop Reference

Can't read from binary file - read some lines in UTF-8, some in binary

I have this code:
import java.io.*;
import java.nio.charset.StandardCharsets;
public class Main {
public static void main(String[] args) {
zero("zero.out");
System.out.println(zeroRead("zero.out"));
}
public static String zeroRead(String name) {
try (FileInputStream fos = new FileInputStream(name);
BufferedInputStream bos = new BufferedInputStream(fos);
DataInputStream dos = new DataInputStream(bos)) {
StringBuffer inputLine = new StringBuffer();
String tmp;
String s = "";
while ((tmp = dos.readLine()) != null) {
inputLine.append(tmp);
System.out.println(tmp);
}
dos.close();
return s;
}
catch (IOException e) {
e.printStackTrace();
}
return null;
}
public static void zero(String name) {
File file = new File(name);
String text = "König" + "\t";
try (FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
DataOutputStream dos = new DataOutputStream(bos)) {
dos.write(text.getBytes(StandardCharsets.UTF_8));
dos.writeInt(50);
} catch (IOException e) {
e.printStackTrace();
}
}
}
zero() method writes data into file: the string is written in UTF-8, while the number is written in binary. zeroRead() read the data from file.
The file looks like this after zero() is executed:
This is what zeroRead() returns:
How do I read the real data König\t50 from the file?

DataInputStream's readLine method has javadoc that is almost yelling that it doesn't want to be used. You should heed this javadoc: That method is bad and you should not use it. It doesn't do charset encoding.
Your file format is impossible as stated: You have no idea when to stop reading the string and start reading the binary numbers. However, the way you've described things, it sounds like the string is terminated by a newline, so, the \n character.
There is no easy 'just make this filter-reader and call .nextLine on it available, as they tend to buffer. You can try this:
InputStreamReader isr = new InputStreamReader(bos, StandardCharsets.UTF_8);
However, basic readers do not have a readLine method, and if you wrap this in a BufferedReader, it may read past the end (the 'buffer' in that name is not just there for kicks). You'd have to handroll a method that fetches one character at a time, appending them to a stringbuilder, ending on a newline:
StringBuilder out = new StringBuilder();
for (int c = isr.read(); c != -1 && c != '\n'; c = isr.read())
out.append((char) c);
String line = out.toString();
will get the job done and won't read 'past' the newline and gobble up your binary number.

Split one large file into multiple InputStream(s) to be processed in multithreading in Java

My codes:
public ArrayList<InputStream> getAllInputStreams() {
ArrayList<InputStream> allStreams = new ArrayList<InputStream>();
InputStream stream = this.getNext();
while (stream != null) {
allStreams.add(stream);
stream = this.getNext();
}
return allStreams;
}
public InputStream getNext() {
if (done()) {
return null;
}
InputStream segment = createInputStream();
this.countStream++;
return segment;
}
protected InputStream createInputStream() {
BoundedInputStream res = new BoundedInputStream(
Channels.newInputStream(this.randomAccessFile.getChannel().position(this.countStream * chunkSize)), chunkSize);
res.setPropagateClose(false) ;
return res ;
}
I am trying to split file into several InputStream(s) (private RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");. All InputStream(s) (got from getAllInputStreams()) to be processed by multiple threads, it seems that most of them are empty. Why?
Any hints welcomed. Thanks
UPDATE
It seems that the following piece of codes working fine. Is the following piece of codes a good way to split the file into several chucks? Should the size of each chuck smaller than memory size?
protected InputStream createInputStream() {
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
BoundedInputStream res = new BoundedInputStream(
Channels.newInputStream(randomAccessFile.getChannel().position(this.countStream * chunkSize)), chunkSize);
res.setPropagateClose(false) ;
return res ;
}

I think that this statement:
res.setPropagateClose(false);
in your "UPDATE" solution will result in a resource leak. The RAF that you have opened is not shared with anything else. When the BoundedInputStream is closed, you need the close to propagate to the RAF.

can try this:
InputStream inputStream = new FileInputStream(file);
inputStream.skip(startPos);
BoundedInputStream partInputStream = new BoundedInputStream(inputStream, curPartSize);
this also need new FileInputStream every time, so it's useless.

How to return value of FileInputStream in Android [duplicate]

This question already has answers here:
How do I read / convert an InputStream into a String in Java?
(62 answers)
Closed 9 years ago.
I have a function based on FileInputStream that should return a string she reads from the file. But why is the return value - empty.
The function's return value should be displayed in MainActivity.
This function is part of the service in my app.
public static String UserLoginFile;
public static String UserPassFile;
public String LoadUserInfopassFromFilePostData() {
String FILENAME = "TeleportSAASPass.txt";
FileInputStream inputStream = null;
if(UserLoginFile != null){
try {
inputStream = openFileInput(FILENAME);
byte[] reader = new byte[inputStream.available()];
while (inputStream.read(reader) != -1) {}
UserPassFile = new String(reader);
// Toast.makeText(getApplicationContext(), "GPSTracker " + UserPassFile, Toast.LENGTH_LONG).show();
} catch(IOException e) {
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
}
}
}
}
return UserPassFile;
}
Please tell me how to fix my function so that it can return a string which she read from the file.

I would suggest using an input stream reader. Check out
http://developer.android.com/reference/java/io/InputStreamReader.html
Your code will look something like:
inputStream = openFileInput(FILENAME);
InputStreamReader( inputstream, "UTF_8"), 8);
StringBuilder response = new StringBuilder();
String line;
//Read the response from input stream line-wise
while((line = reader.readLine()) != null){
response.append(line).append('\n');
}
//Create response string
result = response.toString();
//Close input stream
reader.close();

Consider the difference between your code:
while (inputStream.read(reader) != -1) {}
UserPassFile = new String(reader);
And this alternative:
while (inputStream.read(reader) != -1) {
UserPassFile = new String(reader);
}
In your case, UserPassFile is assigned from reader after every read() call, including the final one which fails. In the second case, UserPassFile is only assigned after a read which succeeds, and so should still contain a non-empty string to return when the end of the method is reached.
However, you seem to be doing one other odd thing. You assume that you will be able to read the entire .available() amount of data in a single read, which is probably true (at least if you were able to allocate a byte[] that large). However, after reading this data, you try again and see if you fail, which seems unnecessary. It would seem that if you are going to make the assumption that you can read the data in a single attempt, then you should simply close the stream and return once you have done so - there doesn't seem to be a compelling reason to try yet another read which would be expected to fail. Otherwise, if there's a possibility that you can't get all the data in a single read(), you should keep track of how much you have gotten and keep appending until you get a failed read or otherwise decide you are done.

Guava equivalent for IOUtils.toString(InputStream)

Apache Commons IO has a nice convenience method IOUtils.toString() to read an InputStream to a String.
Since I am trying to move away from Apache Commons and to Guava: is there an equivalent in Guava? I looked at all classes in the com.google.common.io package and I couldn't find anything nearly as simple.
Edit: I understand and appreciate the issues with charsets. It just so happens that I know that all my sources are in ASCII (yes, ASCII, not ANSI etc.), so in this case, encoding is not an issue for me.

You stated in your comment on Calum's answer that you were going to use
CharStreams.toString(new InputStreamReader(supplier.get(), Charsets.UTF_8))
This code is problematic because the overload CharStreams.toString(Readable) states:
Does not close the Readable.
This means that your InputStreamReader, and by extension the InputStream returned by supplier.get(), will not be closed after this code completes.
If, on the other hand, you take advantage of the fact that you appear to already have an InputSupplier<InputStream> and used the overload CharStreams.toString(InputSupplier<R extends Readable & Closeable>), the toString method will handle both the creation and closing of the Reader for you.
This is exactly what Jon Skeet suggested, except that there isn't actually any overload of CharStreams.newReaderSupplier that takes an InputStream as input... you have to give it an InputSupplier:
InputSupplier<? extends InputStream> supplier = ...
InputSupplier<InputStreamReader> readerSupplier =
CharStreams.newReaderSupplier(supplier, Charsets.UTF_8);
// InputStream and Reader are both created and closed in this single call
String text = CharStreams.toString(readerSupplier);
The point of InputSupplier is to make your life easier by allowing Guava to handle the parts that require an ugly try-finally block to ensure that resources are closed properly.
Edit: Personally, I find the following (which is how I'd actually write it, was just breaking down the steps in the code above)
String text = CharStreams.toString(
CharStreams.newReaderSupplier(supplier, Charsets.UTF_8));
to be far less verbose than this:
String text;
InputStreamReader reader = new InputStreamReader(supplier.get(),
Charsets.UTF_8);
boolean threw = true;
try {
text = CharStreams.toString(reader);
threw = false;
}
finally {
Closeables.close(reader, threw);
}
Which is more or less what you'd have to write to handle this properly yourself.
Edit: Feb. 2014
InputSupplier and OutputSupplier and the methods that use them have been deprecated in Guava 16.0. Their replacements are ByteSource, CharSource, ByteSink and CharSink. Given a ByteSource, you can now get its contents as a String like this:
ByteSource source = ...
String text = source.asCharSource(Charsets.UTF_8).read();

If you've got a Readable you can use CharStreams.toString(Readable). So you can probably do the following:
String string = CharStreams.toString( new InputStreamReader( inputStream, "UTF-8" ) );
Forces you to specify a character set, which I guess you should be doing anyway.

Nearly. You could use something like this:
InputSupplier<InputStreamReader> readerSupplier = CharStreams.newReaderSupplier
(streamSupplier, Charsets.UTF_8);
String text = CharStreams.toString(readerSupplier);
Personally I don't think that IOUtils.toString(InputStream) is "nice" - because it always uses the default encoding of the platform, which is almost never what you want. There's an overload which takes the name of the encoding, but using names isn't a great idea IMO. That's why I like Charsets.*.
EDIT: Not that the above needs an InputSupplier<InputStream> as the streamSupplier. If you've already got the stream you can implement that easily enough though:
InputSupplier<InputStream> supplier = new InputSupplier<InputStream>() {
#Override public InputStream getInput() {
return stream;
}
};

UPDATE: Looking back, I don't like my old solution. Besides it is 2013 now and there are better alternatives available now for Java7. So here is what I use now:
InputStream fis = ...;
String text;
try ( InputStreamReader reader = new InputStreamReader(fis, Charsets.UTF_8)){
text = CharStreams.toString(reader);
}
or if with InputSupplier
InputSupplier<InputStreamReader> spl = ...
try ( InputStreamReader reader = spl.getInput()){
text = CharStreams.toString(reader);
}

Another option is to read bytes from Stream and create a String from them:
new String(ByteStreams.toByteArray(inputStream))
new String(ByteStreams.toByteArray(inputStream), Charsets.UTF_8)
It's not 'pure' Guava, but it's a little bit shorter.

Based on the accepted answer, here is a utility method that mocks the behavior of IOUtils.toString() (and an overloaded version with a charset, as well). This version should be safe, right?
public static String toString(final InputStream is) throws IOException{
return toString(is, Charsets.UTF_8);
}
public static String toString(final InputStream is, final Charset cs)
throws IOException{
Closeable closeMe = is;
try{
final InputStreamReader isr = new InputStreamReader(is, cs);
closeMe = isr;
return CharStreams.toString(isr);
} finally{
Closeables.closeQuietly(closeMe);
}
}

There is much shorter autoclosing solution in case when input stream comes from classpath resource:
URL resource = classLoader.getResource(path);
byte[] bytes = Resources.toByteArray(resource);
String text = Resources.toString(resource, StandardCharsets.UTF_8);
Uses Guava Resources, inspired by IOExplained.

EDIT (2015): Okio is the best abstraction and tools for I/O in Java/Android that I know of. I use it all the time.
FWIW here's what I use.
If I already have a stream in hand, then:
final InputStream stream; // this is received from somewhere
String s = CharStreams.toString(CharStreams.newReaderSupplier(new InputSupplier<InputStream>() {
public InputStream getInput() throws IOException {
return stream;
}
}, Charsets.UTF_8));
If I'm creating a stream:
String s = CharStreams.toString(CharStreams.newReaderSupplier(new InputSupplier<InputStream>() {
public InputStream getInput() throws IOException {
return <expression creating the stream>;
}
}, Charsets.UTF_8));
As a concrete example, I can read an Android text file asset like this:
final Context context = ...;
String s = CharStreams.toString(CharStreams.newReaderSupplier(new InputSupplier<InputStream>() {
public InputStream getInput() throws IOException {
return context.getAssets().open("my_asset.txt");
}
}, Charsets.UTF_8));

For a concrete example, here's how I can read an Android text file asset:
public static String getAssetContent(Context context, String file) {
InputStreamReader reader = null;
InputStream stream = null;
String output = "";
try {
stream = context.getAssets().open(file);
reader = new InputStreamReader(stream, Charsets.UTF_8);
output = CharStreams.toString(reader);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (stream != null) {
try {
stream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return output;
}

Most concise way to read the contents of a file/input stream in Java?

What ist most concise way to read the contents of a file or input stream in Java? Do I always have to create a buffer, read (at most) line by line and so on or is there a more concise way? I wish I could do just
String content = new File("test.txt").readFully();

Use the Apache Commons IOUtils package. In particular the IOUtils class provides a set of methods to read from streams, readers etc. and handle all the exceptions etc.
e.g.
InputStream is = ...
String contents = IOUtils.toString(is);
// or
List lines = IOUtils.readLines(is)

I think using a Scanner is quite OK with regards to conciseness of Java on-board tools:
Scanner s = new Scanner(new File("file"));
StringBuilder builder = new StringBuilder();
while(s.hasNextLine()) builder.append(s.nextLine());
Also, it's quite flexible, too (e.g. regular expressions support, number parsing).

Helper functions. I basically use a few of them, depending on the situation
cat method that pipes an InputStream to an OutputStream
method that calls cat to a ByteArrayOutputStream and extracts the byte array, enabling quick read of an entire file to a byte array
Implementation of Iterator<String> that is constructed using a Reader; it wraps it in a BufferedReader and readLine's on next()
...
Either roll your own or use something out of commons-io or your preferred utility library.

To give an example of such an helper function:
String[] lines = NioUtils.readInFile(componentxml);
The key is to try to close the BufferedReader even if an IOException is thrown.
/**
* Read lines in a file. <br />
* File must exist
* #param f file to be read
* #return array of lines, empty if file empty
* #throws IOException if prb during access or closing of the file
*/
public static String[] readInFile(final File f) throws IOException
{
final ArrayList lines = new ArrayList();
IOException anioe = null;
BufferedReader br = null;
try
{
br = new BufferedReader(new FileReader(f));
String line;
line = br.readLine();
while(line != null)
{
lines.add(line);
line = br.readLine();
}
br.close();
br = null;
}
catch (final IOException e)
{
anioe = e;
}
finally
{
if(br != null)
{
try {
br.close();
} catch (final IOException e) {
anioe = e;
}
}
if(anioe != null)
{
throw anioe;
}
}
final String[] myStrings = new String[lines.size()];
//myStrings = lines.toArray(myStrings);
System.arraycopy(lines.toArray(), 0, myStrings, 0, lines.size());
return myStrings;
}
(if you just want a String, change the function to append each lines to a StringBuffer (or StringBuilder in java5 or 6)

String content = (new RandomAccessFile(new File("test.txt"))).readUTF();
Unfortunately Java is very picky about the source file being valid UTF8 though, or you will get an EOFException or UTFDataFormatException.

You have to create your own function, I suppose. The problem is that Java's read routines (those I know, at least) usually take a buffer argument with a given length.
A solution I saw is to get the size of the file, create a buffer of this size and read the file at once. Hoping the file isn't a gigabyte log or XML file...
The usual way is to have a fixed size buffer or to use readLine and concatenate the results in a StringBuffer/StringBuilder.

I don't think reading using BufferedReader is a good idea because BufferedReader will return just the content of line without the delimeter. When the line contains nothing but newline character, BR will return a null although it still doesn't reach the end of the stream.

String org.apache.commons.io.FileUtils.readFileToString(File file)

Pick one from here.
How do I create a Java string from the contents of a file?
The favorite was:
private static String readFile(String path) throws IOException {
FileInputStream stream = new FileInputStream(new File(path));
try {
FileChannel fc = stream.getChannel();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
/* Instead of using default, pass in a decoder. */
return CharSet.defaultCharset().decode(bb).toString();
}
finally {
stream.close();
}
}
Posted by erickson

Or the Java 8 way:
try {
String str = new String(Files.readAllBytes(Paths.get("myfile.txt")));
...
} catch (IOException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
One may pass an appropriate Charset to the String constructor.