How to read large Base64 file (150MB) on android app? - java

I'm trying to read large base64 Text File with size (~ 150MB) on android application.
The file contains JSON string that i need to decode and transform it to an JSON object, and use it along the app. The problem is that I'm getting an exception Out of Memory while trying to read this data.
The app needs to work offline so I need to download the full data.
Here's the code:
String localPath = getApplicationContext().getFilesDir().getPath().toString() ;
String key = "dataFile.txt" ;
StringBuilder text = new StringBuilder();
File file=new File(localPath+"/"+ key);
byte fileContent[] = new byte[3000];
try ( FileInputStream fin = new FileInputStream(file)) {
while(fin.read(fileContent) >= 0) {
byte[] data = Base64.decode(fileContent, Base64.DEFAULT);
try {
text.append(new String(data, "UTF-8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
obj = new JSONObject(text.toString());
}catch (Exception e){
e.printStackTrace();
}
How can i read this kind of file?

You are trying to read the the whole file into the text object by reading the file, iterating it and appending each line to text. You create the JSONObject out of your text object which is actually useful for your application only in the last step.
Here, by the time your code reaches the line obj = new JSONObject(text.toString()); you have already filled up the heap with nearly the size of your Input File as this complete file is in the memory in the form of the test object. You then make JSONObject of of this text object.
What you can do to eliminate this problem is as follows:
Use BufferedReader to read the file in chunks (Optional). Using read() may be a bit slow and it is nice to have a buffer.
Iterate the file and put the entries into the text object in batches of 1000 or 10000.
Prepare JSONObject out of text and append it to obj.
Clear the text object before processing the next batch and then repeat the whole process.
By doing this you are reading only a small portion of file in the memory and also text object is acting as a buffer, consuming only a small amount of the memory.
Here is the sample code snippet:
int counter = 0;
String temp = null;
final int BATCH_SIZE = 1000;
try (BufferedReader br = new BufferedReader(new FileReader(path)) {
while ((temp = br.readLine()) != null) {
text.append(temp);
++counter;
/* Process In Batches */
if(counter % BATCH_SIZE == 0) {
/* Prepare & Append JSON Objects */
obj = prepareAppendJSON(text.toString(), obj);
/* Clear text */
text.setLength(0);
}
}
/* Last Iteration */
obj = prepareAppendJSON(text, obj);
text = new StringBuilder();
} catch (IOException ex) {
ex.printStackTrace();
}

The only option you have is to use JSON streaming and react on events you are interested in.
import org.codehaus.jackson.*;
.....
JsonParser parser = new JsonFactory().createJsonParser( yourFileInputStream );
parser.configure( Feature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER, true );
parser.configure( Feature.ALLOW_SINGLE_QUOTES, true );
// add more features
for( JsonToken token = parser.nextToken(); null != token; token = parser.nextToken() ){
switch( token ){
case FIELD_NAME:
doStuffWithName();
break;
case START_OBJECT:
doObjectStart();
break;
case END_OBJECT:
processObject();
break;
// other events
}
}
I used the above code on 4.0 device and with JSON file of 10 MB.
PS. You gonna need to decode your original Base64 file first. Not sure if you can do it within a java.io.Stream on the fly. In the worst case unpack the Base64 file into a plain-json and then use JSON streaming code from above

Related

i want to send data from an array to a txt file

String[][] EmployeeArray = new String [1000][25];
this is my array, it has all the info I need in it already but I want to send all the data from here into an text file called EmployeeFile. How should I do this?
You can Serialize it, or even better do some json decorations/formatting and then write that to a file...
jusing json could be as simple as:
String[][] x = { { "0-0", "0-1", "0-2" }, { "1-0", "1-1", "1-2" }, { "2-0", "2-1, "2-2" } };
try (Writer writer = new FileWriter("my2DArray.json")) {
Gson gson = new GsonBuilder().create();
gson.toJson(x, writer);
}
First of all you will need to loop over your array and create a single string containing your data to be written to the file. This is done so you can add the new line characters where you want them.
String fileContents = new String();
for(int i=0; i<1000; i++) {
for(int j=0; j<25; j++) {
fileContents += EmployeeArray[i][j]
}
fileContents += "\n";
}
The code above is very basic and it is ment to demonstrate the basic idea.
It would be more efficient to use a StringBuilder and I guess there are 100 ways to improve those lines further.
Then you can use the following method to write to the file:
public boolean writeFile(String data, String path) {
try {
byte[] encoded = data.getBytes(StandardCharsets.UTF_8);
java.nio.file.Files.write(Paths.get(path),
encoded, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
return true;
} catch(IOException e) {
e.printStackTrace();
}
return false;
}
You should be careful with the options StandardOpenOption.CREATE and StandardOpenOption.TRUNCATE_EXISTING. They will overwrite existing files.
Somtimes you need to append to the file, adjust accordingly.
The StandardOpenOption documentation can be found here. StandardOperation.APPEND comes in handy for logging purposes.
Also note that the character set used is UTF8. It is generally a good idea to use UTF8 if it covers your needs. If you get strange characters in your data you might need to also adjust accordingly.

Loading from a text file

I'm making an android application that allows a user to add collectibles to a shelf, and the name of each collectible gets held in array called owned. I've written some code (with the help of some folks here) that should save my array into a text file.
That code looks like this:
#Override
public void onStop()
{
try{
super.onStop();
FileOutputStream fOut = openFileOutput("savedVinyls", Context.MODE_PRIVATE);
String vinylString="";
OutputStreamWriter osWriter = new OutputStreamWriter(fOut);
for(int i = 0; i < ownedCounter; i++)
{
if(!owned[i].equals(""))
vinylString += owned[i] + " ";
}
fOut.write(vinylString.getBytes());
fOut.close();
}
catch(IOException OE){
OE.getStackTrace();
}
}
This should, I think, save the array to a file so that it can be read back in later (This part appears to be working correctly, as certain parts of the loading code work)
The code for loading this file back and reading it is as follows:
try{
FileInputStream fileStream;
fileStream = openFileInput("savedVinyls");
byte[] buffer = new byte[1024];
StringBuffer line = new StringBuffer("");
while(fileStream.read(buffer) != -1)
{
line.append(new String(buffer));
}
System.out.println(line);
String readVinyls = line.toString();
String[] splitReader = readVinyls.split(" ");
owned = splitReader;
}
catch(IOException OE)
{
OE.printStackTrace();
}
I may be going about the file reading/writing all the wrong way, my goal is to make it so that when the user closes the app it saves their collection string, and then loads it when the app is loaded next time.
Currently, it crashes when it tries to load the text file.
Any help would be greatly appreciated!
With an ArrayIndexOutOfBoundsException it means you are trying to access an invalid index in the array.
Basically you are calling owned[i] at some point, where i is not a valid index. (Greater than the size of owned)

Reading from InflaterInputStream and parsing the result

I am quite new to java, just started yesterday. Since I am a big fan of learning by doing, I am making a small project with it. But I am stucked in this part. I have written a file using this function:
public static boolean writeZippedFile(File destFile, byte[] input) {
try {
// create file if doesn't exist part was here
try (OutputStream out = new DeflaterOutputStream(new FileOutputStream(destFile))) {
out.write(input);
}
return true;
} catch (IOException e) {
// error handlind was here
}
}
Now that I have successully wrote a compressed file using above method, I want to read it back to console. First I need to be able to read the decompressed content and write string representaion of that content to console. However, I have a second problem that I don't want to write characters up to first \0 null character. Here is how I attempt to read the compressed file:
try (InputStream is = new InflaterInputStream(new FileInputStream(destFile))) {
}
and I am completely stuck here. Question is, how to discard first few character until '\0' and then write the rest of the decompressed file to console.
I understand that your data contain text since you want to print a string respresentation. I further assume that the text contains unicode characters. If this is true, then your console should also support unicode for the characters to be displayed correctly.
So you should first read the data byte by byte until you encounter the \0 character and then you can use a BufferedReader to print the rest of the data as lines of text.
try (InputStream is = new InflaterInputStream(new FileInputStream(destFile))) {
// read the stream a single byte each time until we encounter '\0'
int aByte = 0;
while ((aByte = is.read()) != -1) {
if (aByte == '\0') {
break;
}
}
// from now on we want to print the data
BufferedReader b = new BufferedReader(new InputStreamReader(is, "UTF8"));
String line = null;
while ((line = b.readLine()) != null) {
System.out.println(line);
}
b.close();
} catch(IOException e) { // handle }
Skip the first few characters using InputStream#read()
while (is.read() != '\0');

File -> byte[] -> String -> byte[] -> File Conversion

This is basically what I am trying to do.
I wanna take a File
Turn it into a Byte Array
Turn it into a String
Store it in a MySQL Table
Retrieve the String
Turn it back into a Byte Array
Turn it back into a File
Now, I have some code for you, which I tried to comment as best as I could. My problem is, that the file I get at the end of this code, doesn't come out right. It's missing information. It's a text file, so I should be able to tell whether the file is complete or not.
As far as I can see, it looks like I only get the last part of the file, and not the entire file. I am pretty sure I messing something up badly somewhere in this conversion. If you got suggestions on how to do this conversion and retrieval more efficiently (Still keeping the Database and all that in mind), please let me know as well!
The code is listed below
import java.io.*;
import java.util.StringTokenizer;
import java.util.logging.Level;
import java.util.logging.Logger;
public class main {
public static void main(String[] args) {
// The file we want to save.
File f = new File("build.xml");
try {
// Make it into a byte array first
FileInputStream fis = new FileInputStream(f);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
try {
for(int readNum; (readNum = fis.read(buf)) != -1;) {
bos.write(buf, 0, readNum);
System.out.println("read " + readNum + " bytes,");
}
StringBuilder s = new StringBuilder();
// Now we simulate making it into a String, for easier storage
// in a database.
for(byte b : buf) {
// for debugging
s.append(b).append(",");
System.out.print(b +",");
}
// Now we want to retrieve the file from the database as a string
File someFile = new File("build2.xml");
FileOutputStream fos = new FileOutputStream(someFile);
// We count how many bytes there are in this string.
// One byte per Token.
StringTokenizer st = new StringTokenizer(s.toString(),",");
buf = new byte[st.countTokens()];
int i = 0;
StringBuilder t = new StringBuilder();
// Now we parse out all Bytes from the string, and put them into
// the prepared byte array.
while(st.hasMoreTokens()) {
byte b = Byte.parseByte(st.nextToken());
System.out.print(b + ",");
buf[i] = b;
i++;
// for debugging
t.append(b).append(",");
}
// Here I print true if both strings are exactly the same
// which they should be, which means that the bytes are intact
// before and after conversion.
System.out.println("\n" +(t.toString().equals(s.toString()) ? true : false));
// Here we would make the physical file on the machine.
fos.write(buf);
fos.flush();
fos.close();
} catch (IOException ex) {
Logger.getLogger(main.class.getName()).log(Level.SEVERE, null, ex);
}
} catch (FileNotFoundException ex) {
Logger.getLogger(main.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
http://pastebin.com/699yuE8f
Your approach is totally ignoring encodings, which is not a good thing. Characters are not equal to or equivalent to bytes.
If you have to do it in the sequence you describe, then create the string by something like this:
String intermediateString = new String(theByteArray,
theSameEncodingTheFileWasCreatedWith);
Likewise, when you convert the string back into bytes, get the bytes like this:
byte[] bytesToSave = intermediateString.getBytes(theSameEncodingTheFileWasCreatedWith);
But besides any of that, what's the point of using the string at all? Why not just store the bytes right into the database?
You simply messed up the string creation, and you don't read the bos but the buf.
for(byte b : >>buf<<) {
// for debugging
s.append(b).append(",");
System.out.print(b +",");
}
Otherwise I am not convinced that it will work or it is a good solution. Why can't you just store it simply in the database?
The code you shared is IMHO more complicated as it had to be.
Why do you read your text on byte-level if you are only interested in it's String representation?
I would prefer to read the file using an InputStreamReader. That allows you to directly operate on characters.

Modify a .txt file in Java

I have a text file that I want to edit using Java. It has many thousands of lines. I basically want to iterate through the lines and change/edit/delete some text. This will need to happen quite often.
From the solutions I saw on other sites, the general approach seems to be:
Open the existing file using a BufferedReader
Read each line, make modifications to each line, and add it to a StringBuilder
Once all the text has been read and modified, write the contents of the StringBuilder to a new file
Replace the old file with the new file
This solution seems slightly "hacky" to me, especially if I have thousands of lines in my text file.
Anybody know of a better solution?
I haven't done this in Java recently, but writing an entire file into memory seems like a bad idea.
The best idea that I can come up with is open a temporary file in writing mode at the same time, and for each line, read it, modify if necessary, then write into the temporary file. At the end, delete the original and rename the temporary file.
If you have modify permissions on the file system, you probably also have deleting and renaming permissions.
if the file is just a few thousand lines you should be able to read the entire file in one read and convert that to a String.
You can use apache IOUtils which has method like the following.
public static String readFile(String filename) throws IOException {
File file = new File(filename);
int len = (int) file.length();
byte[] bytes = new byte[len];
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
assert len == fis.read(bytes);
} catch (IOException e) {
close(fis);
throw e;
}
return new String(bytes, "UTF-8");
}
public static void writeFile(String filename, String text) throws IOException {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(filename);
fos.write(text.getBytes("UTF-8"));
} catch (IOException e) {
close(fos);
throw e;
}
}
public static void close(Closeable closeable) {
try {
closeable.close();
} catch(IOException ignored) {
}
}
You can use RandomAccessFile in Java to modify the file on one condition:
The size of each line has to be fixed otherwise, when new string is written back, it might override the string in the next line.
Therefore, in my example, I set the line length as 100 and padding with space string when creating the file and writing back to the file.
So in order to allow update, you need to set the length of line a little larger than the longest length of the line in this file.
public class RandomAccessFileUtil {
public static final long RECORD_LENGTH = 100;
public static final String EMPTY_STRING = " ";
public static final String CRLF = "\n";
public static final String PATHNAME = "/home/mjiang/JM/mahtew.txt";
/**
* one two three
Text to be appended with
five six seven
eight nine ten
*
*
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException
{
String starPrefix = "Text to be appended with";
String replacedString = "new text has been appended";
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line = "";
while((line = file.readLine()) != null)
{
if(line.startsWith(starPrefix))
{
file.seek(file.getFilePointer() - RECORD_LENGTH - 1);
file.writeBytes(replacedString);
}
}
}
public static void createFile() throws IOException
{
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line1 = "one two three";
String line2 = "Text to be appended with";
String line3 = "five six seven";
String line4 = "eight nine ten";
file.writeBytes(paddingRight(line1));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line2));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line3));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line4));
file.writeBytes(CRLF);
file.close();
System.out.println(String.format("File is created in [%s]", PATHNAME));
}
public static String paddingRight(String source)
{
StringBuilder result = new StringBuilder(100);
if(source != null)
{
result.append(source);
for (int i = 0; i < RECORD_LENGTH - source.length(); i++)
{
result.append(EMPTY_STRING);
}
}
return result.toString();
}
}
If the file is large, you might want to use a FileStream for output, but that seems pretty much like it is the simplest process to do what you're asking (and without more specificity i.e. on what types of changes / edits / deletions you're trying to do, it's impossible to determine what more complicated way might work).
No reason to buffer the entire file.
Simply write each line as your read it, insert lines when necessary, delete lines when necessary, replace lines when necessary.
Fundamentally, you will not get around having to recreate the file wholesale, especially if it's just a text file.
What kind of data is it? Do you control the format of the file?
If the file contains name/value pairs (or similar), you could have some luck with Properties, or perhaps cobbling together something using a flat file JDBC driver.
Alternatively, have you considered not writing the data so often? Operating on an in-memory copy of your file should be relatively trivial. If there are no external resources which need real time updates of the file, then there is no need to go to disk every time you want to make a modification. You can run a scheduled task to write periodic updates to disk if you are worried about data backup.
In general you cannot edit the file in place; it's simply a very long sequence of characters, which happens to include newline characters. You could edit in place if your changes don't change the number of characters in each line.
Can't you use regular expressions, if you know what you want to change ? Jakarta Regexp should probably do the trick.
Although this question was a time ago posted, I think it is good to put my answer here.
I think that the best approach is to use FileChannel from java.nio.channels package in this scenario. But this, only if you need to have a good performance! You would need to get a FileChannel via a RandomAccessFile, like this:
java.nio.channels.FileChannel channel = new java.io.RandomAccessFile("/my/fyle/path", "rw").getChannel();
After this, you need a to create a ByteBuffer where you will read from the FileChannel.
this looks something like this:
java.nio.ByteBuffer inBuffer = java.nio.ByteBuffer.allocate(100);
int pos = 0;
int aux = 0;
StringBuilder sb = new StringBuilder();
while (pos != -1) {
aux = channel.read(inBuffer, pos);
pos = (aux != -1) ? pos + aux : -1;
b = inBuffer.array();
sb.delete(0, sb.length());
for (int i = 0; i < b.length; ++i) {
sb.append((char)b[i]);
}
//here you can do your stuff on sb
inBuffer = ByteBuffer.allocate(100);
}
Hope that my answer will help you!
I think, FileOutputStream.getFileChannel() will help a lot, see FileChannel api
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html
private static void modifyFile(String filePath, String oldString, String newString) {
File fileToBeModified = new File(filePath);
StringBuilder oldContent = new StringBuilder();
try (BufferedReader reader = new BufferedReader(new FileReader(fileToBeModified))) {
String line = reader.readLine();
while (line != null) {
oldContent.append(line).append(System.lineSeparator());
line = reader.readLine();
}
String content = oldContent.toString();
String newContent = content.replaceAll(oldString, newString);
try (FileWriter writer = new FileWriter(fileToBeModified)) {
writer.write(newContent);
}
} catch (IOException e) {
e.printStackTrace();
}
}
You can change the txt file to java by saving on clicking "Save As" and saving *.java extension.

Categories

Resources