I'm trying to read large base64 Text File with size (~ 150MB) on android application.
The file contains JSON string that i need to decode and transform it to an JSON object, and use it along the app. The problem is that I'm getting an exception Out of Memory while trying to read this data.
The app needs to work offline so I need to download the full data.
Here's the code:
String localPath = getApplicationContext().getFilesDir().getPath().toString() ;
String key = "dataFile.txt" ;
StringBuilder text = new StringBuilder();
File file=new File(localPath+"/"+ key);
byte fileContent[] = new byte[3000];
try ( FileInputStream fin = new FileInputStream(file)) {
while(fin.read(fileContent) >= 0) {
byte[] data = Base64.decode(fileContent, Base64.DEFAULT);
try {
text.append(new String(data, "UTF-8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
obj = new JSONObject(text.toString());
}catch (Exception e){
e.printStackTrace();
}
How can i read this kind of file?
You are trying to read the the whole file into the text object by reading the file, iterating it and appending each line to text. You create the JSONObject out of your text object which is actually useful for your application only in the last step.
Here, by the time your code reaches the line obj = new JSONObject(text.toString()); you have already filled up the heap with nearly the size of your Input File as this complete file is in the memory in the form of the test object. You then make JSONObject of of this text object.
What you can do to eliminate this problem is as follows:
Use BufferedReader to read the file in chunks (Optional). Using read() may be a bit slow and it is nice to have a buffer.
Iterate the file and put the entries into the text object in batches of 1000 or 10000.
Prepare JSONObject out of text and append it to obj.
Clear the text object before processing the next batch and then repeat the whole process.
By doing this you are reading only a small portion of file in the memory and also text object is acting as a buffer, consuming only a small amount of the memory.
Here is the sample code snippet:
int counter = 0;
String temp = null;
final int BATCH_SIZE = 1000;
try (BufferedReader br = new BufferedReader(new FileReader(path)) {
while ((temp = br.readLine()) != null) {
text.append(temp);
++counter;
/* Process In Batches */
if(counter % BATCH_SIZE == 0) {
/* Prepare & Append JSON Objects */
obj = prepareAppendJSON(text.toString(), obj);
/* Clear text */
text.setLength(0);
}
}
/* Last Iteration */
obj = prepareAppendJSON(text, obj);
text = new StringBuilder();
} catch (IOException ex) {
ex.printStackTrace();
}
The only option you have is to use JSON streaming and react on events you are interested in.
import org.codehaus.jackson.*;
.....
JsonParser parser = new JsonFactory().createJsonParser( yourFileInputStream );
parser.configure( Feature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER, true );
parser.configure( Feature.ALLOW_SINGLE_QUOTES, true );
// add more features
for( JsonToken token = parser.nextToken(); null != token; token = parser.nextToken() ){
switch( token ){
case FIELD_NAME:
doStuffWithName();
break;
case START_OBJECT:
doObjectStart();
break;
case END_OBJECT:
processObject();
break;
// other events
}
}
I used the above code on 4.0 device and with JSON file of 10 MB.
PS. You gonna need to decode your original Base64 file first. Not sure if you can do it within a java.io.Stream on the fly. In the worst case unpack the Base64 file into a plain-json and then use JSON streaming code from above
I have made reference from this question and achieved success in setting up encryption. I am trying to however utilize this encryption on a string of array to write into a file. This is how I am setting my method up but I end up writing only one of the string array into the file.
String[] str = new String ["X: Adam", "Y: Barry", "z: Oliver"];
File file = new File(Path + "/EncryptedFile.txt);
Calling method to write the string array into the file: Crypto.WriteEncrypteFile(str, file);
The method
Public void WriteEncrypteFile(String[] str, File file) {
try {
BufferedWriter w = new BufferedWriter(new FileWriter(file));
byte[] tmptxt = Array.toString(str).getbytes(Charset.forName(" UTF-8 "));
byte[] encTxt = cipher.doFinal(tmptxt);
w.write(string.valueOf(encTxt));
w.flush();
w.close();
} catch (Exception e){
e.printStackTrace();
}
My questions is how can I write an encrypted string from my array into the file. Any pointers?
You are just writing the String value of the array to the file (since you use Array.toString(str)). This will usually just be some representation of the reference. You have to either concatenate the values of the array or loop through it and encrypt/write every value individually.
Additionally, you shouldn't use a Writer to write content that does not consist of characters. Writers always try to encode the output which could potentially ruin your carefully set up bytes.
Just use a FileOutputStream and write the bytes with that:
try( FileOutputStream fos = new FileOutputStream(file) ) {
for(String s : str) {
byte[] tmptxt = s.getbytes(Charset.forName("UTF-8"));
byte[] encTxt = cipher.doFinal(tmptxt);
w.write(encTxt);
}
} catch(IOException e) {
// print error or whatever
}
For reading you do the same thing but with a FileInputStream instead.
You can use Arrays.toString(), but this way you will need to parse it to read it. Alternatively you can also write the byte[] directly in the file using an OutputStream. There's no need to convert to a string unless you want a human (e.g. yourself) to read it.
I'm making an android application that allows a user to add collectibles to a shelf, and the name of each collectible gets held in array called owned. I've written some code (with the help of some folks here) that should save my array into a text file.
That code looks like this:
#Override
public void onStop()
{
try{
super.onStop();
FileOutputStream fOut = openFileOutput("savedVinyls", Context.MODE_PRIVATE);
String vinylString="";
OutputStreamWriter osWriter = new OutputStreamWriter(fOut);
for(int i = 0; i < ownedCounter; i++)
{
if(!owned[i].equals(""))
vinylString += owned[i] + " ";
}
fOut.write(vinylString.getBytes());
fOut.close();
}
catch(IOException OE){
OE.getStackTrace();
}
}
This should, I think, save the array to a file so that it can be read back in later (This part appears to be working correctly, as certain parts of the loading code work)
The code for loading this file back and reading it is as follows:
try{
FileInputStream fileStream;
fileStream = openFileInput("savedVinyls");
byte[] buffer = new byte[1024];
StringBuffer line = new StringBuffer("");
while(fileStream.read(buffer) != -1)
{
line.append(new String(buffer));
}
System.out.println(line);
String readVinyls = line.toString();
String[] splitReader = readVinyls.split(" ");
owned = splitReader;
}
catch(IOException OE)
{
OE.printStackTrace();
}
I may be going about the file reading/writing all the wrong way, my goal is to make it so that when the user closes the app it saves their collection string, and then loads it when the app is loaded next time.
Currently, it crashes when it tries to load the text file.
Any help would be greatly appreciated!
With an ArrayIndexOutOfBoundsException it means you are trying to access an invalid index in the array.
Basically you are calling owned[i] at some point, where i is not a valid index. (Greater than the size of owned)
In the last section of the code I print what the Reader gives me. But its just bogus, where did I go wrong?
public static void read_impl(File file, String targetFile) {
// Create zipfile input stream
FileInputStream stream = new FileInputStream(file);
ZipInputStream zipFile = new ZipInputStream(new BufferedInputStream(stream));
// Im looking for a specific file/entry
while (!zipFile.getNextEntry().getName().equals(targetFile)) {
zipFile.getNextEntry();
}
// Next step in api requires a reader
// The target file is a UTF-16 encoded text file
InputStreamReader reader = new InputStreamReader(zipFile, Charset.forName("UTF-16"));
// I cant make sense of what this print
char buf[] = new char[1];
while (reader.read(buf, 0, 1) != -1) {
System.out.print(buf);
}
}
I'd guess that where you went wrong was believing that the file was UTF-16 encoded.
Can you show a few initial byte values if you don't decode them?
Your use of a char array is a bit pointless, though at first glance it should work. Try this instead:
int c;
while ((c = reader.read()) != -1) {
System.out.print((char)c);
}
If that does not work either, then perhaps you got the wrong file, or the file does not contain what you think it does, or the console can't display the characters it contains.
I have a text file that I want to edit using Java. It has many thousands of lines. I basically want to iterate through the lines and change/edit/delete some text. This will need to happen quite often.
From the solutions I saw on other sites, the general approach seems to be:
Open the existing file using a BufferedReader
Read each line, make modifications to each line, and add it to a StringBuilder
Once all the text has been read and modified, write the contents of the StringBuilder to a new file
Replace the old file with the new file
This solution seems slightly "hacky" to me, especially if I have thousands of lines in my text file.
Anybody know of a better solution?
I haven't done this in Java recently, but writing an entire file into memory seems like a bad idea.
The best idea that I can come up with is open a temporary file in writing mode at the same time, and for each line, read it, modify if necessary, then write into the temporary file. At the end, delete the original and rename the temporary file.
If you have modify permissions on the file system, you probably also have deleting and renaming permissions.
if the file is just a few thousand lines you should be able to read the entire file in one read and convert that to a String.
You can use apache IOUtils which has method like the following.
public static String readFile(String filename) throws IOException {
File file = new File(filename);
int len = (int) file.length();
byte[] bytes = new byte[len];
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
assert len == fis.read(bytes);
} catch (IOException e) {
close(fis);
throw e;
}
return new String(bytes, "UTF-8");
}
public static void writeFile(String filename, String text) throws IOException {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(filename);
fos.write(text.getBytes("UTF-8"));
} catch (IOException e) {
close(fos);
throw e;
}
}
public static void close(Closeable closeable) {
try {
closeable.close();
} catch(IOException ignored) {
}
}
You can use RandomAccessFile in Java to modify the file on one condition:
The size of each line has to be fixed otherwise, when new string is written back, it might override the string in the next line.
Therefore, in my example, I set the line length as 100 and padding with space string when creating the file and writing back to the file.
So in order to allow update, you need to set the length of line a little larger than the longest length of the line in this file.
public class RandomAccessFileUtil {
public static final long RECORD_LENGTH = 100;
public static final String EMPTY_STRING = " ";
public static final String CRLF = "\n";
public static final String PATHNAME = "/home/mjiang/JM/mahtew.txt";
/**
* one two three
Text to be appended with
five six seven
eight nine ten
*
*
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException
{
String starPrefix = "Text to be appended with";
String replacedString = "new text has been appended";
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line = "";
while((line = file.readLine()) != null)
{
if(line.startsWith(starPrefix))
{
file.seek(file.getFilePointer() - RECORD_LENGTH - 1);
file.writeBytes(replacedString);
}
}
}
public static void createFile() throws IOException
{
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line1 = "one two three";
String line2 = "Text to be appended with";
String line3 = "five six seven";
String line4 = "eight nine ten";
file.writeBytes(paddingRight(line1));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line2));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line3));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line4));
file.writeBytes(CRLF);
file.close();
System.out.println(String.format("File is created in [%s]", PATHNAME));
}
public static String paddingRight(String source)
{
StringBuilder result = new StringBuilder(100);
if(source != null)
{
result.append(source);
for (int i = 0; i < RECORD_LENGTH - source.length(); i++)
{
result.append(EMPTY_STRING);
}
}
return result.toString();
}
}
If the file is large, you might want to use a FileStream for output, but that seems pretty much like it is the simplest process to do what you're asking (and without more specificity i.e. on what types of changes / edits / deletions you're trying to do, it's impossible to determine what more complicated way might work).
No reason to buffer the entire file.
Simply write each line as your read it, insert lines when necessary, delete lines when necessary, replace lines when necessary.
Fundamentally, you will not get around having to recreate the file wholesale, especially if it's just a text file.
What kind of data is it? Do you control the format of the file?
If the file contains name/value pairs (or similar), you could have some luck with Properties, or perhaps cobbling together something using a flat file JDBC driver.
Alternatively, have you considered not writing the data so often? Operating on an in-memory copy of your file should be relatively trivial. If there are no external resources which need real time updates of the file, then there is no need to go to disk every time you want to make a modification. You can run a scheduled task to write periodic updates to disk if you are worried about data backup.
In general you cannot edit the file in place; it's simply a very long sequence of characters, which happens to include newline characters. You could edit in place if your changes don't change the number of characters in each line.
Can't you use regular expressions, if you know what you want to change ? Jakarta Regexp should probably do the trick.
Although this question was a time ago posted, I think it is good to put my answer here.
I think that the best approach is to use FileChannel from java.nio.channels package in this scenario. But this, only if you need to have a good performance! You would need to get a FileChannel via a RandomAccessFile, like this:
java.nio.channels.FileChannel channel = new java.io.RandomAccessFile("/my/fyle/path", "rw").getChannel();
After this, you need a to create a ByteBuffer where you will read from the FileChannel.
this looks something like this:
java.nio.ByteBuffer inBuffer = java.nio.ByteBuffer.allocate(100);
int pos = 0;
int aux = 0;
StringBuilder sb = new StringBuilder();
while (pos != -1) {
aux = channel.read(inBuffer, pos);
pos = (aux != -1) ? pos + aux : -1;
b = inBuffer.array();
sb.delete(0, sb.length());
for (int i = 0; i < b.length; ++i) {
sb.append((char)b[i]);
}
//here you can do your stuff on sb
inBuffer = ByteBuffer.allocate(100);
}
Hope that my answer will help you!
I think, FileOutputStream.getFileChannel() will help a lot, see FileChannel api
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html
private static void modifyFile(String filePath, String oldString, String newString) {
File fileToBeModified = new File(filePath);
StringBuilder oldContent = new StringBuilder();
try (BufferedReader reader = new BufferedReader(new FileReader(fileToBeModified))) {
String line = reader.readLine();
while (line != null) {
oldContent.append(line).append(System.lineSeparator());
line = reader.readLine();
}
String content = oldContent.toString();
String newContent = content.replaceAll(oldString, newString);
try (FileWriter writer = new FileWriter(fileToBeModified)) {
writer.write(newContent);
}
} catch (IOException e) {
e.printStackTrace();
}
}
You can change the txt file to java by saving on clicking "Save As" and saving *.java extension.