i am getting out of memory error in asynctask which loop to stringbuilder . My target for using this to download image from server and store inside my sd card.My code as below :
HttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
HttpPost httppost = new HttpPost(severPath);
httppost.setEntity(params[0]);
System.out.println("executing request " + httppost.getRequestLine());
HttpResponse response = null;
try {
response = httpclient.execute(httppost);
} catch (ClientProtocolException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
} catch (IOException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
}
String output;
System.out.println("Output from Server .... \n");
BufferedReader br = null;
try {
br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
} catch (IllegalStateException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
} catch (IOException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
}
OutputStreamWriter outputStreamWriter = null;
try {
outputStreamWriter = new OutputStreamWriter(context.openFileOutput("LargeImages.txt", context.MODE_PRIVATE));
} catch (FileNotFoundException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
}
int i = 0;
StringBuilder builder = new StringBuilder();
String Result = "";
try {
for (String line = null; (line = br.readLine()) != null ; ) {
builder.append(line.toString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
outputStreamWriter.close();
i am getting out of memory allocation error. please help. i try many method but also not getting the right.
if you are downloading an image, then you should not use Reader/Writer/StringBuilder to store it's content. Because the file is binary content will be scrambled because of the character encoding used by Reader/Writer classes.
Try using InputStream/OutputStream and store the content directly to sdcard without storing it in memory.
Try out the below code:
InputStream in = response.getEntity().getContent();
OutputStream out = context.openFileOutput("LargeImages.txt", context.MODE_PRIVATE);
byte b[] = new byte[4096];
int i;
while ((i = in.read(b)) >= 0) {
out.write(b, 0, i);
}
There may be two problems.
The first - the cycle for (String line = null; (line = br.readLine()) != null ; ) is not terminated properly. Try to find it out by opening a small file(e.g. with 10 lines total).
The second - it's actually a memory insufficient case. Probably it's not the best idea to get image via strings as images may be very heavy and creating a plenty of Strings causes natural memory error. Try to find another approach.
I don't see code that is actually writing to the output stream. Shouldn't there be a line before the close, that is like outputStreamWriter.print(builder)?
About your question. Instead of collecting the whole data in memory in a StringBuilder and than write it at once, you should write directly each line you get within your for-loop. You don't need the StringBuilder at all. Here's a code snippet:
try {
for (String line = br.readLine(); line != null; line = br.readLine()) {
outputStreamWriter.append(line);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
Three more remarks:
When you get an Exception you should also stop the action, e.g. return from your method. Your code above would print the Stacktrace (which is definitely helpful) but would then continue, which would be not so helpful. Just add return after each printstackTrace.
There's still a chance that one line is too long for memory, but the risk is minimized.
Is the data you download binary image or text? You name it image but you download text. Please be aware that there's a difference between bytes and characters (encoded with character set) and stay within what you actually receive.
Related
I'm learning about Memory Mapped files in java.
I would like to know how to write/read to a MappedByteBuffer.
Here's the code I'm using for writing to MappedByteBuffer.
private static void write(String strFilePath) {
File fl = new File(strFilePath);
FileChannel fChannel = null;
RandomAccessFile rf = null;
MappedByteBuffer mBBuffer = null;
try {
rf = new RandomAccessFile(fl, "rw");
fChannel = rf.getChannel();
mBBuffer = fChannel.map(FileChannel.MapMode.READ_WRITE, 0, 1024 );
for(int i =0;i<10030;i++) {
if(i == mBBuffer.limit()) {
mBBuffer = fChannel.map(FileChannel.MapMode.READ_WRITE,i+1,1024);
}
mBBuffer.put((byte)'a');
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
if(fChannel != null) {
fChannel.close();
}
if(rf != null) {
rf.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I get BufferOverflow Exception while trying to do it.
How do I increase the size of the MappedByteBuffer after reaching the limit while writing, if I do not the know the size of the contents I will write to the file in advance?
For reading, let's take this case,
I create an MappedByteBuffer with an intial buffer size, and if reached the end of that initial size, how do i map a different portion of the file
More generally, I have a file , with byte offsets from one part of the file to another, When I read an offset (y) at a point (x), I would like to jump to (y) from (x). How do I do it?
Thanks in advance for helping me out.
[Update]Problem is different, nothing about csv file format. Question
would be "While using File Writer to write a csv file last few records
are missing.
In my java application i need to append more than 65535 rows in a csv file. but it only writes 65535 rows in a sheet. I haven't used any libraries. some final records missing. how to resolve this..........
public void writeSubmission(){
try {
writer = new FileWriter("res/sample.csv");
writer.append("PhraseId");
writer.append(',');
writer.append("Sentiment");
writer.append('\n');
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public void readTestData(){
String path="res/test.tsv";
Calculation cal=new Calculation();
int counter=0;
try {
BufferedReader bReader = new BufferedReader(new FileReader(path));
String line;
writeSubmission();
bReader.readLine();
while ((line = bReader.readLine()) != null) {
String datavalue[] = line.split("\t");
writer.append(datavalue[0]);
writer.append(',');
try {
double value=cal.calculate(datavalue[2]);
System.out.println(value);
String val;
if(value<-0.4)
{
val="0";
}
else if(value>-0.4 && value<-0.1)
{
val="1";
}
else if(value>-0.1 && value<+0.1)
{
val="2";
}
else if(value>+0.1 && value<+0.40)
{
val="3";
}
else{
val="4";
}
counter++;
writer.append(val);
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println(e);
}
writer.append('\n');
}
System.out.println(counter);
bReader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
System.out.println(e);
} catch (IOException e){
e.printStackTrace();
System.out.println(e);
}
}
I think your issue is probably that the tool you're opening up the CSVs with on the other end doesn't want more than 65535 rows, not that Java's doing anything wrong. It's a bug on the other end, not your Java code, almost certainly. (FileWriter wouldn't care at all about 65535 lines, for example.)
If you're using Excel 2003, for example, you'd see this issue: How to get around 64k row limit in Excel
Finally i fixed the error.File Writer should be flush after used in the code push existing stream. OR should be close the writer it automatically flush and close
The flush method flushes the output stream and forces any buffered
output bytes to be written out. The general contract of flush is that
calling it is an indication that, if any bytes previously written have
been buffered by the implementation of the output stream, such bytes
should immediately be written to their intended destination.
writer = new FileWriter("res/sample.csv");
writer.flush();
writer.close();
I've a web service that prints the following text.
[{"packid":"p101","title":"صفته 1","description":"شسیب: 1\r\nثق س: 50","linkfuntext":"funtext","linkshortstory":"short","linkfunpic":"pic","linkringtone":"ring","linkfungif":"gif","linkwallpaper":"wall","price":"500","buyid":"pack.fun.1","buyed":""},{"packid":"p102","title":"بسته صدا","description":" متن ها: 50\r\nصداها: 120\r\nتصاویر: 100\r\nتصاویر متحرک: 50\r\nداستان کوتاه: 20","linkfuntext":"","linkshortstory":"","linkfunpic":"","linkringtone":"","linkfungif":"","linkwallpaper":"","price":"1200","buyid":"fun.pack.2","buyed":""}]
When I try to read it in java I receive it in the following format
[{"packid":"p101","title":"صفته 1","description":"شسیب: 1\r\nثق س: 50","linkfuntext":"funtext","linkshortstory":"short","linkfunpic":"pic","linkringtone":"ring","linkfungif":"gif","linkwallpaper":"wall","price":"500","buyid":"pack.fun.1","buyed":""},{"packid":"p102","title":"بسته صدا","description":" متن ها: 50\r\nصداها: 120\r\nتصاویر: 100\r\nتصاویر متحرک: 50\r\nداستان کوتاه: 20","linkfuntext":"","linkshortstory":"","linkfunpic":"","linkringtone":"","linkfungif":"","linkwallpaper":"","price":"1200","buyid":"fun.pack.2","buyed":""}]
I've tried changing the character set to UTF-8 as well as ISO-8859-6 but still no luck. When I print the text on console it is printed correctly which means there is no issue in character set of eclipse or console. Also I've tried changing the character set of string that is storing the text, but same issue.
String serverOutput = new String(TEXT_FROM_SERVER.getBytes(), "UTF-8");
Here is my code that gets output from web service
HttpEntity entity = response.getEntity();
InputStream is = entity.getContent();
String serverOutput = convertStreamToString(is);
private String convertStreamToString(InputStream is) {
Reader rd = null;
BufferedReader reader = null;
try {
rd = new InputStreamReader(is,"UTF-8");
} catch (UnsupportedEncodingException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
reader = new BufferedReader(rd);
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append((line + "\n"));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
Any kind of help will be greatly appreciated. Thanks
you need to unescape those HTML characters, and you can do so with a method from Apache Commons Lang called unescapeHtml. More info here.
Example:
String afterDecoding = StringEscapeUtils.unescapeHtml(beforeDecoding);
I want to know the offset of every line present in a text file.
For now I have tried,
path=FileSystems.getDefault().getPath(".",filename);
br=Files.newBufferedReader(path_doc_title_index_path, Charset.defaultCharset());
int offset=0; //offset of first line.
String strline=br.readline();
offset+=strline.length()+1; //offset of second line
In this way I can loop through entire file to know offset of begining of lines in entire text file. But if I use RandomAccessFile to seek through file and access a line using offset calulated by above method then I found myself in the middle of some line. That is it seems that offset are not correct.
What's wrong? Is this method incorrect to calculate offset? Any better and fast methods please?
Your code will only work for ASCII encoded text. Since some characters need more than one byte, you have to change following line
offset += strline.length() + 1;
to
offset += strline.getBytes(Charset.defaultCharset()).length + 1;
As stated in my comments below your question, you have to specifiy the correct encoding of your file. E.g. Charset.forName("UTF-8") here and also where you initialize your BufferedReader.
Apparently, this gives me the expected result. In the following program I print out each line of a file through a set of offsets that I collect through the BufferedReader. Is this your case?
public static void main(String[] args) {
File readFile = new File("/your/file/here");
BufferedReader reader = null;
try
{
reader = new BufferedReader( new FileReader(readFile) );
}
catch (IOException ioe)
{
System.err.println("Error: " + ioe.getMessage());
}
List<Integer> offsets=new ArrayList<Integer>(); //offset of first line.
String strline;
try {
strline = reader.readLine();
while(strline!=null){
offsets.add(strline.length()+System.getProperty("line.separator").length()); //offset of second line
strline = reader.readLine();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
RandomAccessFile raf = new RandomAccessFile(readFile, "rw");
for(Integer offset : offsets){
try {
raf.seek(offset);
System.out.println(raf.readLine());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
I am using the below code to parse the JSON String fetched from Web, (30,000 records)
DefaultHttpClient httpclient = new DefaultHttpClient(new BasicHttpParams());
HttpPost httppost = new HttpPost(params[0]);
httppost.setHeader("Content-type", "application/json");
InputStream inputStream = null;
String result = null;
HttpResponse response = null;
try {
response = httpclient.execute(httppost);
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
HttpEntity entity = response.getEntity();
try {
inputStream = entity.getContent();
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"),8);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
}
result = sb.toString();
I am getting the OutofMemory error in the below code
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
How to get rid of this error.This error does occur when the json string is very huge as it contains data of about 30,000 records.
Any help in this regard is highly appreciated..
Android imposes a memory cap limit (of 16 MB in almost all phones, with some newer tablets have more) for each and every application. Application should make sure they maintain their live memory limit below that level.
So we can't hold a big string, say over 1MB, in full sometimes since the total live memory usage of applicaiton may exceed that limit. Remember, the total memory usage includes all objects (including UI elements) we allocated in our app.
So your only solution is to use a Streaming JSON parser, which takes data as it comes. That is you should not hold on full string in a String object. One option is to use Jackson JSON parser.
EDIT : Android now support JSONReader from API level 11. Never used it, but it seems the way to go..
If data file is too large, you cannot read it all to memory.
Read a line and then write it to a native file. Do not use a StringBuilder to hold all data in memory.
Try to import your data in chuncks, like 1000 records each time. Hopefully you will not experince this issue.
I solved this problem with this library.
There is a very good tutorial here.
With this you will bypass converting entity.getContent() to String and that will solve your problem.
InputStream inputStream = entity.getContent();
JsonReader reader = Json.createReader(inputStream);
JsonObject jsonObject = reader.readObject();
return jsonObject;