Develop Reference

Memory Consumption by Java Applet

In my applet I have GET call to download file from a remote location. When I am trying to download some large file of around 13MB, then my Applet memory consumption is increasing more than 50MB. I am using the below code to get my memory consumption:
public static long getMemoryUsage()
{
long memory = 0;
// Get the Java runtime
Runtime runtime = Runtime.getRuntime();
memory = runtime.totalMemory() - runtime.freeMemory();
return memory;
}
Code for my get call is
public void getFiles(String filePath, long fileSize)throws MyException
{
InputStream objInputStream = null;
HttpURLConnection conn = null;
BufferedReader br = null;
try
{
URL fileUrl=new URL(filePath);
final String strAPICall=fileUrl.getPath();
final String strHost="some.test.com";
final int iPort=1000;
URL url = null;
url = new java.net.URL
( "https",
strHost, iPort , "/" + strAPICall,
new myHandler() );
conn = (HttpURLConnection)new HttpsURLConn(url);
conn.setRequestMethod("GET");
conn.connect();
if (conn.getResponseCode() != 200) {
objInputStream=conn.getInputStream();
br = new BufferedReader(new InputStreamReader(
(objInputStream)));
String output;
while ((output = br.readLine()) != null) {
System.out.println(output);
}
throw new MyException("Bad response from server",
MyError.BAD_RESPONSE_ERROR);
}
else
{
notifyProgressToObservers(0);
System.out.println("conn.getResponseCode()"+conn.getResponseCode());
System.out.println("conn.getResponseMessage()"+conn.getResponseMessage());
objInputStream = conn.getInputStream();
int count=objInputStream.available();
System.out.println("Stream size: "+count);
System.out.println("fileSize size: "+fileSize);
byte []downloadedData = getBytesFromInputStream
(objInputStream, count,fileSize);
notifyChunkToObservers(downloadedData);
notifyIndivisualFileEndToObservers(true, null);
}
}
catch (MyException pm)
{
throw new MyException
(pm, MyError.CONNECTION_TIMEOUT);
}
catch (IOException pm)
{
throw new MyException
(pm, MyError.CONNECTION_TIMEOUT);
}
catch (Exception e)
{
notifyIndivisualFileEndToObservers(false,new MyException(e.toString()));
}
finally
{
System.out.println("Closing all the streams after getting file");
if(conn !=null)
{
try
{
conn.disconnect();
}
catch(Exception e)
{
}
}
if(objInputStream != null)
{
try {
objInputStream.close();
} catch (IOException e) {
}
}
if (br != null)
{
try {
br.close();
} catch (IOException e) {
}
}
}
}
In the above method, I tried putting the log for memory consumption after each line and found that after conn.connect();, the memory consumption of applet increases by atleast 50MB even though the file I am trying to download is only 13MB.
Is there any memory leak anywhere?
EDIT: Added Implementation for getBytesFromInputStream()
public byte[] getBytesFromInputStream(InputStream is, int len, long fileSize)
throws IOException
{
byte[] readBytes= new byte[8192];
ByteArrayOutputStream getBytes= new ByteArrayOutputStream();
int numRead = 0;
while ((numRead = is.read(readBytes)) != -1) {
getBytes.write(readBytes, 0, numRead);
}
return getBytes.toByteArray();
}

it's because of this line:
byte []downloadedData = getBytesFromInputStream(objInputStream, count,fileSize);
here you are reading the complete amount of bytes of file into the heap. After that you need to track down what happens with this array. Maybe you are copying it somewhere and the GC needs some time to kick in even if you do not use the reference to the object anymore.
Large files should never be read completly to memory, but rather streamed directly to some processor of the data.

The only way to optimize getBytesFromInputStream() is if you know beforehand exactly how many by bytes there are to read. Then you allocate a byte[] of the required size, and read from the input directly into the byte[]. For example:
byte[] buffer = new byte[len];
int pos = 0;
while (pos < len) {
int nosRead = is.read(buffer, pos, len - pos);
if (nosRead == -1) {
throw new IOException("incomplete response");
}
pos += nosRead;
}
return buffer;
(For more information, read the javadoc.)
Unfortunately, your (apparent) attempt at getting the size is incorrect.
int count = objInputStream.available();
This doesn't return the total number of bytes that can be read from the stream. It returns the number of bytes that can be read right now without the possibility of blocking.
If the server is setting the Content-Length header in the response, then you could use that; call getContentLength() (or getContentLengthLong() in other use-cases) once you have the response. But be prepared for the case where that gives you -1.

How to write a binary file from a String and retrieve it again to a String?

I have a string and want to persist it into a file and be able to retrieve it again into a String.
Something is wrong with my code because It's supposing that I must write something binary non readable but when i Open the file I can read this:
Original string:
[{"name":"asdasd","etName":"111","members":[]}]
Stored string in binary file:
[ { " n a m e " : " a s d a s d " , " e t N a m e " : " 1 1 1 " , " m e m b e r s " : [ ] } ]
I detect two problems:
Is not stored in binary! I can read it. It's supposed to be a confused binary text unreadable but I can read it.
When i retrieve it it's being retrieved with that strange space between the characters. So it doesn't works.
This is my code for storing the string:
public static void storeStringInBinary(String string, String path) {
DataOutputStream os = null;
try {
os = new DataOutputStream(new FileOutputStream(path));
os.writeChars(string);
os.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (os != null) {
try {
os.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
And this is my code for reading it from binary to a String:
public static String retrieveStringFromBinary(String file) {
String string = null;
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader (file));
String line = null;
StringBuilder stringBuilder = new StringBuilder();
while((line = reader.readLine()) != null) {
stringBuilder.append(line);
}
return stringBuilder.toString();
} catch (Exception e){
e.printStackTrace();
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return string;
}

Firstly, there isn't really a distinction between a text file and a binary file. A text file is just a file who's content falls in the range of byte values that correspond to characters.
If you want to encrypt the content of the file so it is unreadable just by catting the file then you will need to choose an appropriate encryption method.
Secondly Mixing Readers/Writers and Streams in Java is never a good idea, pick one style and stick to it.
The problem with your function that saves the string to a file is that you are using the writeChars() method, which from the doc does the following:
Writes a char to the underlying output stream as a 2-byte value, high byte first. If no exception is thrown, the counter written is incremented by 2.
Since your string is made up of single byte characters this is leading to the padding of your string with null bytes, which are being converted to spaces when read back in. If you change this to writeBytes() then you should get output without the extra null byte.
The null byte will also stop your read function working as the readLine() function will return null on it's first call due to the leading 0x00 in the file.

Try this out:
public static void storeStringInBinary(String string, String path) {
try(ObjectOutputStream os = new ObjectOutputStream(new FileOutputStream(path))) {
os.writeObject(string);
} catch (IOException e) {
e.printStackTrace();
}
}
public static String retrieveStringFromBinary(String file) {
String string = null;
try (ObjectInputStream reader = new ObjectInputStream(new FileInputStream(file))){
string = (String) reader.readObject();
} catch (ClassNotFoundException | IOException e) {
e.printStackTrace();
}
return string;
}

OutOfMemoryError while JSON parsing in android

I am using the below code to parse the JSON String fetched from Web, (30,000 records)
DefaultHttpClient httpclient = new DefaultHttpClient(new BasicHttpParams());
HttpPost httppost = new HttpPost(params[0]);
httppost.setHeader("Content-type", "application/json");
InputStream inputStream = null;
String result = null;
HttpResponse response = null;
try {
response = httpclient.execute(httppost);
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
HttpEntity entity = response.getEntity();
try {
inputStream = entity.getContent();
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"),8);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
}
result = sb.toString();
I am getting the OutofMemory error in the below code
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
How to get rid of this error.This error does occur when the json string is very huge as it contains data of about 30,000 records.
Any help in this regard is highly appreciated..

Android imposes a memory cap limit (of 16 MB in almost all phones, with some newer tablets have more) for each and every application. Application should make sure they maintain their live memory limit below that level.
So we can't hold a big string, say over 1MB, in full sometimes since the total live memory usage of applicaiton may exceed that limit. Remember, the total memory usage includes all objects (including UI elements) we allocated in our app.
So your only solution is to use a Streaming JSON parser, which takes data as it comes. That is you should not hold on full string in a String object. One option is to use Jackson JSON parser.
EDIT : Android now support JSONReader from API level 11. Never used it, but it seems the way to go..

If data file is too large, you cannot read it all to memory.
Read a line and then write it to a native file. Do not use a StringBuilder to hold all data in memory.

Try to import your data in chuncks, like 1000 records each time. Hopefully you will not experince this issue.

I solved this problem with this library.
There is a very good tutorial here.
With this you will bypass converting entity.getContent() to String and that will solve your problem.
InputStream inputStream = entity.getContent();
JsonReader reader = Json.createReader(inputStream);
JsonObject jsonObject = reader.readObject();
return jsonObject;

how to get a value by key from an json (or xml) string?

In the android app I get an xml or json string returned, However, I cant seem to figure out any way on how to get an value from the string in any way by entering an key.
In PHP you just use something like $myArray['parent']['child'] but I have no clue on how this works in java.
Any idea's would be greatly appreciated! (an example for both XML and JSON even more ;) )

Here's what I would do:
locate an XML/JSON library (there's tons) (google-gson for json)
read the documentation to find a parse method ((new JsonParser()).parse(text))
read the documentation to find out what the return value is (JsonElement)
decide what you want to do with the parsed data (myJsonObj.get(...))
write the code

public class parsingjsontest2 extends Activity {
/** Called when the activity is first created. */
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(main);
String str = connect("http://rentopoly.com/ajax.php?query=Bo"));
System.out.println("String::"+str);
}
}
private String connect(String url)
{
// Create the httpclient
HttpClient httpclient = new DefaultHttpClient();
// Prepare a request object
HttpGet httpget = new HttpGet(url);
// Execute the request
HttpResponse response;
// return string
String returnString = null;
try {
// Open the webpage.
response = httpclient.execute(httpget);
if(response.getStatusLine().getStatusCode() == 200){
// Connection was established. Get the content.
HttpEntity entity = response.getEntity();
// If the response does not enclose an entity, there is no need
// to worry about connection release
if (entity != null) {
// A Simple JSON Response Read
InputStream instream = entity.getContent();
// Load the requested page converted to a string into a JSONObject.
JSONObject myAwway = new JSONObject(convertStreamToString(instream));
// Get the query value'
String query = myAwway.getString("query");
**// Make array of the suggestions
JSONArray suggestions = myAwway.getJSONArray("suggestions");
// Build the return string.
returnString = "Found: " + suggestions.length() + " locations for " + query;
for (int i = 0; i < suggestions.length(); i++) {
returnString += "\n\t" + suggestions.getString(i);
}
// Cose the stream.
instream.close();
}
}
else {
// code here for a response othet than 200. A response 200 means the webpage was ok
// Other codes include 404 - not found, 301 - redirect etc...
// Display the response line.
returnString = "Unable to load page - " + response.getStatusLine();
}
}
catch (IOException ex) {
// thrown by line 80 - getContent();
// Connection was not established
returnString = "Connection failed; " + ex.getMessage();
}
catch (JSONException ex){
// JSON errors
returnString = "JSON failed; " + ex.getMessage();
}
return returnString;
}
private static String convertStreamToString(InputStream is) {
/*
* To convert the InputStream to String we use the BufferedReader.readLine()
* method. We iterate until the BufferedReader return null which means
* there's no more data to read. Each line will appended to a StringBuilder
* and returned as String.
*/
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
}

As you didn't specify what kind of xml you are trying to read, I'm answering based on what I know.
In Android, if you were talking about the layout and strings.xml files, you use a dot (.) operator, like R.string.appname.
Please post more details about your specific problem, if this is not what you were looking for.

sockets long reading data

This is my code:
private String receiveData(String sjson) {
Log.i(TAG,"send request: " + sjson);
String jstr="";
try {
OutputStream out = s.getOutputStream();
out.write(sjson.getBytes());
out.flush();
//out.close();
Log.v(TAG,"sended data");
BufferedReader input = new BufferedReader(new InputStreamReader(s.getInputStream()));
char[] cbuf = new char[1];
input.read(cbuf);
String size = new String(cbuf);
while (input.read(cbuf) != 0) {
if((new String(cbuf)).equals("{") == true)
break;
size = size + new String(cbuf);
}
char[] jbuf = new char[Integer.valueOf(size)];
input.read(jbuf);
jstr = "{" + new String(jbuf);
}catch (Exception e) {
Log.e(TAG,e.toString());
}
Log.d(TAG,"responce: " + jstr);
return jstr;
}
public void connectSocket() {
Log.v(TAG,"connecting Socket: "+URL+":"+PORT);
try {
s = new Socket(URL, PORT);
Log.v(TAG,"connect Socket!");
ERROR_CODE = 0;
}catch (Exception e) {
Log.e(TAG,e.toString());
ERROR_CODE = ERROR_SOCKET_CONNECT_SUCCESSFULL;
}
Log.e(TAG,getErrorMsg(ERROR_CODE));
}
public void closeSocket() {
Log.v(TAG,"closeSocket");
try {
s.close();
}catch (Exception e) {
Log.e(TAG,e.toString());
}
}
At server the answer is less than a second. At the client it passes 1 minute before reading data.
Apps stoped at input.read(cbuf); waiting for answer.
Logs:
05-23 06:35:17.540: VERBOSE/Utilits(358): Auth: 77.221.129.100:10598
05-23 06:35:17.660: INFO/Utilits(358): send request: 0119{"data":{"password":"12345","imei":"000000000000001"},"method":"login"}
05-23 06:36:17.909: DEBUG/Utilits(358): responce: {"response":{"success":true,"user":{"id":"6","properties":{"auto":"model":"audi","color":"ffff","number":"td123r"}},"is_driver":"1"}}}
Why does it take so long to read an answer?

What on earth do you expect that method to do? There are bugs in it, and it does things that it should do.
You should specify encoding/charset when you create the InputStreamReader
Why do you read character by character from start to "{"
Why do you create a string for each character that you read before you hit "{"
Why do you append strings in a loop? Use a StringBuilder if you must append.
input.read returns an integer that says how many bytes/character that you have received
It's never guaranteed that it will fill the buffer. So you might not get all data.
Why aren't you closing resources?
.. and now to why it might be slow. Is the server flushing the data? If not, make sure that the server is flushing the data.