Develop Reference

Java Memory Leak in HTTP requests

I have a Java application that is runs constantly. This application makes HTTP requests to a cloud server. The problem is that at each request the memory consumption increases until it reaches the point that the machine complete freezes. I isolated parts of the code and I'm sure the problem is with this code block making this http requests. Analyzing the JVM numbers, via prometheus / Grafana, I see that the use of non-heap memory (codecache and metaspace) are constantly increasing, as shown here
In the image above, whenever there is a drop in the line, it is when 98% of memory consumption reached, and Monit kills the app.
The method that is causing this memory consumption, is below (it is executed approx. 300 times until it exhausts a little more than 1.5GB of available memory in the initialization).
public AbstractRestReponse send(RestRequest request){
BufferedReader in = null;
OutputStream fout = null;
URLConnection conn = null;
InputStreamReader inputStreamReader = null;
String result = "";
try {
MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
for (String key : request.getParams().keySet()) {
String value = (String) request.getParams().get(key);
// System.out.println(key + " = " + value);
mb.addTextBody(key, value);
}
if (request.getFile() != null) {
mb.addBinaryBody("file", request.getFile());
}
org.apache.http.HttpEntity e = mb.build();
conn = new URL(request.getUrl()).openConnection();
conn.setDoOutput(true);
conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
fout = conn.getOutputStream();
e.writeTo(fout);// write multi part data...
inputStreamReader = new InputStreamReader(conn.getInputStream());
in = new BufferedReader(inputStreamReader);
String line;
while ((line = in.readLine()) != null) {
result += line;
}
String text = result.toString();
return objectMapper.readValue(text, FacialApiResult.class);
}catch (Exception e) {
e.printStackTrace();
return null;
}finally {
try {
inputStreamReader.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
conn.getInputStream().close();
} catch (IOException e) {
e.printStackTrace();
}
try {
fout.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

((HttpURLConnection)conn).disconnect() comes to mind. Also String concatenation is time and memory exhaustive. And there was a minor bug in dropping newlines.
NullPointerExceptions may arise in the finally block when an open was not reached due to an exception. But you should have checked that.
public AbstractRestReponse send(RestRequest request) {
URLConnection conn = null;
try {
MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
for (String key : request.getParams().keySet()) {
String value = (String) request.getParams().get(key);
mb.addTextBody(key, value);
}
if (request.getFile() != null) {
mb.addBinaryBody("file", request.getFile());
}
org.apache.http.HttpEntity e = mb.build();
conn = new URL(request.getUrl()).openConnection();
conn.setDoOutput(true);
conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
try (fout = conn.getOutputStream()){
e.writeTo(fout);// write multi part data...
}
StringBuilder resullt = new StringBuilder(2048);
try (BufferedReader in = new InputStreamReader(conn.getInputStream(),
StandardCharsets.UTF_8)) { // Charset
String line;
while ((line = in.readLine()) != null) {
result.append(line).append('\n'); // Newline
}
}
String text = result.toString();
return objectMapper.readValue(text, FacialApiResult.class);
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if (conn != null) {
try {
if (conn instanceof HttpURLConnection) {
((HttpURLConnection) conn).disconnect();
}
} catch (IOException e) {
e.printStackTrace(); //Better logger
}
}
}
return null;
}
I explicitly defined the charset (UTF-8 might be wrong) - momentarily it is the server's default.
Used a StringBuilder, and added the missing newline, which might have lead to wrong parsing.
Try-with-resources for auto-closing, and a bit earlier. Hopefully this does not break anything.
Disconnecting the connection when it is an HttpURLConnection. Mind the instanceof which might play a role in unit tests mocking.

You seems to have handled all possible closing part in the finally block. Anyway it's better to use try-with resources to safely close all Closeable objects, if your application is running on Java 7+. That may isolate the problem further if it doesn't fix.

Read a resource from URL and return directly those bytes as response of REST request, with no memory storing with Java 7 and spring MVC 3.2

I have a REST endpoint that has to be accessed to retrieve a resource (image, document, ...).
#RequestMapping(value = "/attachement", method = RequestMethod.GET)
#ResponseBody
public Object getTrademarkAttachement(HttpServletResponse response, HttpServletRequest request) {
//TODO : Retrieve bytes from microservice url
//TODO : Send bytes to frontend page
}
For retrieving this document, I want to do it via streaming . I don't want to store in memory the info . I want to , as I get the info, send the bytes as a response . My version of spring MVC is Spring MVC 3.2 and my version of java is java 7 . Is it possible to achieve this ? could you give any clue to start investigating ? . I know I'm giving little details about implementation but I'm starting with this point and I would want to get some ideas from you .
EDIT 1 :
I have achieved half of the problem . Retrieving different blocks of the url . I have used the following code
#Override
public byte[] getTrademarkAttachement() {
String urlSample = "http://testUrl.com";
HttpURLConnection httpConn = null;
String line = null;
try {
httpConn = (HttpURLConnection) new URL(urlSample).openConnection();
InputStream ins = httpConn.getInputStream();
BufferedReader is = new BufferedReader(new InputStreamReader(ins));
while ((line = is.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
httpConn.disconnect();
}
return null;
}
Being able to have access to the inputstream , the part that is left is returning each of this lines that I'm reading , so I can stream the response . I have to look for a method in spring MVC that gives a partial response .

Since you can get the InputStream, you should be able to return an OutputStream as a response to the request. Take a look at this (https://stackoverflow.com/a/27742486/):
#RequestMapping(value = "/attachement", method = RequestMethod.GET)
#ResponseBody
public void getAttachment(OutputStream out) {
InputStream in = ; // Set this to the InputStream from HTTP as your provided example
byte[] buffer = new byte[1024]; // You will need a small buffer mem though
int len;
while ((len = in.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
in.close();
out.flush();
}

Ok , I have solved my problem . I attach the solution . Maybe it's useful to anybody.
Controller
#RequestMapping(value="/eutm/{trademarkId}/snapshots/{historyId}/attachements/{attachementId}", method = RequestMethod.GET)
#ResponseBody
public void getTrademarkAttachement(HttpServletResponse response, #PathVariable String trademarkId, #PathVariable String historyId, #PathVariable String attachementId) {
try {
registerService.getTrademarkAttachement(trademarkId, historyId, attachementId, LanguageController.getLocale(), response.getOutputStream());
} catch (IOException e) {
e.printStackTrace();
}
}
Service
#Override
public void getTrademarkAttachement(String trademarkId, String historyId, String attachementId, Locale locale, ServletOutputStream outputStream) {
URI uri = loadHistoryUri(generateUri(REGISTER_BASE_MS_URL, REGISTER_HISTORY_ENTRY_TM_ATTACHEMENT_WS_URL, trademarkId, historyId, attachementId), locale.getLanguage());
HttpURLConnection httpConn = null;
String line = null;
InputStream ins = null;
try {
httpConn = (HttpURLConnection) new URL(uri.toString()).openConnection();
ins = httpConn.getInputStream();
BufferedReader is = new BufferedReader(new InputStreamReader(ins));
while ((line = is.readLine()) != null) {
outputStream.write(line.getBytes());
}
outputStream.flush();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
httpConn.disconnect();
if(ins != null){
try {
ins.close();
} catch (IOException e) {
logger.error("Bad close of inputStream ins");
}
}
}
}
This way, as it reads lines from inputStream ( url to retrieve via GET connection ), it writes it directly to the response via outputStream . It doesn't send bit to bit as in reactive mode , so the user is not getting the info directly, but I think that with Spring MVC 3.2 and Java 7 is the most approximate way to avoid elements in memory .

BufferedReader connection doesn't close

I've created a small scraping class and the method below reads in the text from a page.
However, I've found that the method fails to close the connection properly. This results in a ton of open connections which cause my hosting company to then suspend my account. Is the below correct?
private String getPageText(String urlString) {
String pageText = "";
BufferedReader reader = null;
try {
URL url = new URL(urlString);
reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuilder builder = new StringBuilder();
int read;
char[] chars = new char[1024];
while ((read = reader.read(chars)) != -1)
builder.append(chars, 0, read);
pageText = builder.toString();
} catch (MalformedURLException e) {
Log.e(CLASS_NAME, "getPageText.MalformedUrlException", e);
} catch (IOException e) {
Log.e(CLASS_NAME, "getPageText.IOException", e);
} finally {
if (reader != null)
try {
reader.close();
} catch (IOException e) {
Log.e(CLASS_NAME, "getPageText.IOException", e);
}
}
return pageText;
}

Your code is fine in the success case but will potentially leak connections in the failure cases (when the http server returns a 4xx or 5xx status code). In these cases HttpURLConnection provides the response body via .getErrorStream() rather than .getInputStream() and you should make sure to drain and close that stream as well.
URLConnection conn = null;
BufferedReader reader = null;
try {
conn = url.openConnection();
reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
// ...
} finally {
if(reader != null) {
// ...
}
if(conn instanceof HttpURLConnection) {
InputStream err = ((HttpURLConnection)conn).getErrorStream();
if(err != null) {
byte[] buf = new byte[2048];
while(err.read(buf) >= 0) {}
err.close();
}
}
}
There probably needs to be another layer of try/catch inside that finally but you get the idea. You should not explicitly .disconnect() the connection unless you're sure there won't be any more requests for urls on that host in the near future - disconnect() will prevent subsequent requests from being pipelined over the existing connection, which for https in particular will slow things down considerably.

You are just closing the stream and not the connection, use the following structure:
URL u = new URL(url);
HttpURLConnection conn = (HttpURLConnection)
u.openConnection();
conn.connect();
reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
and then:
} finally {
if (reader != null)
try {
reader.close();
} catch (IOException e) {
Log.e(CLASS_NAME, "getPageText.IOException", e);
}
}
try {
if (conn != null) {
conn.disconnect();
}
} catch (Exception ex) {}
}

Android 2.1 appears to not flush output stream

We are getting some very weird behavior in Android. Our network stack (that talks to a REST server) works fine in almost all situations, except when we do a GET shortly after doing a larger POST. What appears to be happening is that the Output stream is not flushing, and ends up sending the last line that was in there when the new socket is opened. Please note, each connection is a new object created, so this is unexpected behavior. First, the error code that seems to point me to the output stream, these are from the server logs.
10.1.8.195 - - [07/Nov/2012:13:36:28 -0700] "POST /iou/lender HTTP/1.1" 200 28 "-" "Android"
10.1.8.195 - - [07/Nov/2012:13:36:36 -0700] "------------V2ymHFg03ehbqgZCaKO6jy" 400 173 "-" "-"
That attempt after should be a GET that then pulls the data from the server that includes the new entry added via the POST. However, all we get is again what appears to be the last line from the output stream from the POST. Here is our core code for the network stack, if more of the surrounding code is needed, let me know.
public Object serverConnect(String url, String method,
Hashtable<String, Object> params) {
HttpConnection c = null;
InputStream is = null;
OutputStream out = null;
ByteArrayOutputStream postDataByteArrayImage = new ByteArrayOutputStream();
byte[] data;
String boundry = "----------V2ymHFg03ehbqgZCaKO6jy";
try {
if (!url.startsWith("/")) {
url = "/" + url;
}
String uri = Control.URL_Secure + Control.dtserver + ":"
+ Control.port + url;
ByteArrayOutputStream postDataByteArray = new ByteArrayOutputStream();
params.put("sessionId", Control.sessionId);
if (method.equals("GET")) {
uri = uri + "?";
Enumeration enumParams = params.keys();
while (enumParams.hasMoreElements()) {
if (!uri.endsWith("?")) {
uri = uri + "&";
}
String key = (String) enumParams.nextElement();
uri = uri
+ key
+ "="
+ java.net.URLEncoder.encode((String) params
.get(key));
}
} else if (method.equals("POST")) {
Enumeration enumParams = params.keys();
postDataByteArray.write(("--").getBytes());
postDataByteArray.write((boundry).getBytes());
postDataByteArray.write(("\r\n").getBytes());
while (enumParams.hasMoreElements()) {
String key = (String) enumParams.nextElement();
if (!key.equals("image")){
postDataByteArray
.write(("Content-Disposition: form-data; name=\"")
.getBytes());
postDataByteArray.write((key).getBytes());
postDataByteArray.write(("\"").getBytes());
postDataByteArray.write(("\r\n\r\n").getBytes());
postDataByteArray.write(((String) params.get(key))
.getBytes());
postDataByteArray.write(("\r\n").getBytes());
postDataByteArray.write(("--").getBytes());
postDataByteArray.write(boundry.getBytes());
postDataByteArray.write(("\r\n").getBytes());
}
}
postDataByteArray.close();
}
Log.i("URL", uri);
URL urltoConenct = new URL(uri);
URLConnection connection = urltoConenct.openConnection();
HttpURLConnection urlConnection = (HttpURLConnection) connection;
URLConnection.setDefaultRequestProperty("Method", method); // default
urlConnection.setRequestProperty("User-Agent", "Android");
if (method.equals("POST")) {
urlConnection.setDoOutput(true);
urlConnection.setFixedLengthStreamingMode(postDataByteArray.toByteArray().length + postDataByteArrayImage.toByteArray().length);
urlConnection.setRequestProperty("Content-Type",
"multipart/form-data; boundary=" + boundry);
out = urlConnection.getOutputStream();
out.write(postDataByteArray.toByteArray());
out.write(postDataByteArrayImage.toByteArray());
out.close();
}
int response = 0;
try {
response = urlConnection.getResponseCode();
} catch (IOException e) {
if (e.toString()
.equals("java.io.IOException: Received authentication challenge is null"))
throw new RESTException(401, "Invalid Phone or Pin");
else
throw e;
}
if (response == HttpURLConnection.HTTP_OK) {
is = urlConnection.getInputStream();
if (is == null) {
return new IOException(
"Cannot open HTTP InputStream, aborting");
}
ByteArrayOutputStream bo = new ByteArrayOutputStream();
int ch;
int count = 0;
while ((ch = is.read()) != -1) {
bo.write(ch);
count++;
}
data = bo.toByteArray();
return new String(data);
} else if (response == 500) {
return new RESTException(500, "Internal server error");
} else {
RESTException x = new RESTException();
x.setCode(response);
try {
is = urlConnection.getInputStream();
if (is == null) {
x.setMessage("Unable to retrieve message");
return x;
}
ByteArrayOutputStream bo = new ByteArrayOutputStream();
int ch;
int count = 0;
while ((ch = is.read()) != -1) {
bo.write(ch);
count++;
}
data = bo.toByteArray();
String output = new String(data);
JSONObject obj;
try {
obj = new JSONObject(output);
JSONObject err = obj.getJSONArray("errors")
.getJSONObject(0);
x.setMessage(err.getString("message"));
} catch (JSONException e) {
Log.e("stuff", output);
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (FileNotFoundException e) {
// Damn you android! I'm using a REST service here, stop
// trying to interpret my errors!
x.setMessage("Unable to retrieve message");
}
return x;
}
} catch (Exception x) {
x.printStackTrace();
/*
* if (!retried && x.toString().equals(
* "java.io.IOException: Persistent connection dropped after first chunk sent, cannot retry"
* )) { retry = true; } if (!retry) { return x; }
*/
return x;
} finally {
try {
out.close();
} catch (Exception x) {
}
try {
is.close();
} catch (Exception x) {
}
try {
c.close();
} catch (Exception x) {
}
params.clear();
}
// return null;
}

After a very long time of frustration, we discovered that Android tries to keep a connection alive even if you manually call .close() on the connection. This worked fine for our GET methods, but POST methods left the socket in a state that it couldn't then process a GET. Adding the following fixed all our problems:
urlConnection.setRequestProperty("connection", "close");

How to clear the IOExceptionE error in j2me midlet when connect http and send sms vias?

I am working on j2me Mobile application part. I have to send message using http connection and sms format (using sms gateway).
When I am trying to do this, the java.io.IOException: Resource limit exceeded for file handles is throwing in my console.
How to avoid this? This is my connectivity code:
public boolean sendViaHTTP(String message)
{
System.out.println("enter HTTP Via");
HttpConnection httpConn = null;
String url = "http://xxx.com/test.php";
System.out.println("URL="+url);
InputStream is = null;
OutputStream os = null;
try
{
// Open an HTTP Connection object
httpConn = (HttpConnection)Connector.open(url);
// Setup HTTP Request to POST
httpConn.setRequestMethod(HttpConnection.POST);
httpConn.setRequestProperty("User-Agent",
"Profile/MIDP-2.0 Confirguration/CLDC-2.0");
httpConn.setRequestProperty("Accept_Language","en-US");
//Content-Type is must to pass parameters in POST Request
httpConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String value = System.getProperty("com.nokia.network.access");
os = httpConn.openOutputStream();
String params;
params = "message=" + message;
os.write(params.getBytes());// input writes in server side
// Read Response from the Server
StringBuffer sb = new StringBuffer();
is = httpConn.openDataInputStream();
int chr;
while ((chr = is.read()) != -1)
sb.append((char) chr);
Response = sb.toString();
//switchDisplayable("", getForm());
//System.out.println("REsponse="+Response);
}
catch(IOException ex)
{
System.out.println(ex);
return false;
}
catch (Exception ex)
{
System.out.println(ex);
return false;
}
finally
{
try
{
if(is!= null)
is.close();
if(os != null)
os.close();
if(httpConn != null)
httpConn.close();
}
catch (Exception ex)
{
System.out.println(ex);
}
}
return true;
}

That exception is (most likely) happening because somewhere in your application you are not closing your streams after you have finished reading from / writing to them.
To illustrate, if this statement
if (is != null) is.close();
throws an exception (e.g. an IOException), then the remaining statements in the finally block won't be executed. That could leak a file descriptor.
The problem might also be in another part of the code entirely, but the exception message clearly points to a problem with your application using too many file descriptors, and the most likely cause of that is a resource leak.