I have a Java application that is runs constantly. This application makes HTTP requests to a cloud server. The problem is that at each request the memory consumption increases until it reaches the point that the machine complete freezes. I isolated parts of the code and I'm sure the problem is with this code block making this http requests. Analyzing the JVM numbers, via prometheus / Grafana, I see that the use of non-heap memory (codecache and metaspace) are constantly increasing, as shown here
In the image above, whenever there is a drop in the line, it is when 98% of memory consumption reached, and Monit kills the app.
The method that is causing this memory consumption, is below (it is executed approx. 300 times until it exhausts a little more than 1.5GB of available memory in the initialization).
public AbstractRestReponse send(RestRequest request){
BufferedReader in = null;
OutputStream fout = null;
URLConnection conn = null;
InputStreamReader inputStreamReader = null;
String result = "";
try {
MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
for (String key : request.getParams().keySet()) {
String value = (String) request.getParams().get(key);
// System.out.println(key + " = " + value);
mb.addTextBody(key, value);
}
if (request.getFile() != null) {
mb.addBinaryBody("file", request.getFile());
}
org.apache.http.HttpEntity e = mb.build();
conn = new URL(request.getUrl()).openConnection();
conn.setDoOutput(true);
conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
fout = conn.getOutputStream();
e.writeTo(fout);// write multi part data...
inputStreamReader = new InputStreamReader(conn.getInputStream());
in = new BufferedReader(inputStreamReader);
String line;
while ((line = in.readLine()) != null) {
result += line;
}
String text = result.toString();
return objectMapper.readValue(text, FacialApiResult.class);
}catch (Exception e) {
e.printStackTrace();
return null;
}finally {
try {
inputStreamReader.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
conn.getInputStream().close();
} catch (IOException e) {
e.printStackTrace();
}
try {
fout.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
((HttpURLConnection)conn).disconnect() comes to mind. Also String concatenation is time and memory exhaustive. And there was a minor bug in dropping newlines.
NullPointerExceptions may arise in the finally block when an open was not reached due to an exception. But you should have checked that.
public AbstractRestReponse send(RestRequest request) {
URLConnection conn = null;
try {
MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
for (String key : request.getParams().keySet()) {
String value = (String) request.getParams().get(key);
mb.addTextBody(key, value);
}
if (request.getFile() != null) {
mb.addBinaryBody("file", request.getFile());
}
org.apache.http.HttpEntity e = mb.build();
conn = new URL(request.getUrl()).openConnection();
conn.setDoOutput(true);
conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
try (fout = conn.getOutputStream()){
e.writeTo(fout);// write multi part data...
}
StringBuilder resullt = new StringBuilder(2048);
try (BufferedReader in = new InputStreamReader(conn.getInputStream(),
StandardCharsets.UTF_8)) { // Charset
String line;
while ((line = in.readLine()) != null) {
result.append(line).append('\n'); // Newline
}
}
String text = result.toString();
return objectMapper.readValue(text, FacialApiResult.class);
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if (conn != null) {
try {
if (conn instanceof HttpURLConnection) {
((HttpURLConnection) conn).disconnect();
}
} catch (IOException e) {
e.printStackTrace(); //Better logger
}
}
}
return null;
}
I explicitly defined the charset (UTF-8 might be wrong) - momentarily it is the server's default.
Used a StringBuilder, and added the missing newline, which might have lead to wrong parsing.
Try-with-resources for auto-closing, and a bit earlier. Hopefully this does not break anything.
Disconnecting the connection when it is an HttpURLConnection. Mind the instanceof which might play a role in unit tests mocking.
You seems to have handled all possible closing part in the finally block. Anyway it's better to use try-with resources to safely close all Closeable objects, if your application is running on Java 7+. That may isolate the problem further if it doesn't fix.
Related
I i am able to read the lines from csv and download the images from url when the url is not having the image it is showing file not found exception in middle of the program i want to continue the program with out terminating.
public static void main(String[] args) throws IOException {
InputStream inputStream = null;
OutputStream outputStream = null;
try {
BufferedReader br = new BufferedReader(new FileReader("D:\\imgdwnld\\file.csv"));
String line = br.readLine();
while ((line = br.readLine()) !=null){
URL url = new URL(line);
inputStream = url.openStream();
outputStream = new FileOutputStream("D:\\imgdwnld\\" +
line.substring(line.lastIndexOf("/")));
byte[] buffer = new byte[2048];
int length;
while ((length = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, length);
}
}
} catch (MalformedURLException e) {
System.out.println("MalformedURLException :- " + e.getMessage());
} catch (FileNotFoundException e) {
System.out.println("FileNotFoundException :- " + e.getMessage());
} catch (IOException e) {
System.out.println("IOException :- " + e.getMessage());
} finally {
try {
inputStream.close();
outputStream.close();
} catch (IOException e) {
System.out.println("Finally IOException :- " + e.getMessage());
}
}
}
This is the relevant outline of your code:
try {
...
while ((line = br.readLine()) !=null) {
... process each CSV line ...
}
} catch (IOException e) {
... handle it ...
}
At the place where you catch the exception you have already broken out of the main loop. Change the code to have this outline:
while ((line = br.readLine()) !=null) {
try {
... process one CSV line ...
} catch (IOException e) {
... handle it, the loop will proceed with the next line
}
}
You need to remove the "return" instruction from the corresponding catch (or maybe from the whole code). In your case, the instruction allows to "exit" the main method so the rest of instruction (those after the return statement) won't be executed.
Another tip is to separate this instructions in blocs. For example, if reading the CSV and the Img are not bound, you may need to encapsulate each treatment in an individual try-catch block. When parsing/reading the CSV file fails, your code may continue fetching the image.
I've created a small scraping class and the method below reads in the text from a page.
However, I've found that the method fails to close the connection properly. This results in a ton of open connections which cause my hosting company to then suspend my account. Is the below correct?
private String getPageText(String urlString) {
String pageText = "";
BufferedReader reader = null;
try {
URL url = new URL(urlString);
reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuilder builder = new StringBuilder();
int read;
char[] chars = new char[1024];
while ((read = reader.read(chars)) != -1)
builder.append(chars, 0, read);
pageText = builder.toString();
} catch (MalformedURLException e) {
Log.e(CLASS_NAME, "getPageText.MalformedUrlException", e);
} catch (IOException e) {
Log.e(CLASS_NAME, "getPageText.IOException", e);
} finally {
if (reader != null)
try {
reader.close();
} catch (IOException e) {
Log.e(CLASS_NAME, "getPageText.IOException", e);
}
}
return pageText;
}
Your code is fine in the success case but will potentially leak connections in the failure cases (when the http server returns a 4xx or 5xx status code). In these cases HttpURLConnection provides the response body via .getErrorStream() rather than .getInputStream() and you should make sure to drain and close that stream as well.
URLConnection conn = null;
BufferedReader reader = null;
try {
conn = url.openConnection();
reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
// ...
} finally {
if(reader != null) {
// ...
}
if(conn instanceof HttpURLConnection) {
InputStream err = ((HttpURLConnection)conn).getErrorStream();
if(err != null) {
byte[] buf = new byte[2048];
while(err.read(buf) >= 0) {}
err.close();
}
}
}
There probably needs to be another layer of try/catch inside that finally but you get the idea. You should not explicitly .disconnect() the connection unless you're sure there won't be any more requests for urls on that host in the near future - disconnect() will prevent subsequent requests from being pipelined over the existing connection, which for https in particular will slow things down considerably.
You are just closing the stream and not the connection, use the following structure:
URL u = new URL(url);
HttpURLConnection conn = (HttpURLConnection)
u.openConnection();
conn.connect();
reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
and then:
} finally {
if (reader != null)
try {
reader.close();
} catch (IOException e) {
Log.e(CLASS_NAME, "getPageText.IOException", e);
}
}
try {
if (conn != null) {
conn.disconnect();
}
} catch (Exception ex) {}
}
I am working on j2me Mobile application part. I have to send message using http connection and sms format (using sms gateway).
When I am trying to do this, the java.io.IOException: Resource limit exceeded for file handles is throwing in my console.
How to avoid this? This is my connectivity code:
public boolean sendViaHTTP(String message)
{
System.out.println("enter HTTP Via");
HttpConnection httpConn = null;
String url = "http://xxx.com/test.php";
System.out.println("URL="+url);
InputStream is = null;
OutputStream os = null;
try
{
// Open an HTTP Connection object
httpConn = (HttpConnection)Connector.open(url);
// Setup HTTP Request to POST
httpConn.setRequestMethod(HttpConnection.POST);
httpConn.setRequestProperty("User-Agent",
"Profile/MIDP-2.0 Confirguration/CLDC-2.0");
httpConn.setRequestProperty("Accept_Language","en-US");
//Content-Type is must to pass parameters in POST Request
httpConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String value = System.getProperty("com.nokia.network.access");
os = httpConn.openOutputStream();
String params;
params = "message=" + message;
os.write(params.getBytes());// input writes in server side
// Read Response from the Server
StringBuffer sb = new StringBuffer();
is = httpConn.openDataInputStream();
int chr;
while ((chr = is.read()) != -1)
sb.append((char) chr);
Response = sb.toString();
//switchDisplayable("", getForm());
//System.out.println("REsponse="+Response);
}
catch(IOException ex)
{
System.out.println(ex);
return false;
}
catch (Exception ex)
{
System.out.println(ex);
return false;
}
finally
{
try
{
if(is!= null)
is.close();
if(os != null)
os.close();
if(httpConn != null)
httpConn.close();
}
catch (Exception ex)
{
System.out.println(ex);
}
}
return true;
}
That exception is (most likely) happening because somewhere in your application you are not closing your streams after you have finished reading from / writing to them.
To illustrate, if this statement
if (is != null) is.close();
throws an exception (e.g. an IOException), then the remaining statements in the finally block won't be executed. That could leak a file descriptor.
The problem might also be in another part of the code entirely, but the exception message clearly points to a problem with your application using too many file descriptors, and the most likely cause of that is a resource leak.
I can pull the user's statuses with no problem with cURL, but when I connect with Java, the xml comes out truncated and my parser wants to cry. I'm testing with small users so it's not choke data or anything.
public void getRuserHx(){
System.out.println("Getting user status history...");
String https_url = "https://twitter.com/statuses/user_timeline/" + idS.rootUser + ".xml?count=100&page=[1-32]";
URL url;
try {
url = new URL(https_url);
HttpsURLConnection con = (HttpsURLConnection)url.openConnection();
con.setRequestMethod("GET");
con.setReadTimeout(15*1000);
//dump all the content into an xml file
print_content(con);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
System.out.println("Finished downloading user status history.");
}
private void print_content(HttpsURLConnection con){
if(con!=null){
try {
BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
File userHx = new File("/" + idS.rootUser + "Hx.xml");
PrintWriter out = new PrintWriter(idS.hoopoeData + userHx);
String input;
while ((input = br.readLine()) != null){
out.println(input);
}
br.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
This request doesn't need auth. Sorry about my ugly code. My professor says input doesn't matter so my I/O is a trainwreck.
You have to flush the output stream when you write the content out. Did you flush or close the output stream?
I have a method (below) that pulls down and returns the source of a webpage as a String. It all works fine and dandy, but when the connection times out, the program throws an exception and exits. Is there a better method to do this that would allow it to try again on timeout, or is there a way to do it within this method?
public static String getPage(String theURL) {
URL url = null;
try {
url = new URL(theURL);
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
exitprint();
}
InputStream is = null;
try {
is = url.openStream();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
exitprint();
}
int ptr = 0;
StringBuffer buffer = new StringBuffer();
try {
while ((ptr = is.read()) != -1) {
buffer.append((char)ptr);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
exitprint();
}
return buffer.toString();
}
Here's a refactoring of your code that should retry the download N times. Haven't tested it though, but it should kick you off in the right direction.
public static String getPage(String theURL) {
URL url = null;
try {
url = new URL(theURL);
} catch (MalformedURLException e) {
e.printStackTrace();
exitprint();
}
for (int i = 0; i < N; i++) {
try {
InputStream is = url.openStream();
int ptr = 0;
StringBuffer buffer = new StringBuffer();
while ((ptr = is.read()) != -1)
buffer.append((char)ptr);
} catch (IOException e) {
continue;
}
return buffer.toString();
}
throw new SomeException("Failed to download after " + N + " attepmts");
}
I think AOP and Java annotations is a good option. I would recommend to use a read-made mechanism from jcabi-aspects:
#RetryOnFailure(attempts = 2, delay = 10)
public String load(URL url) {
return url.openConnection().getContent();
}
Write a wrapper function around it and allow the connect exception to propogate out. Then you can loop calling your existing function while you receive connect exception upto some max retries.
This is better than embedding a for loop in your existing function because it logically separates retry logic from mainline code. And it's easier to read and understand as a result.
Instead of
try {
is = url.openStream();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
exitprint();
}
you can try set longer timeout and you can still handle timeout exception by catching it
try {
URLConnection con= url.openConnection();
con.setConnectTimeout(5000);
con.setReadTimeout(50000);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
} catch (SocketTimeoutException e) {
//here you can still handle timeout like try again under certain conditions
}
You could put the whole thing in a while loop:
while (true) {
try {
...
} catch (IOException e) {
continue;
}
return buffer.toString();
}
The return statement will break you out of the loop. You might also want to keep track of the number of attempts and stop after 5-10, for politeness, but that's the basic shape of it.
Edit
The better version, based on comments:
int retries = 10;
for (int i = 0 ; i < retries ; i++) {
try {
...
} catch (IOException e) {
continue;
}
return buffer.toString();
}