I faced very strange problem. Writing an application to download some data from Internet with proxy server support I decided to use Apache's HttpClient library. jar binaries were successfully added to NetBeans project and the following code snippet was executed (successfully too) in a simple application:
DefaultHttpClient httpclient = new DefaultHttpClient();
String proxyHost = "192.168.4.10";
Integer proxyPort = 8080;
HttpHost targetHost = new HttpHost("noaasis.noaa.gov", 80, "http");
HttpGet httpget = new HttpGet("/ptbus/ptbus167");
try {
HttpHost proxy = new HttpHost(proxyHost, proxyPort);
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
System.out.println("executing request: " + httpget.getRequestLine());
System.out.println("via proxy: " + proxy);
System.out.println("to target: " + targetHost);
HttpResponse response = httpclient.execute(targetHost, httpget);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
Header[] headers = response.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
}
EntityUtils.consume(entity);
}
catch (IOException ex) {
}
finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
But when I try to do the same thing in Swing application it doesn't work. For example, rewriting default Netbeans desktop application's "about" action listener as follows
#Action
public void showAboutBox() {
new Thread(new Runnable() {
public void run() {
DefaultHttpClient httpclient = new DefaultHttpClient();
......
......
......
finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}).start();
}
causes application's execution to stop somewhere in
HttpResponse response = httpclient.execute(targetHost, httpget);
Leastways, it never returns...
The interesting thing is if I also put this code snippet in application's main method just before creating any Swing instance the mentioned line is passed and HTTP response is received. And calling showAboutBox() doesn't cause the problem anymore then - I receive HTTP response too.
What am I doing wrong, guys? What's the trick? Can I use Apache's library in my Swing application? I cannot understand what happens and didn't find anything similar to this spending hours in the net.
Thank You for attention. Hope for any help!
You're blocking the event dispatch thread (EDT). Use SwingWorker, as shown here.
that only comments but its longer than allowed number of chars....
to avoid wrong directions, Swing based gui doesn't any care that you running any of BackGround Task, Swing is single threaded and all output to the GUI must be done on EDT
1/ wrap output to the GUI to the SwingUtilities.invokeLater(), that's created your own EDT, and if there EDT exist then move actual task to the ends of the EDT
2/ wrap output to the GUI by using javax.swing.Action
3/ or as trashgod suggested let's SwingWorker works for that +1
I solved the problem by excluding org.jdesktop.application.SingleFrameApplication and replacing FrameView by JFrame. Of course, one loses advantages of FrameView but all required things can be implemented extending JFrame.
Unfortunately, I have no enough time to examine why HttpClient doesn't work with SingleFrameApplication so the solution proposed is acceptable for me.
Hope this will help somebody else.
And thanks to trashgod and mKorbel for participation. Thank you, guys. Both +1.
Related
Apache HttpClient 4.2.1
I have a Java process that downloads multiple files from one CouchDB A and uploads the same to another CouchDB B. The downloads and uploads are done using 40 worker threads using an ExecutorService.
When the file count is high (~25k) The process throws a SocketException of "too many open files" due to the Linux open file descriptor limit. I figured this might be due to a resource leak and proceeded to analyze the code which is as follows:
Initiate worker threads
private boolean copyDocumentsFromAtoB(Array ids) {
ExecutorService downloader = Executors.newFixedThreadPool(50);
ExecutorService uploader = Executors.newFixedThreadPool(50);
for (String id: ids) {
downloader.execute(new DownloaderThread(id, downloadedDocs));
}
for (JsonElement doc: downloadedDocs) {
uploader.execute(new UploaderThread(doc));
}
}
Runnable Downloader class (UploaderThread class has a similar implementation)
private static final class DocumentDownloader implements Runnable {
private final String documentId;
private final JsonArray downloadedDocs;
DocumentDownloader(String documentId, JsonArray downloadedDocs) {
this.documentId = documentId;
this.downloadedDocs = downloadedDocs;
}
#Override
public void run() {
InputStream docStream = null;
String url = buildUrl(documentId);
HttpGet doc = new HttpGet(url);
try {
//doc.setHeaders()
HttpClient httpClient = new DefaultHttpClient();
HttpResponse docResponse = httpClient.execute(doc);
docStream = docResponse.getEntity().getContent();
//document parsing and adding to downloadedDocs. Not important
} catch (Exception e) {
//handle exceptions
} finally {
if (docStream != null) {
try {
docStream.close();
} catch (IOException e) {
LOGGER.debug("Cannot close input stream", e);
}
}
}
}
}
Findings:
HttpClient is local to each thread. This is sub-optimal according to many resources and posts I found online.
If I did, httpClient.getConnectionManager().shutdown(); at the end of the Runnable class, the open file count decreased.
Using a global HttpClient (instantiated only once in the copyDocumentsFromAtoB method and passed down to the Runnable classes through the contructors) with a PoolingClientConnectionManager instance and a local context object, decreased the open file count further.
Questions:
Why did the original implementation have a increased number of open file descriptors during the copying process? How does multiple HttpClient instances contribute towards this?
Doesn't the docStream.close(); in the finally block close whatever Http connection that is created between the process and the database?
Does the HttpClient instance get destroyed (releasing any open resources in the process) when the thread that it belongs to terminates? (This would explain why the open file count decreases after the copying process terminates prematurely)
Are there any other optimizations I can do (in terms of resource leaks) other than using a single global HttpClient?
Are there tools that I could use to obtain quantifiable statistics of this scenario for different implementations?
What steps can I follow to find the optimum number of worker
threads, setMaxTotal and setDefaultMaxPerRoute (for the
PoolingClientConnectionManager)?
A bit of background: PropertyFunction is an interface in Jena API that allows doing performing custom operations using SPARQL syntax. Example:
select ?result { ?result f:myPropertyFunction 'someObject' . }
So I made a class Launch that implements this interface and extends a class Client. Within the body of the exec method of my Launch class I establish a connection to a server and, while sending information is no problem, waiting for the server to respond is. Whenever I try to wait() for server response I get the following exception: java.lang.IllegalMonitorStateException.
Here is the body of my exec method for reference:
QueryIterator it = null;
try {
this.connect(); // works well
this.send(algorithmAndArgs); // works well
this.wait(); // exception is thrown
#SuppressWarnings("unused")
ResultSet rs = ResultSetFactory.create(it, Arrays.asList(resultIdentifiers));
} catch (Exception e) {
e.printStackTrace();
}
return it;
Anyone know what the problem may be? Thank you for your answer.
EDIT 1: One thing that I forgot to mention is that the Client class has a method called onObjectReceived(Object o, Socket s) that is triggered each time something is received from the server. I tried using a isDone variable with a while loop in the exec method and set it to true once an object is received, but it did not work.
I solved my own problem: I created an attribute private final CountDownLatch objectWasReceivedLatch = new CountDownLatch(1) and, in the exec method I do boolean objectWasReceived = objectWasReceivedLatch.await(60, TimeUnit.SECONDS); when I want to wait for a response; in the onObjectReceived method I call objectWasReceivedLatch.countDown().
Past few hours I have been working on this without any luck.
public void OnClickEventOnButton(View view)
{
this.commonContext = new BasicHttpContext(); //defined in the class to acihive
HttpClient httpClient=new DefaultHttpClient();
HttpGet get=new HttpGet("http://somesite.com");
HttpResponse response=null;
try{
response=httpClient.execute(get);
if(response.getStatusLine().getStatusCode()==200)
{
}
}catch(ClientProtocolException e){
alert(e.toString(),"エラ");
}
catch (IOException e) {
alert("HTTPHelp : IOException : "+e,"エラ");
}
catch(NullPointerException e){
alert("Null pointer : "+e.getCause(), "sfasfd");
}
}
The line below where my exception is generating
response=httpClient.execute(get);
The error my device showing is Unfortunately, Data Retriver has stopped.
If you need any more information, please let me know. If you need any specific data, please guide me, just started learning, I will provide the data.
Just a guess, but do you
a) don't have the INTERNET permission added in your manifest?
- this means you should see a "IllegalStateException" in your logcat
b) you execute Network on the applications main thread?
- this means you should see a "NetworkOnMainthreadException in your logcat
I have an SQLiteOpenHelper class that implements the singleton pattern and has three methods: addLogs(), getLogs(), and deleteLogs(). Seven threads (triggered by a BroadcastReceiver at certain schedules) access the database--six of them only use addLogs(), and the other one uses getLogs() and calls deleteLogs() before it finishes.
My problem is, because the last thread makes two different calls to the SQLite database, the other threads can call on addLogs() even before deleteLogs() is called, which corrupts my output. How can I make that one thread treat the two calls to the SQLiteOpenHelper as a single, atomic transaction?
What I've tried: Create a static ReentrantLock member in the SQLiteOpenHelper, and call lock() and unlock() from every one of the seven threads before and after their transactions. The other six threads still get through even though the last one still has the lock, which it calls before getLogs() and releases after deleteLogs().
UPDATE: This is the seventh thread that accesses the two methods.
public class Uploader extends Thread {
private Context context;
public Uploader(Context context) {
this.context = context;
}
#Override
public void run() {
Logger.d("STARTED: Uploader");
DB db = DB.getInstance(context);
List<JsonLog> logs = db.getLogs(); // FIRST CALL
String json = new Gson().toJson(logs); // convert to JSON
// connect to the server and upload
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(This.URL);
post.setHeader("content-type", "application/json");
try {
Logger.d("to upload: " + json);
StringEntity se = new StringEntity(json);
post.setEntity(se);
StatusLine status = client.execute(post).getStatusLine();
// if upload is successful
if (status.getStatusCode() == HttpStatus.SC_OK) {
db.deleteLogs(); // SECOND CALL
} else {
Logger.d("Uploader failed. " + status.getStatusCode() + ": " + status.getReasonPhrase());
}
} catch (Exception e) {
e.printStackTrace();
}
Logger.d("FINISHED: Uploader");
}
}
SQLite has transactions as in db.beginTransaction(), db.setTransactionSuccessful() to mark the Tx to be committed and db.endTransaction() to finally mark the Tx as to be done.
So you could in your getLogs() do the beginTransaction and in the deleteLogs to mark it as done (and successful).
[Update]
Now with your code snippet you probably would want to call beginTransaction before your 1st call and in a finally block, that adds to the try-catch block call into endTransaction. At the end of the try part if the status is ok, you would call setTransactionSuccessful (after the delete call.
I am doing HTTP POSTs very frequently (>= 1/sec) to an API endpoint and I want to make sure I'm doing it efficiently. My goal is to succeed or fail as soon as possible, especially since I have separate code to retry failed POSTs. There is a nice page of HttpClient performance tips, but I'm not sure if exhaustively implementing them all will have real benefits. Here is my code right now:
public class Poster {
private String url;
// re-use our request
private HttpClient client;
// re-use our method
private PostMethod method;
public Poster(String url) {
this.url = url;
// Set up the request for reuse.
HttpClientParams clientParams = new HttpClientParams();
clientParams.setSoTimeout(1000); // 1 second timeout.
this.client = new HttpClient(clientParams);
// don't check for stale connections, since we want to be as fast as possible?
// this.client.getParams().setParameter("http.connection.stalecheck", false);
this.method = new PostMethod(this.url);
// custom RetryHandler to prevent retry attempts
HttpMethodRetryHandler myretryhandler = new HttpMethodRetryHandler() {
public boolean retryMethod(final HttpMethod method, final IOException exception, int executionCount) {
// For now, never retry
return false;
}
};
this.method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, myretryhandler);
}
protected boolean sendData(SensorData data) {
NameValuePair[] payload = {
// ...
};
method.setRequestBody(payload);
// Execute it and get the results.
try {
// Execute the POST method.
client.executeMethod(method);
} catch (IOException e) {
// unable to POST, deal with consequences here
method.releaseConnection();
return false;
}
// don't release so that it can be reused?
method.releaseConnection();
return method.getStatusCode() == HttpStatus.SC_OK;
}
}
Would it make sense to disable the check for stale connections? Should I be looking at using the MultiThreadedConnectionManager? Of course, actual benchmarking would help but I wanted to check if my code is on the right track first.
Much of the performance hit of http connections is establishing the socket connection. You can avoid this by using 'keep-alive' http connections. To do this, it's best to use HTTP 1.1 and make sure that "Content-Length: xx" is always set in requests and responses, "Connecction: close" is correctly set when appropriate and is properly acted upon when received.