I'm working on a software that does extensive queries to a database which is has a http interface. So my program parses and handles queries that are in form of long http:// addresses..
I have realized that the bottleneck of this whole system is the querying and the data transfer barely goes above 20KB/s even though I am sitting in the university network with a gigabit connection. Recently a friend of mine mentioned that I might have written my code in an ineffective way and that might be reason for the lack of speed in the process. So my question is what is the fastest/most effective way of getting data from a web source in Java.
Here's the code I have right now:
private void handleQuery(String urlQuery,int qNumber, BufferedWriter out){
BufferedReader reader;
try{
// IO - routines: read from the webservice and print to a log file
reader = new BufferedReader(new InputStreamReader(openURL(urlQuery)));
....
}
}
private InputStream openURL(String urlName)
throws IOException
{
URL url = new URL(urlName);
URLConnection urlConnection = url.openConnection();
return urlConnection.getInputStream();
}
Your code looks good to me. The code snippet doesn't explain the slow read.
Possible problems are,
Network issues. Do an end-end network test to make sure network is as fast as you think.
Server issues. Maybe the server is too slow.
Thread contention. Check if you have any thread issues.
A profiler and network trace will pin-point the problem.
There is nothing in the code that you have provided that should be a bottleneck. The problem is probably somewhere else; e.g. what you are doing with the characters after you read them, how the remote server is writing them, network or webproxy issues, etc.
Related
Following security vulnerability has been reported on our application -
The call to readLine() at xyz.java line 119 might allow an attacker to
crash the program or otherwise make it unavailable to legitimate
users.
Code with vulnerability
Commented line reported -
BufferedReader reader = new BufferedReader(new InputStreamReader(
httpConnection.getInputStream()));
String inputLine;
StringBuffer okResponse = new StringBuffer();
while ((inputLine = reader.readLine()) != null) { //readLine() on this line has been reported
okResponse.append(inputLine);
}
reader.close();
return okResponse.toString();
The remediation for the same says -
Validate user input to ensure that it will not cause inappropriate
resource utilization.
But, it is not clear what exactly can be validated. Any pointers?
In theory an attacker could send you an unlimited amount of data via the httpConnection. Since you try to consume all of it, it might crash your application (OutOfMemory).
I´d assume you have a certain format and length for the okResponse in mind,
so you would be better off checking that.
Make sure you read to read the answer from Subhas linked by Luis Muñoz (Most Robust way of reading a file or stream using Java (to prevent DoS attacks)),
that has some more implementation details for reading content from a stream.
Another issue might be that the attack just keeps the connection open without sending any data. I`d assume there should be a timeout in place to cut the connection at some point, else the thread might be blocked forever.
Edit:
Since you code does not have it explicitly, you also should add a try ... finally ... block to make sure that the resources are properly closed.
I have been searching about this info but since I'm new to web development the answers I'm getting are getting me even more confused.
Basically, I have a webserver established in a Java Modem (which uses 1.3IDE) which will handle requests. These requests were being processed as long as I kept it simple.
http://87.103.87.59/teste.html?a=10&b=10
This request is normally processed.
However, when applying the real deal, my webserver is crashing.
http://5.43.52.4/api.html?ATCOMMAND=AT%5EMTXTUNNEL=SMS,0035111111111,string sending test
The problem is due to two aspects. The "%" character and the string sending test.
To put everything clear, handlers I'm using are these:
public InputStream is = null;
private OutputStream os = null;
private byte buffer[] = new byte[];
String streamAux="";
is = socketX.openInputStream();
os = socketX.openOutputStream();
if ((is.available()>0)||(blockX==true))
{
//Read data sent from remote client
numDadosLidos=is.read(buffer);
for (int i=0;i<numDadosLidos;i++)
streamAux= streamAux + (char)buffer[i]; //where the url will be stored
Basically I will need those parameters so I can use them to operate my Java device so, I think I'll need to do some sort of encoding but there's a lot of information that I can't comprehend and my 1.3 IDE is kind of keeping me stuck.
I apologize for some sort of newbie behaviour in advance.
Hope you can lend me a hand,
Thanks
For those who are interested, I basically went around the issue obliging the message to be sent using '-' character. It doesn't solve the issue, it simply solves the question with the "not-ideal" method.
Still totally interested if someone figures this one out.
Thanks.
I need to know what could cause the InputStream to hang indefinitely on close. Here's my code.
URL url = new URL("ftp://..");
InputSream is = url.openStream();
BufferedReader reader = new InputStreamReader(new GZIPInputStream(is));
try{
while(true){
if(reader.readline() == null){
break;
}
}
}catch(Exception e){
e.printStackTrace();
} finally{
System.out.println("Closing reader");
is.close(); // sometimes hangs indefinitely
System.out.println("Reader closed");
}
Closing InputStream or BufferedReader has the same effect.
I need to know what could cause the InputStream to hang indefinitely on close
I think it is the nature of the particular input stream that you are using.
You have a stream open to a read a file from an FTP server. Depending on the protocol driver that is being used on the Java client side, when you close() the file it may be attempting to close an active FTP session. This could involve exchanging network packets with the remote server, and if there is a network problem or the server has died, then that could take "a long time" ... depending on how long network timeouts have been set to, etc.
It could also be something like this:
Java HttpURLConnection InputStream.close() hangs (or works too long?)
My recommendation you be to capture a thread stacktrace while a connection is hung in close() and then delve into the Java codebase to figure out where exactly it is hung. (There are too many possible places for a problem to occur to investigate this without concrete evidence.)
Also try removing the GZIPInputStream from the stack to see if that makes any difference.
Finally, if you come back to us with more evidence, please tell us the precise Java version you are using and the real URL of the FTP server.
So I've created a program that communicates with both a FTP and SQL server. I run my program on my Windows machine and everything is working fine. But when I run the program on my Ubuntu VM it is not working. My code is as follows:
try {
URL url = new URL(location);
System.out.println("Created url");
URLConnection urlc = url.openConnection();
System.out.println("Created URLConnection");
urlc.setDoOutput(true);
System.out.println("Set Output");
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
System.out.println("Created OutputStreamWriter");
System.out.println("Hello World.");
out.write(Var.pressed);
System.out.println("Wrote String");
out.close();
System.out.println("Closed OutputStreamWriter");
} catch (Exception ex) {
ex.printStackTrace();
}
The program creates a file on my FTP and then writes to it. The file is being created but it is empty. No errors are ever thrown my program prints the following
Created url
Created URLConnection
Set Output
And then does nothing. It doesn't throw an error, it doesn't crash, it doesn't stop. The terminal is just left open. I also tried just running the .jar but nothing happens.
It seems as though the problem is created here:
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
Why is nothing being written to the file? And what is wrong with my program if no errors are ever thrown?
It does throw an error,
From everything else, I think you meant to say it doesn't throw an error. My answer is assuming that anyway.
First there are many reasons for a program to hang or wait and not produce an error. Sometimes if you wait long enough a system threshold will be passed and it will timeout even though your program would be happy to wait forever. Other times the system isn't paying attention either. Error conditions are sort of like a courtesy.
I suggest you should try to make the connection with the Ubuntu ftp line command and compare the same with Windows. FTP protocol does some unusual things with ports. The connection opens on one port, but the data transfer connection is switched over to another port. Firewalls can be tripped up with the change of port and prevent the data transfer. This also has created clients that do "passive" versus "active" FTP.
You need to at least get the response code, or the input stream. Otherwise not even a TCP connection is formed. And otherwise you have no idea whether the server accepted your input.
Your title was wildly inaccurate. Obviously you are creating the OutputStreamWriter. Your issue is that nothing is being sent. And the SQL tag was simply irrelevant.
Without an error message it is nearly impossible to figure out why it fails. A few suggestions, though.
First of all, try adding a System.err.println("test") This will prove that the output log will not only catch System.out AKA standard output stream, but also standard error stream.
I asssume, that in after this fix, the ex.printStackTrace() will be visible.
Otherwise: You do not specify how your location String is filled. Are you sure, you have write access to that directory?
If location is /tmp/foo/bar/myFile.txt, make sure the directory /tmp/foo/bar/ exists (mkdir /tmp/foo/bar/) and that the file is accessible (touch /tmp/foo/bar/myFile.txt).
If both report no error AND the first suggestion would bring up no error, the program might really lock up and wait for something from the outside. Difficult to say what at the moment.
this is my first question on stack overflow, I hope you can help me. I've done a bit of searching online but I keep finding tutorials or answers that talk about reading either text files using a BufferedReader or reading bytes from files on the internet. Ideally, I'd like to have a file on my server called "http://ascistudent.com/scores.data" that stores all of the Score objects made by players of a game I have made.
The game is a simple "block-dropping" game where you try to get 3 of the same blocks touching do increase the score. When time runs out, the scores are loaded from a file, their score is added in the right position of a List of Score objects. After that the scores are saved again to the same file.
At the moment I get an exception, java.io.EOFException on the highlighted line:
URL url = new URL("http://ascistudent.com/scores.data");
InputStream is = url.openStream();
Score s;
ObjectInputStream load;
//if(is.available()==0)return;
load = new ObjectInputStream(is); //----------java.io.EOFException
while ((s = (Score)load.readObject()) != null){
scores.add(s);
}
load.close();
I suspect that this is due to the file being empty. But then when I catch this exception and tell it to write to the file anyway (after changing the Score List) with the following code, nothing appears to be written (the exception continues to happen.)
URL url = new URL("http://ascistudent.com/scores.data");
URLConnection ucon = url.openConnection();
ucon.setDoInput(true);
ucon.setDoOutput(true);
os = ucon.getOutputStream();
ObjectOutputStream save = new ObjectOutputStream(os);
for(Score s:scores){
save.writeObject(s);
}
save.close();
What am I doing wrong? Can anyone point me in the right direction?
Thanks very much,
Luke
Natively you can't write to an URLConnection unless that connection is writable.
What I mean is that you cannot direcly write to an URL unless the otherside accept what you are going to send. This in HTTP is done throug a POST request that attaches data from your client to the request itself.
On the server side you'll have to accept this post request, take the data and add it tothe scores.data. You can't directly write to the file, you need to process the request in the webserver, eg:
http://host/scores.data
provides the data, while
http://host/uploadscores
should be a different URL that accepts a POST request, process it and remotely modifies score.data