If I upload 100K file to certain url of my service, wget takes ~20 seconds to complete:
wget --quiet --post-file data.txt --output-document - --header "Content-Type: text/csv" http://localhost:8080/ingest
But if I do it like this in java, strangely this happens immediately:
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "text/csv;charset=UTF-8");
con.setDoOutput(true);
OutputStream outputStream = con.getOutputStream();
outputStream.write(str.getBytes("UTF-8"));
outputStream.flush();
outputStream.close();
System.out.println("code=" + con.getResponseCode());
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
so my guess is that actually this code is not waiting for data to be submitted, but is doing this in background. How can I force it to block until actual data transfer is finished?
That code should be waiting for the response to complete. The con.getResponseCode() call will not (cannot!) return until the server has at least responded with the HTTP reply header containing the response code.
It may be that the server is sending the HTTP reply header before it has finished reading the data that the client has posted. That would be a mistake. (If the server sends the response too soon, it can't set the response code correctly!)
It is also possible that the server response is not a 2xx response, and there are server error messages / diagnostics on the error stream rather than the input stream. (Read the javadocs on getInputStream versus getErrorStream.)
So the most likely reason that is not blocking for ~20 seconds is because the request has failed ... and this is not being reported properly, due to server or client-side implementation issues.
UPDATE - It turns out that the real issues was that "curl" was behaving strangely on some platforms, probably due to network config issues.
Related
I have a short Android-Java client program which sends a basic information to bottle-python server with POST method. In the first version of code, server does not show anything. However, In the second version it works but I cannot understand what this additional line do because it has anything to do with posting content. I really appreciate if someone helps me figure this out.(There is nothing wrong with the server code since I can properly send request with python requests and my browsers).
This is the first version of client code:
String url = "http://192.168.1.23:8080/";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setDoOutput(true);
OutputStream os = con.getOutputStream();
PrintStream myPusher = new PrintStream(os );
myPusher.print("param1=hey");
Second version:
String url = "http://192.168.1.23:8080/";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setDoOutput(true);
OutputStream os = con.getOutputStream();
PrintStream myPusher = new PrintStream(os );
myPusher.print("param1=hey");
InputStream in= con.getInputStream(); //Nothing changed but only this additional line
Bottle(python) server:
#app.route('/', method="POST")
def hello():
print("it works")
name = request.forms.get("param1")
print(name)
return name
#app.route('/')
def hello():
i=0
print("it works")
run(app, host="192.168.1.23", port=8080)
With first client code server shows nothing.
With second code server shows:
it works
hey
192.168.1.24 - - [31/Dec/2018 17:10:28] "POST / HTTP/1.1" 200 3
Which is as I expected.
With your first code snippet the output stream is still open. So the server does not know if it got the complete request. Probably just closing the stream would work as well.
However, I would make at least a call to getResponseCode to see the outcome of the request.
Your java code seems incomplete for sending a post request.
I think by using this code, you can make it work for yourself.
The PrintStream is a buffered type, this means you should add a flush operation after each print(), or use println() instead.
I am using the following CURL command to save the response in .pdf format:
curl -d "#name_of_xmlFile.xml" -X POST http:url/ -o name_of_response_pdfFile.pdf
how to make the call and save the response generated pdf file in a specific folder using java.
Welcome to stack overflow !
The problem here is that curl is a Linux command. Because Java is intended to be written once and run everywhere (on any machine), then java cannot directly use commands like curl, which do not exist by default on every operating system.
Therefore you need to include additional functionality from elsewhere.
Personally I like to use Unirest which is a nice simple lightweight framework to aid with using Rest in Java.
Good luck !
You can also perform a post in Java without using curl (not sure whether using curl is a MUST have requirement)
taken from this article:
private void doPost() throws Exception {
String url = "http:url/";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
String urlParameters = "sn=C02G8416DRJM&cn=&locale=&caller=&num=12345";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
// could ewually save to file rather than to stdout.
}
For sending the body, you may try this.
If using curl is a hard requirement, you may use Runtime.getRuntime().exec(command); as shown here, but be aware that this isn't necessarily portable (e.g. if curl is not installed) and could be unsafe if the commands being run aren't in some way validated to prevent bad actors running arbitrary commands etc...
I have a core server and a gateway server. Only gateway interacts with CORE.
one of the requests had to interact with an external partner which includes following steps.
CORE sends a request to GATEWAY and waits for response
GATEWAY sends the request to EXTERNAL and sends back response to CORE
In CORE server, I have got this IOException
java.io.IOException: Server returned HTTP response code: 504 for URL: someurl
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
I figured that IOException is because I am using urlConnection.getInputStream() instead of urlConnection.getErrorStream().
urlConnection.connect();
OutputStreamWriter osw = new OutputStreamWriter(urlConnection.getOutputStream());
BufferedWriter buffWriter = new BufferedWriter(osw);
buffWriter.write(request.toString());
buffWriter.flush();
buffWriter.close();
osw.close();
InputStreamReader isr = new InputStreamReader(urlConnection.getInputStream());
BufferedReader buffRead = new BufferedReader(isr);
Can't figure out why I am getting error 504.
GATEWAY server got the response successfully and I see no problem there.
I have handled SocketTimeoutException,ConnectException ,FileNotFoundException exceptions for HTTPSrequestsender.
I have the below code to connect a webserver and get content via http request and everything is Ok. But sometimes that website is really being busy, taking to many request at the same time from different users, since I do not know how the http servers work exactly, Is there any tricks that I can do to get content rapidly, in your mind.
I use the below code in multi threaded, every 5 milisecond to get the content when the website is updated to get data instantly.
Can I keep the connection open? Does it make sense, or anything else that makes me get the data earlier when it is really busy?
URLConnection con = url.openConnection();
url.openConnection();
BufferedReader in = new BufferedReader(new
InputStreamReader(con.getInputStream()));
while ((inputLine = in.readLine()) != null) result.append(inputLine);
in.close();
content = result.toString();
Thanks.
I have a php page in my server that accepts a couple of POST requests and process them. Lets say it's a simple page and the output is simply an echoed statement. With the URLConnection I established from a Java program to send the POST request, I tried to get the input using the input stream got through connection.getInputStream(). But All I get is the source of the page(the whole php script) and not the output it produces. We shall avoid socket connections here. Can this be done with Url connection or HttpRequest? How?
class htttp{
public static void main(String a[]) throws IOException{
URL url=new URL("http://localhost/test.php");
URLConnection conn = url.openConnection();
//((HttpURLConnection) conn).setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write("Hello");
wr.flush();
wr.close();
InputStream ins = conn.getInputStream();
InputStreamReader isr = new InputStreamReader(ins);
BufferedReader in = new BufferedReader(isr);
String inputLine;
String result = "";
while( (inputLine = in.readLine()) != null )
result += inputLine;
System.out.print(result);
}
}
I get the whole source of the webpage test.php in result. But I want only the output of the php script.
The reason you get the PHP source itself, rather than the output it should be rendering, is that your local HTTP server - receiving your request targeted at http://localhost/test.php - decided to serve back the PHP source, rather than forward the HTTP request to a PHP processor to render the output.
Why this happens? that has to do with your HTTP server's configuration; there might be a few reasons for that. For starters, you should validate your HTTP server's configuration.
Which HTTP server are you using on your machine?
What happens when you browse http://localhost/test.php through your browser?
The problem here is not the Java code - the problem lies with the web server. You need to investigate why your webserver is not executing your PHP script but sending it back raw. You can begin by testing using a simple PHP scipt which returns a fixed result and is accessed using a GET request (from a web browser). Once that is working you can test using the one that responds to POST requests.