I'm a QA with desire to learn more about Java programming and problem I'm experiencing is this:
I'm trying to POST Employee data to the database of some fake Rest API, but I'm getting
Cannot write to a URLConnection if doOutput=false - call
setDoOutput(true)"
So far, I tried some ideas from StackOverflow, but inexperienced as I am, I could easily fall deeper into a problem.
So URL is: http://dummy.restapiexample.com/api/v1/create and firstly I created an Employee class of json object:
public class Main {
public static void main(String[] args) {
new Main();
}
public Main() {
Employees em = new Employees();
em.setEmployeeName("Alex");
em.setEmployeeSalary("1234");
em.setEmployeeAge("28");
try{
URL url = new URL("http://dummy.restapiexample.com/api/v1/create");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept", "application/json");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Unsuccessful call: HTTP error : "
+ conn.getResponseCode());
}
// URLConnection urlc = url.openConnection();
// urlc.setDoOutput(true);
PrintWriter pw = new PrintWriter(conn.getOutputStream());
pw.print(new Gson().toJson(em));
pw.close();
pw.flush();
BufferedReader br = new BufferedReader(
new InputStreamReader(conn.getInputStream())
);
String json = "";
String output;
while ((output = br.readLine()) != null) {
json += output;
}
conn.disconnect();
System.out.println("Employee name: " + em.getEmployeeName());
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}
}
Well, using one of your ideas and added next lines of code (it's commented in above code):
URLConnection urlc = url.openConnection();
urlc.setDoOutput(true);
So the code looks like:
public class Main {
public static void main(String[] args) {
new Main();
}
public Main() {
Employees em = new Employees();
em.setEmployeeName("Alex");
em.setEmployeeSalary("1234");
em.setEmployeeAge("28");
try{
URL url = new URL("http://dummy.restapiexample.com/api/v1/create");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept", "application/json");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Unsuccessful call: HTTP error : "
+ conn.getResponseCode());
}
URLConnection urlc = url.openConnection();
urlc.setDoOutput(true);
PrintWriter pw = new PrintWriter(urlc.getOutputStream());
pw.print(new Gson().toJson(em));
pw.close();
pw.flush();
BufferedReader br = new BufferedReader(new InputStreamReader(
(urlc.getInputStream())));
String json = "";
String output;
while ((output = br.readLine()) != null) {
json += output;
}
conn.disconnect();
System.out.println("Employee name: " + em.getEmployeeName());
}
catch (MalformedURLException e) {
e.printStackTrace(); }
catch (IOException e) {
e.printStackTrace();
}
}}
With this second code I'm not getting that error, but there is no inserting to the database(checking that using postman, with GET method)...
Well, what am I missing? I guess, I'm missing something basic...
Using url.openConnection twice means you get two different connections. You send the request to the second connection, and try to read the response from the first connection. You should call doOutput on the connection you open originally.
The second problem is you're calling getResponseCode before the request is sent. In http, the request must be sent entirely before the server sends a response. You should move the code that calls doOutput and writes the request body before the code that tries to check the response code.
I'm trying to use HttpURLConnection for connecting to server from Android app which I'm developing. For now, I'm testing the connection code not in an app but as a plain java program with main class. I guess this doesn't make any difference as far as HttpUrlConnection.
Please examine the code snippet. Another issue is even errorStream is throwing null. This I feel is because of malformed URL.
private static String urlConnectionTry() {
URL url; HttpURLConnection connection = null;
try {
String urlParameters = "email=" + URLEncoder.encode("email", "UTF-8") +
"&pwd=" + URLEncoder.encode("password", "UTF-8");
//Create connection
url = new URL("http://example.com/login");
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("uuid", getUuid());
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(true);
//Send request
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
wr.writeBytes (urlParameters);
wr.flush ();
wr.close ();
//Get Response
InputStream is = connection.getErrorStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
return response.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if(connection != null) {
connection.disconnect();
}
}
}
private static String getUuid() {
try {
Document doc=Jsoup.connect("http://example.com/getUuid").get();
Elements metaElems = doc.select("meta");
for (Element metaElem : metaElems) {
if(metaElem.attr("name").equals("uuid")) {
return metaElem.attr("content");
}
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
You're probably receiving 401 because the credentials that was sent to the server is not authorized- it's probably not registered or the password is incorrect.
As for the null error stream, take a look at this SO answer.
If the connection was not connected, or if the server did not have an error while connecting or if the server had an error but no error data was sent, this method will return null.
It is probably better if you check first the response code using HttpUrlConnection#getResponseCode(). Decide on whether you'll be checking the contents of the error stream based on the response code you get.
I've always been told you should zip the data to be more efficient. On the input size, this is relatively easy, as shown below:
HttpURLConnection urlConnection=(HttpURLConnection) new URL(url).openConnection();
urlConnection.setRequestProperty("Accept-Encoding", "gzip");
InputStream instream=urlConnection.getInputStream();
Map<String, List<String>> headers = urlConnection.getHeaderFields();
List<String> contentEncodings=headers.get("Content-Encoding");
boolean hasGzipHeader=false;
if (contentEncodings!=null) {
for (String header:contentEncodings) {
if (header.equalsIgnoreCase("gzip")) {
hasGzipHeader=true;
break;
}
}
}
if (hasGzipHeader) {
instream = new GZIPInputStream(instream);
}
The output side seems a tad bit trickier. I've found how to just send a general Post statement output, as follows:
HttpsURLConnection conn = (HttpsURLConnection) url
.openConnection();
conn.setReadTimeout(10000);
conn.setConnectTimeout(15000);
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setRequestProperty("Accept-Encoding", "gzip");
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
OutputStream os = conn.getOutputStream();
os.write(getQuery(results).getBytes(Charset.forName("UTF8")));
private String getQuery(List<BasicNameValuePair> params)
throws UnsupportedEncodingException {
StringBuilder result = new StringBuilder();
boolean first = true;
for (BasicNameValuePair pair : params) {
if (first)
first = false;
else
result.append("&");
result.append(URLEncoder.encode(pair.getName(), "UTF-8"));
result.append("=");
result.append(URLEncoder.encode(pair.getValue(), "UTF-8"));
}
return result.toString();
}
But I can't figure out how to get the output stream gziped, if it's even possible. I can't figure out how to tell if the server will accept the connection, and how to tell the server that the data is zipped. It's easy enough to send the data encoded, as follows:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gzipStream = new GZIPOutputStream(baos);
gzipStream.write(getQuery(results).getBytes(
Charset.forName("UTF8")));
os.write(baos.toByteArray());
Yes, for input (download) the Android implementation of HttpURLConnection does gunzip transparently. But for output (upload) this is not done automatically. The client cannot know whether server supports compression. So you have to do it manually and you have to be sure that your servers understand the request.
You can find an example at DavidWebb.
The code to gzip the payload:
static byte[] gzip(byte[] input) {
GZIPOutputStream gzipOS = null;
try {
ByteArrayOutputStream byteArrayOS = new ByteArrayOutputStream();
gzipOS = new GZIPOutputStream(byteArrayOS);
gzipOS.write(input);
gzipOS.flush();
gzipOS.close();
gzipOS = null;
return byteArrayOS.toByteArray();
} catch (Exception e) {
throw new WebbException(e); // <-- just a RuntimeException
} finally {
if (gzipOS != null) {
try { gzipOS.close(); } catch (Exception ignored) {}
}
}
}
And you have to set the following header:
connection.setRequestProperty("Content-Encoding", "gzip");
I'm trying to find Java's equivalent to Groovy's:
String content = "http://www.google.com".toURL().getText();
I want to read content from a URL into string. I don't want to pollute my code with buffered streams and loops for such a simple task. I looked into apache's HttpClient but I also don't see a one or two line implementation.
Now that some time has passed since the original answer was accepted, there's a better approach:
String out = new Scanner(new URL("http://www.google.com").openStream(), "UTF-8").useDelimiter("\\A").next();
If you want a slightly fuller implementation, which is not a single line, do this:
public static String readStringFromURL(String requestURL) throws IOException
{
try (Scanner scanner = new Scanner(new URL(requestURL).openStream(),
StandardCharsets.UTF_8.toString()))
{
scanner.useDelimiter("\\A");
return scanner.hasNext() ? scanner.next() : "";
}
}
This answer refers to an older version of Java. You may want to look at ccleve's answer.
Here is the traditional way to do this:
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static void main(String[] args) throws Exception {
String content = URLConnectionReader.getText(args[0]);
System.out.println(content);
}
}
As #extraneon has suggested, ioutils allows you to do this in a very eloquent way that's still in the Java spirit:
InputStream in = new URL( "http://jakarta.apache.org" ).openStream();
try {
System.out.println( IOUtils.toString( in ) );
} finally {
IOUtils.closeQuietly(in);
}
Or just use Apache Commons IOUtils.toString(URL url), or the variant that also accepts an encoding parameter.
There's an even better way as of Java 9:
URL u = new URL("http://www.example.com/");
try (InputStream in = u.openStream()) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
Like the original groovy example, this assumes that the content is UTF-8 encoded. (If you need something more clever than that, you need to create a URLConnection and use it to figure out the encoding.)
Now that more time has passed, here's a way to do it in Java 8:
URLConnection conn = url.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8))) {
pageText = reader.lines().collect(Collectors.joining("\n"));
}
Additional example using Guava:
URL xmlData = ...
String data = Resources.toString(xmlData, Charsets.UTF_8);
Java 11+:
URI uri = URI.create("http://www.google.com");
HttpRequest request = HttpRequest.newBuilder(uri).build();
String content = HttpClient.newHttpClient().send(request, BodyHandlers.ofString()).body();
If you have the input stream (see Joe's answer) also consider ioutils.toString( inputstream ).
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream)
The following works with Java 7/8, secure urls, and shows how to add a cookie to your request as well. Note this is mostly a direct copy of this other great answer on this page, but added the cookie example, and clarification in that it works with secure urls as well ;-)
If you need to connect to a server with an invalid certificate or self signed certificate, this will throw security errors unless you import the certificate. If you need this functionality, you could consider the approach detailed in this answer to this related question on StackOverflow.
Example
String result = getUrlAsString("https://www.google.com");
System.out.println(result);
outputs
<!doctype html><html itemscope="" .... etc
Code
import java.net.URL;
import java.net.URLConnection;
import java.io.BufferedReader;
import java.io.InputStreamReader;
public static String getUrlAsString(String url)
{
try
{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(true); // we want the response
con.setRequestProperty("Cookie", "myCookie=test123");
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}
Here's Jeanne's lovely answer, but wrapped in a tidy function for muppets like me:
private static String getUrl(String aUrl) throws MalformedURLException, IOException
{
String urlData = "";
URL urlObj = new URL(aUrl);
URLConnection conn = urlObj.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8)))
{
urlData = reader.lines().collect(Collectors.joining("\n"));
}
return urlData;
}
URL to String in pure Java
Example call to get payload from http get call
String str = getStringFromUrl("YourUrl");
Implementation
You can use the method described in this answer, on How to read URL to an InputStream and combine it with this answer on How to read InputStream to String.
The outcome will be something like
public String getStringFromUrl(URL url) throws IOException {
return inputStreamToString(urlToInputStream(url,null));
}
public String inputStreamToString(InputStream inputStream) throws IOException {
try(ByteArrayOutputStream result = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
result.write(buffer, 0, length);
}
return result.toString(UTF_8);
}
}
private InputStream urlToInputStream(URL url, Map<String, String> args) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setReadTimeout(15000);
if (args != null) {
for (Entry<String, String> e : args.entrySet()) {
con.setRequestProperty(e.getKey(), e.getValue());
}
}
con.connect();
int responseCode = con.getResponseCode();
/* By default the connection will follow redirects. The following
* block is only entered if the implementation of HttpURLConnection
* does not perform the redirect. The exact behavior depends to
* the actual implementation (e.g. sun.net).
* !!! Attention: This block allows the connection to
* switch protocols (e.g. HTTP to HTTPS), which is <b>not</b>
* default behavior. See: https://stackoverflow.com/questions/1884230
* for more info!!!
*/
if (responseCode < 400 && responseCode > 299) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, args);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, args);
}
}
/*!!!!!*/
inputStream = con.getInputStream();
return inputStream;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
Pros
It is pure java
It can be easily enhanced by adding different headers as a map (instead of passing a null object, like the example above does), authentication, etc.
Handling of protocol switches is supported
I try to send post request, but webserver returns that I added no post-values. I spent a lot of time trying to solve this issue, but no result. Here is the code:
public static String post(String url, String postParams)
{
URLConnection connection = null;
try
{
connection = initializeConnection(url);
connection.setDoOutput(true);
((HttpURLConnection) connection).setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "text/xml");
connection.setRequestProperty("Accept", "text/xml");
connection.setUseCaches(false);
connection.setDoInput(true);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.write(postParams.getBytes());
wr.flush();
wr.close();
// Get Response
InputStream is = connection.getInputStream();
return inputStreamToString(is);
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
}
protected static HttpURLConnection initializeConnection(String stringUrl)
{
HttpURLConnection connection;
URL url = null;
try
{
url = new URL(stringUrl);
}
catch (MalformedURLException e1)
{
e1.printStackTrace();
}
try
{
connection = (HttpURLConnection) url.openConnection();
connection.setConnectTimeout(5000);
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
return connection;
}
public static String inputStreamToString(InputStream is)
{
BufferedReader r = new BufferedReader(new InputStreamReader(is));
StringBuilder total = new StringBuilder();
String line;
try
{
while ((line = r.readLine()) != null)
{
total.append(line);
}
}
catch (IOException e)
{
e.printStackTrace();
}
return total.toString();
}
I receive a message from webserver where it is told that no post-values are added. As far as I understand from the code, the values are added. I'm stuck.
It turned out that all I had to do was to replace
connection.setRequestProperty("Content-Type", "text/xml");
with
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
So simple and so much time spent to clear it out...
By the way, how could I know that server requires this header? I thought that all the work that is essential to the request would be automatically done by java..
P.S. Installing fiddler helped to solve the issue, thanks for that.
debug the 'postParams' parameter and check what been sent.