Develop Reference

Java tutorial sample code throws: java.net.SocketException: Connection reset - what does cause it?

So this is a beginner's question.
When executing the sample code from the working with urls chapter it throws:
Exception in thread "main" java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189) ...
Origin is the openStream() method.
Here is the code:
import java.net.*;
import java.io.*;
public class URLReader {
public static void main(String[] args) throws Exception {
URL oracle = new URL("http://www.oracle.com/");
BufferedReader in = new BufferedReader(
new InputStreamReader(oracle.openStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
in.close();
}
}
I know there are similar threads regarding that topic, but i could not find an answer that suits me.
What I've tried so far:
I have set the proxy host as suggested
here. Command was: java -Dhttp.proxyHost=dslb-088-071-100-199.pools.arcor-ip.net, I also tried it with inserting System.setProperty("http.proxyHost", "dslb-088-071-100-199.pools.arcor-ip.net"); in the first line of the URLReader class.
I tried JSoup html parser and
org.apache.commons.io.FileUtils.copyURLToFile(URL, File) method to have a similar result.
Whatever I try, I always get the same error: There will happen nothing for 30 seconds or so and then it throws the mentioned SocketException.
I simply dont know how to continue in solving this problem. Helpful would be to get information about what happens in background during the 30seconds before connection reset.
So what could actually cause this Exception?
The smallest hint could help! Thank you!

Your code is working fine for JVM's that can connect to the internet.
Based on the original question and discussion: https://chat.stackoverflow.com/rooms/31264/discussion-between-achingfingers-and-meewok it seems that either:
An intermediate firewall is blocking the JVM from making the connection (or another similar network issue).
An operating system firewall, or antivirus that is causing the problems as well.
My suggestion is to try:
Same app on different computer within same network (to see if it is PC specific).
Same app on different network.

Try Apache HTTPClient. I hope all the imports are included as this code is not tested as it is... Also your 30s is the connection timeout of your client.
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.ProxySelector;
import java.net.SocketAddress;
import java.net.URI;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.CoreConnectionPNames;
public class URLReader {
public static void main(String[] args) throws Exception {
HttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(
CoreConnectionPNames.CONNECTION_TIMEOUT, timeOut);
httpclient.getParams().setParameter(
CoreConnectionPNames.SO_TIMEOUT, 2 * timeOut);
httpclient.getParams().setParameter(
CoreConnectionPNames.STALE_CONNECTION_CHECK, false);
httpclient.getParams().setParameter(
CoreConnectionPNames.TCP_NODELAY, true);
HttpHost proxy = new HttpHost(%proxyhost%, %proxyport%);
HttpGet httpget = new HttpGet("http://www.oracle.com");
HttpResponse resp = httpclient.execute(httpget);
respCode = resp.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(new InputStreamReader(resp
.getEntity().getContent()));
String line = null;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
}
}

Clickatell Connect API - No XML Data found

I'm trying to use Clickatell's Connect API ( http://www.clickatell.com/downloads/Clickatell_Connect_API_User_Guide.pdf?cid=205655 ) in order to create users and send sms. Clickatell's API provides you with a specific URL that you have to submit a case-sensitive XML as an HTTPS form post.
I've written the following simple xml
<?xml version="1.0"?>
<CLICKATELLSDK>
<Action>get_list_country</Action>
</CLICKATELLSDK>
and I have used various code sample online (some taken from answer from stackoverflow, other from google searches - i.e. http://www.java-tips.org/other-api-tips/httpclient/how-to-send-an-xml-document-to-a-remote-web-server-using-http-5.html )on how to submit it as an HTTPS POST, but I always get the following responce.
<?xml version="1.0"?>
<CLICKATELLSDK>
<Result>Error</Result>
<Error>999</Error>
<Description>No XML Data found</Description>
<Timestamp>1353538744</Timestamp>
</CLICKATELLSDK>
Anyone using Clickatell's Connect API that can help me out or does anyone have any ideas?

I managed to get it working using the following code
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
public class HttpPost {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
String postUrl = "https://connect.clickatell.com/[AuthenticationToken]";
try {
String tStatus = "";
URL url = new URL(postUrl + "&XML=<clickatellsdk><action>get_list_country</action></clickatellsdk>");
BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
String line;
while ((line = br.readLine()) != null)
{
tStatus = line;
System.out.println(tStatus);
}
} catch (Exception e)
{
//Handle Exception
}
}
}
Let me know if you don't come right, so we can tackle it further!

How do I work with the Expedia XML API in Java

I am having the Expedia account for getting a hotel list, and they giving the XML format data.
I need to process the XML and display HTML formatted data on my website using the Java programming language. I used the file_get_contents in PHP, but I don't know about penny of link to API in Java. What would be an elaborate explanation?

The code below will read the page contents as file_get_contents in php
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
public class URLExp {
public static void main(String[] args) {
try {
URL google = new URL("http://www.google.com/");
URLConnection yc = google.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc
.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Now after passing your url in the code you should get the xml as you said. Parse the XML and use it.

Use apache httpclient to invoke the URL, get the XML by setting headers as text /XML. Easiest way to parse the XML is to use castor library. Post your code and we can help more...

Communication between my Java Server and Local Client

I've been writing easy Java Server. I'm gonna deploy this code to my student's server and run it there.
public class Demo {
public static void main(String[] args) {
String port = "50000";
ServerAttributes attr = new ServerAttributes();
attr.setPort(Integer.parseInt(port));
Socket socket = null;
ServerSocket serverSocket= null;
try {
serverSocket = new ServerSocket(attr.getPort());
System.out.println("Waiting for accept...");
while(true) {
socket = serverSocket.accept();
// TODO
socket.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I wanna create easy Client code which will be 'talking' with my server. Communication Client->Server is easy. My server is visible for client. But what should I do to provide communication in another way?
Maybe REST is good idea? So, how can I 'teach' my server to answer on REST queries?
I've got piece of code which send data to my GAE server:
package enceladus.server.trash.rest;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
public class RESTGAEServer {
static String httpAddress = "http://*********.appspot.com/sign";
public static void main(String[] args) {
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(httpAddress);
try {
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add(new BasicNameValuePair("guestbookName", "default"));
nameValuePairs.add(new BasicNameValuePair("content", "TEST"));
post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
#SuppressWarnings("unused")
HttpResponse response = client.execute(post);
} catch (IOException e) {
e.printStackTrace();
}
}
}
Thanks in advance

If you are trying to provide RESTFul service from the server, its not an easy task. What you might want to do is user something like Restlet for bootstrapping your RESTFul server and client.
For more information refer to http://wiki.restlet.org/docs_2.0/13-restlet/21-restlet/318-restlet.html

REST is a very simple an easy way of communicating between a client and a server. REST basically says, use HTTP the way it was meant to be used, even when communicating between computer programs.
Read up on HTTP in case you do not have enough knowledge. Here is one good document: http://www.jmarshall.com/easy/http/
Once you understand how to send and receive HTTP messages on the client and on the server, you are ready to develop RESTful server API:s.
What you need to know about REST is that it is mostly a way of thinking when you design your API. Make sure to utilize HTTP to its full extent and send/receive data in whatever format (usually JSON, XML or UrlEncoded key/value pairs).
I would say you are MUCH better off doing this yourself than to try to learn Restlet or some other huge library at the same time you learn REST. REST and HTTP are both easy stuff - once you get down to the "it's just some text going back and fourth". When you understand these things fully, then you could look at some frameworks.
Here is some information about REST:
http://rest.elkstein.org/

Using Java to pull data from a webpage?

I'm attempting to make my first program in Java. The goal is to write a program that browses to a website and downloads a file for me. However, I don't know how to use Java to interact with the internet. Can anyone tell me what topics to look up/read about or recommend some good resources?

The simplest solution (without depending on any third-party library or platform) is to create a URL instance pointing to the web page / link you want to download, and read the content using streams.
For example:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
public class DownloadPage {
public static void main(String[] args) throws IOException {
// Make a URL to the web page
URL url = new URL("http://stackoverflow.com/questions/6159118/using-java-to-pull-data-from-a-webpage");
// Get the input stream through URL Connection
URLConnection con = url.openConnection();
InputStream is = con.getInputStream();
// Once you have the Input Stream, it's just plain old Java IO stuff.
// For this case, since you are interested in getting plain-text web page
// I'll use a reader and output the text content to System.out.
// For binary content, it's better to directly read the bytes from stream and write
// to the target file.
try(BufferedReader br = new BufferedReader(new InputStreamReader(is))) {
String line = null;
// read each line and write to System.out
while ((line = br.readLine()) != null) {
System.out.println(line);
}
}
}
}
Hope this helps.

The Basics
Look at these to build a solution more or less from scratch:
Start from the basics: The Java Tutorial's chapter on Networking, including Working With URLs
Make things easier for yourself: Apache HttpComponents (including HttpClient)
The Easily Glued-Up and Stitched-Up Stuff
You always have the option of calling external tools from Java using the exec() and similar methods. For instance, you could use wget, or cURL.
The Hardcore Stuff
Then if you want to go into more fully-fledged stuff, thankfully the need for automated web-testing as given us very practical tools for this. Look at:
HtmlUnit (powerful and simple)
Selenium, Selenium-RC
WebDriver/Selenium2 (still in the works)
JBehave with JBehave Web
Some other libs are purposefully written with web-scraping in mind:
JSoup
Jaunt
Some Workarounds
Java is a language, but also a platform, with many other languages running on it. Some of which integrate great syntactic sugar or libraries to easily build scrapers.
Check out:
Groovy (and its XmlSlurper)
or Scala (with great XML support as presented here and here)
If you know of a great library for Ruby (JRuby, with an article on scraping with JRuby and HtmlUnit) or Python (Jython) or you prefer these languages, then give their JVM ports a chance.
Some Supplements
Some other similar questions:
Scrape data from HTML using Java
Options for HTML Scraping

Here's my solution using URL and try with resources phrase to catch the exceptions.
/**
* Created by mona on 5/27/16.
*/
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
public class ReadFromWeb {
public static void readFromWeb(String webURL) throws IOException {
URL url = new URL(webURL);
InputStream is = url.openStream();
try( BufferedReader br = new BufferedReader(new InputStreamReader(is))) {
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
}
catch (MalformedURLException e) {
e.printStackTrace();
throw new MalformedURLException("URL is malformed!!");
}
catch (IOException e) {
e.printStackTrace();
throw new IOException();
}
}
public static void main(String[] args) throws IOException {
String url = "https://madison.craigslist.org/search/sub";
readFromWeb(url);
}
}
You could additionally save it to file based on your needs or parse it using XML or HTML libraries.

Since Java 11 the most convenient way it to use java.net.http.HttpClient from the standard library.
Example:
HttpClient client = HttpClient.newBuilder()
.version(Version.HTTP_1_1)
.followRedirects(Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(20))
.proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80)))
.authenticator(Authenticator.getDefault())
.build();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("httpss://foo.com/"))
.timeout(Duration.ofMinutes(2))
.GET()
.build();
HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());

I use the following code for my API:
try {
URL url = new URL("https://stackoverflow.com/questions/6159118/using-java-to-pull-data-from-a-webpage");
InputStream content = url.openStream();
int c;
while ((c = content.read())!=-1) System.out.print((char) c);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException ie) {
ie.printStackTrace();
}
You can catch the characters and convert them to string.