Develop Reference

Java HttpUrlConnection throws Connection Refused

I know there are several question regarding this topic But I did't find an answer in any of them.
I'm trying to open a connection to my local server but I keep getting connection refused.
I have the server running and I tested the connection with the Browser and with a Google App called Postman and it works.
It's failing when opening the connection as if there where nothing to connect to. or maybe something is blocking the connection? I tested with firewall and antivirus down, no luck.
testing in Postman the URL returns a User as it should...
If I replace the url with "http://www.google.com" It Works fine.
here is my code:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
/**
*
* #author Gabriel
*/
public class HttpConnection {
public HttpConnection() {
}
public void makeRequest() throws MalformedURLException, IOException {
String url = "http://localhost:8000/users/1";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip,deflate,sdch");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.8,es;q=0.6");
con.setRequestProperty("Connection", "keep-alive");
con.setRequestProperty("Host", "localhost:8000");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
}

I faced exactly the same problem. Use this instead of localhost:
http://[::1]:8000/index.php

I have similar code that is working, but my request header is a lot simpler. Basically just:
con.setRequestProperty("User-Agent", "Mozilla/5.0");
If simplifying the header does not help, I would capture the traffic when using your browser with something like fiddler and then making the request look exactly like that.

I will make a wild guess what can be the problem. It is possible a IPv4/IPv6 problem.
If so, here is two possible solutions
If the server is only listening on an ipv6 address, change it to listening to ipv4.
If the server is listening to ipv4, then force Java to use ipv4 with
java.net.preferIPv4Stack=true

You can try implementing CORS at the API you are trying to connect by setting access-control-allow-origin:* property in response header.

The code is good and works great. Now the problem must be on the transportation or network part. What I want to mean is you don't request the right server. If you use 127.0.0.1 instead of localhost I think you won't get a problem. So, my guest will be that you have a problem in /etc/hosts or C:\Windows\System32\drivers\etc\hosts.
I advice you to try a simple test: ping the hostname and check in the output if the ip address is good.

Well, put http://localhost:8000/users/1 in your web browser and what do you get? A simple Connection Refused error. It's not you, it's the website. Also, Url returns websites using Protocol Identifiers(http://, https://), Ending Domains(.com, .edu, .gov) that's also another reason why you get an error.

You mentioned that you were opening a connection to your "local server"
I am assuming that you are doing this on the same computer that you're hosting the server on?
Try to open the connection to your local server using a different computer.

Can't read in HTML content from valid URL

I am trying out a simple program for reading the HTML content from a given URL. The URL I am trying in this case doesn't require any cookie/username/password, but still I am getting a io.IOException: Server returned HTTP response code: 403 error. Can anyone tell me what am I doing wrong here? (I know there are similar question in SO, but they didn't help):
import java.net.*;
import java.io.*;
import java.net.MalformedURLException;
import java.io.IOException;
public class urlcont {
public static void main(String[] args) {
try {
URL u = new URL("http://www.amnesty.org/");
URLConnection uc = u.openConnection();
uc.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
uc.connect();
InputStream in = uc.getInputStream();
int b;
File f = new File("C:\\Users\\kausta\\Desktop\\urlcont.txt");
f.createNewFile();
OutputStream s = new FileOutputStream(f);
while ((b = in.read()) != -1) {
s.write(b);
}
}
catch (MalformedURLException e) {System.err.println(e);}
catch (IOException e) {System.err.println(e);}
}
}

If you can fetch the URL in a browser, but not via Java, that indicates, to me, that they are blocking programmatic access to the page via user-agent filtering. Try setting the user-agent on your connection so that your code appears, to the webserver, to be a web-browser.
See this thread for help on that: What is the proper way of setting headers in a URLConnection?

There is a permission problem:
A web server may return a 403 Forbidden HTTP status code in response to a request from a client for a web page or resource to indicate that the server refuses to allow the requested action

you are not doing anything "wrong", the server you are trying to access is blocking your request, as you are not allowed to access the file
Http-Error 403 means Forbidden --> the remote server blocks the request.
check if you need to give authentification to access the document you want and in that case provide it with the request ;)

Suppress NTLM Auth in Java

I am currently pen-testing a web application and came across an interesting phenomenon. During my testing sessions, I gathered URLs using a proxy. Now I wanted to test my URL list for anonymous access, so i wrote this little tool
public static void main(String[] args) {
try {
TrustAllCerts.disableCertChecks();
FileReader fr = new FileReader(new File("urls.txt"));
BufferedReader br = new BufferedReader(fr);
String urlStr = br.readLine();
while (urlStr != null) {
if (urlStr.trim().length() > 0) {
URL url = new URL(urlStr);
HttpsURLConnection urlc = (HttpsURLConnection) url.openConnection();
urlc.connect();
if (urlc.getResponseCode() == HttpURLConnection.HTTP_OK) {
System.out.println(urlStr);
} else {
System.out.println("["+urlc.getResponseCode()+"] "+urlStr);
}
urlc.disconnect();
}
urlStr = br.readLine();
}
br.close();
} catch (Exception e) {
e.printStackTrace();
}
}
It does basically nothing, but opening an URL connection on a given URL and test the HTTP response code (actually I implemented some more tests, if I'm getting redirected to a login page). However, the problem is, that this specific application (some custom MS SQL Server Reporting Services) is configured to use NTLM WWW authentication. If I try to access some of the URLs using Firefox, i get an 401 Unauthorized + login dlg. Internet Exploder performs NTLM auth in the background and grants access. It seems that the Java URLConnection (or URL) class does the same, because I am getting no 401. Is there a way to disable implicit NTLM authentication in Java? This is a bad pitfall for me.

I think the Java Network Documentation is the best resource. Setting the http.auth.preference="basic" should get you what you want. Assuming you don't need digest or something else. I'm not sure if you can go beyond that to disable NTLM.
Another thing to consider is other Java HTTP client implementations, like Apache's or Google's.

I'm not sure that this will help, but I've been stumped by the opposite.
I wanted NTLM auth to take place, so on my local machine I use a free app called CNTLM. It's a local proxy server that will forward (and NT auth) incoming requests. Good for apps that can't use NTLM proxies.
I'm sorry, I know this isn't answering your question, but maybe it proves helpful to someone out there! :)

How to avoid rome exception

I'm doing project using struts1.
I'm fetching RSS feeds using ROME but it fails for two conditions:
When my firewall forbidden rss url (response code 403)
When I insert incorrect rss url
To avoid such conditions what should I do?

Just catch the exceptions and handle them.

There are some situations you simply cannot avoid.
You can't avoid network outages, you can't avoid incorrectly typed URLs.
What you can do, however, is check if network is reachable, and whether the URL is typed correctly or not.
You should catch the exceptions and provide meaningful error messages to the user.

About 403
Some feeds seems have some protection (for DDOS)
So based on user Agent (in your case "Java") they deny you to read the feed
So you have to set your own user agent (like the firefox user agent), before opening connection like this
System.setProperty("http.agent", USER_AGENT);
URLConnection openConnection = url.openConnection();
is = url.openConnection().getInputStream();
if ("gzip".equals(openConnection.getContentEncoding())) {
is = new GZIPInputStream(is);
}
InputSource source = new InputSource(is);
input = new SyndFeedInput();
syndicationFeed = input.build(source);
XmlReader reader = new XmlReader(url);
syndicationFeed = input.build(reader);
My current USER_AGENT String is
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0";

Cookies turned off with Java URLConnection

I am trying to make a request to a webpage that requires cookies. I'm using HTTPUrlConnection, but the response always comes back saying
<div class="body"><p>Your browser's cookie functionality is turned off. Please turn it on.
How can I make the request such that the queried server thinks I have cookies turned on. My code goes something like this.
private String readPage(String page) throws MalformedURLException {
try {
URL url = new URL(page);
HttpURLConnection uc = (HttpURLConnection) url.openConnection();
uc.connect();
InputStream in = uc.getInputStream();
int v;
while( (v = in.read()) != -1){
sb.append((char)v);
}
in.close();
uc.disconnect();
} catch (IOException e){
e.printStackTrace();
}
return sb.toString();
}

You need to add a CookieHandler to the system for it handle cookie. Before Java 6, there is no CookieHandler implementation in the JRE, you have to write your own. If you are on Java 6, you can do this,
CookieHandler.setDefault(new CookieManager());
URLConnection's cookie handling is really weak. It barely works. It doesn't handle all the cookie rules correctly. You should use Apache HttpClient if you are dealing with sensitive cookies like authentication.

I think server can't determine at the first request that a client does not support cookies. So, probably server sends redirects. Try to disable redirects:
uc.setInstanceFollowRedirects(false);
Then you will be able to get cookies from response and use them (if you need) on the next request.

uc.getHeaderFields()
// get cookie (set-cookie) here
URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2");
conn.addRequestProperty("Referer", "http://xxxx");
conn.addRequestProperty("Cookie", "...");

If you're trying to scrape large volumes of data after a login, you may even be better off with a scripted web scraper like WebHarvest (http://web-harvest.sourceforge.net/) I've used it to great success in some of my own projects.