I can easily get authenticated in this site
via browser. But there's NO POSSIBLE way to connect to it using Java. I always get 401 error. I tried to connect via Jsoup, and HTMLUnit and still nothing.. Anyone has at least some explanation as to why this is happening?
Oh, by the way... In just trying to connect to the page WITHOUT authenticating, I get the 401 error. Which in my opinion is pretty damn weird.
Basic URLConnection = new URLConnection(URL); combo (Native Java API) didn't work.
Jsoup with Document = Response.parse(); didn't work
HTMLUnit using WebClient didn't work either
Any of the above trying to emulate a browser didn't work as well
The 401 error is an authentication error. This is likely because the agent you are specifying (or failing to specify) when you connect isn't considered valid. (There could be more than that, depending on how that site is set up.)
You can specify a user agent in JSoup, see this answer to that question on SO.
If you use HttpURLConnection, maybe it helps...
java.net.Authenticator.setDefault( new Authenticator() {
protected java.net.PasswordAuthentication getPasswordAuthentication() {
return new java.net.PasswordAuthentication ("user", "pass".toCharArray());
}
});
Related
Due to company policies, I cannot share the complete code here, but here is where the code fails:
try(InputStream in = new URL("URL with image file").openStream()){
Files.copy(in, Paths.get("file://newfile.png"));
} catch (Exception e) {
}
The Exception is:
java.net.ConnectException: Connection timed out: connect
The things I have tried:
Adding proxy and port number before openstream method
Adding User agent
Using system proxy settings
Note: I'm trying to save an image from the URL. I have tried accessing the URL with my personal system and it's working. With my company system, it's not, even after adding the proxy.
Could someone please tell me why this is happening?
Sorry, it was my mistake. If anyone is seeing this:
The problem came from how I used proxy. If you are NOT able to connect to a website which is otherwise accessible via browser in same system, then you need a proxy in your java code.
For example:
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("google.com", 80));
Hope this helps. If you have any doubts regarding this, I'll be happy to help.
Note: Don't use "https" for proxy. Give only domain name.
I know that there are a lot of questions regarding this subject, but I still don't get it.
I want to get the current URL from my browser in my program. What do I need to succeed?
The connection has to be made with HTTP Connection? A proxy server would help me more? because i have to filter those URLs. Help me please, I am so confused.
request.getRequestURL();
will get you the URL from HttpServletRequest. More documentation can be found in https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html
I am developing an android application and want to connect to a REST service using URLConnection. The resource is protected using Digest Authentication. I can successfully access the REST service via the browser. I do not want to use HttpClient because URLConnection is the more current way to connect for the future.
But when I try to access the resource via the code below, I get a 401 Error with an java.io.FileNotFoundException. I have researched this thoroughly but no success, solutions appreciated.
Note: My REST service has been developed using Grails, I am running this code in the android emulator with code developed in Eclipse on the windows 64 bit OS.
CODE
URL myurl = new URL("http://10.0.2.2:8080/MyREST/customers");
HttpURLConnection myurlConnection = (HttpURLConnection) myurl.openConnection();
String basicAuth = "Basic " + (Base64.encode(userpass.getBytes(),android.util.Base64.DEFAULT)).toString();
myurlConnection.setRequestProperty ("Authorization", basicAuth);
try {
int responseCode1 = ((HttpURLConnection) myurlConnection).getResponseCode();
Log.i("MyLongOperation", "Check connection" +Integer.toString(responseCode1) );
InputStream in = new BufferedInputStream(myurlConnection.getInputStream());
readStream(in);
}
finally {
myurlConnection.disconnect();
}
I have also tried setting authentication at a global level with no effect
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(username, password.toCharArray());
}
}
);
I have referred to this article - with no success. Connecting to remote URL which requires authentication using Java
If the resource is protected by "Digest" then sending a "Basic" authorization scheme in your code will not work because the server would not recognize it.
Secondly, by using a "preemptive" authentication, setting the Authorization header w/o it being requested is kind of a security hole. You will be sending information to the server that it has not requested.
Thirdly, the "Authenticator.setDefault" most likely will not be requested as there was some significant back-and-forth caused by MicroSoft's implementation of HTTP digest authentication (YMMV may vary on my recollection of this). As such, Sun/Oracle decided to leave this behavior disabled by default as per this document.
That said, you may be better off looking into utilizing the Apache HTTP client bundled with Android to do this. There is a bundled implementation Digest Authentication included. There is an example of "preemptive" digest authentication located here.
Couple of caveats to be aware of:
Pay CLOSE attention to the "HttpHost" stored in "target" - this MUST MATCH EXACTLY the host name, protocol port, and protocol scheme used in the URL being retrieved.
The example provided is for HTTP Client 4.2.x. I am not 100% sure of the version included in Android but you should be able to locate working examples.
Update Submitter has provided additional comments with regard to the statement that it is recommended by Google to use the HttpURLConnection with articles here and here.
While I trust the statements made by Tim Bray with regard to the reasoning as to why you should be using the provided HttpURLConnection object for performing these calls, I do not agree that they should be immediately accepted on face value.
There is no indication as to the level of support of digest authentication provided by the implementation in Android. As I mentioned earlier, the HttpURLConnection does not support immediately as it has been known to be buggy.
If you are decided that you are going to use HTTP Digest Authentication, regardless of the fact that it has been deemed unstable by the majority of the community, I would attempt to set the following system properties in your application as EARLY as possible during the Android lifecycle:
http.auth.digest.validateServer="true"
http.auth.digest.validateProxy="true"
By doing so, this should enable the digest authentication scheme.
I am, again, going to re-iterate that the Apache HTTP Client bundled with Android was developed and designed specifically to address short-comings of the basic Java HttpURLConnection, providing much a much broader and robust client for dealing with HTTP(s) based data streams.
I would recommend trying a couple of things as well, see if you can configure your container to provide "Basic" authentication protection. The other, more complex option, would be to possibly provide X.509 Certificate Based authentication.
I hope that this clarification helps you get to your goal.
change
'Base64.encode ...'
to
'Base64.encodeToString ...'
I have a Gingerbread Android application that I'm porting to ICS. This application communicates with a web server sending HTTP POST. My application runs fine on Gingerbread. However, I have been experiencing problems after porting it to ICS. I found out that the POST requests my application is sending are actually changed to GET.
The funny thing is, Android actually reports that POST is indeed used.
URL oURL = new URL(sURL);
HttpURLConnection oHTTPConnection = (HttpURLConnection)(oURL.openConnection());
oHTTPConnection.setDoInput(true);
oHTTPConnection.setDoOutput(true);
oHTTPConnection.setRequestMethod("POST");
// set headers...
int nResponse = oHTTPConnection.getResponseCode();
String sMethod = oHTTPConnection.getRequestMethod(); // Returns "POST"
However, the server would say otherwise. I modified the web server application to check the request method it receives and then put this value in the response body it sends back to my Android application. And what I receive on my Android application is "GET".
I have tried using HttpClient with HttpPost but I get the same issue.
As I mentioned, I didn't have this problem in Gingerbread. Also, I've read from another thread here a similar (but opposite) problem that also only happens in ICS: Android 4.0 ICS turning HttpURLConnection GET requests into POST requests.
Has anyone else experienced this? Can anyone help me solve this?
Thanks in advance!
Rai
Try follow this answer: https://stackoverflow.com/a/8799198/372076
I've found that pre-ICS one could get away with making a body-less
POST without providing a Content-Length value, however post-ICS you
must set Content-Length: 0.
Don't know if you already found a fix for this, but i was having the same problem and just found a work around. In my case it was an issue on server's side with Apache's redirection. I was doing:
Url url = new Url("http://aaaa.bbbb.com/");
Changed to:
Url url = new Url("http://aaaa.bbbb.com/index.php");
Somehow the redirection was turning my POST into a GET with no parameters.
I was expecting this code to return a 404, however it produces the output :
"Response code is 200"
Would it be possible to learn how to differentiate between existent and non-existent web pages . . . thanks so much,
try
{
// create the HttpURLConnection
URL url = new URL("http://www.thisurldoesnotexist");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println("Response code is " + connection.getResponseCode());
}
EDIT: I see you've call openConnection() but not connect() - could that be the problem? I would expect getResponseCode() to actually make the request if it hasn't already, but it's worth just trying that...
That suggests you've possible got some DNS resolver which redirects to a "helper" (spam) page, or something like that.
The easiest way to see exactly what's going on here is to use Wireshark - have that up and capturing traffic (HTTP-only, to make life easier) and then run your code. You should be able to see what's going on that way.
Note that I wouldn't have expected a 404 - because that would involve being able to find a web server to talk to to start with. If you're trying to go to a host which doesn't involve, there shouldn't be an HTTP response at all. I'd expect connect() to throw an exception.
try adding a "connection.connect();" or look at the contents returned...
it could be a dns issue, ie: your dns is being sent to a parking place... for example: freedns does this.
You could:
Resolve the IP from the host of the page
Try to connect to port 80 on the resolved IP using plain sockets
This is a bit low level however and will add complexity since you will need to make a simple GET request through the socket. Then validate the response so you're sure that its actually a HTTP server running on port 80.
NMap might be able to help you here.
Ideally you should be getting this error:
java.net.UnknownHostException: www.thisurldoesnotexist
But it looks like your URL is resolved by you DNS provider.
For instance on my company's network running your code with URI "http://profile/" displays
the employee profile.
Please also check etc.home file if you are on windows to check if any settings have been changed.
Like #spgennard - I think this is most likely a DNS issue.
The URL you have chosen is owned by a DNS speculator.
The URL you have chosen is "parked" by a DNS provider.
Your ISP is messing with your DNS results to send your browser to some search page.
It is also possible that you are accessing the web via a proxy, and the proxy is doing something strange.
The way to diagnose this is to look at the other information in the HTTP responses you are getting, particularly the response body.