Develop Reference

A page returns 200 instead of 401 when login is required

I am tasked with checking whether some URLs are working correctly, I'm using Java to make HTTP get request to get the response code.
So what I did was this.
URL u = new URL("some URL");
HttpURLConnection huc = (HttpURLConnection) u.openConnection();
huc.setRequestMethod("GET");
huc.connect();
int code = huc.getResponseCode();
System.out.println(code + " " + huc.getURL());
The Problem: Some sites require you to login to access the page, but the page doesn't return a 401 code, but 200. Note that the web page doesn't show up until a username and password are provided. It asks for authentication in a pop up window.
So how do I catch these kind of links?
Also, how can I identify if a webpage shows a login page like http://www.example.com/login/? Is it sufficient to just check the URL for the word “login”?

There's no universal way to deal with this. You have to know how the site you're using does authentication - 401? separate login page? multi-factor auth (ie: using RSA token)? Checking for the substring "login" in the URL is a possible way of handling some, but not enough for a general way.
For example, a 401 will only happen when using basic authentication (or when trying to access protected resources directly). There's a lot of other ways to do auth
John sums up the issue quite well in his comment:
If you have to deal with pages that roll their own custom authentication, then it follows that you probably have to write your own custom code to accommodate them. Depending on how the relevant sites work, you might be able to bypass authentication by sending an appropriate cookie in your request, as if you had already authenticated, or by some similar means

How to display MJPEG-stream in Vaadin (Java)?

I`m looking for a way to display a MJPEG-stream (from ip cam) in my vaadin application.
My problem is the necessary authentication to start the stream.
Really easy solution to simply get the stream:
String url = "...urlgoeshere/video.mjpg";
Image image = new Image();
image.setSource(new ExternalResource(url));
content.addComponent(image);
It works like a charm if I allow anonymous connections to the camera, but thats not really my goal.
I tried to authenticate with:
Authenticator.setDefault(new MyAuthenticator(username, password));
but it doesn`t affect the request.
If I manually set up a request like:
String url = "...urlgoeshere/video.mjpg";
Authenticator.setDefault(new MyAuthenticator("username", "password"));
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
the authentication works, but this way I have to handle all the images by myself.
Are there any better ways to display a stream which requires authentication?

I think you should be able to use the username & password in the URL of the external resource.
https://foo:password#example.com
But this depends on the protocol the webcam uses for authentification,
and you will also have the username and password "visible" in the
html / javascript of your application.
And one more note:
Microsoft did disable this some time ago, so if your users are using IE,
this won't work.

Getting the actual text response of a web page in java

I'm using java and trying to get the content of a website so that I can analyze the text on the page, however every time that I "GET" a response from the server, it is from a login page rather than the website page that I am looking at.
I am logged into the website on all my browsers, but my application is not able to see the page as if it were me.
I also tried to use an API called "Yandex" --> http://api.yandex.com/rca/
as a work-around. But when I call the page from Yandex (which would get its content) I only see information based on the login page returned.
Can anyone give me a direction to investigate? I would like to be able to get one item on the page of a website that I work for, but it doesn't seem possible.
m_strseedpath = "http://myUrl.com/mypage.html"; //not https
URLConnection connection = new URL("http://rca.yandex.com/?key={MyActualKeyNotThisText}&url=" + m_strSeedUrlPath + "").openConnection();
connection.setRequestProperty("Accept-Charset", "UTF-8");
InputStream response = connection.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(response, writer, "UTF-8");
String strString = writer.toString();
System.out.println(strString);

The URLConnection object will connect to the page but in a different session. You would have to programmaticaly log in from your Java code.
Create a URLConnection object to the login page, POST the user name and password, receive the content getting the InputStream from the URLConnection object, and finally create a new connection to the page you wish to analyze. You'd have to also work with cookies in order to view the second page.
Hope this helps!

The URL that you are trying to access has access restricted via login. Even if you are logged in via your browser you wont be able to access the page from your Java application because the browser has an Authenticated Session with the target website. The same session is not visible to your Java Application.
You would have to research into the ways to login to the website and then get the page content.

Where does Java store remembered HTTP basic auth passwords and how do you delete them?

I have a Java Swing app that launches by web start and accesses data files on a server through a URL. The files are served by Apache2, with HTTP basic Auth. Java pops up a dialog box prompting for a login, and that works fine.
The trouble comes when a user has checked "save this password in your password list". Then the password changes or was incorrect in the first place and you're stuck. It's apparently not smart enough to give you another chance. If your saved login fails you get a 401 error and that's it.
So, where is it storing saved passwords and how do you delete them?
The code involved looks like this:
// uri is a String
URL url = new URL(uri);
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
// check HTTP response code
int responseCode = urlConnection.getResponseCode();
if (responseCode != HttpURLConnection.HTTP_OK)
throw new IOException("\nHTTP response code: " + responseCode);
// read the file
BufferedReader reader = new BufferedReader
(new InputStreamReader (urlConnection.getInputStream ()));
... etc ...
That code works fine, except in this situation where the user has saved a bad password, in which case you get a 401.
My understanding is that Java WebStart puts its hooks into the java.net classes, so that you get things like the password prompt, which you wouldn't get by running the same code from the command line or from your IDE. So, this question is really about that behavior.
Thanks!

No Code? Now you get a vague answer. Depending on your HttpClient, it's probably stored in the cookies or something. Re-initializing your HttpClient would be a great first debugging step. If that doesn't work, posting a little code here would be very helpful.

In windows the passwords are saved in below file.
{user_name}\AppData\LocalLow\Sun\Java\Deployment\security\auth.dat
You can delete this file to clear the saved passwords.

java, android, resolve an url, get redirected uri

I have an authentication protected url : www.domain.com/alias
that when requested will return another url: www.another.com/resource.mp4 (not protected)
I would like to know if exists a method in Java that will return the real url from a given one.
Something like: second = resolve(first)
I'm thinking of loading the first and try to read into the response maybe the location attribute, but since I'm not a java guru I would like to know if Java already faces this.

This is a problem i used to have concerning URL redirects. Try the following code:
URL url = new URL(url);
HttpURLConnection ucon = (HttpURLConnection) url.openConnection();
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
URLConnection conn = secondURL.openConnection();
The "magic" here happens in these 2 steps:
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
By default InstanceFollowRedirects are set to true, but you want to set it to false to capture the second URL. To be able to get that second URL from the first URL, you need to get the header field called "Location".

I have eliminated this issue on sites where we have a MikroTik router by using a Layer 7 protocol filter as shown below. This doesn't help the devices off the WiFi network (obviously) but at least gives them some reprieve when they are connected to home and/or work WiFi networks.
Firstly, create the protocol definition:
/ip firewall layer7-protocol
add comment="Frigging javascript redirects on chrome browsers" \
name=Javascript_Redirect \
regexp="^.+(spaces.slimspot.com|mostawesomeoffers.com).*\$"
Now, to actually filter this traffic out
/ip firewall filter
add action=drop chain=forward comment=\
"Block and log Javascript_Redirect L7 Protocol" layer7-protocol=\
Javascript_Redirect log=yes log-prefix=JSredirect_
Other firewalls that have Layer 7 filtering capacity could also block these redirects in a similar way.

If you are using Ktor:
import io.ktor.client.statement.*
val resp = HttpClient.get<HttpResponse>(urlString = yourUrl)
val redirectedUrl = resp.request.url