Develop Reference

How to put a HTML link in URLEncoder message

I am trying to create a String in java with a link in it.
The message reads like
String message ="Something happened please go back to Home and start again";
This message is ultimately encoded using
String msg = URLEncoder.encode(message,"UTF-8");
and displayed on a JSP page, but this message when rendered on JSP page looks like this.
Something happened please go back to Home and start again
Plain String without a actual link in it.
I am not sure how to embed a link in a String message in Java.

This seems a lot like the issue discussed on this link:
https://www.talisman.org/~erlkonig/misc/lunatech%5Ewhat-every-webdev-must-know-about-url-encoding/#Donotuse%7B%7Bjava.net.URLEncoder%7D%7Dor%7B%7Bjava.net.URLDecoder%7D%7DforwholeURLs
The article says:
Do not use java.net.URLEncoder or java.net.URLDecoder for whole URLs
We are not kidding. These classes are not made to encode or decode
URLs, as their API documentation clearly says:
Utility class for HTML form encoding. This class contains static
methods for converting a String to the
application/x-www-form-urlencoded MIME format. For more information
about HTML form encoding, consult the HTML specification.
This is not about URLs. At best it resembles the query part encoding.
It is wrong to use it to encode or decode entire URLs. You would think
the standard JDK had a standard class to deal with URL encoding
properly (part by part, that is) but either it is not there, or we
have not found it, which lures a lot of people into using URLEncoder
for the wrong purpose.
I have formatted the relevant code from the page above and adjusted it with regards to your code:
String pathSegment = "link.com";
String message ="Something happened please go back to Home and start again";

how can I clean and sanitize a url submitted by a user for redisplay in java?

I want a user to be able to submit a url, and then display that url to other users as a link.
If I naively redisplay what the user submitted, I leave myself open to urls like
http://somesite.com' ><script>[any javacscript in here]</script>
that when I redisplay it to other users will do something nasty, or at least something that makes me look unprofessional for not preventing it.
Is there a library, preferably in java, that will clean a url so that it retains all valid urls but weeds out any exploits/tomfoolery?
Thanks!

URLs having ' in are perfectly valid. If you are outputting them to an HTML document without escaping, then the problem lies in your lack of HTML-escaping, not in the input checking. You need to ensure that you are calling an HTML encoding method every time you output any variable text (including URLs) into an HTML document.
Java does not have a built-in HTML encoder (poor show!) but most web libraries do (take your pick, or write it yourself with a few string replaces). If you use JSTL tags, you get escapeXml to do it for free by default:
ok
Whilst your main problem is HTML-escaping, it is still potentially beneficial to validate that an input URL is valid to catch mistakes - you can do that by parsing it with new URL(...) and seeing if you get a MalformedURLException.
You should also check that the URL begins with a known-good protocol such as http:// or https://. This will prevent anyone using dangerous URL protocols like javascript: which can lead to cross-site-scripting as easily as HTML-injection can.

I think what you are looking for is output encoding. Have a look at OWASP ESAPI which is tried and tested way to perform encoding in Java.
Also, just a suggestion, if you want to check if a user is submitting malicious URL, you can check that against Google malware database. You can use SafeBrowing API for that.

You can use apache validator URLValidator
UrlValidator urlValidator = new UrlValidator(schemes);
if (urlValidator.isValid("http://somesite.com")) {
//valid
}

Retrieving URL's from webpage in java

I have the most basic java code to do a http request and it works fine. I request data and a ton of html comes back. I want to retrieve all the url's from that page and list them. For a simple first test i made it look like this:
int b = line.indexOf("http://",lastE);
int e = line.indexOf("\"", b);
This works but as you can imagine it's horrible and only works in 80% of the cases. The only alternative i could come up with myself sounded slow and stupid. So my question is pretty mutch do i go from
String html
to
List<Url>
?

Pattern p = Pattern.compile("http://[\w^\"]++");
Matcher m = p.matcher(yourFetchedHtmlString);
while (m.find()) {
nextUrl=m.group();//Do whatever you want with it
}
You may also have to tweak the regexp, as i have just written it without testing. This should be a very fast way to fetch urls.

I would try a library like HTML Parser to parse the html string and extract all url tags from it.

Your thinking is good, you just missing some parts.
Yous should add some known extension for urls.
like .html .aspx .php .htm .cgi .js .pl .asp
And if you like images too then add .gif .jpg .png
I think your doing it the best just need to add more extensions checking.
If you can post the full method code, i will be happy to help you make it better.

Selenium 2: Detect content type of link destinations

I am using the Selenium 2 Java API to interact with web pages. My question is: How can i detect the content type of link destinations?
Basically, this is the background: Before clicking a link, i want to be sure that the response is an HTML file. If not, i need to handle it in another way. So, let's say there is a download link for a PDF file. The application should directly read the contents of that URL instead of opening it in the browser.
The goal is to have an application which automatically knows wheather the current location is an HTML, PDF, XML or whatever to use appropriate parsers to extract useful information out of the documents.
Update
Added bounty: Will reward it to the best solution which allows me to get the content type of a given URL.

As Jochen suggests, the way to get the Content-type without also downloading the content is HTTP HEAD, and the selenium webdrivers does not seem to offer functionality like that. You'll have to find another library to help you with fetching the content type of an url.
A Java library that can do this is Apache HttpComponents, especially HttpClient.
(The following code is untested)
HttpClient httpclient = new DefaultHttpClient();
HttpHead httphead = new HttpHead("http://foo/bar");
HttpResponse response = httpclient.execute(httphead);
BasicHeader contenttypeheader = response.getFirstHeader("Content-Type");
System.out.println(contenttypeheader);
The project publishes JavaDoc for HttpClient, the documentation for the HttpClient interface contains a nice example.

You can figure out the content type will processing the data coming in.
Not sure why you need to figure this out first.
If so, use the HEAD method and look at the Content-Type header.

You can retrieve all the URLs from the DOM, and then parse the last few characters of each URL (using a java regex) to determine the link type.
You can parse characters proceeding the last dot. For example, in the url http://yoursite.com/whatever/test.pdf, extract the pdf, and enforce your test logic accordingly.
Am I oversimplifying your problem?

Running a JavaScript command from MATLAB to fetch a PDF file

I'm currently writing some MATLAB code to interact with my company's internal reports database. So far I can access the HTML abstract page using code which looks like this:
import com.mathworks.mde.desk.*;
wb=com.mathworks.mde.webbrowser.WebBrowser.createBrowser;
wb.setCurrentLocation(ReportURL(8:end));
pause(1);
s={};
while isempty(s)
s=char(wb.getHtmlText);
pause(.1);
end
desk=MLDesktop.getInstance;
desk.removeClient(wb);
I can extract out various bits of information from the HTML text which ends up in the variable s, however the PDF of the report is accessed via what I believe is a JavaScript command (onClick="gotoFulltext('','[Report Number]')").
Any ideas as to how I execute this JavaScript command and get the contents of the PDF file into a MATLAB variable?
(MATLAB sits on top of Java, so I believe a Java solution would work...)

I think you should take a look at the JavaScript that is being called and see what the final request to the webserver looks like.
You can do this quite easily in Firefox using the FireBug plugin.
https://addons.mozilla.org/en-US/firefox/addon/1843
Once you have found the real server request then you can just request this URL or post to this URL instead of trying to run the JavaScript.

Once you have gotten the correct URL (a la the answer from pjp), your next problem is to "get the contents of the PDF file into a MATLAB variable". Whether or not this is possible may depend on what you mean by "contents"...
If you want to get the raw data in the PDF file, I don't think there is a way currently to do this in MATLAB. The URLREAD function was the first thing I thought of to read content from a URL into a string, but it has this note in the documentation:
s = urlread('url') reads the content
at a URL into the string s. If the
server returns binary data, s will
be unreadable.
Indeed, if you try to read a PDF as in the following example, s contains some text intermingled with mostly garbage:
s = urlread('http://samplepdf.com/sample.pdf');
If you want to get the text from the PDF file, you have some options. First, you can use URLWRITE to save the contents of the URL to a file:
urlwrite('http://samplepdf.com/sample.pdf','temp.pdf');
Then you should be able to use one of two submissions on The MathWorks File Exchange to extract the text from the PDF:
Extract text from a PDF document by Dimitri Shvorob
PDF Reader by Tom Gaudette
If you simply want to view the PDF, you can just open it in Adobe Acrobat with the OPEN function:
open('temp.pdf');

wb=com.mathworks.mde.webbrowser.WebBrowser.createBrowser;
wb.executeScript('javascript:alert(''Some code from a link'')');
desk=com.mathworks.mde.desk.MLDesktop.getInstance;
desk.removeClient(wb);