I have been using a java code to retrieve an url content. The code does not work for https://www.amazon.es/. A similar python code does achieve retrieving an amazon url content.
The java code:
URL url = new URL(urlToScan);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream(), StandardCharsets.UTF_8));
StringBuilder builder = new StringBuilder();
for (String temp = reader.readLine(); temp != null; temp = reader.readLine())
builder.append(temp);
webpage = builder.toString();
The python code:
from urllib.request import urlopen
url = "https://www.amazon.es/"
page = urlopen(url)
html_bytes = page.read()
html = html_bytes.decode("utf-8")
print(html)
I searched amazon's html on my own looking for the used charset (in case it was a charset issue) and they are using charset="utf-8".As the html is 22,000+ lines long, I thought it could be some parsing error for long Strings. I also tried with a ByteArrayOutputStream and then instancing using String(byte[], charset) constructor.Java output:
?
Why is not java.net.URL retrieving properly the url content?
Maybe it's because of User-Agent. To set User-Agent, using URLConnection:
URL url = new URL("https://www.amazon.es/");
URLConnection connection = url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)");
BufferedInputStream bufferedInputStream = new BufferedInputStream(connection.getInputStream());
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(bufferedInputStream));
StringBuilder buffer = new StringBuilder();
String inputLine;
while ((inputLine = bufferedReader.readLine()) != null) {
buffer.append(inputLine).append("\n");
}
bufferedReader.close();
System.out.println(buffer.toString());
While Python's urllib should be using certain User-Agent by default.
Related
I'm trying to read the HTML from a particular URL and store it into a String for parsing. I referred to a previous post to help me out. When I print out what was read, all I get are special characters.
Here is my Java code (with try/catches left out) that reads from a URL and prints:
String path = "https://html1-f.scribdassets.com/913q5pjrsw60h9i4/pages/106-6b1bd15200.jsonp";
URL url = new URL(path);
InputStream in = url.openStream();
BufferedReader bw = new BufferedReader(new InputStreamReader(in, "UTF-8");
String line;
while ((line = bw.readLine()) != null) {
System.out.println(line);
}
Program output:
�ĘY106-6b1bd15200.jsonpmP�r� �Ƨ�!�%m�vD"��Ra*��w�%����ݳ�sβ��MK�d�9+%�m��l^��މ����:���� ���8B�Vce�.A*��x$FCo���a�b�<����Xy��m�c�>t����� �Z������Gx�o� �J���oKe�0�5�kGYpb�*l����+|�U���-�N3��jBp�R�z5Cۥjh��o�;�~)����~��)~ɮhy��<c,=;tHW���'�c�=~�w���
Expected output:
window.page106_callback(["<div class=\"newpage\" id=\"page106\" style=\"width: 902px; height:1273px\">\n<div class=image_layer style=\"z-index: 1\">\n<div class=ie_fix>\n<img class=\"absimg\" style=\"left:18px;top:27px;width:860px;height:1077px;clip:rect(1px 859px 1076px 1px)\" orig=\"http://html.scribd.com/913q5pjrsw60h9i4/images/106-6b1bd15200.jpg\"/>\n</div>\n</div>\n</div>\n\n"]);
At first, I thought it was an issue with permissions or something that somehow encrypted the stream, but my friend wrote a small Python script to do the same thing and it worked, thereby ruling this out. This is what he wrote:
import requests
link = 'https://html1-f.scribdassets.com/913q5pjrsw60h9i4/pages/106-
6b1bd15200.jsonp'
f = requests.get(link)
text = (f.text)
print(text)
So the question is, why is the Java version unable to correctly read and print from this particular URL? Note that I tried testing some other URLs from various websites and those worked fine. Maybe I should learn Python.
The response is gzip-encoded. You can do:
InputStream in = new GZIPInputStream(con.getInputStream());
#Maurice Perry is right, I tried with below code
String url = "https://html1-f.scribdassets.com/913q5pjrsw60h9i4/pages/106-6b1bd15200.jsonp";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(new GZIPInputStream(con.getInputStream())));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
System.out.println(response.toString());
I try to read the html content from an URL. When I wan't to print the content to the console "Umlaute" like ä, ö, ü are displayed wrong.
URL url = new URL("http://www.lauftreff.de/laeufe/halbmarathon-1-2017.html");
URLConnection conn = url.openConnection();
InputStreamReader input = new InputStreamReader(conn.getInputStream(),StandardCharsets.ISO_8859_1);
BufferedReader bi = new BufferedReader(input);
String inputLine;
while((inputLine = bi.readLine()) != null){
System.out.println(inputLine);
}
In the header of the html the information of the charset says ISO_8859_1. Also UTF-8 does not work.
Has anyone an Idea what to do?
In the website the Umlaute are decoded as HTML entities. So you would need to decode those. The code below should work, but it is untested.
URL url = new URL("http://www.lauftreff.de/laeufe/halbmarathon-1-2017.html");
URLConnection conn = url.openConnection();
InputStreamReader input = new InputStreamReader(conn.getInputStream(),StandardCharsets.ISO_8859_1);
BufferedReader bi = new BufferedReader(input);
String inputLine;
while((inputLine = bi.readLine()) != null){
inputLine = StringEscapeUtils.unescapeHtml4(inputLine);
System.out.println(inputLine);
}
My code looks like
URL oracle = new URL(calURL);
FileWriter overall = new FileWriter("overall.txt");
HttpURLConnection yc = (HttpURLConnection) oracle.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
overall.append("\n"+inputLine);
}
It seems it is returning only half of content .. Not getting the full content
Note : calURL is dynamically generated
calURL is taking much time to load. Before its my stream starts reading I guess. I included timeout before URL connection it is getting full data now.
I'm building a web crawler. Having read this I understand that DNS resolution is slow and so we should separate out the DNS Resolver.
So say that you have
String urlString http://google.com
you can then convert that into an ip by doing
URL url = new URL(urlString)
InetAddress ip = InetAddress.getByName(url.getHost());
But then how do you download the actual website itself?
With the url, we could just dow something like this:
String htmlDocumentString = new Scanner(new url.openStream(), "UTF-8").useDelimiter("\\A").next();
But if we want to used the resolved IP, do we have to manually reconstruct the URL with an ip? There is no url.setHost() method, it just seems kinda messy?
Reading from URL is simple :
public class URLReader {
public static void main(String[] args) throws Exception {
URL oracle = new URL("http://www.oracle.com/");
BufferedReader in = new BufferedReader(
new InputStreamReader(oracle.openStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
Taken from : http://docs.oracle.com/javase/tutorial/networking/urls/readingURL.html
Try this instead:
URL oracle = new URL("http://www.oracle.com/");
URLConnection urlc = oracle.openConnection();
urlc.setDoInput(true);
urlc.setRequestProperty("Accept", "text/text");
InputStream inputStream = urlc.getInputStream();
String myString = IOUtils.toString(inputStream, "UTF-8");
... using IOUtils from Apache Commons above:
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream,%20java.lang.String)
Im on quite a basic level of android development.
I would like to get text from a page such as "http://www.google.com". (The page i will be using will only have text, so no pictures or something like that)
So, to be clear: I want to get the text written on a page into etc. a string in my application.
I tried this code, but im not even sure if it does what i want.
URL url = new URL(/*"http://www.google.com");
URLConnection connection = url.openConnection();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = "";
I cant get any text from it anyhow. How should I do this?
From the sample code you gave you are not even reading the response from the request. I would get the html with the following code
URL u = new URL("http://www.google.com");
URLConnection conn = u.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
conn.getInputStream()));
StringBuffer buffer = new StringBuffer();
String inputLine;
while ((inputLine = in.readLine()) != null)
buffer.append(inputLine);
in.close();
System.out.println(buffer.toString());
From there you would need to pass the string into some kind of html parser if you want only the text. From what I've heard JTidy would is a good library for this however I have never used any Java html parsing libraries.
You want to extract text from HTML file? You can make use of specialized tool such as the Jericho HTML parser library. I'm not sure if it can be used directly in Android app, it is quite big, but it is open source so you can make use of its code and take only what you need for your task.
Here is one way:
public String scrape(String urlString) throws Exception {
URL url = new URL(urlString);
URLConnection connection = url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(
connection.getInputStream()));
String line = null, data = "";
while ((line = reader.readLine()) != null) {
data += line + "\n";
}
return data;
}
Here is another.