I am trying to get an InputStream from a URL. The URL can be a opened from Firefox. It returns a json and I have installed an addon for viewing json in Firefox so I can view it there.
So I tried to get it from Java by:
URL url = new URL(urlString);
URLConnection urlConnection = url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
But it is throwing an IOException in urlConnection.getInputStream().
I also tried:
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
InputStream inputStream = url.openStream();
But no luck.
Any information is appreciable. Thanks in advance.
Thank you everybody. This is a weird problem but at last I solved it.
The URL I am requesting is
http://api.themoviedb.org/2.1/Movie.search/en/json/api_key/a nightmare on elm street
Now browser replaces the spaces between "a nightmare on elm street" by "%20" internally and parses. That is why the requested server can response by that request. But From Java I didn't replaced that spaces by "%20", so it turns into Bad Request, source.
Now it is working.
BufferedReader reader = new BufferedReader(new InputStreamReader(((HttpURLConnection) (new URL(urlString)).openConnection()).getInputStream(), Charset.forName("UTF-8")));
I had a similar issue and my url was:
http://www.itmat.upenn.edu/assets/user-content/documents/ITMAT17. October 10 2017_.pdf
which obviously contained spaces.
These caused java.io.IOException Server returned HTTP response code: 400 in the following code:
java.net.URL url = new URL(urlString);
java.io.InputStream in = url.openStream();
If you copy the above url and paste in browser, you will realize that browser adds '%20' for the spaces. So I did it manually with the following code and the problem is solved.
if(urlString.contains(" "))
urlString = urlString.replace(" ", "%20");
Complete code/answer should be:
if(urlString.contains(" "))
urlString = urlString.replace(" ", "%20");
java.net.URL url = new URL(urlString);
java.io.InputStream in = url.openStream();
are you setting up the connection correctly? here's some code that illustrates how to do this. Note that I am being lazy about exception handling here, this is not production quality code.
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class URLFetcher {
public static void main(String[] args) throws Exception {
URL myURL = new URL("http://www.paulsanwald.com/");
HttpURLConnection connection = (HttpURLConnection) myURL.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder results = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
results.append(line);
}
connection.disconnect();
System.out.println(results.toString());
}
}
encode the parameters in the URL as follows:
String messageText = URLEncoder.encode(messageText, "UTF-8");
I a encountered same error. In my case, it was because the sizjwt token in the header was larger than acceptable size by mule soft proxy. One option is to increase the size of acceptable header size in mule soft, or reduce the size of token by removing some of the permissions assigned to the user id
Related
I am using core java to retrieve single web page content as String using proxy.
private static HttpURLConnection getConnection(String urlStr) throws MalformedURLException, IOException {
URL url = new URL(urlStr);
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("216.56.48.118", 9000));
HttpURLConnection uc = (HttpURLConnection) url.openConnection(proxy);
return uc;
}
public static void printHomePageContent() throws Exception {
String queryParam = URLEncoder.encode(
"YBuD64pgMuP3hiqNmR4hB6H8xdAebeBPfdEPbdNUq3ptkbhSYGkdhuKaPCXE+KXT3unjfaI3tRJzQno10f/FiC7IzNAdhbrPK9d4smyxHpE=",
"UTF-8");
HttpURLConnection conn = getConnection(
"https://www.example.com?params=" + queryParam);
conn.setRequestMethod("GET");
conn.setReadTimeout(60 * 1000);
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.connect();
Reader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
for (int c; (c = in.read()) >= 0;) {
System.out.print((char) c);
}
}
NOTE: www.example.com is replaced with the actual url as I can't share it in public, all other things are exactly same as my original code.
PROBLEM: When I call printHomePageContent it's printing wrong page(same page if I don't send query param) content that means it's not considering params query parameter value as expected. While if If I hit the same URL on browser or POSTMAN(Rest Client), it's displaying right page. I am using proxy from browser as well using chrome extension.
I know you can't replicate the issue your own as I have replaced the URL, but the description I wrote is exactly what is happening. If anybody can suggest some hints based on their past experienced would be helpful.
Thanks in advance.
I'm writing a small crawler for sites in English only, and doing that by opening a URL connection. I set the encoding to utf-8 both on the request, and the InputStreamReader but I continue to get gobbledigook for some of the requests, while others work fine.
The following code represents all the research I did and advice out there. I have also tried changing URLConnection to HttpURLConnection with no luck. Some of the returned strings continue to look like this:
??}?r?H????P?n?c??]?d?G?o??Xj{?x?"P$a?Qt?#&??e?a#?????lfVx)?='b?"Y(defUeefee=??????.??a8??{O??????zY?2?M???3c??#
What am I missing?
My code:
public static String getDocumentFromUrl(String urlString) throws Exception {
String wholeDocument = null;
URL url = new URL(urlString);
URLConnection conn = url.openConnection();
conn.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
conn.setRequestProperty("Accept-Charset", "utf-8");
conn.setConnectTimeout(60*1000); // wait only 60 seconds for a response
conn.setReadTimeout(60*1000);
InputStreamReader isr = new InputStreamReader(conn.getInputStream(), "utf-8");
BufferedReader in = new BufferedReader(isr);
String inputLine;
while ((inputLine = in.readLine()) != null) {
wholeDocument += inputLine;
}
isr.close();
in.close();
return wholeDocument;
}
The server is sending the document GZIP compressed. You can set the Accept-Encoding HTTP header to make it send the document in plain text.
conn.setRequestProperty("Accept-Encoding", "identity");
Even so, the HTTP client class handles GZIP compression for you, so you shouldn't have to worry about details like this. What seems to be going on here is that the server is buggy: it does not send the Content-Encoding header to tell you the content is compressed. This behavior seems to depend on the User-Agent, so that the site works in regular web browsers but breaks when used from Java. So, setting the user agent also fixes the issue:
conn.setRequestProperty("User-Agent", "Mozilla/5.0"); // for example
I'm writing an Android app. Now I try to get JSON from an URL with this method:
public String getInfo(String adress) throws Exception {
URL url = new URL(adress);
HttpURLConnection uc = (HttpURLConnection) url.openConnection();
int status = uc.getResponseCode();
InputStream in = uc.getInputStream();
InputStreamReader inRead = new InputStreamReader(in);
BufferedReader br = new BufferedReader(inRead);
String line;
String result = "";
while ((line = br.readLine()) != null) {
result += line;
}
return result;
}
On this url: http://www.rtvlansingerland.nl/tag/nieuws/?json=get_posts this method is perfectly working while on this url: http://www.rtvlansingerland.nl/?json=get_post&id=24411 the status variable goes to 400 which gives a url not found exception and uc.getInputStream() returns a filenotfound error.
If I open the URL in my browser it returns perfectly valid JSON (checked with jsonlint).
Does anyone have an option what could be wrong?
Thx in advance.
First of all you should not use network connection on main thread and after that you should check whether the file exist or not on URL if exist then proceed
String album = "http://picasaweb.google.com/data/feed/api/user/"+email;
HttpURLConnection con = (HttpURLConnection) new URL(albumUrl).openConnection();
// request method, timeout and headers
con.setRequestMethod("GET") ;
con.setReadTimeout(15000);
con.setRequestProperty("Authorization", "GoogleLogin auth="+auth);
con.setRequestProperty("GData-Version", "2");
// set timeout and that we will process output
con.setReadTimeout(15000);
con.setDoOutput(true);
// connnect to url
con.connect();
// read output returned for url
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
Problem : Everytime i call con.getInputStream() it gives me file not found exception.
But when i load the same url in the desktop browser then it is displaying correct data.
I am confused why on android it is throwing exception.
Thanks in advance.
Did you get this? Maybe you just missed the https
below example uses default for authenticated user and the experimental fields list.
url = "https://picasaweb.google.com/data/feed/api/user/default?kind=album&access=public&fields="
+ URLEncoder
.encode("entry(title,id,gphoto:numphotosremaining,gphoto:numphotos,media:group/media:thumbnail)",
"UTF-8");
https://developers.google.com/picasa-web/docs/2.0/developers_guide_protocol#ListAlbums
I'm a bit new to Java and more to connections stuff with it. I'm trying to create a program to connect to a website ("www.buybackprofesional.com") where I would like to download pictures and get some text from cars (after the login you have to enter a plate number to access a car's file).
This is what I have right now, but it always says that the session has expired, I need a way to login using the username and password of the mainpage, am I right? can someone give me some advice? Thanks
Note: I want to do it in Java, maybe I was not clear in the question.
//URL web = new URL("http://www.buybackprofesional.com/DetallePeri.asp?mat=9073FCV&fec=27/07/2010&tipo=C&modelo=4582&Foto=0");
URL web = new URL("http://www.buybackprofesional.com/");
HttpURLConnection con = (HttpURLConnection) web.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; JVM)");
con.setRequestProperty("Pragma", "no-cache");
con.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
A colleage helped me with this so I'll post the code that works:
public static URLConnection login(String _url, String _username, String _password) throws IOException, MalformedURLException {
String data = URLEncoder.encode("Usuario", "UTF-8") + "=" + URLEncoder.encode(_username, "UTF-8");
data += "&" + URLEncoder.encode("Contrase", "UTF-8") + "=" + URLEncoder.encode(_password, "UTF-8");
// Send data
URL url = new URL(_url);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
return conn;
}
This will submit the form info on the page I need and after that, using cookies I can stay connected!
To connect to a website using java consider using httpunit or httpcore (offered by apache). They handle sessions much better then you (or I) could do on your own.
Edit: Fixed the location of the link. Thanks for the correction!