I would like to access the link http://www.nation.co.ke/business/seedsofgold/Egg-imports-from-Uganda-hatch-big-losses-for-farmers/-/2301238/2897930/-/dpeqesz/-/index.html
The link is publicly accessible, and can even load using curl
But in Java code it throws Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.nation.co.ke/business/seedsofgold/Egg-imports-from-Uganda-hatch-big-losses-for-farmers/-/2301238/2897930/-/dpeqesz/-/index.html
This is the code:
/**
*
* #param url the HTML page
* #throws IOException
*/
public static String getPage(String url) throws IOException {
URL u = new URL(url);
URLConnection conn = u.openConnection();
String mime = conn.getContentType();
if( !StringUtils.containsIgnoreCase(mime, "text/html") ) {
return null; // don't continue if not HTML
}
else {
// read the response body, using BufferedReader for performance
InputStream in = conn.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in, Charset.defaultCharset()));
int n = 0, totalRead = 0;
char[] buf = new char[1024];
StringBuilder content = new StringBuilder();
// read until EOF or first 16384 characters
while (totalRead < 16384 && (n = reader.read(buf, 0, buf.length)) != -1) {
content.append(buf, 0, n);
totalRead += n;
}
reader.close();
}
The error is thrown at:
InputStream in = conn.getInputStream();
The same code works fine with other URLs.
try to add
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
to your connection right after URLConnection conn = u.openConnection();. Many websites block the site access when no correct agent is set.
If you are getting HTTP 403 status code, it means access to the resource identified by the URL is forbidden for some reason.
A web server may return a 403 Forbidden HTTP status code in response to a request from a client for a web page or resource to indicate that the server can be reached and understood the request, but refuses to take any further action.
You can refer HTTP 403 status code
Related
Im trying to download a file, but for some people running it, the server is giving error 403.
try (BufferedInputStream in = new BufferedInputStream(new URL("http://example.com/test.zip").openStream());
FileOutputStream fileOutputStream = new FileOutputStream("./test.zip")) {
byte dataBuffer[] = new byte[1024];
int bytesRead;
while ((bytesRead = in.read(dataBuffer, 0, 1024)) != -1) {
fileOutputStream.write(dataBuffer, 0, bytesRead);
}
} catch (IOException e18) {
error("Error: "+e18);
e18.printStackTrace();
return false;
}
While researching this error(403 - Forbidden), I found multiple posts saying that a user agent needs to be specified, I believe this may be the case, I am not sure how to easily add a user agent to my code.
Thank You in advance!
URL tgtUrl = new URL("http://example.com/test.zip");
java.net.URLConnection c = tgtUrl .openConnection();
c.setRequestProperty("User-Agent", " USER AGENT STRING HERE ");
ReadableByteChannel tar = Channels.newChannel(c.getInputStream());
OR
URL tgtUrl = new URL("http://example.com/test.zip");
java.net.URLConnection c = tgtUrl .openConnection();
c.setRequestProperty("User-Agent", " USER AGENT STRING HERE ");
BufferedReader br = new BufferedReader(new InputStreamReader(c.getInputStream()));
System.out.println(br.readLine());
Ref : Java: Download from an URL
Might be duplicate question
Simply adding:
System.setProperty("http.agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7");
Fixed it!
Thanks everyone!
I'm using Django server and getting the error Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL"
public static void main(String[] args) throws Exception {
// check if the user enter the right args from the command line.
if(args.length != 2){
System.out.println("Usage: java Reverse "
+ "http://<location of your servlet/script> "
+ "string_to_reverse");// display the error.
System.exit(1); // exit the program.
}
/**the sting that will be reversed may contain spaces or other
* non-alphanumeric characters. These characters must be
* encoded because the string is processed on its way to the server.
* the URLEncoder class methods encode the characters.*/
String stringToReverse = URLEncoder.encode(args[1], "UTF-8");
// create object for the specified url for the command line.
URL url = new URL(args[0]);
// sets the connection so that it can write to it.
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
connection.setReadTimeout(5000);
connection.setConnectTimeout(5000);
// The program then creates an output stream on the connection
// and opens an OutputSteamWriter on it;
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
// the program writes the required information t the output
// stream and closes the stream.
out.write("string = " + stringToReverse);
out.close();
// read the specified url.
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String decodeString;
while((decodeString = in.readLine()) != null ){
System.out.println(decodeString);
}
in.close();
}
Error:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://127.0.0.1:8000/test
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1913)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
at Reverse.main(Reverse.java:52)
I also tried the follow things to fix the error but still not working.
connection.setRequestProperty("http.agent", "Chrome");
connection.setRequestProperty("User-agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.setRequestProperty("User-agent", "Mozilla/5.0");
connection.setRequestProperty("User-agent", "Mozilla");
Can somebody help me fix this?
Using the http Header I'm trying to get the length of a website call from the header using the http connection of java.. But the problem is that my result is not compatible with the true result found in differents httpheader online tools.
This is my Code :
public static URLConnection getConnection(String url) throws IOException {
URL obj = new URL(url);
URLConnection conn = obj.openConnection();
// conn.setRequestProperty("User-Agent",
// "Mozilla/5.0 (Linux ; Android 6.0.1 ; Nexus 5X Build/MMB29P) AppleWebKit/537.36" +
// " (KHTML, par exemple Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible ; Googlebot/2.1 ; +http://www.google.com/bot.html)");
return conn;
}
I call the contentLegth field from header but I get just as result 1037Bytes and the true value have to be : 128241 Kb
public static Boolean PageWeightFromServer(URLConnection c) throws IOException {
int length = c.getContentLength(); // The weight of the page received in the responseHeader
System.err.println(length +"Bytes");
return (length>500);
}
I'm sending a GET request to a URL in a for loop with changing one parameter from java code. Now my request is working fine for 5-6 times but after a certain point the status code for my connection becomes 404 and all the way down from that point my requests are failing. but when I try to connect to the URL from browser with following the sites direction(clicking the buttons that gets me to the same url) it works without any problem.
Here is the function I'm using for Httpconnection:
private static void getFile(String name) throws MalformedURLException, InterruptedException, UnsupportedEncodingException{
String url = "http://<this is not showed>"+URLEncoder.encode(name.trim(),"UTF-8")+".mp3";
URL obj = new URL (url);
try {
HttpURLConnection connection = (HttpURLConnection) obj.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", USER_AGENT);
connection.setDoOutput(true);
connection.connect();
//saving the file returned from request
FileOutputStream fileOutputStream = new FileOutputStream(new File(System.getProperty("user.dir") + "/audios/"+name+".mp3"));
InputStream inputStream = connection.getInputStream();
byte[] buffer = new byte[1024];
int len1 = 0;
while((len1 = inputStream.read(buffer)) > 0){
fileOutputStream.write(buffer, 0, len1);
}
fileOutputStream.close();
connection.disconnect();
break;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
and here is the part where I'm using the getFile funciton:
for (int i = 0; i < length; i++) {
getFile(names[i]);
}
Here is the output of the loop:
HTML connection status: 200
a| File downloaded succesfully.
HTML connection status: 200
ab| File downloaded succesfully.
HTML connection status: 200
ba| File downloaded succesfully.
HTML connection status: 200
dî| File downloaded succesfully.
HTML connection status: 200
jur| File downloaded succesfully.
HTML connection status: 200
küs| File downloaded succesfully.
HTML connection status: 200
ban| File downloaded succesfully.
HTML connection status: 404
java.io.FileNotFoundException:
and it keeps like that.
I have the following code to perform a GET request on the following URL:
http://rt.hnnnglmbrg.de/server.php/someReferenceNumber
However, here is my output from Logcat:
java.io.FileNotFoundException: http://rt.hnnnglmbrg.de/server.php/6
Why does it return 404 when the URL is clearly valid?
Here is my connect code:
/**
* Performs an HTTP GET request that returns base64 data from the server
*
* #param ref
* The Accident's reference
* #return The base64 data from the server.
*/
public static String performGet(String ref) {
String returnRef = null;
try {
URL url = new URL(SERVER_URL + "/" + ref);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder builder = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
builder.append(line);
}
returnRef = builder.toString();
} catch (IOException e) {
e.printStackTrace();
}
return returnRef;
}
When you request the URL, it actually return HTTP code 404 which mean not found. If you have control to the PHP script, set the header to 200 to indicate file is found.
You are getting a 404, as said above. To avoid an exception, try something like this:
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
con.connect () ;
int code = con.getResponseCode() ;
if (code == HttpURLConnection.HTTP_NOT_FOUND)
{
// Handle error
}
else
{
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
// etc...
}
Never trust what you see in your browser. Always try to mimic your request using something like curl, and you'll clearly see that you're getting an HTTP 404 response code.
java.net will translate the HTTP 404 code to a FileNotFoundException
curl -v http://rt.hnnnglmbrg.de/server.php/4
* About to connect() to rt.hnnnglmbrg.de port 80 (#0)
* Trying 217.160.115.112... connected
* Connected to rt.hnnnglmbrg.de (217.160.115.112) port 80 (#0)
> GET /server.php/4 HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: rt.hnnnglmbrg.de
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Mon, 11 Jun 2012 07:34:55 GMT
< Server: Apache
< X-Powered-By: PHP/5.2.17
< Transfer-Encoding: chunked
< Content-Type: text/html
<
* Connection #0 to host rt.hnnnglmbrg.de left intact
* Closing connection #0
0
From the javadocs at http://docs.oracle.com/javase/6/docs/api/java/net/HttpURLConnection.html
Returns the error stream if the connection failed but the server sent useful data nonetheless. The typical example is when an HTTP server responds with a 404, which will cause a FileNotFoundException to be thrown in connect, but the server sent an HTML help page with suggestions as to what to do.