Get the true length of a url server content - java

Using the http Header I'm trying to get the length of a website call from the header using the http connection of java.. But the problem is that my result is not compatible with the true result found in differents httpheader online tools.
This is my Code :
public static URLConnection getConnection(String url) throws IOException {
URL obj = new URL(url);
URLConnection conn = obj.openConnection();
// conn.setRequestProperty("User-Agent",
// "Mozilla/5.0 (Linux ; Android 6.0.1 ; Nexus 5X Build/MMB29P) AppleWebKit/537.36" +
// " (KHTML, par exemple Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible ; Googlebot/2.1 ; +http://www.google.com/bot.html)");
return conn;
}
I call the contentLegth field from header but I get just as result 1037Bytes and the true value have to be : 128241 Kb
public static Boolean PageWeightFromServer(URLConnection c) throws IOException {
int length = c.getContentLength(); // The weight of the page received in the responseHeader
System.err.println(length +"Bytes");
return (length>500);
}

Related

how to fix "Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL" Error?

I'm using Django server and getting the error Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL"
public static void main(String[] args) throws Exception {
// check if the user enter the right args from the command line.
if(args.length != 2){
System.out.println("Usage: java Reverse "
+ "http://<location of your servlet/script> "
+ "string_to_reverse");// display the error.
System.exit(1); // exit the program.
}
/**the sting that will be reversed may contain spaces or other
* non-alphanumeric characters. These characters must be
* encoded because the string is processed on its way to the server.
* the URLEncoder class methods encode the characters.*/
String stringToReverse = URLEncoder.encode(args[1], "UTF-8");
// create object for the specified url for the command line.
URL url = new URL(args[0]);
// sets the connection so that it can write to it.
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
connection.setReadTimeout(5000);
connection.setConnectTimeout(5000);
// The program then creates an output stream on the connection
// and opens an OutputSteamWriter on it;
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
// the program writes the required information t the output
// stream and closes the stream.
out.write("string = " + stringToReverse);
out.close();
// read the specified url.
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String decodeString;
while((decodeString = in.readLine()) != null ){
System.out.println(decodeString);
}
in.close();
}
Error:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://127.0.0.1:8000/test
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1913)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
at Reverse.main(Reverse.java:52)
I also tried the follow things to fix the error but still not working.
connection.setRequestProperty("http.agent", "Chrome");
connection.setRequestProperty("User-agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.setRequestProperty("User-agent", "Mozilla/5.0");
connection.setRequestProperty("User-agent", "Mozilla");
Can somebody help me fix this?

Java GET request with HttpURLConnection not working as expected

I am using core java to retrieve single web page content as String using proxy.
private static HttpURLConnection getConnection(String urlStr) throws MalformedURLException, IOException {
URL url = new URL(urlStr);
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("216.56.48.118", 9000));
HttpURLConnection uc = (HttpURLConnection) url.openConnection(proxy);
return uc;
}
public static void printHomePageContent() throws Exception {
String queryParam = URLEncoder.encode(
"YBuD64pgMuP3hiqNmR4hB6H8xdAebeBPfdEPbdNUq3ptkbhSYGkdhuKaPCXE+KXT3unjfaI3tRJzQno10f/FiC7IzNAdhbrPK9d4smyxHpE=",
"UTF-8");
HttpURLConnection conn = getConnection(
"https://www.example.com?params=" + queryParam);
conn.setRequestMethod("GET");
conn.setReadTimeout(60 * 1000);
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.connect();
Reader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
for (int c; (c = in.read()) >= 0;) {
System.out.print((char) c);
}
}
NOTE: www.example.com is replaced with the actual url as I can't share it in public, all other things are exactly same as my original code.
PROBLEM: When I call printHomePageContent it's printing wrong page(same page if I don't send query param) content that means it's not considering params query parameter value as expected. While if If I hit the same URL on browser or POSTMAN(Rest Client), it's displaying right page. I am using proxy from browser as well using chrome extension.
I know you can't replicate the issue your own as I have replaced the URL, but the description I wrote is exactly what is happening. If anybody can suggest some hints based on their past experienced would be helpful.
Thanks in advance.

HTTPS request hangs only on Android APIs below 20 even with a connect timeout set

I made a java.net.HttpURLConnection and it hang on the line connection.connect() even though I’ve set a connect timeout. “b4 connect” gets logged and “after connect” never gets logged. I’ve tested on API 21 and above and things work, but I get this issue with my test on API 16-19. Here is my code below. The request is using https and the backend uses a standard nginx https configuration.
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
try {
connection.setRequestMethod("GET");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
connection.setConnectTimeout('\uea60');
connection.setReadTimeout('\uea60');
connection.setInstanceFollowRedirects(false);
Log.d(TAG, "b4 connect");
connection.connect();
Log.d(TAG, "after connect");
if(connection.getResponseCode() == 200) {
return IOUtils.toString(connection.getInputStream(), "UTF-8");
}
} catch (Exception var) {
throw new Exception(var.getMessage());
} finally {
connection.disconnect();
}
return null;
You're specifying timouts using some unicode charachters. Please try regular numbers like that:
connection.setConnectTimeout(30000);
сonnection.setReadTimeout(30000);
Keep in mind "after connect" will not be logged on timeout. Exception will be thrown instead.

Publicly accessible URL throwing IOException

I would like to access the link http://www.nation.co.ke/business/seedsofgold/Egg-imports-from-Uganda-hatch-big-losses-for-farmers/-/2301238/2897930/-/dpeqesz/-/index.html
The link is publicly accessible, and can even load using curl
But in Java code it throws Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.nation.co.ke/business/seedsofgold/Egg-imports-from-Uganda-hatch-big-losses-for-farmers/-/2301238/2897930/-/dpeqesz/-/index.html
This is the code:
/**
*
* #param url the HTML page
* #throws IOException
*/
public static String getPage(String url) throws IOException {
URL u = new URL(url);
URLConnection conn = u.openConnection();
String mime = conn.getContentType();
if( !StringUtils.containsIgnoreCase(mime, "text/html") ) {
return null; // don't continue if not HTML
}
else {
// read the response body, using BufferedReader for performance
InputStream in = conn.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in, Charset.defaultCharset()));
int n = 0, totalRead = 0;
char[] buf = new char[1024];
StringBuilder content = new StringBuilder();
// read until EOF or first 16384 characters
while (totalRead < 16384 && (n = reader.read(buf, 0, buf.length)) != -1) {
content.append(buf, 0, n);
totalRead += n;
}
reader.close();
}
The error is thrown at:
InputStream in = conn.getInputStream();
The same code works fine with other URLs.
try to add
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
to your connection right after URLConnection conn = u.openConnection();. Many websites block the site access when no correct agent is set.
If you are getting HTTP 403 status code, it means access to the resource identified by the URL is forbidden for some reason.
A web server may return a 403 Forbidden HTTP status code in response to a request from a client for a web page or resource to indicate that the server can be reached and understood the request, but refuses to take any further action.
You can refer HTTP 403 status code

java http 403 error on upload

I am definitely not a computer guru. :)
I get an http 403 error when trying to upload a file on my website through an applet.
This means that it is forbidden. Maybe it is a regular behavior, for uploading through http protocol may be not allowed. Is it true? Then how to do it? I would like my applet to upload little files on a specific folder of the server.
here is the code :
private static String folder = "http://..." //URL of the folder to upload to
public void saveScore(Item hs) { //Item is a serializable object to save
String filename = "s"+Integer.toString(hs.getScore())+".sco" ; // name of the file
System.out.println("*** Trying to save file to : " + filename) ;
try {
//setting connection
HttpURLConnection con = (HttpURLConnection) new URL(folder+"/"+filename).openConnection() ;
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestProperty ("Content-Type", "multipart/form-data");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
con.setChunkedStreamingMode(1024);
con.setRequestMethod("PUT") ;
ObjectOutputStream oos = new ObjectOutputStream(con.getOutputStream()) ;
//uploading
oos.writeObject(hs) ;
oos.flush() ;
oos.close();
//getting answer
DataInputStream is = new DataInputStream(con.getInputStream());
String s = is.readLine();
is.close();
System.out.println("** Answer **");
System.out.println(s) ;
} catch (IOException e) {
e.printStackTrace(System.out) ; // gives me a 403 error
}
}
Thanks for helping...
Go through the link. Hope it will help you.
http://stackoverflow.com/questions/1599018/java-applet-to-upload-a-file?rq=1

Categories

Resources