Develop Reference

Why am I getting 403 status code in Java after a while?

When I try to check status codes within sites I face off 403 response code after a while. First when I run the code every sites send back datas but after my code repeat itself with Timer I see one webpage returns 403 response code. Here is my code.
public class Main {
public static void checkSites() {
Timer ifSee403 = new Timer();
try {
File links = new File("./linkler.txt");
Scanner scan = new Scanner(links);
ArrayList<String> list = new ArrayList<>();
while(scan.hasNext()) {
list.add(scan.nextLine());
}
File linkStatus = new File("LinkStatus.txt");
if(!linkStatus.exists()){
linkStatus.createNewFile();
}else{
System.out.println("File already exists");
}
BufferedWriter writer = new BufferedWriter(new FileWriter(linkStatus));
for(String link : list) {
try {
if(!link.startsWith("http")) {
link = "http://"+link;
}
URL url = new URL(link);
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection http = (HttpURLConnection)url.openConnection();
http.setRequestMethod("HEAD");
http.setConnectTimeout(5000);
http.setReadTimeout(8000);
int statusCode = http.getResponseCode();
if (statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Hello, here we go again");
}
http.disconnect();
System.out.println(link + " " + statusCode);
writer.write(link + " " + statusCode);
writer.newLine();
} catch (Exception e) {
writer.write(link + " " + e.getMessage());
writer.newLine();
System.out.println(link + " " +e.getMessage());
}
}
try {
writer.close();
} catch (Exception e) {
System.out.println(e.getMessage());
}
System.out.println("Finished.");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
public static void main(String[] args) throws Exception {
Timer myTimer = new Timer();
TimerTask sendingRequest = new TimerTask() {
public void run() {
checkSites();
}
};
myTimer.schedule(sendingRequest,0,150000);
}
}
How can I solve this? Thanks
Edited comment:
I've added http.disconnect(); for closing connection after checked status codes.
Also I've added
if(statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Test message);
}
But it didn't work. Compiler returned current thread is not owner error. I need to fix this and change 200 with 403 and say ifSee403.wait(5000) and try it again the status code.

One "alternative" - by the way - to IP / Spoofing / Anonymizing would be to (instead) try "obeying" what the security-code is expecting you to do. If you are going to write a "scraper", and are aware there is a "bot detection" that doesn't like you debugging your code while you visit the site over and over and over - you should try using the HTML Download which I posted as an answer to the last question you asked.
If you download the HTML and save it (save it to a file - once an hour), and then write you HTML Parsing / Monitoring Code using the HTML contents of the file you have saved, you will (likely) be abiding by the security-requirements of the web-site and still be able to check availability.
If you wish to continue to use JSoup, that A.P.I. has an option for receiving HTML as a String. So if you use the HTML Scrape Code I posted, and then write that HTML String to disk, you can feed that to JSoup as often as you like without causing the Bot Detection Security Checks to go off.
If you play by their rules once in a while, you can write your tester without much hassle.
import java.io.*;
import java.net.*;
...
// This line asks the "url" that you are trying to connect with for
// an instance of HttpURLConnection. These two classes (URL and HttpURLConnection)
// are in the standard JDK Package java.net.*
HttpURLConnection con = (HttpURLConnection) url.openConnection();
// Tells the connection to use "GET" ... and to "pretend" that you are
// using a "Chrome" web-browser. Note, the User-Agent sometimes means
// something to the web-server, and sometimes is fully ignored.
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Chrome/61.0.3163.100");
// The classes InputStream, InputStreamReader, and BufferedReader
// are all JDK 1.0 package java.io.* classes.
InputStream is = con.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
StringBuffer sb = new StringBuffer();
String s;
// This reads each line from the web-server.
while ((s = br.readLine()) != null) sb.append(s + "\n");
// This writes the results from the web-server to a file
// It is using classes java.io.File and java.io.FileWriter
File outF = new File("SavedSite.html");
outF.createNewFile();
FileWriter fw = new FileWriter(outF);
fw.write(sb.toString());
fw.close();
Again, this code is very basic stuff that doesn't use any special JAR Library Code at all. The next method uses the JSoup library (which you have explicitly requested - even though I don't use it... It is just fine!) ... This is the method "parse" which will parse the String you have just saved. You may load this HTML String from disk, and send it to JSoup using:
Method Documentation: org.jsoup.Jsoup.parse​(File in, String charsetName, String baseUri)
If you wish to invoke JSoup just pass it a java.io.File instance using the following:
File f = new File("SavedSite.html");
Document d = Jsoup.parse(f, "UTF-8", url.toString());
I do not think you need timers at all...
AGAIN: If you are making lots of calls to the server. The purpose of this answer is to show you how to save the response of the server to a file on disk, so you don't have to make lots of calls - JUST ONE! If you restrict your calls to the server to once per hour, then you will (likely, but not a guarantee) avoid getting a 403 Forbidden Bot Detection Problem.

Uploadcare: Error getting file group info

I set up a free account with Uploadcare a couple days ago. I've been trying to test out there REST API. Hence, I have to use this URL:
https://api.uploadcare.com/files/:uuid/
with a GET request.
I've tried sending a request from JAVA.
public String getResponse(String urlToRead) {
URL url;
HttpURLConnection conn;
BufferedReader rd;
String line;
String result = "";
try {
url = new URL(urlToRead);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
//add request header
conn.setRequestProperty("Accept", "application/vnd.uploadcare-v0.3+json");
conn.setRequestProperty("Date", "Fri, 09 Feb 2013 01:08:47 -0000");
conn.setRequestProperty("Authorization", "Uploadcare.Simple publicKey:privateKey");
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = rd.readLine()) != null) {
result += line;
}
rd.close();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
public static void main(String args[]) {
UploadCareTesting c = new UploadCareTesting();
String url = "https://api.uploadcare.com/files/4100d0d6-fa27-475d-9f7a-ef218d718b5e~1/";
System.out.println(c.getHTML(url));
}
Yet I keep getting this Error:
java.io.FileNotFoundException:
https://api.uploadcare.com/files/4100d0d6-fa27-475d-9f7a-ef218d718b5e~1/
I looked at the code trying to see if I have any errors. However, the Uploadcare documentation says that I can just got to https://api.uploadcare.com. and make sample request, but the page comes back with a error saying "Something isn't working. This is our fault, not yours.
We’re sorry."
Anybody else have this problem, and found how to get around it? (I have messaged Uploadcare support and I haven't heard anything back yet)

There are two parts to this:
4100d0d6-fa27-475d-9f7a-ef218d718b5e~1 is not file UUID, but file group UUID. So you want to request group info at https://api.uploadcare.com/groups/4100d0d6-fa27-475d-9f7a-ef218d718b5e~1/ instead.
This seems to be a bug, and I'd like to ask you to contact us directly to figure this out.
p.s.: our REST API was not down

Source not Found Error in Eclipse BlackBerry App

I'm new to BlackBerry application development. I have one silly error while during HttpConnection in httpcon.getResponseCode() method giving source not found error.
Please, can any one figure out this error?
Here is my method:
net.rim.device.api.io.transport.ConnectionFactory cf = new net.rim.device.api.io.transport.ConnectionFactory();
httpConn = (HttpConnection) cf.getConnection(url).getConnection();
httpConn.setRequestMethod(HttpConnection.POST);
httpConn.setRequestProperty("User-Agent",
"Profile/MIDP-2.0 Confirguration/CLDC-1.0");
httpConn.setRequestProperty("Accept_Language", "en-US");
httpConn.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
httpConn.setRequestProperty("Content-Length",
Integer.toString(postData.length));
os = httpConn.openOutputStream();
os.write(("LoginID=yahoo#sol.com&Password=yah123")
.getBytes("UTF-8"));
os.flush();
os.close();
try {
responseCode = httpConn.getResponseCode();
} catch (IOException ex1) {
//check if it's eof, if yes retrieve code again
if (-1 != ex1.getMessage().indexOf("EOF")) {
try {
responseCode = httpConn.getResponseCode();
} catch (IOException ex2) {
System.out.println(ex2.getMessage());
// handle exception
}
} else {
System.out.println(ex1.getMessage());
// handle exception
}
}
int status = httpConn.getResponseCode();
if (status == HttpConnection.HTTP_OK) {
InputStream input = httpConn.openInputStream();
byte[] bytes = IOUtilities.streamToBytes(input);
StringBuffer raw = new StringBuffer(new String(bytes));
raw.insert(0, "bytes received]\n");
raw.insert(0, bytes.length);
raw.insert(0, '[');
url = raw.toString();
input.close();
} else {
url = "response code = " + status;
}
httpConn.close();
} catch (IOCancelledException e) {
System.out.println(e.toString());
return "";
} catch (IOException e) {
return "";
}
return "";
}
Update: I am not trying to step into getResponseCode(). Eclipse is stopping execution at that point, and showing the Source Not Found error.

#Nate yes we are step into method that time only we are getting Run Time Error.Error is displayed When Getresponsecode() is called
Does your eclipse error look like this?
getResponseCode() is a method in the HttpConnection class. This is RIM's code, not yours. You normally shouldn't need to step into that code. Just step over that line while debugging. The only thing you should want to see is the result of that method, not what happens inside.
Even if net_rim_api.jar is included in your project, that simply gives you the binary version of RIM's classes, including HttpConnection. It doesn't provide the Java source code for that class. In order to step into a method, you would need to have the source code, too.

I have found my mistake.When calling HttpConnection. getResponseCode() in that URL String we have to add deviceside=true then only getresponsecode() will be called without throwing any http exceptions.
For Example:
httpClient ht = new httpClient();
String str = ht.getHttpClientResponse("https://www.google.co.in;deviceside=true",post);
Note:
(From #Nate Comment)
we don't always want deviceside=true in your connection strings. It depends on whether you're running on a device or a simulator, and what kind of network's available.

Cannot write output after reading input

I'm writing a program that connects to a servlet thanks to a HttpURLConnection but I stuck while checking the url
public void connect (String method) throws Exception {
server = (HttpURLConnection) url.openConnection ();
server.setDoInput (true);
server.setDoOutput (true);
server.setUseCaches (false);
server.setRequestMethod (method);
server.setRequestProperty ("Content-Type", "application / xml");
server.connect ();
/*if (server.getResponseCode () == 200)
{
System.out.println ("Connection OK at the url:" + url);
System.out.println ("------------------------------------------- ------- ");
}
else
System.out.println ("Connection failed");
}*/
I got the error :
java.net.ProtocolException: Cannot write output after reading input.
if i check the url with the code in comments but it work perfectly without it
unfortunately, I need to check the url so i think the problem comes from the getResponseCode method but i don t know how to resolve it
Thank you very much

The HTTP protocol is based on a request-response pattern: you send your request first and the server responds. Once the server responded, you can't send any more content, it wouldn't make sense. (How could the server give you a response code before it knows what is it you're trying to send?)
So when you call server.getResponseCode(), you effectively tell the server that your request has finished and it can process it. If you want to send more data, you have to start a new request.
Looking at your code you want to check whether the connection itself was successful, but there's no need for that: if the connection isn't successful, an Exception is thrown by server.connect(). But the outcome of a connection attempt isn't the same as the HTTP response code, which always comes after the server processed all your input.

I think the exception is not due toprinting url. There should some piece of code which is trying to write to set the request body after the response is read.
This exception will occur if you are trying to get HttpURLConnection.getOutputStream() after obtaining HttpURLConnection.getInputStream()
Here is the implentation of sun.net.www.protocol.http.HttpURLConnection.getOutputStream:
public synchronized OutputStream getOutputStream() throws IOException {
try {
if (!doOutput) {
throw new ProtocolException("cannot write to a URLConnection"
+ " if doOutput=false - call setDoOutput(true)");
}
if (method.equals("GET")) {
method = "POST"; // Backward compatibility
}
if (!"POST".equals(method) && !"PUT".equals(method) &&
"http".equals(url.getProtocol())) {
throw new ProtocolException("HTTP method " + method +
" doesn't support output");
}
// if there's already an input stream open, throw an exception
if (inputStream != null) {
throw new ProtocolException("Cannot write output after reading
input.");
}
if (!checkReuseConnection())
connect();
/* REMIND: This exists to fix the HttpsURLConnection subclass.
* Hotjava needs to run on JDK.FCS. Do proper fix in subclass
* for . and remove this.
*/
if (streaming() && strOutputStream == null) {
writeRequests();
}
ps = (PrintStream)http.getOutputStream();
if (streaming()) {
if (strOutputStream == null) {
if (fixedContentLength != -) {
strOutputStream =
new StreamingOutputStream (ps, fixedContentLength);
} else if (chunkLength != -) {
strOutputStream = new StreamingOutputStream(
new ChunkedOutputStream (ps, chunkLength), -);
}
}
return strOutputStream;
} else {
if (poster == null) {
poster = new PosterOutputStream();
}
return poster;
}
} catch (RuntimeException e) {
disconnectInternal();
throw e;
} catch (IOException e) {
disconnectInternal();
throw e;
}
}

I have this problem too, what surprises me is that the error is caused by my added code System.out.println(conn.getHeaderFields());
Below is my code:
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setRequestMethod("POST");
configureConnection(conn);
//System.out.println(conn.getHeaderFields()); //if i comment this code,everything is ok, if not the 'Cannot write output after reading input' error happens
conn.connect();
OutputStream os = conn.getOutputStream();
os.write(paramsContent.getBytes());
os.flush();
os.close();

I had the same problem.
The solution for the problem is that you need to use the sequence
openConnection -> getOutputStream -> write -> getInputStream -> read
That means..:
public String sendReceive(String url, String toSend) {
URL url = new URL(url);
URLConnection conn = url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.sets...
OutputStreamWriter out = new OutputStreamWriter(conn.getOutputStream());
out.write(toSend);
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String receive = "";
do {
String line = in.readLine();
if (line == null)
break;
receive += line;
} while (true);
in.close();
return receive;
}
String results1 = sendReceive("site.com/update.php", params1);
String results2 = sendReceive("site.com/update.php", params2);
...

Closing URLConnection and InputStream correctly?

I've seen many different examples of using HttpURLConnection + InputStream, and closing them (or not closing them) after use. This is what I came up with to make sure everything is closed after finished, whether there's an error or not. Is this valid?:
HttpURLConnection conn = null;
InputStream is = null;
try {
URL url = new URL("http://example.com");
// (set connection and read timeouts on the connection)
conn = (HttpURLConnection)url.openConnection();
is = new BufferedInputStream(conn.getInputStream());
doSomethingWithInputStream(is);
} catch (Exception ex) {
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
}
}
if (conn != null) {
conn.disconnect();
}
}
Thanks

Yep.. Doing the end part in finally would be best idea because if code fails somewhere, program won't reach till .close(), .disconnect() statements that we keep before catch statements...
If the code fails somewhere and exception is thrown in between of the program, still finally get executed regardless of exception thrown...

There is also the new (with Java 7) 'try()' technique
try (OutputStream os = http.getOutputStream()) {
os.write(out);
}
Basically, it will auto-close anything in the try() statement, regardless of whether it is successful or not.