Develop Reference

Why am I getting 403 status code in Java after a while?

When I try to check status codes within sites I face off 403 response code after a while. First when I run the code every sites send back datas but after my code repeat itself with Timer I see one webpage returns 403 response code. Here is my code.
public class Main {
public static void checkSites() {
Timer ifSee403 = new Timer();
try {
File links = new File("./linkler.txt");
Scanner scan = new Scanner(links);
ArrayList<String> list = new ArrayList<>();
while(scan.hasNext()) {
list.add(scan.nextLine());
}
File linkStatus = new File("LinkStatus.txt");
if(!linkStatus.exists()){
linkStatus.createNewFile();
}else{
System.out.println("File already exists");
}
BufferedWriter writer = new BufferedWriter(new FileWriter(linkStatus));
for(String link : list) {
try {
if(!link.startsWith("http")) {
link = "http://"+link;
}
URL url = new URL(link);
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection http = (HttpURLConnection)url.openConnection();
http.setRequestMethod("HEAD");
http.setConnectTimeout(5000);
http.setReadTimeout(8000);
int statusCode = http.getResponseCode();
if (statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Hello, here we go again");
}
http.disconnect();
System.out.println(link + " " + statusCode);
writer.write(link + " " + statusCode);
writer.newLine();
} catch (Exception e) {
writer.write(link + " " + e.getMessage());
writer.newLine();
System.out.println(link + " " +e.getMessage());
}
}
try {
writer.close();
} catch (Exception e) {
System.out.println(e.getMessage());
}
System.out.println("Finished.");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
public static void main(String[] args) throws Exception {
Timer myTimer = new Timer();
TimerTask sendingRequest = new TimerTask() {
public void run() {
checkSites();
}
};
myTimer.schedule(sendingRequest,0,150000);
}
}
How can I solve this? Thanks
Edited comment:
I've added http.disconnect(); for closing connection after checked status codes.
Also I've added
if(statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Test message);
}
But it didn't work. Compiler returned current thread is not owner error. I need to fix this and change 200 with 403 and say ifSee403.wait(5000) and try it again the status code.

One "alternative" - by the way - to IP / Spoofing / Anonymizing would be to (instead) try "obeying" what the security-code is expecting you to do. If you are going to write a "scraper", and are aware there is a "bot detection" that doesn't like you debugging your code while you visit the site over and over and over - you should try using the HTML Download which I posted as an answer to the last question you asked.
If you download the HTML and save it (save it to a file - once an hour), and then write you HTML Parsing / Monitoring Code using the HTML contents of the file you have saved, you will (likely) be abiding by the security-requirements of the web-site and still be able to check availability.
If you wish to continue to use JSoup, that A.P.I. has an option for receiving HTML as a String. So if you use the HTML Scrape Code I posted, and then write that HTML String to disk, you can feed that to JSoup as often as you like without causing the Bot Detection Security Checks to go off.
If you play by their rules once in a while, you can write your tester without much hassle.
import java.io.*;
import java.net.*;
...
// This line asks the "url" that you are trying to connect with for
// an instance of HttpURLConnection. These two classes (URL and HttpURLConnection)
// are in the standard JDK Package java.net.*
HttpURLConnection con = (HttpURLConnection) url.openConnection();
// Tells the connection to use "GET" ... and to "pretend" that you are
// using a "Chrome" web-browser. Note, the User-Agent sometimes means
// something to the web-server, and sometimes is fully ignored.
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Chrome/61.0.3163.100");
// The classes InputStream, InputStreamReader, and BufferedReader
// are all JDK 1.0 package java.io.* classes.
InputStream is = con.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
StringBuffer sb = new StringBuffer();
String s;
// This reads each line from the web-server.
while ((s = br.readLine()) != null) sb.append(s + "\n");
// This writes the results from the web-server to a file
// It is using classes java.io.File and java.io.FileWriter
File outF = new File("SavedSite.html");
outF.createNewFile();
FileWriter fw = new FileWriter(outF);
fw.write(sb.toString());
fw.close();
Again, this code is very basic stuff that doesn't use any special JAR Library Code at all. The next method uses the JSoup library (which you have explicitly requested - even though I don't use it... It is just fine!) ... This is the method "parse" which will parse the String you have just saved. You may load this HTML String from disk, and send it to JSoup using:
Method Documentation: org.jsoup.Jsoup.parse​(File in, String charsetName, String baseUri)
If you wish to invoke JSoup just pass it a java.io.File instance using the following:
File f = new File("SavedSite.html");
Document d = Jsoup.parse(f, "UTF-8", url.toString());
I do not think you need timers at all...
AGAIN: If you are making lots of calls to the server. The purpose of this answer is to show you how to save the response of the server to a file on disk, so you don't have to make lots of calls - JUST ONE! If you restrict your calls to the server to once per hour, then you will (likely, but not a guarantee) avoid getting a 403 Forbidden Bot Detection Problem.

How to go about debugging HttpURLConnection?

I'm trying to connect a java application to an external api for GuildWars2.
The link I am trying to test is:
http://api.guildwars2.com/v2/commerce/listings
A list of int IDs are returned when navigating to that page within a browser.
As a learning practice, I am trying to get that list of id's when running my java application.
I use the following code (hopefully it formats correct, currently on my phone, trying to program remotely to my desktop):
public class GuildWarsAPI
{
public static void main(String[] args)
{
GuildWarsAPI api = new GuildWarsAPI();
api.getAPIResponse("http://api.guildwars2.com/v2/commerce/listings");
}
public void getAPIResponse(String URLString)
{
URL url = null;
try {
url = new URL(URLString);
} catch (MalformedURLException e1) {
e1.printStackTrace();
}
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) url.openConnection();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
if (connection != null)
{
System.out.println("connection success");
connection.setRequestProperty("Accept", "application/json");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setReadTimeout(10000);
connection.setConnectTimeout(10000);
try {
/*BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder input = new StringBuilder();
String nextLine = null;
while ((nextLine = in.readLine()) != null)
{
System.out.println("adding output");
input.append(nextLine);
}*/
InputStream in = new BufferedInputStream(connection.getInputStream());
int b = 0;
while ((b = in.read()) != -1)
{
System.out.println("byte:" + b);
}
System.out.println("done");
}
catch (Exception e) {
e.printStackTrace();
}
finally {
connection.disconnect();
System.out.println("closed");
}
}
}
}
Upon running my class, it immediately prints out connection success, done, closed. It definitely isnt waiting for the timeouts, and i've been trying to play with that, the request header, and the DoInput/DoOutput. I stepped through it, and it appears as if it connects, and just doesnt receive any bytes of information back. (doesnt go into the while loop)
So, while my ultimate question is: How do I get the id's back like I expect?, my other question is: how can I figure out how to get the other id's back like I expect?

Your code is getting response code 302 Found. It should follow the Location: header to the new location, as followRedirects is true by default, but it isn't. The server is however returning a Location: header of https://api.guildwars2.com/v2/commerce/listings. I don't know why HttpURLConnection isn't following that, but the simple fix is to use https: in the original URL.
You're setting doOutput(true) but you aren't sending any output.
Your code is poorly structured. Code that depends on the success of code in a prior try block should be inside that same try block. I would have the method throw MalformedURLException and IOException and not have any internal try/catch blocks at all.

In my experience, wrestling with HttpUrlConnection is more trouble than it's worth.
It's hard to debug, hard to use, and provides very little support for complex http operations.
There are a bunch of better options.
My default choice is Apache HttpConponents Client (http://hc.apache.org/). It's not necessarily any better than all the other options, but it's quite well documented and widely used.

URLConnection Reading in as null in java

I'm a building a basic program to query Target's API with a store ID and Product ID which returns the aisle location. I think I'm using the URL constructor incorrectly, however (I've had trouble with it in the past and still don't fully understand them). Below is the code I have, redacted the API Key for obvious reasons. The URL I create is valid when put into a browser and no exceptions are thrown but at the the end when I print out the contents of the page it is null. What am I missing? Any help is really appreciated!
package productVerf;
import java.net.*;
import java.io.*;
public class Verify {
public static void main(String args[]) {
// first input is store id second input is product id
String productID = args[0];
String storeID = args[1];
String file = "/v2/products/storeLocations?productId=" + productID
+ "&storeId=" + storeID
+ "&storeId=694&key=REDACTED";
URL locQuery;
URLConnection lqConection = null;
try {
locQuery = new URL("http", "api.target.com", file);
lqConection = locQuery.openConnection();
} catch (IOException e) {
e.printStackTrace();
}
BufferedReader response;
String responseString = "";
try {
response = new BufferedReader(new InputStreamReader(
lqConection.getInputStream()));
while (response.readLine() != null) {
responseString += response.readLine();
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(responseString);
}
}

Maybe you are reading only even lines
you are reading a line twice? (in while statement...), it looks you reads the first line which is dropped in while condition test. If your response contains only one line, nothing will be readed
use this:
String line;
while ((line=response.readLine()) != null) {
responseString += line;
}

Bing API error 1002

First of all I'm sorry for my English.
I am developing an application in java and I want to use search Bing API, So I opened user-centered development of Bing (http://www.bing.com/dev/en-us/dev-center) and accept key number then I wrote the following code to get results Bing
String q = "http://api.bing.net/json.aspx?Appid=MyClientId=girls&sources=web&web.count=40&web.offset=41";
URL searchURL;
try {
searchURL = new URL(q);
HttpURLConnection httpURLConnection = (HttpURLConnection) searchURL.openConnection();
if(httpURLConnection.getResponseCode() == HttpURLConnection.HTTP_OK){
InputStreamReader inputStreamReader = new InputStreamReader(httpURLConnection.getInputStream());
BufferedReader bufferedReader = new BufferedReader(inputStreamReader, 8192);
String line = null;
String result = "";
while((line = bufferedReader.readLine()) != null){
result += line;
}
bufferedReader.close();
}
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Why do I get the following error 1002?
{"SearchResponse":{
"Version":"2.2",
"Query":{"SearchTerms":"girls"},
"Errors":[
{"Code":1002,
"Message":"Parameter has invalid value.",
"Parameter":"SearchRequest.AppId",
"Value":"MyClientId",
"HelpUrl":"http:\/\/msdn.microsoft.com\/en-us\/library\/dd251042.aspx"}]
}}

It looks like you've got a typo in the address
This looks very suspicious:
Appid=MyClientId=girls
You should see the documentation http://msdn.microsoft.com/en-us/library/dd250882.aspx, but I guess that you need to replace the MyClientId with something and also you haven't spearated the query and the clientId i.e. &q=girls
EDIT: You need to get the AppId somewhere Steps of creating appid for bing search
Here's some question which can help you:
Bing search API and Azure

Cannot write output after reading input

I'm writing a program that connects to a servlet thanks to a HttpURLConnection but I stuck while checking the url
public void connect (String method) throws Exception {
server = (HttpURLConnection) url.openConnection ();
server.setDoInput (true);
server.setDoOutput (true);
server.setUseCaches (false);
server.setRequestMethod (method);
server.setRequestProperty ("Content-Type", "application / xml");
server.connect ();
/*if (server.getResponseCode () == 200)
{
System.out.println ("Connection OK at the url:" + url);
System.out.println ("------------------------------------------- ------- ");
}
else
System.out.println ("Connection failed");
}*/
I got the error :
java.net.ProtocolException: Cannot write output after reading input.
if i check the url with the code in comments but it work perfectly without it
unfortunately, I need to check the url so i think the problem comes from the getResponseCode method but i don t know how to resolve it
Thank you very much

The HTTP protocol is based on a request-response pattern: you send your request first and the server responds. Once the server responded, you can't send any more content, it wouldn't make sense. (How could the server give you a response code before it knows what is it you're trying to send?)
So when you call server.getResponseCode(), you effectively tell the server that your request has finished and it can process it. If you want to send more data, you have to start a new request.
Looking at your code you want to check whether the connection itself was successful, but there's no need for that: if the connection isn't successful, an Exception is thrown by server.connect(). But the outcome of a connection attempt isn't the same as the HTTP response code, which always comes after the server processed all your input.

I think the exception is not due toprinting url. There should some piece of code which is trying to write to set the request body after the response is read.
This exception will occur if you are trying to get HttpURLConnection.getOutputStream() after obtaining HttpURLConnection.getInputStream()
Here is the implentation of sun.net.www.protocol.http.HttpURLConnection.getOutputStream:
public synchronized OutputStream getOutputStream() throws IOException {
try {
if (!doOutput) {
throw new ProtocolException("cannot write to a URLConnection"
+ " if doOutput=false - call setDoOutput(true)");
}
if (method.equals("GET")) {
method = "POST"; // Backward compatibility
}
if (!"POST".equals(method) && !"PUT".equals(method) &&
"http".equals(url.getProtocol())) {
throw new ProtocolException("HTTP method " + method +
" doesn't support output");
}
// if there's already an input stream open, throw an exception
if (inputStream != null) {
throw new ProtocolException("Cannot write output after reading
input.");
}
if (!checkReuseConnection())
connect();
/* REMIND: This exists to fix the HttpsURLConnection subclass.
* Hotjava needs to run on JDK.FCS. Do proper fix in subclass
* for . and remove this.
*/
if (streaming() && strOutputStream == null) {
writeRequests();
}
ps = (PrintStream)http.getOutputStream();
if (streaming()) {
if (strOutputStream == null) {
if (fixedContentLength != -) {
strOutputStream =
new StreamingOutputStream (ps, fixedContentLength);
} else if (chunkLength != -) {
strOutputStream = new StreamingOutputStream(
new ChunkedOutputStream (ps, chunkLength), -);
}
}
return strOutputStream;
} else {
if (poster == null) {
poster = new PosterOutputStream();
}
return poster;
}
} catch (RuntimeException e) {
disconnectInternal();
throw e;
} catch (IOException e) {
disconnectInternal();
throw e;
}
}

I have this problem too, what surprises me is that the error is caused by my added code System.out.println(conn.getHeaderFields());
Below is my code:
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setRequestMethod("POST");
configureConnection(conn);
//System.out.println(conn.getHeaderFields()); //if i comment this code,everything is ok, if not the 'Cannot write output after reading input' error happens
conn.connect();
OutputStream os = conn.getOutputStream();
os.write(paramsContent.getBytes());
os.flush();
os.close();

I had the same problem.
The solution for the problem is that you need to use the sequence
openConnection -> getOutputStream -> write -> getInputStream -> read
That means..:
public String sendReceive(String url, String toSend) {
URL url = new URL(url);
URLConnection conn = url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.sets...
OutputStreamWriter out = new OutputStreamWriter(conn.getOutputStream());
out.write(toSend);
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String receive = "";
do {
String line = in.readLine();
if (line == null)
break;
receive += line;
} while (true);
in.close();
return receive;
}
String results1 = sendReceive("site.com/update.php", params1);
String results2 = sendReceive("site.com/update.php", params2);
...