Develop Reference

Why am I getting 403 status code in Java after a while?

When I try to check status codes within sites I face off 403 response code after a while. First when I run the code every sites send back datas but after my code repeat itself with Timer I see one webpage returns 403 response code. Here is my code.
public class Main {
public static void checkSites() {
Timer ifSee403 = new Timer();
try {
File links = new File("./linkler.txt");
Scanner scan = new Scanner(links);
ArrayList<String> list = new ArrayList<>();
while(scan.hasNext()) {
list.add(scan.nextLine());
}
File linkStatus = new File("LinkStatus.txt");
if(!linkStatus.exists()){
linkStatus.createNewFile();
}else{
System.out.println("File already exists");
}
BufferedWriter writer = new BufferedWriter(new FileWriter(linkStatus));
for(String link : list) {
try {
if(!link.startsWith("http")) {
link = "http://"+link;
}
URL url = new URL(link);
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection http = (HttpURLConnection)url.openConnection();
http.setRequestMethod("HEAD");
http.setConnectTimeout(5000);
http.setReadTimeout(8000);
int statusCode = http.getResponseCode();
if (statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Hello, here we go again");
}
http.disconnect();
System.out.println(link + " " + statusCode);
writer.write(link + " " + statusCode);
writer.newLine();
} catch (Exception e) {
writer.write(link + " " + e.getMessage());
writer.newLine();
System.out.println(link + " " +e.getMessage());
}
}
try {
writer.close();
} catch (Exception e) {
System.out.println(e.getMessage());
}
System.out.println("Finished.");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
public static void main(String[] args) throws Exception {
Timer myTimer = new Timer();
TimerTask sendingRequest = new TimerTask() {
public void run() {
checkSites();
}
};
myTimer.schedule(sendingRequest,0,150000);
}
}
How can I solve this? Thanks
Edited comment:
I've added http.disconnect(); for closing connection after checked status codes.
Also I've added
if(statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Test message);
}
But it didn't work. Compiler returned current thread is not owner error. I need to fix this and change 200 with 403 and say ifSee403.wait(5000) and try it again the status code.

One "alternative" - by the way - to IP / Spoofing / Anonymizing would be to (instead) try "obeying" what the security-code is expecting you to do. If you are going to write a "scraper", and are aware there is a "bot detection" that doesn't like you debugging your code while you visit the site over and over and over - you should try using the HTML Download which I posted as an answer to the last question you asked.
If you download the HTML and save it (save it to a file - once an hour), and then write you HTML Parsing / Monitoring Code using the HTML contents of the file you have saved, you will (likely) be abiding by the security-requirements of the web-site and still be able to check availability.
If you wish to continue to use JSoup, that A.P.I. has an option for receiving HTML as a String. So if you use the HTML Scrape Code I posted, and then write that HTML String to disk, you can feed that to JSoup as often as you like without causing the Bot Detection Security Checks to go off.
If you play by their rules once in a while, you can write your tester without much hassle.
import java.io.*;
import java.net.*;
...
// This line asks the "url" that you are trying to connect with for
// an instance of HttpURLConnection. These two classes (URL and HttpURLConnection)
// are in the standard JDK Package java.net.*
HttpURLConnection con = (HttpURLConnection) url.openConnection();
// Tells the connection to use "GET" ... and to "pretend" that you are
// using a "Chrome" web-browser. Note, the User-Agent sometimes means
// something to the web-server, and sometimes is fully ignored.
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Chrome/61.0.3163.100");
// The classes InputStream, InputStreamReader, and BufferedReader
// are all JDK 1.0 package java.io.* classes.
InputStream is = con.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
StringBuffer sb = new StringBuffer();
String s;
// This reads each line from the web-server.
while ((s = br.readLine()) != null) sb.append(s + "\n");
// This writes the results from the web-server to a file
// It is using classes java.io.File and java.io.FileWriter
File outF = new File("SavedSite.html");
outF.createNewFile();
FileWriter fw = new FileWriter(outF);
fw.write(sb.toString());
fw.close();
Again, this code is very basic stuff that doesn't use any special JAR Library Code at all. The next method uses the JSoup library (which you have explicitly requested - even though I don't use it... It is just fine!) ... This is the method "parse" which will parse the String you have just saved. You may load this HTML String from disk, and send it to JSoup using:
Method Documentation: org.jsoup.Jsoup.parse​(File in, String charsetName, String baseUri)
If you wish to invoke JSoup just pass it a java.io.File instance using the following:
File f = new File("SavedSite.html");
Document d = Jsoup.parse(f, "UTF-8", url.toString());
I do not think you need timers at all...
AGAIN: If you are making lots of calls to the server. The purpose of this answer is to show you how to save the response of the server to a file on disk, so you don't have to make lots of calls - JUST ONE! If you restrict your calls to the server to once per hour, then you will (likely, but not a guarantee) avoid getting a 403 Forbidden Bot Detection Problem.

java TCP socket message breaks

i have a java client-server app in java, both using the same connection class that contains both send/receive messages.
for some reason, some of the messages i send are received in a malformed order:
here's the code
//set up
_in = new BufferedReader(new InputStreamReader(this._socket.getInputStream()));
_out = new BufferedWriter(new OutputStreamWriter(this._socket.getOutputStream()));
this._socket.setSoTimeout(S_TIMEOUT);
public synchronized boolean send(String message){
try {
_out.write(message);
_out.write(Connection.DELIMITER);
_out.flush();
return true;
} catch (IOException e) {
}
return false;
}
public String receive(){
int c;
try {
String message = "";
System.out.println("Getting message:");
c = _in.read();
while(c != -1 && c != Connection.DELIMITER) {
message += (char) c;
c = _in.read();
}
if (c == -1) {
return null;
}
return message;
} catch (IOException e) { }
return null;
}
some messages, for example "new_order" will might return with "ew_ord".
some characters are lost, others are sent separately. this seems odd as its TCP
could this be an encoding related issue?
Delimiter is (char) 0
socket timeout is 20000 (ie 20 senconds). every 10 seconds i send an empty message to make sure socket does not close
EDIT:
although it was solved using the Scanner, i must say that the original code worked fine for many messages/various machines for a very long time (a few weeks), and then suddenly failed to work with one specific message on one specific machine (other messages went through just fine). i've done socket data transfer in java MANY times and i've written many read/write methods to handle the sockets. it's the first time i ran into this.
although in the original code i set the encoding (in the posted code i didn't), i believe that the problem was encoding related. at one point, the message that was received had every second character missing. afterwards i changed it a bit, and the first/second character of the message were received in a separate message. from my understanding, it's either an encoding issue or some firewall/other security program that was running on the message sender machine, that decided to filter outgoing packets.

Try replacing your receive with a Scanner and let it do the work for you.
// in your setup
Scanner sc = new Scanner(_in).useDelimiter(Connection.DELIMETER);
public String receive() {
try {
return sc.next();
} catch(IOException e) {
return "";
}
}

For starters, I would make sure you're printing exceptions in those catch blocks.
Then, you're using the platform default encoding for converting characters to bytes. If these two processes are running on different machines, it's possible they're using different encodings. I would make sure you're specifying an encoding when you set up the Reader and Writer.

You can use UTF encoding for getting Full String of Message.
U can try this code and I am Sure About this code because i used it in My Chat Application.
String data=" ";
socket = new Socket("localhost",999);
while(true)
{
dos = new DataOutputStream(socket.getOutputStream());
dis = new DataInputStream(socket.getInputStream());
data = dis.readUTF();
jta.append(data +"\n");
}
Where jta is JTextArea.
It's for Client Side
Now For Server Side:
try
{
server = new ServerSocket(999);
Socket soc = server.accept();
while(true)
{
String data="";
try
{
dis = new DataInputStream(soc.getInputStream());
dos = new DataOutputStream(soc.getOutputStream());
data = dis.readUTF();
}
catch(Exception e)
{ }
jta.append(data + "\n");
}
}
catch (IOException e)
{
JOptionPane.showMessageDialog(this, e);
System.exit(-1);
}

How to use java to fill in a web page and pull back the results?

Hey I am relatively new to java, and I am trying to make an application that does the following:
Sends a request to a live website
Retrieves the data of that page
For example, assume the following site displays game results, where 'game=500' shows the results for game number 324 of 500 different games. http://www.some-site.com/results.php?game=324
I would like to use a Java program to automatically cycle through the game=1 to game=500, posting to the website and retrieving the results of the page.
What is the best way to do this? Can anyone give me a simple example? If I knew the correct java 'key words', I would google for some tutorials on this concept.
Note: the target-page in question is php

URL url;
InputStream is = null;
DataInputStream dis;
String line;
for(int i=1;i<=500;i++){
try {
url = new URL("http://www.some-site.com/results.php?game="+i);
is = url.openStream(); // throws an IOException
dis = new DataInputStream(new BufferedInputStream(is));
while ((line = dis.readLine()) != null) {
//do sth with the datea
}
} catch (MalformedURLException mue) {
mue.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
}

Do something like the answer in this other stackoverflow page
and then you want to use a for loop for loop through pages 1 through 500.

Apache has some really good Java libraries for accessing HTTP. See this for more details.

Simple Java form that would send data to webpage?

I know virtually no java, but I need to make a simple java application for mobiles that would display a form.On being submitted The data would then be sent to a webpage. As I am a php programmer, I would prefer to have it sent to a php file, which would then use the form's data. I only need a couple of input text areas, would anybody be able to help me with the java part?
Thanks in advance.
Niall

You don't mention what platform but no matter what you choose you will need to look into HTTP Client and here is a good example of mimicking a form based submission.
As far as presenting a form, that's very platform dependent.

If you can call your PHP File by calling an URL via GET, than you should be able to solve your problem with the following piece of code:
int variable1 = 4;
String variable2 = "My Phone Service";
URL url = null;
try {
url = new URL("http://myserver.com/service.php?var1=" + variable1 + "&variable2=" + variable2);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openConnection().getInputStream()));
String s = null;
while((s= reader.readLine())!=null){
System.out.println(s);
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

BlackBerry java.io.IOException: No tunnels to open

I wrote a simple BB app that does an HTTP Request and downloads some JSON encoded information.
When I run it on the emulator with the ";deviceSide=true" after the URL it works OK. But when I tried it in my BlackBerry Storm actual device, I got the following exception:
java.io.IOException: No tunnels to open
Here's the code I use to do the HTTP Request:
public static String doHttpRequest(String addr) {
HttpConnection hConn = null;
DataInputStream dis = null;
String strData = "";
try {
hConn = (HttpConnection)Connector.open( addr );
dis = new DataInputStream(hConn.openInputStream());
//Get the string in the stream
int c;
while ( (c = dis.read()) != -1 ) {
strData = strData + (char) c;
}
}catch (Exception e) {
net.rim.device.api.ui.component.Dialog.alert( e.toString() );
} finally {
try{
if(dis != null) dis.close();
if(hConn != null) hConn.close();
} catch(Exception e) {
net.rim.device.api.ui.component.Dialog.alert( e.toString() );
}
}
return strData;
}

It seems you have some APN issues, you should reconfigure this part.
In general if your code works well on an emulator the chances are that your will need some configuration tweaks on your phone.
Resources :
blackberryforums - Java.io.IOException:could not open tunnel - failed exception
crackberry.com - Youtube and Storm 2?

Colin, Please append this string - ";deviceside=true;interface=wifi" after the url. It should work. I found a problem similar to this. I change this setting. And now, It is working. :-)