I am trying to implement a web-server which serves static pages only.
Here is the code of my ultra mini web server.
import java.io.*;
import java.net.*;
class MyWebServer{
static ServerSocket serSocket = null ;
static{
try {
serSocket = new ServerSocket(80) ;
System.out.println("Server started successfully....\n\n") ;
} catch (IOException io) {
System.out.println( io.getMessage() );
System.exit(1) ;
}
}
public static void main(String []args){
while(true){
try {
new Thread( new ServingThread( serSocket.accept() ) ).start() ;
} catch (IOException io) {
System.out.println(io.getMessage());
}
}
}
}
class ServingThread implements Runnable{
private Socket socket ;
public ServingThread(Socket socket){
this.socket = socket ;
System.out.println("Receives a new browser request from " + socket);
}
public void run() {
BufferedReader in = null ;
PrintWriter out = null ;
try {
//reading request headers from browser starts here
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String str = ".";
while (!str.equals("")){
str = in.readLine();
//System.out.println(str);
}
System.out.println("\n\n");
//reading request headers from browser ends here
//writing output on outputstream starts here
out = new PrintWriter( socket.getOutputStream(), true ) ;
int i = (int) ( Math.random() * 10) ;
out.println(i) ;
System.out.println(i) ;
//writing output on outputstream ends here
socket.close() ;
System.out.println("Request successfully fulfilled.\n") ;
}
catch (IOException io) {
System.out.println(io.getMessage());
}
}
}
It works fine, but some times (not always)
my mozilla firefox client
automatically sends a request, when i
add a new tab (Ctrl + T) or closes an
existing tab.
When i request for
http://localhost/ , single digit gets displayed on my browser. But the
server's console shows that it get 2
or 3 requests, so, it shows 2 or 3 numbers
accordingly.
I googled a lot but don't get an satisfactory answer for this.
Any thoughts/explanations are highly appreciated. :)
First of all, I'm assuming you're doing this as an exercise to learn about sockets / http / etc... Because if you're really trying to implement a web server, you should seriously consider using an existing one or at least looking at its source - there's a LOT of stuff involved in even the most simplistic HTTP server implementation.
That said, your "server" :-) doesn't implement HTTP properly (doesn't return status code, for one) so browser's behavior is really undetermined here - it may attempt to retry the same request multiple times or it may not. Caching issue may be at play here too - once you do implement HTTP, you'll need to send appropriate caching headers if you don't want "new tab" to repeat request.
Related
When I try to check status codes within sites I face off 403 response code after a while. First when I run the code every sites send back datas but after my code repeat itself with Timer I see one webpage returns 403 response code. Here is my code.
public class Main {
public static void checkSites() {
Timer ifSee403 = new Timer();
try {
File links = new File("./linkler.txt");
Scanner scan = new Scanner(links);
ArrayList<String> list = new ArrayList<>();
while(scan.hasNext()) {
list.add(scan.nextLine());
}
File linkStatus = new File("LinkStatus.txt");
if(!linkStatus.exists()){
linkStatus.createNewFile();
}else{
System.out.println("File already exists");
}
BufferedWriter writer = new BufferedWriter(new FileWriter(linkStatus));
for(String link : list) {
try {
if(!link.startsWith("http")) {
link = "http://"+link;
}
URL url = new URL(link);
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection http = (HttpURLConnection)url.openConnection();
http.setRequestMethod("HEAD");
http.setConnectTimeout(5000);
http.setReadTimeout(8000);
int statusCode = http.getResponseCode();
if (statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Hello, here we go again");
}
http.disconnect();
System.out.println(link + " " + statusCode);
writer.write(link + " " + statusCode);
writer.newLine();
} catch (Exception e) {
writer.write(link + " " + e.getMessage());
writer.newLine();
System.out.println(link + " " +e.getMessage());
}
}
try {
writer.close();
} catch (Exception e) {
System.out.println(e.getMessage());
}
System.out.println("Finished.");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
public static void main(String[] args) throws Exception {
Timer myTimer = new Timer();
TimerTask sendingRequest = new TimerTask() {
public void run() {
checkSites();
}
};
myTimer.schedule(sendingRequest,0,150000);
}
}
How can I solve this? Thanks
Edited comment:
I've added http.disconnect(); for closing connection after checked status codes.
Also I've added
if(statusCode == 200) {
ifSee403.wait(5000);
System.out.println("Test message);
}
But it didn't work. Compiler returned current thread is not owner error. I need to fix this and change 200 with 403 and say ifSee403.wait(5000) and try it again the status code.
One "alternative" - by the way - to IP / Spoofing / Anonymizing would be to (instead) try "obeying" what the security-code is expecting you to do. If you are going to write a "scraper", and are aware there is a "bot detection" that doesn't like you debugging your code while you visit the site over and over and over - you should try using the HTML Download which I posted as an answer to the last question you asked.
If you download the HTML and save it (save it to a file - once an hour), and then write you HTML Parsing / Monitoring Code using the HTML contents of the file you have saved, you will (likely) be abiding by the security-requirements of the web-site and still be able to check availability.
If you wish to continue to use JSoup, that A.P.I. has an option for receiving HTML as a String. So if you use the HTML Scrape Code I posted, and then write that HTML String to disk, you can feed that to JSoup as often as you like without causing the Bot Detection Security Checks to go off.
If you play by their rules once in a while, you can write your tester without much hassle.
import java.io.*;
import java.net.*;
...
// This line asks the "url" that you are trying to connect with for
// an instance of HttpURLConnection. These two classes (URL and HttpURLConnection)
// are in the standard JDK Package java.net.*
HttpURLConnection con = (HttpURLConnection) url.openConnection();
// Tells the connection to use "GET" ... and to "pretend" that you are
// using a "Chrome" web-browser. Note, the User-Agent sometimes means
// something to the web-server, and sometimes is fully ignored.
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Chrome/61.0.3163.100");
// The classes InputStream, InputStreamReader, and BufferedReader
// are all JDK 1.0 package java.io.* classes.
InputStream is = con.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
StringBuffer sb = new StringBuffer();
String s;
// This reads each line from the web-server.
while ((s = br.readLine()) != null) sb.append(s + "\n");
// This writes the results from the web-server to a file
// It is using classes java.io.File and java.io.FileWriter
File outF = new File("SavedSite.html");
outF.createNewFile();
FileWriter fw = new FileWriter(outF);
fw.write(sb.toString());
fw.close();
Again, this code is very basic stuff that doesn't use any special JAR Library Code at all. The next method uses the JSoup library (which you have explicitly requested - even though I don't use it... It is just fine!) ... This is the method "parse" which will parse the String you have just saved. You may load this HTML String from disk, and send it to JSoup using:
Method Documentation: org.jsoup.Jsoup.parse(File in, String charsetName, String baseUri)
If you wish to invoke JSoup just pass it a java.io.File instance using the following:
File f = new File("SavedSite.html");
Document d = Jsoup.parse(f, "UTF-8", url.toString());
I do not think you need timers at all...
AGAIN: If you are making lots of calls to the server. The purpose of this answer is to show you how to save the response of the server to a file on disk, so you don't have to make lots of calls - JUST ONE! If you restrict your calls to the server to once per hour, then you will (likely, but not a guarantee) avoid getting a 403 Forbidden Bot Detection Problem.
I'm trying to create a single-threaded echo-server, but I can't figure out how to send the input from the client back to the client again from the server. The client connects to the server alright, but it's when it's waiting for a response that it throws an IOException. I have tried connecting my chat client to other chat servers, hence I'm sure the fault is in the chat server implementation.
I'm guessing the "villain of the piece" is this method presented below that takes the input from the connected client and sends it back, but I'm not sure why it doesn't work.
private void processClientRequest(Socket clientSocket) throws IOException {
InputStream in = clientSocket.getInputStream();
PrintWriter out =
new PrintWriter(clientSocket.getOutputStream());
long time = System.currentTimeMillis();
out.write("Server - " + time + ": " + in);
out.close();
in.close();
}
Please tell me if this method isn't the issue and you need other parts of my code instead.
Thanks in advance!
Edit: I have now managed to get my server to respond to the client. I did this by changing the processClientRequest method to this:
try {
BufferedReader in =
new BufferedReader(
new InputStreamReader(clientSocket.getInputStream()));
PrintWriter out =
new PrintWriter(clientSocket.getOutputStream());
long time = System.currentTimeMillis();
out.write("Server - " + time + ": " + in.readLine());
out.close();
try {
in.close();
} catch (IOException e) {
System.err.println("Couldn't close input stream");
}
} catch(IOException e) {
System.err.println("Got an IOException error while reading or writing from/to client");
}
But as of now my server can only respond the client once. If the client writes to the server a second time it does get a response, but instead of the clients message, it's "null". And since I want my echo server to be able to answer the client until the client shuts down or something like that, I need to figure out how to change that. I figure I'm going to need some kind of while loop to make this happen, but I have tried for example putting the whole try statement into an infinite while loop, but that just gave me loads of IOExceptions.
You're not reading the clients input. You're just passing the toString to your out. You need to make a while loop and read from in. Maybe in a seperate thread unless you're sure the input is short. In test code should be okay:
private void processClientRequest(Socket clientSocket) throws IOException {
InputStream in = clientSocket.getInputStream();//wrap this in a object stream or other to read lines/ utf
int i = 0;
while(i++ < 1000 && in.available() > 0){
int read = in.read();//use it if u want, like to a string builder declared out of loop and echo it back after u read input
}
PrintWriter out =
new PrintWriter(clientSocket.getOutputStream());
long time = System.currentTimeMillis();
out.write("Server - " + time + ": " + in);
out.close();//should be in seperate try catches so if 1 close fails u still close other and finally close socket in 3rd try catch
in.close();
}
Socket server with working echo server, mini web server and helpful in many more ways.
I'm kinda new to Java language, but have a good experience in other technologies.
Right now Im working on creating a proxy server and I have come up with the code that seems to work fine for single requests, but when I'm trying to open some html page that loads lots of styles, images etc, part of requests just got aborted ( this is what firebug tells me )
So, will try to simplify the code and be as much specific as possible in showing up the exact problem
Here's the main class :
public class ProxyThread {
public static void main(String[] args) {
ProxyThread Proxy = new ProxyThread();
Proxy.start();
}
public void start() {
ServerSocket server = new ServerSocket(this.portNumber, 1);
while(true) {
Socket serverClient = server.accept();
this._threads[threadCount] = new Thread( new RequestProcess( serverClient, threadCount++ ));
}
}
}
Here's the code of RequestProcess that does processing of each request
public class RequestProcess implements Runnable {
public void start() throws InterruptedException {
this.serverIn = new BufferedReader( new InputStreamReader( this.serverClient.getInputStream() ) );
this.serverOut = this.serverClient.getOutputStream() ;
String currentBuffer;
String request = "";
String host = "";
int port = 0;
while ((currentBuffer = this.serverIn.readLine()) != null) {
if (currentBuffer.length() == 0) {
break;
}
request += currentBuffer + "\r\n";
if (currentBuffer.startsWith("CONNECT ") || currentBuffer.startsWith("GET ")) {
host = this.parseHost( currentBuffer );
port = this.parsePort( currentBuffer );
}
}
request += "\r\n";
if (host.isEmpty() || request.isEmpty()) {
throw new InterruptedException("request or host empty, so exiting ");
}
clientRequestProcess clientProcess = new clientRequestProcess( host, port, request, this.threadNum );
byte[] response = clientProcess.processRequest();
this.serverOut.write(response);
this.serverOut.close();
}
}
And here's the class that process request and sending it to the actual server
public class clientRequestProcess {
public byte[] processRequest()
{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte buf[] = new byte[256*1024];
Socket clientSocket = new Socket( host, port );
clientSocket.getOutputStream().write( request.getBytes() );
InputStream is = clientSocket.getInputStream();
int r = 1;
while (r > 0) {
r = is.read(buf);
if (r > 0) {
bos.write(buf, 0, r);
}
}
return bos.toByteArray();
}
}
All the code is draft one and is simplified to show the big picture of how it works. All try-catch blocks, debugging info etc are missed here.
steps
- Set up browser to use JAVA proxy
- Open some site
- Part of http requests are successfully processed and correct response is returned
-part of requests shows up as aborted in firebug. The thing is that part is absolutely random, so one time some file is loading, another it is not
Code troubleshooting shows me that first line of request (that comes from the browser) are empty, so my condition
if (currentBuffer.length() == 0) {
break;
}
breaks reading the socket and thus it ruturns nothing to the browser and connection gets aborted
I read rfc on http protocol and figured out that https requests considered to be over once \r\n meets, so that's why I'm using that condition
If I just open that file in separate tab - it loads successfully, don;t matter how many times im trying to reload it. But when a bunch of files are loaded at one time - some random of them gets aborted. So it is happening only when a lot of files loading, when one or even 3-5 - all files loads fine
Any ideas?
Thanks
i have a java client-server app in java, both using the same connection class that contains both send/receive messages.
for some reason, some of the messages i send are received in a malformed order:
here's the code
//set up
_in = new BufferedReader(new InputStreamReader(this._socket.getInputStream()));
_out = new BufferedWriter(new OutputStreamWriter(this._socket.getOutputStream()));
this._socket.setSoTimeout(S_TIMEOUT);
public synchronized boolean send(String message){
try {
_out.write(message);
_out.write(Connection.DELIMITER);
_out.flush();
return true;
} catch (IOException e) {
}
return false;
}
public String receive(){
int c;
try {
String message = "";
System.out.println("Getting message:");
c = _in.read();
while(c != -1 && c != Connection.DELIMITER) {
message += (char) c;
c = _in.read();
}
if (c == -1) {
return null;
}
return message;
} catch (IOException e) { }
return null;
}
some messages, for example "new_order" will might return with "ew_ord".
some characters are lost, others are sent separately. this seems odd as its TCP
could this be an encoding related issue?
Delimiter is (char) 0
socket timeout is 20000 (ie 20 senconds). every 10 seconds i send an empty message to make sure socket does not close
EDIT:
although it was solved using the Scanner, i must say that the original code worked fine for many messages/various machines for a very long time (a few weeks), and then suddenly failed to work with one specific message on one specific machine (other messages went through just fine). i've done socket data transfer in java MANY times and i've written many read/write methods to handle the sockets. it's the first time i ran into this.
although in the original code i set the encoding (in the posted code i didn't), i believe that the problem was encoding related. at one point, the message that was received had every second character missing. afterwards i changed it a bit, and the first/second character of the message were received in a separate message. from my understanding, it's either an encoding issue or some firewall/other security program that was running on the message sender machine, that decided to filter outgoing packets.
Try replacing your receive with a Scanner and let it do the work for you.
// in your setup
Scanner sc = new Scanner(_in).useDelimiter(Connection.DELIMETER);
public String receive() {
try {
return sc.next();
} catch(IOException e) {
return "";
}
}
For starters, I would make sure you're printing exceptions in those catch blocks.
Then, you're using the platform default encoding for converting characters to bytes. If these two processes are running on different machines, it's possible they're using different encodings. I would make sure you're specifying an encoding when you set up the Reader and Writer.
You can use UTF encoding for getting Full String of Message.
U can try this code and I am Sure About this code because i used it in My Chat Application.
String data=" ";
socket = new Socket("localhost",999);
while(true)
{
dos = new DataOutputStream(socket.getOutputStream());
dis = new DataInputStream(socket.getInputStream());
data = dis.readUTF();
jta.append(data +"\n");
}
Where jta is JTextArea.
It's for Client Side
Now For Server Side:
try
{
server = new ServerSocket(999);
Socket soc = server.accept();
while(true)
{
String data="";
try
{
dis = new DataInputStream(soc.getInputStream());
dos = new DataOutputStream(soc.getOutputStream());
data = dis.readUTF();
}
catch(Exception e)
{ }
jta.append(data + "\n");
}
}
catch (IOException e)
{
JOptionPane.showMessageDialog(this, e);
System.exit(-1);
}
I'm requesting a webpage with sockets like this:
Socket sock = null;
PrintWriter out = null;
BufferedReader in = null;
try {
sock = new Socket( "www.overvprojects.nl", 80 );
out = new PrintWriter( sock.getOutputStream(), true );
in = new BufferedReader( new InputStreamReader( sock.getInputStream() ) );
} catch ( UnknownHostException e ) {
e.printStackTrace();
} catch ( IOException e ) {
e.printStackTrace();
}
out.println( "GET /ip.php HTTP/1.1" );
out.println( "Host: www.overvprojects.nl" );
out.println( "" );
out.flush();
String buf = "";
while ( buf != null )
{
try {
buf = in.readLine();
} catch ( IOException e ) {
e.printStackTrace();
}
if ( buf != null && Pattern.matches( "^(25[0-5]|2[0-4]\\d|[0-1]?\\d?\\d)(\\.(25[0-5]|2[0-4]\\d|[0-1]?\\d?\\d)){3}$", buf ) ) {
ipText.setText( buf );
break;
}
}
try {
in.close();
out.close();
sock.close();
} catch ( IOException e ) {
e.printStackTrace();
}
However, the program seems to wait until the connection times out. I've debugged the loop and I'm absolutely sure break is called, yet the UI doesn't update until the connection is terminated. (This takes over 10 seconds.) After that the IP is visible, so I'm sure break has been called at some point. I've also tried using the website www.whatismyip.org instead and that does finish within two seconds.
You need to check the content length header returned by the server. Only read that many bytes then close the connection after that.
You are waiting until the server closes the connection which is why it is taking 20 seconds.
The activity of (re)painting the screen happens to be done by the same thread that is responsible for reading the contents of the reader object. This is a classic example of using multi-threading so that the screen rendering is not affected by the process of reading the response.
EDIT: Based on the comments received, the behavior can be explained by the fact that both the client and the server need to perform cleanup operations when the socket is abruptly closed. In simple terms, the client does not appear to be reading all data from the input stream, and hence it takes far longer than usual for the JVM and hence the OS to perform the appropriate operations that will actually release resources, resulting in an apparent lockup of some variety. So the advice provided by others of using URL/URLConnection is very valid in this case.
i might throw a break in the catch block after the read
readLine is waiting for a newline which the server never sends