I'm kinda new to Java language, but have a good experience in other technologies.
Right now Im working on creating a proxy server and I have come up with the code that seems to work fine for single requests, but when I'm trying to open some html page that loads lots of styles, images etc, part of requests just got aborted ( this is what firebug tells me )
So, will try to simplify the code and be as much specific as possible in showing up the exact problem
Here's the main class :
public class ProxyThread {
public static void main(String[] args) {
ProxyThread Proxy = new ProxyThread();
Proxy.start();
}
public void start() {
ServerSocket server = new ServerSocket(this.portNumber, 1);
while(true) {
Socket serverClient = server.accept();
this._threads[threadCount] = new Thread( new RequestProcess( serverClient, threadCount++ ));
}
}
}
Here's the code of RequestProcess that does processing of each request
public class RequestProcess implements Runnable {
public void start() throws InterruptedException {
this.serverIn = new BufferedReader( new InputStreamReader( this.serverClient.getInputStream() ) );
this.serverOut = this.serverClient.getOutputStream() ;
String currentBuffer;
String request = "";
String host = "";
int port = 0;
while ((currentBuffer = this.serverIn.readLine()) != null) {
if (currentBuffer.length() == 0) {
break;
}
request += currentBuffer + "\r\n";
if (currentBuffer.startsWith("CONNECT ") || currentBuffer.startsWith("GET ")) {
host = this.parseHost( currentBuffer );
port = this.parsePort( currentBuffer );
}
}
request += "\r\n";
if (host.isEmpty() || request.isEmpty()) {
throw new InterruptedException("request or host empty, so exiting ");
}
clientRequestProcess clientProcess = new clientRequestProcess( host, port, request, this.threadNum );
byte[] response = clientProcess.processRequest();
this.serverOut.write(response);
this.serverOut.close();
}
}
And here's the class that process request and sending it to the actual server
public class clientRequestProcess {
public byte[] processRequest()
{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte buf[] = new byte[256*1024];
Socket clientSocket = new Socket( host, port );
clientSocket.getOutputStream().write( request.getBytes() );
InputStream is = clientSocket.getInputStream();
int r = 1;
while (r > 0) {
r = is.read(buf);
if (r > 0) {
bos.write(buf, 0, r);
}
}
return bos.toByteArray();
}
}
All the code is draft one and is simplified to show the big picture of how it works. All try-catch blocks, debugging info etc are missed here.
steps
- Set up browser to use JAVA proxy
- Open some site
- Part of http requests are successfully processed and correct response is returned
-part of requests shows up as aborted in firebug. The thing is that part is absolutely random, so one time some file is loading, another it is not
Code troubleshooting shows me that first line of request (that comes from the browser) are empty, so my condition
if (currentBuffer.length() == 0) {
break;
}
breaks reading the socket and thus it ruturns nothing to the browser and connection gets aborted
I read rfc on http protocol and figured out that https requests considered to be over once \r\n meets, so that's why I'm using that condition
If I just open that file in separate tab - it loads successfully, don;t matter how many times im trying to reload it. But when a bunch of files are loaded at one time - some random of them gets aborted. So it is happening only when a lot of files loading, when one or even 3-5 - all files loads fine
Any ideas?
Thanks
Related
I have written a piece of software in Java that checks if proxies are working by sending a HTTP request using the proxy.
It takes around 30,000 proxies from a database, then attempts to check if they are operational. The proxies received from the database used to be returned as an ArrayList<String>, but have been changed to Deque<String> for reasons stated below.
The way the program works is there is a ProxyRequest object that stores the IP & Port as a String and int respectively. The ProxyRequest object has a method isWorkingProxy() which attempts to send a request using a proxy and returns a boolean on whether it was successful.
This ProxyRequest object is wrapped around by a RunnableProxyRequest object that calls super.isWorkingProxy() in the overrided run() method. Based on the response from super.isWorkingProxy(), the RunnableProxyRequest object updates a MySQL database.
Do note that the updating of the MySQL database is synchronized().
It runs on 750 threads using a FixedThreadPool (on a VPS), but towards
the end, it becomes very slow (stuck on ~50 threads), which obviously
implies the garbage collector is working. This is the problem.
I have attempted the following to improve the lag, it does not seem to work:
1) Using a Deque<String> proxies and using Deque.pop() to obtain the String in which the proxy is. This (I believe), continuously makes the Deque<String> smaller, which should improve lag caused by the GC.
2) Set the con.setConnectTimeout(this.timeout);, where this.timeout = 5000; This way, the connection should return a result in 5 seconds. If not, the thread is completed and should no longer be active in the threadpool.
Besides this, I don't know any other way I can improve performance.
Can anyone recommend a way for me to improve performance to avoid / stop lagging towards the end of the threads by the GC? I know there is a Stackoverflow question about this (Java threads slow down towards the end of processing), but I have tried everything in the answer and it has not worked for me.
Thank you for your time.
Code snippets:
Loop adding threads to the FixedThreadPool:
//This code is executed recursively (at the end, main(args) is called again)
//Create the threadpool for requests
//Threads is an argument that is set to 750.
ThreadPoolExecutor executor = (ThreadPoolExecutor)Executors.newFixedThreadPool(threads);
Deque<String> proxies = DB.getProxiesToCheck();
while(proxies.isEmpty() == false) {
try {
String[] split = proxies.pop().split(":");
Runnable[] checks = new Runnable[] {
//HTTP check
new RunnableProxyRequest(split[0], split[1], Proxy.Type.HTTP, false),
//SSL check
new RunnableProxyRequest(split[0], split[1], Proxy.Type.HTTP, true),
//SOCKS check
new RunnableProxyRequest(split[0], split[1], Proxy.Type.SOCKS, false)
//Add more checks to this list as time goes...
};
for(Runnable check : checks) {
executor.submit(check);
}
} catch(IndexOutOfBoundsException e) {
continue;
}
}
ProxyRequest class:
//Proxy details
private String proxyIp;
private int proxyPort;
private Proxy.Type testingType;
//Request details
private boolean useSsl;
public ProxyRequest(String proxyIp, String proxyPort, Proxy.Type testingType, boolean useSsl) {
this.proxyIp = proxyIp;
try {
this.proxyPort = Integer.parseInt(proxyPort);
} catch(NumberFormatException e) {
this.proxyPort = -1;
}
this.testingType = testingType;
this.useSsl = useSsl;
}
public boolean isWorkingProxy() {
//Case of an invalid proxy
if(proxyPort == -1) {
return false;
}
HttpURLConnection con = null;
//Perform checks on URL
//IF any exception occurs here, the proxy is obviously bad.
try {
URL url = new URL(this.getTestingUrl());
//Create proxy
Proxy p = new Proxy(this.testingType, new InetSocketAddress(this.proxyIp, this.proxyPort));
//No redirect
HttpURLConnection.setFollowRedirects(false);
//Open connection with proxy
con = (HttpURLConnection)url.openConnection(p);
//Set the request method
con.setRequestMethod("GET");
//Set max timeout for a request.
con.setConnectTimeout(this.timeout);
} catch(MalformedURLException e) {
System.out.println("The testing URL is bad. Please fix this.");
return false;
} catch(Exception e) {
return false;
}
try(
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
) {
String inputLine = null; StringBuilder response = new StringBuilder();
while((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
//A valid proxy!
return con.getResponseCode() > 0;
} catch(Exception e) {
return false;
}
}
RunnableProxyRequest class:
public class RunnableProxyRequest extends ProxyRequest implements Runnable {
public RunnableProxyRequest(String proxyIp, String proxyPort, Proxy.Type testingType, boolean useSsl) {
super(proxyIp, proxyPort, testingType, useSsl);
}
#Override
public void run() {
String test = super.getTest();
if(super.isWorkingProxy()) {
System.out.println("-- Working proxy: " + super.getProxy() + " | Test: " + test);
this.updateDB(true, test);
} else {
System.out.println("-- Not working: " + super.getProxy() + " | Test: " + test);
this.updateDB(false, test);
}
}
private void updateDB(boolean success, String testingType) {
switch(testingType) {
case "SSL":
DB.updateSsl(super.getProxyIp(), super.getProxyPort(), success);
break;
case "HTTP":
DB.updateHttp(super.getProxyIp(), super.getProxyPort(), success);
break;
case "SOCKS":
DB.updateSocks(super.getProxyIp(), super.getProxyPort(), success);
break;
default:
break;
}
}
}
DB class:
//Locker for async
private static Object locker = new Object();
private static void executeUpdateQuery(String query, String proxy, int port, boolean toSet) {
synchronized(locker) {
//Some prepared statements here.
}
}
Thanks to Peter Lawrey for guiding me to the solution! :)
His comment:
#ILoveKali I have found network libraries are not aggressive enough in
shutting down a connection when things go really wrong. Timeouts tend
to work best when the connection is fine. YMMV
So I did some research, and found that I had to also use the method setReadTimeout(this.timeout);. Previously, I was only using setConnectTimeout(this.timeout);!
Thanks to this post (HttpURLConnection timeout defaults) that explained the following:
Unfortunately, in my experience, it appears using these defaults can
lead to an unstable state, depending on what happens with your
connection to the server. If you use an HttpURLConnection and don't
explicitly set (at least read) timeouts, your connection can get into
a permanent stale state. By default. So always set setReadTimeout to
"something" or you might orphan connections (and possibly threads
depending on how your app runs).
So the final answer is: The GC was doing just fine, it was not responsible for the lag. The threads were simply stuck FOREVER at a single number because I did not set the read timeout, and so the isWorkingProxy() method never got a result and kept reading.
I'm creating an app that sends and receive data with an arduino via bluetooth. Sending works fine, however when receiving data I don't get the first few characters of the string sent. I always don't get the first character, the second I sometimes get it, the third I almost always get it, etc.
So for example if the arduino sends "OK 1" I receive "K 1" or " 1" or "1", but never the complete string. An easy fix would be to add a few dummy characters, but that's a shit fix.
Here's the method which listens to incoming connections, directly copy/pasted from Android sample bluetooth code to send a simple string via bluetooth (though with a few fixes):
void beginListenForData()
{
final Handler handler = new Handler();
final byte delimiter = 10; //This is the ASCII code for a newline character
final boolean stopWorker = false;
final int readBufferPosition = 0;
final byte[]readBuffer = new byte[1024];
Thread workerThread = new Thread(new Runnable()
{
public void run()
{
while(!Thread.currentThread().isInterrupted() && !stopWorker)
{
try
{
int bytesAvailable = inStream.available();
int readBufferPosition2 = readBufferPosition;
if(bytesAvailable > 0)
{
byte[] packetBytes = new byte[bytesAvailable];
inStream.read(packetBytes);
for(int i=0;i<bytesAvailable;i++)
{
byte b = packetBytes[i];
if(b == delimiter)
{
byte[] encodedBytes = new byte[readBufferPosition2];
System.arraycopy(readBuffer, 0, encodedBytes, 0, encodedBytes.length);
final String data = new String(encodedBytes);
readBufferPosition2 = 0;
handler.post(new Runnable()
{
public void run()
{
result.setText(data);
}
});
}
else
{
readBuffer[readBufferPosition2++] = b;
}
}
}
}
catch (IOException ex)
{
}
}
}
});
workerThread.start();
}
Here's all my code in case you want to test it (warning: lots of dummy and outdated code):
MainActivity.java http://pastebin.com/cdjW4Y1V
XML layout file http://pastebin.com/Ruf5euPP
Click on the first button to connect to the arduino and click on the second button to send a string and to begin receiving data.
So yep, I have absolutely no idea why it doesn't work. It works fine with TerminalBT so it's not a problem with the arduino, it's a problem with my app, but why do I receive characters randomly?
one thing i noticed is that you should not use 10 for the delimiter.
I have encountered this error before if you use 10 then some times it does not get recognized properly.
you should use standard Java Function to parse the delimiter.
System.getProperty("line.separator");
//OR
System.lineSeparator();
I'm working on a project for school. We are are making a harbour where you can load and unload ships. The control part is made in Netbeans and the simulation in JME.
We send data from Netbeans to JME via a socket. JME is running a serversocket who is liseting to the input from Netbeans.
For example Netbeans sends an ID of a container and the crane in JME gets that container and puts it on the shore so a verhicle can pick it up.
We change a count in the main (Main.count = 2) so the SimpleUpdate can call a method. The problem is that sometimes stuff is getting skipped. Also I think it's getting worse when we send more information for instance a vehicle that's getting the container. How can I fix this? And are there other ways to get a good connection?
The code:
Netbeans
Send client
public static void run() throws Exception
{
Socket socket = new Socket("Localhost", 4321);
out = new ObjectOutputStream(socket.getOutputStream());
}
//Sent arraystring to Simulation
public void sent(String sentString){
try {
out.writeObject(sentString);
} catch (IOException ex) {
Logger.getLogger(CommunicationWithSimulatoin.class.getName()).log(Level.SEVERE, null, ex);
}
}
Main send some stuff example
for(int i = Calculator.getContainersFromMaritime(); i > 1; i--)
{
Thread.sleep(50);
sim.sent("craneCon;" + i + ";");
System.out.println(i);
}
JME
Listener
public static void Listener() throws Exception {
boolean isRunning = true;
//Creates the server socket
ServerSocket sSocket = new ServerSocket(4321);
//Acception a connection from the client
Socket socket = sSocket.accept();
ObjectInputStream in = new ObjectInputStream(socket.getInputStream());
//Get the input from the client
while (isRunning) {
//Reads and prints the input
test = (String) in.readObject();
System.out.println(test);
String[] parts = receivedString.split(";");
if(parts[0].equals("ContainerPositionsMaritime"))
{
Maritime.ContainersOnBoard = receivedString.split(";");
Main.count = 0;
}
if(parts[0].equals("craneCon"))
{
int containerId = Integer.parseInt(parts[1]);
SeagoingCranes.idContainer = containerId;
Main.count = 2;
}
}
}
Main simpleupdate
public void simpleUpdate(float tpf) {
if(count == 0)
{
InitContainers();
//martime.setLocalTranslation(0, 500.0f, 0);
count = 999;
}
if(count == 2)
{
InitCrane(SeagoingCranes.idContainer);
count = 999;
}
if(martime != null)
{
martime.move(0,0,0.25f*tpf);
}
}
There are a number of problems with your program.
Firstly - you have potential race and thread contention issues as you have "count" which I assume is an integer value inside the SimpleApplication is being modified from one thread and read from another. Unless the value is declared as volatile this can cause all sorts of unexpected problems and odd behaviour and even declaring it as volatile is not recommended.
Your main issue though (even leaving aside the subtle problems) is being caused by the fact that in simpleUpdate() you are scanning count and then taking an action based on count. simpleUpdate() is called once for each frame as your jME3 application is running.
If you receive more than one message in a frame then only the last one will be acted on as the count will be modified again before the next simpleUpdate() runs.
The best way to do this is to use app.enqueue().
if(parts[0].equals("ContainerPositionsMaritime"))
{
final ContainersOnBoard containers = receivedString.split(";");
mainApp.enqueue(new Callable<Spatial>() {
public Spatial call() throws Exception {
mainApp.InitContainers(containers);
return null;
}
});
}
}
You can remove all the existing code from your simpleUpdate().
The Callable you enqueue will be called back from the JME3 thread in the next update and process the addition of the containers. By doing a similar thing for every different commands it will enqueue and process all the commands as the time comes. You can enqueue as many commands as you like and they will all be processed.
In general you should read up on AppStates, Controls and the threading model as they will allow you to make your code much more structured and organised.
http://hub.jmonkeyengine.org/wiki/doku.php/jme3:advanced:multithreading
http://hub.jmonkeyengine.org/wiki/doku.php/jme3:advanced:application_states
http://hub.jmonkeyengine.org/wiki/doku.php/jme3:advanced:custom_controls
P.S. You should try and follow Java style/coding conventions - for example methods should begin with lowercase. initContainers not InitContainers.
I know there are many people who already asked this Question, but in all the threads I read I couldn't find 1 solution for my problem (even if others had the same one, it didn't work for me).
As the Title says, I'm trying to connect from a Flash/SWF-Application to a small Java server I wrote via Sockets. It works fine offline (on the same machine), but as soon as I put the .swf on a Webspace and open it from there, Flash requests the Policy file from the server. There's nothing bad with that, but my problem is that Flash disconnects after (hopefully) getting the policy-file but doesn't reconnect again.
My server always opens a new Thread when a client connects, but that's not where the trouble is made, as I already tried it without opening a new Thread.
Here's my code:
while (true) {
connection = providerSocket.accept();
System.out.println("Incoming connection from " +
connection.getInetAddress().getHostName());
BufferedReader in = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
String request = in.readLine();
if (request != null && request.contains("<policy-file-request/>")) {
System.out.println("Authorization request.");
PrintStream out = new PrintStream(connection.getOutputStream(), true);
out.println("<?xml version=\"1.0\"?><cross-domain-policy><!DOCTYPE cross-domain-policy SYSTEM \"http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd\"><allow-access-from domain=\"*\" to-ports=\"3002\" /></cross-domain-policy>\u0000");
out.flush();
System.out.println("AuthData sent.");
connection.close();
System.out.println("Authorization complete.");
connection = providerSocket.accept();
System.out.println("TEST");
RequestProcessor c = new RequestProcessor(connection, connectionCounter++);
Thread t = new Thread(c);
t.start();
} else {
RequestProcessor c = new RequestProcessor(connection, connectionCounter++);
Thread t = new Thread(c);
t.start();
}
}
You will surely notice that I am using "\u0000" at the end instead of "\0", but don't worry, I also tested that case, didn't change anything. :/
I dont even reach the "TEST"-Ouput, because I don't get a new connection. And if I don't close the connection myself, flash automatically disconnects me.
The last thing I tried was just sending the xml without any request (right at the beginning, after the connection is established). I got a "recv failed" - Error for that.
PS: RequestProcessor is my new Thread, in which I would process the Strings/Commands sent from my.swf-File...
Thanks for helping me! :)
I had this problem before, you can not just use in.readLine() to get the policy file request string, because there're zero character.
To make sure you read the whole policy file request:
private String read(BufferedReader in) throws IOException {
StringBuilder builder = new StringBuilder();
int codePoint;
boolean zeroByteRead = false;
System.out.println("Reading...");
do {
codePoint = in.read();
if (codePoint == -1) {
return null;
}
if (codePoint == 0) {
zeroByteRead = true;
} else {
builder.appendCodePoint(codePoint);
}
} while (!zeroByteRead);
return builder.toString();
}
In the calling method:
BufferedReader in = new BufferedReader(new InputStreamReader(
clientSocket.getInputStream()));
String inputLine;
while ((inputLine = read(in)) != null) {
System.out.println("Receive from client: " + inputLine);
if ("<policy-file-request/>".equals(inputLine)) {
// Serve policy file, like the one in your question.
out.println(buildPolicy() +"\u0000");
} else {
// Do your job.
}
}
You can find the policy file project in java which can be downloaded. I myself thank to the guys over there.
I am trying to implement a web-server which serves static pages only.
Here is the code of my ultra mini web server.
import java.io.*;
import java.net.*;
class MyWebServer{
static ServerSocket serSocket = null ;
static{
try {
serSocket = new ServerSocket(80) ;
System.out.println("Server started successfully....\n\n") ;
} catch (IOException io) {
System.out.println( io.getMessage() );
System.exit(1) ;
}
}
public static void main(String []args){
while(true){
try {
new Thread( new ServingThread( serSocket.accept() ) ).start() ;
} catch (IOException io) {
System.out.println(io.getMessage());
}
}
}
}
class ServingThread implements Runnable{
private Socket socket ;
public ServingThread(Socket socket){
this.socket = socket ;
System.out.println("Receives a new browser request from " + socket);
}
public void run() {
BufferedReader in = null ;
PrintWriter out = null ;
try {
//reading request headers from browser starts here
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String str = ".";
while (!str.equals("")){
str = in.readLine();
//System.out.println(str);
}
System.out.println("\n\n");
//reading request headers from browser ends here
//writing output on outputstream starts here
out = new PrintWriter( socket.getOutputStream(), true ) ;
int i = (int) ( Math.random() * 10) ;
out.println(i) ;
System.out.println(i) ;
//writing output on outputstream ends here
socket.close() ;
System.out.println("Request successfully fulfilled.\n") ;
}
catch (IOException io) {
System.out.println(io.getMessage());
}
}
}
It works fine, but some times (not always)
my mozilla firefox client
automatically sends a request, when i
add a new tab (Ctrl + T) or closes an
existing tab.
When i request for
http://localhost/ , single digit gets displayed on my browser. But the
server's console shows that it get 2
or 3 requests, so, it shows 2 or 3 numbers
accordingly.
I googled a lot but don't get an satisfactory answer for this.
Any thoughts/explanations are highly appreciated. :)
First of all, I'm assuming you're doing this as an exercise to learn about sockets / http / etc... Because if you're really trying to implement a web server, you should seriously consider using an existing one or at least looking at its source - there's a LOT of stuff involved in even the most simplistic HTTP server implementation.
That said, your "server" :-) doesn't implement HTTP properly (doesn't return status code, for one) so browser's behavior is really undetermined here - it may attempt to retry the same request multiple times or it may not. Caching issue may be at play here too - once you do implement HTTP, you'll need to send appropriate caching headers if you don't want "new tab" to repeat request.