I want to check an huge amount (thousands) of Websites, if they are still running. Because I want to get rid of unececarry entries in my HostFile Wikipage about Hostfiles.
I want to do it in a 2 Stage process.
Check if something is running on Port 80
Check the HTTP response code (if it's not 200 I have to check the site)
I want to multithread, because if I want to check thousands of addresses, I cant wait for timeouts.
This question is just about Step one.
I have the problem, that ~1/4 of my connect attempts don't work. If I retry the not working ones about ~3/4 work? Do I not close the Sockets correctly? Do I run into a limit of open Sockets?
Default I run 16 threads, but I have the same problems with 8 or 4.
Is there something I'm missing
I have simplified the code a little.
Here is the code of the Thread
public class SocketThread extends Thread{
int tn;
int n;
String[] s;
private ArrayList<String> good;
private ArrayList<String> bad;
public SocketThread(int tn, int n, String[] s) {
this.tn = tn;
this.n = n;
this.s = s;
good = new ArrayList<String>();
bad = new ArrayList<String>();
}
#Override
public void run() {
int answer;
for (int i = tn * (s.length / n); i < ((tn + 1) * (s.length / n)) - 1; i++) {
answer = checkPort80(s[i]);
if (answer == 1) {
good.add(s[i]);
} else {
bad.add(s[i]);
}
System.out.println(s[i] + " | " + answer);
}
}
}
And here is the checkPort80 Method
public static int checkPort80(String host)
Socket socket = null;
int reachable = -1;
try {
//One way of doing it
//socket = new Socket(host, 80);
//socket.close();
//Another way I've tried
socket = new Socket();
InetSocketAddress ina = new InetSocketAddress(host, 80);
socket.connect(ina, 30000);
socket.close();
return reachable = 1;
} catch (Exception e) {
} finally {
if (socket != null) {
if (socket.isBound()) {
try {
socket.close();
return reachable;
} catch (Exception e) {
e.getMessage();
return reachable;
}
}
}
}
}
About Threads, I make a ArrayList of Threads, create them and .start() them and right afterwards I .join() them, get the "God" and the "Bad" save them to files.
Help is appreciated.
PS: I rename the Hosts-file first so that it doesn't affect the process, so this is not an issue.
Edit:
Thanks to Marcelo Hernández Rishr I discovered, that HttpURLConnection seems to be the better solution. It works faster and I can also get the HttpResponseCode, which I was also interested anyways (just thought it would be much slower, then just checking Port 80). I still after a while suddenly get Errors, I guess this has to do with the DNS server thinking this is a DOS-Attack ^^ (but I should examine futher if the error lies somewhere else) also fyi I use OpenDNS, so maybe they just don't like me ^^.
x4u suggested adding a sleep() to the Threads, which seems to make things a little better, but will it help me raise entries/second i don't know.
Still, I can't (by far) get to the speed I wanted (10+ entries/second), even 6 entries per second doesn't seem to work.
Here are a few scenarios I tested (until now all without any sleep()).
number of time i get first round how many entries where entries/second
threads of errors processed until then
10 1 minute 17 seconds ~770 entries 10
8 3 minute 55 seconds ~2000 entries 8,51
6 6 minute 30 seconds ~2270 entries 5,82
I will try to find a sweet spot with Threads and sleep (or maybe simply pause all for one minute if I get many errors).
Problem is, there are Hostfiles with one million entries, which at one entry per second would take 11 Days, which I guess all understand, is not expectable.
Are there ways to switch DNS-Servers on the fly?
Any other suggestions?
Should I post the new questions as separate questions?
Thanks for the help until now.
I'll post new results in about a week.
I have 3 suggestions that may help you in your task.
Maybe you can use the class HttpURLConnection
Use a maximum of 10 threads because you are still limited by cpu, bandwidth, etc.
The lists good and bad shouldn't be part of your thread class, maybe they can be static members of the class were you have your main method and do static synchronized methods to add members to both lists from any thread.
Sockets usually try to shut down gracefully and wait for a response from the destination port. While they are waiting they are still blocking resources which can make successive connection attempts fail if they were executed while there have still been too many open sockets.
To avoid this you can turn off the lingering before you connect the socket:
socket.setSoLinger(false, 0);
Related
I just started learn Java and I'm stuck with this problem: I have an infinite while-loop which creates a message to send over a socket; currently the message is not send until a number of elements is poll from a queue and read them.
String msg = null;
String toSend = "";
String currentNumOfMsg = 0;
String MAX_MSG_TO_SEND = 200;
while(true) {
if ((msg = messageQueue.poll()) != null) { // if there is an element in the list
toSend += (msg + "#");
currentNumOfMsg++;
if (currentNumOfMsg == MAX_MSG_TO_SEND) {
try {
sendMessage(toSend); // send to socket
} finally {
msg = null;
toSend = "";
currentNumOfMsg = 0;
}
}
}
}
My goal is to send the message after N seconds, without waiting to reach the MAX_MSG_TO_SEND... Is it possible to do it or I shall continue with this approach?
While the other answer is perfectly valid, I thought it may be valuable to tell you that ScheduledExecutorService (documentation found here), lets you call a function foo() every n seconds using the method scheduleAtFixedRate().
Basically, the actually setting up the executor is as easy as:
ScheduledExecutorService ses = Executors.newScheduledThreadPool(1);
ses.scheduleAtFixedRate(foo, 0, n, TimeUnit.SECONDS);
I think putting any more code in here is bit unnecessary, but to see how to do this in more detail, look here, here, or here. These links give some basic examples. I would really recommend doing it this way as this class is part of the java util library (so no extra dependencies) and you don't actually have to worry very much about the multithreading/scheduling part of it, it takes care of all that for you. But thats just my $.02.
Leave a question/comment if you have one, I'll try to answer it.
Yeah, definitely you can do such a thing. But at first you should store your receive messages in a data structure and when you want to send the data via the socket, send the data in the data structure.
also, you can use guava stopWatch to send the message exactly on time. for further information, you can see https://dzone.com/articles/guava-stopwatch
Otherwise, you can use a long variable which stores System.currentTimeMillis() and each time checks if the expected elapsed time is received or not like below sample code:
long l = System.currentTimeMillis();
if(System.currentTimeMillis() - l >= 10000) {
//send data
}
everyone!
I have just created a brute force bot which uses WebDriver and multithreading to brute force a 4-digit code. 4-digit means a range of 0000 - 9999 possible String values. In my case, after clicking the "submit" button, not less than 7 seconds passes before the client gets a response from the server. So, I have decided to use Thread.sleep(7200) to let the page with a response be fully loaded. Then, I found out that I couldn't afford to wait for 9999*7,5 seconds for the task to be accomplished, so I had to use multithreading. I have a Quad-Core AMD machine with 1 virtual core per 1 hardware one, which gives me the opportunity to run 8 threads simultaneously. Ok, I have separated the whole job of 9999 combinations between 8 threads equally, each had got a scope of work of 1249 combinations + remainder thread starting at the very end. Ok, now I'm getting my job done in 1,5 hours (because the right code appears to be in the middle of the scope of work). That is much better, BUT it could be even more better! You know, the Thread.sleep(7500) is a pure waste of time. My machine could be switching to other threads which are wait() because of limited amount of hardware cores. How to do this? Any ideas?
Below are two classes to represent my architecture approach:
public class BruteforceBot extends Thread {
// All the necessary implementation, blah-blah
public void run() {
brutforce();
}
private void brutforce() {
initDriver();
int counter = start;
while (counter <= finish) {
try {
webDriver.get(gatewayURL);
webDriver.findElement(By.name("code")).sendKeys(codes.get(counter));
webDriver.findElement(By.name("code")).submit();
Thread.sleep(7200);
String textFound = "";
try {
do {
textFound = Jsoup.parse(webDriver.getPageSource()).text();
//we need to be sure that the page is fully loaded
} while (textFound.contains("XXXXXXXXXXXXX"));
} catch (org.openqa.selenium.JavascriptException je) {
System.err.println("JavascriptException: TypeError: "
+ "document.documentElement is null");
continue;
}
// Test if the page returns XXXXXXXXXXXXX below
if (textFound.contains("XXXXXXXXXXXXXXXx") && !textFound.contains("YYYYYYY")) {
System.out.println("Not " + codes.get(counter));
counter++;
// Test if the page contains "YYYYYYY" string below
} else if (textFound.contains("YYYYYYY")) {
System.out.println("Correct Code is " + codes.get(counter));
botLogger.writeTheLogToFile("We have found it: " + textFound
+ " ... at the code of " + codes.get(counter));
break;
// Test if any other case of response below
} else {
System.out.println("WTF?");
botLogger.writeTheLogToFile("Strange response for code "
+ codes.get(counter));
continue;
}
} catch (InterruptedException intrrEx) {
System.err.println("Interrupted exception: ");
intrrEx.printStackTrace();
}
}
destroyDriver();
} // end of bruteforce() method
And
public class ThreadMaster {
// All the necessary implementation, blah-blah
public ThreadMaster(int amountOfThreadsArgument,
ArrayList<String> customCodes) {
this();
this.codes = customCodes;
this.amountOfThreads = amountOfThreadsArgument;
this.lastCodeIndex = codes.size() - 1;
this.remainderThread = codes.size() % amountOfThreads;
this.scopeOfWorkForASingleThread
= codes.size()/amountOfThreads;
}
public static void runThreads() {
do {
bots = new BruteforceBot[amountOfThreads];
System.out.println("Bots array is populated");
} while (bots.length != amountOfThreads);
for (int j = 0; j <= amountOfThreads - 1;) {
int finish = start + scopeOfWorkForASingleThread;
try {
bots[j] = new BruteforceBot(start, finish, codes);
} catch (Exception e) {
System.err.println("Putting a bot into a theads array failed");
continue;
}
bots[j].start();
start = finish;
j++;
}
try {
for (int j = 0; j <= amountOfThreads - 1; j++) {
bots[j].join();
}
} catch (InterruptedException ie) {
System.err.println("InterruptedException has occured "
+ "while a Bot was joining a thread ...");
ie.printStackTrace();
}
// if there are any codes that are still remain to be tested -
// this last bot/thread will take care of them
if (remainderThread != 0) {
try {
int remainderStart = lastCodeIndex - remainderThread;
int remainderFinish = lastCodeIndex;
BruteforceBot remainderBot
= new BruteforceBot(remainderStart, remainderFinish, codes);
remainderBot.start();
remainderBot.join();
} catch (InterruptedException ie) {
System.err.println("The remainder Bot has failed to "
+ "create or start or join a thread ...");
}
}
}
I need your advise on how to improve the architecture of this app to make it successfully run with say, 20 threads instead of 8. My problem is - when I simply remove Thread.sleep(7200) and at the same time order to run 20 Thread instances instead of 8, the thread constantly fails to get a response from the server because it doesn't wait for 7 seconds for it to come. Therefore, the performance becomes not just less, it == 0; Which approach would you choose in this case?
P.S.: I order the amount of threads from the main() method:
public static void main(String[] args)
throws InterruptedException, org.openqa.selenium.SessionNotCreatedException {
System.setProperty("webdriver.gecko.driver", "lib/geckodriver.exe");
ThreadMaster tm = new ThreadMaster(8, new CodesGenerator().getListOfCodesFourDigits());
tm.runThreads();
Okay, so everyone can't wait until my question will get a response so I decided to answer it as soon as I can (now!).
If you would like to increase a performance of a Selenium WebDriver-based brute force bot like this one, you need to reject using the Selenium WebDriver. Because the WebDriver is a separate process in the OS, it does not even need a JVM to run. So, every single instance of the Bot was not only a thread managed by my JVM, but a Windows process also! This was the reason why I could hardly use my PC when this app was running with more than 8 threads (each thread was invoking a Windows process geckodriver.exe or chromedriver.exe). Okay, so what you really need to do to increase performance of such a brute force bot is to use HtmlUnit instead of Selenium! HtmlUnit is a pure Java framework, its jar could be found at Maven Central, its dependency could be added to your pom.xml. This way, brute forcing a 4-digit code takes 15 - 20 minutes, taking into account that after each attempt the website responds not faster than 7 seconds after each attempt. To compare, with Selenium WebDriver it took 90 minutes to accomplish the task.
And thanks again to #MartinJames, who has pointed that Thread.sleep() does let the hardware core to switch to other threads!
My result
Expected result
public void run () {
try {
handlers.addElement (this);
broadcast("Welcome " + name);
while(handlers.size() != 2){
if(handlers.size() > 2){
this.out.writeUTF ("The Room is full!");
this.out.flush();
handlers.removeElement(this);
socket.close();
}
}
broadcast("No of Player: " + handlers.size());
for(int i = 0; i < handlers.size(); i++){
GameHandler player = (GameHandler) handlers.get(i);
broadcast("Player " + (i + 1) + ": " + player.name);
}
System.out.println("Game starts!");
startGame(4);
....
}
protected static void broadcast (String message) {
synchronized (handlers) {
Enumeration e = handlers.elements ();
while (e.hasMoreElements ()) {
GameHandler handler = (GameHandler) e.nextElement ();
try {
handler.out.writeUTF (message);
handler.out.flush ();
} catch (IOException ex) {
handler.stop ();
}
}
}
}
The problem is the difference between the expected result and my actual result. I have no idea why the broadcast before the while loop runs normally but others run twice
Your problem is that in your case, each of the thread is sending the broadcast. Either you need to have a "master" / "server" of games thread that does the "system announcements" broadcasting, -or- elect one of the client threads (maybe the "player 1" thread?) to send the announcements.
The problem is the difference between the expected result and my actual result. I have no idea why the broadcast before the while loop runs normally but others run twice
You really don't give enough details on your problem but I see these issues:
You talk about TCP and the code mentions sockets but you are processing a local elements collection. Unless you are talking to the same JVM over TCP (which is strange) the elements collection is going to start 2 players on each client. Is that really what you expect?
Even though you says elements is a Vector you still need to synchronize on it at the start of the run() method because you are performing multiple operations on it and there are race conditions. For example, if 3 handlers are added, they will all remove themselves and close their own sockets.
Vector really is an outdated collection. You should be using something else.
When the first thread adds itself to elements it then enters a spin loop waiting for the second person to join the game. Seems like a waste there. Some small Thread.sleep(...) would be appropriate.
If the room is full I suspect that the thread should return; from the run() method. Instead it continues on which I suspect is not good.
Hope something here helps.
Basic
I have an app that is sending packets using DatagramChannel.send in multiple threads each to its own IP address/port and each of them keeping constant bit-rate/bandwidth. Every now and then I get this error:
java.net.SocketException: Invalid argument: no further information
at sun.nio.ch.DatagramChannelImpl.send0(Native Method)
at sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(Unknown Source)
at sun.nio.ch.DatagramChannelImpl.send(Unknown Source)
at sun.nio.ch.DatagramChannelImpl.send(Unknown Source)
...
It happens on random - sometimes 5 minutes after start sometimes after a day - so I really have problems reproducing it for testing. And on my home machine I can't reproduce it at all.
Environments
Windows 7, 8 and Server 2012 (all 64bit)
64bit Java 7 update 45
More information
The app is sending SI/EIT data to DVB-C network. I'm creating a list of 188-byte arrays for each of 80-120 threads and giving it to use. The thread takes the list and is looping over the list until new list is provided.
The error usually happens on multiple channels at once. But it can happen on just one also.
The error never happened until we had 40+ threads.
The error happens while looping over the list, not when I'm binding new list to thread.
The app it not running out of memory. Its usually running up to 70% of memory given to JVM.
Strange part: If I run multiple instance of app each handling ~10 threads problems are the same.
Simplified code sample
for(int i = 0; i < 100; ++i) {
final int id = i;
new Thread(new Runnable() {
#Override
public void run() {
final Random r = new Random();
final List<byte[]> buffer = Lists.newArrayList();
for(int i = 0; i < 200; ++i) {
final byte[] temp = new byte[188];
r.nextBytes(temp);
buffer.add(temp);
}
final SocketAddress target = new InetSocketAddress("230.0.0.18", 1000 + id);
try (final DatagramChannel channel = DatagramChannel.open(StandardProtocolFamily.INET)) {
channel.configureBlocking(false);
channel.setOption(StandardSocketOptions.IP_MULTICAST_IF, NetworkInterface.getByName("eth0"));
channel.setOption(StandardSocketOptions.IP_MULTICAST_TTL, 8);
channel.setOption(StandardSocketOptions.SO_REUSEADDR, true);
channel.setOption(StandardSocketOptions.SO_SNDBUF, 1024 * 64);
int counter = 0;
int index = 0;
while(true) {
final byte[] item = buffer.get(index);
channel.send(ByteBuffer.wrap(item), target);
index = (index + 1) % buffer.size();
counter++;
Thread.sleep(1);
}
}
catch(Exception e) {
LOG.error("Fail at " + id, e);
}
}
}).start();
}
Edits:
1) #EJP: I'm setting setting multicast properties as the actual app that I use was doing joins (and reading some data). But the problems persisted even after I removed them.
2) Should I be using some other API if I just need to send UDP packets? All the samples I could find use DatagramChannel (or its older alternative).
3) I'm still stuck with this. If anyone has an idea what can I even try, please let me know.
I had exactly the same problem, and it was caused by a zero port in the target InetSocketAddress, when calling the send method.
In your code, the target port is defined as 1000 + i, so it doesn't seem to be the problem. Anyway, I'd log the target parameters that are used when the exception is thrown, just in case.
My code is here from which I get result either true or false that weather it ping to the host I mention in it or not,
try
{
InetAddress address = InetAddress.getByName("192.168.1.125");
boolean reachable=address.isReachable(10000));
out.print(PingHost.DrawTable());
out.print("Is host reachable? " + reachable);
}
catch(Exception e)
{
out.print(e.printStackTrace());
}
I want to count the no of times it try to ping the host if it is not ping success fully fro the first time and the max no of count for ping would be 10
Hopes for your suggestions
Thanks in Advance
final static int MAX_PINGS = 10;
final static int TIMEOUT= 10000;
int countFailed = 0;
for (int i=0; i<MAX_PINGS; i++){
if (address.isReachable(TIMEOUT)){
System.out.println("Pinged successfully");
break;
}else{
countFailed++;
}
}
Note: giving 10000ms (10 seconds) as timeout is too much. I suggest it should be around 1000 ms.
Assuming that address.isReachable(10000)) is doing the ping, and returns true or false, then you want something like this:
int counter = 0;
do
{
counter ++;
if(address.isReachable(10000))
{
break;
}
}
while (counter < 10)
// now counter contains the number of attempts
I think you'd do well to find a good book on programming, to come up with a solution similar to this should not be something you need to ask about.
I would first question why this code needs to reside in a JSP. A request to this JSP will take forever to get back to you if the host is unreachable. Any solution that uses a member variable to track the count will also be problematic since it will run into concurrency issues.
You are better off writing LaceySnr's code on a servlet and spawning that code on a separate thread.