I'm trying to setup a producers/consumers using the selenium webdriver as queries/browsers.
While running the following code, not all the queries are actually entered even though the output says they are. Some queries also either get doubled ( i.e. "Fish" becomes "FisFishh" ).
APP
public class App {
public static void main(String[] args) {
DriverHashMap.init();
Executor exec = new Executor();
// Add queries to list
Query query = new Query(exec);
query.addQuery("Cats");
query.addQuery("Dogs");
query.addQuery("Fish");
// query.addQuery("Bats");
// query.addQuery("Rats");
// query.addQuery("Geese");
ExecutorService threadPool = Executors.newFixedThreadPool(4);
// Submit queries to pool
Future queryStatus = threadPool.submit(query);
// Start browsers
threadPool.execute(new Browser("1", exec));
threadPool.execute(new Browser("2", exec));
// this will wait for the producer to finish its execution.
try {
queryStatus.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
// Wait for pool to finish
threadPool.shutdown();
}
}
BROWSER
public class Browser implements Runnable {
private String name;
private Executor exec;
private static WebDriver driver;
private static String baseURL = "http://www.google.com";
private static String query;
public Browser(String name, Executor exec) {
synchronized (this) {
this.name = name;
this.exec = exec;
System.out.println("\tStart Browser-" + this.name);
driver = new FirefoxDriver();
// Wait up to 30 seconds for a response
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
DriverHashMap.addDriver(name, driver);
System.out.println("\tAdded Browser-" + name + " to hashmap");
}
}
private void enterQuery() {
synchronized (this) {
// Get driver for this browser
driver = DriverHashMap.getDriver(this.name);
System.out.println("Browser " + this.name
+ " entering query from executor: " + query);
// Search field qbqfq
driver.findElement(By.id("gbqfq")).clear();
// Enter query
driver.findElement(By.id("gbqfq")).sendKeys(query);
// Click search button
driver.findElement(By.id("gbqfb")).click();
}
}
public void run() {
try {
synchronized (this) {
// Receive first query
driver = DriverHashMap.getDriver(this.name);
query = exec.get();
System.out.println("Browser " + this.name
+ "\n\rReceived first query from executor: " + query);
}
do {
synchronized (this) {
// Process query
driver = DriverHashMap.getDriver(this.name);
driver.get(baseURL);
enterQuery();
}
synchronized (this) {
// Receive next query
query = exec.get();
driver = DriverHashMap.getDriver(this.name);
System.out.println("Browser " + this.name
+ "\n\rReceived new query from executor: " + query);
}
} while (query != null);
// Close this browser
synchronized (this) {
driver = DriverHashMap.getDriver(this.name);
System.out.println("Browser " + this.name
+ " finished its job; terminating.");
driver.close();
}
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
QUERY
public class Query implements Runnable {
private Executor exec;
private List<String> queries = new ArrayList<String>();
public Query(Executor exec) {
this.exec = exec;
}
public void addQuery( String query ) {
System.out.println("Adding " + query + " to queries");
queries.add( query );
}
public void run() {
Iterator it = queries.iterator();
String currQuery = new String();
try {
while( it.hasNext()) {
currQuery = (String)it.next();
System.out.println("Adding " + currQuery + " to Executor");
exec.put(currQuery);
}
this.exec.continueProducing = Boolean.FALSE;
System.out.println("Query has finished its job; terminating.");
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
EXECUTOR
public class Executor {
public ArrayBlockingQueue<String> queue = new ArrayBlockingQueue<String>(
100);
public Boolean continueProducing = Boolean.TRUE;
public void put(String query) throws InterruptedException {
this.queue.put(query);
}
public String get() throws InterruptedException {
return this.queue.poll(1, TimeUnit.SECONDS);
}
}
Webdriver is not threadsafe per se. Try wrapping it into a ThreadLocal.
That has helped me a lot in a similar situation.
Related
Below is the code implementation...where I need to schedule tasks at random interval and update the time of the particular tasks :
class Task implements Callable<String>
{
private final String name;
public Task(String name) {
this.name = name;
}
#Override
public String call() throws Exception {
return "Task [" + name + "] executed on : " + LocalDateTime.now().toString();
}
}
public class Main
{
public static void main(String[] args) throws InterruptedException
{
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
List<ScheduledFuture<String>> results = new ArrayList<ScheduledFuture<String>>();
for (int i = 1; i <= 5; i++)
{
Task task = new Task("Task-" + i);
ScheduledFuture<String> result = executor.schedule(task, i*2, TimeUnit.SECONDS);
results.add(result);
}
executor.shutdown();
try {
executor.awaitTermination(1, TimeUnit.DAYS);
for(ScheduledFuture<String> result : results) {
System.out.println(result.get());
}
} catch (Exception e) {
e.printStackTrace();
}
I am trying to execute 2 jobs parallel from main thread but if a callback method take long time to give response rest of requests are pause and wait to complete first.
Here is my code:
private final ExecutorService executorService = Executors.newFixedThreadPool(10);
private void executeService(String uuid) {
System.out.println("query executed done: " + uuid);
}
private String getAsynchTest(final String uuid) throws Exception {
testAsynchF = executorService.submit(
new Callable<String>() {
public String call() throws Exception {
executeService(uuid);
return getFutuerResult(uuid, Thread.currentThread()); // long processing
}
});
return testAsynchF.get();
}
public String getFutuerResult(String uuid, Thread t) {
String dummy = "your result for request: "+uuid;
if (uuid.equalsIgnoreCase("112")) {
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return dummy;
}
public static void main(String[] args) {
SynchronousTimeoutTester tester = new SynchronousTimeoutTester();
try {
String one = "112"
System.out.println("Result sync call:*** " + tester.getAsynchTest(one));
String two = "115";
System.out.println("Result sync call:**** " + tester.getAsynchTest(two));
} catch (Exception e) {
System.out.println("catched as Exception: " + e);
}
}
Why is this stopping to execute request 115 if request 112 thread is pause?
Since you pass Thread.currentThread() to getFutureResult and that method calls join() on its thread argument (in case uuid is "112"), the method will wait for its own thread to end, which it can't since it's waiting.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I am pretty new to Object Programming and Java, so I am here to gather your advice and feedback. Basically I am trying to write a background service which performs different tasks at different intervals. I'm just not 100% sure of what I am doing is following the coding standards or is efficient.
Main / Start Class:
public class Start {
public static void main(String[] args) {
Service s = new Service();
s.Start();
}
}
Database Class:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class Database {
/* Database settings */
private final String HOSTNAME = "localhost";
private final String DATABASE = "java_database";
private final String USERNAME = "java_username";
private final String PASSWORD = "java_password";
/* Database connection */
public Connection getConnection() {
try {
return DriverManager.getConnection("jdbc:mysql://" + HOSTNAME + "/" + DATABASE + "?user=" + USERNAME + "&password=" + PASSWORD + "&useSSL=false&useUnicode=true&characterSetResults=utf8");
} catch (SQLException ex) {
ex.printStackTrace();
}
return null;
}
}
Service Class:
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class Service {
private int taskId;
private int taskType;
/* Start Service */
public void Start() {
try {
System.out.println("Starting Service...");
while(true) {
System.out.print("Checking for tasks... ");
getNextTask();
if (this.taskId > 0) {
System.out.println("Task ID " + this.taskId + " found.");
switch (this.taskType) {
case 1:
System.out.println("Task 1");
SampleTask s = new SampleTask();
s.Start();
s = null;
break;
default:
System.out.println("Error: Unknown Task");
}
setUsedTask();
} else {
System.out.println("No tasks to perform at this time.");
}
this.taskId = 0;
this.taskType = 0;
Thread.sleep(5000);
}
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
/* Gets the next available task from the database */
public void getNextTask() {
try {
Database db = new Database();
String query = "select taskId, taskType "
+ "from tasks "
+ "where (time_to_sec(timediff(now(), taskLastRun)) > taskFrequency or taskLastRun is null) and taskEnabled = 1 "
+ "limit 1";
Statement stmt = db.getConnection().createStatement();
ResultSet rset = stmt.executeQuery(query);
if (rset.next()) {
this.taskId = rset.getInt(1);
this.taskType = rset.getInt(2);
}
} catch (SQLException ex) {
ex.printStackTrace();
}
}
/* Set task as complete */
public void setUsedTask() {
try {
Database db = new Database();
String query = "update tasks "
+ "set taskLastRun = now() "
+ "where taskId = ? "
+ "limit 1";
PreparedStatement pstmt = db.getConnection().prepareStatement(query);
pstmt.setInt(1, this.taskId);
pstmt.executeUpdate();
} catch (SQLException ex) {
ex.printStackTrace();
}
}
}
Consider replacing your Thread.sleep() approach with a wait() and notify() approach as discussed here.
public class Service {
private int taskId;
private int taskType;
private final Object serviceMonitor;
/* Start Service */
public void Start() {
synchronized(serviceMonitor){
try {
System.out.println("Starting Service...");
while(true) {
System.out.print("Checking for tasks... ");
getNextTask();
if (this.taskId > 0) {
System.out.println("Task ID " + this.taskId + " found.");
switch (this.taskType) {
case 1:
System.out.println("Task 1");
SampleTask s = new SampleTask();
s.Start();
s = null;
break;
default:
System.out.println("Error: Unknown Task");
}
setUsedTask();
} else {
System.out.println("No tasks to perform at this time.");
}
this.taskId = 0;
this.taskType = 0;
serviceMonitor.wait();
}
}
}
catch (InterruptedException ex) {
ex.printStackTrace();
}
}
public void getNextTask() {
synchronized(serviceMonitor){
try {
Database db = new Database();
String query = "select taskId, taskType "
+ "from tasks "
+ "where (time_to_sec(timediff(now(), taskLastRun)) > taskFrequency or taskLastRun is null) and taskEnabled = 1 "
+ "limit 1";
Statement stmt = db.getConnection().createStatement();
ResultSet rset = stmt.executeQuery(query);
if (rset.next()) {
this.taskId = rset.getInt(1);
this.taskType = rset.getInt(2);
serviceMonitor.notifyAll();
}
}
} catch (SQLException ex) {
ex.printStackTrace();
}
}
public class Main {
public static void main(String[] args) {
Service service = new Service(Arrays.asList(new SampleTask(),new AnotherTask()));
service.execute();
}
}
class Service {
private List<Task> taskList;
public Service(List<Task> taskList) {
this.taskList = taskList;
}
public void addTask(Task task) {
taskList.add(task);
}
public void execute() {
for (Task task : taskList) {
new Timer().schedule(task, task.getDelay(), task.getPeriod());
}
}
public void clearTasks() {
taskList.clear();
}
}
abstract class Task extends TimerTask {
abstract long getDelay();
abstract long getPeriod();
}
class SampleTask extends Task {
public void run() {
System.out.println("Sample task executed");
}
long getDelay() {
return 1000;
}
long getPeriod() {
return 60000;
}
}
class AnotherTask extends Task {
public void run() {
System.out.println("Another task is executed");
}
long getDelay() {
return 1000;
}
long getPeriod() {
return 500000;
}
}
I want to use threading to stream the tweets but I am getting rate-limited. What am I doing wrong?
public class AppThread {
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(5);
for (int i = 0; i < 5; i++) {
Runnable worker = new App("Thread" + i);
executor.execute(worker);//calling execute method of ExecutorService
}
executor.shutdown();
while (!executor.isTerminated()) { }
System.out.println("Finished all threads");
}
In the following class, another class(not included) is called to fetch the tweets.
public class App implements Runnable {
private String threadName;
TwitterStream twitterStream = new TwitterStreamFactory().getTwitterStreamInstance();
TwitterStreamWorker streamWorker = new TwitterStreamWorker();
public App(String name) {
this.threadName = name;
}
public void appTwitter() {
String[] queryParams = { "Delhi" };
String[] queryLang = { "en" };
FilterQuery filterQuery = new FilterQuery();
FilterQuery query = filterQuery.getFilterQueryInstance(queryParams, queryLang);
System.out.println("Query " + query);
streamWorker.startStreamingWorker(twitterStream, query);
}
public void run() {
System.out.println("Running thread " + threadName);
appTwitter();
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void start() {
// TODO Auto-generated method stub
}
}
I have also tried using newSingleThreadExecutor() but the problem remains the same.
I am creating a system that will have multiple suite deployments and each deployment will have a queue of test suites. Since I want the test suites to run concurrently on their individual suite deployment, I need to add concurrency to the code. I have created a simplified version of the code I am using, but the concurrency portion doesn't work when I try to shut it down.
When the Runner.stopEverything() gets called, the result is that the queue gets emptied, and it waits for the threads to complete, but even when the tests all complete, the wait never finishes even with the notifyAll(). The result is that the process just sits there never ending. I go look at it in debug mode and the result is that all 3 threads show waiting.
Main:
public static void main(String args[]) throws Exception {
Runner.queueTestSuites("SD1", Arrays.asList("A", "B", "C"));
Runner.queueTestSuites("SD2", Arrays.asList("D", "E", "F"));
Runner.queueTestSuites("SD3", Arrays.asList("G", "H", "I"));
Thread.sleep(5000);
System.out.println("~~~~~~~~~~~~~~~~~~~~~~~~");
Runner.stopEverything();
}
Runner:
public class Runner {
private static Map<String, TestQueue> runnerQueueMap = new ConcurrentHashMap<>();
public synchronized static void queueTestSuites(String suiteDeployment, List<String> testSuiteQueueAsJSON) throws Exception {
TestQueue queue;
if(runnerQueueMap.containsKey(suiteDeployment)) {
queue = runnerQueueMap.get(suiteDeployment);
} else {
queue = new TestQueue(suiteDeployment);
}
for (int i = 0; i < testSuiteQueueAsJSON.size(); i++) {
String name = testSuiteQueueAsJSON.get(i);
queue.addToQueue(name);
}
runnerQueueMap.put(suiteDeployment,queue);
}
public synchronized static void stopEverything() throws InterruptedException {
for (String s : runnerQueueMap.keySet()) {
TestQueue q = runnerQueueMap.get(s);
q.saveAndClearQueue();
}
for (String s : runnerQueueMap.keySet()) {
TestQueue q = runnerQueueMap.get(s);
q.waitForThread();
}
System.out.println("All done at " + new Date());
}
}
TestQueue:
public class TestQueue {
private Consumer consumer;
private Thread consumerThread;
private java.util.concurrent.BlockingQueue<String> queue;
private String suiteDeployment;
public TestQueue(String suiteDeployment) {
this.suiteDeployment = suiteDeployment;
queue = new ArrayBlockingQueue<>(100);
startConsumer();
}
public void addToQueue(String testSuite) {
try {
queue.put(testSuite);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public synchronized void waitForThread() {
try {
if (consumer.running.get()) {
synchronized (consumerThread) {
System.out.println("Waiting for " + consumerThread.getName());
consumerThread.wait();
}
}
System.out.println("Thread complete at " + new Date());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public void saveAndClearQueue() {
List<String> suiteNames = new ArrayList<>();
for (String suite : queue) {
suiteNames.add(suite);
}
queue.clear();
}
private void startConsumer() {
consumer = new Consumer(queue,suiteDeployment);
consumerThread = new Thread(consumer);
consumerThread.start();
}
private class Consumer implements Runnable{
private BlockingQueue<String> queue;
private String suiteDeployment;
public AtomicBoolean running;
public Consumer(BlockingQueue<String> queue, String suiteDeployment){
this.queue = queue;
this.suiteDeployment = suiteDeployment;
this.running = new AtomicBoolean(false);
}
#Override
public void run() {
try{
while(!Thread.currentThread().isInterrupted()) {
String testSuite = queue.take();
this.running.set(true);
new Test(testSuite, suiteDeployment).run();
this.running.set(false);
}
notifyAll();
}catch(Exception e) {
e.printStackTrace();
}
}
}
}
Test:
public class Test {
String testSuite = "";
String suiteDeployment = "";
public Test(String testSuite, String suiteDeployment) {
this.testSuite = testSuite;
this.suiteDeployment = suiteDeployment;
}
public void run() {
int time = new Random().nextInt() % 10000;
time = Math.max(time, 3000);
System.out.println("Test Started: " + testSuite + " on " + suiteDeployment + " at " + new Date() + " running for " + time + " on thread " + Thread.currentThread().getName());
try {
Thread.sleep(time);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Test Completed: " + testSuite + " on " + suiteDeployment + " at " + new Date());
}
}
Inside run method of your consumer, you have a blocking call to queue.take() which means it will block until there is an item inside your queue. You run out of elements inside the queue eventually and all your thread are blocked by the queue.take() call waiting for more elements to become available to process.
Although your call is in a while loop where it check if the thread is interrupted, you actually never interrupt the threads so it never gets to the while loop evaluation & blocked at the call to queue.take()
So your threads stay in wait as they are waiting for input to become avilable inside your blocking queue
Also your saveAndClear method must lock on the correct object which is the queue itself, like below:
public void saveAndClearQueue() {
List<String> suiteNames = new ArrayList<String>();
synchronized (queue) {
for (String suite : queue) {
suiteNames.add(suite);
}
queue.clear();
}
System.out.println("Saved(not executed) : "+suiteNames);
}
And your waitForThread method should do sth like below:
public void waitForThread() {
synchronized (consumerThread) {
while (consumer.running.get()) {
try {
consumerThread.wait(100);
} catch (InterruptedException e) {
break;
}
}
}
if (!consumer.running.get()) {
consumerThread.interrupt();
}
System.out.println("Thread complete at " + new Date());
}