I want to use threading to stream the tweets but I am getting rate-limited. What am I doing wrong?
public class AppThread {
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(5);
for (int i = 0; i < 5; i++) {
Runnable worker = new App("Thread" + i);
executor.execute(worker);//calling execute method of ExecutorService
}
executor.shutdown();
while (!executor.isTerminated()) { }
System.out.println("Finished all threads");
}
In the following class, another class(not included) is called to fetch the tweets.
public class App implements Runnable {
private String threadName;
TwitterStream twitterStream = new TwitterStreamFactory().getTwitterStreamInstance();
TwitterStreamWorker streamWorker = new TwitterStreamWorker();
public App(String name) {
this.threadName = name;
}
public void appTwitter() {
String[] queryParams = { "Delhi" };
String[] queryLang = { "en" };
FilterQuery filterQuery = new FilterQuery();
FilterQuery query = filterQuery.getFilterQueryInstance(queryParams, queryLang);
System.out.println("Query " + query);
streamWorker.startStreamingWorker(twitterStream, query);
}
public void run() {
System.out.println("Running thread " + threadName);
appTwitter();
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void start() {
// TODO Auto-generated method stub
}
}
I have also tried using newSingleThreadExecutor() but the problem remains the same.
Related
I'm trying to get data from four different sources with four scanners. I do realize that I need to use threads. But here's the error message:
P.S = paths for the files were fine before using thread. ( I was using one file, path was ok.)
null
null
null
null
java.lang.ArrayIndexOutOfBoundsException: 41705
at getText.getCities(getText.java:132)
at getText$1.run(getText.java:23)
at java.lang.Thread.run(Unknown Source)
java.lang.ArrayIndexOutOfBoundsException: 41705
at getText.getNames(getText.java:112)
at getText$2.run(getText.java:30)
at java.lang.Thread.run(Unknown Source)
Code Itself
public class getText {
static int ln = 41705;
static String [] icaos = new String[ln];
static String [] iatas = new String[ln];
static String [] names = new String[ln];
static String [] cities = new String[ln];
public final Runnable typeA;
public final Runnable typeB;
public final Runnable typeC;
public final Runnable typeD;
public getText() {
typeA = new Runnable() {
public void run() {
getText.this.getCities();
}
};
typeB = new Runnable() {
public void run() {
getText.this.getNames();
}
};
typeC = new Runnable() {
public void run() {
getText.this.getIcao();
}
};
typeD = new Runnable() {
public void run() {
getText.this.getIata();
}
};
}
public static void main(String[] args) {
// TODO Auto-generated method stub
getText x = new getText();
new Thread(x.typeA).start();
new Thread(x.typeB).start();
new Thread(x.typeC).start();
new Thread(x.typeD).start();
System.out.println(icaos[32541]);
System.out.println(iatas[32541]);
System.out.println(names[32541]);
System.out.println(cities[32541]);
}
public void getIcao () {
try {
int i=0;
InputStream icao_stream = new FileInputStream("src/icao.txt");
Scanner icao_s = new Scanner(icao_stream);
icao_s.useDelimiter(",");
while(icao_s.hasNext()) {
icaos[i] = icao_s.next();
i++;
}
icao_s.close();
icao_stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public void getIata() {
try {
int i=0;
InputStream iata_stream = new FileInputStream("src/iata.txt");
Scanner iata_s = new Scanner(iata_stream);
iata_s.useDelimiter(",");
while(iata_s.hasNext()) {
iatas[i] = iata_s.next();
i++;
}
iata_s.close();
iata_stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public void getNames() {
try {
int i=0;
InputStream names_stream = new FileInputStream("src/names.txt");
Scanner names_s = new Scanner(names_stream);
names_s.useDelimiter(",");
while(names_s.hasNext()) {
names[i] = names_s.next();
i++;
}
names_s.close();
names_stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public void getCities() {
try {
int i=0;
InputStream cities_stream = new FileInputStream("src/cities.txt");
Scanner cities_s = new Scanner(cities_stream);
cities_s.useDelimiter(",");
while(cities_s.hasNext()) {
cities[i] = cities_s.next();
i++;
}
cities_s.close();
cities_stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
So the problem now is you have started your threads, but the control returns immediately to the main thread and your other threads are not finished working by then. What you need is to wait for those threads to finish working and then check for results. There are multiple ways to achieve that. A simple one is to change your main to:
public static void main(String[] args) {
getText x = new getText();
List<Thread> threads = new ArrayList<>();
threads.add(new Thread(x.typeA));
threads.add(new Thread(x.typeB));
threads.add(new Thread(x.typeC));
threads.add(new Thread(x.typeD));
threads.forEach(t -> t.start());
threads.forEach(t -> {
try {
//this makes the main thread to wait for thread "t" to finish
t.join();
} catch (InterruptedException e) {
// log or throw or anything you need to do
}
});
System.out.println(icaos[32541]);
System.out.println(iatas[32541]);
System.out.println(names[32541]);
System.out.println(cities[32541]);
}
This is not the most optimal solution but the simplest one. Others better options would be to use a CyclicBarrier or ExecutorService among others.
Below is the code implementation...where I need to schedule tasks at random interval and update the time of the particular tasks :
class Task implements Callable<String>
{
private final String name;
public Task(String name) {
this.name = name;
}
#Override
public String call() throws Exception {
return "Task [" + name + "] executed on : " + LocalDateTime.now().toString();
}
}
public class Main
{
public static void main(String[] args) throws InterruptedException
{
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
List<ScheduledFuture<String>> results = new ArrayList<ScheduledFuture<String>>();
for (int i = 1; i <= 5; i++)
{
Task task = new Task("Task-" + i);
ScheduledFuture<String> result = executor.schedule(task, i*2, TimeUnit.SECONDS);
results.add(result);
}
executor.shutdown();
try {
executor.awaitTermination(1, TimeUnit.DAYS);
for(ScheduledFuture<String> result : results) {
System.out.println(result.get());
}
} catch (Exception e) {
e.printStackTrace();
}
I have an object A on which I'm updating some data every second and other objects B and C which want to use the data only once per update.
Every object work in parallel.
How can I make B and C wait for the update in A ?
I've seen some similar questions but their responses didn't help me.
I've seen that I could use a "synchronized" bloc on an object D, but they just put the bloc without telling how to instanciate or share that object.
The following code is what I use for my tests. I managed to get them working in parallel but I'm stuck with the suspending part.
This is the class for A
public class Master{
public static void main(String[] args) throws Exception {
Worker B = new Worker("B");
B.start();
Worker C = new Worker("C");
C.start();
while(true)
{
Thread.sleep(1000);
// update data
// notify every thread waiting that they can resume
}
}
}
This is the class used for B and C
public class Worker extends Thread
{
Worker(String name)
{
super("Worker " + name);
}
public void run()
{
int i = 0;
while(!this.isInterrupted())
{
// wait for A to update data
System.out.println(i);
i++;
}
System.out.println("thread interrupted");
}
}
From there, what do I need to add for the purpose I'm looking for ?
To do it very low level, only using the lang APIs, you should use wait/notifyAll.
Not that I used Main.class as an arbitrary object to synchronize
public class Main {
public static void main(String[] args) {
SharedData sharedData = new SharedData();
Worker w1 = new Worker("Worker 1", sharedData);
Worker w2 = new Worker("Worker 2", sharedData);
w1.start();
w2.start();
while (true) {
try {
Thread.sleep(1000);
sharedData.increase();;
System.out.println("Master: " + sharedData.value());
synchronized (Main.class) {
Main.class.notifyAll();
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
class SharedData {
private int data = 0;
public void increase () {
data++;
}
public int value() {
return data;
}
}
class Worker extends Thread {
private String workerName;
private SharedData sharedData;
public Worker(String workerName, SharedData sharedData) {
super();
this.workerName = workerName;
this.sharedData = sharedData;
}
#Override
public void run() {
while (true) {
try {
synchronized (Main.class) {
Main.class.wait();
}
System.out.println(workerName + ": " + sharedData.value());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
Not sure if I understand you correctly, but this might be worth checking out for you:
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CountDownLatch.html
Why use threads at all? Why not just do this?
public class Master {
public static void main(String[] args) {
Worker B = new Worker("B");
Worker C = new Worker("C");
while(true) {
Thread.sleep(1000);
updateData();
B.doWork();
C.doWork();
}
}
}
public class Worker
{
public void doWork() {
System.out.println(i);
i++;
}
private int i = 0;
}
i am trying to solve consumer-producer problem with multi-Threading by using ConcurrentHashMap and newFixedThreadPool.
my goal is to make diffrerent threads to put and remove from hashmap simultaneously and Ensure that map size will not be bigger than MAXQUEUE,
unique key to each element.
the program below isn't behave as i descirbed, it's fill the map until the size is 20 and then it removes 20 and so on.
I need some help to make it bahave as the description,
also i will be glad to get suggestion to improve the code.
this is my Producer Class:
public class Producer extends Thread
{
static final int MAXQUEUE = 20;
private ConcurrentHashMap<Long, String> myMap = new ConcurrentHashMap<Long, String>();
private AtomicLong m_Key = new AtomicLong(0);
public void run()
{
try
{
while (true)
{
putMessage();
}
} catch (InterruptedException e)
{
}
}
private void putMessage() throws InterruptedException
{
synchronized(this)
{
while (myMap.size() == MAXQUEUE)
{
wait();
}
myMap.put(this.m_Key.incrementAndGet(), "Hello");
System.out.println(Thread.currentThread().getName() + " put message; key " + this.m_Key);
notify();
//Later, when the necessary event happens, the thread that is running it calls notify() from a block synchronized on the same object.
}
}
// Called by Consumer
public void removeElementFromMap() throws InterruptedException
{
synchronized(this)
{
notify();
while (myMap.size() == 0)
{
wait();
}
for (Iterator<Entry<Long, String>> iter = this.myMap.entrySet().iterator() ; iter.hasNext() ; )
{
Map.Entry<Long, String> entry = iter.next();
System.out.println("Removed element with key " + entry.getKey() );
iter.remove();
}
}
}
}
Consumer Class:
public class Consumer extends Thread
{
Producer producer;
public void Consumer(Producer p)
{
producer = p;
}
public void run()
{
try
{
while (true)
{
producer.removeElementFromMap();
}
} catch (InterruptedException e)
{
e.printStackTrace();
}
}
Main Class:
public class Main
{
public static void main(String[] args)
{
Producer producer = new Producer();
ExecutorService producersExecutors = Executors.newFixedThreadPool(5);
Consumer consumer = new Consumer(producer);
ExecutorService consumersExecutors = Executors.newFixedThreadPool(5);
for(int i=0;i<5;i++)
{
producersExecutors.execute(producer);
consumersExecutors.execute(consumer);
}
}
}
You can use blocking queues available in Java e.g. ArrayBlockingQueue.
But if you still want to use map and your own way of handling this then may be you can do it like below -
public class MapTest {
public static void main(String[] args) {
DataStore dataStore = new DataStore(100);
ExecutorService executorService = Executors.newFixedThreadPool(10);
for (int i = 0; i < 5; i++) {
executorService.execute(new Producer(dataStore));
executorService.execute(new Consumer(dataStore));
}
}
}
class DataStore {
private final int maxQueueSize;
private Lock lock = new ReentrantLock();
private AtomicInteger queueSize = new AtomicInteger(0);
private AtomicLong keyGenerator = new AtomicLong(0);
private Map<Long, String> map = new HashMap<Long, String>();
public DataStore(int maxQueueSize) {
this.maxQueueSize = maxQueueSize;
}
public void putMessage() throws InterruptedException {
while (queueSize.get() == maxQueueSize) {
Thread.sleep(10);
}
lock.lock();
try {
if (queueSize.get() < maxQueueSize) {
map.put(keyGenerator.incrementAndGet(), "Hello");
queueSize.incrementAndGet();
System.out.println(Thread.currentThread().getName() + " put message; key " + keyGenerator.get() + ", queue size: " + queueSize.get());
}
} finally {
lock.unlock();
}
}
public void removeMessage() throws InterruptedException {
while (queueSize.get() == 0) {
Thread.sleep(10);
}
lock.lock();
try {
if (queueSize.get() > 0) {
Iterator<Long> keyIterator = map.keySet().iterator();
if (keyIterator.hasNext()) {
Long key = keyIterator.next();
map.remove(key);
queueSize.decrementAndGet();
System.out.println(Thread.currentThread().getName() + " removed message; key: " + key + ", queue size: " + queueSize.get());
}
}
} finally {
lock.unlock();
}
}
}
class Producer implements Runnable {
private DataStore dataStore;
public Producer(DataStore dataStore) {
this.dataStore = dataStore;
}
public void run() {
try {
while (true) {
dataStore.putMessage();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
class Consumer implements Runnable {
private DataStore dataStore;
public Consumer(DataStore dataStore) {
this.dataStore = dataStore;
}
public void run() {
try {
while (true) {
dataStore.removeMessage();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Problem description : -
Step 1: Take input FILE_NAME from user at main thread.
Step 2: Perform 10 operations on that file (i.e count chars, count lines etc.. ), and all those 10 operations must be in septate threads. It means there must be 10 child threads.
Step 3: Main thread waits until all those child threads completed.
Step 4: Print result.
What I did :-
I did a sample code with 3 threads. I don't want file operation code from your side.
public class ThreadTest {
// This is object to synchronize on.
private static final Object waitObject = ThreadTest.class;
// Your boolean.
private static boolean boolValue = false;
public final Result result = new Result();
public static void main(String[] args) {
final ThreadTest mytest = new ThreadTest();
System.out.println("main started");
new Thread(new Runnable() {
public void run() {
System.out.println("Inside thread");
//Int initialiser
new Thread(new Runnable() {
public void run() {
System.out.println("Setting integer value");
mytest.result.setIntValue(346635);
System.out.println("Integer value seted");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
//String initialiser
new Thread(new Runnable() {
public void run() {
System.out.println("Setting string value");
mytest.result.setStringValue("Hello hi");
System.out.println("String value seted");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
//Boolean initialiser
new Thread(new Runnable() {
public void run() {
System.out.println("Setting boolean value");
mytest.result.setBoolValue(true);
System.out.println("Boolean value seted");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
System.out.println("Thread is finished");
//Notify to main thread
synchronized (ThreadTest.waitObject) {
ThreadTest.boolValue = true;
ThreadTest.waitObject.notifyAll();
}
}
}).start();
try {
synchronized (ThreadTest.waitObject) {
while (!ThreadTest.boolValue) {
ThreadTest.waitObject.wait();
}
}
} catch (InterruptedException ie) {
ie.printStackTrace();
}
System.out.println("main finished");
System.out.println("Result is : " + mytest.result.toString());
}
}
Problem :-
My above code is not giving correct answer. How can I do that?
Alternate solutions:
CountDownLatch class does the same. But I don't want to use that class.
I looked this similar solution and I want to use methods of Thread only.
You can do:
Thread t = new Thread() {
public void run() {
System.out.println("text");
// other complex code
}
};
t.start();
t.join();
This way you will wait until the thread finishes and just then continue. You can join multiple threads:
for (Thread thread : threads) {
thread.join();
}
I would recommend looking at the Executors framework first, and then look into the CompletionService.
Then you can write something like this:
ExecutorService executor = Executors.newFixedThreadPool(maxThreadsToUse);
CompletionService completion = new ExecutorCompletionService(executor);
for (each sub task) {
completion.submit(new SomeTaskYouCreate())
}
// wait for all tasks to complete.
for (int i = 0; i < numberOfSubTasks; ++i) {
completion.take(); // will block until the next sub task has completed.
}
executor.shutdown();
In Java 8 a far better approach is to use parallelStream()
Note: it is far easier to see exactly what these background tasks are doing.
public static void main(String[] args) {
Stream.<Runnable>of(
() -> mytest.result.setIntValue(346635),
() -> mytest.result.setStringValue("Hello hi"),
() -> mytest.result.setBoolValue(true) )
.parallel()
.forEach(Runnable::run);
System.out.println("main finished");
System.out.println("Result is : " + mytest.result.toString());
}
I took out the debug information and the sleep as these don't alter the outcome.
You may want to choose CountDownLatch from java.util.concurrent. From JavaDocs:
A synchronization aid that allows one or more threads to wait until a
set of operations being performed in other threads completes.
Sample code:
import java.util.concurrent.CountDownLatch;
public class Test {
private final ChildThread[] children;
private final CountDownLatch latch;
public Test() {
this.children = new ChildThread[4];
this.latch = new CountDownLatch(children.length);
children[0] = new ChildThread(latch, "Task 1");
children[1] = new ChildThread(latch, "Task 2");
children[2] = new ChildThread(latch, "Task 3");
children[3] = new ChildThread(latch, "Task 4");
}
public void run() {
startChildThreads();
waitForChildThreadsToComplete();
}
private void startChildThreads() {
Thread[] threads = new Thread[children.length];
for (int i = 0; i < threads.length; i++) {
ChildThread child = children[i];
threads[i] = new Thread(child);
threads[i].start();
}
}
private void waitForChildThreadsToComplete() {
try {
latch.await();
System.out.println("All child threads have completed.");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private class ChildThread implements Runnable {
private final String name;
private final CountDownLatch latch;
protected ChildThread(CountDownLatch latch, String name) {
this.latch = latch;
this.name = name;
}
#Override
public void run() {
try {
// Implementation
System.out.println(name + " has completed.");
} finally {
latch.countDown();
}
}
}
public static void main(String[] args) {
Test test = new Test();
test.run();
}
}
Output:
Task 1 has completed.
Task 4 has completed.
Task 3 has completed.
Task 2 has completed.
All child threads have completed.
There are many ways to approach this. Consider CountDownLatch:
import java.util.concurrent.CountDownLatch;
public class WorkerTest {
final int NUM_JOBS = 3;
final CountDownLatch countDownLatch = new CountDownLatch(NUM_JOBS);
final Object mutex = new Object();
int workData = 0;
public static void main(String[] args) throws Exception {
WorkerTest workerTest = new WorkerTest();
workerTest.go();
workerTest.awaitAndReportData();
}
private void go() {
for (int i = 0; i < NUM_JOBS; i++) {
final int fI = i;
Thread t = new Thread() {
public void run() {
synchronized(mutex) {
workData++;
}
try {
Thread.sleep(fI * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
countDownLatch.countDown();
}
};
t.start();
}
}
private void awaitAndReportData() throws InterruptedException {
countDownLatch.await();
synchronized(mutex) {
System.out.println("All workers done. workData=" + workData);
}
}
}
Check if all child threads are dead, every n seconds. Simple, yet effective method:
boolean allDead=false;
while(! allDead){
allDead=true;
for (int t = 0; t < threadCount; t++)
if(threads[t].isAlive()) allDead=false;
Thread.sleep(2000);
}