I have a data structure like this:
BlockingQueue mailbox = new LinkedBlockingQueue();
I'm trying to do this:
for(Mail mail: mailbox)
{
if(badNews(mail))
{
mailbox.remove(mail);
}
}
Obviously the contents of the loop interfere with the bounds and a error is triggered, so I would normally do this:
for(int i = 0; i < mailbox.size(); i++)
{
if(badNews(mailbox.get(i)))
{
mailbox.remove(i);
i--;
}
}
But sadly BlockingQueue's don't have a function to get or remove an element by index, so I'm stuck. Any ideas?
Edit - A few clarifications:
One of my goals is the maintain the same ordering so popping from the head and putting it back into the tail is no good. Also, although no other threads will remove mail from a mailbox, they will add to it, so I don't want to be in the middle of an removal algorithm, have someone send me mail, and then have an exception occur.
Thanks in advance!
You may p̶o̶p̶ poll and p̶u̶s̶h̶ offer all the elements in your queue until you make a complete loop over your queue. Here's an example:
Mail firstMail = mailbox.peek();
Mail currentMail = mailbox.pop();
while (true) {
//a base condition to stop the loop
Mail tempMail = mailbox.peek();
if (tempMail == null || tempMail.equals(firstMail)) {
mailbox.offer(currentMail);
break;
}
//if there's nothing wrong with the current mail, then re add to mailbox
if (!badNews(currentMail)) {
mailbox.offer(currentMail);
}
currentMail = mailbox.poll();
}
Note that this approach will work only if this code is executed in a single thread and there's no other thread that removes items from this queue.
Maybe you need to check if you really want to poll or take the elements from the BlockingQueue. Similar for offer and put.
More info:
Java BlockingQueue take() vs poll()
LinkedBlockingQueue put vs offer
Another less buggy approach is using a temporary collection, not necessarily concurrent, and store the elements you still need in the queue. Here's a kickoff example:
List<Mail> mailListTemp = new ArrayList<>();
while (mailbox.peek() != null) {
Mail mail = mailbox.take();
if (!badNews(mail)) {
mailListTemp.add(mail);
}
}
for (Mail mail : mailListTemp) {
mailbox.offer(mail);
}
I looked over the solutions posted and I think I found a version that serves my purposes. What do you think about this one?
int size = mailbox.size();
for(int i = 0; i < size; i++)
{
Mail currentMail = mailbox.poll();
if (!badNews(currentMail))
mailbox.offer(currentMail);
}
Edit: A new solution that may be problem free. What you guys think?
while(true)
{
boolean badNewRemains = false;
for(Mail mail: mailbox)
{
if(badNews(mail))
{
badNewRemains = true;
mailbox.remove(mail);
break;
}
}
if(!badNewRemains)
break;
}
You can easily implement queue for your need. And you will need to, if API provided doesn't have such features.
One like:
import java.util.Iterator;
import java.util.LinkedList;
class Mail {
boolean badMail;
}
class MailQueue {
private LinkedList<Mail> backingQueue = new LinkedList<>();
private final Object lock = new Object();
public void push(Mail mail){
synchronized (lock) {
backingQueue.addLast(mail);
if(backingQueue.size() == 1){
// this is only element in queue, i.e. queue was empty before, so invoke if any thread waiting for mails in queue.
lock.notify();
}
}
}
public Mail pop() throws InterruptedException{
synchronized (lock) {
while(backingQueue.isEmpty()){
// no elements in queue, wait.
lock.wait();
}
return backingQueue.removeFirst();
}
}
public boolean removeBadMailsInstantly() {
synchronized (lock) {
boolean removed = false;
Iterator<Mail> iterator = backingQueue.iterator();
while(iterator.hasNext()){
Mail mail = iterator.next();
if(mail.badMail){
iterator.remove();
removed = true;
}
}
return removed;
}
}
}
The implemented queue will be thread-safe, whether push or pop. Also you can edit queue for more operations. And it will allow to access removeBadMailsInstantly method by multiple threads (thread-safe). And you will also learn concepts of multithreading.
Related
I have a Hashmap that is created for each "mailer" class and each "agent" class creates a mailer.
My problem is that each of my "agents" creates a "mailer" that in turn creates a new hashmap.
What I'm trying to do is to create one Hashmap that will be used by all the agents(every agent is a thread).
This is the Agent class:
public class Agent implements Runnable {
private int id;
private int n;
private Mailer mailer;
private static int counter;
private List<Integer> received = new ArrayList<Integer>();
#Override
public void run() {
System.out.println("Thread has started");
n = 10;
if (counter < n - 1) {
this.id = ThreadLocalRandom.current().nextInt(0, n + 1);
counter++;
}
Message m = new Message(this.id, this.id);
this.mailer.getMap().put(this.id, new ArrayList<Message>());
System.out.println(this.mailer.getMap());
for (int i = 0; i < n; i++) {
if (i == this.id) {
continue;
}
this.mailer.send(i, m);
}
for (int i = 0; i < n; i++) {
if (i == this.id) {
continue;
}
if (this.mailer.getMap().get(i) == null) {
continue;
} else {
this.received.add(this.mailer.readOne(this.id).getContent());
}
}
System.out.println(this.id + "" + this.received);
}
}
This is the Mailer class :
public class Mailer {
private HashMap<Integer, List<Message>> map = new HashMap<>();
public void send(int receiver, Message m) {
synchronized (map) {
while (this.map.get(receiver) == null) {
this.map.get(receiver);
}
if (this.map.get(receiver) == null) {
} else {
map.get(receiver).add(m);
}
}
}
public Message readOne(int receiver) {
synchronized (map) {
if (this.map.get(receiver) == null) {
return null;
} else if (this.map.get(receiver).size() == 0) {
return null;
} else {
Message m = this.map.get(receiver).get(0);
this.map.get(receiver).remove(0);
return m;
}
}
}
public HashMap<Integer, List<Message>> getMap() {
synchronized (map) {
return map;
}
}
}
I have tried so far :
Creating the mailer object inside the run method in agent.
Going by the idea (based on your own answer to this question) that you made the map static, you've made 2 mistakes.
do not use static
static means there is one map for the entire JVM you run this on. This is not actually a good thing: Now you can't create separate mailers on one JVM in the future, and you've made it hard to test stuff.
You want something else: A way to group a bunch of mailer threads together (these are all mailers for the agent), but a bit more discerning than a simple: "ALL mailers in the ENTIRE system are all the one mailer for the one agent that will ever run".
A trivial way to do this is to pass the map in as argument. Alternatively, have the map be part of the agent, and pass the agent to the mailer constructor, and have the mailer ask the agent for the map every time.
this is not thread safe
Thread safety is a crucial concept to get right, because the failure mode if you get it wrong is extremely annoying: It may or may not work, and the JVM is free to base whether it'll work right this moment or won't work on the phase of the moon or the flip of a coin: The JVM is given room to do whatever it feels like it needs to, in order to have a JVM that can make full use of the CPU's powers regardless of which CPU and operating system your app is running on.
Your code is not thread safe.
In any given moment, if 2 threads are both referring to the same field, you've got a problem: You need to ensure that this is done 'safely', and the compiler nor the runtime will throw errors if you fail to do this, but you will get bizarre behaviour because the JVM is free to give you caches, refuse to synchronize things, make ghosts of data appear, and more.
In this case the fix is near-trivial: Use java.util.concurrent.ConcurrentHashMap instead, that's all you'd have to do to make this safe.
Whenever you're interacting with a field that doesn't have a convenient 'typesafe' type, or you're messing with the field itself (one thread assigns a new value to the field, another reads it - you don't do that here, there is just the one field that always points at the same map, but you're messing with the map) - you need to use synchronized and/or volatile and/or locks from the java.util.concurrent package and in general it gets very complicated. Concurrent programming is hard.
I was able to solve this by changing the mailer to static in the Agent class
I'm building a webcrawler to download files from websites. I've a producer (the link fetcher) and a consumer (the downloader).
They both can be summarized as followed :
//Fetcher implements Runnable
public void run(){
while(String link = getLinkFromDatabase != null){
String htmlContent = HTTPrequest.getHTMLtoString(link);
ArrayList<String> links = HTTPrequest.getUrlsFromString(htmlContent); //Custom Parser/Extractor
ArrayList<String> files = HTTPrequest.getFilesFromString(htmlContent);//Custom Parser/Extractor
String SqlQueryAddLinks = "INSERT IGNORE DUPLICATE INTO [...]"; //Insert query for Links with unique key : sha256 of the url.
String SqlQUeryAddFiles = "INSERT IGNORE DUPLICATE INTO [...]"; //Insert query for Files with unique key : sha256 of the url.
Queries.sqlExec(SqlQueryAddLinks);
int RowAffected = Queries.sqlExec(SqlQueryAddFiles);
Queries.archiveLink(link);
Monitor.append(RowAffected);
}
}
//Downloader implements Runnable
public void run(){
while(String link = getFileFromeDatabase != null){
//You don't care of steps here I just download the file
if(fileDownloaded){
Queries.archiveFile(link);
Monitor.take();
}
}
}
Now i'm trying to synch both thread to assure that links cannot be too old. To do so I'm using Monitor (as described in Operating Systems : Internals and design principles wrote by William Stallings)
public class Monitor{
int N = 10;
int count;
Condition notfull, notempty;
public Monitor(){
count = 0;
}
public void append(int nbr) throws InterruptedException{
if(count >= N){
notfull.wait();
}
count+=nbr;
notempty.signal();
}
public void take() throws InterruptedException{
if(count == 0){
notempty.wait();
}
count--;
notfull.signal();
}
Now the thing is that I want to launch multiples couples of fetcher and downloader sync by a monitor. Do I need to create a new Monitors object and add a Monitor into the class of my Downloader and Fetcher or is there a better way ? The book isn't talking about multiples Producer/Consumer and is using the function parbegin(producer, consumer); in C++ (I presume it's C++).
Just by eyeballing, this code doesn't compile for many reasons and has guaranteed runtime failures.
a) you try to call a static method take/append but they are not static.
b) you try to have 2 Condition objects but you have no reentrant lock.
c) you don't even lock/unlock the reentrant lock behind the condition before waiting/notifying
d) you use Condition.wait() instead of the .await().
e) you are using Condition.signal() instead of the .signalAll()
I have set of connection objects (library code I cannot change) that have a send method. If the sending fails, they call back a generic onClosed listener which I implement that calls removeConnection() in my code, which will remove the connection from the collection.
The onClosed callback is generic and can be called at any time. It is called when the peer closes the connection, for example, and not just when a write fails.
However, if I have some code that loops over my connections and sends, then the onClosed callback will attempt to modify a collection during iteration.
My current code creates a copy of the connections list before each iteration over it; however, in profiling this has shown to be very expensive.
Set<Connection> connections = new ....;
public void addConnection(Connection conn) {
connections.add(conn);
conn.addClosedListener(this);
}
#Override void onClosed(Connection conn) {
connections.remove(conn);
}
void send(Message msg) {
// how to make this so that the onClosed callback can be safely invoked, and efficient?
for(Connection conn: connections)
conn.send(msg);
}
How can I efficiently cope with modifying collections during iteration?
To iterate a collection with the concurrent modification without any exceptions use List Iterator.
http://www.mkyong.com/java/how-do-loop-iterate-a-list-in-java/ - example
If you use simple for or foreach loops, you will receive ConcurrentModificationException during the element removing - be careful on that.
As an addition, you could override the List Iterator with your own one and add the needed logic. Just implement the java.util.Iterator interface.
A ConcurrentSkipListSet is probably what you want.
You could also use a CopyOnWriteArraySet. This of course will still make a copy, however, it will only do so when the set is modified. So as long as Connection objects are not added or removed regularly, this would be more efficient.
You can also use ConcurrentHashMap.
ConcurrentHashMap is thread-safe, so you don't need to make a copy in order to be able to iterate.
Take a look at this implementation.. http://www.java2s.com/Tutorial/Java/0140__Collections/Concurrentset.htm
I would write a collection wrapper that:
Keeps a set of objects that are to be removed. If the iteration across the underlying collection comes across one of these it is skipped.
On completion of iteration, takes a second pass across the list to remove all of the gathered objects.
Perhaps something like this:
class ModifiableIterator<T> implements Iterator<T> {
// My iterable.
final Iterable<T> it;
// The Iterator we are walking.
final Iterator<T> i;
// The removed objects.
Set<T> removed = new HashSet<T>();
// The next actual one to return.
T next = null;
public ModifiableIterator(Iterable<T> it) {
this.it = it;
i = it.iterator();
}
#Override
public boolean hasNext() {
while ( next == null && i.hasNext() ) {
// Pull a new one.
next = i.next();
if ( removed.contains(next)) {
// Not that one.
next = null;
}
}
if ( next == null ) {
// Finished! Close.
close();
}
return next != null;
}
#Override
public T next() {
T n = next;
next = null;
return n;
}
// Close down - remove all removed.
public void close () {
if ( !removed.isEmpty() ) {
Iterator<T> i = it.iterator();
while ( i.hasNext() ) {
if ( removed.contains(i.next())) {
i.remove();
}
}
// Clear down.
removed.clear();
}
}
#Override
public void remove() {
throw new UnsupportedOperationException("Not supported.");
}
public void remove(T t) {
removed.add(t);
}
}
public void test() {
List<String> test = new ArrayList(Arrays.asList("A","B","C","D","E"));
ModifiableIterator i = new ModifiableIterator(test);
i.remove("A");
i.remove("E");
System.out.println(test);
while ( i.hasNext() ) {
System.out.println(i.next());
}
System.out.println(test);
}
You may need to consider whether your list could contain null values, in which case you will need to tweak it somewhat.
Please remember to close the iterator if you abandon the iteration before it completes.
E.g.
I have a queue
void someMethod() {
history.add(new Sample(time, data));
...
traverse(history);
}
void traverse(Queue<Sample> history) {
for(int i=0; i<history.size(); i=i+10) {
history.get(i)... // ???
}
}
class Sample {
long time;
double data;
}
The concerns are that
I don't want to destroy this queue by calling traverse().
Traverse the queue in a given step, say 10 here.
Any simple and nice solution?
for (Sample s : history)
doStuff(s);
This is called the enhanced for-loop; you can read more about it here.
Queue implements Iterable, so a simple loop will traverse it:
for (Sample sample : history)
An Iterator is another way to do it, with more control (can destroy it if you want to), but more verbose.
If you just want to iterate, use a for-each loop or directly a for loop with an Iterator. This doesn't consume the queue.
If you need to iterate with a step, you can use this pattern. It works generally with any Iterable. Putting the skipping into a separate reusable method makes the code more clear than having two nested for loops.
public static void main(String[] args) {
Queue<Sample> history = ...
int step = 10;
for (Iterator<Sample> it = history.iterator();
it.hasNext(); skip(it, step - 1)) {
// note that we skipped 1 less elements than the size of the step
Sample sample = it.next();
// stuff
}
}
static void skip(Iterator<?> iterator, int count) {
for (int i = 0; i < count && iterator.hasNext(); i++) {
iterator.next();
}
}
LinkedList<Sample> h = (LinkedList<Sample>) history;
for(int i=0; i < h.size(); i+=step) {
h.get(i).memory ...
}
I just realized this approach, haven't tried it yet.
As nullptr pointed out, the condition for above code is that the Queue is implemented as a LinkedList. (which is my case: Queue<Sample> history = new LinkedList<Sample>();)
I receive this exception
Exception in thread "Thread-3" java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761)
at java.util.LinkedList$ListItr.next(LinkedList.java:696)
at ServerMultiThread.run(ServerMultiThread.java:89)
at java.lang.Thread.run(Thread.java:680)
from this code:
synchronized(itemsList)
{
if(itemsList.isEmpty())
{
item.initCounter();
itemsList.add(item);
pw.println("It's the first time you connect to server!");
}
else
{
for(ItemClient itm : itemsList)
{
if(itm.equals(item))
{
int value = itm.getCounter();
value++;
itm.setCounter(value);
pw.println(itm.getCounter());
}
else
{
item.initCounter();
itemsList.add(item);
pw.println("It's the first time you connect to server!");
}
}
}
}
the row 89 corresponds to this for(ItemClient itm : itemsList). Why I receive this error?
You are changing the LinkedList content inside the for-each loop. The implementation of the LinkedList iterator checks on the next call to next() if the list has changed and throws the exception (sad I know ...).
The enhanced for loop is used to iterate over the given set of values, but during iteration you are modifying the contents of the same, that's why you getting that error, instead use a normal for loop to do your stuff for this thing.
Regards
Sadly, there is no easy way around it. As the others have said, you cannot modify a collection inside this kind of loop. Your other option is to use a normal for loop. However, accessing a LinkedList by index like:
for(int i = 0; i < list.size(); i++) {
list.get(i);
}
takes O(N) time for each item, because the linked list needs to be traversed from the beginning each time.
If the linked list is not essential to your algorithm, I suggest you to use an ArrayList instead and change your code as follows:
for(int i = 0; i < itemsList.size(); i++) {
itm = itemsList.get(i);
if(itm.equals(item)) {
int value = itm.getCounter();
value++;
itm.setCounter(value);
pw.println(itm.getCounter());
} else {
item.initCounter();
itemsList.add(item);
pw.println("It's the first time you connect to server!");
}
}
This will not throw the exception, but it's still not a nice piece of code because you are adding to the list while iterating and that is never a good idea.
I hope you had patience to read so far!
My final suggestion for you is to hold a temporary list of elements that you need to add and append them to the initial list at the end of the loop. This way you can keep all your original code and the LinkedList:
LinkedList<ItemClient> tempList = new LinkedList<ItemClient>();
for(ItemClient itm: itemsList) {
itm = itemsList.get(i);
if(itm.equals(item)) {
int value = itm.getCounter();
value++;
itm.setCounter(value);
pw.println(itm.getCounter());
} else {
item.initCounter();
tempList.add(item);
pw.println("It's the first time you connect to server!");
}
}
itemsList.addAll(tempList);