Basically I've got a little threading class used by ExecutorService and a fixed thread pool. Each thread instantiates my threading class and the call method is fired, works great!
However I really need to call another class (via instantiation or static means) to process & return some data within the call method, however when trying this I understandably get concurrent.ExecutionException, along with related methods.
I think it will be easier to paste all my code here, note its very rough
MainController
package com.multithreading.excutorservice;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
public class MainController {
private static List<String> urls;
public static void main(String[] args) {
populateList();
// futures to retrieve task results
List<Future<ArrayList>> futures = new ArrayList<Future<ArrayList>>();
// results
List<ArrayList> results = new ArrayList<ArrayList>();
// pool with 5 threads
ExecutorService exec = Executors.newFixedThreadPool(5);
// enqueue tasks
for(String url: urls) {
futures.add(exec.submit(new ThreadTask(url)));
}
// attempt to move ArrayLists within Future<ArrayList> into a normal ArrayList
for(Future<ArrayList> future: futures) {
try {
results.add(future.get());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
// for(ArrayList<String> s: results) {
// System.out.println(s);
// }
}
private static void populateList() {
urls = new ArrayList<String>();
urls.add("http://www.google.com");
urls.add("http://www.msn.co.uk");
urls.add("http://www.yahoo.co.uk");
urls.add("http://www.google.com");
urls.add("http://www.msn.co.uk");
urls.add("http://www.yahoo.co.uk");
}
}
ThreadTask
package com.multithreading.excutorservice;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
public class ThreadTask implements Callable<ArrayList> {
private String url;
HtmlParser parseHtml;
public ThreadTask(String url) {
this.url = url;
}
public ArrayList call() {
int counter = 0;
String html = null;
try {
URL myUrl = new URL(url);
BufferedReader reader = new BufferedReader(new InputStreamReader(myUrl.openStream()));
while ((html = reader.readLine()) != null) {
//counter += inputLine.length();
html += html;
}
}
catch (Exception ex) {
System.out.println(ex.toString());
}
ArrayList<String> storeLinks = new ArrayList<String>();
HtmlParser par = new HtmlParser();
storeLinks = par.returnNewUrls(html);
// for(String s: parseHtml) {
// System.out.println(s);
// }
//returns an ArrayList of URLS which is stored in a List<Future<ArrayList>> temporarily
return storeLinks;
}
}
HtmlParser
package com.multithreading.excutorservice;
import java.util.ArrayList;
import java.util.concurrent.Callable;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class HtmlParser {
private final String regex_links = "\\s*(?i)href\\s*=\\s*(\"([^\"]*\")|'[^']*'|([^'\">\\s]+))";
private ArrayList<String> extractedUrls;
public ArrayList<String> returnNewUrls (String data) {
extractedUrls = new ArrayList<String>();
Pattern p = Pattern.compile(regex_links);
Matcher m = p.matcher(data);
System.out.println("Test");
while (m.find()) {
System.out.println("Test");
extractedUrls.add(m.group(1));
}
return getLinks();
}
//returns the links
public ArrayList getLinks() {
return extractedUrls;
}
}
You're doing some pretty weird stuff here. Multiple threads are accessing the same static extractedUrls field, and each call to returnNewUrls creates a new field. In your returnNewUrls method, create a new ArrayList which is local to the method scope. Something along the lines of:
public static ArrayList<String> returnNewUrls(String data) {
ArrayList<String> urls = new ArrayList<String>();
addStuffToUrlsList();
return urls;
}
Another thing - not a bug, but you're doing unnecessary stuff - in the call method you don't need to create a new list if you're just assigning to a variable:
ArrayList<String> parseHtml = new ArrayList<String>();
parseHtml = HtmlParser.returnNewUrls(html);
This is better:
ArrayList<String> parseHtml = HtmlParser.returnNewUrls(html);
You have several concurrent tasks, but they use the same variable HtmlParser.extractedUrls? without any synchronization. Move this variable inside the returnNewUrls method.
BTW even without concurrency, using static variables is not encouraged, especially in a case like that, where it can easily be replaced with a local variable.
Related
I have set up a java class that I want to use for an xe:beanNamePicker. Somehow I am not able to add a created SimplePickerResult into the result set.
package se.myorg.myproject.app;
import java.io.IOException;
import java.util.List;
import java.util.Properties;
import java.util.TreeSet;
import se.sebank.namis.utils.Utils;
import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.DocumentCollection;
import lotus.domino.NotesException;
import lotus.domino.View;
import com.ibm.xsp.complex.ValueBindingObjectImpl;
import com.ibm.xsp.extlib.component.picker.data.INamePickerData;
import com.ibm.xsp.extlib.component.picker.data.IPickerEntry;
import com.ibm.xsp.extlib.component.picker.data.IPickerOptions;
import com.ibm.xsp.extlib.component.picker.data.IPickerResult;
import com.ibm.xsp.extlib.component.picker.data.SimplePickerResult;
public class DirectoryNamePicker extends ValueBindingObjectImpl implements INamePickerData {
private Utils utils;
Properties props;
public DirectoryNamePicker(){
//constructor
utils = new Utils();
utils.printToConsole(this.getClass().getSimpleName().toString() + " - DirectoryNamePicker() // constructor");
try {
props = utils.getDataSourceProperties();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public String[] getSourceLabels () {
// TODO Auto-generated method stub
return null;
}
public boolean hasCapability (final int arg0) {
// TODO Auto-generated method stub
return false;
}
public List<IPickerEntry> loadEntries (final Object[] arg0, final String[] arg1) {
// TODO Auto-generated method stub
return null;
}
#SuppressWarnings("unchecked")
public IPickerResult readEntries (final IPickerOptions options) {
String startKey = options.getStartKey();
int count = options.getCount();
TreeSet<IPickerEntry> entries = new TreeSet<IPickerEntry>();
if (startKey != null) {
// User is performing a search
try {
entries = this.dirLookup(startKey, count);
} catch (NotesException e) {
System.err.println("Exception trying to perform directory lookup: " + e.getMessage());
e.printStackTrace();
}
}
return new SimplePickerResult((List<IPickerEntry>) entries, -1);
}
public TreeSet<IPickerEntry> dirLookup(final String search, final int limit) throws NotesException {
TreeSet<IPickerEntry> result = new TreeSet<IPickerEntry>();
String server = props.getProperty("server_notesname");
String filepath = props.getProperty("db_project_data");
Database db = utils.getSession().getDatabase(server, filepath);
View vw = db.getView("vw_all_todo_namespicker");
vw.setAutoUpdate(false);
DocumentCollection dc = vw.getAllDocumentsByKey(search, false);
int count = 0;
Document tmpdoc;
Document doc = dc.getFirstDocument();
while (doc != null && count < limit) {
String person = doc.getItemValueString("app_ProjMgrName");
IPickerEntry entry = new SimplePickerResult.Entry(person, person);
result.add(entry);
// result.add(entry does not seem to work
tmpdoc = dc.getNextDocument();
doc.recycle();
doc = tmpdoc;
count = count +1;
}
vw.setAutoUpdate(true);
return result;
}
}
Is there anyone that can tell me what I m doing wrong? I have choosen a treeset instead of an arraylist. this is because I go to a view with lots of multiple entries so I do not want duplicates and have it sorted by values.
You're casting TreeSet to (List) at the line:
return new SimplePickerResult((List<IPickerEntry>) entries, -1);
because the SimplePickerResult needs a List (it won't accept a Collection), but TreeSet does not implement List, so that cast will fail.
You'll probably have to change it back to an ArrayList.
To sort, try using java.util.Collections.sort(List list, Comparator c) with a custom comparator that compares the entry.getLabel() value, as SimplePickerResult.Entry doesn't have an in-built compare method.
I have 2 class files in my simple project - sorry another newbee here!
But I get a compilation error on the last part where I am trying to print the hopefully stored configuration settings from a file for my project that will be referred to throughout the project.
The file is just rows of values like this 'ButtonConfig,8,V,NULL,bunny,mpg'
I basically want to be able to used the contents of this arraylist to dynamicly set up the configuration of a Raspberry pi GPO pins i.e. for the above values button attached to GPO pin 8 will play video (V) "<..other value...>_bunny.mpg"
Any help greatly appreciated - just telling me why I can't access the getExtension method would be nice!
Contents of first java file is -
package bpunit;
public class ButtonConfig {
private String keyword;
private String gponumber;
private String buttontype;
private String language;
private String filename;
private String extension;
public String getKeyword() {
return keyword;
}
public void setKeyword(String keyword) {
this.keyword = keyword;
}
...............
public String getExtension() {
return extension;
}
public void setExtension(String extension) {
this.extension = extension;
}
}
The second contains this -
package bpunit;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class Read_ini {
public void Read_ini_toObject()
{
String csvFileToRead = "configs/BPUnit.properties";
BufferedReader br = null;
String line;
String splitBy = ",";
List buttonList = new ArrayList();
try {
br = new BufferedReader(new FileReader(csvFileToRead));
while ((line = br.readLine()) != null) {
// split on comma(',')
String[] buttonconfig = line.split(splitBy);
// create button object to store values
ButtonConfig buttonObject = new ButtonConfig();
// add values from csv to car object
buttonObject.setKeyword(buttonconfig[0]);
buttonObject.setGponumber(buttonconfig[1]);
buttonObject.setButtontype(buttonconfig[2]);
buttonObject.setLanguage(buttonconfig[3]);
buttonObject.setFilename(buttonconfig[4]);
buttonObject.setExtension(buttonconfig[5]);
// adding button object to a list
buttonList.add(buttonObject);
}
// print values stored in buttonList
printButtonList(buttonList);
} catch (FileNotFoundException e) {
System.out.print(e);
} catch (IOException e) {
System.out.print(e);
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
System.out.print(e);
}
}
}
}
public void printButtonList(List buttonListToPrint) {
for (int i = 0; i < buttonListToPrint.size(); i++) {
// THE LINE BELOW FAILS - getExtension() does not exist
// and all other attempts give me pointer references
//instead of the text //
System.out.println(buttonListToPrint.get(i).getExtension());
}
}
}
You have to add the parameterized type ButtonConfig to your ArrayList. It ends up being List<ButtonConfig> instead of just List.
package bpunit;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class Read_ini {
public void Read_ini_toObject()
{
String csvFileToRead = "configs/BPUnit.properties";
BufferedReader br = null;
String line;
String splitBy = ",";
List<ButtonConfig> buttonList = new ArrayList<ButtonConfig>();
try {
br = new BufferedReader(new FileReader(csvFileToRead));
while ((line = br.readLine()) != null) {
// split on comma(',')
String[] buttonconfig = line.split(splitBy);
// create button object to store values
ButtonConfig buttonObject = new ButtonConfig();
// add values from csv to car object
buttonObject.setKeyword(buttonconfig[0]);
buttonObject.setGponumber(buttonconfig[1]);
buttonObject.setButtontype(buttonconfig[2]);
buttonObject.setLanguage(buttonconfig[3]);
buttonObject.setFilename(buttonconfig[4]);
buttonObject.setExtension(buttonconfig[5]);
// adding button object to a list
buttonList.add(buttonObject);
}
// print values stored in buttonList
printButtonList(buttonList);
} catch (FileNotFoundException e) {
System.out.print(e);
} catch (IOException e) {
System.out.print(e);
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
System.out.print(e);
}
}
}
}
public void printButtonList(List<ButtonConfig> buttonListToPrint) {
for (int i = 0; i < buttonListToPrint.size(); i++) {
// THE LINE BELOW FAILS - getExtension() does not exist
// and all other attempts give me pointer references
//instead of the text //
System.out.println(buttonListToPrint.get(i).getExtension());
}
}
}
The reason why the compilation is failing is because when you add an object to the ArrayList it is upcast as an object of the class Object. Now when you extract it you simply have to typecast it back to the original type. so all you have to do is this :
public void printButtonList(List buttonListToPrint) {
for (int i = 0; i < buttonListToPrint.size(); i++) {
// THE LINE BELOW FAILS - getExtension() does not exist
// and all other attempts give me pointer references
//instead of the text
ButtonConfig buttonObject =(ButtonConfig)buttonListToPrint.get(i);
System.out.println(buttonObject.getExtension());
}
}
Or as mentioned in the comments and answers above you could use generics and create an List of type ButtonConfig
public void Read_ini_toObject()
{
String csvFileToRead = "configs/BPUnit.properties";
BufferedReader br = null;
String line;
String splitBy = ",";
List<ButtonConfig> buttonList = new ArrayList<ButtonConfig>();
and pass it in the function printButtonList
public void printButtonList(List<ButtonConfig> buttonListToPrint) {
for (int i = 0; i < buttonListToPrint.size(); i++) {
// THE LINE BELOW FAILS - getExtension() does not exist
// and all other attempts give me pointer references
//instead of the text
System.out.println(buttonListToPrint.get(i).getExtension());
}
}
How can re-implement this using concurrent executor, or just a much better way. meaning threadpool executor .
Basically i want the crawler to crawl the given url and maybe later follow the urls found to another website and so one.
package Mainpackge;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class main {
public static void main(String[] args) {
//List of urs to collect data from
String[] urls = new String[]{
"http://www.answers.com/",
"http://www.britannica.com/",
"https://ie.yahoo.com/?p=us",
"https://en.wikipedia.org/wiki/Main_Page",
"http://ww w.worldbook.com/",
"http://www.computerlanguage.com/",
"http://www.howstuffworks.com/",
"http://www.dmoz.org/Computers/Computer_Science/"
};
// Create and start workers
List<Worker> workers = new ArrayList<>(urls.length);
for (String url : urls) {
Worker w = new Worker(url);
workers.add(w);
new Thread(w).start();
}
// Retrieve results
for (Worker w : workers) {
Elements results = w.waitForResults();
if (results != null)
for (Element result : results) { result.absUrl("a") ;
System.out.println(w.getName()+": "+result.absUrl("href"));
}
else
System.err.println(w.getName()+" had some error!");
}
}
}
class Worker implements Runnable {
private String url;
private Elements results;
private String name;
private static int number = 0;
private final Object lock = new Object();
public Worker(String url) {
this.url = url;
this.name = "Worker-" + (number++);
}
public String getName() {
return name;
}
#Override
public void run() {
try {
Document doc = Jsoup.connect(this.url).get();
Elements links = doc.select("a");
// Update results
synchronized (lock) {
this.results = links;
lock.notifyAll();
}
} catch (IOException e) {
// You should implement a better error handling code..
System.err.println("Error while parsing: "+this.url);
e.printStackTrace();
}
}
public Elements waitForResults() {
synchronized (lock) {
try {
while (this.results == null) {
lock.wait();
}
return this.results;
} catch (InterruptedException e) {
// Again better error handling
e.printStackTrace();
}
return null;
}
}
}
Full example using an ExecutorService and Callable implementation for your threads.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
public class ThreadPoolExample {
public static void main(String[] args) throws InterruptedException, ExecutionException {
List<String> urls = Arrays.asList(new String[]{
"http://www.answers.com/",
"http://www.britannica.com/",
"https://ie.yahoo.com/?p=us",
"https://en.wikipedia.org/wiki/Main_Page",
"http://ww w.worldbook.com/",
"http://www.computerlanguage.com/",
"http://www.howstuffworks.com/",
"http://www.dmoz.org/Computers/Computer_Science/"
});
ExecutorService ex = Executors.newFixedThreadPool(10);
ex.awaitTermination(2, TimeUnit.SECONDS);
List<Future<Element>> results = new ArrayList<>();
for (String string : urls) {
results.add(ex.submit(new Crawler(string)));
}
for (Future<Element> future : results) {
// Get will wait for the thread to be done
for (String url : future.get().urls) {
// ADD A NEW THREAD FOR EACH URLS YOU FOUND !
ex.submit(new Crawler(url));
}
}
ex.shutdown();
}
public static class Crawler implements Callable<Element>{
String url;
public Crawler(String url) {
this.url = url;
}
#Override
public Element call() throws Exception {
// Implement your crawling logic and return your elements
return new Element(Arrays.asList(new String[]{"all new urls", "that you found while crwaling"}));
}
}
public static class Element{
List<String> urls;
public Element(List<String> urls) {
this.urls = urls;
}
#Override
public String toString() {
return "Elements found : " + urls.size();
}
}
}
I wan't to mock the CSVReader. So my mock should return a new array each time, this should be generic.
the last value should be null.
ex
nextLine() -> ["a","b","c"]
nextLine() -> ["a","b","c"]
nextLine() -> null
My MockClass:
import au.com.bytecode.opencsv.CSVReader;
import com.sun.javafx.beans.annotations.NonNull;
import org.mockito.Mockito;
import org.mockito.stubbing.OngoingStubbing;
import java.io.IOException;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class CSVReaderMock {
CSVReader reader;
private boolean linesCorrectInitialized;
public CSVReaderMock()
{
reader = mock(CSVReader.class);
}
public CSVReaderMock returnLines(#NonNull List<String> lines) {
// the last value has to be null
lines.add(null);
try {
for (String line : lines) {
String[] lineArr = null;
if (line != null) {
lineArr = line.split(",");
}
when(reader.readNext()).thenReturn(lineArr);
}
linesCorrectInitialized = true;
} catch (IOException e) {
e.printStackTrace();
};
return this;
}
public CSVReader create() {
if (!linesCorrectInitialized) { throw new RuntimeException("lines are not initialized correct"); }
return reader;
}
}
and here a testcase (i only writed to check my mock builder):
#Test
public void testImportLines() throws Exception {
CSVReader reader;
List<String> list = new LinkedList<>();
list.add("some,lines,for,testing");
reader = new CSVReaderMock().returnLines(list).create();
System.out.println(reader.readNext()); // should return [Ljava.lang.String;#xxxx with conent-> ["some","lines","for","testing"]
System.out.println(reader.readNext()); // should return null
}
the actual output is :
null
null
So my question is, how can i pass a list of return values without knowing in advance how the list will look? I know i could pass the "csv lines" via .thenReturn(line1,line2,line3) but this will break my approach.
Mockito has a ReturnsElementsOf Answer for just such an occasion.
Returns elements of the collection. Keeps returning the last element forever. Might be useful on occasion when you have a collection of elements to return.
So now you just need to prepare the elements and then pass that in. Because the null call will need to be added at the very end, it'll keep you from reusing the CSVReaderMock builder, but that's the same whether or not you use the answer.
List<String[]> returnList = new ArrayList<>();
public CSVReaderMock returnLines(#NonNull List<String> lines) {
try {
for (String line : lines) {
String[] lineArr = null;
if (line != null) {
lineArr = line.split(",");
}
returnList.add(lineArr);
}
linesCorrectInitialized = true;
} catch (IOException e) { /* ... */ };
return this;
}
public CSVReader create() {
if (!linesCorrectInitialized) { /* ... */ }
// Return null repeatedly after all stubs are exhausted.
returnList.add(null);
when(reader.readNext()).thenAnswer(new ReturnsElementsOf(returnList));
return reader;
}
You need to chain the return values into a single when() result by using the OngoingStubbing reference, e.g:
Iterator<String> ls = org.mockito.Mockito.mock(Iterator.class);
OngoingStubbing<String> stubbing = when(ls.next());
for(String s: new String[] { "ABC", "DEF" }) {
stubbing = stubbing.thenReturn(s);
}
System.out.println(ls.next());
System.out.println(ls.next());
... prints "ABC", then "DEF".
I have something like this:
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;
import javax.xml.stream.*;
import javax.xml.stream.events.*;
public class MyClass implements javax.xml.stream.StreamFilter
{
private Map myMap= new HashMap();
public Map getMap()
{
return this.myMap;
}
public void addElement(String text)
{
this.myMap.put(text, "value");
}
public void doSomething(String strValue)
{
try
{
XMLInputFactory xmlif = null;
xmlif = XMLInputFactory.newInstance();
FileInputStream fis = new FileInputStream("myFile.xml");
XMLStreamReader xmlr = xmlif.createFilteredReader(xmlif.createXMLStreamReader(fis),new MyClass());
XMLOutputFactory outputFactory = XMLOutputFactory.newInstance();
OutputStream fileStream = new FileOutputStream("myFileOutput");
XMLStreamWriter xmlWriter = outputFactory.createXMLStreamWriter(fileStream);
addElement(strValue);
System.out.println(getMap().size()+"Before");
while (xmlr.hasNext())
{
write(xmlr, xmlWriter);
xmlr.next();
}
System.out.println(getMap().size()+"After");
}
catch(Exception e)
{
e.printStackTrace();
}
}
public boolean accept(XMLStreamReader reader)
{
System.out.println(getMap().size()+"inside");
if ( getMap().containsKey("Something") )
{
System.out.println("The value is there");
}
return true;
}
private void write(XMLStreamReader xmlr, XMLStreamWriter writer) throws XMLStreamException
{
switch (xmlr.getEventType()) {
case XMLEvent.START_ELEMENT:
String localName = xmlr.getLocalName();
writer.writeStartElement(localName);
break;
}
}
}
When I create the tagMap as static, it works, but since I will put this class in threads, all the maps will point to the same map, making the comparison useless. Specifically I have seen that inside the function accept, the map has 0 values.
The function accept is part of the interface SteamFilter, so it is called automatically when an event occurs. I have tried changing the map to public, and even setting the values with a constructor, but it is the same.
The way it is being called in the class that creates the class, is this:
Runnable runClass = new Runnable(){
public void run(){
try
{
MyClass myC = new MyClass();
myC.doSomething("value");
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Thread myTask = new Thread(runClass);
myTask.start();
Do you know how can I check for the value of the map inside the function accept?
You instanciate your FilterReader with a new Instance of MyClass (one that hasn't been called doSomething yet). If you instead start it with the same instance that is currently be worked on it should work.
Change:
xmlif.createFilteredReader(xmlif.createXMLStreamReader(fis),new MyClass())
to
xmlif.createFilteredReader(xmlif.createXMLStreamReader(fis),this)
I presume every thread has a reference to an instance (presumably the same instance) of MyMap.
They should just be able to call getMap() on the reference.
Q: Am I missing something????