Develop Reference

Threads are not running simultaneously to read files

I want to read multiple files through multi threading I wrote the code for the same but my threads are executing one by one which is very time consuming. I wants them to run simultaneously.
Please correct me what I am doing wrong in the below code where I am doing this by implementing the callable interface because I have to read the file and set its data into the variable of Model object and after that I am returning the list of objects.
Thanks In advance.
Class A{
ExecutorService executor = getExecuterService();
private ExecutorService getExecuterService() {
int threadPoolSize = Runtime.getRuntime().availableProcessors() - 1;
System.out.println("Number of COre" + threadPoolSize);
return Executors.newFixedThreadPool(threadPoolSize);
}
#SuppressWarnings({ "unused", "unchecked" })
FutureTask<List<DSection>> viewList = (FutureTask<List<DSection>>) executor
.submit(new MultiThreadedFileReadForDashboard(DashboardSectionList, sftpChannel,customQuery));
executor.shutdown();
while (!executor.isTerminated()) {
}
}
Class for task:
public class MultiThreadedFileReadForDashboard implements Callable {
public MultiThreadedFileReadForDashboard(List<DSection> dashboardSectionList, ChannelSftp sftpChannel,
CustomQueryImpl customQuery) {
this.dashboardSectionList = dashboardSectionList;
this.sftpChannel = sftpChannel;
this.customQuery = customQuery;
}
public List<DSection> call() throws Exception {
for (int i = 0; i < dashboardSectionList.size(); ++i) {
DSection DSection = dashboardSectionList.get(i);
List<LView> linkedViewList = new ArrayList<LView>(DSection.getLinkedViewList());
LView lView;
for (int j = 0; j < linkedViewList.size(); ++j) {
lView = linkedViewList.get(j);
int UserQueryId = Integer.parseInt(lView.getUserQueryId());
outputFileName = customQuery.fetchTableInfo(UserQueryId);
if ((outputFileName != null) && (!outputFileName.equalsIgnoreCase(""))) {
String data = readFiles(outputFileName);
lView.setData(data);
} else {
lView.setData("No File is present");
}
}
if (size == dashboardSectionList.size()) {
break;
}
}
return dSectionList;
}
private String readFiles(String outputFileName) {
String response = null;
try {
InputStream in = sftpChannel.get(outputFileName);
BufferedReader br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
StringBuilder inputData = new StringBuilder("");
String line = null;
while ((line = br.readLine()) != null) {
inputData.append(line).append("\n");
}
JSONArray array = null;
if (outputFileName.toLowerCase().contains("csv")) {
array = CDL.toJSONArray(inputData.toString());
} else {
}
response = array.toString();
} catch (Exception e) {
e.printStackTrace();
}
return response;
// TODO Auto-generated method stub
}
}
}

I do not see read multiple files through multi threading. I see one task invoked by the ExecuterService and it is reading all the files. the multi threading feature is achieved by submitting multiple tasks to the ExecuterService, each is given one file to process (can be by constructor).
Here is what I think you should do:
inside the inner for loop, you construct a task that is given outputFileName in constructor and submit it to the executor, getting back a Future instance. after all tasks were submitted, you will have a List<Future> that you can query to see when they are done and get result. the task will call readFiles() (odd name for a method that reads one file...)

Wait in ForkJoin Pool ( Java )

I am using Fork join pool in java for multitasking. Now i came across a situation where, for every task, I need to hit a url then wait for 10 minutes and then again hit another url to read the data. Now the problem is that for those 10 minutes my CPU is idle and not starting another tasks ( more than those defined in fork join pool).
static ForkJoinPool pool = new ForkJoinPool(10);
public static void main(String[] args){
List<String> list = new ArrayList<>();
for(int i=1; i<=100; i++){
list.add("Str"+i);
}
final Tasker task = new Tasker(list);
pool.invoke(task);
public class Tasker extends RecursiveAction{
private static final long serialVersionUID = 1L;
List<String> myList;
public Tasker(List<String> checkersList) {
super();
this.myList = checkersList;
}
#Override
protected void compute() {
if(myList.size()==1){
System.out.println(myList.get(0) + "start");
//Date start = new Date();
try {
Thread.sleep(10*60*1000);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(myList.get(0) + "Finished");
}
else{
List<String> temp = new ArrayList<>();
temp.add( myList.get( myList.size()-1 ) );
myList.remove( myList.size()-1 );
Tasker left = new Tasker(myList);
Tasker right = new Tasker(temp);
left.fork();
right.compute();
left.join();
}
}
Now What should I do so that CPU picks all the tasks and then wait parallaly for them.

Unfortunately, ForkJoinPool does not work well in the face of Thread.sleep(), because it designed for many short tasks that finish quickly, rather than tasks that block for a long time.
Instead, for what you are trying to accomplish, I would recommend using ScheduledThreadPoolExecutor and dividing your task into two parts.
import java.util.*;
import java.util.concurrent.*;
public class Main {
static ScheduledThreadPoolExecutor pool = new ScheduledThreadPoolExecutor(10);
public static void main(String[] args){
for(int i=1; i<=100; i++){
pool.schedule(new FirstHalf("Str"+i), 0, TimeUnit.NANOSECONDS);
}
}
static class FirstHalf implements Runnable {
String name;
public FirstHalf(String name) {
this.name = name;
}
public void run() {
System.out.println(name + "start");
pool.schedule(new SecondHalf(name), 10, TimeUnit.MINUTES);
}
}
static class SecondHalf implements Runnable {
String name;
public SecondHalf(String name) {
this.name = name;
}
public void run() {
System.out.println(name + "Finished");
}
}
}
If Java provides a thread pool which allows releasing the underlying resources (that is, the kernel thread participating in the thread pool) during a Thread.sleep(), you should use that instead, but I currently do not know of one.

According to docs forkJoin basic use section tells:
if (my portion of the work is small enough)
do the work directly
else
split my work into two pieces
invoke the two pieces and wait for the results
Hopefully this meets your need if you are using forkjoin
public class Tasker extends RecursiveAction {
static ForkJoinPool pool = new ForkJoinPool(10);
static int threshold = 10;
public static void main(String[] args){
List<String> list = new ArrayList<>();
for(int i=1; i<=100; i++){
list.add("Str"+i);
}
final Tasker task = new Tasker(list);
pool.invoke(task);
}
private static final long serialVersionUID = 1L;
List<String> myList;
public Tasker(List<String> checkersList) {
super();
this.myList = checkersList;
}
void computeDirectly() {
for(String url : myList){
System.out.println(url + " start");
}
//Date start = new Date();
try {
//keep hitting url
while (true) {
for(String url : myList) {
//url hitting code here
System.out.println(url + " hitting");
}
Thread.sleep(10 * 60 * 1000);
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
for(String url : myList){
System.out.println(url + " Finished");
}
}
#Override
protected void compute() {
if (myList.size() <= threshold) {
computeDirectly();
return;
}
//temp list have only one url
//List<String> temp = new ArrayList<>();
//temp.add( myList.get( myList.size()-1 ) );
//myList.remove( myList.size()-1 );
//Tasker left = new Tasker(myList);
//Tasker right = new Tasker(temp);
//left.fork();
//right.compute();
//left.join();
List<String> first = new ArrayList<>();
List<String> second = new ArrayList<>();
//divide list
int len = myList.size();
int smHalf = len / 2;//smaller half
first = myList.subList(0, smHalf);
second = myList.subList(smHalf + 1, len);
invokeAll(new Tasker(first), new Tasker(second));
}
}

initialize all the metadata at once

I have a Singleton class which connects to Cassandra. I want to initialize processMetadata, procMetadata and topicMetadata all at once not one by one. If they gets initialize all at once then I will see consistent values from all those three not different values for either of them.
In the below code, processMetadata, procMetadata and topicMetadata is initialized for the first time inside initializeMetadata method and then it gets updated every 15 minutes.
public class CassUtil {
private static final Logger LOGGER = Logger.getInstance(CassUtil.class);
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
// below are my three metedata which I need to update all three at once not one by one
private List<ProcessMetadata> processMetadata = new ArrayList<>();
private List<ProcMetadata> procMetadata = new ArrayList<>();
private List<String> topicMetadata = new ArrayList<>();
private Session session;
private Cluster cluster;
private static class Holder {
private static final CassUtil INSTANCE = new CassUtil();
}
public static CassUtil getInstance() {
return Holder.INSTANCE;
}
private CassUtil() {
List<String> servers = TestUtils.HOSTNAMES;
String username = TestUtils.USERNAME;
String password = TestUtils.PASSWORD;
PoolingOptions opts = new PoolingOptions();
opts.setCoreConnectionsPerHost(HostDistance.LOCAL,
opts.getCoreConnectionsPerHost(HostDistance.LOCAL));
Builder builder = Cluster.builder();
cluster =
builder
.addContactPoints(servers.toArray(new String[servers.size()]))
.withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)
.withPoolingOptions(opts)
.withReconnectionPolicy(new ConstantReconnectionPolicy(100L))
.withLoadBalancingPolicy(
DCAwareRoundRobinPolicy
.builder()
.withLocalDc(
!TestUtils.isProduction() ? "DC2" : TestUtils.getCurrentLocation()
.get().name().toLowerCase()).build())
.withCredentials(username, password).build();
try {
session = cluster.connect("testkeyspace");
} catch (NoHostAvailableException ex) {
LOGGER.logError("error= ", ExceptionUtils.getStackTrace(ex));
} catch (Exception ex) {
LOGGER.logError("error= " + ExceptionUtils.getStackTrace(ex));
}
}
// start a background thread which runs every 15 minutes
public void startScheduleTask() {
scheduler.scheduleAtFixedRate(new Runnable() {
public void run() {
try {
processMetadata = processMetadata(true);
topicMetadata = listOfTopic(TestUtils.GROUP_ID);
procMetadata = procMetadata();
} catch (Exception ex) {
LOGGER.logError("error= ", ExceptionUtils.getStackTrace(ex));
}
}
}, 0, 15, TimeUnit.MINUTES);
}
// called from main thread to initialize the metadata
// and start the background thread where it gets updated
// every 15 minutes
public void initializeMetadata() {
processMetadata = processMetadata(true);
topicMetadata = listOfTopic(TestUtils.GROUP_ID);
procMetadata = procMetadata();
startScheduleTask();
}
private List<String> listOfTopic(final String consumerName) {
List<String> listOfTopics = new ArrayList<>();
String sql = "select topics from topic_metadata where id=1 and consumerName=?";
try {
// get data from cassandra
} catch (Exception ex) {
LOGGER.logError("error= ", ExceptionUtils.getStackTrace(ex), ", Consumer Name= ",
consumerName);
}
return listOfTopics;
}
private List<ProcessMetadata> processMetadata(final boolean flag) {
List<ProcessMetadata> metadatas = new ArrayList<>();
String sql = "select * from process_metadata where id=1 and is_active=?";
try {
// get data from cassandra
} catch (Exception ex) {
LOGGER.logError("error= ", ExceptionUtils.getStackTrace(ex), ", active= ", flag);
}
return metadatas;
}
private List<ProcMetadata> procMetadata() {
List<ProcMetadata> metadatas = new ArrayList<>();
String sql = "select * from schema where id=1";
try {
// get data from cassandra
} catch (SchemaParseException ex) {
LOGGER.logError("schema parsing error= ", ExceptionUtils.getStackTrace(ex));
} catch (Exception ex) {
LOGGER.logError("error= ", ExceptionUtils.getStackTrace(ex));
}
return metadatas;
}
public List<ProcessMetadata> getProcessMetadata() {
return processMetadata;
}
public List<String> getTopicMetadata() {
return topicMetadata;
}
public List<ProcMetadata> getProcMetadata() {
return procMetadata;
}
}
So from my main thread, I call initializeMetadata method only once which initializes those three metadata and then it starts a background thread which updates them every 15 minutes. Afer that I was using them like below from my multiple threads:
CassUtil.getInstance().getProcessMetadata();
CassUtil.getInstance().getTopicMetadata();
CassUtil.getInstance().getProcMetadata();
Problem Statement:-
Now I want to see same state of processMetadata, topicMetadata and procMetadata. Meaning these three metadata should be updated at same time not one after other bcoz I don't want to see mix state value for them after I do get on them.
How can I avoid this issue? Do I need to create another class which will hold these three metadata as constructor parameter?

The most efficient way to keep a consistent state of your lists can be to use an immutable class that will hold your 3 lists, you will then have a field of this type in your class that you will define volatile to make sure that all threads see the latest update of this field.
Here is for example the immutable class that we use to hold the state of the lists (it could be an ordinary class but as it is implementation specific it could be a static inner class):
private static class State {
private final List<ProcessMetadata> processMetadata;
private final List<ProcMetadata> procMetadata;
private final List<String> topicMetadata;
public State(final List<ProcessMetadata> processMetadata,
final List<ProcMetadata> procMetadata, final List<String> topicMetadata) {
this.processMetadata = new ArrayList<>(processMetadata);
this.procMetadata = new ArrayList<>(procMetadata);
this.topicMetadata = new ArrayList<>(topicMetadata);
}
// Getters
}
Then your class would be something like that:
public class CassUtil {
...
private volatile State state = new State(
new ArrayList<>(), new ArrayList<>(), new ArrayList<>()
);
...
public void startScheduleTask() {
...
this.state = new State(
processMetadata(true), listOfTopic(TestUtils.GROUP_ID),
procMetadata()
);
...
}
...
public void initializeMetadata() {
this.state = new State(
processMetadata(true), listOfTopic(TestUtils.GROUP_ID), procMetadata()
);
startScheduleTask();
}
...
public List<ProcessMetadata> getProcessMetadata() {
return this.state.getProcessMetadata();
}
public List<String> getTopicMetadata() {
return this.state.getTopicMetadata();
}
public List<ProcMetadata> getProcMetadata() {
return this.state.getProcMetadata();
}

Need help in multi-threading

I am creating a crawler like app which will find images in the web pages. In here the producer produces the links and the consumer connects to that link for finding the images, but consumer was taking a lot of time because of huge number of links produced by the consumer. So I placed the consumer in the executor service but I could not find decrease in time taken by the consumer. Kindly help me out. Below is my code.
#Service
#Qualifier("crawlerService")
public class CrawlerService {
#Autowired
#Qualifier("loggerService")
LoggerService loggerService;
#Autowired
#Qualifier("imageTypeExtensionCombo")
ImageTypeExtensionCombo imageTypeExtensionCombo;
public List<String> startCrawler(List<String> links, List<String> images, URL url, String protocol, String protocolHost) throws Exception{
LinkQueue queue = new LinkQueue(links);
LinkProducer producer = new LinkProducer(links, url, protocol, protocolHost, queue, loggerService);
LinkConsumer consumer = new LinkConsumer(links, images, url, protocol, protocolHost, loggerService, queue);
ExecutorService executorService = Executors.newFixedThreadPool(4);
executorService.submit(consumer);
producer.start();
//consumer.start();
Thread.currentThread().join();
executorService.shutdown();
return images;
}
}
LinkProducer class
public class LinkProducer extends Thread {
private List<String> anchorList;
private URL url;
private String protocol;
private String protocolHost;
private UrlValidator urlValidator = new UrlValidator();
private LinkQueue queue;
private LoggerService loggerService;
private int MAX_QUEUE_SIZE = 2;
private int counter = 0;
private boolean stopThread = false;
private String HTML_TYPE = "HTML";
private String HTML_CONTENT_TYPE = "text/html";
private String IMAGE_TYPE = "IMAGE";
private String NON_HTML_NON_IMAGE_TYPE = "OTHERS";
public LinkProducer(List<String> anchorList, URL url, String protocol,String protocolHost, LinkQueue queue, LoggerService loggerService) {
super(protocolHost.replace(protocol, "").replaceAll("/", ""));
this.anchorList = anchorList;
this.url = url;
this.protocol = protocol;
this.protocolHost = protocolHost;
this.queue = queue;
this.loggerService = loggerService;
}
public void run() {
int i = 0;
while(true) {
List<String> anchors = null;
loggerService.log("Producer Thread : " + (++i));
try {
anchors = produce();
} catch (Exception ex) {
loggerService.log("Exception occured in producer thread : "+ ex.getMessage());
ex.printStackTrace();
if(stopThread){
break;
}
}
if(stopThread){
break;
}
if(anchors != null && anchors.size() > 0){
Iterator<String> iter = anchors.iterator();
while(iter.hasNext()){
synchronized (queue) {
queue.enQueue(iter.next());
}
}
}
}
}
}
LinkConsumer class
public class LinkConsumer extends Thread {
private List<String> anchorList;
private List<String> imageList;
private URL url;
private String protocol;
private String protocolHost;
private LinkQueue queue;
private LoggerService loggerService;
private UrlValidator urlValidator = new UrlValidator();
private String HTML_TYPE = "HTML";
private String HTML_CONTENT_TYPE = "text/html";
private String IMAGE_TYPE = "IMAGE";
private String NON_HTML_NON_IMAGE_TYPE = "OTHERS";
public LinkConsumer(List<String> anchorList, List<String> imageList, URL url, String protocol,String protocolHost, LoggerService loggerService, LinkQueue queue) {
super(protocolHost.replace(protocol, "").replaceAll("/", ""));
this.anchorList = anchorList;
this.imageList = imageList;
this.url = url;
this.protocol = protocol;
this.protocolHost = protocolHost;
this.queue = queue;
this.loggerService = loggerService;
}
public void run() {
int i = 0;
while (!queue.isEmpty()) {
List<String> images = null;
loggerService.log("Consumer Thread : " + (++i));
try {
images = consume();
} catch (Exception ex) {
loggerService.log("Exception occured in consumer thread : "+ ex.getMessage());
ex.printStackTrace();
}
if (images != null && images.size() > 0) {
Iterator<String> iter = images.iterator();
while (iter.hasNext()) {
imageList.add(iter.next());
}
}
}
}
}
Thanks

You create and submit only a single LinkConsumer and therefore you have only one worker.
To achieve real parallel performance you will need to create and submit more LinkConsumer.

Multi-threading doesn't give you much of an advantage. Infact, it increases the complexity, when you are creating too many threads and when your hardware isn't capable enough of processing these threads.
Multi-threading would give you significant gains, only if you use it efficiently. If you keep creating threads this way, you aren't going to gain any performance improvements.
Your hardware, especially processor specs and the volume of data you are writing to your disk are the primary constraints, which are going to decide the performance that you would get.
I would suggest the following.
Have multiple machines. One machine, which acts as producer writes all URL's or images or what ever you want to a database. The client systems take the URL's from DB and fetch the data from the source.
Technically, you have multiple systems which are working and each machine could have ~10 active threads at a time. And you need to code only once and run the same code on multiple machines. You could use the same producer machine as consumer too.

You can try something like this to create new threads. But I'm not sure that creating new threads will increase time too much. You will need better Hardware as well.
public boolean secondThread(){
Thread t = new Thread(){
public void run(){
//do somehting
}
};
t.start();
return true;
}

Use threading to process file chunk by chunk

I have a arraylist which consists of 5000 IP Addresses. For each IP Address, I want to execute a SNMPGet request and a FTPDownload command. I want to implement it in a fashion, wherein at a time 2 different threads run simultaneously for the first five IP Addresses. After the execution of these IP Addresses, next 2 IP Addresses will be executed on these threads. Can anyone help how to do it?
Here, connection is a class which extends the thread and the work to be implemented is written in its run() method. Please help.
Connection newConnection =new Connection(0);
Connection newConnection1 =new Connection(1);
for(int i = 0; i < NE_list.getRowCount(); i=i+2)
{
if(NE_list.getValueAt(i, 0).toString().equals("true")) //Some condition here for the IP Address
{
newConnection.i=i;
newConnection1.i=i+1;
newConnection.runprogram();
newConnection1.runprogram();
}
}
class Connection extends Thread{
int i;
Connection(int val){
i=val;
}
void runprogram(){
start();
}
public void run(){
//SNMP and FTP Code here for IP Address in index i of NE_list
}
}

Executor Framework will be best suit for your solution. I have created one example here. You can increase the number of threads as per your requirement.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
class SomeRunnable implements Runnable {
int threadNo = -1 ;
List<String> list = new ArrayList<String>();
public SomeRunnable(List list, int threadNo ) {
this.list.addAll(list);
this.threadNo =threadNo;
}
#Override
public void run() {
for (String element : list) {
System.out.println("By Thread:" + threadNo+", Processed Element:" +element);
}
}
}
public class ExecutorDemo {
public static void main(String[] args) {
List<String> list = new ArrayList<String>();
for (int i = 0; i < 100; i++) {
list.add("Elem:"+i);
}
// Divide list
int divideIndex = list.size()/2;
//Create objects of Runnable
SomeRunnable obj1 = new SomeRunnable(list.subList(0, divideIndex),1);
SomeRunnable obj2 = new SomeRunnable(list.subList(divideIndex,list.size()),2);
//Create fixed Thread pool, here pool of 2 thread will created
ExecutorService pool = Executors.newFixedThreadPool(2);
pool.execute(obj1);
pool.execute(obj2);
pool.shutdown();
}
}

Herewith adding working example having 5 threads. Just put the test.txt in your CLASS_PATH of the application.
class MyRunnable implements Runnable {
List<List<String>> records;
MyRunnable(List<List<String>> records){
this.records = records;
}
public void run(){
for(List list : records){
System.out.println(Thread.currentThread().getName() + " : "+list.toString());
}
}}
Main Class -
public class FileProcessThreads {
public List<List<String>> process(String fileName) throws IOException {
List<List<String>> records = new ArrayList<>();
BufferedReader br = new BufferedReader(new FileReader(fileName));
String line = null;
while((line = br.readLine()) != null){
List<String> listValues = Arrays.asList(line.split(" "));
records.add(listValues);
}
System.out.println(records.size());
return records;
}
public static void main(String[] args) throws IOException {
FileProcessThreads fp = new FileProcessThreads();
List<List<String>> records = fp.process("test.txt");
ExecutorService es = Executors.newFixedThreadPool(5);
int recordsInEachThread = (int) (records.size()/5);
System.out.println(recordsInEachThread);
MyRunnable my1 = new MyRunnable(records.subList(0, recordsInEachThread));
MyRunnable my2 = new MyRunnable(records.subList(recordsInEachThread+1, recordsInEachThread*2));
MyRunnable my3 = new MyRunnable(records.subList(recordsInEachThread*2 + 1, recordsInEachThread*3));
MyRunnable my4 = new MyRunnable(records.subList(recordsInEachThread*3 + 1, recordsInEachThread*4));
MyRunnable my5 = new MyRunnable(records.subList(recordsInEachThread*4 + 1, records.size() - 1));
es.execute(my1);
es.execute(my2);
es.execute(my3);
es.execute(my4);
es.execute(my5);
es.shutdown();
}}