I need some input from you regarding a scenario for which I am doing some POC(Proof of Concept).
I am novice to multithreading in java and trying to some test. My requirement is I want to load millions of records using simple
java and then after doing some conversion with the data will insert in a database table.
For that I want to perform a simple test related to finishing of all task.
Currently I want to try that my main method finishes only after my executor services finishes. Below is the code which I have tried.
Can anyone please help me to know if thats the correct way to finish the main method after finishing the executor threads.
Your suggestion will be highly appreciated.
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class SimpleThreadPool {
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(5);
for (int i = 0; i < 10; i++) {
executor = Executors.newFixedThreadPool(5);
// Runnable worker = new WorkerThread("Thread executor :" + i);
executor.execute(new WorkerThread("Thread executor :" + i));
}
executor.shutdown();
while (!executor.isTerminated()) {
//System.out.println("Waiting");
}
System.out.println("Will start for Executor 1");
ExecutorService executor1 = Executors.newFixedThreadPool(5);
for (int i = 0; i < 10; i++) {
// Runnable worker = new WorkerThread("Thread executor1 :" + i);
executor1.execute(new WorkerThread("Thread executor1 :" + i));
}
executor1.shutdown();
while (!executor1.isTerminated()) {
//System.out.println("Waiting");
}
System.out.println("Finished all threads");
//
String s=null;
s.toUpperCase();
}
}
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class WorkerThread implements Runnable {
private String command;
public WorkerThread(String s){
this.command=s;
}
public void run() {
ExecutorService executor2 = Executors.newFixedThreadPool(5);
Future loadData=executor2.submit(new LoadData());
System.out.println(" Start. Command = "+command);
try {
List listOfData=(List) loadData.get();
for(int i=0;i<listOfData.size();i++){
//Thread.sleep(500);
//System.out.println("Printing the value from list:"+listOfData.get(i));
ConversionLogic conversion= new ConversionLogic();
conversion.doConversion(command);
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(" End."+command);
}
public String toString(){
return this.command;
}
}
class LoadData implements Callable{
public List call() throws Exception {
List<String> dataList= new ArrayList<String>();
for(int i=0;i<100;i++){
String data="Data_"+i;
//System.out.println("Data Added in List:"+data);
dataList.add(data);
}
Thread.sleep(10000);
return dataList;
}
}
public class ConversionLogic {
public void doConversion(String threadName){
try {
System.out.println("Converting Data for Thread starts:"+threadName);
Thread.sleep(5000);
System.out.println("Converting Data for Thread ends:"+threadName);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
SO this is what I have understood from the answers provided below:
package stackoverflow.test;
import java.util.List;
import java.util.concurrent.*;
class SimpleThreadPool {
public static void main(String[] args) throws InterruptedException, ExecutionException {
ExecutorService executor = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor = new ExecutorCompletionService<List>(executor);
ExecutorService executor2 = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor2 = new ExecutorCompletionService<List>(executor2);
//start loading data
int procCount = 0;
for (int i = 0; i < 10; i++) {
Future loadData = processor.submit(new LoadData("Org"));
procCount++;
}
//now all loading tasks have been submitted and are being executed
System.out.println("All LOADING tasks have been submitted and are being executed");
//new work queue using the same executor (or another one if you want more parallelism)
ExecutorCompletionService<Void> converter = new ExecutorCompletionService<Void>(executor);
while (procCount-- > 0) {
Future<List> listOfDataFuture = processor.take(); //blocks until a 'LoadData' finishes
System.out.println("A loading task just completed");
List listOfData = listOfDataFuture.get();
for (int i = 0; i < listOfData.size(); i++) {
ConversionLogic conversion = new ConversionLogic(procCount + "_" + i);
converter.submit(conversion);
}
}
System.out.println("All LOADING tasks have been COMPLETED for Org");
//now all conversion tasks have been submitted and are being executed
System.out.println("All CONVERSION task have been submitted and are being executed for Org:");
/* You don't need to wait for conversion tasks to complete:
* they will be completed nonetheless. Wait for them (with take())
* if you need the results.
* */
executor.shutdown();
try {
System.out.println("Waiting for finish");
executor.awaitTermination(1000, TimeUnit.SECONDS);
System.out.println("Stopped nicely");
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
System.out.println("Fund Data Loading Starts:");
//___________________________________________________________________//
// Some argument to get Fund Data
int procCount1 = 0;
for (int i = 0; i < 5; i++) {
Future loadData = processor2.submit(new LoadData("Fund"));
procCount1++;
}
//now all loading tasks have been submitted and are being executed
System.out.println("All LOADING tasks have been submitted and are being executed for Fund:");
//new work queue using the same executor (or another one if you want more parallelism)
ExecutorCompletionService<Void> converter1 = new ExecutorCompletionService<Void>(executor2);
while (procCount1-- > 0) {
Future<List> listOfDataFuture = processor2.take(); //blocks until a 'LoadData' finishes
System.out.println("A loading task just completed for Fund:");
List listOfData = listOfDataFuture.get();
for (int i = 0; i < listOfData.size(); i++) {
ConversionLogic conversion = new ConversionLogic(procCount + "_" + i);
converter1.submit(conversion);
}
}
System.out.println("All LOADING tasks have been COMPLETED for Org");
//now all conversion tasks have been submitted and are being executed
System.out.println("All CONVERSION task have been submitted and are being executed for Org:");
/* You don't need to wait for conversion tasks to complete:
* they will be completed nonetheless. Wait for them (with take())
* if you need the results.
* */
executor2.shutdown();
try {
System.out.println("Waiting for finish");
executor.awaitTermination(1000, TimeUnit.SECONDS);
System.out.println("Stopped nicely");
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
System.out.println("<<<<<<<<<< End>>>>>>>>");
System.exit(0);
}
}
package stackoverflow.test;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
class LoadData implements Callable {
String dataType;
public List call() throws Exception {
List<String> dataList = new ArrayList<String>();
for (int i = 0; i < 20; i++) {
String data = "Data_" + i;
System.out.println("Processing Data of Type :" + dataType + "Data is:"+data);
dataList.add(data);
}
Thread.sleep(2000);
return dataList;
}
LoadData(String type){
this.dataType=type;
}
}
package stackoverflow.test;
import java.util.concurrent.Callable;
class ConversionLogic implements Callable {
private String threadName;
public ConversionLogic(String threadName) {
this.threadName = threadName;
}
public Void call() throws Exception {
try {
System.out.println("Converting Data for Thread starts:" + threadName);
Thread.sleep(1000);
System.out.println("Converting Data for Thread ends:" + threadName);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
}
Updating the code for one set of requirement.Any suggestion to improve the performance is welcome.
package stackoverflow.tesst;
import java.sql.SQLException;
import java.util.List;
import java.util.concurrent.*;
import connection.java.JdbcConnection;
class SimpleThreadPool {
public static void main(String[] args) throws InterruptedException,
ExecutionException {
ExecutorService executor = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor = new ExecutorCompletionService<List>(
executor);
ExecutorService executor2 = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor2 = new ExecutorCompletionService<List>(
executor2);
System.out.println("Connecting to DB...");
try {
System.out.println("Connection is :"
+ JdbcConnection.getConnection());
} catch (ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (SQLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
// start loading data
int priceRange1 = 200;
int priceRange2 = priceRange1 + 200;
int procCount = 0;
for (int i = 0; i < 10; i++) {
String query = "select code,name,price from Product where price ";
if (i == 0) {
String finalQuery = query + " <= " + priceRange1;
Future loadData = processor.submit(new LoadData("Org",
finalQuery));
} else {
String finalQuery = query + " <= " + priceRange2
+ " and price > " + priceRange1;
Future loadData = processor.submit(new LoadData("Org",
finalQuery));
}
priceRange1 = priceRange2;
priceRange2 = priceRange2 + 200;
procCount++;
}
System.out.println("All LOADING tasks have been COMPLETED for Org");
ExecutorCompletionService<Void> converter = new ExecutorCompletionService<Void>(
executor);
while (procCount-- > 0) {
Future<List> listOfDataFuture = processor.take();
System.out.println("A loading task just completed");
List listOfData = listOfDataFuture.get();
for (int i = 0; i < listOfData.size(); i++) {
ConversionLogic conversion = new ConversionLogic(procCount
+ "_" + i, listOfData);
converter.submit(conversion);
}
}
System.out
.println("All CONVERSION task have been submitted and are being executed for Org:");
executor.shutdown();
try {
System.out.println("Waiting for finish");
executor.awaitTermination(1000, TimeUnit.SECONDS);
System.out.println("Stopped nicely for Org");
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
System.out.println("<<<<<<<<<< End>>>>>>>>");
System.exit(0);
}
}
package stackoverflow.tesst;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
----------------------------------------------
import connection.java.JdbcConnection;
class LoadData implements Callable {
String dataType;
Connection conn;
String query;
public List call() throws Exception {
List<Product> dataList = new ArrayList<Product>();
try {
conn=JdbcConnection.getConnection();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(this.query);
while(rs.next()){
System.out.println(rs.getString("code"));
System.out.println(rs.getString("name"));
System.out.println(rs.getInt("price"));
Product p= new Product(rs.getString("code"),rs.getString("name"),rs.getInt("price"));
dataList.add(p);
}
rs.close();//conn.close();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Thread.sleep(2000);
return dataList;
}
LoadData(String type,String query){
this.dataType=type;
this.query=query;
}
}
}
}
---------------------------
package stackoverflow.tesst;
import java.util.List;
import java.util.concurrent.Callable;
class ConversionLogic implements Callable {
private String threadName;
List<Product> productList;
public ConversionLogic(String threadName,List<Product> productList) {
this.threadName = threadName;
this.productList=productList;
}
public Void call() throws Exception {
try {
System.out.println("Converting Data for Thread starts:" + threadName);
Thread.sleep(1000);
int listSize=productList.size();
for(int i=0;i<listSize;i++){
//Do conversion for product let say
System.out.println("Print product in Conversion:"+productList.get(i).getPrice());
}
System.out.println("Converting Data for Thread ends:" + threadName);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
}
------------------------------------
package connection.java;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class JdbcConnection {
static Connection conn;
static String user;
static String pass;
static String dbURL;
public static Connection getConnection() throws ClassNotFoundException,
SQLException {
Class.forName("org.postgresql.Driver");
dbURL = "jdbc:postgresql://localhost:5433:postgres";
user = "postgres";
pass = "root";
Connection conn = DriverManager.getConnection(dbURL, user, pass);
Statement stmt = conn.createStatement();
System.out.println("Created DB Connection....");
return conn;
}
}
package stackoverflow.tesst;
import java.io.Serializable;
import java.math.BigDecimal;
public class Product implements Serializable {
/**
* Product Class using Annotation
*/
private static final long serialVersionUID = 1L;
private Integer id;
private String code;
private String name;
private int price;
Product(String code,String name,int price){
this.code=code;
this.name=name;
this.price=price;
}
public Integer getId() {
return id;
}
public void setId(Integer id) {
this.id = id;
}
public String getCode() {
return code;
}
public void setCode(String code) {
this.code = code;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getPrice() {
return price;
}
public void setPrice(int price) {
this.price = price;
}
#Override
public String toString() {
return "Product [id=" + id + ", code=" + code + ", name="
+ name + ", price=" + price + "]";
}
}
Here is the final version of my POC, which I am actually looking for. Only suggestion required is do I suppose to synchronize the insert query part?
package stackoverflow.tesst;
import java.sql.SQLException;
import java.util.List;
import java.util.concurrent.*;
import connection.java.JdbcConnection;
class SimpleThreadPool {
public static void main(String[] args) throws InterruptedException,
ExecutionException {
ExecutorService executor = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor = new ExecutorCompletionService<List>(
executor);
ExecutorService executor2 = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor2 = new ExecutorCompletionService<List>(
executor2);
/*System.out.println("Connecting to DB...");
try {
System.out.println("Connection is :"
+ JdbcConnection.getConnection());
} catch (ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (SQLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}*/
// start loading data
int priceRange1 = 200;
int priceRange2 = priceRange1 + 200;
int procCount = 0;
for (int i = 0; i < 10; i++) {
String query = "select code,name,price from Product where price ";
if (i == 0) {
String finalQuery = query + " <= " + priceRange1 + " order by price";
Future loadData = processor.submit(new LoadData("Org",
finalQuery));
} else {
String finalQuery = query + " <= " + priceRange2
+ " and price > " + priceRange1 + " order by price";
Future loadData = processor.submit(new LoadData("Org",
finalQuery));
}
priceRange1 = priceRange2;
priceRange2 = priceRange2 + 200;
procCount++;
}
System.out.println("All LOADING tasks have been COMPLETED for Org");
ExecutorCompletionService<Void> converter = new ExecutorCompletionService<Void>(
executor);
while (procCount-- > 0) {
Future<List> listOfDataFuture = processor.take();
System.out.println("A loading task just completed");
List listOfData = listOfDataFuture.get();
for (int i = 0; i < listOfData.size(); i++) {
ConversionLogic conversion = new ConversionLogic("<<Org>>"
+ "_" + i, listOfData);
converter.submit(conversion);
}
}
System.out
.println("All CONVERSION task have been submitted and are being executed for Org:");
executor.shutdown();
try {
System.out.println("Waiting for finish");
executor.awaitTermination(1000, TimeUnit.SECONDS);
System.out.println("Stopped nicely for Org");
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
System.out.println("Fund Data Loading Starts:");
// ___________________________________________________________________//
int fundRange1 = 200;
int fundRange2 = fundRange1 + 200;
int procCount1 = 0;
for (int i = 0; i < 10; i++) {
String query = "select code,name,price from Product where price ";
if (i == 0) {
String finalQuery = query + " <= " + fundRange1;
Future loadData = processor2.submit(new LoadData("Fund",
finalQuery));
} else {
String finalQuery = query + " <= " + fundRange2
+ " and price > " + fundRange1;
Future loadData = processor2.submit(new LoadData("Fund",
finalQuery));
}
fundRange1 = fundRange2;
fundRange2 = fundRange2 + 200;
procCount1++;
}
System.out.println("All LOADING tasks have been COMPLETED for Fund");
ExecutorCompletionService<Void> converter1 = new ExecutorCompletionService<Void>(
executor2);
while (procCount1-- > 0) {
Future<List> listOfDataFuture = processor2.take();
System.out.println("A loading task just completed");
List listOfData = listOfDataFuture.get();
for (int i = 0; i < listOfData.size(); i++) {
ConversionLogic conversion = new ConversionLogic("<<Fund>>"
+ "_" + i, listOfData);
converter1.submit(conversion);
}
}
System.out
.println("All CONVERSION task have been submitted and are being executed for Fund:");
executor2.shutdown();
try {
System.out.println("Waiting for finish");
executor.awaitTermination(1000, TimeUnit.SECONDS);
System.out.println("Stopped nicely for Fund");
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
System.out.println("<<<<<<<<<< End>>>>>>>>");
System.exit(0);
}
}
package stackoverflow.tesst;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import connection.java.JdbcConnection;
class LoadData implements Callable {
String dataType;
Connection conn;
String query;
public List call() throws Exception {
List<Product> dataList = new ArrayList<Product>();
try {
System.out.println("Connection establishing for Loading Org Data:");
conn = JdbcConnection.getConnection();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(this.query);
while (rs.next()) {
System.out.println(rs.getString("code"));
System.out.println(rs.getString("name"));
System.out.println(rs.getInt("price"));
Product p = new Product(rs.getString("code"),
rs.getString("name"), rs.getInt("price"));
dataList.add(p);
}
rs.close();
conn.close();
System.out.println("Connection Closed While loading for :"+dataType);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Thread.sleep(2000);
return dataList;
}
LoadData(String type, String query) {
this.dataType = type;
this.query = query;
}
}
package stackoverflow.tesst;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.concurrent.Callable;
import connection.java.JdbcConnection;
class ConversionLogic implements Callable {
private String threadName;
List<Product> productList;
Connection conn;
public ConversionLogic(String threadName, List<Product> productList) {
this.threadName = threadName;
this.productList = productList;
}
public Void call() throws Exception {
int listSize = productList.size();
try {
conn = JdbcConnection.getConnection();
System.out.println("Connection establishing for Converting Org Data:");
String insertTableSQL = "INSERT INTO item"
+ "(code, name, price) VALUES" + "(?,?,?)";
PreparedStatement preparedStatement = conn
.prepareStatement(insertTableSQL);
for (int i = 0; i < listSize; i++) {
preparedStatement.setString(1, productList.get(i)
.getCode());
preparedStatement.setString(2, productList.get(i)
.getName());
preparedStatement.setInt(3, productList.get(i)
.getPrice());
//Guess we suppose to synchronize the insert part in case
// In case mutiple threads trying to insert some records in a table and we might end up loosing data
//
preparedStatement.executeUpdate();
}
preparedStatement.close();
conn.close();
System.out.println("Connection Closed While Converting for Org :");
} catch (ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (SQLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
return null;
}
}
Instead of that while loop I would advise you to use the built-in await function like this:
executor.shutdown();
try {
System.out.println("Waiting for finish");
executor.awaitTermination(1000, TimeUnit.SECONDS);
System.out.println("Stopped nicely");
} catch (InterruptedException e) {
System.out.println("Could not stop in alloted time");
}
System.exit(0);
Place this code wherever you want to make sure that your executorService finished before you continue any further.
There are way to many thread pools in your code and, in general, spawning new threads inside a spawned thread is not a good idea: it can easily get out of control. In your case, you don't need the WorkerThread: you already have the thread pool provided by the Java framework (ExecutorService).
Since you need the result from a thread (LoadData) to be processed (ConversionLogic) I would use also an ExecutorCompletionService to help with gathering results from LoadData.
Following the refactored code. I've ditched the WorkerThread and used only a threadpool (altough you can use two if you want more parallelism), also the ConversionLogic now implements Callable so that it can be easily processed by the thread pool.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
class SimpleThreadPool {
public static void main(String[] args) throws InterruptedException, ExecutionException {
ExecutorService executor = Executors.newFixedThreadPool(10);
ExecutorCompletionService<List> processor = new ExecutorCompletionService<List>(executor);
//start loading data
int procCount = 0;
for (int i = 0; i < 10; i++) {
Future loadData = processor.submit(new LoadData());
procCount++;
}
//now all loading tasks have been submitted and are being executed
System.out.println("All LOADING tasks have been submitted and are being executed");
//new work queue using the same executor (or another one if you want more parallelism)
ExecutorCompletionService<Void> converter = new ExecutorCompletionService<Void>(executor);
while (procCount-- > 0) {
Future<List> listOfDataFuture = processor.take(); //blocks until a 'LoadData' finishes
System.out.println("A loading task just completed");
List listOfData = listOfDataFuture.get();
for (int i = 0; i < listOfData.size(); i++) {
ConversionLogic conversion = new ConversionLogic(procCount + "_" + i);
converter.submit(conversion);
}
}
System.out.println("All LOADING tasks have been COMPLETED");
//now all conversion tasks have been submitted and are being executed
System.out.println("All CONVERSION task have been submitted and are being executed");
/* You don't need to wait for conversion tasks to complete:
* they will be completed nonetheless. Wait for them (with take())
* if you need the results.
* */
executor.shutdown();
System.out.println(" End.");
}
}
class LoadData implements Callable {
public List call() throws Exception {
List<String> dataList = new ArrayList<String>();
for (int i = 0; i < 20; i++) {
String data = "Data_" + i;
System.out.println("Data Added in List:" + data);
dataList.add(data);
}
Thread.sleep(2000);
return dataList;
}
}
class ConversionLogic implements Callable {
private String threadName;
public ConversionLogic(String threadName) {
this.threadName = threadName;
}
#Override
public Void call() throws Exception {
try {
System.out.println("Converting Data for Thread starts:" + threadName);
Thread.sleep(1000);
System.out.println("Converting Data for Thread ends:" + threadName);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
}
executor.awaitTermination() method sounds convenient, but this is not an efficient one because what if all your executor services finishes before/after 1000 seconds (or any time you mention as an argument in awaitTermination() method).Either you are wasting the time or not letting your services finish their task.
Instead i'd suggest you use CountDownLatch as explained in below example.
public class app {
public static void main(String[] args) throws InterruptedException {
CountDownLatch latch = new CountDownLatch(3); // here assuming you know number of threads
ExecutorService executor = Executors.newFixedThreadPool(3);
for (int i = 0; i < 3; i++) {
executor.submit(new processor(latch));
}
latch.await();
System.out.println("main resume");
}
}
class processor extends Thread{
private CountDownLatch latch;
processor(CountDownLatch latch){
this.latch=latch;
}
public void run(){
System.out.println("in thread");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
latch.countDown();
}
}
Thus, you don't waste single ms of time.
Related
I want to write a program that can calculate the size of a directory and its subdirectories with multi-thread.
I write this:
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;
import java.util.concurrent.ThreadPoolExecutor;
public class Innerclass {
Semaphore semaphore = new Semaphore(1);
private List<String> availableConnections = new ArrayList();
public Innerclass(){
this.availableConnections.add("A");
}
public String acquireConnection() throws InterruptedException {
semaphore.acquire();
System.out.println("Acquiring connection " + Thread.currentThread().getName());
return availableConnections.remove(0);
}
static Long count = 0L;
static File file1 ;
public static void main(String[] args) {
System.out.println(countFilesInDirectory(new File("target directory address")));
}
public static Long countFilesInDirectory(File directory) {
Innerclass connection = new Innerclass();
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(20);
Task task1 = new Task();
//Long count = 0L;
for (File file : directory.listFiles()) {
if (file.isFile()) {
executor.execute(new Runnable() {
#Override
public void run() {
try {
String far = connection.acquireConnection();
count += printFileSizeNIO(String.valueOf(file));
connection.acquireConnection();
} catch (InterruptedException e) {
e.printStackTrace();
}
//System.out.println(printFileSizeNIO(String.valueOf(file)));
}
});
}
if (file.isDirectory()) {
count += countFilesInDirectory(file);
}
}
executor.shutdown();
//System.out.println(task1.getCount());
//return task1.getCount();
return count;
}
public static Long printFileSizeNIO(String fileName) {
Path path = Paths.get(fileName);
Long bytes = 0L;
try {
bytes = Files.size(path);
} catch (IOException e) {
e.printStackTrace();
}
return bytes;
}
}
This program gives results close to reality but can not calculate exactly. What do you think is the problem?
and with single thread this program work exactly!
Receives the current path and the number of fragments from the user and performs the calculation. Do you think there is another way besides the thread pool to write this program. Thank you, dear professors.
With executor service, your main thread wont wait for threads inside pool to finish.
See here:
Wait main thread until all the thread pools task complete of ExecutorService?
I have updated your code a little bit. It will give the same answer always.
public class TestSF
{
Semaphore semaphore = new Semaphore(1);
public void acquireConnection() throws InterruptedException
{
semaphore.acquire();
}
public void releaseConnection() throws InterruptedException
{
semaphore.release();
}
static AtomicLong count = new AtomicLong(0);
public static void main(String[] args)
{
long bytes = countFilesInDirectory(new File("C:\\Users\\ashish\\Desktop\\Loader"));
System.out.println(humanReadableByteCountBin(bytes));
}
public static Long countFilesInDirectory(File directory)
{
TestSF connection = new TestSF();
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(20);
for (File file : Objects.requireNonNull(directory.listFiles()))
{
executor.execute(() -> {
if (file.isFile())
{
try
{
connection.acquireConnection();
count.addAndGet(printFileSizeNIO(String.valueOf(file)));
connection.releaseConnection();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
if (file.isDirectory())
{
countFilesInDirectory(file);
}
});
}
executor.shutdown();
try {
if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
executor.shutdownNow();
}
} catch (InterruptedException ex) {
executor.shutdownNow();
Thread.currentThread().interrupt();
}
return count.get();
}
public static Long printFileSizeNIO(String fileName)
{
Path path = Paths.get(fileName);
long bytes = 0L;
try
{
bytes = Files.size(path);
}
catch (IOException e)
{
e.printStackTrace();
}
return bytes;
}
public static String humanReadableByteCountBin(long bytes)
{
long absB = bytes == Long.MIN_VALUE ? Long.MAX_VALUE : Math.abs(bytes);
if (absB < 1024) {
return bytes + " B";
}
long value = absB;
CharacterIterator ci = new StringCharacterIterator("KMGTPE");
for (int i = 40; i >= 0 && absB > 0xfffccccccccccccL >> i; i -= 10)
{
value >>= 10;
ci.next();
}
value *= Long.signum(bytes);
return String.format("%.1f %ciB", value / 1024.0, ci.current());
}
}
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I am pretty new to Object Programming and Java, so I am here to gather your advice and feedback. Basically I am trying to write a background service which performs different tasks at different intervals. I'm just not 100% sure of what I am doing is following the coding standards or is efficient.
Main / Start Class:
public class Start {
public static void main(String[] args) {
Service s = new Service();
s.Start();
}
}
Database Class:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class Database {
/* Database settings */
private final String HOSTNAME = "localhost";
private final String DATABASE = "java_database";
private final String USERNAME = "java_username";
private final String PASSWORD = "java_password";
/* Database connection */
public Connection getConnection() {
try {
return DriverManager.getConnection("jdbc:mysql://" + HOSTNAME + "/" + DATABASE + "?user=" + USERNAME + "&password=" + PASSWORD + "&useSSL=false&useUnicode=true&characterSetResults=utf8");
} catch (SQLException ex) {
ex.printStackTrace();
}
return null;
}
}
Service Class:
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class Service {
private int taskId;
private int taskType;
/* Start Service */
public void Start() {
try {
System.out.println("Starting Service...");
while(true) {
System.out.print("Checking for tasks... ");
getNextTask();
if (this.taskId > 0) {
System.out.println("Task ID " + this.taskId + " found.");
switch (this.taskType) {
case 1:
System.out.println("Task 1");
SampleTask s = new SampleTask();
s.Start();
s = null;
break;
default:
System.out.println("Error: Unknown Task");
}
setUsedTask();
} else {
System.out.println("No tasks to perform at this time.");
}
this.taskId = 0;
this.taskType = 0;
Thread.sleep(5000);
}
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
/* Gets the next available task from the database */
public void getNextTask() {
try {
Database db = new Database();
String query = "select taskId, taskType "
+ "from tasks "
+ "where (time_to_sec(timediff(now(), taskLastRun)) > taskFrequency or taskLastRun is null) and taskEnabled = 1 "
+ "limit 1";
Statement stmt = db.getConnection().createStatement();
ResultSet rset = stmt.executeQuery(query);
if (rset.next()) {
this.taskId = rset.getInt(1);
this.taskType = rset.getInt(2);
}
} catch (SQLException ex) {
ex.printStackTrace();
}
}
/* Set task as complete */
public void setUsedTask() {
try {
Database db = new Database();
String query = "update tasks "
+ "set taskLastRun = now() "
+ "where taskId = ? "
+ "limit 1";
PreparedStatement pstmt = db.getConnection().prepareStatement(query);
pstmt.setInt(1, this.taskId);
pstmt.executeUpdate();
} catch (SQLException ex) {
ex.printStackTrace();
}
}
}
Consider replacing your Thread.sleep() approach with a wait() and notify() approach as discussed here.
public class Service {
private int taskId;
private int taskType;
private final Object serviceMonitor;
/* Start Service */
public void Start() {
synchronized(serviceMonitor){
try {
System.out.println("Starting Service...");
while(true) {
System.out.print("Checking for tasks... ");
getNextTask();
if (this.taskId > 0) {
System.out.println("Task ID " + this.taskId + " found.");
switch (this.taskType) {
case 1:
System.out.println("Task 1");
SampleTask s = new SampleTask();
s.Start();
s = null;
break;
default:
System.out.println("Error: Unknown Task");
}
setUsedTask();
} else {
System.out.println("No tasks to perform at this time.");
}
this.taskId = 0;
this.taskType = 0;
serviceMonitor.wait();
}
}
}
catch (InterruptedException ex) {
ex.printStackTrace();
}
}
public void getNextTask() {
synchronized(serviceMonitor){
try {
Database db = new Database();
String query = "select taskId, taskType "
+ "from tasks "
+ "where (time_to_sec(timediff(now(), taskLastRun)) > taskFrequency or taskLastRun is null) and taskEnabled = 1 "
+ "limit 1";
Statement stmt = db.getConnection().createStatement();
ResultSet rset = stmt.executeQuery(query);
if (rset.next()) {
this.taskId = rset.getInt(1);
this.taskType = rset.getInt(2);
serviceMonitor.notifyAll();
}
}
} catch (SQLException ex) {
ex.printStackTrace();
}
}
public class Main {
public static void main(String[] args) {
Service service = new Service(Arrays.asList(new SampleTask(),new AnotherTask()));
service.execute();
}
}
class Service {
private List<Task> taskList;
public Service(List<Task> taskList) {
this.taskList = taskList;
}
public void addTask(Task task) {
taskList.add(task);
}
public void execute() {
for (Task task : taskList) {
new Timer().schedule(task, task.getDelay(), task.getPeriod());
}
}
public void clearTasks() {
taskList.clear();
}
}
abstract class Task extends TimerTask {
abstract long getDelay();
abstract long getPeriod();
}
class SampleTask extends Task {
public void run() {
System.out.println("Sample task executed");
}
long getDelay() {
return 1000;
}
long getPeriod() {
return 60000;
}
}
class AnotherTask extends Task {
public void run() {
System.out.println("Another task is executed");
}
long getDelay() {
return 1000;
}
long getPeriod() {
return 500000;
}
}
I am trying to insert close to 30 million rows from oracle into cassandra.
Here is little snippet of code,which I put together quickly.
I am getting java.sql.SQLException: Exhausted Resultset and in case I make my dbread smaller
(that is fetching just 500000 records) it only inserts close to 3500000 or lower,it is forcing me to run cassandra insertion in single thread which is taking close to 2.5 hrs for 30 millions)
Here is code :
import java.math.BigInteger;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import com.datastax.driver.core.BoundStatement;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
import com.pega.dbutils.DataBaseUtils;
import cassandraconnect.learncassandra.ConnectionToDB;
public class DumpMultiThreadedDumpInCassandra {
Connection connection = null;
Cluster cluster = null;
Session session = null;
public static void main(String[] args) throws SQLException {
// TODO Auto-generated method stub
DumpMultiThreadedDumpInCassandra dumpInCassandra = new DumpMultiThreadedDumpInCassandra();
dumpInCassandra.openConnectionWithCassadnra();
try {
dumpInCassandra.getDataFromDbInToMemoryAndInsertInCassandra();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally {
System.out.println("Closing the cassandra cluster");
dumpInCassandra.session.close();
dumpInCassandra.cluster.close();
dumpInCassandra.connection.close();
}
}
private void openConnectionWithCassadnra() {
cluster = Cluster.builder().addContactPoint(ipaddress).build();
session = cluster.connect("data");
System.out.println(session.getCluster().getClusterName());
}
private void getDataFromDbInToMemoryAndInsertInCassandra() throws SQLException, InterruptedException {`enter code here`
String insertSQL = "insert into table(pysubjectid,pyissue,pygroup,pyname,pxoutcometime,price,sequence) values(?,?,?,?,?,?,?)";
final com.datastax.driver.core.PreparedStatement preparedStatement = session.prepare(insertSQL);
int threads = 3;
connection = ConnectionToDB.openConnection();
String Query = "select SUBJECTID,ISSUE,GROUPISSUE,NAMEPROP,OUTCOMETIME,PRICE,factidint from oracletable where factidint>="+startrange +" "+"and"+" "+ "factidint<="+endstartrange;
//String Query = "select SUBJECTID,ISSUE,GROUPISSUE,NAMEPROP,OUTCOMETIME,PRICE,factidint from oracletable ";
System.out.println("Query is"+">>>"+Query);
java.sql.Statement statement = connection.createStatement();
final BlockingQueue<IHFactRecords> blockingQueue = new LinkedBlockingQueue<IHFactRecords>();
final ResultSet resultSet = statement.executeQuery(Query);
long countprocessed=0;
while (resultSet.next()) { blockingQueue.offer(new
IHFactRecords(resultSet.getString(1), resultSet.getString(2),
resultSet.getString(3), resultSet.getString(4),
Timestamp.valueOf(resultSet.getString(5)), resultSet.getLong(6),
resultSet.getLong(7)));
countprocessed++;
if(countprocessed%1000000==0){
System.out.println("million put in memory");
}
}
DataBaseUtils.close(resultSet);
DataBaseUtils.close(statement);
/*ExecutorService readThreadservice = Executors.newFixedThreadPool(1);
readThreadservice.submit(new Runnable() {
public void run() {
// TODO Auto-generated method stub
try {
if (resultSet!= null){
while (resultSet.next()) {
blockingQueue.offer(new IHFactRecords(resultSet.getString(1), resultSet.getString(2),
resultSet.getString(3), resultSet.getString(4),
Timestamp.valueOf(resultSet.getString(5)), resultSet.getLong(6), resultSet.getLong(7)));
}}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});*/
System.out.println("Sleeping for 5 seconds");
Thread.sleep(5000);
System.out
.println("size of blocking queue populated in separated thread is now simultaneously starting ingestion"
+ ">>>" + blockingQueue.size());
long starttime = System.currentTimeMillis();
ExecutorService executorService = Executors.newFixedThreadPool(threads);
final AtomicInteger totalcountinserted = new AtomicInteger(0);
System.out.println("starting threads for parallel ingestion");
for (int i = 0; i < threads; i++) {
executorService.submit(new Runnable() {
public void run() {
// TODO Auto-generated method stub
IHFactRecords ihrecord;
while ((ihrecord = blockingQueue.poll()) != null) {
BoundStatement boundStatement = preparedStatement.bind();
boundStatement.setString(0, ihrecord.getSubjectID());
boundStatement.setString(1, ihrecord.getIssue());
boundStatement.setString(2, ihrecord.getGroup());
boundStatement.setString(3, ihrecord.getPropsition());
boundStatement.setTimestamp(4, ihrecord.getTime());
boundStatement.setVarint(5, BigInteger.valueOf(ihrecord.getPrice()));
boundStatement.setVarint(6, BigInteger.valueOf(ihrecord.getSequence()));
session.execute(boundStatement);
int createdtillnow = totalcountinserted.incrementAndGet();
if (createdtillnow % 100000 == 0)
System.out.println("Total records inserted till now are" + ">>" + createdtillnow);
}
}
});
}
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.MINUTES);
//readThreadservice.shutdown();
//readThreadservice.awaitTermination(2, TimeUnit.MINUTES);
long endTime = System.currentTimeMillis();
System.out.println("time in seconds" + ">>>" + TimeUnit.MILLISECONDS.toSeconds(endTime - starttime));
System.out.println("time in minutes" + ">>>" + TimeUnit.MILLISECONDS.toMinutes(endTime - starttime));
System.out.println("time in hrs" + ">>>" + TimeUnit.MILLISECONDS.toHours(endTime - starttime));
;
DataBaseUtils.close(resultSet);
DataBaseUtils.close(statement);
}
}
Here I have two run methods which should synchronize each other.
Poller Class:
*/
public void run() {
int seqId = 0;
while(true) {
List<KpiMessage> list = null;
try{
if(!accumulator.isUsed){
try {
list = fullPoll(seqId);
if (!list.isEmpty()) {
seqId = list.get(0).getSequence();
accumulator.manageIngoing(list);
}
System.out.println("Updated");
wait();
} catch (Exception e1) {
e1.printStackTrace();
}
}
} catch (Exception e){
// TODO:
System.err.println(e.getMessage());
e.printStackTrace();
}
}
}
/**
* Method which defines polling of the database and also count the number of Queries
* #param lastSeq
* #return pojo col
* #throws Exception
*/
public List<KpiMessage> fullPoll(int lastSeq) throws Exception {
Statement st = dbConnection.createStatement();
System.out.println("Polling");
ResultSet rs = st.executeQuery("Select * from msg_new_to_bde where ACTION = 814 and
STATUS = 200 order by SEQ DESC");
List<KpiMessage> pojoCol = new ArrayList<KpiMessage>();
try {
while (rs.next()) {
KpiMessage filedClass = convertRecordsetToPojo(rs);
pojoCol.add(filedClass);
}
for (KpiMessage pojoClass : pojoCol) {
System.out.print(" " + pojoClass.getSequence());
System.out.print(" " + pojoClass.getTableName());
System.out.print(" " + pojoClass.getAction());
System.out.print(" " + pojoClass.getKeyInfo1());
System.out.print(" " + pojoClass.getKeyInfo2());
System.out.print(" "+ pojoClass.getStatus());
System.out.println(" " + pojoClass.getEntryTime());
}
} finally {
try {
st.close();
rs.close();
} catch (Exception e) {
System.err.println(e.getMessage());
e.printStackTrace();
}
}
Processing and Updating Class:
public void run() {
while(true){
try {
while(!accumulator.isUsed)
{
try {
System.out.println("Waiting for new outgoingmessages");
this.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Collection<KpiMessage> outgoingQueue = generate(accumulator.outgoingQueue);
accumulator.manageOutgoing(outgoingQueue, dbConnection);
} catch (Exception e) {
System.err.println(e.getMessage());
e.printStackTrace();
}
}
}
}
I have a logical error:
The poller is polling not only for new messsage but also reads the DB again and again from the first.
Also Updates again and again.
How to solve this synchronization problem.
Alternatively you could use a BlockingQueue to transfer the data between threads.
See BlockingQueue for details.
// The end of the list.
private static final Integer End = -1;
static class Producer implements Runnable {
final Queue<Integer> queue;
private int i = 0;
public Producer(Queue<Integer> queue) {
this.queue = queue;
}
#Override
public void run() {
try {
for (int i = 0; i < 1000; i++) {
queue.add(i++);
Thread.sleep(1);
}
// Finish the queue.
queue.add(End);
} catch (InterruptedException ex) {
// Just exit.
}
}
}
static class Consumer implements Runnable {
final Queue<Integer> queue;
private int i = 0;
public Consumer(Queue<Integer> queue) {
this.queue = queue;
}
#Override
public void run() {
boolean ended = false;
while (!ended) {
Integer i = queue.poll();
if ( i != null ) {
ended = i == End;
System.out.println(i);
}
}
}
}
public void test() throws InterruptedException {
Queue queue = new LinkedBlockingQueue();
Producer p = new Producer(queue);
Consumer c = new Consumer(queue);
Thread pt = new Thread(p);
Thread ct = new Thread(c);
// Start it all going.
pt.start();
ct.start();
// Close it down
pt.join();
ct.join();
}
You should synchronize or rather hold the lock or monitor for the object that you are calling wait() or notify() on.
Here is what will help you : wait() throwing IllegalArgumentException
synchronized(lockObject){
lockObject.wait(); //you should hold the lock to be able to call wait()
}
Earlier I posted a problem about implementing wait and notify, but I wasn't very clear, so here is a more specific question.
In the long code block below there is one wait and one notify. The notify is supposed to stop the wait and cause it to stop waiting. At the moment, i think the wait works, but the notify does not. Could someone explain why the notify doesn't notify the wait? Thanks!
Note: the rest of the code works i'm only interested in these two specific parts.
import com.fmr.ipgt.email.*;
import java.io.File;
import java.io.IOException;
import java.util.List;
import org.jdom.Document;
import org.jdom.Element;
import org.jdom.JDOMException;
import org.jdom.input.SAXBuilder;
import javax.mail.MessagingException;
class MyQuery {
synchronized void qQuery() throws Exception {
String query = ".z.k"; // The query that is used to query q; this can be changed here.
int version = 0;
c qConn = null;
qConn = new c(Main.host,Main.port); // Connect to the q database
while (Main.healthy) {
Object o = qConn.k(query); // Query q
version = c.t(o);
if(!(version==0)) {
System.out.println(version);
System.out.println("database healthy");
NewThread.suspendFlag = false;
notify();
break; // End the process if the database responds
}
}
System.out.println("reaches loop end");
}
}
class MyThread implements Runnable {
MyQuery myResource;
MyThread(String name, MyQuery so) {
myResource = so;
new Thread(this, name).start();
}
public void run() {
try {
myResource.qQuery(); // Begin a method to query q.
} catch (Exception e) {
e.printStackTrace();
}
}
}
class NewThread implements Runnable {
String name; // name of thread
Thread t;
static boolean suspendFlag;
private int minutes;
NewThread(int minutes) {
this.minutes = minutes;
System.out.println("reaches constructor");
t = new Thread(this);
suspendFlag = true;
t.start(); // Start the thread
}
// This is the entry point for thread.
public void run() {
try {
synchronized(this) {
while(suspendFlag) {
System.out.println("reaches wait");
wait(minutes*60000);
System.out.println("reaches end");
if(suspendFlag) {
Main.setHealth(false);
Main.sendMessages(); // The database has not responded for the given time. Report that it is unhealthy.
}
break;
}
}
} catch (InterruptedException e) {
System.out.println(name + " interrupted.");
} catch (MessagingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public class Main {
private static String[] recipients;
private static String subject = "Database Failure";
private static String message = "The database has failed or is in a hung state";
private static String from;
static String host;
static int port;
private static String emails;
private static int minutes;
static boolean healthy = true;
public static void main(String args[]) throws Exception {
// Import information from the configuration file
SAXBuilder builder = new SAXBuilder();
File xmlFile = new File("/export/home/rhadm/file.xml"); // Note: The directory for the configuration file may need to be changed
try {
Document document = (Document) builder.build(xmlFile);
Element rootNode = document.getRootElement();
List list = rootNode.getChildren("parameters");
Element node = (Element) list.get(0);
host = node.getChildText("host");
port = Integer.parseInt(node.getChildText("port"));
emails = node.getChildText("emails");
String delims = "[ ]+";
recipients = emails.split(delims); // parse email list
minutes = Integer.parseInt(node.getChildText("time"));
from = node.getChildText("from");
} catch (IOException io) {
System.out.println(io.getMessage());
} catch (JDOMException jdomex) {
System.out.println(jdomex.getMessage());
}
MyQuery unhealthy = new MyQuery();
NewThread ob1 = new NewThread(minutes);
new MyThread("MyThread", unhealthy); // Create new Thread
}
public static void setHealth(boolean health){
System.out.println("database unhealthy");
healthy = health;
}
public static void sendMessages() throws MessagingException {
System.out.println("sending emails");
FCAPMailSender.postMail(recipients,subject,message,from);
}
}
You are synchronizing on different objects. The notify will only effect objects synchronized-waiting on the same object & instance.
The waiting thread is synchronized & waiting on a NewThread while the notifying thread is doing so on a MyQuery instance
Have a shared object.
private final Object LOCK = new Object();
synchronized(LOCK){
LOCK.wait();
}
synchronized(LOCK){
LOCK.notify();
}