I am not getting how to solve this exception.
I am passing Resultset object to each thread.and doing some logic
My sample program is:
public class dataimport
{
public dataimport()
{
connect(); /*connect to database
}
private void connect()
{
try
{
/* connected data base
str="SELECT * FROM tablename where rownum<=5";
rs1 = statement.executeQuery(str);
while (rs1.next())
{
Runnable r = new MyThreadClass(rs1);
System.out.println(new Thread().getName());
new Thread(r).start();
}
}
Thread class
class MyThreadClass implements Runnable
{
public MyThreadClass(ResultSet rs1)
{
rs2=rs1;
}
public void run()
{
int i=1;
try
{
while (rs2.next())
{
date=rs2.getString("mydate");
System.out.println("mydate="+date);
}
}
catch (Exception ex)
{
ex.printStackTrace();
}
}
}
If i add while (rs2.next()) in thread class,there will be no exception,but mydate it will
print only two insight of few records.
please can any one help me on this.
thank you in advance
No way. You are getting concurrency errors.
You get N results -> start N threads. Inside each thread, you loop again over the resultset. It makes little sense, and worse, it can happen that one thread checks that rs.next is true, enter to iterate, get suspended by another thread that finish the result set data and, when resuming, tries to read data from a resultset with no data.
The simplest (without changing the mechanics) solution would be putting the threads main loop in a synchronized loop, but that will cause a massive deadlock so you will end with the same performance as no threads.
As for you code, I think that what you really want is, in the principal loop, extract all the results into a data structure, and pass that data structure to each thread so it can deal directly with it (and no passing a shared result set object which causes such problems).
Related
I have a piece of legacy code which is basically like:
// instance variable
List<Future<Object>> futureList;
methodA() {
List list = getListOfMessages();
for (Object o : list) {
methodB(o);
}
}
void methodB(Object o) {
// after multi-threading, this statement takes ~2 mins
someDAO.update(o.value);
// some other tasks
}
This works fine, except that I have about million records that are retrieved into list via getListOfMessages(). So I was asked to multithread it and I changed it to something like...
methodA() {
List list = getListOfMessages();
// created executorservice here
for(Object o : list) {
Future future = executorService.submit(methodB(o));
futureList.add(future);
}
// call another method to see the status of each ask
checkFutureStatus(futureList);
}
void checkFutureStatus(List<Future<Object>> list) {
for(Future<Object> future : list) {
try {
future.get(1000, TimeUnit.Milliseconds);
} catch (InterrupException | ExecutionExecption e) {
} catch (TimeoutException e) {
}
}
}
So basically, for each list item, I pass it to methodB but have it handled by separate threads. Once all threads have been submitted, I check the status of the threads but every thread throws a TimeoutException. On debugging, I see that the threads take too long for DB updates...like 1-2 min.
Just to be sure that the threads are not competing with each other, I had the getListOfMessages() return just one message. And even that is taking 1-2 min, If I just revert everything and go non threaded approach, the DB update takes 1ms. I can't really figure out why the multi-thread implementation is causing the db update to take so long.
I'm using Postgres 10 and the db update is via jdbctemplate.
Thank you in advance.
Edit:
Added method to explain how I'm checking the status of each thread.
I am setting up a simulator (for testing) of a server (Radius) which sends queries to another server (LDAP) using threads.
The queries need to be executed on a x per second basis.
I am using a scheduled thread pool executor with callable for this purpose so that I can create callables and submit them to the thread pool for execution.
Each thread should open its own connection and use it to query.
The thing is that I want the connection to be re-used by the same thread every time it is used.
To clarify:
If I have lets say a thread pool of 20 I want 20 connections to be created and used. (so I can send lets say 10.000 queries which will be processed in turn by the 20 threads/connections).
Now the (LDAP) server information to connect to is sent to the constructor of the callable and the callable sets up the connection for execution. Thereafter I retrieve the result using the future system of callable.
The problem with this is each time I create a callable the connection is being opened (and later closed of course).
I am looking for the best practice to keep the connections alive and them being re-used for each thread.
I have thought of some ways to implement this but they dont seem very efficient:
Use a connection pool inside my threadpool to retrieve a free connection when needed (Creates deadlock and other thread safety issues)
Use a static (or so) array with connections and using the thread number to retrieve its connection (Not foul proof either, see link)
What is the most efficient way of implementing this? <- old question, see edit for new question.
EDIT:
I was thinking because I cannot safely get a thread number, but the threadId is always unique, I can just use a
map<String/threadId, connection>
And pass the whole map (reference) to the callable. This way I can use something like: (pseudo code)
Connection con = map.get(this.getThreadId());
If (con == null){
con = new Connection(...);
map.put(this.getThreadId(), con)
}
It would also be possible to make the map static and just access it statically. This way I don't have to pass the map to the Callable.
This would be at least safe and doesnt force me to restructure my code.
New question:
What would be closer aligned with best practices; The above solution or Zim-Zam's solution?
And if the above is best, would it be better to go for the static solution or not?
I would implement this using a BlockingQueue that is shared between Callables, with the ScheduledThreadPoolExecutor putting x queries into the BlockingQueue every second
public class Worker implements Runnable {
private final BlockingQueue<Query> inbox;
private final BlockingQueue<Result> outbox;
public Worker(BlockingQueue<Query> inbox, BlockingQueue<Result> outbox) {
// create LDAP connection
this.inbox = inbox;
this.outbox = outbox;
}
public void run() {
try {
while(true) {
// waits for a Query to be available
Query query = inbox.take();
// execute query
outbox.add(new Result(/* result */));
}
} catch(InterruptedException e) {
// log and restart? close LDAP connection and return?
}
}
}
public class Master {
private final int x; // number of queries per second
private final BlockingQueue<Query> outbox = new ArrayBlockingQueue<>(4 * x);
private final BlockingQueue<Result> inbox = new ArrayBlockingQueue<>(4 * x);
private final ScheduledThreadPoolExecutor executor;
private final List<Future<?>> workers = new ArrayList<>(20);
private final Future<?> receiver;
public Master() {
// initialize executor
for(int i = 0; i < 20; i++) {
Worker worker = new Worker(inbox, outbox);
workers.add(executor.submit(worker));
}
receiver = executor.submit(new Runnable() {
public void run() {
while(!Thread.interrupted()) {
try {
Result result = inbox.take();
// process result
} catch(InterruptedException e) {
return;
}
}
}
}
}
executor.scheduleWithFixedDelay(new Runnable() {
public void run() {
// add x queries to the queue
}
}, 0, 1, TimeUnit.SECONDS);
}
Use BlockingQueue#add to add new Queries to outbox, if this throws an exception then your queue is full and you'll need to reduce the rate of query creation and/or create more workers. To break out of a worker's infinite loop call cancel(true) on its Future, this will throw an InterruptedException inside of the Worker.
I've got a problem I can't really figure out. I have my main thread, and in it I want to
Send an email with an attachment
Delete the files that were attached
in that order. My problem is that I am using an email helper that I don't have control over, and it generates another thread that does the sending. Because of this, my files are being deleted before they are done being attached, and I am getting a FNF error in the mailer. I am looking to find a way to make my main thread wait until the files are done being attached. I don't know how long that will take. I don't have control over the other thread creation, so I can't use join(). Is there something I can use with Transport maybe, or a way to wait for all threads made in a method/class to stop?
The layout of my program is
//do stuff
MailHelper.sendMail(...); //thread is created somewhere in this method
deleteFiles(); //this happens before sendMail is finished!
I need to use Java 6. Worst-case I can have my main thread sleep for a few seconds, but that's unideal. Any help is appreciated
This is an interesting question! Basically you want to wait for all child threads to complete, but have no control over them.
Here is a demonstration of the technique using ThreadGroup:
Assuming you have a MailHelper class like this:
public class MailHelper {
public void sendMail(){
Thread t = new Thread(new Runnable() {
#Override
public void run() {
System.out.println("MailHelper: Sending mail for 6s");
for(int i = 0; i < 6; i++){
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(".");
}
System.out.println("MailHelper: Sent mail!");
}
});
t.start();
}
}
then our Main class demonstrates how use it:
public class Main {
public static void main(String[] args) throws InterruptedException {
final MailHelper mh = new MailHelper();
ThreadGroup mailThreadGroup = new ThreadGroup("mailGroup");
Thread callSendmailThread = new Thread(mailThreadGroup, new Runnable() {
#Override
public void run() {
System.out.println("Calling sendMail().");
mh.sendMail();
System.out.println("sendMail() returned.");
}
});
callSendmailThread.start();
callSendmailThread.join();
System.out.println("callSendmailThread joined. Waiting for rest of ThreadGroup to finish.");
// We cannot rely on ThreadGroup.activeCount() to give us an accurate number, and it could be zero!
Thread[] mailThreads = new Thread[mailThreadGroup.activeCount() + 1];
//Therefore retry enumerate until our array was large enough to hold all
while ( mailThreadGroup.enumerate( mailThreads, true ) == mailThreads.length ) {
mailThreads = new Thread[ mailThreads.length * 2 ];
}
for(Thread t : mailThreads){
if(t != null && t.isAlive()){
System.out.println("Joining thread " + t.getName());
t.join();
System.out.println("Thread " + t.getName() + " joined.");
}
}
mailThreadGroup.destroy();
System.out.println("Done!");
}
}
The output:
Calling sendMail().
sendMail() returned.
callSendmailThread joined. Waiting for rest of ThreadGroup to finish.
Joining thread Thread-1
MailHelper: Sending mail for 6s
.
.
.
.
.
.
MailHelper: Sent mail!
Thread Thread-1 joined.
Done!
Note that you must ensure that the Thread-1 is actually started by the time you enumerate the ThreadGroup, thus joining the callSendMailThread is absolutely necessary. Otherwise you'd get a race condition.
Also note that the quirky behaviour of ThreadGroup.enumerate() must be accounted for, by retrying to enumerate all the items several times.
The easy way to solve the issue logically is to track if the mail is send sucessfully.
It can be done by any of the below
1) Set a global variable with some value after mail is send from other thread and reset it value once the attachement is deleted.
2) Instead of variable you can also try creating a file.
Thank you,
Mukeshkoshym
I have a Java game that uses networking, and I have a client (using a Socket) fetching objects from an ObjectInputStream, running in its own thread.
From Client.java:
Object input = null;
while(true) {
input = in.readObject();
if(input != null) {
listener.gotObject(input);
}
}
This works pretty well. The object is gotten and is passed to the listener, which is a class linked to a my main GameApp class.
From the listener (NetControl.java):
public void gotObject(Object o) {
System.out.println(o);
app.gotObject(o);
}
"app" is the instance that handles all new objects received and deals with them.
From the app (GameApp.java) (edit: the non-abstract CardGameApp.java gives greater context):
public void gotObject(Object o) {
// select instance:
if(o instanceof GameList) {
GameList gameList = (GameList) o;
System.out.println("gamelist: " + gameList);
this.lobbyControl.gotGameList(gameList);
}
}
I've run this code in the debugger, one step at a time, and it works perfectly. When I run it normally though, I get a null pointer (output is as follows:)
Game ID: 0. Name: game1. Players: 1 / 1. // the object, as it is printed in Client.java
gamelist: Game ID: 0. Name: game1. Players: 1 / 1. // the object, as it is printed again in GameApp.java
Exception in thread "Thread-1" java.lang.NullPointerException
at com.lgposse.game.app.GameApp.gotObject(GameApp.java:61)
at com.lgposse.game.net.NetControl.gotObject(NetControl.java:47)
at com.lgposse.net.client.Client.run(Client.java:49)
Now, I see the object being printed twice, so I know it has been received... but I get a null pointer.
I added a sleep function in the middle of the function:
else if(o instanceof GameList) {
GameList gameList = (GameList) o;
System.out.println("gamelist: " + gameList);
try {
Thread.sleep(1000); // sleep 100 still gave null pointer
} catch (InterruptedException e) {}
this.lobbyControl.gotGameList(gameList);
}
And setting it to sleep for a while, it all finally worked.
Any idea why I need to sleep the thread like this? Is there something I should do differently? I'm not sure why I was able to print the object while it was still considered null.
Edit: added some more context.
It looks like lobbyControl is null, not gameList. If gameList were null, the top of the stack would be the gotGameList() method, not gotObject().
If sleeping helps the problem, then you must be manipulating the lobbyControl member without proper concurrency safeguards. An ObjectInputStream won't return an object until it's been fully read from the stream, so your problem has nothing to do with not having completely read the object.
Update: I can't follow all the code, but it appears that a reference to the object being constructed is leaked to a thread (the client in the NetControl), which is started before the constructor completes. If that is the case, that's very, very bad. You should never allow a partially constructed object to become visible to another thread.
Well, I'll start off by saying that the code snippets posted seem to help illustrate the issue, but i don't think the full picture is painted. I'd ask for a bit more code, to help get a full context.
That being said, I'd offer the following guidance:
Don't lean on java's built in object serialization. It's nice and
easy to use, but can be very unstable and error prone at runtime.
I'd suggest a custom object serialization and deserialization
scheme.
Depending on the scope of the game you're making, NIO may be a
netter choice. If you stick with regular IO, then make sure you
have a rock solid Thread Manager in place to properly handle the
threads dealing with the socket IO.
..without more code, that's the most I can offer.
Just to improve my comment...When i need to wait for one or more threads to finish, i like to use java.util.concurrent.CountDownLatch. Its very simple:
//game class
public class DummyGame
{
CountDownLatch signal;
public DummyGame( CountDownLatch signal)
{
this.signal = signal;
}
public void run()
{
doLogic();
signal.countDown();
}
}
//game controller class
public void run()
{
while (! gameOver)
{
CountDownLatch signal = new CountDownLatch(1); //wait one thread to finish
new thread(newGame(signal)).start();
//wait for game run() to finish
signal.await();
updateInterface();
}
}
That's just an idea, hope it helps.
The Following class DoStuff starts a thread and syncs to protect the listener object from being accessed when null.
Now when accessing the DoStuff class function setOnProgressListener() externally I'm having issues because the call is getting held for a long time before it exits the function call. I'm not sure why this happens? I seems as if the synchronization has queued up a lot of calls? Any input on this would help!
I'm essentially passing null to the listener because I no longer wish to get updated for this status. I do this as part of my process to kill the DoStuff Thread.
Thanks!
public class DoStuff extends Runnable
{
Object MUTEX = new Object();
private OnProgressListener mOnProgressListener = null;
public DoStuff()
{
new Thread(this).start();
}
public void setOnProgressListener( OnProgressListener onProgressListener )
{
synchronized (MUTEX)
{
mOnProgressListener = onProgressListener;
}
}
private void reportStatus( int statusId )
{
synchronized (MUTEX)
{
if (null != mOnStatusListener)
{
mOnStatusListener.setStatusMessage(new OnStatusEvent(this, statusId));
}
}
}
// this is the run of a thread
public void run()
{
int status = 0;
do
{
// do some work and report the current work status
status = doWork();
reportStatus( status );
} while(true);
}
}
You should use wait/notify. here is sample;
public class DoStuff {
Object MUTEX = new Object();
String data = null;
public void setData(String data) {
synchronized (MUTEX) {
this.data = data;
System.out.println(Thread.currentThread());
MUTEX.notifyAll();
}
}
public void run() {
do {
synchronized (MUTEX) {
if (null == data) {
return;
} else {
System.out.println(data);
}
try {
MUTEX.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
} while (true);
}
}
The trouble with this code is that your while() loop is constantly trying to grab the monitor for MUTEX immediately after releasing it or even yield()-ing to help the scheduler put another thread in. So there's a very good chance that anyone else trying to obtain that monitor will be starved, because your while() loop will consume most of your CPU time and even when other threads could run, they might not get the monitor they're waiting for.
Ideally a wait()/notify() pair should be used or failing that, you should at least call a Thread.yield() in your while loop, outside the synchronized block. (But I this second "solution" really isn't a very good one, you should consider using the first one instead.)
UPDATE: I read the code again and I think I believe to see what you wanted to achieve: printing the value of data every time you set a new value. If that's true, you should definitely go for the wait/notify solution, although if you want to absolutely guarantee that every single value is printed, you need to do even more work, possibly using a queue.
I'm a little confused about your code, can you provide the full listing?
First, where does DoStuff start a thread? Why are you quitting if your data is still null? (you might actually be out of the thread before setData even executes).
But the main thing here is that you're doing essentially a busy-waiting loop, in which you synchronize on the mutex. This is pretty wasteful and will generally block cores of your CPU.
Depending on what you are trying to do, you might want to use a wait-notify scheme, in which the thread goes to sleep until something happens.
Thanks all for your help. I was able to determine why the indefinite lock. Something important and obvious is that once I run the reportStatus() function call it will hold the lock MUTEX until it is completely done executing the callback. My fault was that at the registered callback I was calling setOnProgressListener(null) by mistake. Yes, I admit didn't post enough code, and most likely all of you would have catched the bug... So calling setOnProgressListener(null) would wait until the MUTEX object has been released, and the reportStatus() was held waiting to call setOnProgressListener(null), therefore I was in a deadlock!
Again the main point I learned is to remember that triggering a callback message will hold until the registered callback function is done processing it's call.
Thanks all!