I want to write a JUnitTest, which ensures that my Karaf Server is started fine and all (needed) Bundles are installed and active.
For this I have a Test, calling a helper method "assertBundleState" which ensures, that the given Bundle is in the given State. The test looks like following:
#Test (timeout=30000L)
public void testBundlesStarted() throws Exception {
assertBundleState("bundle.Name", BundleLifecycleState.Active);
... <other bundles in similar way>
}
This worked fine in the past. The timeout was never reached.
Unfortunately I now have to load a bundle, which need a bit longer for startup. So the Bundles are not yet started, when the test is executed. And so the test fails.
I have tried it with a sleep in a BeforeClass method, but I am not really happy with this solution. Because I can't ensure that the given sleep time is enough on every machine at every time. So I am looking for some event-based solution to interact with Karaf.
#BeforeClass
public static void init() throws Exception {
Thread.sleep(120000L);
}
Does anybody have an idea how I can solve it in a better way?
Thanks
It sounds like assertBundleState can successfully determine whether the Karaf server has started and 'is ready'. But the issue is that you don't know how long to wait before checking this; if you don't wait long enough you risk a false negative, if you wait too long then your build elapsed time is falsely prolonged.
I presume you cannot register some sort of listener or callback hook which the Karaf server will invoke when it 'is ready'. Indeed, even if that was possible you would still have to cater for the failure case i.e. where the Karaf server does not start (and hence could never invoke a listener).
So, I think you just have to wait for the Karaf server to become available and hide the ugliness of Thread.sleep calls and the potential lost time in waiting too long. You could use Awaitility to tick those boxes. For example:
#Test
public void aTest() {
// run your test
// this will wait until Karaf is available for at most 10 seconds
// and will check every 100ms so if Karaf becomes available earlier
// than 10 seconds then the test will complete almost as soon as
// it becomes available but if Karaf does not become available
// within 10 seconds then the test will fail
await().atMost(10, SECONDS).until(karafIsAvailable());
// assert
// ...
}
private Callable<Boolean> karafIsAvailable() {
return new Callable<Boolean>() {
public Boolean call() throws Exception {
// return true if your condition has been met
// e.g. assertBundleState
return ...;
}
};
}
Related
Since I upgraded to Ignite 2.8.1 I'm frequently seeing timeouts on Ignite compute for Callables. This hasn't been the case with v2.7.6. I even never used the timeout option before because I never saw any delay in compute.
Since it's now often stuck in the compute().call() method I added withTimeout(60000) but still had to add a loop to try this several times before the function is actually called on any of the client nodes. This doesn't happen all times, but sometimes.
When it happens I see in the Ignite log messages like this:
[01:14:27,237][WARNING][grid-timeout-worker-#119][GridTaskWorker] Task has timed out: GridTaskSessionImpl [taskName=de.my.comp.net.ignite.CreateNewMedia, dep=LocalDeployment [super=GridDeployment [ts=1609753866994, depMode=SHARED, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader#5c29bfd, clsLdrId=06faeccc671-65368413-cf29-4a85-9f51-9b124c6a3351, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=de.my.comp.net.ignite.CreateNewMedia, sesId=7bccb2ec671-65368413-cf29-4a85-9f51-9b124c6a3351, startTime=1610586807229, endTime=1610586867229, taskNodeId=65368413-cf29-4a85-9f51-9b124c6a3351, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader#5c29bfd, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=65368413-cf29-4a85-9f51-9b124c6a3351, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=154761191]], execName=null]
This is now my code (shortened to be more readable):
while(something)
{
...
...
final long TASK_TIMEOUT_CREATENEWMEDIA = 60 * 1000;
ClusterGroup servers = IgniteHelper.getComputeNodes(ignite);
UUID thisMasterID = IgniteHelper.getLocalNodeID(ignite);
IgniteSemaphore semaphore = ignite.semaphore("masterFindNodeSempahore", 1, true, true);
try
{
semaphore.acquire();
CreateNewMedia cnm = new CreateNewMedia(currentConfig, thisMasterID, currentJobID, msResubmit);
CreateNewMediaResult result = ignite.compute(servers).withTimeout(TASK_TIMEOUT_CREATENEWMEDIA).call(cnm);
...
...
}
catch (IgniteException e)
{
if (e instanceof ComputeTaskTimeoutException)
{
// log task timeout, try again if not max tries reached
...
...
}
else
{
// if some other issue, we better leave
throw new Exception(e);
}
}
finally
{
semaphore.release();
}
}
The goal of this code is to find a node that is free to perform a specific action. The function returns if the node where it was executed is available. If it returns negative result, I remove this node ID from the list and try the remaining servers (this logic is not in the above code for simplicity).
It seems that setting publicThreadPoolSize to 512 reduced chances for the timeout to happen, but it still sometimes occur. If it occurs, in most cases it eventually calls the function after a few minutes, i.e. several tries to compute().call() in loop.
Is this known in Ignite 2.8.1? This is running on a server farm with 25 Ignite compute clients.
Workload on the client nodes is high but definitely not overloaded. Actually the nodes are mostly running an external cmdline executable while the Java portion is only waiting for its console output.
Any help appreciated.
Additional note: I've seen the timeout with v2.8.1 on other Callable's as well, so I don't think it's particularly related to the implementation of CreateNewMedia or of how it's been called in my sample code above.
I'm trying to implement excel export for some amount of data. After 5 minutes I receive a 504 Gateway timeout. In the backend the process continues with its work.
For the whole service to finish, I need approximately 15 minutes. Is there anything I can do to prevent this? I dont have access to the servers in production.
The app is Spring boot with Oracle database. I'm using POI for this export.
One common way to handle these kinds of problems is to have the first request start the process in the background, and when the file has been generated, download the results from another place. The first request finishes immediately, and the user can then check another view to see if the file has been generated, and download the results.
You can export the data in smaller chunks. Run a test with say 10K records, make a note of the id of the last record and repeat the export starting at the next record. If 10K finishes quickly, then try 50K. If you have a timer that might come in handy. Good luck.
I had the same situation where the timeout of the network calls wasn't in our hand, so I guess you have something where it is 5 mins to receive the 1st byte and then the timeout is gone.
My solution was, let's assume you have a controller and a query layer to talk to the database. In this case, you make your process in the Async way. The call to this controller should just trigger that async execution and return the success status immediately, without waiting. Here execution will happen in the background. Futures can be used here as they are async and you can also handle the result once completed by using callback methods of Future.
You can implement using Future and callback methods in java8 like below:
Futures.addCallback(
exportData,
new FutureCallback<String>() {
public void onSuccess(String message) {
System.out.println(message);
}
public void onFailure(Throwable thrown) {
thrown.getCause();
}
},
service)
and in Scala like:
val result = Future {
exportData(data)
}
result.onComplete {
case Success(message) => println(s"Got the callback result:
$message")
case Failure(e) => e.printStackTrace
}
How does async JMS work? I've below sample code:
public class JmsAdapter implements MessageListener, ExceptionListener
{
private ConnectionFactory connFactory = null;
private Connection conn = null;
private Session session = null;
public void receiveMessages()
{
try
{
this.session = this.conn.createSession(true, Session.SESSION_TRANSACTED);
this.conn.setExceptionListener(this);
Destination destination = this.session.createQueue("SOME_QUEUE_NAME");
this.consumer = this.session.createConsumer(destination);
this.consumer.setMessageListener(this);
this.conn.start();
}
catch (JMSException e)
{
//Handle JMS Exceptions Here
}
}
#Override
public void onMessage(Message message)
{
try
{
//Do Message Processing Here
//Message sucessfully processed... Go ahead and commit the transaction.
this.session.commit();
}
catch(SomeApplicationException e)
{
//Message processing failed.
//Do whatever you need to do here for the exception.
//NOTE: You may need to check the redelivery count of this message first
//and just commit it after it fails a predefined number of times (Make sure you
//store it somewhere if you don't want to lose it). This way you're process isn't
//handling the same failed message over and over again.
this.session.rollback()
}
}
}
But I'm new to Java & JMS. I'll probably consume messages in onMessage method. But I don't know how does it work exactly.
Do I need to add main method in JmsAdapter class? After adding main method, do I need to create a jar & then run the jar as "java -jar abc.jar"?
Any help is much appreciated.
UPDATE: What I want to know is that if I add main method, should I simply call receiveMessages() in main? And then after running, will the listener keep on running? And if there are messages, will it retrieve automatically in onMessage method?
Also, if the listener is continuously listening, doesn't it take CPU??? In case of threads, when we create a thread & put it in sleep, the CPU utilization is zero, how doe it work in case of listener?
Note: I've only Tomcat server & I'll not be using any jms server. I'm not sure if listener needs any specific jms server such as JBoss? But in any case, please assume that I'll not be having anything except tomcat.
Thanks!
You need to learn to walk before you start trying to run.
Read / do a tutorial on Java programming. This should explain (among other things) how to compile and run a Java program from the command line.
Read / do a tutorial on JMS.
Read the Oracle material on how to create an executable JAR file.
Figure out what it is you are trying to do ... and design your application.
Looking at what you've shown and told us:
You could add a main method to that class, but to make an executable JAR file, you've got to create your JAR file with a manifest entry that specifies the name of the class with the main method.
There's a lot more that you have to do before that code will work:
add code to (at least) log the exceptions that you are catching
add code to process the messages
add code to initialize the connection factory and connection objects
And like I said above, you probably need some kind of design ... so that you don't end up with everything in a "kitchen sink" class.
if I add main method, should I simply call receiveMessages() in main?
That is one approach. But like I said, you really need to design your application.
And then after running, will the listener keep on running?
It is not entirely clear. It should keep running as long as the main thread is alive, but it is not immediately obvious what happens when your main method returns. (It depends on whether the JMS threads are created as daemon threads, and that's not specified.)
And if there are messages, will it retrieve automatically in onMessage method?
It would appear that each message is retrieved (read from the socket) before your onMessage method is called.
Also, if the listener is continuously listening, doesn't it take CPU???
Not if it is implemented properly.
In case of threads, when we create a thread & put it in sleep, the CPU utilization is zero, how doe it work in case of listener?
At a certain level, a listener thread will make a system call that waits for data to arrive on a network socket. I don't know how it is exactly implemented, but this could be as simple as an read() call on the network socket's InoutStream. No CPU is used by a thread while it waits in a blocking system call.
This link looks like a pretty good place with examples using Oracle AQ. There's an examples section that tells you how to setup the examples and run them. Hopefully this can help.
Link to Oracle Advanced Queueing
How can I create a timeout for each command that is running in parallel using java.util.concurrent.ExecutorService?
My code is something like this:
For example in the code below I need obj1 run for maximum 1 min, and obj2 for 2 mins and others 5 mins.
ExecutorService exService;
exService = Executors.newCachedThreadPool();
exService.execute(obj1);
exService.execute(obj2);
exService.execute(obj3);
exService.execute(obj4);
exService.shutdown();
boolean finshed = exService.awaitTermination(5, TimeUnit.MINUTES);
if (finshed) {
//Doing something
}
EDIT:
Unfortunately the class of obj1 - obj4 is scraping some web pages using WebHarvest that uses jakarta HttpClient for reading web pages and HttpClient (And neither WebHarvest itself) doesn't have any feature for timeout on entire page reading and/or scraping job.
This is my time consuming task and I thought about killing ExecutorService thread after a timeout to handle this problem.
In general, there is no reliable way to make a separate thread quit. In particular, there is no reliable way to interrupt and stop your task after a timeout from outside that task. What you need to do is make the tasks themselves responsible for stopping after their time runs out. Depending on what they do, you might be able to abstract this behaviour into a superclass something like:
public abstract class TimeoutRunnable implements Runnable {
private final long timeLimitMillis;
private long startTimeMillis;
public TimeoutRunnable(long timeLimitMillis) {
this.timeLimitMillis = timeLimitMillis;
}
public final void run() {
startTimeMillis = System.currentTimeMillis();
while (System.currentTimeMillis() - startTimeMillis < timeLimitMillis) {
runIteration();
}
}
protected abstract void runIteration();
}
Then in your subclass override, runIteration() and perform a single "step" of the task.
The only reasonably reliable way to kill a task is to run it in a separate process and kill that process if it times out. Using any other approach with a library which does not support timeouts is likely to be error prone at best.
from my point of view I think that such stuff requires some more robust foundations than plain Java standard classes , that 's why I would suggest to use any scheduler infrastructure (Quartz or any other project) which may gives you handles (job identifiers) to kill your time consuming tasks .
You may have something like this :
Main Thread launches the Quartz Scheduler , receive subscriptions from different jobs
saying : Job1,Job 2 and TimeCheckerJob
TimeCheckerJob would be a forever job ,notified to any new job and would check for living time for each any new job... In this job you would have to deal with start time of each job, beware of the OS clocks and don't try to setup too hard constraints (nanoseconds is pure fiction).
HTH
My 2 cents
Jerome
Edit
This question has gone through a few iterations by now, so feel free to look through the revisions to see some background information on the history and things tried.
I'm using a CompletionService together with an ExecutorService and a Callable, to concurrently call the a number of functions on a few different webservices through CXF generated code.. These services all contribute different information towards a single set of information I'm using for my project. The services however can fail to respond for a prolonged period of time without throwing an exception, prolonging the wait for the combined set of information.
To counter this I'm running all the service calls concurrently, and after a few minutes would like to terminate any of the calls that have not yet finished, and preferably log which ones weren't done yet either from within the callable or by throwing an detailed Exception.
Here's some highly simplified code to illustrate what I'm doing already:
private Callable<List<Feature>> getXXXFeatures(final WiwsPortType port,
final String accessionCode) {
return new Callable<List<Feature>>() {
#Override
public List<Feature> call() throws Exception {
List<Feature> features = new ArrayList<Feature>();
//getXXXFeatures are methods of the WS Proxy
//that can take anywhere from second to never to return
for (RawFeature raw : port.getXXXFeatures(accessionCode)) {
Feature ft = convertFeature(raw);
features.add(ft);
}
if (Thread.currentThread().isInterrupted())
log.error("XXX was interrupted");
return features;
}
};
}
And the code that concurrently starts the WS calls:
WiwsPortType port = new Wiws().getWiws();
List<Future<List<Feature>>> ftList = new ArrayList<Future<List<Feature>>>();
//Counting wrapper around CompletionService,
//so I could implement ccs.hasRemaining()
CountingCompletionService<List<Feature>> ccs =
new CountingCompletionService<List<Feature>>(threadpool);
ftList.add(ccs.submit(getXXXFeatures(port, accessionCode)));
ftList.add(ccs.submit(getYYYFeatures(port accessionCode)));
ftList.add(ccs.submit(getZZZFeatures(port, accessionCode)));
List<Feature> allFeatures = new ArrayList<Feature>();
while (ccs.hasRemaining()) {
//Low for testing, eventually a little more lenient
Future<List<Feature>> polled = ccs.poll(5, TimeUnit.SECONDS);
if (polled != null)
allFeatures.addAll(polled.get());
else {
//Still jobs remaining, but unresponsive: Cancel them all
int jobsCanceled = 0;
for (Future<List<Feature>> job : ftList)
if (job.cancel(true))
jobsCanceled++;
log.error("Canceled {} feature jobs because they took too long",
jobsCanceled);
break;
}
}
The problem I'm having with this code is that the Callables aren't actually canceled when waiting for port.getXXXFeatures(...) to return, but somehow keep running. As you can see from the if (Thread.currentThread().isInterrupted()) log.error("XXX was interrupted"); statements the interrupted flag is set after port.getFeatures returns, this is only available after the Webservice call completes normally, instead of it having been interrupted when I called Cancel.
Can anyone tell me what I am doing wrong and how I can stop the running CXF Webservice call after a given time period, and register this information in my application?
Best regards, Tim
Edit 3 New answer.
I see these options:
Post your problem on the Apache CXF as feature request
Fix ACXF yourself and expose some features.
Look for options for asynchronous WS call support within the Apache CXF
Consider switching to a different WS provider (JAX-WS?)
Do your WS call yourself using RESTful API if the service supports it (e.g. plain HTTP request with parameters)
For über experts only: use true threads/thread group and kill the threads with unorthodox methods.
The CXF docs have some instructions for setting the read timeout on the HTTPURLConnection:
http://cwiki.apache.org/CXF20DOC/client-http-transport-including-ssl-support.html
That would probably meet your needs. If the server doesn't respond in time, an exception is raised and the callable would get the exception. (except there is a bug where is MAY hang instead. I cannot remember if that was fixed for 2.2.2 or if it's just in the SNAPSHOTS right now.)