I have a Java GAE instance that runs TaskQueue jobs. In the last 48 hours I have been getting lots of CancellationException errors (about 36 in a 24 hour period).
The error always occurs on a DataStore query to a particular table. The query is on an indexed column, projectId,and the query is a composite OR (i.e. projectId = -1 or projectId = 23). The entire table in question has 96 records and any one query will pull back 20 records at most.
Does anyone have any idea why I'm getting these problems
Thanks
Paul
Update: Sorry my mistake, this isn't part of a Task Queue call it is the code that sets up the task queue call. The time the call took is on average about 59.5 secs (thanks for pointing that out Alex). Looking at the trace (thanks Alex) it seems that the delay is in the datastore call. It can be as quick as 21ms or over 59000ms, which is way to long. I just checked the logs again and there hasn't been any of those exceptions in the last 24 hours so this might be a 48 hour glitch.
Understanding "CancellationException: Task was cancelled" error while doing a Google Datastore query
Based on this question, it sounds like CancellationException means that you are hitting the 10-minute timeout that TaskQueue Jobs have to complete execution.
Do your logs show how long these requests are running for before they throw an exception (for example stackdriver shows that this deferred task ran for 902ms)?
Querying a table of 96 records should be fine, but it's hard to be certain without being able to see what the code of your task looks like.
You could split your task up into smaller 'sub-tasks' and have sub-task launch the next sub-task before returning.
EDIT: you could also try going here https://console.cloud.google.com/traces/traces to see a break down of where your request is spending its time
Related
I have around 1000 entries in my datastore and this is likely to increase with time to around 10,000 entries. My task is to update each row's certain properties and save it back and this task has to be performed every 24 hours.
So, what should I use?
First, you create a cron job that runs every 24 hours.
Second, you need to decide what this cron job will do. The simplest option is to update all 1,000 records. You can retrieve and save all entities in large batches (i.e. 500 per call). If this is a simple update of values, it will take just a few seconds.
Since cron jobs are not retried if they fail, a better option is to create a task and add it to the queue. All updates will happen within that task.
NB: Make sure that if your task is retried, it won't mess the data. If this is not possible, you will have to use some kind of flag (i.e. timestamp of last update) to separate updated entities from those that still need updates.
As your data set grows, your cron job can start multiple tasks to update, for example, 1,000 records in each task.
In the task queue the tasks have to be added to the queue manually though code. If you want to do this task automatically every x time, what you need is a cron job.
You need both,
Cron job to start your batch update job every 24 hours
Task-queues to process you records.
I have one method execute(data) which takes considerable time (depending on data like 10 seconds or 20 seconds), it has timeout feature which is 30 seconds default. I want to test that method. One way of doing it is to collect enough data which lasts more than 30 seconds and then see whether I get timeout exception. Other way of doing it is to use threads. What I intend to do is to run method for some milliseconds and then put thread on wait before I get timeout exception or make it last for some seconds.Can any one please suggest how can I achieve that.
You should walk through the Java Threads Tutorial (Concurrency). Any answer on Stack Overflow would need to be really long to help you here, and the Threads/Concurrency tutorials already cover this well.
http://docs.oracle.com/javase/tutorial/essential/concurrency/
You could use
Thread.sleep( millis );
to put the thread to sleep for the required time.
Or, you could put your data processing code into a loop, so that it processes it multiple times. This would recreate the scenario of the thread actually processing data for longer than 30 seconds.
Or, you could test your code with a shorter timeout value.
I am trying to chase down a problem with CAS that is causing the following exception to be thrown:
javax.naming.TimeLimitExceededException: [LDAP: error code 3 - Timelimit Exceeded]; remaining name ''
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3097)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2987)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2794)
at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:129)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:198)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:171)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:295)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:361)...
The error is returned virtually instantly. The client side timeout is set to 10 seconds, but that isn't occuring as, based on looking through the com.sun.jndi.ldap code, it appears that the domain controller is returning a response with a status of 3, indicating a time limit exceeded.
We are hitting an Active Directory global catalog, and our filter and base are pretty broad: base = '', filter = (proxyAddresses=*:someone#somewhere.com)
However, the query succeeds sometimes, but returns an immediate status code 3.
Does anyone know what might be causing this kind of behavior? Or perhaps how to go about determining what exactly is occurring?
Turns out our search filter was too broad.
As you can see, we were using a wildcard in the filter, and the query took a little less than 2 seconds.
However, 2 seconds is far shorter than the Active Directory configured time limit so I couldn't figure out why the error was occurring immediately (not even taking 2 seconds when it failed).
I assume AD must have been accruing the time taken by multiple requests from the same account, and at some point began returning the time limit exceeded error.
To solve it, we modified the search filter so that it no longer included a wildcard. The search then runs almost instantaneously, and the time limit exceeded no longer occurs.
I have cron setup in appengine project:
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
<cron>
<url>/cron/someurl</url>
<description>cron</description>
<schedule>every monday 8:00</schedule>
<timezone>Asia/Singapore</timezone>
</cron>
</cronentries>
I am getting the error:
com.google.apphosting.api.DeadlineExceededException: This request (40811df3b6350a70) started at 2012/11/26 00:00:00.404 UTC and was still executing at 2012/11/26 00:09:59.917 UTC.
It's a 1-minute limit to run the task? I though cron doesn't have that limit. How to avoid the error in cron entry?
Thanks.
An HTTP request invoked by cron can run for up to 10 minutes, as per the documentation. If you notice the exception log closely in the HH:MM:SS value, you will find that a total of 10 minutes have passed since the job was started.
You might want to look at your code to see why it is taking that long. In case you have requirements that make your tasks run longer than 10 minutes, I suggest that you look at trapping the exception and then inserting another request to run a job with some request parameter that tells the job to start from where it left off last time.
Alternately, you could also look at Backends.
Also you can try using task queues (10 minute maximun) and re-enqueue when you are near to time limit, using for that any state variable stored in datastore, or if you are depending of a iteration in datastore you can pass the datastore cursor to the other re-enqueued task. For me that works very well.
I'm about to create a small application which will be responsible for sending out various reports to various users at various intevals. We might be talking about 50 or 100 different reports going to different people. Some reports needs to be generated every day, some every week, and some every month.
I've been using the Quartz library earlier to run tasks at regular intervals. However, in order to keep things simple I like the thought of having a single Quartz thread taking care of all reports. That is, the thread should loop through all reports, say every 15 minutes, and determine wether it is time for one or more to be generated and sent. It does not matter if a report is generated at 12:00 or 12:15.
I'm thinking about wether it would be possible, somehow, for each report to set up specific times such as "mon#12:00,wed#12:00" or "fri#09:30". Then, based on that, the thread would determine if it was time to send a report or not.
My question is; has anyone else done something like this and does any libraries exist which can make it easy to implement this task?
why not simply register a separate quartz task instance for each report and let Quartz handle all the scheduling for you? That is after all the point behind it.
you can create just single thread and it would ping a "job schedule data structure" at some time interval to see if it needs to run a report. If yes, it would run the report, otherwise, it would go for a short nap and ping again after specified sleep time.
It will cause problem if one job takes too much time to complete and you start accumulating jobs.
The job schedule data structure would keep its record sorted by time stamp.