I am having some issues with an app I have deployed on GAE. Specifically, I am intermittently running into the DeadlineExceededException where the server is not responding within the 30 seconds required.
What is odd is that the code is not overly complex, it should run in milliseconds. My guess is that the delay is in dealing with the persistence manager and accessing the datastore.
2 questions:
1) What is the best way to track where all of the CPU time on the server is being used up? Log files do not seem helpful and to make things more complicated the code runs very fast when I am running it locally
2) Any tips / best practices in dealing with the 30 second exception? What are the biggest drivers of this? Datastore? HTTP requests / responses?
Thanks
These SO threads might be helpful,
How is DeadlineExceededException implemented in Google App Engine for Java?
Techniques to avoid DeadlineExceededException in GAE/J?
The Google App engine blog covers this exception in the entry about the 1.2.8 release (current release is 1.3.0) of GAE:
Request performance in Java
The new class-loading aptimization option reduces the length of loading requests, which occur when App Engine is preparing a new instance of your code to respond an incoming request:
First, we're introducing a new
class-loading optimization in 1.2.8
called precompilation. Precompilation
makes loading requests faster by doing
class-loading work ahead of time in
the App Engine environment.
Related
In Play's documention about ThreadPools it's recommended for highly synchronized Java web applications with a lot of blocking IO to 'use a very large number of threads in its pool'. In the example they use 200 to 300 parallel threads. My app reads files from a hard disk in most of the requests and nearly each request access a MySQL database via JPA, so I'd say it's pretty synchronized.
Now I did some stress test with Gatling on my development laptop and compared the Play's default thread pool with the one recommended for synchronized web applications. Surprisingly I couldn't find any difference between both.
Do I the wrong kind of stress test? What kind of stress test should I do to see the difference in the two configs? Or do I misunderstand Play's documentation?
Regarding your load test, make sure to you don't just run a few concurrent users that execute requests at fast pace. This way, you might get the number of requests per sec you want, but not the proper level of concurrency.
Make sure you run the proper number of concurrent users to match what you're expecting on your live system.
As you can see in the following snapshot the load in some of the dynamic instances is huge (more than 20k requests) while in other are very small.
Why is this happening? Shouldn't GAE distribute uniformly the load??
If the load would be balanced across the active dynamic instances then they'd rarely become idle (only when the entire app's traffic would drop to almost nothing) thus it'd be difficult to dynamically shut them down.
More info here:
https://cloud.google.com/appengine/docs/scaling#scaling_dynamic_instances
https://cloud.google.com/appengine/docs/managing-resources#instances
This is what I got from a Google App Engine expert:
App Engine request scheduling uses several heuristics for routing requests to application instances. At low QPS it stays in affinity scheduling mode and routes majority of requests to instances that have most recently responded to the health check and handled requests successfully. That would explain why you see this variation in number of requests for each instance. As you ramp up the application traffic, load should even out across all instances.
I also asked what was the policy GAE follow to shut down the instances. I see that many of them are up even if they are not receiving any request
Dynamic instances that are not serving requests get garbage collected eventually. However, you only get billed for 15 additional minutes after they serve the last request. Please refer to this doc for additional information on instance billing.
https://cloud.google.com/appengine/kb/billing#different_on_demand_instance_resident
I read that Google App Engine (GAE) will shut down your application if it goes idle, and startup/boot everything again when it gets a request. And i know that Spring startup is slow, like 2-3 seconds even for a small web app. Is working on GAE using Spring really suffer from this badly?
Thanks in advance.
It's really not that bad but considering your instances are being shutdown and started constantly, you should work on getting your startup as fast as possible. A few pointers to consider:
Enable warmup requests
Enable resident instances
Optimize Spring config (There are great suggestions in this article)
We are implementing a bespoke 3rd party J2EE application on a 6 server weblogic cluster (latest versions of Oracle products - running on SuSE). The supplier is suggesting to me that we schedule a restart of each WebLogic instance every week on a Monday morning at 3am.
I'm no weblogic expert and I can't seem to track down any best practice guidelines on the subject of regular restarts, but I'm used to working in environments where other clustered app server instances have uptime is measured in much longer periods than 7 days...
My concern is that this is intended to mask issues in the J2EE app itself. Can anyone point me towards best practice guidance related to Weblogic which I may have missed, or confirm that this may be a legitimate suggestion from the application vendor?
We don't always get perfect codes, no-wrong applications, and best programmers to work with you, in fact, many codes are written by junior programmers with low cost. So it is reasonable there are some bugs in these J2EE applications (depend on OS patch level, java version, application itself, etc). Memory leak is one of the problem to ask regular restart to avoid the applications go down at business time. Some other problems are hide in and can't be easily found out.
That's the reason to recommend to restart the application fortnight, weekly, or daily (I DO see some business java application restart every night).
If you really want to troubleshooting the application, maybe you can install some APM (application performance management) application to help you to find out why the application have memory leak, unstable behavior, etc.
You can search in google or read this URL for a starting : http://en.wikipedia.org/wiki/Application_performance_management
I'm new here and I'm not that very good in CPU consumption and Multi Threading. But I was wondering why my web app is consuming too much of the CPU process? What my program does is update values in the background so that users don't have to wait for the processing of the data and will only need to fetch it upon request. The updating processes are scheduled tasks using executor library that fires off 8 threads every 5 seconds to update my data.
Now I'm wondering why my application is consuming too much of the CPU. Is it because of bad code or is it because of a low spec server? (2 cores with 2 database and 1 major application running with my web app)
Thank you very much for your help.
You need to profile your application to find out where the CPU is actually being consumed. Java has some basic profiling methods built in, or if your environment permits it, you could run the built in "hprof" compiler:
java -Xrunhprof ...
(In reality, you probably want to set some extra options: Google "hprof" for more details.)
The latter is easier in principle, but I mention the possibility of adding your own profiling routine because it's more flexible and you can do it e.g. in a Servlet environment where running another profiler is more cumbersome.
Paulo,
It is not possible for someone here to say whether the problem is that your code is inefficient or the server is under spec. It could be either or both of those, or something else.
You are going to need to do some research of your own:
Profile the code. This will allow you to identify where your webapp is spending most of its time.
Look at the OS-level stats that are available to you. This might tell you that the real problem is memory usage or disk I/O.
Look at the performance of the back-end database. Is it using a lot of CPU?
Once you have identified the area(s) where the CPU is being used, you need to figure out the real cause of the problem is and work out how to fix it. And once you've got a potential fix implemented, you can rerun your profiling, etc to see it has helped.