Develop Reference

Java process in linux needs initial warmup

we have a legacy java multithreaded process on a RHEL 6.5 which is very time critical (low latency), and it processes hundreds of thousands of message a day. It runs in a powerful Linux machine with 40cpus. What we found is the process has a high latency when it process the first 50k messages with average of 10ms / msg , and after this 'warmup' time, the latency starts to drop and became about 7ms, then 5ms and eventually stops at about 3-4ms / second at day end.
this puzzles me , and one of possibility that i can think of is maps are being resized at the beginning till it reaches a very big capacity - and it just doesn't exceed the load factor anymore. From what I see, the maps are not initialized with initial capacity - so that is why i say that may be the case. I tried to put it thru profiler and pump millions of messages inside, hoping to see some 'resize' method from the java collections, but i was unable to find any of them. It could be i am searching for wrong things, or looking into wrong direction. As a new joiner and existing team member left, i am trying to see if there are other reasons that i haven't thought of.
Another possibility that i can think of is kernel settings related, but i am unsure what it could be.
I don't think it is a programming logic issue, because it runs in acceptable speed after the first 30k-50k of messages.
Any suggestion?

It sounds like it takes some time for the operating system to realize that your application is a big resource consumer. So after few seconds it sees that there is a lot of activity with files of your application, and only then the operating system deals with the activity by populating the cache and action like this.

Hashing Password in Google App Engine and Instance Hours Quota

I've been reading a lot about password storing, hashing, salting, "peppering", MAC, etc because I'm about to make a new website and security it's really important to me, however there are some reasons why I'm considering not using Google Authentication (or Facebook, OpenID or any other) which are not relevant right now, but it brings me to this point.
I'm new to Google App Engine, this is going to be my first project on it, and I'm a little confused about the "Instance Hours" and how it no longer has "CPU time" but the aforementioned quota. Even worst, I haven't been able to understand what is the Instance Hours Free Quota.
Here's why I'm worried about the quotas and what does that has anything to do with my security concerns: One recommendation I've read everywhere is to make multiple iterations and hash the password several times, because that would make and attacker spend much much much more time (I don't have numbers, but they are everywhere on https://security.stackexchange.com/).
Multiple iterations have direct impact on CPU time, and if GAE had a CPU time quota I think making 1000 iterations every time a user logs in could be a problem, however if what they count is Instance Hours from the moment the request is done to up to fifteen minutes later and as read on GAE quota docs is:
In general, instance usage is billed on an hourly basis based on the
instance's uptime. Billing begins when the instance starts and ends
fifteen minutes after the instance shuts down. You will be billed only
for idle instances up to the number of maximum idle instances set in
the Performance Settings tab of the Admin Console. Runtime overhead is
counted against the instance memory.
then it means that if my users log in (hash 1000 times), then they continue to use the site, the Instance Hours will continue to sum until all of them leave the page + 15 minutes? If this is true, then making it iterate 1000 times wouldn't have a significant impact on my quota, other than the "extra" time it takes for the user to log in, but I'm aware of that and it's a price I'm willing to pay.
The number of iterations I'll make will be the ones that make the time to log in acceptable and imperceptible to the user, so don't worry about this.
My questions are:
Will making MANY iterations have a direct impact on the Instance Hours, or my assumptions about how the Instance Hours are summed are correct?
Is there a CPU time quota on Google App Engine I'm missing somehow? Does it have a Free Quota?
What is the Instance Hours Free Quota?
Answers:
Look Moishe accepted answer and the other question he asked (which has not been answered but has usefull comments) When does the App Engine scheduler use a new thread vs. a new instance?
According to Google there is no CPU time quota: http://googleappengine.blogspot.com.es/2009/02/skys-almost-limit-high-cpu-is-no-more.html
Found an answer to question number 3 here: Google App Engine Frontend Instance Hours Limit Reached

If it takes a long time to process a request, because eg. you're doing something very computationally intensive, and you don't want other users to wait a long time, the App Engine scheduler may spin up another instance of your application to serve incoming requests.
Imagine that computing the hash for a password takes 1 minute and during that minute your application gets a request from another user. That user could wait for a minute to get a response to their request, or the App Engine scheduler could spin up another instance to service that request and get a response back much sooner. You can tune whether or not another instance will come up using the Performance sliders on your Application Settings page in the admin console.
Basically the question you need to ask about instance hours is: is it likely you'll get overlapping requests (ie. a new request coming in before the current request is complete). If this happens not-infrequently, and you want snappy response for your users, you'll need to budget more instance hours.
I suspect that the big computation you'll need to do will be infrequent -- only on initial sign-in to generate a cookie, say, rather than for every request.
To explicitly answer your question #1, making many iterations will only have an effect on your instance hours if it causes overlapping requests. If you only get one request every 30 seconds, you could spend 30 seconds serving each request (including calculating each hash, and doing other operations) and not exceed your free instance-hours quota. Conversely if you get 10 requests per second and spend any more than 100ms serving the request, then you'll start to exceed your instance hours fairly quickly.

Instance hours are for long as the server is running, answering requests, etc. If your server isn't running, it can't wake up on a request or anything.
Imagine instance hours as having the computer on. You are billed when it's on, and not when it's off.
You could have multiple instances, so let's say you have two instances, you're burning twice as many instance hours.
Your password hashing won't affect this because it will only incur instance hours when the instance is on, and when its off, it won't be incurring any instance hours, but it won't be hashing either.

There are multiple sources covering passwords. You evidently have read some that encourage multi-pass hashing. Consider the first link below before finalizing this decision. Excerpt from this page: "It's easy to get carried away and try to combine different hash functions, hoping that the result will be more secure. In practice, though, there is no benefit to doing it. All it does is create interoperability problems, and can sometimes even make the hashes less secure."
Two valuable links to consider( first has quote above, second is good "how to" source):
http://crackstation.net/hashing-security.htm
http://throwingfire.com/storing-passwords-securely/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+throwingfire+%28Throwing+Fire%29#notpasswordhashes

Change the thread count of test plan in JMeter, at run time

I want to change the number of threads for a JMeter test plan at runtime.
I have Googled my problem and found a proposed solution to use JMeter plugins. But in this solution I would have to schedule the thread group before running the test plan, which I don't want. I also found another potential solution which changes the property, but doesn't affect test plan behavior at run time.
Ultimately, what I am trying to do is change the thread number given in a thread group and have it immediately increase or decrease the number of threads in the current running test plan.
Is this possible?

IMHO that's just a fancy feature that has no real benefit when doing proper performance testing.
In order to generate relevant test output (report), you need repeatability, and clearly defined test methodology and scenarios. In order to compare impact of any application/server/infrastructure changes, you need repeatability.
What do you mean by
We can't predict the user of our site
That's why we do performance testing at the first place. To find out what is our application/infrastructure limit.
I.e. the most significant metric you can produce is how your application response time changes when number of parallel users change. But not change erratically, in run time.
With jMeter plugins' Ultimate thread group you can cover any imaginable scenario.

The short answer is: no, you cannot change the number of threads dynamically during runtime. Each thread count value is only read once when the test plan is first compiled and is not resolved again after this point, so it remains fixed.

This feature is indeed useful, and surprisingly difficult to implement even with commercial tools such as Loadrunner. I would compare it to finding a loudspeakers maximum volume. You would manually turn the volume up until it started to crackle, then turn it back down slightly to maintain that maximum volume. In the same way, to find the peak capacity of an application, you want control to 'turn the volume up' until errors are seen, then back it down slightly to see if it stabilizes. You can then maintain that load to find where the bottleneck is.
Anyway, to answer the question, what I have done in the past is use an external influence, such as a file name or similar. Then combine that with the thread unique reference you can control which threads run and which are held (by pausing or similar).
For example, if you start with 100 threads, then create a file called '5.txt' in a specific location, you can add code such that if the threads sees that it's own reference is equal to or lower than the number then it can run. If not then it drops into a pause. At the start of this example 5 threads would run, and 95 would pause. You can then rename the file to '25.txt', and threads 6 to 25 would start running. It would work the other way too, changing it to '20.txt' would mean threads 21-25 pause again.
The key is to start enough threads to exceed your expected peak.

you can change it based on a variable which you set in a startup thread. See below.
In Jmeter how do I set a variable number of threads using a beanshell sampler variable?
However once the thread group has started you can not modify it. To the guy who said this feature would not be useful I disagree. There are many types of load tests and they do not all have the same number of users running for the duration. Here are just 2 example types of enterprise load tests we conduct at the bank where I work:
duration test - same number of users run the entire time (possibly
with a short ramp-up period)
Break point test - ramp up the number of users incrementally till the
application breaks
Spike test - run with a constant number of users but sporadically
throw in a large number of users
A break point test ramps up the number of users until the application breaks (the point being to see how high your app can scale). You can sort of do this using the thread groups "ramp up period" property. If you set ramp up time to 1000 and the number of threads to 100 it will add 1 thread every 10 seconds.
Spike tests are like duration tests but at some intervals a large number of users log in. This is used to guage the applications response time during peak hours or how it will respond if you all of a sudden get a large number of users (a very real scenario).
I find that Jmeter does not handle all load test scenarios that are needed in enterprise load testing. One work around Im considering is to just start all the threads but find a way to make some of them sleep. So you could set the number of threads to 1000 but somehow make 980 of them sleep or do nothing. Then maybe when the time_in_seconds%5==0 (every 5 minutes) you allow the other threads to run - simulating a spike test. The idea is you can hard code the threads to 1000 and will always have 1000 threads running - but they don't all have to be doing something at all times.
(in other words you can probably find a way but you have to get creative)
Update:
I just found this plugin which allows different types of testing. Have not tried it yet but looks promising:
http://jmeter-plugins.org/wiki/ThroughputShapingTimer/

You can set/change the number of threads at runtime using command line option...
you can use function calls, or variable references to User Parameters (which in turn could be functions), or variable references to variables set up by functions earlier in the test. There's more than one way to do it.
Suppose you want to be able to vary the number of threads in a test plan. Choose a suitable property name, say group1.threads. Replace the thread count in the GUI (or the JMX, if you're feeling brave!) with the following function call:
Please set below property in JMeter thread group as below
${__property(group1.threads)}
Then, when starting JMeter, define the property on the command line:
jmeter -Jgroup1.threads=10

We can't predict the user of our site.
Sure you can. This is what the HTTP logs of your existing site are for. You can also use logs from tools like Omniture or your CDN logs. If you look at the combination of Actual user IP address, request and referer tags in the logs you will be able to build a traversal map of ever single user on your site. You will be able to profile the high hit unique leaf node pages of a given business process to understand how many times a particular business process happens an hour. You will be able to examine abandonment by taking a look at the funnel in tools such as Omniture. If you need tools for this analysis I recommend Splunk. It's easy to install and configure. Time to value is very fast.
The more log data you have which you are using to profile the closer you can come to actual for what users do during a day/week/month/spot sale/end of quarter/end of year/etc....You need to combine actual at a point in time with actual from earlier points in time to project growth over time since you will need to allow for growth in your performance testing model.
If you don't get the values right then the value of your test as a predictor of what will/can happen in production will be quite low. This is not a failure of any given tool, but a failure in process on the planning front for the actual load model used as part of the test requirements. If you cannot build these models then you need to pull someone into your team who can.
This ability to produce a valid load model independent of tool is the difference between tests which reduce risk and throwing load.

By enabling the BeanShell server you can vary properties at runtime.
Just enable it and telnet on port 9001 (warning: not secure!)
Based on a test I did, unfortnately, it appears that the thread count it's not applied at runtime. However you can still manipulate the load of the test by other means, for example apply a costant throughput timer parametrized with a property named "throughput" and vary it at runtime like this:
setprop("throughput","2000");
It's well explained in the guide.

Cons of using Thread.sleep to slow down script?

This pertains to Google App Engine, using Java.
("threadsafe" in this scenario would be set to true, so please include that in your thought process.)
Which of the App Engine quotas are affected?
The goal would be to slow down some users (non-paying users), in such a way that their requests would take longer to arrive. (The time spent on the website/app is in the range of hours, per user.) So in a situation where the user is slowed down by double the wait, half of the theoretical bandwidth usage is saved, per user, since they don't spend even more time on to compensate; they go to bed at the end of the day, in this scenario.
Specifically, my goal is for this to be more cost effective, more than anything.
Are CPU quotas affected by thread delays?
(Excuse my ignorance on this topic. I assume that such a Java thread delay doesn't incur CPU usage itself, but I could be wrong, which is why I'm asking you guys, who are more knowledgeable.)
Any other things I should consider to be detrimental?
(Make sure to factor in that, in this scenario, the user themselves aren't negatively impacted by the site loading slower, and that they patiently wait for the page to load. It is the type of application where the user isn't bothered by a little bit of lag because they spend a lot of time on the site just to kill time. Hope this is understood.)
Thanks in advance for any thoughts, and thanks for at least reading my question.

Under the current billing model, your app will not scale up if your handlers take more than 1000ms on average to return a response. Deliberately adding sleep calls will slow your app down, and make it less likely to be autoscaled.
Under the new billing model, as Chris points out, you will be charged by instance-hours, which means that although your app will scale fine, you will pay more to slow your users down.
When most people are trying to shave milliseconds off their response times, you're asking about ways to artificially slow requests down; this seems very odd, and likely to drive your users off. Wouldn't you be better off serving them a page informing them they're over-quota and inviting them to come back later (or, say, pay you)?

How do you determine an acceptable response time for App Engine DB requests?

According to this discussion of Google App Engine on Hacker News,
A DB (read) request takes over 100ms on the
datastore. That's insane and unusable
for about 90% of applications.
How do you determine what is an acceptable response time for a DB read request?
I have been using App Engine without noticing any issues with DB responsiveness. But, on the other hand, I'm not sure I would even know what to look for in that regard :)

You can measure precisely how much each RPC call (datastore or otherwise) is taking, thanks to Guido van Rossum's AppStats relatively-new component (it's part of the standard SDK since 1.3.1). See here for more. 100 milliseconds is fine for most well-designed apps -- if you need to make two or three queries to serve a page, you can still serve in less than half a seconds even if there's lots of processing and rendering involved... not too shabby. Plus, you can use memcache to reduce many of those latencies, etc.

The poster is wrong. Datastore get operations are much faster - about 15-20ms each, currently. Datastore query operations can be slower, because they're much more involved and return more data, but they still complete in anywhere from 30-100ms for a typical query. Other posters have amply addressed whether that's "acceptable" or not.

What do you mean by acceptable? What kind of application are you writing? Acceptable means different things for different domains/applications/people. First, you should decide how quickly you want your app to respond to a request. Let's pick 1 second, just for argument's sake. Now, how many DB requests do you need to make to fulfill that request? Let's say 5. Let's also say that we also have 400ms worth of other processing to do. OK, so that's 5 reads times 100ms each, plus 400ms of other stuff. 900ms total, which is less than our goal of 1 second. Perfect! 100ms is an acceptable read rate. In fact, 120ms would still be acceptable, just barely.
Now, let's generalize:
numberOfReads * readTime + otherStuffTime = TotalTime
Fill in your numbers, and you can see what is an acceptable time for your particular situation.

If you haven't noticed any issues then it is by definition an acceptable response time. The only question is how long your users are happy to wait.

An "an acceptable response time for a DB read request" depends entirely on your application and your users.
If the net result is that your site runs fast enough to satisfy you and your users then the slow response time of the services provided by Google in their AppEngine are acceptable.
Now, looking deeper at this particular issue, it sounds like we are talking about GET's. Here are the figures for GET latency and it looks to me that the average latency is closer to 50ms then 100. I'm not saying that is good, but I don't think it is accurate to say 100ms.