Develop Reference

During Performance Test (Jmeter) of Spring Boot application the tested Method has some extremely short runtimes

I am currently working on my Bachelors Thesis on the advantages/disadvantages of GraalVM Native Image compared to a Jar running on the JVM.
During one of the Tests i am calling a certain Method which allocates and populates an Array of size 10^6. After that the function loops the array and performs arithmetic operations (It is a variant of the ackley function). The usual runtime of this method was between 3 and 4 seconds, but sometimes the method would complete after just 50 ms (when running as either the Native Image or as the jar file running on the JVM).
Since the array is populated by using the Math.Random() function I dont think it is due to caching and the Native Image rules out JIT compilation as the source for these outliers.
The endpoint looks like this, where dtno is the Data Transfer Object containing the "range" variable:
#PostMapping(path="/ackley")
public static #ResponseBody long calculateackley (#RequestBody dtno d) {
long start = System.nanoTime();
double res = ackley(d.range);
long end = System.nanoTime();
System.out.println("Ackley funtion took: "+res);
return end - start;
}
The ackley function looks like this:
public static long ackley(int range){
long start = System.nanoTime();
if(range!=0){
double[] a = new double[range];
int counter = 0;
for(int i=-range/2;i<range/2;i++){
a[counter++] = Math.random()*range*i;
}
double sum1 = 0.0;
double sum2 = 0.0;
for (int i = 0 ; i < a.length ; i ++) {
sum1 += a[i]*a[i];
sum2 += (Math.cos(2*Math.PI*a[i]));
}
double result = -20.0*Math.exp(-0.2*Math.sqrt(sum1 / ((double )a.length))) + 20
- Math.exp(sum2 /((double)a.length)) + Math.exp(1.0);
}
long end = System.nanoTime();
return end - start;
}
As already mentioned the range variable in the test was 10^6. What I was also suspecting is that, since the result and sum are never actually used to calculate the return value, the programm decides to skip everything between for loop and the decleration of "end".
In the graph from the JMeter test you can see that these fast execution times where all during ramp up and the very end of the testrun.
Test Results Performance Test
In the summary Report you can see the huge deviation from the average runtime.
Performance Test Summary Report
If anyone could give me a hint or a good source, where I could find a hint as to what is going on, I would be very thankfull.

for loop performance in Java

After playing around with a simple palindrome function, I was surprised to find the performance difference between two different approaches.
public static boolean checkPalindrome(String inputString) {
String[] arr = inputString.split("");
for (int i = 0; i < arr.length / 2; i++) {
if(!arr[i].equals(arr[arr.length - (i + 1)]))
return false;
}
return true;
}
In this function I am only iterating through half of the array.
And in the following I would imagine that the whole array is iterated and a new object is created through the builder pattern.
public static boolean checkPalindrome2(String inputString) {
return inputString.equals(new StringBuilder(inputString).reverse().toString());
}
I was extremely surprised to find that the first function has an average execution time of 550146 nano seconds, measure using System.nanoTime(), and the second has an average execution time of 61665 nano seconds, which is almost a tenfold increase in performance.
Could anybody help explain what is happening here?

Why is my java program becoming gradually slower?

I recently built a Fibonacci generator that uses recursion and hashmaps to reduce complexity. I am using the System.nanoTime() to keep track of the time it takes for my program to print 10000 Fibonacci number. It started out good with less than a second but gradually became slower and now it takes more than 4 seconds. Can someone explain why this might be happening. The code is down here-
import java.util.*;
import java.math.*;
public class FibonacciGeneratorUnlimited {
static int numFibCalls = 0;
static HashMap<Integer, BigInteger> d = new HashMap<Integer, BigInteger>();
static Scanner fibNumber = new Scanner(System.in);
static BigInteger ans = new BigInteger("0");
public static void main(String[] args){
d.put(0 , new BigInteger("0"));
d.put(1 , new BigInteger("1"));
System.out.print("Enter the term:\t");
int n = fibNumber.nextInt();
long startTime = System.nanoTime();
for (int i = 0; i <= n; i++) {
System.out.println(i + " : " + fib_efficient(i, d));
}
System.out.println((double)(System.nanoTime() - startTime) / 1000000000);
}
public static BigInteger fib_efficient(int n, HashMap<Integer, BigInteger> d) {
numFibCalls += 1;
if (d.containsKey(n)) {
return (d.get(n));
} else {
ans = (fib_efficient(n-1, d).add(fib_efficient(n-2, d)));
d.put(n, ans);
return ans;
}
}
}

If you are restarting the program every time you make a new fibonacci sequence, then your program most likely isn't the problem. It might just be the your processor got hot after running the program a few times, or a background process in your computer suddenly started, causing your program to slow down.

More memory java -Xmx=... or less caching
public static BigInteger fib_efficient(int n, HashMap<Integer, BigInteger> d) {
numFibCalls++;
if ((n & 3) <= 1) { // Every second is cached.
BigInteger cached = d.get(n);
if (cached != null) {
return cached;
} else {
BigInteger ans = fib_efficient(n-1, d).add(fib_efficient(n-2, d));
d.put(n, ans);
return ans;
}
} else {
return fib_efficient(n-1, d).add(fib_efficient(n-2, d));
}
}
Two subsequent numbers are cached out of four in order to stop the
recursion on both branches for:
fib(n) = fib(n-1) + fib(n-2)
BigInteger isn't the nicest class where performance and memory is concerned.

It started out good with less than a second but gradually became slower and now it takes more than 4 seconds.
What do you mean by this? Do you mean that you ran this exact same program with the same input and its run-time changed from < 1 second to > 4 seconds?
If you have the same exact code running with the same exact inputs in a deterministic algorithm...
then the differences are probably external to your code - maybe other processes are taking up more CPU on one run.
Do you mean that you increased the inputs from some value X to 10,000 and now it takes > 4 seconds?
Then that's just a matter of the algorithm taking longer with larger inputs, which is perfectly normal.
recursion and hashmaps to reduce complexity
That's not quite how complexity works. You have improved the best-case and the average-case, but you have done nothing to change the worst-case.
Now for some actual performance improvement advice
Stop printing out the results... that's eating up over 99% of your processing time. Seriously, though, switch out "System.out.println(i + " : " + fib_efficient(i, d))" with "fib_efficient(i,d)" and it'll execute over 100x faster.
Concatenating strings and printing to console are very expensive processes.

It happens because the complexity for Fibonacci is Big-O(n^2). This means that, the larger the input the time increases exponentially, as you can see in the graph for Big-O(n^2) in this link. Check this answer to see a complete explanation about it´s complexity.
Now, the complexity of your algorithm increases because you are using a HashMap to search and insert elements each time that function is invoked. Consider remove this HashMap.

Empirical analysis for binary search not matching Theoretical Analysis

I'm currently doing a test for the Binary searches average case. Simply all I do is I generate a random variable and then search for this random variable in different sized arrays using the binary search. Below is my code used:
public static void main(String[] args)
{
//This array keeps track of the times of the linear search
long[] ArrayTimeTaken = new long[18];
//Values of the array lengths that we test for
int[] ArrayAssignValues = new int[18];
ArrayAssignValues[0] = 1000000;
ArrayAssignValues[1] = 10000000;
ArrayAssignValues[2] = 20000000;
ArrayAssignValues[3] = 30000000;
ArrayAssignValues[4] = 40000000;
ArrayAssignValues[5] = 50000000;
ArrayAssignValues[6] = 60000000;
ArrayAssignValues[7] = 70000000;
ArrayAssignValues[8] = 80000000;
ArrayAssignValues[9] = 90000000;
ArrayAssignValues[10] = 100000000;
ArrayAssignValues[11] = 110000000;
ArrayAssignValues[12] = 120000000;
ArrayAssignValues[13] = 130000000;
ArrayAssignValues[14] = 140000000;
ArrayAssignValues[15] = 150000000;
ArrayAssignValues[16] = 160000000;
ArrayAssignValues[17] = 170000000;
//Code that runs the linear search
for (int i = 0; i < ArrayAssignValues.length; i++)
{
float[] arrayExperimentTest = new float[ ArrayAssignValues[i]];
//We fill the array with ascending numbers
for (int j = 0; j < arrayExperimentTest.length; j++)
{
arrayExperimentTest[j] = j;
}
Random Generator = new Random();
int ValuetoSearchfor = (int) Generator.nextInt(ArrayAssignValues[i]);
System.out.println(ValuetoSearchfor);
ValuetoSearchfor = (int) arrayExperimentTest[ValuetoSearchfor];
//Here we perform a the Linear Search
ArrayTimeTaken[i] = BinarySearch(arrayExperimentTest,ValuetoSearchfor);
}
ChartCreate(ArrayTimeTaken);
System.out.println("Done");
}
Here is my code for the binary search:
static long BinarySearch(float[] ArraySearch,int ValueFind)
{
System.gc();
long startTime = System.nanoTime();
int low = 0;
int high = ArraySearch.length-1;
int mid = Math.round((low+high)/2);
while (ArraySearch[mid] != ValueFind )
{
if (ValueFind <ArraySearch[mid])
{
high = mid-1;
}
else
{
low = mid+1;
}
mid = (low+high)/2;
}
long TimeTaken = System.nanoTime() - startTime;
return TimeTaken;
}
Now the problem is that the results aren't making sense. Below is a graph:
Can some explain how the 1st array takes so much time? I've run the code a good few times and its basically the same graph created every time. Does Java some how cache results? Can anyone explain the result why the 1st binary search takes so long relativve to the others even though the array size is tiny compared to the rest?

It looks like you're doing these searches one after another, starting with the lowest values. If that's true, then the code will be running much slower, because the JIT compiler won't have had a chance to warm up yet. Generally, for benchmarking like this, you want to run through all relevant code to give the JIT compiler time to compile it and optimize before you do the real testing.
For more information on the JIT compiler, read this
You should also see this question to learn more about benchmarking.
Another possible cause of the slowness is that the JVM could still be in the process of starting up, and running it' own background code while you're timing it, causing slowdown.

Benchmarking is not done this way, you should run at least 1000 cycles as a "warmup" and only then start measuring. Benchmarking could be more complicated than it seems, it should be carefully constructed to not be affected by other programs that run in the memory at the same time etc. Here and here you can find some good tips.

Jvm native code compilation crazyness - I seem to suffer odd performance penalties for some time even after the code is compiled. Why?

Issue
When benchmarking a simple QuickSort implementation in Java, I faced unexpected humps in the n vs time graphics I was plotting:
I know HotSpot will attempt to compile code to native after it seems certain methods are being heavily used, so I ran the JVM with -XX:+PrintCompilation. After repeated trials, it seems to be compiling the algorithm's methods always in the same way:
# iteration 6 -> sorting.QuickSort::swap (15 bytes)
# iteration 7 -> sorting.QuickSort::partition (66 bytes)
# iteration 7 -> sorting.QuickSort::quickSort (29 bytes)
I am repeating the above graphic with this added info, just to make things a bit clearer:
At this point, we must all be asking ourselves : why are we still getting those ugly humps AFTER the code is compiled? Maybe it has something to do with the algorithm itself? It sure could be, and luckily for us there's a quick way to sort that out, with -XX:CompileThreshold=0:
Bummer! It really must be something the JVM is doing in the background. But what?
I theorized that although code is being compiled, it may take a while until the compiled code actually starts to be used. Maybe adding a couple of Thread.sleep()s here and there could help us a bit sorting this issue out?
Ouch! The green colored function is the QuickSort's code ran with a 1000ms internal between each run (details in the appendix), while the blue colored function is our old one (just for comparison).
At fist, giving time to the HotSpot only seems to make matters worse! Maybe it only seems worse by some other factor, such as caching issues?
Disclaimer : I am running 1000 trials for each point of the shown graphics, and using System.nanoTime() to measure the results.
EDIT
Some of you may at this stage wonder how the use of sleep() might distort the results. I ran the Red Plot (no native compilation) again, now with the sleeps in-between:
Scary!
Appendix
Here I present the QuickSort code I am using, just in case:
public class QuickSort {
public <T extends Comparable<T>> void sort(int[] table) {
quickSort(table, 0, table.length - 1);
}
private static <T extends Comparable<T>> void quickSort(int[] table,
int first, int last) {
if (first < last) { // There is data to be sorted.
// Partition the table.
int pivotIndex = partition(table, first, last);
// Sort the left half.
quickSort(table, first, pivotIndex - 1);
// Sort the right half.
quickSort(table, pivotIndex + 1, last);
}
}
/**
* #author http://en.wikipedia.org/wiki/Quick_Sort
*/
private static <T extends Comparable<T>> int partition(int[] table,
int first, int last) {
int pivotIndex = (first + last) / 2;
int pivotValue = table[pivotIndex];
swap(table, pivotIndex, last);
int storeIndex = first;
for (int i = first; i < last; i++) {
if (table[i]-(pivotValue) <= 0) {
swap(table, i, storeIndex);
storeIndex++;
}
}
swap(table, storeIndex, last);
return storeIndex;
}
private static <T> void swap(int[] a, int i, int j) {
int h = a[i];
a[i] = a[j];
a[j] = h;
}
}
as well the code I am using to run my benchmarks:
public static void main(String[] args) throws InterruptedException, IOException {
QuickSort quickSort = new QuickSort();
int TRIALS = 1000;
File file = new File(Long.toString(System.currentTimeMillis()));
System.out.println("Saving # \"" + file.getAbsolutePath() + "\"");
for (int x = 0; x < 30; ++x) {
// if (x > 4 && x < 17)
// Thread.sleep(1000);
int[] values = new int[x];
long start = System.nanoTime();
for (int i = 0; i < TRIALS; ++i)
quickSort.sort(values);
double duration = (System.nanoTime() - start) / TRIALS;
String line = x + "\t" + duration;
System.out.println(line);
FileUtils.writeStringToFile(file, line + "\r\n", true);
}
}

Well, it seems that I sorted the issue out on my own.
I was right about the idea that compiled code could take a while to kick in. The problem was a flaw in the way I actually implemented my benchmarking code:
if (x > 4 && x < 17)
Thread.sleep(1000);
in here I assumed that as the only "affected" area would be between 4 and 17, I could go on and just do a sleep over those values. This is simply not so. The following plot may be enlightening:
Here I am comparing the original no compilation function (red) to another no compilation function, but separed with sleeps in-between. As you may see, they work in different orders of magnitude, that meaning that mixing results of code with and without sleeps will yield unsound results, as I was guilty of doing.
The original question remains unaswered, yet. What causes the humps to ocurr even after compilation took place? Let's try to find that out, putting a 1s sleep in ALL points taken:
That yields the expected result. The odd humps were happening the native code still didn't kick in.
Comparing a sleep 50ms with a sleep 1000ms function yields yet again, the expected result:
(the gray one seems to still show a bit of delay)