A few weeks ago I wrote an exam. The first task was to find the right approximation of a function with the given properties: Properties of the function
I had to check every approximation with the tests i write for the properties. The properties 2, 3 and 4 were no problem. But I don't know how to check for property 1 using a JUnit test written in Java. My approach was to do it this way:
#Test
void test1() {
Double x = 0.01;
Double res = underTest.apply(x);
for(int i = 0; i < 100000; ++i) {
x = rnd(0, x);
Double lastRes = res;
res = underTest.apply(x);
assertTrue(res <= lastRes);
}
}
Where rnd(0, x) is a function call that generates a random number within (0,x].
But this can't be the right way, because it is only checking for the x's getting smaller, the result is smaller than the previous one. I.e. the test would also succeed if the first res is equal to -5 and the next a result little smaller than the previous for all 100000 iterations. So it could be that the result after the 100000 iteration is -5.5 (or something else). That means the test would also succeed for a function where the right limit to 0 is -6. Is there a way to check for property 1?
Is there a way to check for property 1?
Trivially? No, of course not. Calculus is all about what happens at infinity / at infinitely close to any particular point – where basic arithmetic gets stuck trying to divide a number that grows ever tinier by a range that is ever smaller, and basic arithmetic can't do '0 divided by 0'.
Computers are like basic arithmetic here, or at least, you are applying basic arithmetic in the code you have pasted, and making the computer do actual calculus is considerably more complicated than what you do here. As in, years of programming experience required more complicated; I very much doubt you were supposed to write a few hundred thousand lines of code to build an entire computerized math platform to write this test, surely.
More generally, you seem to be labouring under the notion that a test proves anything.
They do not. A test cannot prove correctness, ever. Just like science can never prove anything, they can only disprove (and can build up ever more solid foundations and confirmations which any rational person usually considers more than enough so that they will e.g. take as given that the Law of Gravity will hold, even if there is no proof of this and there never will be).
A test can only prove that you have a bug. It can never prove that you don't.
Therefore, anything you can do here is merely a shades of gray scenario: You can make this test catch more scenarios where you know the underTest algorithm is incorrect, but it is impossible to catch all of them.
What you have pasted is already useful: This test proves that as you use double values that get ever closer to 0, your test ensures that the output of the function grows ever smaller. That's worth something. You seem to think this particular shade of gray isn't dark enough for you. You want to show that it gets fairly close to infinity.
That's... your choice. You cannot PROVE that it gets to infinity, because computers are like basic arithmetic with some approximation sprinkled in.
There isn't a heck of a lot you can do here: Yes, you can test if the final res value is for example 'less than -1000000'. You can even test if it is literally negative infinity, but there is no guarantee that this is even correct; a function that is defined as 'as the input goes to 0 from the positive end, the output will tend towards negative infinity' is free to do so only for an input so incredibly tiny, that double cannot represent it at all (computers are not magical; doubles take 64 bit, that means there are at most 2^64 unique numbers that double can even represent. 2^64 is a very large number, but it is nothing compared to the doubly-dimensioned infinity that is the concept of 'all numbers imaginable' (there are an infinite amount of numbers between 0 and 1, and an infinite amount of such numbers across the whole number line, after all). Thus, there are plenty of very tiny numbers that double just cannot represent. At all.
For your own sanity, using randomness in a unit test is a bad idea, and for some definitions of the word 'unit test', literally broken (some labour under the notion that unit tests must be reliable or they cannot be considered a unit test, and this is not such a crazy notion if you look at what 'unit test' pragmatically speaking ends up being used for: To have test environments automatically run the unit tests, repeatedly and near continuously, in order to flag down ASAP when someone breaks one. If the CI server runs a unit test 1819 times a day, it would get quite annoying if by sheer random chance, one in 20k times it fails; it would then assume the most recent commit is to blame and no framework out there I know of repeats a unit test a few more times. In the end programming works best if you move away from the notion of proofs and cold hard definitions, and move towards 'do what the community thinks things mean'. For unit tests, that means: Don't use randomness).
Firstly: you cannot reliably test ANY of the properties
function f could break one of the properties in a point x which is not even representable as a double due to limited precision
there is too much points too test, realistically you need to pick a subset of the domain
Secondly:
your definition of a limit is wrong. You check that the function is monotonically decreasing. This is not required by limit definition - the function could fluctuate when approaching the limit. In the general case, I would probably follow Weierstrass definition
But:
By eyeballing the conditions you can quickly notice that a logarithmic function (with any base) meets the criteria. (so the function is indeed monotonically decreasing).
Let's pick natural logarithm, and check its value at smallest x that can be represented as a double:
System.out.println(Double.MIN_VALUE);
System.out.println(Math.log(Double.MIN_VALUE));
System.out.println(Math.log(Double.MIN_VALUE) < -1000);
// prints
4.9E-324
-744.4400719213812
false
As you can see, the value is circa -744, which is really far from minus infinity. You cannot get any closer on double represented on 64 bits.
Related
Considering this code which calculates a power of a double x:
public static double F1 (double x, int k){
if (k==1) { return x; } // O(1)
return x * F1(x, k-1); // O(k)
}
I have concluded that
the nr of operations in if (k==1) { return x; } : 2 operations, the if-statement and the return-statement. Thus, T(1) = 2
the nr of operations in return x * F1(x, k-1); : 4 operations, the return-statement = 1, the *-operator = 1, and F1(x, k-1); = 2. So the first part of the equation = 4
We have one recursive call in x * F1(x, k-1), so x = 1.
We reduce the problem by 1 in each recursive call, so y = k-1. So the second part of the equation = T(k-1)
Putting this all together, we get:
4 + T(k-1), T(1) = 2
But how do I proceed from here to find the exact runtime?
I tried to look at this question for an explanation, but it focused on how to calculate the Big-O notation, and not the exact time complexity. How do I proceed to find the exact time-complexity?
The answer here should be:
Exact: 4k-2
Tilde: 4k
Big-O: O(k)
But I don't know what they did to arrive at this.
But how do I proceed from here to find the exact runtime?
You toss everything you did so far in the garbage and fire up JMH instead, see later for more on that.
It is completely impossible to determine exact runtime based on such academic analysis. Exact runtime depends on which song is playing in your music player, whether your OS is busy doing some disk cleanup, sending a ping to the network time server, which pages so happen to be on the on-die caches, which CPU core your code ends up being run on, and the phase of the moon.
Let me say this as clear as I can: Something like 4k - 2 is utterly irrelevant and misguided - that's just not how computers work. You can't say that an algorithm with 'exact runtime' 4k - 2 will be faster than a 6k + 2 algorithm. It is equally likely to be slower: It holds zero predictive power. It's a completely pointless 'calculation'. It means nothing. There's a reason big-O notation exist: That does mean something regardless of hardware vagary: Given 2 algorithms such that one has a 'better' big-O notation than the other, then there exists some input size such that the better algorithm WILL be faster, regardless of hardware concerns. It might be a really big number and big-O does nothing whatsoever to tell you at what number this occurs.
The point of big-O notation is that it dictates with mathematical certainty what will eventually happen if you change the size of the input to your algorithm, in very broad strokes. It is why you remove all constants and everything but the largest factor when showing a big-O notation.
Take a graph; on the X-axis, there's 'input size', which is the 'k' in O(k). On the Y-axis, there's execution time (or if you prefer, max. memory load). Then, make up some input size and run your algorithm a few times. Average the result, and place a dot on that graph. For example, if you are running your algorithm on an input of k=5, and it takes 27ms on average, put a dot on x=5, y=27.
Keep going. Lots of dots. Eventually those dots form a graph. The graph will, near the x=0 point, be all over the place. As if a drunk with a penchant for randomness is tossing darts at a board.
But, eventually (and when 'eventually' kicks in is impossible to determine, as, again, it depends on so many OS things, don't bother attempting to predict such things), it'll start looking like a recognizable shape. We define these shapes in terms of simplistic formulas. For example, if it eventually (far enough to the right) coalesces into something that looks like what you'd get if you graph y=x^2, then we call that O(x^2).
Now, y=5x^2 looks exactly like y=x^2. For that matter, y=158*x^2 + 25000x + 2134931239, if you look far enough to the right on that curve, looks exactly like y=x^2. Hence why O(158x^2+20x) is completely missing the point, and therefore incorrect. The point of O is merely to tell you what it'll look like 'far enough to the right'.
This leaves us with precisely 2 useful performance metrics:
O(k) notation. Which you correctly determined here: This algorithm has an O(k) runtime.
A timing report. There is no point trying to figure this out by looking at the code, you need to run the code. Repeatedly, with all sorts of guards around it to ensure that hotspot optimization isn't eliminating your code completely, re-running lots of times to get a good average, and ensuring that we're past the JVM's JIT step. You use JMH to do this, and note that the result of JMH, naturally, depends on the hardware you run it on, and that's because programs can have wildly different performance characteristics depending on hardware.
For the first k-1 steps you execute:
the comparison k==1
the subtraction k-1
the product x * ...
the return instruction
In the last step you execute:
the comparison k==1
the return instruction
So you have 4*(k-1)+2 = 4k-2 overall instructions.
EDIT: As #rzwitserloot correctly pointed out, the quantity that you are searching for is not very significant, but it depends on how the code is compiled and executed. Above I've just tried to figure out what your teacher meant with "exact time-complexity".
I know that for primitive floating point types (floats and doubles), you're not supposed to compare them directly via ==. But what if you're using the wrapper class for double? Will something like
Double a = 5.05;
Double b = 5.05;
boolean test = a.equals(b);
compare the two values properly?
You need to fully understand the reason for why == comparison is a bad idea. Without understanding, you're just fumbling in the dark.
Let's talk about how computers (and doubles) work.
Imagine you enter a room; it has 3 lightswitches, otherwise it is bare. You will enter the room, you can fiddle with the switches, but then you have to leave. I enter the room later and can look at the switches.
How much information can you convey?
The answer is: You can convey 8 different states: DDD, DDU, DUD, DUU, UDD, UDU, UUD, and UUU. That's it.
Computers work exactly like this when they store a double. Except instead of 3 switches, you get 64 switches. That means 2^64 different states you can convey with a single double, and that's a ton of states: That's a 19 digit number.
But it's still a finite amount of states, and that's problematic: There are an infinite amount of numbers between 0 and 1. Let alone between -infinity and +infinity, which double dares to cover. How do you store one of an infinite infinity of choices when you only get to represent 2^64 states? That 19-digit number starts to look pretty small when it's tasked to differentiate from an infinite infinity of possibilities, doesn't it?
The answer, of course, is that this is completely impossible.
So doubles don't actually work like that. Instead, someone took some effort and hung up a gigantic numerline, from minus infinite to plus infinite, in a big room, and threw 2^64 darts at this line. The numbers they landed on are the 'blessed numbers' - these are representable by a double value. That does mean there are an infinite amount of numbers between any 2 darts that therefore are not representable. The darts aren't quite random: The closer you are to 0, the denser the darts. Once you get beyond about 2^52 or so, the distance between 2 darts exceeds 1.0 even.
Here's a trivial example of a non-representable number: 0.3. Amazing, isn't it? Something that simple. It means a computer literally cannot calculate 0.1 + 0.2 using double. So what happens when you try? The rules state that the result of any calculation is always silently rounded to the nearest blessed number.
And therein lies the rub: You can run the math:
double x = 0.1 + 0.2;
and later do:
double y = 0.9 - 0.8 + 0.15 + 0.05;
and us humans would immediately notice that x and y are naturally identical. But not so for computers - because of that silent rounding to the nearest blessed number, it's possible that x is 0.29999999999999999785, and y is 0.300000000000000000012.
Thus we get to four crucial aspects when using double (or float which is just about worse in every fashion, don't ever use those):
If you need absolute precision, don't use them at all.
When printing them, always round them down. System.out.println does this out of the box, but you should really use .printf("%.5f") or similar: Pick the # of digits you need.
Be aware that the errors will compound, and it gets worse as you are further away from 1.0.
Do not ever compare with ==, instead always use a delta-compare: The notion of "lets consider the 2 numbers equal if they are within 0.0000000001 of each other".
There is no universal magic delta value; it depends on your precision needs, how far you away from 1.0, etc. Therefore, just asking the computer: Hey, just figure this stuff out I just wanna know if these 2 doubles are equal is impossible. The only definition available that doesn't require your input as to 'how close' they can be, is the notion of sheer perfection: They are equal only if they are precisely identical. This definition would fail you in that trivial example above. It makes not one iota of difference if you use Double.equals instead of double == double, or any other utility class for that matter.
So, no, Double.equals is not suitable. You will have to compare Math.abs(d1 - d2) < epsilon, where epsilon is your choice. Mostly, if equality matters at all you're already doing it wrong and shouldn't be using double in the first place.
NB: When representing money, you don't want unpredictable rounding, so never use doubles for these. Instead figure out what the atomic banking unit is (dollarcents, eurocents, yen, satoshis for bitcoin, etc), and store that as a long. You store $4.52 and long x = 452;, not as double x = 4.52;.
I am writing a basic neural network in Java and I am writing the activation functions (currently I have just written the sigmoid function). I am trying to use doubles (as apposed to BigDecimal) with hopes that training will actually take a reasonable amount of time. However, I've noticed that the function doesn't work with larger inputs. Currently my function is:
public static double sigmoid(double t){
return (1 / (1 + Math.pow(Math.E, -t)));
}
This function returns pretty precise values all the way down to when t = -100, but when t >= 37 the function returns 1.0. In a typical neural network when the input is normalized is this fine? Will a neuron ever get inputs summing over ~37? If the size of the sum of inputs fed into the activation function vary from NN to NN, what are some of the factors the affect it? Also, is there any way to make this function more precise? Is there an alternative that is more precise and/or faster?
Yes, in a normalized network double is fine to use. But this depend on your input, if your input layer is bigger, your input sum will be bigger of course.
I have encountered the same problem using C++, after t become big, the compiler/rte does not even take into account E^-t and returns plain 1, as it only calculates the 1/1 part. I tried to divide the already normalized input by 1000-1000000 and it worked sometimes, but sometimes it did not as I was using a randomized input for the first epoch and my input layer was a matrix 784x784. Nevertheless, if your input layer is small, and your input is normalized this will help you
The surprising answer is that double is actually more precision than you need. This blog article by Pete Warden claims that even 8 bits are enough precision. And not just an academic idea: NVidia's new Pascal chips emphasize their single-precision performance above everything else, because that is what matters for deep learning training.
You should be normalizing your input neuron values. If extreme values still happen, it is fine to set them to -1 or +1. In fact, this answer shows doing that explicitly. (Other answers on that question are also interesting - the suggestion to just pre-calculate 100 or so values, and not use Math.exp() or Math.pow() at all!)
I have a situation where I might need to apply a multiplier to a value in order to get the correct results. This involves computing the value using floating point division.
I'm thinking it would be a good idea to check the values before I perform floating point logic on them to save processor time, however I'm not sure how efficient it will be at run-time either way.
I'm assuming that the if check is 1 or 2 instructions (been a while since assembly class), and that the floating point operation is going to be many more than that.
//Check
if (a != 10) { //1 or 2 instructions?
b *= (float) a / 10; //Many instructions?
}
Value a is going to be '10' most of the time, however there are a few instances where it wont be. Is the floating point division going to take very many cycles even if a is equal to the divisor?
Will the previous code with the if statement execute more efficiently than simply the next one without?
//Don't check
b *= (float) a / 10; //Many instructions?
Granted there wont be any noticable difference either way, however I'm curious as to the behavior of the floating point multiplication when the divisor is equal to the dividend in case things get processor heavy.
Assuming this is in some incredibly tight loop, executed billions of times, so the difference of 1-2 instructions matters, since otherwise you should probably not bother --
Yes you are right to weigh the cost of the additional check each time, versus the savings when the check is true. But my guess is that it has to be true a lot to overcome not only the extra overhead, but the fact that you're introducing a branch, which will ultimately do more to slow you down via a pipeline stall in the CPU in the JIT-compiled code than you'll gain otherwise.
If a == 10 a whole lot, I'd imagine there's a better and faster way to take advantage of that somehow, earlier in the code.
IIRC, floating-point multiplication is much less expensive than division, so this might be faster than both:
b *= (a * 0.1);
If you do end up needing to optimize this code I would recommend using Caliper to do micro benchmarks of your inner loops. It's very hard to predict accurately what sort of effect these small modifications will have. Especially in Java where how the VM behaves is bit of an unknown since in theory it can optimize the code on the fly. Best to try several strategies and see what works.
http://code.google.com/p/caliper/
Ok so I'm trying to use Apache Commons Math library to compute a double integral, but they are both from negative infinity (to around 1) and it's taking ages to compute. Are there any other ways of doing such operations in java? Or should it run "faster" (I mean I could actually see the result some day before I die) and I'm doing something wrong?
EDIT: Ok, thanks for the answers. As for what I've been trying to compute it's the Gaussian Copula:
So we have a standard bivariate normal cumulative distribution function which takes as arguments two inverse standard normal cumulative distribution functions and I need integers to compute that (I know there's a Apache Commons Math function for standard normal cumulative distribution but I failed to find the inverse and bivariate versions).
EDIT2: as my friend once said "ahhh yes the beauty of Java, no matter what you want to do, someone has already done it" I found everything I needed here http://www.iro.umontreal.ca/~simardr/ssj/ very nice library for probability etc.
There are two problems with infinite integrals: convergence and value-of-convergence. That is, does the integral even converge? If so, to what value does it converge? There are integrals which are guaranteed to converge, but whose value it is not possible to determine exactly (try the integral from 1 to infinity of e^(-x^2)). If it can't be exactly returned, then an exact answer is not possible mathematically, which leaves only approximation. Apache Commons uses several different approximation schemes, but all require the use of finite bounds for correctness.
The best way to get an appropriate answer is to repeatedly evaluate finite integrals, with ever increasing bounds, and compare the results. In pseudo-code, it would look something like this:
double DELTA = 10^-6//your error threshold here
double STEP_SIZE = 10.0;
double oldValue=Double.MAX_VALUE;
double newValue=oldValue;
double lowerBound=-10; //or whatever you want to start with--for (-infinity,1), I'd
//start with something like -10
double upperBound=1;
do{
oldValue = newValue;
lowerBound-= STEP_SIZE;
newValue = integrate(lowerBound,upperBound); //perform your integration methods here
}while(Math.abs(newValue-oldValue)>DELTA);
Eventually, if the integral converges, then you will get enough of the important stuff in that widening the bounds further will not produce meaningful information.
A word to the wise though: this kind of thing can be explosively bad if the integral doesn't converge. In that case, one of two situations can occur: Either your termination condition is never satisfied and you fall into an infinite loop, or the value of the integral oscillates indefinitely around a value, which may cause your termination condition to be incorrectly satisfied (giving incorrect results).
To avoid the first, the best way is to put in some maximum number of steps to take before returning--doing this should stop the potentially infinite loop that can result.
To avoid the second, hope it doesn't happen or prove that the integral must converge (three cheers for Calculus 2, anyone? ;-)).
To answer your question formally, no, there are no other such ways to perform your computation in java. In fact, there are no guaranteed ways of doing it in any language, with any algorithm--the mathematics just don't work out the way we want them to. However, in practice, a lot (though by no means all!) of the practical integrals do converge; its been my experience that only about ~20 iterations will give you an approximation of reasonable accuracy, and Apache should be fast enough to handle that without taking absurdly long.
Suppose you are integrating f(x) over -infinity to 1, then substitute x = 2 - 1/(1-t), and evaluate over the range 0 .. 1. Note check a maths text for how to do the substition, I'm a little rusty and its too late here.
The result of a numerical integration where one of the bounds is infinity has a good chance to be infinity as well. And it will take infinite time to prove it ;)
So you either find an equivalent formula (using real math) that can be computed or your replace the lower bound with a reasonable big negative value and look, if you can get a good estimation for the integral.
If Apache Commons Math could do numerical integration for integrals with infinite bounds in finite time, they wouldn't give it away for free ;-)
Maybe it's your algorithm.
If you're doing something naive like Simpson's rule it's likely to take a very long time.
If you're using Gaussian or log quadrature you might have better luck.
What's the function you're trying to integrate, and what's the algorithm you're using?