I'm using an implementation of this answer to calculate the distance between two points for my GPS tracking software.
However I've read on numerous StackOverFlow that using double is bad because of the inaccurate float representation of values. Therefore, I'm asking :
Would using BigDecimal to calculate distances be more accurate ?
Would converting the result from BigDecimal back to double produce additional inaccuracies ? (to store or retrieve the value)
No, using BigDecimal instead of double will not significantly improve the accuracy of this algorithm.
The main source of error is a modeling error: you're using a formula that assumes the Earth is a perfect sphere. This means the result can be wrong by as much as 0.5%. The rounding error you get from using double is on the order of 0.000000000001%.
To get more accurate results, you could use for example Vincenty's formulae, which assumes the Earth is an ellipsoid, but it is harder to implement than the great circle distance. To get even more accurate results, you should take into account the local topography, which is even harder.
For further reading: https://en.m.wikipedia.org/wiki/Geographical_distance
Yes, in theory doing any calculations in BigDecimal could lead to better precisions, but that is very unlikely to actually help for several reasons:
as you found out there are non-naive algorithms to calculate the distance using only double without an excessive numerical error.
the precision of double values for this use case is many orders of magnitude higher than the precision and accuracy of any positional information you might start out with. That means that the measurement error will definitely be bigger than the numerical error caused by using double.
And if you did your calculations in BigDecimal anyway and then converted the result to double, then you'd lose precision, of course, since BigDecimal can be effectively arbitrarily precise, but double can't.
But again: that loss of precision is irrelevant for the use case you describe.
Related
I know float type is A IEEE floating point, and it's not accuracy in calculation, for example, if I'd like to sum two floats 8.4 and 2.4, what I get is 10.7999999 rather than 10.8. I also know BigDecimal can solve this problem, but BigDecimal is much slower than float type.
In most real productions we'd like an accuracy value like above 10.8 not a 10.7999.. so my question is shall I prevent to use float as much as I can in programming? if not is there any use cases? I mean in a real production.
If you're handling monetary amounts, then numbers like 8.4 and 2.4 are exact values, and you'll want to use BigDecimal for those. However, if you're doing a physics calculation where you're dealing with measurements, the values 8.4 and 2.4 aren't going to be exact anyway, since measurements aren't exact. That's a use case where using double is better. Also, a scientific calculation could involve things like square roots, trigonometric functions, logarithms, etc., and those can be done only using IEEE floats. Calculations involving money don't normally involve those kinds of functions.
By the way, there's very little reason to ever use the float type; stick with double.
You use float when the percision is enough. It is generally faster to do calculations with float and requires less memory. Sometimes you just need the performance.
What you describe is caused by the fact that binary floating point numbers cannot exactly represent many numbers that can be exactly represented by decimal floating point numbers, like 8.4 or 2.4.
This affects not only the float type in Java but also double.
In many cases you can do calculations with integers and then rescale to get the deciamls correctly. But if you require numbers with equal relative accurracies, no matter how large they are, floating point is far superior.
So yes, if you can, you should prefer integers over floats, but there are many applications where floating point is required. This includes many scientific and mathematical algorithms.
You should also consider that 10.7999999 instead of 10.8 looks weird when displayed but actually the difference is really small. So it's not so much an accurracy issue but more related to number formatting. In most cases this problem is resolved by rounding the number appropriately when converting it to a string for output, for example:
String price = String.format("%.2f", floatPrice);
BigDecimals are very precise (you can determine their precision -- it is mainly limited by memory) but pretty slow and memory intensive. You use them when you need exact results, i.e. in financial applications, or when you otherwise need very precise results and when speed is not too critical.
Floating point types (double and float) are not nearly as precise, but much faster and they only take up limited memory. Typically, a float takes up 4 bytes and a double takes up 8 bytes. You use them with measurements that can't be very exact anyway, but also if you need the speed or the memory. I use them for (real time) graphics and real time music. Or when otherwise precision of the result is not so important, e.g. when measuring time or percentages when downloading or some such.
I have a Java project that deals with a lot money values and the project mainly involves:
reading the data from database,
calculations (process data)
showing to users (no inserts or updates in database are required).
I need precision for only some of the money values and not for all. So here I can do:
using doubles when precision not required or
using BigDecimals for ALL.
I want to know if there will be any performance issues if I use BigDecimal for all the variables? Can I save execution time if I opt for choice 1?
Which way is best? (I am using java 6)
Don't use double for money Why not use Double or Float to represent currency?
Using Big Decimal is 100x slower than the built in primitives and you can't use + and -, / and * with BigDecimal but must use the equivalent BigDecimal method calls.
An alternative is to use int instead of double where you are counting cents or whatever fractional currency equivalent and then when formatting the output to the user, do the appropriate conversions back to show the values the way the user expects.
If you have really large values, you can use long instead of int
It's a trade-off.
With BigDecimal you are working with immutable objects. This means that each operation will cause the creation of new objects and this, for sure, will have some impact on the memory. How much - it depends on a lot of things - execution environment, number and complexity of the calculations, etc. But you are getting precision, which is the most important thing when working with money.
With double you can use primitive values, but the precision is poor and they are not suitable for money calculation at all.
If I had to suggest a way - I would say for sure use BigDecimal when dealing with money.
Have you considered moving some of the calculation logic to the DB layer? This can save you a lot in terms of memory and performance, and you will still keep the precision requirement in tact.
BigDecimal and double are very different types, with very different purposes. Java benefits from having both, and Java programmers should be using both of them appropriately.
The floating point primitives are based on binary to be both space and time efficient. They can be, and typically are, implemented in very fast hardware. double should be used in contexts in which there is nothing special about terminating decimal fractions, and all that is needed is an extremely close approximation to a value that may be fractional, irrational, very big, or very small. There are good implementations of many trig and similar functions for double. See java.lang.Math for some examples.
BigDecimal can represent any terminating decimal fraction exactly, given enough memory. That is very, very good in situations in which terminating decimal fractions have special status, such as many money calculations.
In addition to exact representation of terminating decimal fractions, it could also be used to get a closer approximation to e.g. one third than is possible with double. However, situations in which you need an approximation that is closer than double supplies are very rare. The closest double to one third is 0.333333333333333314829616256247390992939472198486328125, which is close enough for most practical purposes. Try measuring the difference between one third of an inch and 0.3333333333333333 inches.
BigDecimal is a only supported in software, does not have the support for mathematical functions that double has, and is much less space and time efficient.
If you have a job for which either would work functionally, use double. If you need exact representation of terminating decimal fractions, use BigDecimal.
The Java™ Tutorials state that "this data type [double] should never be used for precise values, such as currency." Is the fact that an ORM / DSL is returning floating point numbers for database columns storing values to be used to calculate monetary amounts a problem? I'm using QueryDSL and I'm dealing with money. QueryDSL is returning a Double for any number with a precision up to 16 and a BigDecimal thereafter. This concerns me as I'm aware that floating point arithmetic isn't suitable for currency calculations.
From this QueryDSL issue I'm led to believe that Hibernate does the same thing; see OracleDialect. Why does it use a Double rather than a BigDecimal? Is it safe to retrieve the Double and construct a BigDecimal, or is there a chance that a number with a precision of less than 16 could be incorrectly represented? Is it only when performing arithmetic operations that a Double can have floating-point issues, or are there values to which it cannot be accurately initialised?
Using floating point numbers for storing money is a bad idea indeed. Floating points can approximate an operation result, but that's not what you want when dealing with money.
The easiest way to fix it, in a database portable way, is to simply store cents. This is the proffered way of dealing with currency operations in financial operations. Pay attention that most databases use the half-away from zero rounding algorithm, so make sure that's appropriate in your context.
When it comes to money you should always ask a local accountant, especially for the rounding part. Better safe then sorry.
Now back to your questions:
Is it safe to retrieve the Double and construct a BigDecimal, or is
there a chance that a number with a precision of less than 16 could be
incorrectly represented?
This is a safe operation as long as your database uses at most a 16 digit precision. If it uses a higher precision, you'd need to override the OracleDialect and
Is it only when performing arithmetic operations that a Double can
have floating-point issues, or are there values to which it cannot be
accurately initialised?
When performing arithmetic operations you must always take into consideration the monetary rounding anyway, and that applies to BigDecimal as well. So if you can guarantee that the database value doesn't loose any decimal when being cast to a java Double, you are fine to create a BigDecimal from it. Using BigDecimal pays off when applying arithmetic operations to the database loaded value.
As for the threshold of 16, according to Wiki:
The 11 bit width of the exponent allows the representation of numbers
with a decimal exponent between 10−308 and 10308, with full 15–17
decimal digits precision. By compromising precision, subnormal
representation allows values smaller than 10−323.
There seems to be several concerns mentioned in the question, comments, and answers by Robert Bain. I've collected and paraphrased some of these.
Is it safe to use a double to store a precise value?
Yes, provided the number of significant-digits (precision) is small enough.
From wikipedia
If a decimal string with at most 15 significant digits is converted to IEEE 754 double precision representation and then converted back to a string with the same number of significant digits, then the final string should match the original.
But new BigDecimal(1000.1d) has the value 1000.1000000000000227373675443232059478759765625, why not 1000.1?
In the quote above I added emphasis - when converted from a double the number of significant digits must be specified, e.g.
new BigDecimal(1000.1d, new MathContext(15))
Is it safe to use a double for arbitrary arithmetic on precise values?
No, each intermediate value used in the calculation could introduce additional error.
Using a double to store exact values should be seen as an optimization. It introduces risk that if care is not taken, precision could be lost. Using a BigDecimal is much less likely to have unexpected consequences and should be your default choice.
Is it correct that QueryDSL returns a double for precise value?
It is not necessarily incorrect, but is probably not desirable. I would suggest you engage with the QueryDSL developers... but I see you have already raised an issue and they intend to change this behavior.
After much deliberation, I must conclude that the answer to my own question:
Is the fact that an ORM / DSL is returning floating point numbers for database columns storing values to be used to calculate monetary amounts a problem?
put simply, is yes. Please read on.
Is it safe to retrieve the Double and construct a BigDecimal, or is there a chance that a number with a precision of less than 16 could be incorrectly represented?
A number with a precision of less than 16 decimal digits is incorrectly represented in the following example.
BigDecimal foo = new BigDecimal(1000.1d);
The BigDecimal value of foo is 1000.1000000000000227373675443232059478759765625. 1000.1 has a precision of 1 and is being misrepresented from precision 14 of the BigDecimal value.
Is it only when performing arithmetic operations that a Double can have floating-point issues, or are there values to which it cannot be accurately initialised?
As per the example above, there are values to which it cannot be accurately initialised. As The Java™ Tutorials clearly states, "This data type [float / double] should never be used for precise values, such as currency. For that, you will need to use the java.math.BigDecimal class instead."
Interestingly, calling BigDecimal.valueOf(someDouble) appeared at first to magically resolve things but upon realising that it calls Double.toString() then reading Double's documentation it became apparent that this is not appropriate for exact values either.
In conclusion, when dealing with exact values, floating point numbers are never appropriate. As such, in my mind, ORMs / DSLs should be mapping to BigDecimal unless otherwise specified, given that most database use will involve the calculation of exact values.
Update:
Based on this conclusion, I've raised this issue with QueryDSL.
It is not only about arithmetic operations, but also about pure read&write.
Oracle NUMBER and BigDecimal do both use decadic base. So when you read number from database and then you store it back you can be sure, that the same number was written. (Unless it exceeds Oracle's limit of 38 digits).
If you convert NUMBER into binary base (Double) and then you convert it back do decadic then you might expect problems. And also this operation must be much slower.
I am trying to establish some concise overview of what options for precise caluclations we have in JAVA+SQL. So far I have found following options:
use doubles accepting their drawbacks, no go.
use BigDecimals
using them in complicated formulas is problematic for me
use String.format/Decimal.format to round doubles
do i need to round each variable in formula or just result to get BigDecimal precision?
how can this be tweaked?
use computed fields option in SQL.
drawback is that I'd need dynamic SQL to pull data from different tables + calculate fields on other calculated fields and that would get messy
any other options?
Problem statement:
I need precise financial calculations that would involve using very big (billions) and very small numbers (0.0000004321), and also dividing values that are very similar to each other, so for sure I need precision of BigDecimal.
On the other side, I want to retain ease of use that doubles have in functions (i work on arrays from decimal SQL data), so calculations like: (a[i] - b[i])/b[i] etc. etc. that are further used in other calculations. and I'd like to have users to be able to desing their own formulas as they need them (using common math statements)
i am keen to use "formatting" solution for String.format, but this makes code not very readable ( using String.format() for each variable...).
Many thanks for suggestion of how to deal with the stuff.
There is nothing you can do to avoid floating point erros in float and double.
No free cheese here - use BigDecimal.
From Effective Java (2nd ED):
Item 48: Avoid float and double if exact answers are required
Float and double do not provide exact results and should not be used where exact results are required.
The float and double types are particularly ill-suited for monetary claculations because is impossible to represent 0.1 (or any other negative power of ten) as a float or double exactly.
The right way to solve this problem is to ouse BigDecimal, int, or long for monetary calculations.
...
An alternative is to use int or long and to keep track of the decimal point yourself.
There is no way to get BigDecimal precision on a double. doubles have double precision.
If you want to guarantee precise results use BigDecimal.
You could create your own variant using a long to store the integer part and an int to store the fractional part - but why reinvent the wheel.
Any time use doubles you stand to stuffer from double precision issues. If you use them in a single place you might as well use them everywhere.
Even if you only use them to represent data from the database then will round the data to double precision and you will lose information.
If I understand your question, you want to use Data Types with more precision than the native Java ones without loosing the simple mathematical syntax (e.g. / + * - and so on). As you cannot overload operators in Java, I think this is not possible.
Is there a generally accepted way to get Euler's number in BigDecimal?
I basically want to do Math.exp(double a) and I basically want the same function but calculated using BigDecimal to prevent precision loss with other calculations.
I thought of just doing,
BigDecimal.valueOf(Math.E).pow(BigDecimal.SOMEOTHERNUMBER)
But not sure if that's the right way to approach the problem.
The problem is that Euler's number is irrational; in BigDecimal, you'd have to have infinite memory to hold it. The reason there's no previously-stored value is that however many decimal places you needed, you wouldn't have the right number for your current application.
You're going to have to determine how many decimals you want, then construct the BigDecimal representation using the String constructor (new BigDecimal("2.71828...")).
Well, Math.E is a double. If you want a higher precision representation of Euler's number, you can create it with some arbitrary precision of decimal places, or there are BigDecimal math libraries with a really high precision version of E.
You could approximate one yourself with something like this: Better approximation of e with Java
While the double Math.E should be accurate enough for most calculations, there is a way to calculate e to greater accuracy.
If you want to get e that has more accuracy that double can contain, you calculate it by adding up the sum of 1/0!, 1/1!, 1/2!, 1/3!...
Where ! means factorial. Just add up the reciprocal of all the factorials starting from zero. The more terms you add up, the more precise you will get. Of course since e is irrational, you can never get the real value, but you can get pretty darn close.