Develop Reference

Is it sufficient to convert a double to a BigDecimal just before addition to retain original precision?

We are solving a numeric precision related bug. Our system collects some numbers and spits their sum.
The issue is that the system does not retain the numeric precision, e.g. 300.7 + 400.9 = 701.599..., while expected result would be 701.6. The precision is supposed to adapt to the input values so we cannot just round results to fixed precision.
The problem is obvious, we use double for the values and addition accumulates the error from the binary representation of the decimal value.
The path of the data is following:
XML file, type xsd:decimal
Parse into a java primitive double. Its 15 decimal places should be enough, we expect values no longer than 10 digits total, 5 fraction digits.
Store into DB MySql 5.5, type double
Load via Hibernate into a JPA entity, i.e. still primitive double
Sum bunch of these values
Print the sum into another XML file
Now, I assume the optimal solution would be converting everything to a decimal format. Unsurprisingly, there is a pressure to go with the cheapest solution. It turns out that converting doubles to BigDecimal just before adding a couple of numbers works in case B in following example:
import java.math.BigDecimal;
public class Arithmetic {
public static void main(String[] args) {
double a = 0.3;
double b = -0.2;
// A
System.out.println(a + b);//0.09999999999999998
// B
System.out.println(BigDecimal.valueOf(a).add(BigDecimal.valueOf(b)));//0.1
// C
System.out.println(new BigDecimal(a).add(new BigDecimal(b)));//0.099999999999999977795539507496869191527366638183593750
}
}
More about this:
Why do we need to convert the double into a string, before we can convert it into a BigDecimal?
Unpredictability of the BigDecimal(double) constructor
I am worried that such a workaround would be a ticking bomb.
First, I am not so sure that this arithmetic is bullet proof for all cases.
Second, there is still some risk that someone in the future might implement some changes and change B to C, because this pitfall is far from obvious and even a unit test may fail to reveal the bug.
I would be willing to live with the second point but the question is: Would this workaround provide correct results? Could there be a case where somehow
Double.valueOf("12345.12345").toString().equals("12345.12345")
is false? Given that Double.toString, according to javadoc, prints just the digits needed to uniquely represent underlying double value, so when parsed again, it gives the same double value? Isn't that sufficient for this use case where I only need to add the numbers and print the sum with this magical Double.toString(Double d) method? To be clear, I do prefer what I consider the clean solution, using BigDecimal everywhere, but I am kind of short of arguments to sell it, by which I mean ideally an example where conversion to BigDecimal before addition fails to do the job described above.

If you can't avoid parsing into primitive double or store as double, you should convert to BigDecimal as early as possible.
double can't exactly represent decimal fractions. The value in double x = 7.3; will never be exactly 7.3, but something very very close to it, with a difference visible from the 16th digit or so on to the right (giving 50 decimal places or so). Don't be mislead by the fact that printing might give exactly "7.3", as printing already does some kind of rounding and doesn't show the number exactly.
If you do lots of computations with double numbers, the tiny differences will eventually sum up until they exceed your tolerance. So using doubles in computations where decimal fractions are needed, is indeed a ticking bomb.
[...] we expect values no longer than 10 digits total, 5 fraction digits.
I read that assertion to mean that all numbers you deal with, are to be exact multiples of 0.00001, without any further digits. You can convert doubles to such BigDecimals with
new BigDecimal.valueOf(Math.round(doubleVal * 100000), 5)
This will give you an exact representation of a number with 5 decimal fraction digits, the 5-fraction-digits one that's closest to the input doubleVal. This way you correct for the tiny differences between the doubleVal and the decimal number that you originally meant.
If you'd simply use BigDecimal.valueOf(double val), you'd go through the string representation of the double you're using, which can't guarantee that it's what you want. It depends on a rounding process inside the Double class which tries to represent the double-approximation of 7.3 (being maybe 7.30000000000000123456789123456789125) with the most plausible number of decimal digits. It happens to result in "7.3" (and, kudos to the developers, quite often matches the "expected" string) and not "7.300000000000001" or "7.3000000000000012" which both seem equally plausible to me.
That's why I recommend not to rely on that rounding, but to do the rounding yourself by decimal shifting 5 places, then rounding to the nearest long, and constructing a BigDecimal scaled back by 5 decimal places. This guarantees that you get an exact value with (at most) 5 fractional decimal places.
Then do your computations with the BigDecimals (using the appropriate MathContext for rounding, if necessary).
When you finally have to store the number as a double, use BigDecimal.doubleValue(). The resulting double will be close enough to the decimal that the above-mentioned conversion will surely give you the same BigDecimal that you had before (unless you have really huge numbers like 10 digits before the decimal point - the you're lost with double anyway).
P.S. Be sure to use BigDecimal only if decimal fractions are relevant to you - there were times when the British Shilling currency consisted of twelve Pence. Representing fractional Pounds as BigDecimal would give a disaster much worse than using doubles.

It depends on the Database you are using. If you are using SQL Server you can use data type as numeric(12, 8) where 12 represent numeric value and 8 represents precision. similarly, for my SQL DECIMAL(5,2) you can use.
You won't lose any precision value if you use the above-mentioned datatype.
Java Hibernate Class :
You can define
private double latitude;
Database:

Floating point types returned in ORM / DSL

The Java™ Tutorials state that "this data type [double] should never be used for precise values, such as currency." Is the fact that an ORM / DSL is returning floating point numbers for database columns storing values to be used to calculate monetary amounts a problem? I'm using QueryDSL and I'm dealing with money. QueryDSL is returning a Double for any number with a precision up to 16 and a BigDecimal thereafter. This concerns me as I'm aware that floating point arithmetic isn't suitable for currency calculations.
From this QueryDSL issue I'm led to believe that Hibernate does the same thing; see OracleDialect. Why does it use a Double rather than a BigDecimal? Is it safe to retrieve the Double and construct a BigDecimal, or is there a chance that a number with a precision of less than 16 could be incorrectly represented? Is it only when performing arithmetic operations that a Double can have floating-point issues, or are there values to which it cannot be accurately initialised?

Using floating point numbers for storing money is a bad idea indeed. Floating points can approximate an operation result, but that's not what you want when dealing with money.
The easiest way to fix it, in a database portable way, is to simply store cents. This is the proffered way of dealing with currency operations in financial operations. Pay attention that most databases use the half-away from zero rounding algorithm, so make sure that's appropriate in your context.
When it comes to money you should always ask a local accountant, especially for the rounding part. Better safe then sorry.
Now back to your questions:
Is it safe to retrieve the Double and construct a BigDecimal, or is
there a chance that a number with a precision of less than 16 could be
incorrectly represented?
This is a safe operation as long as your database uses at most a 16 digit precision. If it uses a higher precision, you'd need to override the OracleDialect and
Is it only when performing arithmetic operations that a Double can
have floating-point issues, or are there values to which it cannot be
accurately initialised?
When performing arithmetic operations you must always take into consideration the monetary rounding anyway, and that applies to BigDecimal as well. So if you can guarantee that the database value doesn't loose any decimal when being cast to a java Double, you are fine to create a BigDecimal from it. Using BigDecimal pays off when applying arithmetic operations to the database loaded value.
As for the threshold of 16, according to Wiki:
The 11 bit width of the exponent allows the representation of numbers
with a decimal exponent between 10−308 and 10308, with full 15–17
decimal digits precision. By compromising precision, subnormal
representation allows values smaller than 10−323.

There seems to be several concerns mentioned in the question, comments, and answers by Robert Bain. I've collected and paraphrased some of these.
Is it safe to use a double to store a precise value?
Yes, provided the number of significant-digits (precision) is small enough.
From wikipedia
If a decimal string with at most 15 significant digits is converted to IEEE 754 double precision representation and then converted back to a string with the same number of significant digits, then the final string should match the original.
But new BigDecimal(1000.1d) has the value 1000.1000000000000227373675443232059478759765625, why not 1000.1?
In the quote above I added emphasis - when converted from a double the number of significant digits must be specified, e.g.
new BigDecimal(1000.1d, new MathContext(15))
Is it safe to use a double for arbitrary arithmetic on precise values?
No, each intermediate value used in the calculation could introduce additional error.
Using a double to store exact values should be seen as an optimization. It introduces risk that if care is not taken, precision could be lost. Using a BigDecimal is much less likely to have unexpected consequences and should be your default choice.
Is it correct that QueryDSL returns a double for precise value?
It is not necessarily incorrect, but is probably not desirable. I would suggest you engage with the QueryDSL developers... but I see you have already raised an issue and they intend to change this behavior.

After much deliberation, I must conclude that the answer to my own question:
Is the fact that an ORM / DSL is returning floating point numbers for database columns storing values to be used to calculate monetary amounts a problem?
put simply, is yes. Please read on.
Is it safe to retrieve the Double and construct a BigDecimal, or is there a chance that a number with a precision of less than 16 could be incorrectly represented?
A number with a precision of less than 16 decimal digits is incorrectly represented in the following example.
BigDecimal foo = new BigDecimal(1000.1d);
The BigDecimal value of foo is 1000.1000000000000227373675443232059478759765625. 1000.1 has a precision of 1 and is being misrepresented from precision 14 of the BigDecimal value.
Is it only when performing arithmetic operations that a Double can have floating-point issues, or are there values to which it cannot be accurately initialised?
As per the example above, there are values to which it cannot be accurately initialised. As The Java™ Tutorials clearly states, "This data type [float / double] should never be used for precise values, such as currency. For that, you will need to use the java.math.BigDecimal class instead."
Interestingly, calling BigDecimal.valueOf(someDouble) appeared at first to magically resolve things but upon realising that it calls Double.toString() then reading Double's documentation it became apparent that this is not appropriate for exact values either.
In conclusion, when dealing with exact values, floating point numbers are never appropriate. As such, in my mind, ORMs / DSLs should be mapping to BigDecimal unless otherwise specified, given that most database use will involve the calculation of exact values.
Update:
Based on this conclusion, I've raised this issue with QueryDSL.

It is not only about arithmetic operations, but also about pure read&write.
Oracle NUMBER and BigDecimal do both use decadic base. So when you read number from database and then you store it back you can be sure, that the same number was written. (Unless it exceeds Oracle's limit of 38 digits).
If you convert NUMBER into binary base (Double) and then you convert it back do decadic then you might expect problems. And also this operation must be much slower.

Is BigDecimal an overkill in some cases?

I'm working with money so I need my results to be accurate but I only need a precision of 2 decimal points (cents). Is BigDecimal needed to guarantee results of multiplication/division are accurate?

BigDecimal is a very appropriate type for decimal fraction arithmetic with a known number of digits after the decimal point. You can use an integer type and keep track of the multiplier yourself, but that involves doing in your code work that could be automated.
As well as managing the digits after the decimal point, BigDecimal will also expand the number of stored digits as needed - many business and government financial calculations involve sums too large to store in cents in an int.
I would consider avoiding it only if you need to store a very large array of amounts of money, and are short of memory.

One common option is to do all your calculation with integer or long(the cents value) and then simply add two decimal places when you need to display it.
Similarly, there is a JODA Money library that will give you a more full-featured API for money calculations.

It depends on your application. One reason to use that level of accuracy is to prevent errors accumulated over many operations from percolating up and causing loss of valuable information. If you're creating a casual application and/or are only using it for, say, data entry, BigDecimal is very likely overkill.

+1 for Patricias answer, but I very strongly discourage anyone to implement own classes with an integer datatype with fixed bitlength as long as someone really do not know what you are doing. BigDecimal supports all rounding and precision issues while a long/int has severe problems:
Unknown number of fraction digits: Trade exchanges/Law/Commerce are varying in their amount
of fractional digits, so you do not know if your chosen number of digits must be changed and
adjusted in the future. Worse: There are some things like stock evaluation which need a ridiculous amount of fractional digits. A ship with 1000 metric tons of coal causes e.g.
4,12 € costs of ice, leading to 0,000412 €/ton.
Unimplemented operations: It means that people are likely to use floating-point for
rounding/division or other arithmetic operations, hiding the inexactness and leading to
all the known problems of floating-point arithmetic.
Overflow/Underflow: After reaching the maximum amount, adding an amount results in changing the sign. Long.MAX_VALUE switches to Long.MIN_VALUE. This can easily happen if you are doing fractions like (a*b*c*d)/(e*f) which may perfectly valid results in range of a long, but the intermediate nominator or denominator does not.

You could write your own Currency class, using a long to hold the amount. The class methods would set and get the amount using a String.
Division will be a concern no matter whether you use a long or a BigDecimal. You have to determine on a case by case basis what you do with fractional cents. Discard them, round them, or save them (somewhere besides your own account).

BigDecimal Euler's number calculation to emulate Math.exp(double a)

Is there a generally accepted way to get Euler's number in BigDecimal?
I basically want to do Math.exp(double a) and I basically want the same function but calculated using BigDecimal to prevent precision loss with other calculations.
I thought of just doing,
BigDecimal.valueOf(Math.E).pow(BigDecimal.SOMEOTHERNUMBER)
But not sure if that's the right way to approach the problem.

The problem is that Euler's number is irrational; in BigDecimal, you'd have to have infinite memory to hold it. The reason there's no previously-stored value is that however many decimal places you needed, you wouldn't have the right number for your current application.
You're going to have to determine how many decimals you want, then construct the BigDecimal representation using the String constructor (new BigDecimal("2.71828...")).

Well, Math.E is a double. If you want a higher precision representation of Euler's number, you can create it with some arbitrary precision of decimal places, or there are BigDecimal math libraries with a really high precision version of E.
You could approximate one yourself with something like this: Better approximation of e with Java

While the double Math.E should be accurate enough for most calculations, there is a way to calculate e to greater accuracy.
If you want to get e that has more accuracy that double can contain, you calculate it by adding up the sum of 1/0!, 1/1!, 1/2!, 1/3!...
Where ! means factorial. Just add up the reciprocal of all the factorials starting from zero. The more terms you add up, the more precise you will get. Of course since e is irrational, you can never get the real value, but you can get pretty darn close.

ArithmeticException thrown during BigDecimal.divide

I thought java.math.BigDecimal is supposed to be The Answer™ to the need of performing infinite precision arithmetic with decimal numbers.
Consider the following snippet:
import java.math.BigDecimal;
//...
final BigDecimal one = BigDecimal.ONE;
final BigDecimal three = BigDecimal.valueOf(3);
final BigDecimal third = one.divide(three);
assert third.multiply(three).equals(one); // this should pass, right?
I expect the assert to pass, but in fact the execution doesn't even get there: one.divide(three) causes ArithmeticException to be thrown!
Exception in thread "main" java.lang.ArithmeticException:
Non-terminating decimal expansion; no exact representable decimal result.
at java.math.BigDecimal.divide
It turns out that this behavior is explicitly documented in the API:
In the case of divide, the exact quotient could have an infinitely long decimal expansion; for example, 1 divided by 3. If the quotient has a non-terminating decimal expansion and the operation is specified to return an exact result, an ArithmeticException is thrown. Otherwise, the exact result of the division is returned, as done for other operations.
Browsing around the API further, one finds that in fact there are various overloads of divide that performs inexact division, i.e.:
final BigDecimal third = one.divide(three, 33, RoundingMode.DOWN);
System.out.println(three.multiply(third));
// prints "0.999999999999999999999999999999999"
Of course, the obvious question now is "What's the point???". I thought BigDecimal is the solution when we need exact arithmetic, e.g. for financial calculations. If we can't even divide exactly, then how useful can this be? Does it actually serve a general purpose, or is it only useful in a very niche application where you fortunately just don't need to divide at all?
If this is not the right answer, what CAN we use for exact division in financial calculation? (I mean, I don't have a finance major, but they still use division, right???).

If this is not the right answer, what CAN we use for exact division in financial calculation? (I mean, I don't have a finance major, but they still use division, right???).
Then I was in primary school1, they taught me that when you divide by 1 by 3 you get a 0.33333... i.e. a recurring decimal. Division of numbers represented in decimal form is NOT exact. In fact for any fixed base there will be fractions (the result of dividing one integer by another) that cannot be represented exactly as a finite precision floating point number in that base. (The number will have a recurring part ...)
When you do financial calculations involving division, you have to consider the what to do with a recurring fraction. You can round it up, or down, or to the nearest whole number, or something else, but basically you cannot just forget about the issue.
The BigDecimal javadoc says this:
The BigDecimal class gives its user complete control over rounding behavior. If no rounding mode is specified and the exact result cannot be represented, an exception is thrown; otherwise, calculations can be carried out to a chosen precision and rounding mode by supplying an appropriate MathContext object to the operation.
In other words, it is your responsibility to tell BigDecimal what to do about rounding.
EDIT - in response to these followups from the OP.
How does BigDecimal detect infinite recurring decimal?
It does not explicitly detect the recurring decimal. It simply detects that the result of some operation cannot be represented exactly using the specified precision; e.g. too many digits are required after the decimal point for an exact representation.
It must keep track of and detect a cycle in the dividend. It COULD HAVE chosen to handle this another way, by marking where the recurring portion is, etc.
I suppose that BigDecimal could have been specified to represent a recurring decimal exactly; i.e. as a BigRational class. However, this would make the implementation more complicated and more expensive to use2. And since most people expect numbers to be displayed in decimal, and the problem of recurring decimal recurs at that point.
The bottom line is that this extra complexity and runtime cost would be inappropriate for typical use-cases for BigDecimal. This includes financial calculations, where accounting conventions do not allow you to use recurring decimals.
1 - It was an excellent primary school. You may have been taught this in high school.
2 - Either you try to remove common factors of the divisor and dividend (computationally expensive), or allow them to grow without bounds (expensive in space usage and computationally expensive for subsequent operations).

The class is BigDecimal not BigFractional. From some of your comments it sounds like you just want to complain that someone didn't build in all possible number handling algorithms into this class. Financial apps do not need infinite decimal precision; just perfectly accurate values to the precision required (typically 0, 2, 4, or 5 decimal digits).
Actually I have dealt with many financial applications that use double. I don't like it but that was the way they are written (not in Java either). When there are exchange rates and unit conversions then there are both the potential of rounding and bruising problems. BigDecimal eliminates the later but there is still the former for division.

If you want to work with decimals, not rational numbers, and you need exact arithmetics before the final rounding (rounding to cents or something), here's a little trick.
You can always manipulate your formulas so that there's only one final division. That way you won't lose precision during calculations and you'll always get the correctly rounded result. For instance
a/b + c
equals
(a + bc) / b.

By the way, I'd really appreciate
insight from people who've worked with
financial software. I often heard
BigDecimal being advocated over double
In financial reports we use alwasy BigDecimal with scale = 2 and ROUND_HALF_UP, since all printed values in a report must be lead to a reproducable result. If someone checks this using a simple calculator.
In switzerland they round to 0.05 since they no longer have 1 or 2 Rappen coins.

You should prefer BigDecimal for finance calculations. Rounding should be specified by the business. E.g. an amount (100,00$) has to be split equally across three accounts. There has to be a business rule which account takes the extra cent.
Double, floats are not approriate for use in financial applications because they can not represent fractions of 1 precisely that are not exponentials of 2. E.g. consider 0.6 = 6/10 = 1*1/2 + 0*1/4 + 0*1/8 + 1*1/16 + ... = 0.1001...b
For mathematic calculations you can use a symbolic number, e.g. storing denominator and numerator or even a whole expression (e.g. this number is sqrt(5)+3/4). As this is not the main use case of the java api you won' find it there.

Is there a need for
a=1/3;
b=a*3;
resulting in
b==1;
in financial systems? I guess not. In financial systems it is defined, which roundmode and scale has to be used, when doing calculations. In some situations, the roundmode and scale is defined in the law. All components can rely on such a defined behaviour. Returning b==1 would be a failure, because it would not fulfill the specified behaviour. This is very important when calculating prices etc.
It is like the IEEE 754 specifications for representing floats in binary digits. A component must not optimize a "better" representation without loss of information, because this will break the contract.

To divide save, you have to set the MATHcontext,
BigDecimal bd = new BigDecimal(12.12, MathContext.DECIMAL32).divide(new BigDecimal(2)).setScale(2, RoundingMode.HALF_UP);

I accept that Java doesn't have great support for representing fractions, but you have to realise that it is impossible to keep things entirely precise when working with computers. At least in this case, the exception is telling you that precision is being lost.
As far as I know, "infinite precision arithmetic with decimal numbers" just isn't going to happen. If you have to work with decimals, what you're doing is probably fine, just catch the exceptions. Otherwise, a quick google search finds some interesting resources for working with fractions in Java:
http://commons.apache.org/math/userguide/fraction.html
http://www.merriampark.com/fractions.htm
Best way to represent a fraction in Java?

Notice we are using a computer... A computer has a lot of ram and precision takes ram. So when you want an infinite precision you need
(infinite * infinite) ^ (infinite * Integer.MAX_VALUE) terrabyte ram...
I know 1 / 3 is 0.333333... and it should be possible to store it in ram like "one divided by three" and then you can multiply it back and you should have 1. But I don't think Java has something like that...
Maybe you have to win the Nobel Price for writing something doing that. ;-)