When 5.95 does not equal 5.95
People like nicely formatted numbers on their reports and web pages. For example, if I needed to display the results of a division where the dividend is 1 and the divisor is 3, typically I would display that as 0.3 (the number formatted to 1 decimal place) instead of 0.3333333.
So imagine my surprise when I had a single detail line in my report where a value is shown as 6.0, but in the total line right below it (which utilizes the same data since there is only the one detail line), the same value is displayed as 5.9.
We finally tracked the problem down to a conversion between a float to a double. It appears that the .NET runtime is not able to take a float value of 5.95 and convert it exactly to a double of 5.95, the conversion sets the double to 5.94999980926514. As a result, when this value was sent into the string formatting function, the value would round down instead of rounding up. Keep in mind that the divisions actually come out right, whether they are floats or doubles. As I stepped through the code, all looked well since the float variable was calculating as 5.95 (119 divided by 20), but this was of course before the conversion and formatting.
As I think about it, I was pretty lucky to find this little glitch, as it will only manifest itself when for certain dividends when the divisor is a multiple of 20. (The 1 decimal place round off error happens at multiples of 0.05, or 1 / 20.) Feeling interested, I decided to do a little experimentation. I am married and over the hill, what else am I going to do on a Friday night?
I set up a .NET C# console application to test out the divisions, conversions, and string formatting, and reporting on when the formatted string of the double varies from the formatted string of the float converted to a double. The divisor would loop from 20 to 1000 by 20, and the dividend would loop from 1 to 1000. The final results were:
Divisions tested: 50000
Divisions failed: 916
Failure percentage: 1.83 %
As expected, the divisor of 20 shows the most variations, and dropped off to only 4 variations with a divisor of 1000 (at 350, 450, 650, and 950).
The moral of the story is when it comes to floats, just say no. We went into the code and removed all the float variables and casts and changed them all to doubles.