vendredi 16 août 2019

loss of precision when using sqrt()

loss of precision when using sqrtl() with long long int

I need to use a variable to store the value since I would expect the user to type in their own value in range 0 to 10^18, so it makes sense to use long long int in this situation. But this is where the error occurs which I could not solve it properly.

unsigned long long int a = 999999999999999999;
cout << fixed;
cout << (floor(sqrtl(999999999999999999)) == sqrtl(999999999999999999)) << endl;
cout << floor(sqrtl(999999999999999999)) << endl;
cout << (sqrtl(999999999999999999)) << endl;

cout << a << endl;
cout << (floor(sqrtl(a)) == sqrtl(a)) << endl;
cout << floor(sqrtl(a)) << endl;
cout << (sqrtl(a)) << endl;

In my machine, the output is

0
999999999.000000
1000000000.000000
999999999999999999
1
1000000000.000000
1000000000.000000

which shows the loss precision when using sqrtl(a) compare to sqrtl(999999999999999999)

I want to be able to get the correct value while using the variable

Aucun commentaire:

Enregistrer un commentaire