Skip to main content

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

4
  • 3
    Note that at least newer versions of GNU awk have features for Getting the Accuracy You Need e.g. gawk -M -v PREC="quad" -v OFMT="%.0f" 'BEGIN{ a = 11111111111111111111; print a, a/22; }' Commented Aug 17, 2018 at 0:58
  • @Isaac, well, the online manual says the default for OFMT is %.6g, so I get 5.05051e+17 for gawk -M -vPREC=quad 'BEGIN{ a = 11111111111111111111; print a/22; }'. As for just using -M, gawk -M 'BEGIN{ a = 11111111111111111111; print a/22; }' gives me the wrong answer 505050505050505024. I'm not sure if that's a bug or if it automatically converts the number to a float when it's no longer an integer (a/11 is an integer, and it gives the right answer with just -M). Commented Aug 21, 2018 at 8:46
  • And there's of course ROUNDMODE for setting the rounding mode if "round to even" isn't what you want Commented Aug 21, 2018 at 8:50
  • @ilkkachu (1) Yes, Sorry about OFMT, yes, it is g not f. (2) If the result of a division require a decimal (some non-zero digit after the dot) then that result is internally converted to a float with the precision given by PREC. Any integer gets as many digits (before the dot) as it may need. Try gawk -M -vPREC=20 -vOFMT='%.30f' 'BEGIN{ a=22^22; b=101; print a,a/22,b/10; }' The first two numbers are integers and are exact, the last is as imprecise as given by PREC=20. Change it to PREC=200 to see the change on the last number only.(4)Yes, rounding could be adjusted. Commented Aug 21, 2018 at 14:53