How to find numbers in a textfile that are not divisible by 4096, round them up and write new file?

Question

In Linux there is a file numbers.txt. It contains a few numbers, all separated with space.

There are numbers like this: 5476089856 71788143 9999744134 114731731 3179237376

In this example only the first and the last numbers are divisible by 4096. All others are not divisible by 4096. I need all numbers divisible by 4096.

The numbers which are not divisble by 4096 should be rounded up. The other ones should be untouched.

All of them I need written to a new file numbers_4096.txt.

Numbers like the second 71788143 must be rounded up to 71790592. 9999744134 to 9999745024 and so on...

Numbers like 5476089856 and 3179237376 should not be changed, they are divisible by 4096.

This script can do this with a single number:

#!/bin/sh

number=9999744134
divisor=4096

remainder=$((number % divisor))
new_number=$((number - remainder))
roundedup=$((new_number + divisor))

echo "Original number: $number"
echo "Divisor: $divisor"
echo "New number (divisible by $divisor): $new_number"
echo "Roundedup Number (divisible by $divisor): $roundedup"

But how to do that with a whole list that has to be rewritten and rounded up into a new file?

This script adds 4096 to numbers which are divisible by 4096, that is not what I want.

Answer 1

in awk it is possible to avoid logical checks and rely on pure math in "one line":

awk  '{ for(i=1;i<=NF;i++) printf "%i ", int(($i+4095)/4096)*4096 ;print ""}' input_file

The logic is:

1. add 4096-1
2. get integer part of division to 4096
3. multiply to 4096

---

With default Awk precision settings, this incorrectly rounds up for 2^n - 4096 for n >= 54, due to the limitations of double-precision floating point which Awk uses for all its numbers. When the FP exponent field = 53, only even numbers are representable, so adding 4095 actually rounds to the nearest-even number, which is upwards to 2^54. For larger numbers, the representable values are even farther apart, larger powers of 2.

Also, rounding can occur on input into Awk for huge integers (greater than 2^54), so the same problem is possible even with huge inputs in the original text. For a large-enough exponent, 2^n - 4096 wouldn't be representable and would have rounded up to 2^n in the first place, before we try adding 4095.

GNU Awk can do arbitrary-precision arithmetic with -M / --bignum, but that actually doesn't help here (GNU Awk 5.3.1). So be careful with this if your numbers are much bigger than the example.

Adding 4095 can't overflow to +Infinity, but incorrectly rounding up when a huge number was 2^n - 4096 is possible.

The smallest number with that problem is 2^54 - 4096 = 0x3ffffffffff000 = 18014398509477888. This answer works for all smaller numbers, so should be fine for many use-cases. But not for the full range of [u]int64_t, let alone of double.

Answer 2

With perl, finding all sequences of decimal digits and rounding them up to multiple of 4096:

$ echo 'foo123bar 4095-4097 1e+4 0x1f2 004123 3.4' |
   perl -pe 's{\d+}{int(($& + 4095)/4096)*4096}ge'
foo4096bar 4096-8192 4096e+4096 0x4096f4096 8192 4096.4096

If you wanted to consider those -4097, 1e+4, 0x1f2, 3.4 as the negative / floating point / hexadecimal numbers they represent instead of only considering the sequences of decimal digits within, you could change it to:

$ echo '0x1p13 foo123bar 4095-4097 4096.001 1e+4 0x1f2 004123 3.4 infinity nan' |
   perl -MPOSIX=floor,strtold -pe '
     s{0x[\da-f]+(p[-+]?\d+)?|[-+]?(\d+\.?\d*|\.\d+)(e[-+]?\d+)?|\b(inf(inity)?|nan)\b}{
       ($x) = strtold($&);
       $y = floor($x / 4096) * 4096;
       $y == $x ? $y : $y+4096
     }gie'
8192 foo4096bar 4096-4096 8192 12288 4096 8192 4096 Inf NaN

Change ? $y to ? $& to not reformat the numbers that are multiples of 4096 as decimal integer (as in keep for instance 0x1000, 0X1P12, +40.96E+2, 4096.000 as is and not change them to 4096).

Answer 3

shell (for one number)

#!/bin/sh

number=9999744134
divisor=4096

remainder=$((number % divisor))
echo "Original number: $number"
echo "Divisor: $divisor"

if [ "$remainder" -eq 0 ]; then
    echo "Unchanged number (divisible by $divisor): ${number}"
else
    new_number=$((number - remainder))
    roundedup=$((new_number + divisor))
    echo "New number (divisible by ${divisor}): ${new_number}"
    echo "Roundedup Number (divisible by ${divisor}): ${roundedup}"
fi

awk (for the whole file)

echo 5476089856 71788143 9999744134 114731731 3179237376 |
  awk -v divisor=4096 '
    { for (i=1;i<=NF;i++)  {
        num=$i; remainder = num % divisor;
        if (remainder == 0) new_num = num;
        else new_num = num + divisor - remainder;
        printf "%i ", new_num;
      }
      print "";
    }'
5476089856 71790592 9999745024 114733056 3179237376

Answer 4

If you are always using a power of 2, like 4096, which has 12 bits.

For bash:

for i in $(cat numbers.txt); do
   echo $(( (i & ~4095) + ((!(!(i & 4095))) << 12) ))
done

where the first mask rounds down. The second mask checks the lower bits and the logical nots create a 0 or 1 for the remainder. The left shift converts the remainder to 0 or 4096.

In my experience, some Linux distributions implicitly replace bash for sh.

Answer 5

numfmt from GNU coreutils can do this in 2 passes:

< numbers.txt \
  numfmt --to-unit=4096 --round=up --field=- |
  numfmt --from-unit=4096 --field=-

The first pass does the division (rounding up):

% print 5476089856 71788143 9999744134 114731731 3179237376 |
> numfmt --to-unit=4096 --round=up --field=-
1336936    17527    2441344     28011     776181

The second pass does the multiplication:

% print 5476089856 71788143 9999744134 114731731 3179237376 |
> numfmt --to-unit=4096 --round=up --field=- |
> numfmt --from-unit=4096 --field=-
5476089856 71790592 9999745024 114733056 3179237376

The above assumes numbers.txt only has space/tab separated decimal integers (no trailing space, optional - sign). If there are non-numeric fields, --invalid=ignore (for both numfmt invocations) should leave those fields unchanged. Or perhaps you could specify which fields are to be converted (e.g. --field=2,3 to convert only fields 2 and 3). If the input may contain a decimal point as in 71788143.0, then add --format=%.0f to the first numfmt invocation. In this case, if that decimal point character shall be . (dot), then you may want to use the C locale (e.g. with LC_ALL=C numfmt ...). Otherwise the decimal point should match the locale (see locale decimal_point).

Answer 6

You can avoid the rounding issues from other answers using bc, which supports arbitrarily large numbers. In interactive mode, you could enter an expression like (5476089856+4095)/4096*4096 to round up to the nearest multiple of 4096; so for non-interactive mode we'll just convert all the numbers into that form using sed.

sed 's^\S\+^(\0+4095)/4096*4096;^g' | bc

Quick explanation of the sed argument: s^X^Y^g says to replace all the occurrences of the pattern X with the result Y. The X part in the above command says to look for non-whitespace characters (\S) in a non-empty sequence (\+); the Y part says to copy in whatever the pattern matched (\0) and surround it with our rounding computation (( and +4095)/4096*4096;).

For example, to read numbers from in.txt and write them to out.txt, you would do

sed 's^\S\+^(\0+4095)/4096*4096;^g' <in.txt | bc >out.txt

The outputs of this will be on newlines; if it's important to have spaces instead of newlines, you can do that using the (GNU-specific) bc extension print:

sed 's^\S\+^print (\0+4095)/4096*4096, " ";^g' | bc

Answer 7

Using Raku (formerly known as Perl_6)

~$ echo "4095 4096 5476089856 71788143 9999744134 114731731 3179237376\n18014398509477887 18014398509477888" |\
    raku -ne '.words.map( { my $remainder = $_ % 4096; 
              $remainder == 0 ?? $_ !! $_ + (4096 - $remainder) }).put;'
4096 4096 5476089856 71790592 9999745024 114733056 3179237376
18014398509477888 18014398509477888

OR:

~$ echo "4095 4096 5476089856 71788143 9999744134 114731731 3179237376\n18014398509477887 18014398509477888" |\
    raku -e '.words.map( { my $remainder = $_ % 4096; 
             $remainder == 0 ?? $_ !! $_ + (4096 - $remainder) }).put for lines();'
4096 4096 5476089856 71790592 9999745024 114733056 3179237376
18014398509477888 18014398509477888

Raku is a programming language in the Perl-family. Raku uses $_ as topic variable--same as Perl. Above also uses Raku's new built-in ternary operator (which is spelled slightly differently than Perl's): Test ?? True !! False. Some people find Raku's ternary easier to read.

Spurious input: Only a tiny bit of testing above--adding an alphabetic character to the end of one number gives error Cannot convert string to number... (with the description trailing characters after number and the location of the alphabetic character). Not a bad error message!

ALSO, Raku does reasonably well with alternative numeric representations, e.g. the forms given in the Perl answer by @StéphaneChazelas. See below:

~$ echo "-4097 4096.001 1e+4 0x1f2 004123 3.4 Inf NaN \n $(seq 4095 4097)" |\
    raku -e '.words.map( { my $remainder = $_ % 4096;
             $remainder == 0 ?? $_ !! $_ + (4096 - $remainder) }).put for lines();'
-4096 8192 12288 4096 8192 4096 NaN NaN
4096
4096
8192

https://docs.raku.org/language/operators#infix_??_!!
https://docs.raku.org
https://raku.org