Arithmetic between 2 files generating a series of new files using awk

Question

I have a tab separated model input file i would like to vary for ensemble analysis formatted such as this

cat input.txt

#############################################   
###  Parameter file for the program ### 
#############################################   
### GENERAL PARAMETERS
4   /* nbout # Number of outputs */ 
46  /* numesp # Number of species */
0.05    /* p # light incidence param (diff through turbid medium) */
0.1357158   0.2446549   0.3535940   0.4992873   0.6449806   0.6957850   0.7465893   0.8130218   0.8794543   0.9397271   1.0000000   0.9397271   0.8794543   0.8078294   0.7362045   0.6899817   0.6437589   0.5989616   0.5541642   0.4617186   0.3692730   0.3633708   0.3574686   0.2426215   /* normalized daily light course (from 7am to 7pm, with a half-hour time-step */
1   /* vox_la_max. The max voxel leaf area. */
0   /* l_growth_scheme. 0 = top down; 1 = random; 2 = homogeneous; 3 = bottom up */
0.1 /* knockout_max. Parameter controlling the extent to which lianas can knock out trees */
0.05    /* shed_prob. With this probability, the liana is completely shed from the voxel. */

### Species description                                 
****    Nmass   LMA wsg dmax    hmax    ah  tmax    seedmass    Fregdistgr  Pmass   g1  s_liana
Alvaradoa_amorphoides   0.0214  74.775  0.584   0.5 24.44   0.892   1   0.0078  40  0.00145 3.77    0
Annona_reticulata   0.0350  74.529  0.503   0.5 24.44   0.892   1   0.2392  40  0.00142 3.77    0
Brosimum_alicastrum 0.0201  104.281 0.760   0.5 17.31   0.117   1   1.2486  40  0.00097 3.77    0

### Climate (input environment)
25.47447    26.02723    26.87827    27.58436    26.95839    25.63987    25.61669    25.26543    24.99990    24.10808    24.71997    24.67287    /*Temperature in degree C*/

I have another tab separated file of multipliers selected from a distribution formatted such as this

cat multipliers.txt

2   3   4
3   2   2
4   3   3

I am trying to multiply 3 specific input fields by multipliers to generate a series of new input files equal to the number of multipliers (in this case 3), while leaving the rest of the input file unchanged. In this scenario, I would like to multiply vox_la_max,knockout_max, and shed_prob by 2, 3, and 4 respectively for the first file, 3, 2, and 2 for the second file, and 4, 3, and 3 for the third. I would have 3 new files generated such as this

cat input1.txt

#############################################   
###  Parameter file for the program ### 
#############################################   
### GENERAL PARAMETERS
4   /* nbout # Number of outputs */ 
46  /* numesp # Number of species */
0.05    /* p # light incidence param (diff through turbid medium) */
0.1357158   0.2446549   0.3535940   0.4992873   0.6449806   0.6957850   0.7465893   0.8130218   0.8794543   0.9397271   1.0000000   0.9397271   0.8794543   0.8078294   0.7362045   0.6899817   0.6437589   0.5989616   0.5541642   0.4617186   0.3692730   0.3633708   0.3574686   0.2426215   /* normalized daily light course (from 7am to 7pm, with a half-hour time-step */
2   /* vox_la_max. The max voxel leaf area. */
0   /* l_growth_scheme. 0 = top down; 1 = random; 2 = homogeneous; 3 = bottom up */
0.3 /* knockout_max. Parameter controlling the extent to which lianas can knock out trees */
0.2 /* shed_prob. With this probability, the liana is completely shed from the voxel. */

### Species description                                 
****    Nmass   LMA wsg dmax    hmax    ah  tmax    seedmass    Fregdistgr  Pmass   g1  s_liana
Alvaradoa_amorphoides   0.0214  74.775  0.584   0.5 24.44   0.892   1   0.0078  40  0.00145 3.77    0
Annona_reticulata   0.0350  74.529  0.503   0.5 24.44   0.892   1   0.2392  40  0.00142 3.77    0
Brosimum_alicastrum 0.0201  104.281 0.760   0.5 17.31   0.117   1   1.2486  40  0.00097 3.77    0

### Climate (input environment)
25.47447    26.02723    26.87827    27.58436    26.95839    25.63987    25.61669    25.26543    24.99990    24.10808    24.71997    24.67287    /*Temperature in degree C*/

cat input2.txt

#############################################   
###  Parameter file for the program ### 
#############################################   
### GENERAL PARAMETERS
4   /* nbout # Number of outputs */ 
46  /* numesp # Number of species */
0.05    /* p # light incidence param (diff through turbid medium) */
0.1357158   0.2446549   0.3535940   0.4992873   0.6449806   0.6957850   0.7465893   0.8130218   0.8794543   0.9397271   1.0000000   0.9397271   0.8794543   0.8078294   0.7362045   0.6899817   0.6437589   0.5989616   0.5541642   0.4617186   0.3692730   0.3633708   0.3574686   0.2426215   /* normalized daily light course (from 7am to 7pm, with a half-hour time-step */
3   /* vox_la_max. The max voxel leaf area. */
0   /* l_growth_scheme. 0 = top down; 1 = random; 2 = homogeneous; 3 = bottom up */
0.2 /* knockout_max. Parameter controlling the extent to which lianas can knock out trees */
0.1 /* shed_prob. With this probability, the liana is completely shed from the voxel. */

### Species description                                 
****    Nmass   LMA wsg dmax    hmax    ah  tmax    seedmass    Fregdistgr  Pmass   g1  s_liana
Alvaradoa_amorphoides   0.0214  74.775  0.584   0.5 24.44   0.892   1   0.0078  40  0.00145 3.77    0
Annona_reticulata   0.0350  74.529  0.503   0.5 24.44   0.892   1   0.2392  40  0.00142 3.77    0
Brosimum_alicastrum 0.0201  104.281 0.760   0.5 17.31   0.117   1   1.2486  40  0.00097 3.77    0

### Climate (input environment)
25.47447    26.02723    26.87827    27.58436    26.95839    25.63987    25.61669    25.26543    24.99990    24.10808    24.71997    24.67287    /*Temperature in degree C*/

cat input3.txt

#############################################   
###  Parameter file for the program ### 
#############################################   
### GENERAL PARAMETERS
4   /* nbout # Number of outputs */ 
46  /* numesp # Number of species */
0.05    /* p # light incidence param (diff through turbid medium) */
0.1357158   0.2446549   0.3535940   0.4992873   0.6449806   0.6957850   0.7465893   0.8130218   0.8794543   0.9397271   1.0000000   0.9397271   0.8794543   0.8078294   0.7362045   0.6899817   0.6437589   0.5989616   0.5541642   0.4617186   0.3692730   0.3633708   0.3574686   0.2426215   /* normalized daily light course (from 7am to 7pm, with a half-hour time-step */
4   /* vox_la_max. The max voxel leaf area. */
0   /* l_growth_scheme. 0 = top down; 1 = random; 2 = homogeneous; 3 = bottom up */
0.3 /* knockout_max. Parameter controlling the extent to which lianas can knock out trees */
0.15    /* shed_prob. With this probability, the liana is completely shed from the voxel. */

### Species description                                 
****    Nmass   LMA wsg dmax    hmax    ah  tmax    seedmass    Fregdistgr  Pmass   g1  s_liana
Alvaradoa_amorphoides   0.0214  74.775  0.584   0.5 24.44   0.892   1   0.0078  40  0.00145 3.77    0
Annona_reticulata   0.0350  74.529  0.503   0.5 24.44   0.892   1   0.2392  40  0.00142 3.77    0
Brosimum_alicastrum 0.0201  104.281 0.760   0.5 17.31   0.117   1   1.2486  40  0.00097 3.77    0

### Climate (input environment)
25.47447    26.02723    26.87827    27.58436    26.95839    25.63987    25.61669    25.26543    24.99990    24.10808    24.71997    24.67287    /*Temperature in degree C*/

I think I should be using awk, but have only been successful at varying one parameter with a one-column multipliers file at a time so far, and I need to be able to vary these 3 parameters simultaneously. What sort of script could I set up to generate these outputs?

How many lines (roughly) in multipliers.txt? And is input.txt just that or does it contain several models, all to undergo the same multipliers? — LL3
– LL3, Commented Mar 22, 2021 at 18:26
@LL3 My actual multipliers.txt is 1000 rows and 3 columns of numeric values. The actual input.txt has many more parameters. In this example, vox_la_max,knockout_max, and shed_prob are on lines 9, 11, and 12. In the actual file they are on lines 36, 38, and 39. It is the parameter file for a single ecosystem model — sethparker
– sethparker, Commented Mar 22, 2021 at 18:35

LL3 · Accepted Answer · 2021-03-24 00:24:14Z

TL; DR: A compact awk script hardcoded for your example

NR != FNR {
    out = "out" FNR ".txt"
    printf "" > out
    for (l=m=1; l <= nl; l++)
        printf tmpl[l] ORS, l in vals ? $(m++)*vals[l] : 0 >> out
    close(out)
    next
}

{
    gsub(/%/, "%%")
# here is the regex that selects the fields by their name
    if ($3 ~ /^(vox_la_max|knockout_max|shed_prob)[^[:alnum:]_]*$/) {
        vals[NR] = $1
        sub(/^[0-9]+(\.[0-9]+)?/, OFMT)
    }
    tmpl[NR] = $0; nl++
}

Use it as:

LC_NUMERIC=C awk -f script input.txt multipliers.txt

It generates output files named outX.txt.

The LC_NUMERIC=C bit is required if your locale would use a comma instead of a dot as decimal separator for floating-point values.

For simplicity I made some assumptions which looked reasonable:

wanted input fields are always lone values with adjacent comment indicating the field's name as one word, which must be separated by blanks (at least one space) from the /*
there are no fields with same name
floating-point values are expressed with only digits and (possibly) one dot, i.e. no exponents or other scientific representations

The same script as above but verbose, described, and expanded to allow:

arbitrary specification of wanted fields by line number
arbitrary specification of wanted fields by their name as referred to by the comment on the input line belonging to each field
output files are automatically named after the input file name, which may have one extension (eg .txt) and whose indicated path (if any) must not have dots; in other words, better run the script from the directory containing the input file

# some preparations
BEGIN {
    # output files named as the input file name
    split(ARGV[1], f, ".")
    outpfx = f[1]
    # remember wanted fields specified on command line as comma-separated line numbers
    if (nums) {
        # split variable "nums" on comma into helper array "r"
        n = split(nums, r, ",")
        # loop over helper array to build final array, thus indexed by wanted line numbers
        while (n) rows[r[n--]]
    }
}

# here we operate on multipliers file
NR != FNR {
    # output file name for this set of multipliers
    out = outpfx FNR ".txt"
    # create/overwrite this output file
    printf "" > out
    # loop over template lines scanned from input file
    for (linenum = multnum = 1; linenum <= numlines; linenum++)
        # use the template line as printf format string to consume values to be multiplied (if any)
        printf tmpl[linenum] ORS, linenum in wanted_values ? $(multnum++)*wanted_values[linenum] : 0 >> out
    close(out)
    next
}

# here we scan the input file to build a template for printf
{
    # escape existing % chars as we are going to leverage printfs own format string which is %-based
    gsub(/%/, "%%")
    # on specified line numbers or named fields:
    if (NR in rows || names && match($3, "^("names")[^[:alnum:]_]*$")) {
        # remember this value
        wanted_values[NR] = $1
        # replace the original value with the printfs conversion specification for floating-point values
        # it will be used by printf later on while processing the multipliers file
        sub(/^[0-9]+(\.[0-9]+)?/, OFMT)
    }
    # remember this whole line as a template
    tmpl[NR] = $0; numlines++
}

Use it like:

# specify fields by their line numbers, each separated by a comma
LC_NUMERIC=C awk -f script -v nums=36,38,39 input.txt multipliers.txt
# or specify fields by their names, each separated by the | character (NOTE it's a regexp)
LC_NUMERIC=C awk -f script -v names='vox_la_max|knockout_max|shed_prob' input.txt multipliers.txt
# or also use both ways of specifying fields
LC_NUMERIC=C awk -f script -v nums=15,112,234,71,5 -v names='vox_la_max|numesp' input.txt multipliers.txt

If you specify more fields than multipliers, the fields in excess become 0 (are multiplied by 0).

If you specify less fields than multipliers, the multipliers in excess are simply ignored.

In any case the fields consume multipliers always in the order of the line number they appear on, i.e. the first field encountered in the input file consumes the first multiplier no matter how you specified the field.

guest_7 · Accepted Answer · 2021-03-24 02:20:01Z

Using Perl to generate the output files.

perl -wMstrict -pale '
BEGIN {
  ## variables declaration
  use vars qw($name $extn);
  use vars qw($header $template $footer);
  use vars qw(@glob_params $num_re);

  ## pick apart include.txt filename into its components
  ($name, $extn) =
    pop =~ m{(.*?)(\.[^.]*|)$}x;

  ## split and stuff into variables the include.txt file
  ($header, $template, $footer) =
    do{local $/;<STDIN>;} =~
    m{\A
    (.*?\n)
    (\#+\h*GENERAL\h+PARAMETERS\h*\n.*?\n\n)
    (.*)
    }xms;

  ## names of global parameters to vary
  @glob_params =
    qw( vox_la_max knockout_max shed_prob);

  # cinstruct the regex to search for numbers
  my $sign = qr{ [-+] }x;
  my $float = qr{ \d+(?:\.\d*)?|\.\d+ }x;
  my $exponent = qr{ [eE][-+]?\d+ }x;
  $num_re =
    qr{ $sign? $float $exponent? }x;
}

##### multiplier.txt processed from here onwards
  my %mult;
  @mult{ @glob_params } = @F;
  my $template_copy = $template;
  for (@glob_params) {
    $template_copy =~
      s{^($num_re)(?=\h+/\*\h*\Q$_\E\b)}{$1 * $mult{$_}}xme;
  }

  my $out = sprintf "%s%d%s",
    $name, $., $extn;
  open my $fh, ">", $out
    or die "Opening $out for writing:$!";
  select $fh;
  s/.*/$header$template_copy$footer/;
' multipliers.txt < include.txt include.txt

How are you running this? When I put your answer in a file named tst.perl and run sh tst.perl It generates 3 new files, but they are all identical to the initial file — sethparker
– sethparker, Commented Mar 23, 2021 at 16:18
Since you had stated your files are TAB separated so I assumed both include.txt and multipliers.txt are tab separated. Made a tweak and it should run now. — guest_7
– guest_7, Commented Mar 24, 2021 at 2:22

Stack Exchange Network

Arithmetic between 2 files generating a series of new files using awk

2 Answers 2

You must log in to answer this question.

Hot Network Questions

Arithmetic between 2 files generating a series of new files using awk

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions