Apply a single filter expression to many fields

[NikosAlexandris]

The following filter expression works just fine (for #608).

filter 'is_numeric($ln) || is_numeric($le) || is_numeric($mn)'

What about automagically applying is_numeric() to all of the input fields?
And, would this be possible to limit only to fields after cutting?

I am unsuccessfully trying something like

filter 'for (k,v in $*) { is_numeric(v) }'

[aborruso]

Does your goal is to filter every row in which at least one cell is a number?

Are all your fields numeric?

[NikosAlexandris]

Does your goal is to filter every row in which at least one cell is a number?

Filter every record of selected fields/columns.

Are all your fields numeric?

The ones I am interested to fill, yes.
I want to apply 'your' trick for selected columns. I have got my way to select the columns of interest. I want to avoid writing is_numeric() for every column I want. There are tenths or can be more of them.

[johnkerl]

@NikosAlexandris

I am unsuccessfully trying something like
filter 'for (k,v in $*) { is_numeric(v) }'

Can you please include the full input and output of your unsuccessful attempt?

[NikosAlexandris]

@aborruso correctly mentioned elsewhere to loop over each column. I just thought it's possible and worth learning how to do this with Miller only.

[NikosAlexandris]

Overall I think I would like to understand a bit more on how to work in a column-wise manner.

[johnkerl]

Ah OK @NikosAlexandris I get it! This will require https://johnkerl.org/miller6/operating-on-all-records/ as it's non-streaming (we need to see all rows to be sure b is an all-numeric column) -- we can make an example for sure. :)

[aborruso]

@NikosAlexandris it's not what you want. A stupid bash way could be the one below.

Starting from this csv input

a,b,c
1,2,a
3,5,t
4,7,o

I calculate the mean for each field. If I have 1 status, that field is not a number.

#!/bin/bash

set -x

if [ -f ./test ]; then
  rm ./test
fi

echo -e "$(mlr --c2n -N cat input.csv | head -n +1 | sed -r 's/ /\n/g')" | while read line; do
  test=$(mlr --csv stats1 -a mean -f "$line" input.csv)
  status=$?
  echo "field=$line,t=$status" >>./test
done

numericFields=$(mlr --ocsv filter '$t==0' then reshape -s field,t test | head -n +1)

mlr --csv cut -f $numericFields input.csv

The output is

a,b
1,2
3,5
4,7

[johnkerl]

@NikosAlexandris

$ cat mixed.csv
a,b,c
1,2,3
1,2,3
1,2,3
1,2,3
1,2,3
1,x,3
1,2,3
1,2,3

mlr --csv --from mixed.csv put -q '
  # Accumulate a list of all records
  @records[NR] = $*;
  # Accumulate a map of all field names seen
  for (k in $*) {
    @column_is_numeric[k] = true;
  }

  end {
    # See which column names are entirely numeric
    for (nr, record in @records) {
      for (k, v in record) {
        if (!is_numeric(v)) {
          @column_is_numeric[k] = false;
        }
      }
    }

    # Loop over retained records, unsetting the column
    # name if any record has non-numeric data in that column.
    for (nr, record in @records) {
      for (k, v in record) {
        if (!@column_is_numeric[k]) {
          unset record[k]
        }
      }
      emit record;
    }
  }
'

a,c
1,3
1,3
1,3
1,3
1,3
1,3
1,3
1,3