I have a data file of numbers seperated by tabs, like this
1 2 3 4
2 4 6 8
My real file is 50000 columns wide and I only need every 100th column (column 100, 200, 300, 400, ...). Now I would like to remove all the other columns.
How can I do that?
That's what awk is for:
awk '{for(i=100;i<=NF;i+=100){printf "%s ",$i;} print ""}' file > output
Or, if you can have spaces inside your fields, specify tab as the field separator:
awk -F'\t' '{for(i=100;i<=NF;i+=100){printf "%s ",$i;} print ""}' file > output
Alternatively, you could use Perl:
perl -ane 'for($i=99;$i<=$#F;$i+=100){print "$F[$i] "}' file > output
To do this for multiple files, you can use a shell loop (assuming you want to run this on all files in the current directory):
for f in *; do
awk '{for(i=100;i<=NF;i+=100){printf "%s ",$i;} print ""}' "$f" > "$f".new;
done
Although I don't know if it's appropriated with big files, you can do this with cut:
cut -d " " -f -100 < [your file]
cut you could use cut -d' ' -f$(seq -s, 0 100 N) where N is the total number of columns but as you said, this won't work with big numbers (you'll most likely get an argument list too long).