I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?
If it's any help, I'm using RedHat and an Ubuntu linux systems.
As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.
$ cat t.txt
12
34
56
78
90
$ sed -e '1,3d' < t.txt
78
90
You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.
$ tail -n +4 t.txt
78
90
$ awk 'NR > 3 { print }' < t.txt
78
90
sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.
sed -i specifically. Most implementations just delete the file and replace it with a new one, which doesn't work for links since you end up leaving the original at its other location.
sed doesn't do it in-place at all, it creates a temp file.
sed -i '1,3d' file.txt
This deletes first 3 line from file.txt.
If the tabulated lines are the ones that have a tab character:
grep '␉' <input_file >output_file
(␉ being a literal tab character) or equivalently
sed -n '/␉/p' <input_file >output_file
In a bash/ksh/zsh script, you can write $'\t' for a tab, e.g. grep $'\t' or sed -n $'/\t/p'.
If you want to eliminate 10 lines at the beginning of the file:
tail -n +11 <input_file >output_file
(note that it's +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or
sed '1,10d' <input_file >output_file
On Linux, you can take advantage of GNU sed's -i option to modify files in place:
sed -i -n '/\t/p' *.txt
Or you can use a shell loop and temporary files:
for x in *.txt; do
tail -n +11 <"$x" >"$x.tmp"
mv "$x.tmp" "$x"
done
Or if you don't want to modify the files in place, but instead give them a different name:
for x in *.txt; do
tail -n +11 <"$x" >"${x%.txt}.data"
done
You can use Vim in Ex mode:
ex -s -c '1d5|x' file
1 move to first line
5 select 5 lines
d delete
x save and close
Using bash, to clean up a file using a percentage number instead of an absolute number of lines:
sed -i -e 1,$( printf "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log
Watch out because that command can be destructive since it deletes content in-place, without creating a copy.
It deletes the first 75% of lines from the mentioned file.
# deletes first line
echo "a\nb" | sed '1d'
# read list.txt and write list.csv without first line
cat list.txt | sed '1d' > list.csv
Other useful commands:
# finds first character (pipe|)
grep '^|'
# deletes pipe
sed 's/|//g'
# deletes space
sed 's/ //g'
i created a small script which will delete everything from the /var/log/messages except last 4 lines.
# cat remove-range-of-lines.sh
#!/usr/bin/bash
#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`
#exclude last 4 lines
remove_line=`expr $line_count - 4`
#remove everything except last 4 lines
sed -i '1,'$remove_line'd' /var/log/messages
using this script we can always keep latest entry in the /var/log/messages based on our requirement
if you want to keep last 10000 entries in the /var/log/messages below is the modified script.
#!/usr/bin/bash
#print total line number
line_count=`awk 'END{print NR}' /var/log/messages`
#exclude last 10000 lines
remove_line=`expr $line_count - 10000`
#remove everything except last 10000 lines
sed -i '1,'$remove_line'd' /var/log/messages
n lines, not to keep the last n lines.