2

I commonly append to text files using cat:

cat >> FILE

I use an alias to avoid accidentally overwriting the file (with a single >):

alias a='cat >>'

Enter changes the line and Ctrl + D terminates the command. I write to multiple text files in my home folder, all of which I've created, own and can edit.

On several occasions, both on my desktop Linux system (Fedora 39) and Termux (Android), the command has ceased redirecting to a file while accepting input seemingly normally. I've lost hundreds of lines, mainly URLs I've pasted. Appears to only occur when the command has been running for a while.

Are there any reasons why redirection with cat >> may cease to function? Do for instance any special characters (in the input) have an effect?

UPDATE: I've confirmed that the inode number for the respective files keeps changing ($ ls -li or see the exact time with $ stat -c '%w') — this occurs since Syncthing by design recreates the synchronized files. I'll have to re-evaluate how I'm using the software in the future. Sorry for not mentioning Syncthing in the first place.

Of the commands I've automated, at least sed -i (edit a file in-place) replaces the inode as well.

The command cat >> FILE was due for a replacement as well (suggestions have been given, see the comments and the answer).

9
  • When you say "ceased redirecting", do you mean that it didn't append any of the text you typed, or just some of it? Maybe the next time it happens, try to copy the exact command you've used, then show the content of the file you were trying to append to with the missing text. edit your question and paste that information from your console. Commented Jan 28, 2024 at 11:54
  • @aviro Certainly none of it, the last time the issue occurred. I successfully wrote the same pasted lines to another file using the same command. Sadly I don't know if this can be a Syncthing-related error, haven't yet been able to reproduce elsewhere. Commented Jan 28, 2024 at 11:57
  • 2
    if it just so happens that anything removes the file cat is writing to, all consequent writes go to that now-deleted and impossible-to-access file, regardless of if a new file with the same name is created. You'd have to check the output of e.g. ls -li file at the start and when the writes start disappearing to check if it's still the same file (same inode number) Commented Jan 28, 2024 at 12:59
  • 2
    if that indeed appears the to the issue, you could switch from using cat to e.g. a shell script that loops reading and writing lines of input and reopens the file for each line. (You don't really need more than while IFS= read -r line; do echo "$line" >> "$file"; done, with the redirection on the echo, not around the whole loop.) Commented Jan 28, 2024 at 13:00
  • 2
    @GordonDavisson, a regular write could overwrite something written previously, but an append-mode write won't, as it's supposed to be an atomic seek to the end plus the write (with no opportunity for anything to get in between). So at least two instances of cat >> here should not overwrite each other. Commented Jan 28, 2024 at 19:53

2 Answers 2

5

Upon running cat >> file, the shell opens file in the current working directory in O_WRONLY|O_CREAT|O_APPEND in a child process on fd 1 and if successful executes cat in that process.

If not, the shell will output an error message and not run cat.

If it can't find a cat command it will also output an error message.

cat in turn, in a loop reads its fd 0 and writes what it has read on fd 1.

Again, if it fails to do so, it will output an error message.

If the process running cat is killed or suspended by a signal, the shell should also report it on stderr (like File size limit exceeded if killed with SIGXFSZ or suspended (tty input) if suspended with SIGTTIN).

If fd 0 is opened on a tty device and that tty device is the same the shell reads its commands from which will typically be the case, before running cat, the shell will configure the tty's line discipline the way it was before entering its own line editor.

Generally that means it will be in icanon mode where the line discipline implements a crude line editor.

In all of that, that line editor is really the only thing that will recognise special characters. stty -a will give you a list. In my case here:

$ stty -a
speed 38400 baud; rows 43; columns 159; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

You'll see the icanon mentioned above. ^C, ^\, ^?, ^U, ^D, ^Q, ^S, ^Z, ^R, ^W, ^V, ^O characters are handled specially. ^C/^Z/^\ subject to isig (in my case enabled), ^S/^Q subject to ixon, discard (^O above) not supported on Linux.

For instance, if you enter foobar^Ubaz, the foobar is killed, but then again, you'd see it being erased in the echo of that you type.

Entering the ^C character would kill cat in the middle of its read.

But most terminal emulators these days remove those characters when pasting. So for that to happen upon pasting, you'd need to be using a terminal that still doesn't do that stripping or for some of those kill/werase/intr... settings to be set to some non-control characters, which would be a pathological condition.

That line editor also has a limit on the size of the line it can edit. On Linux, IIRC it's 4095 bytes. So if you're pasting a line larger than that, anything past the 4095th byte will be discarded.

In the icanon mode, cat's read() will return when the tty line editor exits which will happen when you enter ^M (which it translates to ^J with icrnl on) or ^J or any of the eol, eol2 or eof characters.

But in any case, if cat exits normally, that is if it's not killed by SIGINT upon ^C or SIGQUIT upon ^\ or suspended with SIGTSTP upon ^Z, and doesn't report an error, it will have read the input from the terminal and written it to the file.

So in, summary, what you're experimenting should not happen under normal condition, and the only cases I can think of would pathological ones where:

  • the tty settings as reported by stty -a are all buggered.
  • or the data that you paste contains control characters interpreted by the tty driver's line editor and for some reason your terminal editor doesn't remove them.

Don't dismiss human errors like failing to see the error reported by cat or the shell or the files created in a different directory from what you thought, or removed, replaced, renamed or truncated whilst cat is running (like by that Syncthing that you mention using in comments).

With that a='cat >>' alias, if you enter a some file instead of a 'some file', that becomes cat >> some file which is the same as cat file >> some and cat will read file instead of stdin and write the output in some instead of some file.

A possible improvement over that alias (assuming GNU tee) would be:

a() {
  rlwrap -pblue -S 'add> ' tee --output-error=warn -a -- "$@" > /dev/null
}

Where rlwrap gives you a more advanced line editor than the crude tty driver one (and one you'll already be familiar with if using bash as your shell), and a prompt which makes it clearer that it's expecting some input.

And using tee allows you to write the input to several files at once.

3

If the intent is to avoid accidentally overwriting files, consider setting an appropriate shell variable. For example, for bash,

set -o noclobber    # Do not allow > to overwrite a file
set -C              # The same

Example

set -o noclobber
echo >ddd
echo >ddd
-bash: ddd: cannot overwrite existing file

set +o noclobber
echo >ddd

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.