Why does bash still keep duplicate lines with erasedups?

Question

I have the following line in my ~/.bashrc

export HISTCONTROL=ignoreboth:erasedups

However whenever I run somecommand it keeps adding the duplicated command into .bash_history like this

grep -w somecommand ~/.bash_history
somecommand
somecommand abc xyz
somecommand 123
somecommand
...

Why is this and how to prevent it?

Answer 1

The erasedups option will remove duplicated lines limited by this two conditions:

• The lines will be erased from the list in memory.
• Only when a new command is going to be added to the list.

The list in memory is the output of history.
The whole list "in memory" will be written to file with history -w or when bash is closed (if histappend is not set).

So, if the list in memory is:

$ history
1  cmd1
2  cmd2
3  cmd3
4  cmd1
5  cmd2
6  export HISTCONTROL=ignoreboth:erasedups
7  history

It will become:

$ history
1  cmd2
2  cmd3
3  cmd2
4  export HISTCONTROL=ignoreboth:erasedups
5  cmd1
6  history

All cmd1 commands (except the one being appended) are removed.

If cmd2 is executed, all repeats of cmd2 are also removed (also history):

$ cmd2
$ history
1  cmd3
2  export HISTCONTROL=ignoreboth:erasedups
3  cmd1
4  cmd2
5  history

But even now, the file ~/.bash_history has not been modified. It still contains the list of commands from a previous session. Using grep on the file may show many duplicates (dups), including the commands just used (cmd1 and cmd2). A grep will only work correctly on the memory list:

$ history | grep cmd1

Once the list gets written to file (history -w) is it correct that only one instance of each command line used in the session.will be found with grep.

You can force an update of the disk file every time with:

$ PROMPT_COMMAND='history -w'

Answer 2

According to man bash:

A value of ignoredups causes lines matching the previous history entry to not be saved.

If you run somecommand twice in a row with no arguments, you should see only one consecutive entry.

Likewise if you run somecommand 123 twice in a row, it will only be added to the history once.

The commands in your list show no consecutive duplicates, so this feature is working as intended.

Answer 3

To uniqely record every new command is tricky. First you need to add to ~/.profile or similar:

HISTCONTROL=erasedups
PROMPT_COMMAND='history -w'

Then you need to add to ~/.bash_logout:

history -a
history -w