Explanation:

trap command signals executes command when one of signals is "raised".

history | tail -1l | cut -c -5 outputs history number of last command saved into history.
^{(Note: if command which failed is not saved into history due to HISTCONTROL, HISTIGNORE, etc., you get history number of previous command, marking it for deletion.)}

Enclosing command in in $(), like $(command), executes the command and captures its output.
 

Captured output is then prepended (not appended!) into FAILED_COMMANDS variable.

It's probably good idea to hook it at EXIT, but you can also call it anytime earlier.

trap command signals executes command when one of signals is "raised".

$(command), executes the command and captures its output.

When command fails, this snippet of code captures history number of last command saved into history, and stores it in variable for future deletion.

Simple, but works incorrectly with HISTCONTROL and HISTIGNORE – when command is not saved into history due to one of the variables, history number of last command saved into history is previous command's one; so, if incorrect command is not saved into history, previous command is going to be deleted.

Slightly more complicated version, which works correctly in that case:

debug_handler() {
    LAST_COMMAND=$BASH_COMMAND;
}

error_handler() {
    local LAST_HISTORY_ENTRY=$(history | tail -1l)

    # if last command is in history (HISTCONTROL, HISTIGNORE)...
    if [ "$LAST_COMMAND" == "$(cut -d ' ' -f 2- <<< $LAST_HISTORY_ENTRY)" ]
    then
        # ...prepend it's history number into FAILED_COMMANDS,
        # marking the command for deletion.
        FAILED_COMMANDS="$(cut -d ' ' -f 1 <<< $LAST_HISTORY_ENTRY) $FAILED_COMMANDS"
    fi
}

trap error_handler ERR
trap debug_handler DEBUG

Because of that, when using this code, you cannot manually call history -d <n>
^{where n is smaller or equal to greatest number stored in FAILED_COMMANDS}
and expect the code to work properly.

It's probably good idea to hook exit_handler at EXIT, but you can also call it anytime earlier.

It's good to have last incorrect comment to correct it, but soon after that, it becomes potentially confusing garbage.

My approach is two-step: store commands that fail when they do, and remove them sometime later.

Store commands that fail when they do:

error_handler() {
    FAILED_COMMANDS="$(history | tail -1l | cut -c -5) $FAILED_COMMANDS"
}

trap error_handler ERR

Explanation:

trap command signals executes command when one of signals is "raised".

history | tail -1l | cut -c -5 outputs history number of last command saved into history.
^{(Note: if command which failed is not saved into history due to HISTCONTROL, HISTIGNORE, etc., you get history number of previous command, marking it for deletion.)}

Enclosing command in in $(), like $(command), executes the command and captures its output.

Captured output is then prepended (not appended!) into FAILED_COMMANDS variable.

Remove stored commands sometime later:

exit_handler() {
    for i in $(echo $FAILED_COMMANDS | tr ' ' '\n' | uniq)
    do
        history -d $i
    done
    FAILED_COMMANDS=
}

trap exit_handler EXIT

Explanation:

When exiting Bash, for each unique history number remove corresponding history entry,
then clear FAILED_COMMANDS to not remove commands which inherited history numbers from already deleted commands.

If you're sure that FAILED_COMMANDS will be free from duplicates, you can simple iterate over it
(ie. write for i in $FAILED_COMMANDS). If, however, you expect it to be not sorted from greatest to smallest (in this case it always is), replace uniq with sort -rn.

History numbers in FAILED_COMMANDS must be unique and sorted from greatest to smallest, because when you delete entry, next commands' numbers are shifted – ie. when you issue history -d 2, 3rd entry becomes 2nd, 4th becomes 3rd, etc.

It's probably good idea to hook it at EXIT, but you can also call it anytime earlier.

It's good to have last incorrect comment to correct it, but soon after that, it becomes potentially confusing garbage.

My approach is two-step: store commands that fail when they do, and remove them sometime later.

Store commands that fail when they do:

error_handler() {
    FAILED_COMMANDS="$(history | tail -1l | cut -c -5) $FAILED_COMMANDS"
}

trap error_handler ERR

Explanation:

trap command signals executes command when one of signals is "raised".

history | tail -1l | cut -c -5 outputs history number of last command saved into history.
^{(Note: if command which failed is not saved into history due to HISTCONTROL, HISTIGNORE, etc., you get history number of previous command, marking it for deletion.)}

Enclosing command in in $(), like $(command), executes the command and captures its output.

Captured output is then prepended (not appended!) into FAILED_COMMANDS variable.

Remove stored commands sometime later:

exit_handler() {
    for i in $(echo $FAILED_COMMANDS | tr ' ' '\n' | uniq)
    do
        history -d $i
    done
    FAILED_COMMANDS=
}

trap exit_handler EXIT

Explanation:

When exiting Bash, for each unique history number remove corresponding history entry,
then clear FAILED_COMMANDS to not remove commands which inherited history numbers from already deleted commands.

If you're sure that FAILED_COMMANDS will be free from duplicates, you can simple iterate over it
(ie. write for i in $FAILED_COMMANDS). If, however, you expect it to be not sorted from greatest to smallest (in this case it always is), replace uniq with sort -rnu.

History numbers in FAILED_COMMANDS must be unique and sorted from greatest to smallest, because when you delete entry, next commands' numbers are shifted – ie. when you issue history -d 2, 3rd entry becomes 2nd, 4th becomes 3rd, etc.

It's probably good idea to hook it at EXIT, but you can also call it anytime earlier.