It's a trade-off. You should optimize when there is a likely or clear potential benefit from optimizing the code, and stop optimizing when you no longer estimate that benefit to be worth your time.

The cost, as you mentioned, can be frustration, finding and fixing more obscure bugs, creating more thorough tests, clearly documenting obscure refactorings, giving up and retreating to a previous working version, and etc.

The benefits can be a little as just from learning how to write (or not write!) "tight" efficient code for later situations where it is required (tiny embedded IoT, or zillion of costly AWS instances, or deep space network uploads, and etc.)

In other situations, removing duplicated code might even cause disadvantages (tutorial code readability, weird processor cache/pipeline hazards, and etc.)

It's a trade-off. You should optimize when there is a likely or clear potential benefit from optimizing the code, and stop optimizing when you no longer estimate that benefit to be worth your time.

The cost, as you mentioned, can be frustration, more time finding and fixing obscure bugs, creating more thorough tests, clearly documenting obscure refactorings, giving up and retreating to a previous working version, and etc.

The benefits can be a little as just from learning how to write (or not write!) "tight" efficient code for later situations where it is required (tiny embedded IoT, or zillion of costly AWS instances, or deep space network uploads, mobile user's battery life, and etc.)

In other situations, removing duplicated code might even cause disadvantages (tutorial code readability, weird processor cache/pipeline hazards, and etc.)