In general, you should focus on readability first, and performance much later. Most of the time, those performance optimizations are negligible anyway, but the maintenance cost can be huge.

Certainly all the "little" things should be changed in favor of clarity since, as you pointed out, most of them will get optimized by the compiler anyway.

As for the larger optimizations, there might be a chance that the optimizations are actually critical to reaching reasonable performance (though this isn't the case surprisingly often). I would make your changes and then profile the code before and after the changes. If the new code has significant performance issues, you can always roll-back to the optimized version, and if not you can just stick with the cleaner code version.

Change only one part of the code at a time and see how it affects performance after each round of refactoring.