While it's true that the controller to interactor boundary isn't helping you invert a dependency, that isn't why it's there. The reason it's there is this:
• High-level modules should not depend on low-level modules. Both should depend on abstractions.
That's from the Dependency Inversion Principle. All it means is that it's nice to have a stable definition of the mini language that the two volatile concrete modules will communicate through. It means they don't have to know about each other. Just about the mini language. A fully abstract class that isn't bound to any destabilizing implementation details satisfies this just fine.
DIP and Clean Architecture are very closely linked. I mapped the Clean Architecture UML onto DIP here