To decide the point at which you should start over, you need to work out where the value of continuing on your current project falls short of the value in starting over.

The reason this is so difficult is that you make an accurate estimation of the team's development velocity on the new project until you actually start it. That said, if, using a VERY conservative estimate of your development velocity on the new project, the project is estimated to give a worthwhile return on investment, then you have a business case for starting afresh.