I ran into the same problem today but the solution of @Mark-Nutter was incomplete to remove the hashbang from my angularjs application.
In fact you have to go to Edit Permissions, click on Add more permissions and then add the right List on your bucket to everyone. With this configuration, AWS S3 will now, be able to return 404 error and then the redirection rule will properly catch the case.
And then you can go to Edit Redirection Rules and add this rule :
<RoutingRules>
<RoutingRule>
<Condition>
<HttpErrorCodeReturnedEquals>404</HttpErrorCodeReturnedEquals>
</Condition>
<Redirect>
<HostName>subdomain.domain.fr</HostName>
<ReplaceKeyPrefixWith>#!/</ReplaceKeyPrefixWith>
</Redirect>
</RoutingRule>
</RoutingRules>
<RoutingRules>
<RoutingRule>
<Condition>
<HttpErrorCodeReturnedEquals>404</HttpErrorCodeReturnedEquals>
</Condition>
<Redirect>
<HostName>subdomain.domain.fr</HostName>
<ReplaceKeyPrefixWith>#!/</ReplaceKeyPrefixWith>
</Redirect>
</RoutingRule>
</RoutingRules>
Here you can replace the HostName subdomain.domain.fr with your domain and the KeyPrefix #!/ if you don't use the hashbang method for SEO purpose.
Of course, all of this will only work if you have already have setup html5mode in your angular application.
$locationProvider.html5Mode(true).hashPrefix('!');
$locationProvider.html5Mode(true).hashPrefix('!');
