auth [success=done default=ignore] pam_access.so accessfile=/etc/security/access-local.conf

Seems to be the answer.

If someone else wants to right a better answer that explains what's happening - I'll happily accept that. I'll try to work out what's happening and edit this.

Thanks to Hostfission for pointing me at the right part. Not sure why his comment didn't work at present and broke the authentication

auth [success=done default=ignore] pam_access.so accessfile=/etc/security/access-local.conf

Seems to be the answer. Editing this as I learn more.

From http://linux-pam.org/Linux-PAM-html/sag-configuration-file.html

success=done

done equivalent to ok with the side effect of terminating the module stack and PAM immediately returning to the application.

default=ignore

ignore when used with a stack of modules, the module's return status will not contribute to the return code the application obtains.

from - https://serverfault.com/questions/134471/success-n-control-syntax-in-pam-conf-pam-d-files#134488

success=1 skips a line.

If someone else wants to write a better answer that explains what's happening - I'll happily accept that.

Thanks to Hostfission for pointing me at the right part.