It depends largely on how you expect users to move through your app.
If the typical use case is to touch many pages in a session, then it might be a good idea to combine all the files -- this will result in a longer initial load, but subsequent transitions will be near-instant.
On the other hand, if you expect users to only hit one or two pages of the app in a typical visit (i.e. the pages are somehow "separate" from each other) then lazy loading could be to your advantage.
Personally, I always concat/minify into one package. Unless your modules are huge (10s/100s of KB huge) or there are sufficient metrics to back up a better user experience with lazy loading, it's not really worth the effort to implement it. And also, your users are already loading Angular which is itself pretty big anyway.