A few papers that will help you get a better understating of browser architecture:

• A Reference Architecture for Web Browsers
• Architecture and evolution of the modern web browser
• Conceptual Architecture of Firefox
• The Security Architecture of the Chromium Browser

Be warned, the links are to pdf files. The last one is not exactly what you are looking for, but it’s a good read. As for specific layout engines:

Gecko

• Mozilla’s Layout Engine, presentation by David Baron
• Gecko Overview, in the Mozilla Wiki

Webkit

You can find some basic architectural diagrams in this presentation on Developing hybrid Web GTK+ applications by Alp Toker.

Trident

• Internet Explorer Architecture, on MSDN