I am looking to add button (highlighted with yellow in the screenshot) in the header when the document is opened in SPO mode.
Is it possible through SPFX or any other way in SPO ?
The document isn't actually opened in Sharepoint, it's opened in Office Web Apps, which is the same framework that opens the docs in OneDrive for business or from Outlook.. Which is outside the SharePoint context. Hence you don't actually see the SharePoint theme applied there.