Not sure the issue with this will be the crypto. All of that seems fine.
Transporting the password for a user in plaintext over the internet feels weird. But there's no reason a hashed password couldn't be used as the key instead.
Depending on the type of asymmetric key used, storing an encrypted copy for each user might get a bit heavy (RSA-4096 for example). Not massively heavy, Walmart with 2.2Mil employees would be 8.8gig of duplicated encrypted key data but still.
Key rotation (of encryption/decryption key if it is ever exposed) would require every user to log in and update their encrypted store of the key. By the sound of it, this would require an invitation from a user which was part of the key rotation. That email would probably be pretty confusing to the uninitiated.
Not sure how much this really gains. The key instead could be stored in a KMS/HSM where it can never be seen again (compared to in-memory) and requests for decryption only asked for after the user has been verified as part of the tenant.