The solution above didn't work for me. It caused an error on boot to the effect of unw_get_proc_name failed. I don't know if this happened for anythinganyone else, but I found a different solution.
I modified /usr/share/X11/xorg.conf.d/10-nvidia.conf. For me, the file contained the following pre-edit:
Section "OutputClass"
Identifier "nvidia"
MatchDriver "nvidia-drm"
Driver "nvidia"
Option "AllowEmptyInitialConfiguration"
Option "Accel" "off"
ModulePath "/usr/lib/x86_64-linux-gnu/nvidia/xorg"
EndSection
I added a line below Option "AllowEmptyInitialConfiguration": Option "Accel" "off". Using this modification instead of the solution above, my system booted and nvidia-smi didn't show Xorg anymore.
Hope this helps somebody.