There used to be a lighter-weight way of displaying a single graphical application without the added weight of support for multiple windows and multiple applications. But since there's very little demand, I don't think there are any maintained drivers for modern PC graphics cards any more, and there are extremely few still-maintained applications that support graphics without X (or Wayland, with some graphical interface libraries). SVGAlib has not been updated in a very long time. So you'd be constrained to a slow graphical mode, possibly with less than your screen resolution. And of course you'd be constrained to applications that support a different interface.
An X server is lightweight, by the standards of a PC manufactured in the 21st century.
(If your application runs in text mode in a terminal, then this still works on modern PC hardware. But I guess you wouldn't be asking in that case.)