You can use xdotool to get the root window geometry and the VLC window geometry. If they are equal, then VLC window is fullscreen.

gRoot=$(xdotool search --maxdepth 0 '.*' getwindowgeometry | grep 'Geometry:')
gActive=$(xdotool getactivewindow getwindowgeometry | grep 'Geometry:')
if [ "$gRoot" = "$gActive" ]; then
    echo "The active window is fullscreened."
else
    echo "The active window is not fullscreened."
fi

The first command uses --maxdepth 0 so that only the root window is searched.

You can use xdotool to get the root window geometry and the VLC window geometry. If they are equal, then VLC window is fullscreen.

gRoot=$(xdotool search --maxdepth 0 '.*' getwindowgeometry | grep 'Geometry:')
gActive=$(xdotool getactivewindow getwindowgeometry | grep 'Geometry:')
if [ "$gRoot" = "$gActive" ]; then
    echo "The active window is fullscreened."
else
    echo "The active window is not fullscreened."
fi

The first command uses --maxdepth 0 so that only the root window is searched.

A related command is xdotool getdisplaygeometry. It is undocumented in the manual but it has a mention on the changelist. Bear in mind it returns the display dimensions in the form [Width] [Height] instead of [Width]x[Height] as getwindowgeometry does, so using it requires further processing to make the equality comparison.