With the ps command from procps-ng, the output width is limited to the width of the terminal when it goes to a terminal and to $COLUMNS (or to the value given to --columns or --width) otherwise (like when it goes to a pipe instead in your case).
Several shells set the $COLUMNS shell variable to the width of the terminal (some of them, such as bash, only when interactive), but few of them if any export it to the environment. You can however do something like:
COLUMNS=$COLUMNS ps -Ao pid,rss,args --sort=-rss | head -n5n11
To have the COLUMNS environment variable set to value of the COLUMNS shell variable.
Or:
ps -Ao pid,rss,args --sort=-rss --width="$COLUMND"width="$COLUMNS" | head -n5n11