First off; I'm not looking to enforce this via systems; just as a community guideline.

First, as an example, here's a review that may pop up in the review queue for Close Votes. It's close voted as "broken code". Take a moment to consider why or how it is broken. I'm not going to give you any context here; there wasn't any on the real question.

import java.util.*;
public class Stack2{
    private int[] stack;
    private int size;

    public Stack2(){
        stack = new int[10];
        size = 0;
    }

    public Stack2(int height){
        if(height <= 0){
            throw new EmptyStackException();
        }
        stack = new int[height];
        size = 0;
    }

    public void add(int value){
        if (size == stack.length){
            throw new StackOverflowException();
        }
        stack[size] = value;
        size++;
    }


    public int pop(){
        if(size == 0){
            throw new EmptyStackException();
        }
        size--;
        return stack[size + 1];
    }

    public int peek(){
        if(size == 0){
            throw new EmptyStackException();
        }
        return stack[size - 1];
    }

    public boolean isEmpty(){
        if(size == 0){
            return true;
        }
        return false;
    }

    public int getSize(){
        return this.size;
    }

    class StackOverflowException extends RuntimeException{
        public StackOverflowException(){
            super("Nothing can be added to the stack. The stack was full and has overflowed");
        }

        public StackOverflowException(String message){
            super(message);
        }
    }
}

It's a javadata-structuresstack question, if that helps.

Alright. Found the bug? How long did it take?

Personally, I didn't find the bug. I took about 20 seconds to look at this code, look it through a bit. Didn't see anything obviously wrong.

The bug is, apparently, quite obvious if you test the code:

This code does not behave as expected (add(3);pop() returns 0).

(Now that we've seen this question in detail, here's the link for the question in review and here's the question itself)

The close voter has, luckily, added a comment!

It has been demonstrated in an answer, that this code does not behave as expected (add(3);pop() returns 0). As such I'm voting to close this question as off-topic because the code is not working as intended. For more information, please see the help center. Thanks - Vogel612

As a result, after I didn't see the bug on my first pass, I read the comments, including the comment containing the reason to vote to close. I scrolled back up and verified the bug, and could then vote to close with certainty. A stack that doesn't give the right items back is just broken.

However...

There are cases where someone votes to close a question because it contains broken code... and doesn't add a reason. And I'm left baffled. The code looks fine. In some of those cases I'll skip; in others I'll vote to leave open because there was no reason provided.

My question; should we make it a community guideline that if you vote to close a question based off the fact that the code is broken, you add a comment explaining why you do so?

This would allow for two things: One, to weigh by your own merits whether the bug is a major flaw or an edge case ("Your calculator fails for MAX_INT!") and two, to more easily SEE what's wrong with the code. Helps streamline reviews.