The second statement says Select records when student name matches @name or when no name was specified If @name is specified, select for that name. If @name is empty, select all records. This doesn't care about what value is in the table when @name is NULL
For the first statement, if @name is NULL, it will use student.name which is the same as the current row's student.name
When both @name and student.name are NULL
WHERE student.name = @name OR @name IS NULL -- the other part of the OR is true so records will be returned
and
WHERE student.name = ISNULL(@name, student.name) becomes WHERE NULL = NULL
because NULL = NULL returns false, no records will be returned for that row