11.8.5 is the abstract relational expression, the production relational expression is what you need to look at, which includes the abstract one.
11.8.1 The Less-than Operator ( < ) // pretty much the same as >
The production RelationalExpression : RelationalExpression < ShiftExpression is evaluated as follows:
1) Let lref be the result of evaluating RelationalExpression.
2) Let lval be GetValue(lref).
3) Let rref be the result of evaluating ShiftExpression.
4) Let rval be GetValue(rref).
--> 5) Let r be the result of performing abstract relational comparison
lval < rval. (see 11.8.5)
--> 6) If r is undefined, return false. Otherwise, return r.
When you look at the abstract relation comparison, the result will be undefined because Number('a') => NaN (Rule 3) which will set off these rules: If nx is NaN, return undefined or If ny is NaN, return undefined.
So 6) Let r be the result of performing abstract relational comparison, in your cases r is then undefined. The last step says if r is undefined then return false, hence your answer.