If you're going to be reading from stdin, you should read the manpage for tcsetattr, and specifically the section about ‘Canonical and noncanonical mode’ (ICANON). Unless you disable ICANON on stdin, input is line-oriented (you get nothing until Enter is pressed). That's the case even for O_NONBLOCK.
If in doubt, strace -v stty raw >&log; stty sane; grep TCSETSW logand see what it does to stdin. You can use the same system call with identical parameters to uncook the terminal. Be careful, unless you restore the settings on exit, the terminal will be useless after your program terminates. (this is why I did stty sane after stty raw)
Using raw hardware I/O is probably a very bad idea, unless you're absolutely certain about your target hardware, and/or you want to anticipate every type of keyboard there is.
Oh, despite it being ‘raw’, you're still reading somewhat cooked input from the keyboard: you won't get scancodes, you'll get ASCII (or UTF-8, most likely). So Escape is decimal 27, not 1 (as on the original IBM PC keyboard).
Alternative suggestion that requires superuser access: read up on event devices, open the right one and read Linux event structures from it. This method returns raw press/release events and key (they're the kernel's internal key codes, though, not hardware scancodes). Check out /usr/include/linux/input.h for the specifics. Finding the right eventX device to open in 256 bytes of code is another matter altogether, of course.