It depends on how many instructions per second you can run to read the buffer, and update pointers and clear the buffer. Even as a ISR it will consume instruction cycles.
Install the code you need to do everything else first, then use software or hardware to capture your 'spare' time in which you can manage a buffer. It only needs to be created once, plus a status byte and read and write pointers. Unless you have some built-in automation you will need to poll the buffer often to see how full it is, and compare this to how long it takes to read it and clear it. Some built-in buffers offer a half-full flag to trigger an ISR.
You start with 32 bytes and see how much 'spare' time is left. I would not go over 256 bytes/words/dwords just because of the time it takes to clear it and read from it. No CPU or MPU maker I know of goes beyond that, even for Ethernet and USB.