It is generally possible to roll a system with Busybox; busybox's web site details how to do this.
A statically linked busybox binary will require just a couple of megs of memory (over what the kernel requires, of course). I've been able to boot and log into a machine with 8M of ram.
However, it is relatively complicated to get all the system services you may require working, using a small existing distribution might be better.
How much is "little memory"? Are you on a really tiny embedded system? Unless you have less than 64M, or your process needs to use a lot of the available ram (and no swap), I'd recommend going with a minimal standard distro.
Edit: The "buildroot" tool is a companion of Busybox which helps you to build very small usable filesystems.