Amongst other things, you want to remove everything you don't need. Make sure the filesystem has atime fully enabled; you can set this in /etc/fstab. The current default is relatime but you want to use just atime. Every time a file is accessed, the timestamp will get updated. Then do some usage for a few days to see which files have never had their atime updated. I would do all of this in a VM, and very carefully, because I imagine there are a few files that are read when the system is in read-only mode. Note: set it to noatime once you're ready for production, otherwise you'll do a write every time you read; this is inefficient.
Though to be honest, I'd look at Damn Small Linux; do you really need to be smaller than that? Build yours based on their distro and simply remove the window manager and a few extra programs. Leave all the command-line tools, that way if you ever need to repair or reload you have the shell.
