Creating large metadata table to map out storage

This is too long for a comment, but not good enough for an answer. Sorry about that.

---What is the best way I could get the file type and size for all files, recursively? This would be a quick way to check for intermediate files which are useless

Not sure about the "best" part, but first hit on Google searching for "advanced disk usage linux" leads to this.

---I was thinking about outputting all file metadata (i.e. data user, file type, file size, timestamp) into one large tsv file, so I could subset this in Python/R based on size and file type. Is this possible? How would we go about getting this information?

With Python, sure: just use os.walk() to recurse the directory and print out all relevant information about each file. I'd probably use SQLite as an output format instead of TSV though.

---Are there methods to look at storage versus time?

You can run du -s periodically and feed the results to a RRD database. Then you can make nice graphs out of it.

---Are there any other diagnostics/fancy visualizations which could help me figure out how to organize/archive/delete the data currently stored?

There should be, but I'm skeptical about how useful they are for your particular situation.