This will show you top 10 process that using the most memory:
ps aux --sort=-%mem | head -n 10
Using top: when you open top, pressing m will sort processes based on memory usage.
But this will not solve your problem, in Linux everything is either file or process. So the files you opened will eating the memory too. So this will not help.
lsof will give you all opened files with the size of the file or the file offset in bytes.