1

I have an 11GB word-list file which is already sorted as each word is on its own line.

I need to remove duplicates and lines starting from 077.

I guess I need to run sed and sort -u together but I also want a live output display what's happening in terminal and if possible display the time left.

All of this in one command and it must be able to run optimally at full performance under a Live CD or possibly installed Backtrack 5 rc3.

Time is not very important but if there is a way for me to calculate the ETA, I might be able to borrow my dad's i7 based CPU which should process it faster obviously otherwise I'll have to use an older core 2 CPU.

The problem I'm facing with sort command is that under a VMware player running it live, it doesn't have enough space so I have to specify temp files on my 32GB USB by using the -T command. I guess this won't be a problem had I installed Linux.

So please give me the complete command, be it sed,sort,awk to do this (whichever is most optimal).

Improve this question
2
  • 2
    You might want to include a sample of the file, so people can see what you are dealing with. And any approaches you have tried yourself. Commented Nov 27, 2012 at 0:59
  • @jasonwryan umm how do i give a sample of 11gb file lol. Anyways just imagine the file contains 1 password on each line like a list. You would be familiar if you ever used a password dictionary file. Also i know how to run those command individually but they take a lot of time since its a huge file, i just want simply it and get the additional options listed in one command. Commented Nov 27, 2012 at 3:00

3 Answers 3

Reset to default
7

Use pv for progress. You don't need a tempfile since you're only removing text. Just overwrite the file in place. If the file is already sorted, you don't need sort -u, just uniq.

pv file | {
  uniq | grep -v '^077'
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> file

The perl line is to truncate the file at the point were grep finished to write.

Note that since you're writing the file in-place, if you make a mistake, you won't be able to go back.

Improve this answer
2
  • do i copy this whole script and paste into terminal after changing the input/output file ? I also need to install pv for this to work. Is this script any better then what Leonid has suggested below ? Because i rather not sacrifice performance. Commented Nov 27, 2012 at 11:24
  • Yes. Mine is a simplified version of Leonid's (uniq < infile | greo -v '^077' > outfile) with the addition of the progress bar (pv), and inplace editing (so you don't need extra free space). Commented Nov 27, 2012 at 12:00
2

I'm not sure you can have ETA or any progress display without significantly increasing the complexity of the sed/awk/shell script (and slowing the whole thing as a result). If you just want it to be as fast as possible, just try cat source_file | uniq | sed -n -e '/^077/!p' > dest_file. For an approximation of the progress display, you might want to watch the growing size of the dest_file with that command running in background or in another terminal.

Improve this answer
4
  • Thanks, this one looks pretty straight forward, its running uniq and then piping into sed to remove numbers starting from 077 and saving the output in the destination file. I have also heard of awk with this command: awk '!x[$0]++' is it any good compared to uniq ? Commented Nov 27, 2012 at 11:19
  • If the file is already sorted, you won't be able to beat uniq. That awk command will end up storing all unique lines in memory which for so big a file is probably not a good idea and has no benefit otherwise. That sed line is a convoluted (double-negation) way to write sed '/^077/d' though you don't need sed here as grep is more than enough. You don't need cat either (UUOC). Commented Nov 27, 2012 at 11:57
  • I completely agree with Stephane: just like it's hard to beat uniq in doing its specific job, grep shall be faster than sed in just filtering out the '^077'-s. Commented Nov 27, 2012 at 13:44
  • Thats great, i will run this script and see how it performs. Thank you Stephane and Leonid for your time and information :) Commented Nov 27, 2012 at 15:30
-1 
awk '!a[$0]++' "filename" > /tmp/dup
mv -f /tmp/dup "filename"
Improve this answer
1
  • please include a short description of what your commands do. It looks like you do not take the requirement '077' into account. Why is filename quoted? Commented Dec 24, 2017 at 10:13

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.