2

I am mirroring a website using wget. On the server, there is a dump of the find / command, which includes filenames in the form :

/dir1/ /dir1/page1.txt /dir1/page2.txt ...

The mirrorring is very large (100GB+) and I would like to compare the local output of find to the text file containing the server-side dump, so as to have an idea of how many files are left.

I was thinking of using a grep -v ... or awk but I'm not quite sure how one would go about doing that.

Can anyone help me please?

Thanks !

Improve this question

4 Answers 4

Reset to default
2

You can do this with comm (using bash process substitution):

comm -13 <(find / | sort) <(sort server_dump)

This will show the files which are unique to the server. For files that are unique to the local system:

comm -23 <(find / | sort) <(sort server_dump)

You may also wan to add the -xdev option to find to stop it from comparing files on other filesystems, such as /proc and /sys.

Improve this answer
3
  • This is pretty good in theory, but it doesn't work in practice – is there a way to make it more flexible ? E.g. that the files don't need to be exactly, line by line, in the same order, and for example the out put of find is in an absolute format /pathToMirrorDirectory/dir1... ; so there is a mismatch. Commented Apr 11, 2014 at 19:40
  • This ended up being the way to go. I just dupplicated the dump and appended /PathToMirrorDirectory/ before each line. Kind of ugly – but it worked. Thank you ! Commented Apr 11, 2014 at 19:50
  • @AlexandreCassagne, unfortunately sorting is necessary for comm. You can always use -printf '%P\n' with find to strip the location argument from you path or worst case add a sed process to alter the paths. It depends on the exact commands you are actually using though. Commented Apr 11, 2014 at 19:58
2

I don't think you need to use wget's recursive downloading options because you already have a list of the files that need downloading:

wget --input-file=./path/to/your/list --base=URL

In this way you would just follow the list in the first place, and so every file downloaded would be just the next in the list. So you could always find your progress by merely checking against the last file downloaded and its position in the list.

But if that doesn't work:

{ find . ; sed 's/.*/.&/' list.file ; } | 
    sort | uniq -u
Improve this answer
9
  • Thanks, but I already launched the mirror. Besides – the recursive download is useful, in case the dump wasn't updated or something ! :) Commented Apr 11, 2014 at 19:51
  • @AlexandreCassagne I put another solution up, and I'd be curious just to know if it worked. Commented Apr 11, 2014 at 20:15
  • I upvoted because this seems like a good way to tackle this, but the code doesn't run atm; after searching a bit why it seems to say the Mac OS X (which I'm trying to run this on) doesn't have the same sed as the GNU standard. But thanks :) Commented Apr 11, 2014 at 22:40
  • @AlexandreCassagne I think its because of /dev/stdin - either that or you're right and the problem is -f - which breaks the streamed model - or s:. You can - and probably should - get gnu tools for the mac - in my opinion. Commented Apr 11, 2014 at 22:43
  • @AlexandreCassagne if the process substitution in Graeme's answer works you could try: sed -f <(find /./path/from/root | sed 's:/./path/from/root/to/base/dir(.*):\\|\1|d:') ./path/to/list but you already have a working solution, so it might not be worth the effort. Commented Apr 11, 2014 at 22:51
0

Why not just dump the output of find to a file, on both hosts, and then do a diff on the two files? To get the number of files, you could count using wc.

Improve this answer
0

It isn't clear from the question if there is SSH access available to the server. If there is, I would simply use rsync for the job.

Improve this answer
1
  • Would be useful — but the mirrored server isn't under my control. This is clandestine to say the least, haha Commented Apr 13, 2014 at 16:23

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.