1

According to the man pages, when I specify --convert-links to wget:

After the download is complete, convert the links in the document to make them suitable for local viewing [...] Note that only at the end of the download can Wget know which links have been downloaded. Because of that, the work done by -k will be performed at the end of all the downloads.

Let's say I do something like wget --convert-links http://stackoverflow.com. This site is large enough that I may never complete downloading everything. In this case, will I never get my links converted over?

When I run this locally from /tmp/wget, I see a directory called stackoverflow.com with files like index.html and tour.html. But when I load up the index page and hover over the link for the tour, it takes me to file:///tour.html, instead of file:///tmp/wget/stackoverflow.com/tour.html (the actual file location).

Improve this question

1 Answer 1

Reset to default
1

Your only question seems to be:

In this case, will I never get my links converted over?

I guess not. You could try this with a partition limited in size and see what wget does when it runs out of space (or use the --quota option, presumably it will still convert with that).

I can understand why this is the strategy. If this were done on the fly, then we would have to go through every document for every item that is downloaded. Consider:

  • A.html is downloaded.
  • B.html is downloaded, now check A.html for links.
  • C.html is downloaded, now check A.html and B.html for links.
  • ...and so on.

Instead, presumably a manifest is kept and then at the end all documents are scanned once. Of course, my on-the-fly method could be improved by keeping an index of all links, but I still think it will be orders of magnitude slower -- and require orders of magnitude more frequent writing to disk. Instead of saving a file, then changing it once at the end, you would commonly be rewriting it dozens or hundreds of times.

Improve this answer
2
  • Thanks. I assumed the same thing -- that you need the whole site to efficiently parse links. So there's no solution to this as far as using wget itself? Commented Nov 13, 2014 at 15:08
  • That line in the man page seems pretty definitive. If I were going to implement an alternative, I'd have one to allow it to use intelligent guesses on-the-fly rather than wait to do it all decisively. In the general case though, that would not be desirable. You could suggest a feature to whoever maintains wget; it's mostly a matter of human resources. Commented Nov 13, 2014 at 15:21

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.