Using wget to get file names from a text file

Question

I have found a website that hosts a few files that I'm after, there's too many to download them all individually. The filenames take a fairly standard and reproduceable form i.e. 1_a, 1_b, 1_c etc

Is there a way, using the Linux command line, to use wget to automate downloading them all? I can easily put the filenames in a 1 entry per line text file and direct the command line to look up from there, but it wouldn't be the whole URL, just the bit that changes so the command would need to look something like:

wget url.com/files/(bit from file).doc sourcefile.txt

And basically be able to substitute in an entry from the sourcefile to the bit in the brackets.

Also, at one stage a large chunk (a few hundred) of the files are simply sequentially numbered, so could I use a for loop for that bit? If so, how would I do this syntactically in the command line?

Answer 1

You can use a for loop:

for x in a b c d e f g h ; do
    wget http://url.com/$x.doc
done

which can be shortened in bash to

for x in {a..h} ; do
    wget ...
done

But you can use this shorthand even without a for loop:

wget http://url.com/{a..h}.doc

If the names are not sequential and you need to read them from a file, use a while loop:

while read x ; do
    wget http://url.com/$x.doc
done < sourcefile.txt

Answer 2

Provided you have created the file with relative URLs, you can do this:

wget -i /path/to/file -B http://base_url_here

So, for example, if you want to download http://www.myfiles.com/*.txt where * can be many different things like a, a_1, ... you simply create a file with entries

a.txt
a_1.txt
a_2.txt
...

and enter the command:

wget -i /path/to/file -B http://www.myfiles.com

The -i switch instructs wget to read its links from a file and the -B switch instructs it to prepend the given argument as a base URL to all links read from the file.

Answer 3

You can use httrack to download either entire directories of files (essentially mirror everything from a site) or you can specify to httrack a filter along with specific file extensions, such as download only .pdf files.

You can read more about httrack's filter capability which is what you'd need to use if you were interested in only downloading files that were named in a specific way.

Here are some examples of the wildcard capability:

• *[file] or *[name] - any filename or name, e.g. not /,? and ; characters
• *[path] - any path (and filename), e.g. not ? and ; characters
• *[a,z,e,r,t,y] - any letters among a,z,e,r,t,y
• *[a-z] - any letters
• *[0-9,a,z,e,r,t,y] - any characters among 0..9 and a,z,e,r,t,y

Example

$ httrack http://url.com/files/ -* +1_[a-z].doc -O /dir/to/output

The switches are as follows:

• -* - remove everything from list of things to download
• +1_[a-z].doc - download files named 1_a.doc, 1_b.doc, etc.
• -O /dir/to/output - write results here