Download files from different directories with wildcards using wget?

Question

I need to download all files with "h11v10" in the name from a http server, from different directories (DF). I tried different ways but none of them worked.

wget --http-user user --http-password pass \
  'https://e4ftl01.cr.usgs.gov/MOLT/MOD16A2.006/DF/*h11v10*'

wget --http-user user --http-password pass \
  -r -l1 --no-parent -A 'h11v10' \
  'https://e4ftl01.cr.usgs.gov/MOLT/MOD16A2.006/DF/'

How can I deal with different directories and wildcards using wget? I also tried curl but it did not work either.

Answer 1

Since you need a login to download the files, I can't test it for your situation.

So another example: if you want to download all files containing “SHA” from the directories here:

https://cdimage.debian.org/cdimage/weekly-builds/

you can use:

wget -r --no-parent 'https://cdimage.debian.org/cdimage/weekly-builds/' --accept '*SHA*'

So for your problem, it should be:

wget -r --no-parent --http-user user --http-password pass \
  'https://e4ftl01.cr.usgs.gov/MOLT/MOD16A2.006/' --accept '*h11v10*'

Does that work?

It's not very efficient since it has to download the index.html pages first.