Skip to main content
Formatting: put fixed-width text in code blocks
Source Link
Michael Homer
  • 78.9k
  • 17
  • 221
  • 239

I would like to create multiple 'wget' shell scripts to download a batch of considerably large (~3GB) files using our HPC cluster. The names of the files are stored in filenames.txt, like so:

$cat filenames.txt
file1
file2
file3
...

$cat filenames.txt
file1
file2
file3
...

and the urls I would like to wget from are structured like so:
ftp: //host.com/dir1/dir2/file1/file1.sra
ftp: //host.com/dir1/dir2/file2/file2.sra
ftp: //host.com/dir1/dir2/file3/file3.sra

ftp://host.com/dir1/dir2/file1/file1.sra
ftp://host.com/dir1/dir2/file2/file2.sra
ftp://host.com/dir1/dir2/file3/file3.sra

I'd like to create a shell script for each wget and write it out into a file named as the variable itself. For example, file1.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

and file2.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

As you can see, the pattern to match is 1) The URL and 2) the filename to be written out. How would I 'append' the URL to the filename, then write it out into a .sh file named after it?

Thanks in advance, a UNIX beginner here.

I would like to create multiple 'wget' shell scripts to download a batch of considerably large (~3GB) files using our HPC cluster. The names of the files are stored in filenames.txt, like so:

$cat filenames.txt
file1
file2
file3
...

and the urls I would like to wget from are structured like so:
ftp: //host.com/dir1/dir2/file1/file1.sra
ftp: //host.com/dir1/dir2/file2/file2.sra
ftp: //host.com/dir1/dir2/file3/file3.sra

I'd like to create a shell script for each wget and write it out into a file named as the variable itself. For example, file1.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

and file2.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

As you can see, the pattern to match is 1) The URL and 2) the filename to be written out. How would I 'append' the URL to the filename, then write it out into a .sh file named after it?

Thanks in advance, a UNIX beginner here.

I would like to create multiple 'wget' shell scripts to download a batch of considerably large (~3GB) files using our HPC cluster. The names of the files are stored in filenames.txt, like so:

$cat filenames.txt
file1
file2
file3
...

and the urls I would like to wget from are structured like so:

ftp://host.com/dir1/dir2/file1/file1.sra
ftp://host.com/dir1/dir2/file2/file2.sra
ftp://host.com/dir1/dir2/file3/file3.sra

I'd like to create a shell script for each wget and write it out into a file named as the variable itself. For example, file1.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

and file2.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

As you can see, the pattern to match is 1) The URL and 2) the filename to be written out. How would I 'append' the URL to the filename, then write it out into a .sh file named after it?

Source Link

Append and write out variable-named file from list of variables

I would like to create multiple 'wget' shell scripts to download a batch of considerably large (~3GB) files using our HPC cluster. The names of the files are stored in filenames.txt, like so:

$cat filenames.txt
file1
file2
file3
...

and the urls I would like to wget from are structured like so:
ftp: //host.com/dir1/dir2/file1/file1.sra
ftp: //host.com/dir1/dir2/file2/file2.sra
ftp: //host.com/dir1/dir2/file3/file3.sra

I'd like to create a shell script for each wget and write it out into a file named as the variable itself. For example, file1.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

and file2.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

As you can see, the pattern to match is 1) The URL and 2) the filename to be written out. How would I 'append' the URL to the filename, then write it out into a .sh file named after it?

Thanks in advance, a UNIX beginner here.