11.2. Download tools

11.2.1. wget

11.2.1.1. Usage

• Normal download wget http://example.com/file.iso

• Specify the save file name wget ‐‐output-document=myname.iso http://example.com/file.iso

• Save to the specified directory wget ‐‐directory-prefix=folder/subfolder http://example.com/file.iso

• Resume upload of large files wget ‐‐continue http://example.com/big.file.iso

• Download a list of urls in the specified file wget ‐‐input list-of-file-urls.txt

• Download multiple files of a specified list of numbers wget http://example.com/images/{1..20}.jpg

• Download all resources of a web page wget ‐‐page-requisites ‐‐span-hosts ‐‐convert-links ‐‐adjust-extension http://example.com/dir/file

11.2.1.2. Download whole site

• Download all linked pages and files wget ‐‐execute robots=off ‐‐recursive ‐‐no-parent ‐‐continue ‐‐no-clobber http://example.com/

• Download the file with the specified suffix wget ‐‐level=1 ‐‐recursive ‐‐no-parent ‐‐accept mp3,MP3 http://example.com/mp3/

• Exclude download from specified directory wget ‐‐recursive ‐‐no-clobber ‐‐no-parent ‐‐exclude-directories /forums,/support http://example.com

11.2.1.3. Specifying parameters

• user agent ‐‐user-agent="Mozilla/5.0 Firefox/4.0.1"

• basic auth ‐‐http-user=user ‐‐http-password=pwd

• save cookies ‐‐cookies=on ‐‐save-cookies cookies.txt ‐‐keep-session-cookies

• Use cookies ‐‐cookies=on ‐‐load-cookies cookies.txt ‐‐keep-session-cookies

11.2.2. curl

11.2.2.1. Common

• display directly curl www.example.com

• save the specified name -o newname

• no name specified -O

11.2.2.2. Regularization

• file name curl ftp://example.com/file[1-100].txt

• domain name curl http://site.{one,two,three}.com