wget¶

wget downloads files over HTTP(S) and FTP from the command line. Where curl is the tool for talking to APIs, wget is the tool for fetching things — especially large files, recursive site downloads, and mirrors. It is installed by default on most Linux systems and in Git Bash on Windows.

Installation / Setup¶

# Debian/Ubuntu
sudo apt install wget

# macOS (via Homebrew)
brew install wget

# Windows: built into Git Bash; standalone builds available
wget --version    # verify

Most Linux distributions ship with wget already installed.

Core Syntax¶

Goal	Command
Download a file	`wget URL`
Save under a different name	`wget -O model.zip URL`
Resume an interrupted download	`wget -c URL`
Follow redirects (default)	`wget URL` (redirects are followed by default)
Quiet output	`wget -q URL`
Show progress bar	`wget --show-progress URL`
Set a User-Agent	`wget -U "Mozilla/5.0" URL`
Send headers	`wget --header="Authorization: Bearer TOKEN" URL`
Retry on failure	`wget --tries=5 URL`
Timeout	`wget --timeout=30 URL`
Limit download rate	`wget --limit-rate=1m URL`

Common Operations¶

Download with progress¶

# The classic large-file download, with a progress bar
wget --show-progress https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin

# Quiet, for scripts
wget -q https://example.com/dataset.csv

Resume a broken download¶

# -c (continue) resumes from where it stopped — essential for big files
wget -c https://example.com/large-model.zip

Authentication¶

# Basic auth
wget --user=myuser --password=mypass URL

# Bearer token via header
wget --header="Authorization: Bearer YOUR_TOKEN" URL

# Cookies from a session file
wget --load-cookies cookies.txt URL

Download an entire directory recursively¶

wget -r -np -nH --cut-dirs=1 https://example.com/data/

Practical Examples¶

Example 1 — Download a model from Hugging Face¶

wget -q --show-progress \
  https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/pytorch_model.bin

Expected output:

pytorch_model.bin                 100%[===================>]  90.9M   12.3MB/s

Example 2 — Mirror a documentation site for offline reading¶

wget --mirror \
  --convert-links \
  --adjust-extension \
  --page-requisites \
  --no-parent \
  https://docs.example.com/guide/

--mirror — recursive with timestamps, no duplicates
--convert-links — rewrite links for local browsing
--page-requisites — also fetch CSS/images/JS
--no-parent — never go above the start URL

Example 3 — Download a Kaggle-style authenticated file with a token¶

wget --header="Authorization: Bearer YOUR_TOKEN" \
  -O train.csv \
  "https://api.example.com/files/train.csv?download=1"

Common Patterns¶

Resumable big-file download with retries (for training data)¶

wget -c --tries=5 --timeout=60 --show-progress \
  https://example.com/datasets/imagenet-subset.tar.gz

Batch download from a URL list¶

# urls.txt contains one URL per line
wget -i urls.txt --show-progress

Respectful mirroring (rate limit, wait between files)¶

wget --mirror --limit-rate=2m --wait=1 --random-wait https://example.com/docs/

Download with timestamp preservation¶

wget --timestamping URL   # only downloads if the remote is newer

Best Practices¶

Use -c for large files — resumes save you from restarting a 20 GB download.
Use -q in scripts, --show-progress interactively — clean CI logs, human-friendly terminals.
Respect robots and rate limits — add --wait/--limit-rate for recursive downloads of sites you don't own.
Verify downloads with checksums — after downloading, compare sha256sum file against the published hash.
Prefer HTTPS URLs — wget warns on plain HTTP; sensitive data must never travel in cleartext.

Common Mistakes¶

Mistake	Consequence	Correct approach
No `-c` on a 10 GB file	One disconnect and you restart from zero	`wget -c`
Mirroring a site without `--no-parent`	Crawls the entire domain (or worse)	Always `--no-parent` on mirrors
Downloading with `--user`/`--password` on the command line	Password visible in shell history	Use `--header` with a token, or a `~/.netrc` file with `chmod 600`
Forgetting `--show-progress` in a pipeline	Huge stderr output, hard-to-read logs	`-q` in scripts; progress only interactively
Assuming wget follows redirects to other hosts	Can fetch the wrong resource	Verify final URL with `wget --server-response`

Troubleshooting¶

Symptom	Cause	Fix
`ERROR 403: Forbidden`	Server blocks default wget User-Agent	`wget -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"` or set your own UA
`Resolving ... failed: Name or service not known`	DNS / network issue	Check connectivity: `ping` the host, verify proxy settings
Download stops mid-way	Network drop or server timeout	`wget -c --tries=5` to resume and retry
`SSL certificate problem`	Self-signed or outdated CA	Fix the cert; avoid `--no-check-certificate` except for controlled internal hosts
`File has already been retrieved`	`-O` + resume confusion	Remove the partial file or use a fresh name
Downloading HTML instead of the file	Server returned a login/redirect page	Add auth headers/cookies; check the URL with `-S` (server response)

Home — repository landing page
Central Navigation Index — find related categories
curl — the API-focused sibling; when to use which
jq.md — process the JSON you just downloaded
ssh.md — secure file transfer with scp/sftp

Official Documentation¶

Last Updated¶

2026-09-07 — Initial version.

Found a mistake or something outdated? Open an issue or submit a pull request — see CONTRIBUTING.md.