Marcus Summers
56f1014f73
Covers shell quoting for URLs containing &, ?, #, and other characters that Bash interprets as operators. Common gotcha when downloading from CDNs with token-based URLs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
101 lines
3.1 KiB
Markdown
101 lines
3.1 KiB
Markdown
---
|
|
title: "wget/curl: URLs with Special Characters Fail in Bash"
|
|
domain: troubleshooting
|
|
category: general
|
|
tags: [wget, curl, bash, shell, quoting, url]
|
|
status: published
|
|
created: 2026-04-08
|
|
updated: 2026-04-08
|
|
---
|
|
# wget/curl: URLs with Special Characters Fail in Bash
|
|
|
|
## Problem
|
|
|
|
Downloading a URL that contains `&`, `=`, `#`, `?`, or other shell-meaningful characters fails with cryptic errors when the URL is not properly quoted:
|
|
|
|
```bash
|
|
wget -O output.mp4 https://cdn.example.com/video%20file.mp4?secure=abc123&token=xyz
|
|
```
|
|
|
|
Bash interprets `&` as a background operator, splitting the command:
|
|
|
|
```
|
|
bash: token=xyz: command not found
|
|
```
|
|
|
|
The download either fails outright or downloads only a partial/error page (e.g., 868 bytes instead of 2 GB).
|
|
|
|
---
|
|
|
|
## Root Cause
|
|
|
|
Bash treats several URL-common characters as shell operators:
|
|
|
|
| Character | Shell Meaning | URL Meaning |
|
|
|---|---|---|
|
|
| `&` | Run previous command in background | Query parameter separator |
|
|
| `?` | Single-character glob wildcard | Start of query string |
|
|
| `#` | Comment (rest of line ignored) | Fragment identifier |
|
|
| `=` | Variable assignment (in some contexts) | Key-value separator |
|
|
| `%` | Job control (`%1`, `%2`) | URL encoding prefix |
|
|
| `!` | History expansion (in interactive shells) | Rarely used in URLs |
|
|
|
|
If the URL is unquoted, Bash processes these characters before `wget` or `curl` ever sees them.
|
|
|
|
---
|
|
|
|
## Fix
|
|
|
|
**Always single-quote URLs** passed to `wget` or `curl`:
|
|
|
|
```bash
|
|
wget -O '/path/to/output file.mp4' 'https://cdn.example.com/path/video%20file.mp4?secure=abc123&token=xyz'
|
|
```
|
|
|
|
Single quotes prevent **all** shell interpretation — no variable expansion, no globbing, no operator parsing. The URL reaches `wget` exactly as written.
|
|
|
|
### When to use double quotes instead
|
|
|
|
If the URL contains a shell variable (e.g., a token stored in `$TOKEN`), use double quotes:
|
|
|
|
```bash
|
|
wget -O output.mp4 "https://cdn.example.com/file.mp4?secure=${TOKEN}&expires=9999"
|
|
```
|
|
|
|
Double quotes allow variable expansion but still protect `&`, `?`, and `#` from shell interpretation.
|
|
|
|
### Output filename quoting
|
|
|
|
The `-O` filename also needs quoting if it contains spaces or special characters:
|
|
|
|
```bash
|
|
wget -O '/plex/plex/DF Direct Q+A #258.mp4' 'https://example.com/video.mp4'
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
| Scenario | Quoting |
|
|
|---|---|
|
|
| Static URL, no variables | Single quotes: `'https://...'` |
|
|
| URL with shell variable | Double quotes: `"https://...${VAR}"` |
|
|
| Output path with spaces | Single or double quotes around `-O` path |
|
|
| URL in a script variable | Assign with double quotes: `URL="https://..."`, then `wget "$URL"` |
|
|
|
|
---
|
|
|
|
## Common Symptoms of Unquoted URLs
|
|
|
|
- `bash: <partial-url>: command not found` — `&` split the command
|
|
- Download completes instantly with a tiny file (error page, not the real content)
|
|
- `wget` reports success but the file is corrupt or truncated
|
|
- `No such file or directory` errors on URL fragments
|
|
- History expansion errors (`!` in URL triggers `bash: !...: event not found`)
|
|
|
|
---
|
|
|
|
## See Also
|
|
|
|
- [Bash Scripting Patterns](../01-linux/shell-scripting/bash-scripting-patterns.md) — general shell quoting and safety patterns
|