From 56f1014f73148be799494da7f47d43d8561c20ff Mon Sep 17 00:00:00 2001 From: Marcus Summers Date: Wed, 8 Apr 2026 10:18:34 -0400 Subject: [PATCH] Add troubleshooting article: wget/curl URLs with special characters Covers shell quoting for URLs containing &, ?, #, and other characters that Bash interprets as operators. Common gotcha when downloading from CDNs with token-based URLs. Co-Authored-By: Claude Opus 4.6 (1M context) --- 05-troubleshooting/index.md | 3 +- .../wget-url-special-characters.md | 100 ++++++++++++++++++ README.md | 8 +- SUMMARY.md | 5 + index.md | 8 +- 5 files changed, 117 insertions(+), 7 deletions(-) create mode 100644 05-troubleshooting/wget-url-special-characters.md diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index 56d5dc1..d50be83 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -1,6 +1,6 @@ --- created: 2026-03-15T06:37 -updated: 2026-04-07T10:48 +updated: 2026-04-08 --- # 🔧 General Troubleshooting @@ -17,6 +17,7 @@ Practical fixes for common Linux, networking, and application problems. - [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) - [ISP SNI Filtering & Caddy](isp-sni-filtering-caddy.md) - [yt-dlp YouTube JS Challenge Fix](yt-dlp-fedora-js-challenge.md) +- [wget/curl: URLs with Special Characters Fail in Bash](wget-url-special-characters.md) ## ⚙️ Ansible & Fleet Management - [SSH Timeout During dnf upgrade on Fedora Hosts](ansible-ssh-timeout-dnf-upgrade.md) diff --git a/05-troubleshooting/wget-url-special-characters.md b/05-troubleshooting/wget-url-special-characters.md new file mode 100644 index 0000000..09e4ba6 --- /dev/null +++ b/05-troubleshooting/wget-url-special-characters.md @@ -0,0 +1,100 @@ +--- +title: "wget/curl: URLs with Special Characters Fail in Bash" +domain: troubleshooting +category: general +tags: [wget, curl, bash, shell, quoting, url] +status: published +created: 2026-04-08 +updated: 2026-04-08 +--- +# wget/curl: URLs with Special Characters Fail in Bash + +## Problem + +Downloading a URL that contains `&`, `=`, `#`, `?`, or other shell-meaningful characters fails with cryptic errors when the URL is not properly quoted: + +```bash +wget -O output.mp4 https://cdn.example.com/video%20file.mp4?secure=abc123&token=xyz +``` + +Bash interprets `&` as a background operator, splitting the command: + +``` +bash: token=xyz: command not found +``` + +The download either fails outright or downloads only a partial/error page (e.g., 868 bytes instead of 2 GB). + +--- + +## Root Cause + +Bash treats several URL-common characters as shell operators: + +| Character | Shell Meaning | URL Meaning | +|---|---|---| +| `&` | Run previous command in background | Query parameter separator | +| `?` | Single-character glob wildcard | Start of query string | +| `#` | Comment (rest of line ignored) | Fragment identifier | +| `=` | Variable assignment (in some contexts) | Key-value separator | +| `%` | Job control (`%1`, `%2`) | URL encoding prefix | +| `!` | History expansion (in interactive shells) | Rarely used in URLs | + +If the URL is unquoted, Bash processes these characters before `wget` or `curl` ever sees them. + +--- + +## Fix + +**Always single-quote URLs** passed to `wget` or `curl`: + +```bash +wget -O '/path/to/output file.mp4' 'https://cdn.example.com/path/video%20file.mp4?secure=abc123&token=xyz' +``` + +Single quotes prevent **all** shell interpretation — no variable expansion, no globbing, no operator parsing. The URL reaches `wget` exactly as written. + +### When to use double quotes instead + +If the URL contains a shell variable (e.g., a token stored in `$TOKEN`), use double quotes: + +```bash +wget -O output.mp4 "https://cdn.example.com/file.mp4?secure=${TOKEN}&expires=9999" +``` + +Double quotes allow variable expansion but still protect `&`, `?`, and `#` from shell interpretation. + +### Output filename quoting + +The `-O` filename also needs quoting if it contains spaces or special characters: + +```bash +wget -O '/plex/plex/DF Direct Q+A #258.mp4' 'https://example.com/video.mp4' +``` + +--- + +## Quick Reference + +| Scenario | Quoting | +|---|---| +| Static URL, no variables | Single quotes: `'https://...'` | +| URL with shell variable | Double quotes: `"https://...${VAR}"` | +| Output path with spaces | Single or double quotes around `-O` path | +| URL in a script variable | Assign with double quotes: `URL="https://..."`, then `wget "$URL"` | + +--- + +## Common Symptoms of Unquoted URLs + +- `bash: : command not found` — `&` split the command +- Download completes instantly with a tiny file (error page, not the real content) +- `wget` reports success but the file is corrupt or truncated +- `No such file or directory` errors on URL fragments +- History expansion errors (`!` in URL triggers `bash: !...: event not found`) + +--- + +## See Also + +- [Bash Scripting Patterns](../01-linux/shell-scripting/bash-scripting-patterns.md) — general shell quoting and safety patterns diff --git a/README.md b/README.md index 0a6ffea..a3a69a6 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,8 @@ updated: 2026-04-07T21:59 > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > -**Last updated:** 2026-04-03 -**Article count:** 76 +**Last updated:** 2026-04-08 +**Article count:** 74 ## Domains @@ -17,7 +17,7 @@ updated: 2026-04-07T21:59 | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 21 | | 🔓 Open Source Tools | `03-opensource/` | 10 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 2 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 26 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 29 | --- @@ -152,6 +152,7 @@ updated: 2026-04-07T21:59 - [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md) — blocking directory scanners and junk HTTP methods - [ClamAV Safe Scheduling on Live Servers](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — preventing clamscan CPU spikes with nice and ionice - [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md) — fixing session-cN.scope failures during login +- [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) — fixing broken downloads caused by unquoted URLs with &, ?, # characters --- @@ -159,6 +160,7 @@ updated: 2026-04-07T21:59 | Date | Article | Domain | |---|---|---| +| 2026-04-08 | [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) | Troubleshooting | | 2026-04-07 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | | 2026-04-07 | [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) | Troubleshooting | | 2026-04-07 | [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md) | Troubleshooting | diff --git a/SUMMARY.md b/SUMMARY.md index 6d22432..ba10bb1 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,6 +1,10 @@ --- created: 2026-04-02T16:03 +<<<<<<< HEAD updated: 2026-04-07T10:48 +======= +updated: 2026-04-08 +>>>>>>> 4dc77d4 (Add troubleshooting article: wget/curl URLs with special characters) --- * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) @@ -81,3 +85,4 @@ updated: 2026-04-07T10:48 * [Ansible: SSH Timeout During dnf upgrade on Fedora Hosts](05-troubleshooting/ansible-ssh-timeout-dnf-upgrade.md) * [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md) * [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md) + * [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) diff --git a/index.md b/index.md index 0c8614b..d0de3c7 100644 --- a/index.md +++ b/index.md @@ -6,8 +6,8 @@ updated: 2026-04-07T21:59 > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > -> **Last updated:** 2026-04-07 -> **Article count:** 73 +> **Last updated:** 2026-04-08 +> **Article count:** 74 ## Domains @@ -17,7 +17,7 @@ updated: 2026-04-07T21:59 | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 21 | | 🔓 Open Source Tools | `03-opensource/` | 10 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 2 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 28 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 29 | --- @@ -155,6 +155,7 @@ updated: 2026-04-07T21:59 - [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md) — blocking directory scanners and junk HTTP methods - [ClamAV Safe Scheduling on Live Servers](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — preventing clamscan CPU spikes with nice and ionice - [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md) — fixing session-cN.scope failures during login +- [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) — fixing broken downloads caused by unquoted URLs with &, ?, # characters --- @@ -163,6 +164,7 @@ updated: 2026-04-07T21:59 | Date | Article | Domain | |---|---|---| +| 2026-04-08 | [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) | Troubleshooting | | 2026-04-07 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | | 2026-04-07 | [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) | Troubleshooting | | 2026-04-07 | [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md) | Troubleshooting |