From 639b23f861a3401b3ab593cb7deee3c7653c95b9 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 01:31:25 -0400 Subject: [PATCH 01/12] vault backup: 2026-03-13 01:31:25 --- ...cker-caddy-selinux-post-reboot-recovery.md | 135 +++++++++++++ 05-troubleshooting/index.md | 1 + 05-troubleshooting/isp-sni-filtering-caddy.md | 139 ++----------- .../fail2ban-self-ban-apache-outage.md | 186 ++++++++++++++++++ .../yt-dlp-fedora-js-challenge.md | 39 ++++ MajorWiki-Deploy-Status.md | 20 +- README.md | 81 ++++---- SUMMARY.md | 15 +- index.md | 10 +- 9 files changed, 446 insertions(+), 180 deletions(-) create mode 100644 05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md create mode 100644 05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md diff --git a/05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md b/05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md new file mode 100644 index 0000000..a54a7a6 --- /dev/null +++ b/05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md @@ -0,0 +1,135 @@ +# Docker & Caddy Recovery After Reboot (Fedora + SELinux) + +## 🛑 Problem + +After a system reboot on **majorlab** (Fedora 43, SELinux Enforcing), Docker containers and all Caddy-proxied services become unreachable. Browsers may show connection errors or 502 Bad Gateway responses. + +## 🔍 Diagnosis + +Three separate failures occur in sequence: + +### 1. Docker fails to start + +```bash +systemctl status docker.service +# → Active: inactive (dead) +# → Dependency failed for docker.service + +systemctl status docker.socket +# → Active: failed (Result: resources) +# → Failed to create listening socket (/run/docker.sock): Invalid argument +``` + +**Cause:** `docker.socket` is disabled, so Docker's socket activation fails and `docker.service` never starts. All containers are down. + +--- + +### 2. Caddy fails to bind ports + +```bash +journalctl -u caddy -n 20 +# → Error: listen tcp :4443: bind: permission denied +# → Error: listen tcp :8448: bind: permission denied +``` + +**Cause:** SELinux's `http_port_t` type does not include ports `4443` (Tailscale HTTPS) or `8448` (Matrix federation), so Caddy is denied when trying to bind them. + +--- + +### 3. Caddy returns 502 Bad Gateway + +Even after Caddy starts, all reverse proxied services return 502. + +```bash +journalctl -u caddy | grep "permission denied" +# → dial tcp 127.0.0.1:: connect: permission denied +``` + +**Cause:** The SELinux boolean `httpd_can_network_connect` is off, preventing Caddy from making outbound connections to upstream services. + +--- + +## ✅ Solution + +### Step 1 — Re-enable and start Docker + +```bash +sudo systemctl enable docker.socket +sudo systemctl start docker.socket +sudo systemctl start docker.service +``` + +Verify containers are up: + +```bash +sudo docker ps -a +``` + +--- + +### Step 2 — Add missing ports to SELinux http_port_t + +```bash +sudo semanage port -m -t http_port_t -p tcp 4443 +sudo semanage port -a -t http_port_t -p tcp 8448 +``` + +Verify: + +```bash +sudo semanage port -l | grep http_port_t +# Should include 4443 and 8448 +``` + +--- + +### Step 3 — Enable httpd_can_network_connect + +```bash +sudo setsebool -P httpd_can_network_connect on +``` + +The `-P` flag makes this persistent across reboots. + +--- + +### Step 4 — Start Caddy + +```bash +sudo systemctl restart caddy +systemctl is-active caddy +# → active +``` + +--- + +## 🔁 Why This Happens + +| Issue | Root Cause | +|---|---| +| Docker down | `docker.socket` was disabled (not just stopped) — survives reboots until explicitly enabled | +| Port bind denied | SELinux requires non-standard ports to be explicitly added to `http_port_t` — this is not automatic on upgrades or reinstalls | +| 502 on all proxied services | `httpd_can_network_connect` defaults to `off` on Fedora — must be set once per installation | + +--- + +## 🔎 Quick Diagnostic Commands + +```bash +# Check Docker +systemctl status docker.socket docker.service +sudo docker ps -a + +# Check Caddy +systemctl status caddy +journalctl -u caddy -n 30 + +# Check SELinux booleans +getsebool httpd_can_network_connect + +# Check allowed HTTP ports +sudo semanage port -l | grep http_port_t + +# Test upstream directly (bypass Caddy) +curl -sv http://localhost:8086 +``` diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index 3d44700..9bfc177 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -6,3 +6,4 @@ Practical fixes for common Linux, networking, and application problems. - [Obsidian Cache Hang Recovery](obsidian-cache-hang-recovery.md) - [yt-dlp Fedora JS Challenge](yt-dlp-fedora-js-challenge.md) - [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) +- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) diff --git a/05-troubleshooting/isp-sni-filtering-caddy.md b/05-troubleshooting/isp-sni-filtering-caddy.md index 81de28f..26701dc 100644 --- a/05-troubleshooting/isp-sni-filtering-caddy.md +++ b/05-troubleshooting/isp-sni-filtering-caddy.md @@ -1,129 +1,22 @@ ---- -title: ISP SNI Filtering Blocking Caddy Reverse Proxy -domain: troubleshooting -category: networking -tags: - - caddy - - tls - - sni - - isp - - google-fiber - - reverse-proxy - - troubleshooting -status: published -created: '2026-03-11' -updated: '2026-03-11' ---- +# ISP SNI Filtering & Caddy Troubleshooting -# ISP SNI Filtering Blocking Caddy Reverse Proxy +## 🛑 Problem +When deploying the MajorWiki at `wiki.majorshouse.com`, the site was unreachable over HTTPS. Browsers reported a `TLS_CONNECTION_REFUSED` error. -Some ISPs — including Google Fiber — silently block TLS handshakes for certain hostnames at the network level. The connection reaches your server, TCP completes, but the TLS handshake never finishes. The symptom looks identical to a misconfigured Caddy setup or a missing certificate, which makes it a frustrating thing to debug. +## 🔍 Diagnosis +1. **Direct IP Check:** Accessing the server via IP on port 8092 worked fine. +2. **Tailscale Check:** Accessing via the Tailscale magic DNS worked fine. +3. **SNI Analysis:** Using `openssl s_client -connect :443 -servername wiki.majorshouse.com` resulted in an immediate reset by peer. +4. **Root Cause:** Google Fiber (the local ISP) appears to be performing SNI-based filtering on hostnames containing the string "wiki". -## What Happened +## ✅ Solution +The domain was changed from `wiki.majorshouse.com` to `notes.majorshouse.com`. -Deployed a new Caddy vhost for `wiki.majorshouse.com` on a Google Fiber residential connection. Everything on the server was correct: - -- Let's Encrypt cert provisioned successfully -- Caddy validated clean with `caddy validate` -- `curl --resolve wiki.majorshouse.com:443:127.0.0.1 https://wiki.majorshouse.com` returned 200 from loopback -- iptables had ACCEPT rules for ports 80 and 443 -- All other Caddy vhosts on the same IP and port worked fine externally - -But from any external host, `curl` timed out with no response. `ss -tn` showed SYN-RECV connections piling up on port 443 — the TCP handshake was completing, but the TLS handshake was stalling. - -## The Debugging Sequence - -**Step 1: Ruled out Caddy config issues** - -```bash -caddy validate --config /etc/caddy/Caddyfile -curl --resolve wiki.majorshouse.com:443:127.0.0.1 https://wiki.majorshouse.com +### Caddy Configuration Update +```caddy +notes.majorshouse.com { + reverse_proxy :8092 +} ``` -Both clean. Loopback returned 200. - -**Step 2: Ruled out certificate issues** - -```bash -ls /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/wiki.majorshouse.com/ -openssl x509 -in wiki.majorshouse.com.crt -noout -text | grep -E "Subject:|Not Before|Not After" -``` - -Valid cert, correct subject, not expired. - -**Step 3: Ruled out firewall** - -```bash -iptables -L INPUT -n -v | grep -E "80|443" -ss -tlnp | grep ':443' -``` - -Ports open, Caddy listening on `*:443`. - -**Step 4: Ruled out hairpin NAT** - -Testing `curl https://wiki.majorshouse.com` from the server itself returned "No route to host" — the server can't reach its own public IP. This is normal for residential connections without NAT loopback. It's not the problem. - -**Step 5: Confirmed external connectivity on port 443** - -```bash -# From an external server (majormail) -curl -sk -o /dev/null -w "%{http_code}" https://git.majorshouse.com # 200 -curl -sk -o /dev/null -w "%{http_code}" https://wiki.majorshouse.com # 000 -``` - -Same IP, same port, same Caddy process. `git` works, `wiki` doesn't. - -**Step 6: Tested a different subdomain** - -Added `notes.majorshouse.com` as a new Caddyfile entry pointing to the same upstream. Cert provisioned via HTTP-01 challenge successfully (proving port 80 is reachable). Then: - -```bash -curl -sk -o /dev/null -w "%{http_code}" https://notes.majorshouse.com # 200 -curl -sk -o /dev/null -w "%{http_code}" https://wiki.majorshouse.com # 000 -``` - -`notes` worked immediately. `wiki` still timed out. - -**Conclusion:** Google Fiber is performing SNI-based filtering and blocking TLS connections where the ClientHello contains `wiki.majorshouse.com` as the server name. - -## The Fix - -Rename the subdomain. Use anything that doesn't trigger the filter. `notes.majorshouse.com` works fine. - -```bash -# Remove the blocked entry -sed -i '/^wiki\.majorshouse\.com/,/^}/d' /etc/caddy/Caddyfile -systemctl reload caddy -``` - -Update `mkdocs.yml` or whatever service's config references the domain, add DNS for the new subdomain, and done. - -## How to Diagnose This Yourself - -If your Caddy vhost works on loopback but times out externally: - -1. Confirm other vhosts on the same IP and port work externally -2. Test the specific domain from multiple external networks (different ISP, mobile data) -3. Add a second vhost with a different subdomain pointing to the same upstream -4. If the new subdomain works and the original doesn't, the hostname is being filtered - -```bash -# Quick external test — run from a server outside your network -curl -sk -o /dev/null -w "%{http_code}" --max-time 10 https://your-domain.com -``` - -If you get `000` (connection timeout, not a TLS error like `curl: (35)`), the TCP connection isn't completing — pointing to network-level blocking rather than a Caddy or cert issue. - -## Gotchas & Notes - -- **`curl: (35) TLS error` is different from `000`.** A TLS error means TCP connected but the handshake failed — usually a missing or invalid cert. A `000` timeout means TCP never completed — a network or firewall issue. -- **SYN-RECV in `ss -tn` means TCP is partially open.** If you see SYN-RECV entries for your domain but the connection never moves to ESTAB, something between the client and your TLS stack is dropping the handshake. -- **ISP SNI filtering is uncommon but real.** Residential ISPs sometimes filter on SNI for terms associated with piracy, proxies, or certain categories of content. "Wiki" may trigger a content-type heuristic. -- **Loopback testing isn't enough.** Always test from an external host before declaring a service working. The server can't test its own public IP on most residential connections. - -## See Also - -- [[setting-up-caddy-reverse-proxy]] -- [[linux-server-hardening-checklist]] -- [[tailscale-homelab-remote-access]] +Once the hostname was changed to one without the "wiki" keyword, the TLS handshake completed successfully. diff --git a/05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md b/05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md new file mode 100644 index 0000000..35d8593 --- /dev/null +++ b/05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md @@ -0,0 +1,186 @@ +# Apache Outage: Fail2ban Self-Ban + Missing iptables Rules + +## 🛑 Problem + +A web server running Apache2 becomes completely unreachable (`ERR_CONNECTION_TIMED_OUT`) despite Apache running normally. SSH access via Tailscale is unaffected. + +--- + +## 🔍 Diagnosis + +### Step 1 — Confirm Apache is running + +```bash +sudo systemctl status apache2 +``` + +If Apache is `active (running)`, the problem is at the firewall layer, not the application. + +--- + +### Step 2 — Test the public IP directly + +```bash +curl -I --max-time 5 http:// +``` + +A **timeout** means traffic is being dropped by the firewall. A **connection refused** means Apache is down. + +--- + +### Step 3 — Check the iptables INPUT chain + +```bash +sudo iptables -L INPUT -n -v +``` + +Look for ACCEPT rules on ports 80 and 443. If they're missing and the chain policy is `DROP`, HTTP/HTTPS traffic is being silently dropped. + +**Example of broken state:** +``` +Chain INPUT (policy DROP) + ACCEPT tcp -- lo * ... # loopback only + ACCEPT tcp -- tailscale0 * ... tcp dpt:22 + # no rules for port 80 or 443 +``` + +--- + +### Step 4 — Check the nftables ruleset for Fail2ban + +```bash +sudo nft list tables +``` + +Look for `table inet f2b-table` — this is Fail2ban's nftables table. It operates at **priority `filter - 1`**, meaning it is evaluated *before* the main iptables INPUT chain. + +```bash +sudo nft list ruleset | grep -A 10 'f2b-table' +``` + +Fail2ban rejects banned IPs with rules like: +``` +tcp dport { 80, 443 } ip saddr @addr-set-wordpress-hard reject with icmp port-unreachable +``` + +A banned admin IP will be rejected here regardless of any ACCEPT rules downstream. + +--- + +### Step 5 — Check if your IP is banned + +```bash +for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do + echo "=== $jail ==="; sudo fail2ban-client get $jail banip | tr ',' '\n' | grep +done +``` + +--- + +## ✅ Solution + +### Fix 1 — Add missing iptables ACCEPT rules for HTTP/HTTPS + +If ports 80/443 are absent from the INPUT chain: + +```bash +sudo iptables -I INPUT -i eth0 -p tcp --dport 80 -j ACCEPT +sudo iptables -I INPUT -i eth0 -p tcp --dport 443 -j ACCEPT +``` + +Persist the rules: + +```bash +sudo netfilter-persistent save +``` + +If `netfilter-persistent` is not installed: + +```bash +sudo apt install -y iptables-persistent +sudo netfilter-persistent save +``` + +--- + +### Fix 2 — Unban your IP from all Fail2ban jails + +```bash +for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do + sudo fail2ban-client set $jail unbanip 2>/dev/null && echo "Unbanned from $jail" +done +``` + +--- + +### Fix 3 — Add your IP to Fail2ban's ignore list + +Edit `/etc/fail2ban/jail.local`: + +```bash +sudo nano /etc/fail2ban/jail.local +``` + +Add or update the `[DEFAULT]` section: + +```ini +[DEFAULT] +ignoreip = 127.0.0.1/8 ::1 +``` + +Restart Fail2ban: + +```bash +sudo systemctl restart fail2ban +``` + +--- + +## 🔁 Why This Happens + +| Issue | Root Cause | +|---|---| +| Missing port 80/443 rules | iptables INPUT chain left incomplete after a manual firewall rework (e.g., SSH lockdown) | +| Still blocked after adding iptables rules | Fail2ban uses a separate nftables table at higher priority — iptables ACCEPT rules are never reached for banned IPs | +| Admin IP gets banned | Automated WordPress/Apache probes trigger Fail2ban jails against the admin's own IP | + +--- + +## ⚠️ Key Architecture Note + +On servers running both iptables and Fail2ban, the evaluation order is: + +1. **`inet f2b-table`** (nftables, priority `filter - 1`) — Fail2ban ban sets; evaluated first +2. **`ip filter` INPUT chain** (iptables/nftables, policy DROP) — explicit ACCEPT rules +3. **UFW chains** — IP-specific rules; evaluated last + +A banned IP is stopped at step 1 and never reaches the ACCEPT rules in step 2. Always check Fail2ban *after* confirming iptables looks correct. + +--- + +## 🔎 Quick Diagnostic Commands + +```bash +# Check Apache +sudo systemctl status apache2 + +# Test public connectivity +curl -I --max-time 5 http:// + +# Check iptables INPUT chain +sudo iptables -L INPUT -n -v + +# List nftables tables (look for inet f2b-table) +sudo nft list tables + +# Check Fail2ban jail status +sudo fail2ban-client status + +# Check a specific jail's banned IPs +sudo fail2ban-client status wordpress-hard + +# Unban an IP from all jails +for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do + sudo fail2ban-client set $jail unbanip 2>/dev/null && echo "Unbanned from $jail" +done +``` diff --git a/05-troubleshooting/yt-dlp-fedora-js-challenge.md b/05-troubleshooting/yt-dlp-fedora-js-challenge.md index e998e47..648580b 100644 --- a/05-troubleshooting/yt-dlp-fedora-js-challenge.md +++ b/05-troubleshooting/yt-dlp-fedora-js-challenge.md @@ -135,3 +135,42 @@ This is a YouTube-side experiment. yt-dlp falls back to other clients automatica yt-dlp --version pip show yt-dlp ``` + +### Format Not Available: Strict AVC+M4A Selector + +The format selector `bestvideo[vcodec^=avc]+bestaudio[ext=m4a]` will hard-fail if YouTube doesn't serve H.264 (AVC) video for a given video: + +``` +ERROR: [youtube] Requested format is not available. Use --list-formats for a list of available formats +``` + +This is separate from the n-challenge issue — the format simply doesn't exist for that video (common with newer uploads that are VP9/AV1-only). + +**Fix 1 — Relax the selector to mp4 container without enforcing codec:** + +```bash +yt-dlp -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio' \ + --merge-output-format mp4 \ + -o "/plex/plex/%(title)s.%(ext)s" \ + --write-auto-subs --embed-subs \ + https://youtu.be/VIDEO_ID +``` + +**Fix 2 — Let yt-dlp pick best and re-encode to H.264 via ffmpeg (Plex-safe, slower):** + +```bash +yt-dlp -f 'bestvideo+bestaudio' \ + --merge-output-format mp4 \ + --recode-video mp4 \ + -o "/plex/plex/%(title)s.%(ext)s" \ + --write-auto-subs --embed-subs \ + https://youtu.be/VIDEO_ID +``` + +Use `--recode-video mp4` when Plex direct play is required and the source stream may be VP9/AV1. Requires ffmpeg. + +**Inspect available formats first:** + +```bash +yt-dlp --list-formats https://youtu.be/VIDEO_ID +``` diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index e2f6f00..cd03aca 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -2,7 +2,7 @@ title: MajorWiki Deployment Status status: deployed project: MajorTwin -updated: '2026-03-11' +updated: '2026-03-12' --- # MajorWiki Deployment Status @@ -31,9 +31,9 @@ DNS record and Caddy entry have been removed. ## Content -- 15 articles across 4 domains +- 23 articles across 5 domains - Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/` -- Deployed via rsync from MajorRig WSL2 +- Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab) ## Update Workflow @@ -48,5 +48,15 @@ rsync -av --include="*.md" --include="*/" --exclude="*" \ ## Backlog -- [ ] Gitea webhook for auto-deploy on push -- [ ] Add `03-opensource` and `05-troubleshooting` articles +- [✅] Gitea webhook for auto-deploy on push (Active as of 2026-03-11) +- [✅] Git repo initialized on MajorAir — push to deploy (2026-03-12) +- [✅] Troubleshooting section seeded with articles +- [ ] Add `03-opensource` articles + +## Incident Log + +### 2026-03-11 — 502 Error & Webhook Repair +- **Issue:** Wiki returned 502 Bad Gateway. +- **Cause:** `majwiki` container was in a restart loop due to a `LiterateNavParseError` in `SUMMARY.md` (invalid globbing syntax). +- **Repair:** Rewrote `SUMMARY.md` with explicit links. Repaired `majwiki-webhook.service` by killing an orphaned process and correcting the Caddy reverse proxy routing for `/webhook`. +- **Result:** Site live and auto-deployment functional. diff --git a/README.md b/README.md index d9fd861..e9f7697 100644 --- a/README.md +++ b/README.md @@ -1,70 +1,67 @@ - # MajorLinux Tech Wiki — Index > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > -> **Last updated:** 2026-03-11 -> **Article count:** 20 - ---- +> **Last updated:** 2026-03-12 +> **Article count:** 22 ## Domains | Domain | Folder | Articles | |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 8 | -| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 7 | +| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | | 🔓 Open Source Tools | `03-opensource/` | 0 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 4 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 5 | --- ## 🐧 Linux & Sysadmin ### Files & Permissions -- [[linux-file-permissions]] — chmod, chown, special bits, finding permission problems +- [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md) — chmod, chown, special bits, finding permission problems ### Process Management -- [[managing-linux-services-systemd-ansible]] — systemctl, journalctl, writing service files, Ansible service management +- [Managing Linux Services with systemd](01-linux/process-management/managing-linux-services-systemd-ansible.md) — systemctl, journalctl, writing service files, Ansible service management ### Networking -- [[ssh-config-key-management]] — key generation, ssh-copy-id, ~/.ssh/config, managing multiple keys +- [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) — key generation, ssh-copy-id, ~/.ssh/config, managing multiple keys ### Package Management -- [[package-management-reference]] — apt, dnf, pacman side-by-side reference, Flatpak/Snap +- [Package Management Reference](01-linux/packages/package-management-reference.md) — apt, dnf, pacman side-by-side reference, Flatpak/Snap ### Shell & Scripting -- [[ansible-getting-started]] — inventory, ad-hoc commands, playbooks, handlers, roles -- [[bash-scripting-patterns]] — set -euo pipefail, logging, error handling, argument parsing, common patterns +- [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) — inventory, ad-hoc commands, playbooks, handlers, roles +- [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) — set -euo pipefail, logging, error handling, argument parsing, common patterns ### Distro-Specific -- [[linux-distro-guide-beginners]] — Ubuntu recommendation, distro comparison, desktop environments -- [[wsl2-instance-migration-fedora43]] — moving WSL2 VHDX from C: to another drive +- [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) — Ubuntu recommendation, distro comparison, desktop environments +- [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md) — moving WSL2 VHDX from C: to another drive --- ## 🏠 Self-Hosting & Homelab ### Docker & Containers -- [[self-hosting-starter-guide]] — hardware options, Docker install, first services, networking basics -- [[docker-vs-vms-homelab]] — when to use containers vs VMs, KVM setup, how to run both -- [[debugging-broken-docker-containers]] — logs, inspect, exec, port conflicts, permission errors +- [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) — hardware options, Docker install, first services, networking basics +- [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md) — when to use containers vs VMs, KVM setup, how to run both +- [Debugging Broken Docker Containers](02-selfhosting/docker/debugging-broken-docker-containers.md) — logs, inspect, exec, port conflicts, permission errors ### Reverse Proxies -- [[setting-up-caddy-reverse-proxy]] — Caddyfile basics, automatic HTTPS, local TLS, DNS challenge +- [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) — Caddyfile basics, automatic HTTPS, local TLS, DNS challenge ### DNS & Networking -- [[tailscale-homelab-remote-access]] — installation, MagicDNS, making services accessible, subnet router, ACLs +- [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) — installation, MagicDNS, making services accessible, subnet router, ACLs ### Storage & Backup -- [[rsync-backup-patterns]] — flags reference, remote backup, incremental with hard links, cron/systemd +- [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) — flags reference, remote backup, incremental with hard links, cron/systemd ### Monitoring -- [[tuning-netdata-web-log-alerts]] — tuning web_log_1m_redirects threshold for HTTPS-forcing servers +- [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md) — tuning web_log_1m_redirects threshold for HTTPS-forcing servers ### Security -- [[linux-server-hardening-checklist]] — non-root user, SSH key auth, sshd_config, firewall, fail2ban +- [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban --- @@ -77,16 +74,17 @@ ## 🎙️ Streaming & Podcasting ### OBS Studio -- [[obs-studio-setup-encoding]] — installation, NVENC/x264 settings, scene setup, audio filters, Linux Wayland notes +- [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) — installation, NVENC/x264 settings, scene setup, audio filters, Linux Wayland notes --- ## 🔧 General Troubleshooting -- [[isp-sni-filtering-caddy]] — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber -- [[obsidian-cache-hang-recovery]] — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts -- [[yt-dlp-fedora-js-challenge]] — fixing YouTube JS challenge solver errors and missing formats on Fedora -- [[majwiki-setup-and-pipeline]] — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline +- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot +- [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber +- [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts +- [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora +- [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline --- @@ -94,19 +92,20 @@ | Date | Article | Domain | |---|---|---| -| 2026-03-11 | [[majwiki-setup-and-pipeline]] | Troubleshooting | -| 2026-03-11 | [[obsidian-cache-hang-recovery]] | Troubleshooting | -| 2026-03-11 | [[yt-dlp-fedora-js-challenge]] | Troubleshooting | -| 2026-03-08 | [[obs-studio-setup-encoding]] | Streaming | -| 2026-03-08 | [[linux-file-permissions]] | Linux | -| 2026-03-08 | [[rsync-backup-patterns]] | Self-Hosting | -| 2026-03-08 | [[tailscale-homelab-remote-access]] | Self-Hosting | -| 2026-03-08 | [[package-management-reference]] | Linux | -| 2026-03-08 | [[bash-scripting-patterns]] | Linux | -| 2026-03-08 | [[setting-up-caddy-reverse-proxy]] | Self-Hosting | -| 2026-03-08 | [[ssh-config-key-management]] | Linux | -| 2026-03-08 | [[ansible-getting-started]] | Linux | -| 2026-03-08 | [[self-hosting-starter-guide]] | Self-Hosting | +| 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | +| 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | +| 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | +| 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | +| 2026-03-08 | [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) | Streaming | +| 2026-03-08 | [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md) | Linux | +| 2026-03-08 | [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) | Self-Hosting | +| 2026-03-08 | [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) | Self-Hosting | +| 2026-03-08 | [Package Management Reference](01-linux/packages/package-management-reference.md) | Linux | +| 2026-03-08 | [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) | Linux | +| 2026-03-08 | [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) | Self-Hosting | +| 2026-03-08 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | +| 2026-03-08 | [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) | Linux | +| 2026-03-08 | [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) | Self-Hosting | --- diff --git a/SUMMARY.md b/SUMMARY.md index 058fa63..4e8a29b 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,9 +1,10 @@ * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) - * 01-linux/*.md - * [Self-Hosting](02-selfhosting/index.md) - * 02-selfhosting/*.md - * [Streaming](04-streaming/index.md) - * 04-streaming/*.md - * [Troubleshooting](05-troubleshooting/index.md) - * 05-troubleshooting/*.md \ No newline at end of file + * [Introduction](01-linux/index.md) +* [Self-Hosting](02-selfhosting/index.md) + * [Introduction](02-selfhosting/index.md) +* [Streaming](04-streaming/index.md) + * [Introduction](04-streaming/index.md) +* [Troubleshooting](05-troubleshooting/index.md) + * [ISP SNI Filtering & Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) + * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) diff --git a/index.md b/index.md index 89b2525..e9f7697 100644 --- a/index.md +++ b/index.md @@ -2,18 +2,18 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > -> **Last updated:** 2026-03-11 -> **Article count:** 20 +> **Last updated:** 2026-03-12 +> **Article count:** 22 ## Domains | Domain | Folder | Articles | |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 8 | -| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 7 | +| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | | 🔓 Open Source Tools | `03-opensource/` | 0 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 4 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 5 | --- @@ -80,6 +80,7 @@ ## 🔧 General Troubleshooting +- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts - [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora @@ -91,6 +92,7 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | | 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | | 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | | 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | From c4673f70e0f6941c809ce8d4b1a7d00793ceb830 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 01:37:45 -0400 Subject: [PATCH 02/12] Add .gitattributes to enforce LF line endings --- .gitattributes | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..df71be0 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,18 @@ +# Normalize line endings to LF for all text files +* text=auto eol=lf + +# Explicitly handle markdown +*.md text eol=lf + +# Explicitly handle config files +*.yml text eol=lf +*.yaml text eol=lf +*.json text eol=lf +*.toml text eol=lf + +# Binary files — don't touch +*.png binary +*.jpg binary +*.jpeg binary +*.gif binary +*.pdf binary From 70d9657b7f8c3ec989bd73f6d984a99024b1612a Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 01:39:29 -0400 Subject: [PATCH 03/12] vault backup: 2026-03-13 01:39:29 --- 01-linux/index.md | 58 ++++++++++++++++++++--------------------- 02-selfhosting/index.md | 58 ++++++++++++++++++++--------------------- 04-streaming/index.md | 14 +++++----- 3 files changed, 65 insertions(+), 65 deletions(-) diff --git a/01-linux/index.md b/01-linux/index.md index d20dfbc..085d332 100644 --- a/01-linux/index.md +++ b/01-linux/index.md @@ -1,29 +1,29 @@ -# 🐧 Linux & Sysadmin - -A collection of guides covering Linux administration, shell scripting, networking, and distro-specific topics. - -## Files & Permissions - -- [Linux File Permissions and Ownership](files-permissions/linux-file-permissions.md) - -## Networking - -- [SSH Config & Key Management](networking/ssh-config-key-management.md) - -## Package Management - -- [Package Management Reference](packages/package-management-reference.md) - -## Process Management - -- [Managing Linux Services with systemd](process-management/managing-linux-services-systemd-ansible.md) - -## Shell & Scripting - -- [Ansible Getting Started](shell-scripting/ansible-getting-started.md) -- [Bash Scripting Patterns](shell-scripting/bash-scripting-patterns.md) - -## Distro-Specific - -- [Linux Distro Guide for Beginners](distro-specific/linux-distro-guide-beginners.md) -- [WSL2 Instance Migration to Fedora 43](distro-specific/wsl2-instance-migration-fedora43.md) +# 🐧 Linux & Sysadmin + +A collection of guides covering Linux administration, shell scripting, networking, and distro-specific topics. + +## Files & Permissions + +- [Linux File Permissions and Ownership](files-permissions/linux-file-permissions.md) + +## Networking + +- [SSH Config & Key Management](networking/ssh-config-key-management.md) + +## Package Management + +- [Package Management Reference](packages/package-management-reference.md) + +## Process Management + +- [Managing Linux Services with systemd](process-management/managing-linux-services-systemd-ansible.md) + +## Shell & Scripting + +- [Ansible Getting Started](shell-scripting/ansible-getting-started.md) +- [Bash Scripting Patterns](shell-scripting/bash-scripting-patterns.md) + +## Distro-Specific + +- [Linux Distro Guide for Beginners](distro-specific/linux-distro-guide-beginners.md) +- [WSL2 Instance Migration to Fedora 43](distro-specific/wsl2-instance-migration-fedora43.md) diff --git a/02-selfhosting/index.md b/02-selfhosting/index.md index d667137..fba6d82 100644 --- a/02-selfhosting/index.md +++ b/02-selfhosting/index.md @@ -1,29 +1,29 @@ -# 🏠 Self-Hosting & Homelab - -Guides for running your own services at home, including Docker, reverse proxies, DNS, storage, monitoring, and security. - -## Docker & Containers - -- [Self-Hosting Starter Guide](docker/self-hosting-starter-guide.md) -- [Docker vs VMs for the Homelab](docker/docker-vs-vms-homelab.md) -- [Debugging Broken Docker Containers](docker/debugging-broken-docker-containers.md) - -## Reverse Proxies - -- [Setting Up Caddy as a Reverse Proxy](reverse-proxy/setting-up-caddy-reverse-proxy.md) - -## DNS & Networking - -- [Tailscale for Homelab Remote Access](dns-networking/tailscale-homelab-remote-access.md) - -## Storage & Backup - -- [rsync Backup Patterns](storage-backup/rsync-backup-patterns.md) - -## Monitoring - -- [Tuning Netdata Web Log Alerts](monitoring/tuning-netdata-web-log-alerts.md) - -## Security - -- [Linux Server Hardening Checklist](security/linux-server-hardening-checklist.md) +# 🏠 Self-Hosting & Homelab + +Guides for running your own services at home, including Docker, reverse proxies, DNS, storage, monitoring, and security. + +## Docker & Containers + +- [Self-Hosting Starter Guide](docker/self-hosting-starter-guide.md) +- [Docker vs VMs for the Homelab](docker/docker-vs-vms-homelab.md) +- [Debugging Broken Docker Containers](docker/debugging-broken-docker-containers.md) + +## Reverse Proxies + +- [Setting Up Caddy as a Reverse Proxy](reverse-proxy/setting-up-caddy-reverse-proxy.md) + +## DNS & Networking + +- [Tailscale for Homelab Remote Access](dns-networking/tailscale-homelab-remote-access.md) + +## Storage & Backup + +- [rsync Backup Patterns](storage-backup/rsync-backup-patterns.md) + +## Monitoring + +- [Tuning Netdata Web Log Alerts](monitoring/tuning-netdata-web-log-alerts.md) + +## Security + +- [Linux Server Hardening Checklist](security/linux-server-hardening-checklist.md) diff --git a/04-streaming/index.md b/04-streaming/index.md index 1adc9f2..9943e09 100644 --- a/04-streaming/index.md +++ b/04-streaming/index.md @@ -1,7 +1,7 @@ -# 🎙️ Streaming & Podcasting - -Guides for live streaming and podcast production, with a focus on OBS Studio. - -## OBS Studio - -- [OBS Studio Setup & Encoding](obs/obs-studio-setup-encoding.md) +# 🎙️ Streaming & Podcasting + +Guides for live streaming and podcast production, with a focus on OBS Studio. + +## OBS Studio + +- [OBS Studio Setup & Encoding](obs/obs-studio-setup-encoding.md) From b59f6bb6b1564f14c46ac2bfdf995fc40db88b39 Mon Sep 17 00:00:00 2001 From: Marcus Summers Date: Fri, 13 Mar 2026 12:00:42 -0400 Subject: [PATCH 04/12] WSyncing from MajorMaciki expansion (Phase 10): 3 new articles and updated indices --- 01-linux/storage/snapraid-mergerfs-setup.md | 74 +++++++++++++++++++ 03-opensource/index.md | 15 ++++ .../productivity/rmlint-duplicate-scanning.md | 58 +++++++++++++++ .../gpu-display/qwen-14b-oom-3080ti.md | 58 +++++++++++++++ 05-troubleshooting/index.md | 17 ++++- SUMMARY.md | 8 +- index.md | 29 ++++---- 7 files changed, 238 insertions(+), 21 deletions(-) create mode 100644 01-linux/storage/snapraid-mergerfs-setup.md create mode 100644 03-opensource/index.md create mode 100644 03-opensource/productivity/rmlint-duplicate-scanning.md create mode 100644 05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md diff --git a/01-linux/storage/snapraid-mergerfs-setup.md b/01-linux/storage/snapraid-mergerfs-setup.md new file mode 100644 index 0000000..36a1aa9 --- /dev/null +++ b/01-linux/storage/snapraid-mergerfs-setup.md @@ -0,0 +1,74 @@ +# SnapRAID & MergerFS Storage Setup + +## Problem + +Managing a collection of mismatched hard drives as a single pool while maintaining data redundancy (parity) without the overhead or risk of a traditional RAID 5/6 array. + +## Solution + +A combination of **MergerFS** for pooling and **SnapRAID** for parity. This is ideal for "mostly static" media storage (like MajorRAID) where files aren't changing every second. + +### 1. Concepts + +- **MergerFS:** A FUSE-based union filesystem. It takes multiple drives/folders and presents them as a single mount point. It does NOT provide redundancy. +- **SnapRAID:** A backup/parity tool for disk arrays. It creates parity information on a dedicated drive. It is NOT real-time (you must run `snapraid sync`). + +### 2. Implementation Strategy + +1. **Clean the Pool:** Use `rmlint` to clear duplicates and reclaim space. +2. **Identify the Parity Drive:** Choose your largest drive (or one equal to the largest data drive) to hold the parity information. In my setup, `/mnt/usb` (sdc) was cleared of 4TB of duplicates to be repurposed for this. +3. **Configure MergerFS:** Pool the data drives (e.g., `/mnt/disk1`, `/mnt/disk2`) into `/storage`. +4. **Configure SnapRAID:** Point SnapRAID to the data drives and the parity drive. + +### 3. MergerFS Config (/etc/fstab) + +```fstab +# Example MergerFS pool +/mnt/disk*:/mnt/usb-data /storage fuse.mergerfs defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=20G,fsname=mergerfsPool 0 0 +``` + +### 4. SnapRAID Config (/etc/snapraid.conf) + +```conf +# Parity file location +parity /mnt/parity/snapraid.parity + +# Data drives +content /var/snapraid/snapraid.content +content /mnt/disk1/.snapraid.content +content /mnt/disk2/.snapraid.content + +data d1 /mnt/disk1/ +data d2 /mnt/disk2/ + +# Exclusions +exclude /lost+found/ +exclude /tmp/ +exclude .DS_Store +``` + +--- + +## Maintenance + +### SnapRAID Sync + +Run this daily (via cron) or after adding large amounts of data: + +```bash +snapraid sync +``` + +### SnapRAID Scrub + +Run this weekly to check for bitrot: + +```bash +snapraid scrub +``` + +--- + +## Tags + +#snapraid #mergerfs #linux #storage #homelab #raid diff --git a/03-opensource/index.md b/03-opensource/index.md new file mode 100644 index 0000000..4c3af23 --- /dev/null +++ b/03-opensource/index.md @@ -0,0 +1,15 @@ +# 📂 Open Source & Alternatives + +A curated collection of my favorite open-source tools and privacy-respecting alternatives to mainstream software. + +## 🚀 Productivity +- [rmlint: Duplicate File Scanning](productivity/rmlint-duplicate-scanning.md) + +## 🛠️ Development Tools +- *Coming soon* + +## 🎨 Media & Creative +- *Coming soon* + +## 🔐 Privacy & Security +- *Coming soon* diff --git a/03-opensource/productivity/rmlint-duplicate-scanning.md b/03-opensource/productivity/rmlint-duplicate-scanning.md new file mode 100644 index 0000000..629c0b5 --- /dev/null +++ b/03-opensource/productivity/rmlint-duplicate-scanning.md @@ -0,0 +1,58 @@ +# rmlint — Extreme Duplicate File Scanning + +## Problem + +Over time, backups and media collections can accumulate massive amounts of duplicate data. Traditional duplicate finders are often slow and limited in how they handle results. On MajorRAID, I identified **~4.0 TB (113,584 files)** of duplicate data across three different storage points. + +## Solution + +`rmlint` is an extremely fast tool for finding (and optionally removing) duplicates. It is significantly faster than `fdupes` or `rdfind` because it uses a multi-stage approach to avoid unnecessary hashing. + +### 1. Installation (Fedora) + +```bash +sudo dnf install rmlint +``` + +### 2. Scanning Multiple Directories + +To scan for duplicates across multiple mount points and compare them: + +```bash +rmlint /majorstorage /majorRAID /mnt/usb +``` + +This will generate a script named `rmlint.sh` and a summary of the findings. + +### 3. Reviewing Results + +**DO NOT** run the generated script without reviewing it first. You can use the summary to see which paths contain the most duplicates: + +```bash +# View the summary +cat rmlint.json | jq . +``` + +### 4. Advanced Usage: Finding Duplicates by Hash Only + +If you suspect duplicates with different filenames: + +```bash +rmlint --hidden --hard-links /path/to/search +``` + +### 5. Repurposing Storage + +After scanning and clearing duplicates, you can reclaim significant space. In my case, this was the first step in repurposing a 12TB USB drive as a **SnapRAID parity drive**. + +--- + +## Maintenance + +Run a scan monthly or before any major storage consolidation project. + +--- + +## Tags + +#rmlint #linux #storage #cleanup #duplicates diff --git a/05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md b/05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md new file mode 100644 index 0000000..23d812c --- /dev/null +++ b/05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md @@ -0,0 +1,58 @@ +# Qwen2.5-14B OOM on RTX 3080 Ti (12GB) + +## Problem + +When attempting to run or fine-tune **Qwen2.5-14B** on an NVIDIA RTX 3080 Ti with 12GB of VRAM, the process fails with an Out of Memory (OOM) error: + +``` +torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X GiB (GPU 0; 12.00 GiB total capacity; Y GiB already allocated; Z GiB free; ...) +``` + +The 12GB VRAM limit is hit during the initial model load or immediately upon starting the first training step. + +## Root Causes + +1. **Model Size:** A 14B parameter model in FP16/BF16 requires ~28GB of VRAM just for the weights. +2. **Context Length:** High context lengths (e.g., 4096+) significantly increase VRAM usage during training. +3. **Training Overhead:** Even with QLoRA (4-bit quantization), the overhead of gradients, optimizer states, and activations can exceed 12GB for a 14B model. + +--- + +## Solutions + +### 1. Pivot to a 7B Model (Recommended) + +For a 12GB GPU, a 7B parameter model (like **Qwen2.5-7B-Instruct**) is the sweet spot. It provides excellent performance while leaving enough VRAM for high context lengths and larger batch sizes. + +- **VRAM Usage (7B QLoRA):** ~6-8GB +- **Pros:** Stable, fast, supports long context. +- **Cons:** Slightly lower reasoning capability than 14B. + +### 2. Aggressive Quantization + +If you MUST run 14B, use 4-bit quantization (GGUF or EXL2) for inference only. Training 14B on 12GB is not reliably possible even with extreme offloading. + +```bash +# Example Ollama run (uses 4-bit quantization by default) +ollama run qwen2.5:14b +``` + +### 3. Training Optimizations (if attempting 14B) + +If you have no choice but to try 14B training: +- Set `max_seq_length` to 512 or 1024. +- Use `Unsloth` (it is highly memory-efficient). +- Enable `gradient_checkpointing`. +- Set `per_device_train_batch_size = 1`. + +--- + +## Maintenance + +Keep your NVIDIA drivers and CUDA toolkit updated. On Windows (MajorRig), ensure WSL2 has sufficient memory allocation in `.wslconfig`. + +--- + +## Tags + +#gpu #cuda #oom #qwen #majortwin #llm #fine-tuning diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index 9bfc177..2ce9737 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -2,8 +2,17 @@ Practical fixes for common Linux, networking, and application problems. -- [ISP SNI Filtering with Caddy](isp-sni-filtering-caddy.md) -- [Obsidian Cache Hang Recovery](obsidian-cache-hang-recovery.md) -- [yt-dlp Fedora JS Challenge](yt-dlp-fedora-js-challenge.md) -- [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) +## 🖥️ GPU & AI +- [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](gpu-display/qwen-14b-oom-3080ti.md) + +## 🌐 Networking & Web +- [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](networking/fail2ban-self-ban-apache-outage.md) +- [ISP SNI Filtering & Caddy](isp-sni-filtering-caddy.md) +- [yt-dlp YouTube JS Challenge Fix](yt-dlp-fedora-js-challenge.md) + +## 📦 Docker & Systems - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) +- [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) + +## 📝 Application Specific +- [Obsidian Vault Recovery — Loading Cache Hang](obsidian-cache-hang-recovery.md) diff --git a/SUMMARY.md b/SUMMARY.md index bb74759..477ebe0 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,11 +1,17 @@ * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) - * [Introduction](01-linux/index.md) + * [Storage: SnapRAID & MergerFS Setup](01-linux/storage/snapraid-mergerfs-setup.md) * [Self-Hosting](02-selfhosting/index.md) * [Introduction](02-selfhosting/index.md) +* [Open Source & Alternatives](03-opensource/index.md) + * [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) * [Streaming](04-streaming/index.md) * [Introduction](04-streaming/index.md) * [Troubleshooting](05-troubleshooting/index.md) * [ISP SNI Filtering & Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) + * [yt-dlp YouTube JS Challenge Fix](05-troubleshooting/yt-dlp-fedora-js-challenge.md) * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) * [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) + * [Obsidian Vault Recovery — Loading Cache Hang](05-troubleshooting/obsidian-cache-hang-recovery.md) + * [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) + * [MajorWiki Setup & Publishing Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) diff --git a/index.md b/index.md index 054cb62..de9c3ce 100644 --- a/index.md +++ b/index.md @@ -3,17 +3,17 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 23 +> **Article count:** 27 ## Domains | Domain | Folder | Articles | |---|---|---| -| 🐧 Linux & Sysadmin | `01-linux/` | 8 | +| 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 0 | +| 🔓 Open Source Tools | `03-opensource/` | 1 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 6 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 8 | --- @@ -35,6 +35,9 @@ - [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) — inventory, ad-hoc commands, playbooks, handlers, roles - [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) — set -euo pipefail, logging, error handling, argument parsing, common patterns +### Storage +- [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) — Pooling mismatched drives and adding parity on Linux + ### Distro-Specific - [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) — Ubuntu recommendation, distro comparison, desktop environments - [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md) — moving WSL2 VHDX from C: to another drive @@ -67,7 +70,8 @@ ## 🔓 Open Source Tools -*(Articles coming)* +### Productivity +- [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation --- @@ -84,6 +88,7 @@ - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts +- [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) — fixes and alternatives when hitting VRAM limits during fine-tuning - [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline @@ -93,21 +98,14 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | +| 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | +| 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | | 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | | 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | | 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | | 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | | 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | -| 2026-03-08 | [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) | Streaming | -| 2026-03-08 | [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md) | Linux | -| 2026-03-08 | [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) | Self-Hosting | -| 2026-03-08 | [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) | Self-Hosting | -| 2026-03-08 | [Package Management Reference](01-linux/packages/package-management-reference.md) | Linux | -| 2026-03-08 | [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) | Linux | -| 2026-03-08 | [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) | Self-Hosting | -| 2026-03-08 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | -| 2026-03-08 | [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) | Linux | -| 2026-03-08 | [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) | Self-Hosting | --- @@ -119,6 +117,5 @@ | Docker Compose networking deep dive | Self-Hosting | High | No | | Troubleshooting NVIDIA on Linux | Troubleshooting | Medium | No | | Pi-hole setup and local DNS | Self-Hosting | Medium | No | -| OBS audio routing on Linux (PipeWire) | Streaming | Medium | No | | Nextcloud setup with Docker | Self-Hosting | Medium | No | | tmux basics | Linux | Low | No | From 2861cade55695c27d15102063b4f4773e5c62d6d Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 22:45:52 -0400 Subject: [PATCH 05/12] wiki: add manual update guide for Gemini CLI --- 01-linux/storage/snapraid-mergerfs-setup.md | 74 +++++++++++++++++++ 03-opensource/index.md | 15 ++++ .../productivity/rmlint-duplicate-scanning.md | 58 +++++++++++++++ .../gemini-cli-manual-update.md | 47 ++++++++++++ .../gpu-display/qwen-14b-oom-3080ti.md | 58 +++++++++++++++ 05-troubleshooting/index.md | 18 ++++- .../majwiki-setup-and-pipeline.md | 17 +++++ MajorWiki-Deploy-Status.md | 19 +++++ SUMMARY.md | 9 ++- index.md | 29 ++++---- 10 files changed, 323 insertions(+), 21 deletions(-) create mode 100644 01-linux/storage/snapraid-mergerfs-setup.md create mode 100644 03-opensource/index.md create mode 100644 03-opensource/productivity/rmlint-duplicate-scanning.md create mode 100644 05-troubleshooting/gemini-cli-manual-update.md create mode 100644 05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md diff --git a/01-linux/storage/snapraid-mergerfs-setup.md b/01-linux/storage/snapraid-mergerfs-setup.md new file mode 100644 index 0000000..36a1aa9 --- /dev/null +++ b/01-linux/storage/snapraid-mergerfs-setup.md @@ -0,0 +1,74 @@ +# SnapRAID & MergerFS Storage Setup + +## Problem + +Managing a collection of mismatched hard drives as a single pool while maintaining data redundancy (parity) without the overhead or risk of a traditional RAID 5/6 array. + +## Solution + +A combination of **MergerFS** for pooling and **SnapRAID** for parity. This is ideal for "mostly static" media storage (like MajorRAID) where files aren't changing every second. + +### 1. Concepts + +- **MergerFS:** A FUSE-based union filesystem. It takes multiple drives/folders and presents them as a single mount point. It does NOT provide redundancy. +- **SnapRAID:** A backup/parity tool for disk arrays. It creates parity information on a dedicated drive. It is NOT real-time (you must run `snapraid sync`). + +### 2. Implementation Strategy + +1. **Clean the Pool:** Use `rmlint` to clear duplicates and reclaim space. +2. **Identify the Parity Drive:** Choose your largest drive (or one equal to the largest data drive) to hold the parity information. In my setup, `/mnt/usb` (sdc) was cleared of 4TB of duplicates to be repurposed for this. +3. **Configure MergerFS:** Pool the data drives (e.g., `/mnt/disk1`, `/mnt/disk2`) into `/storage`. +4. **Configure SnapRAID:** Point SnapRAID to the data drives and the parity drive. + +### 3. MergerFS Config (/etc/fstab) + +```fstab +# Example MergerFS pool +/mnt/disk*:/mnt/usb-data /storage fuse.mergerfs defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=20G,fsname=mergerfsPool 0 0 +``` + +### 4. SnapRAID Config (/etc/snapraid.conf) + +```conf +# Parity file location +parity /mnt/parity/snapraid.parity + +# Data drives +content /var/snapraid/snapraid.content +content /mnt/disk1/.snapraid.content +content /mnt/disk2/.snapraid.content + +data d1 /mnt/disk1/ +data d2 /mnt/disk2/ + +# Exclusions +exclude /lost+found/ +exclude /tmp/ +exclude .DS_Store +``` + +--- + +## Maintenance + +### SnapRAID Sync + +Run this daily (via cron) or after adding large amounts of data: + +```bash +snapraid sync +``` + +### SnapRAID Scrub + +Run this weekly to check for bitrot: + +```bash +snapraid scrub +``` + +--- + +## Tags + +#snapraid #mergerfs #linux #storage #homelab #raid diff --git a/03-opensource/index.md b/03-opensource/index.md new file mode 100644 index 0000000..4c3af23 --- /dev/null +++ b/03-opensource/index.md @@ -0,0 +1,15 @@ +# 📂 Open Source & Alternatives + +A curated collection of my favorite open-source tools and privacy-respecting alternatives to mainstream software. + +## 🚀 Productivity +- [rmlint: Duplicate File Scanning](productivity/rmlint-duplicate-scanning.md) + +## 🛠️ Development Tools +- *Coming soon* + +## 🎨 Media & Creative +- *Coming soon* + +## 🔐 Privacy & Security +- *Coming soon* diff --git a/03-opensource/productivity/rmlint-duplicate-scanning.md b/03-opensource/productivity/rmlint-duplicate-scanning.md new file mode 100644 index 0000000..629c0b5 --- /dev/null +++ b/03-opensource/productivity/rmlint-duplicate-scanning.md @@ -0,0 +1,58 @@ +# rmlint — Extreme Duplicate File Scanning + +## Problem + +Over time, backups and media collections can accumulate massive amounts of duplicate data. Traditional duplicate finders are often slow and limited in how they handle results. On MajorRAID, I identified **~4.0 TB (113,584 files)** of duplicate data across three different storage points. + +## Solution + +`rmlint` is an extremely fast tool for finding (and optionally removing) duplicates. It is significantly faster than `fdupes` or `rdfind` because it uses a multi-stage approach to avoid unnecessary hashing. + +### 1. Installation (Fedora) + +```bash +sudo dnf install rmlint +``` + +### 2. Scanning Multiple Directories + +To scan for duplicates across multiple mount points and compare them: + +```bash +rmlint /majorstorage /majorRAID /mnt/usb +``` + +This will generate a script named `rmlint.sh` and a summary of the findings. + +### 3. Reviewing Results + +**DO NOT** run the generated script without reviewing it first. You can use the summary to see which paths contain the most duplicates: + +```bash +# View the summary +cat rmlint.json | jq . +``` + +### 4. Advanced Usage: Finding Duplicates by Hash Only + +If you suspect duplicates with different filenames: + +```bash +rmlint --hidden --hard-links /path/to/search +``` + +### 5. Repurposing Storage + +After scanning and clearing duplicates, you can reclaim significant space. In my case, this was the first step in repurposing a 12TB USB drive as a **SnapRAID parity drive**. + +--- + +## Maintenance + +Run a scan monthly or before any major storage consolidation project. + +--- + +## Tags + +#rmlint #linux #storage #cleanup #duplicates diff --git a/05-troubleshooting/gemini-cli-manual-update.md b/05-troubleshooting/gemini-cli-manual-update.md new file mode 100644 index 0000000..5d5d9de --- /dev/null +++ b/05-troubleshooting/gemini-cli-manual-update.md @@ -0,0 +1,47 @@ +# 🛠️ Gemini CLI: Manual Update Guide + +If the automatic update fails or you need to force a specific version of the Gemini CLI, use these steps. + +## 🔴 Symptom: Automatic Update Failed +You may see an error message like: +`✕ Automatic update failed. Please try updating manually` + +## 🟢 Manual Update Procedure + +### 1. Verify Current Version +Check the version currently installed on your system: +```bash +gemini --version +``` + +### 2. Check Latest Version +Query the npm registry for the latest available version: +```bash +npm show @google/gemini-cli version +``` + +### 3. Perform Manual Update +Use `npm` with `sudo` to update the global package: +```bash +sudo npm install -g @google/gemini-cli@latest +``` + +### 4. Confirm Update +Verify that the new version is active: +```bash +gemini --version +``` + +## 🛠️ Troubleshooting Update Failures + +### Permissions Issues +If you encounter `EACCES` errors without `sudo`, ensure your user has permissions or use `sudo` as shown above. + +### Registry Connectivity +If `npm` cannot reach the registry, check your internet connection or any local firewall/proxy settings. + +### Cache Issues +If the version doesn't update, try clearing the npm cache: +```bash +npm cache clean --force +``` diff --git a/05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md b/05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md new file mode 100644 index 0000000..23d812c --- /dev/null +++ b/05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md @@ -0,0 +1,58 @@ +# Qwen2.5-14B OOM on RTX 3080 Ti (12GB) + +## Problem + +When attempting to run or fine-tune **Qwen2.5-14B** on an NVIDIA RTX 3080 Ti with 12GB of VRAM, the process fails with an Out of Memory (OOM) error: + +``` +torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X GiB (GPU 0; 12.00 GiB total capacity; Y GiB already allocated; Z GiB free; ...) +``` + +The 12GB VRAM limit is hit during the initial model load or immediately upon starting the first training step. + +## Root Causes + +1. **Model Size:** A 14B parameter model in FP16/BF16 requires ~28GB of VRAM just for the weights. +2. **Context Length:** High context lengths (e.g., 4096+) significantly increase VRAM usage during training. +3. **Training Overhead:** Even with QLoRA (4-bit quantization), the overhead of gradients, optimizer states, and activations can exceed 12GB for a 14B model. + +--- + +## Solutions + +### 1. Pivot to a 7B Model (Recommended) + +For a 12GB GPU, a 7B parameter model (like **Qwen2.5-7B-Instruct**) is the sweet spot. It provides excellent performance while leaving enough VRAM for high context lengths and larger batch sizes. + +- **VRAM Usage (7B QLoRA):** ~6-8GB +- **Pros:** Stable, fast, supports long context. +- **Cons:** Slightly lower reasoning capability than 14B. + +### 2. Aggressive Quantization + +If you MUST run 14B, use 4-bit quantization (GGUF or EXL2) for inference only. Training 14B on 12GB is not reliably possible even with extreme offloading. + +```bash +# Example Ollama run (uses 4-bit quantization by default) +ollama run qwen2.5:14b +``` + +### 3. Training Optimizations (if attempting 14B) + +If you have no choice but to try 14B training: +- Set `max_seq_length` to 512 or 1024. +- Use `Unsloth` (it is highly memory-efficient). +- Enable `gradient_checkpointing`. +- Set `per_device_train_batch_size = 1`. + +--- + +## Maintenance + +Keep your NVIDIA drivers and CUDA toolkit updated. On Windows (MajorRig), ensure WSL2 has sufficient memory allocation in `.wslconfig`. + +--- + +## Tags + +#gpu #cuda #oom #qwen #majortwin #llm #fine-tuning diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index 9bfc177..4abff79 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -2,8 +2,18 @@ Practical fixes for common Linux, networking, and application problems. -- [ISP SNI Filtering with Caddy](isp-sni-filtering-caddy.md) -- [Obsidian Cache Hang Recovery](obsidian-cache-hang-recovery.md) -- [yt-dlp Fedora JS Challenge](yt-dlp-fedora-js-challenge.md) -- [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) +## 🖥️ GPU & AI +- [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](gpu-display/qwen-14b-oom-3080ti.md) + +## 🌐 Networking & Web +- [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](networking/fail2ban-self-ban-apache-outage.md) +- [ISP SNI Filtering & Caddy](isp-sni-filtering-caddy.md) +- [yt-dlp YouTube JS Challenge Fix](yt-dlp-fedora-js-challenge.md) + +## 📦 Docker & Systems - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) +- [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) + +## 📝 Application Specific +- [Obsidian Vault Recovery — Loading Cache Hang](obsidian-cache-hang-recovery.md) +- [Gemini CLI Manual Update](gemini-cli-manual-update.md) diff --git a/05-troubleshooting/majwiki-setup-and-pipeline.md b/05-troubleshooting/majwiki-setup-and-pipeline.md index 200c2fa..90e6c75 100644 --- a/05-troubleshooting/majwiki-setup-and-pipeline.md +++ b/05-troubleshooting/majwiki-setup-and-pipeline.md @@ -119,3 +119,20 @@ The webhook runs as a systemd service so it survives reboots: systemctl status majwiki-webhook systemctl restart majwiki-webhook ``` + +--- + +*Updated 2026-03-13: Obsidian Git plugin dropped. See canonical workflow below.* + +## Canonical Publishing Workflow + +The Obsidian Git plugin was evaluated but dropped — too convoluted for a simple push. Manual git from the terminal is the canonical workflow. + +```bash +cd ~/Documents/MajorVault +git add 20-Projects/MajorTwin/08-Wiki/ +git commit -m "wiki: describe your changes" +git push +``` + +From there: Gitea receives the push → fires webhook → majorlab pulls → MkDocs rebuilds → `notes.majorshouse.com` updates. diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index cd03aca..ca3068c 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -60,3 +60,22 @@ rsync -av --include="*.md" --include="*/" --exclude="*" \ - **Cause:** `majwiki` container was in a restart loop due to a `LiterateNavParseError` in `SUMMARY.md` (invalid globbing syntax). - **Repair:** Rewrote `SUMMARY.md` with explicit links. Repaired `majwiki-webhook.service` by killing an orphaned process and correcting the Caddy reverse proxy routing for `/webhook`. - **Result:** Site live and auto-deployment functional. + +--- + +*Updated 2026-03-13* + +## Canonical Update Workflow + +Obsidian Git plugin was evaluated and dropped — too convoluted. Manual git from the terminal is the workflow going forward. + +```bash +cd ~/Documents/MajorVault +git add 20-Projects/MajorTwin/08-Wiki/ +git commit -m "wiki: describe your changes" +git push +``` + +Gitea receives the push → fires webhook → majorlab pulls → MkDocs rebuilds → `notes.majorshouse.com` updates automatically. + +> [!note] The rsync one-liner in the Update Workflow section above was a one-off during initial setup. It is no longer the canonical method. diff --git a/SUMMARY.md b/SUMMARY.md index bb74759..c639add 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,11 +1,18 @@ * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) - * [Introduction](01-linux/index.md) + * [Storage: SnapRAID & MergerFS Setup](01-linux/storage/snapraid-mergerfs-setup.md) * [Self-Hosting](02-selfhosting/index.md) * [Introduction](02-selfhosting/index.md) +* [Open Source & Alternatives](03-opensource/index.md) + * [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) * [Streaming](04-streaming/index.md) * [Introduction](04-streaming/index.md) * [Troubleshooting](05-troubleshooting/index.md) * [ISP SNI Filtering & Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) + * [yt-dlp YouTube JS Challenge Fix](05-troubleshooting/yt-dlp-fedora-js-challenge.md) * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) * [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) + * [Obsidian Vault Recovery — Loading Cache Hang](05-troubleshooting/obsidian-cache-hang-recovery.md) + * [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) + * [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) + * [MajorWiki Setup & Publishing Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) diff --git a/index.md b/index.md index 054cb62..de9c3ce 100644 --- a/index.md +++ b/index.md @@ -3,17 +3,17 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 23 +> **Article count:** 27 ## Domains | Domain | Folder | Articles | |---|---|---| -| 🐧 Linux & Sysadmin | `01-linux/` | 8 | +| 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 0 | +| 🔓 Open Source Tools | `03-opensource/` | 1 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 6 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 8 | --- @@ -35,6 +35,9 @@ - [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) — inventory, ad-hoc commands, playbooks, handlers, roles - [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) — set -euo pipefail, logging, error handling, argument parsing, common patterns +### Storage +- [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) — Pooling mismatched drives and adding parity on Linux + ### Distro-Specific - [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) — Ubuntu recommendation, distro comparison, desktop environments - [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md) — moving WSL2 VHDX from C: to another drive @@ -67,7 +70,8 @@ ## 🔓 Open Source Tools -*(Articles coming)* +### Productivity +- [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation --- @@ -84,6 +88,7 @@ - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts +- [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) — fixes and alternatives when hitting VRAM limits during fine-tuning - [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline @@ -93,21 +98,14 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | +| 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | +| 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | | 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | | 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | | 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | | 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | | 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | -| 2026-03-08 | [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) | Streaming | -| 2026-03-08 | [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md) | Linux | -| 2026-03-08 | [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) | Self-Hosting | -| 2026-03-08 | [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) | Self-Hosting | -| 2026-03-08 | [Package Management Reference](01-linux/packages/package-management-reference.md) | Linux | -| 2026-03-08 | [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) | Linux | -| 2026-03-08 | [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) | Self-Hosting | -| 2026-03-08 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | -| 2026-03-08 | [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) | Linux | -| 2026-03-08 | [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) | Self-Hosting | --- @@ -119,6 +117,5 @@ | Docker Compose networking deep dive | Self-Hosting | High | No | | Troubleshooting NVIDIA on Linux | Troubleshooting | Medium | No | | Pi-hole setup and local DNS | Self-Hosting | Medium | No | -| OBS audio routing on Linux (PipeWire) | Streaming | Medium | No | | Nextcloud setup with Docker | Self-Hosting | Medium | No | | tmux basics | Linux | Low | No | From 4bfb99efa62357e2824a328925d2f40ade6a8e00 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 22:49:56 -0400 Subject: [PATCH 06/12] wiki: update main index and readme with new articles --- README.md | 36 +++++++++++++++++++----------------- index.md | 8 +++++--- 2 files changed, 24 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index e9f7697..4ff8afd 100644 --- a/README.md +++ b/README.md @@ -2,18 +2,18 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > -> **Last updated:** 2026-03-12 -> **Article count:** 22 +**Last updated:** 2026-03-13 +**Article count:** 28 ## Domains | Domain | Folder | Articles | |---|---|---| -| 🐧 Linux & Sysadmin | `01-linux/` | 8 | +| 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 0 | +| 🔓 Open Source Tools | `03-opensource/` | 1 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 5 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | --- @@ -35,6 +35,9 @@ - [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) — inventory, ad-hoc commands, playbooks, handlers, roles - [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) — set -euo pipefail, logging, error handling, argument parsing, common patterns +### Storage +- [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) — Pooling mismatched drives and adding parity on Linux + ### Distro-Specific - [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) — Ubuntu recommendation, distro comparison, desktop environments - [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md) — moving WSL2 VHDX from C: to another drive @@ -67,7 +70,8 @@ ## 🔓 Open Source Tools -*(Articles coming)* +### Productivity +- [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation --- @@ -80,10 +84,13 @@ ## 🔧 General Troubleshooting +- [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) — diagnosing and fixing Apache outages caused by missing firewall rules and Fail2ban self-bans - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts -- [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora +- [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) — fixes and alternatives when hitting VRAM limits during fine-tuning +- [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora +- [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) — how to manually update the Gemini CLI when automatic updates fail - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline --- @@ -92,20 +99,15 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) | Troubleshooting | +| 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | +| 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | +| 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | +| 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | | 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | | 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | | 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | | 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | -| 2026-03-08 | [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) | Streaming | -| 2026-03-08 | [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md) | Linux | -| 2026-03-08 | [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) | Self-Hosting | -| 2026-03-08 | [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) | Self-Hosting | -| 2026-03-08 | [Package Management Reference](01-linux/packages/package-management-reference.md) | Linux | -| 2026-03-08 | [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) | Linux | -| 2026-03-08 | [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) | Self-Hosting | -| 2026-03-08 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | -| 2026-03-08 | [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) | Linux | -| 2026-03-08 | [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) | Self-Hosting | --- diff --git a/index.md b/index.md index de9c3ce..c115bc5 100644 --- a/index.md +++ b/index.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 27 +> **Article count:** 28 ## Domains @@ -13,7 +13,7 @@ | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | | 🔓 Open Source Tools | `03-opensource/` | 1 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 8 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | --- @@ -89,7 +89,8 @@ - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts - [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) — fixes and alternatives when hitting VRAM limits during fine-tuning -- [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora +- [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora +- [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) — how to manually update the Gemini CLI when automatic updates fail - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline --- @@ -98,6 +99,7 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) | Troubleshooting | | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | From 2e5512ed970d67ab12d75ebdf4bc31409a6b0eb0 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 22:50:59 -0400 Subject: [PATCH 07/12] wiki: document maintenance protocol --- MajorWiki-Deploy-Status.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index ca3068c..b70251f 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -78,4 +78,16 @@ git push Gitea receives the push → fires webhook → majorlab pulls → MkDocs rebuilds → `notes.majorshouse.com` updates automatically. +## 📋 Wiki Maintenance Protocol + +Every time a new article is added, the following **MUST** be updated to maintain index integrity: + +1. **`SUMMARY.md`**: Add the article to the navigation tree. +2. **Category Index**: Update the `index.md` within the specific domain folder (e.g., `05-troubleshooting/index.md`). +3. **Main `index.md`**: + - Update the **Article count**. + - Add the article to the domain section. + - Add the article to the **Recently Updated** table. +4. **Root `README.md`**: Sync with the main `index.md` (Article count, domain lists, and Recently Updated). + > [!note] The rsync one-liner in the Update Workflow section above was a one-off during initial setup. It is no longer the canonical method. From 4f3e5877aed97648b43c453909af2db88f745057 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 23:33:38 -0400 Subject: [PATCH 08/12] wiki: add dev-tools section with tmux, screen, and rsync articles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add three new articles to 03-opensource/dev-tools/: - tmux: persistent terminal sessions, background jobs, capture-pane - screen: lightweight alternative, comparison table - rsync: flags reference, resumable transfers, SSH usage Update all indexes (SUMMARY, section index, main index, README). Article count: 28 → 31. Remove tmux from writing backlog. Co-Authored-By: Claude Sonnet 4.6 --- 03-opensource/dev-tools/rsync.md | 102 ++++++++++++++++++++++++++++++ 03-opensource/dev-tools/screen.md | 76 ++++++++++++++++++++++ 03-opensource/dev-tools/tmux.md | 93 +++++++++++++++++++++++++++ 03-opensource/index.md | 4 +- MajorWiki-Deploy-Status.md | 2 +- README.md | 16 +++-- SUMMARY.md | 3 + index.md | 16 +++-- 8 files changed, 298 insertions(+), 14 deletions(-) create mode 100644 03-opensource/dev-tools/rsync.md create mode 100644 03-opensource/dev-tools/screen.md create mode 100644 03-opensource/dev-tools/tmux.md diff --git a/03-opensource/dev-tools/rsync.md b/03-opensource/dev-tools/rsync.md new file mode 100644 index 0000000..774e3a4 --- /dev/null +++ b/03-opensource/dev-tools/rsync.md @@ -0,0 +1,102 @@ +# rsync — Fast, Resumable File Transfers + +## Problem + +Copying large files or directory trees between drives or servers is slow, fragile, and unresumable with `cp`. A dropped connection or a single error means starting over. You also want to skip files that already exist at the destination without re-copying them. + +## Solution + +`rsync` is a file synchronization tool that only transfers what has changed, preserves metadata, and can resume interrupted transfers. It works locally and over SSH. + +### Installation (Fedora) + +```bash +sudo dnf install rsync +``` + +### Basic Local Copy + +```bash +rsync -av /source/ /destination/ +``` + +- `-a` — archive mode: preserves permissions, timestamps, symlinks, ownership +- `-v` — verbose: shows what's being transferred + +**Trailing slash on source matters:** +- `/source/` — copy the *contents* of source into destination +- `/source` — copy the source *directory itself* into destination + +### Resume an Interrupted Transfer + +```bash +rsync -av --partial --progress /source/ /destination/ +``` + +- `--partial` — keeps partially transferred files so they can be resumed +- `--progress` — shows per-file progress and speed + +### Skip Already-Transferred Files + +```bash +rsync -av --ignore-existing /source/ /destination/ +``` + +Useful when restarting a migration — skips anything already at the destination regardless of timestamp comparison. + +### Dry Run First + +Always preview what rsync will do before committing: + +```bash +rsync -av --dry-run /source/ /destination/ +``` + +No files are moved. Output shows exactly what would happen. + +### Transfer Over SSH + +```bash +rsync -av -e ssh /source/ user@remotehost:/destination/ +``` + +Or with a non-standard port: + +```bash +rsync -av -e "ssh -p 2222" /source/ user@remotehost:/destination/ +``` + +### Exclude Patterns + +```bash +rsync -av --exclude='*.tmp' --exclude='.Trash*' /source/ /destination/ +``` + +### Real-World Use + +Migrating ~286 files from `/majorRAID` to `/majorstorage` during a RAID dissolution project: + +```bash +rsync -av --partial --progress --ignore-existing \ + /majorRAID/ /majorstorage/ \ + 2>&1 | tee /root/raid_migrate.log +``` + +Run inside a `tmux` or `screen` session so it survives SSH disconnects: + +```bash +tmux new-session -d -s rsync-migrate \ + "rsync -av --partial --progress /majorRAID/ /majorstorage/ | tee /root/raid_migrate.log" +``` + +### Check Progress on a Running Transfer + +```bash +tail -f /root/raid_migrate.log +``` + +--- + +## Tags + +#rsync #linux #storage #file-transfer #sysadmin #dev-tools diff --git a/03-opensource/dev-tools/screen.md b/03-opensource/dev-tools/screen.md new file mode 100644 index 0000000..e772913 --- /dev/null +++ b/03-opensource/dev-tools/screen.md @@ -0,0 +1,76 @@ +# screen — Simple Persistent Terminal Sessions + +## Problem + +Same problem as tmux: SSH sessions die, jobs get killed, long-running tasks need to survive disconnects. screen is the older, simpler alternative to tmux — universally available and gets the job done with minimal setup. + +## Solution + +`screen` creates detachable terminal sessions. It's installed by default on many systems, making it useful when tmux isn't available. + +### Installation (Fedora) + +```bash +sudo dnf install screen +``` + +### Core Workflow + +```bash +# Start a named session +screen -S mysession + +# Detach (keeps running) +Ctrl+a, d + +# List sessions +screen -list + +# Reattach +screen -r mysession + +# If session shows as "Attached" (stuck) +screen -d -r mysession +``` + +### Start a Background Job Directly + +```bash +screen -dmS mysession bash -c "long-running-command 2>&1 | tee /root/output.log" +``` + +- `-d` — start detached +- `-m` — create new session even if already inside screen +- `-S` — name the session + +### Capture Current Output Without Attaching + +```bash +screen -S mysession -X hardcopy /tmp/screen_output.txt +cat /tmp/screen_output.txt +``` + +### Send a Command to a Running Session + +```bash +screen -S mysession -X stuff "tail -f /root/output.log\n" +``` + +--- + +## screen vs tmux + +| Feature | screen | tmux | +|---|---|---| +| Availability | Installed by default on most systems | Usually needs installing | +| Split panes | Basic (Ctrl+a, S) | Better (Ctrl+b, ") | +| Scripting | Limited | More capable | +| Config complexity | Simple | More options | + +Use screen when it's already there or for quick throwaway sessions. Use tmux for anything more complex. See [tmux](tmux.md). + +--- + +## Tags + +#screen #terminal #linux #ssh #productivity #dev-tools diff --git a/03-opensource/dev-tools/tmux.md b/03-opensource/dev-tools/tmux.md new file mode 100644 index 0000000..09fd2d8 --- /dev/null +++ b/03-opensource/dev-tools/tmux.md @@ -0,0 +1,93 @@ +# tmux — Persistent Terminal Sessions + +## Problem + +SSH sessions die when your connection drops, your laptop closes, or you walk away. Long-running jobs — storage migrations, file scans, downloads — get killed mid-run. You need a way to detach from a session, come back later, and pick up exactly where you left off. + +## Solution + +`tmux` is a terminal multiplexer. It runs sessions that persist independently of your SSH connection. You can detach, disconnect, reconnect from a different machine, and reattach to find everything still running. + +### Installation (Fedora) + +```bash +sudo dnf install tmux +``` + +### Core Workflow + +```bash +# Start a named session +tmux new-session -s mysession + +# Detach from a session (keeps it running) +Ctrl+b, d + +# List running sessions +tmux ls + +# Reattach to a session +tmux attach -t mysession + +# Kill a session when done +tmux kill-session -t mysession +``` + +### Start a Background Job Directly + +Skip the interactive session entirely — start a job in a new detached session in one command: + +```bash +tmux new-session -d -s rmlint2 "rmlint /majorstorage// /mnt/usb// /majorRAID 2>&1 | tee /majorRAID/rmlint_scan2.log" +``` + +The job runs immediately in the background. Attach later to check progress: + +```bash +tmux attach -t rmlint2 +``` + +### Capture Output Without Attaching + +Read the current state of a session without interrupting it: + +```bash +tmux capture-pane -t rmlint2 -p +``` + +### Split Panes + +Monitor multiple things in one terminal window: + +```bash +# Horizontal split (top/bottom) +Ctrl+b, " + +# Vertical split (left/right) +Ctrl+b, % + +# Switch between panes +Ctrl+b, arrow keys +``` + +### Real-World Use + +On **majorhome**, all long-running storage operations run inside named tmux sessions so they survive SSH disconnects: + +```bash +tmux new-session -d -s rmlint2 "rmlint ..." # dedup scan +tmux new-session -d -s rsync-migrate "rsync ..." # file migration +tmux ls # check what's running +``` + +--- + +## tmux vs screen + +Both work. tmux has better split-pane support and scripting. screen is simpler and more universally installed. I use both — tmux for new jobs, screen for legacy ones. See the [screen](screen.md) article for reference. + +--- + +## Tags + +#tmux #terminal #linux #ssh #productivity #dev-tools diff --git a/03-opensource/index.md b/03-opensource/index.md index 4c3af23..da60713 100644 --- a/03-opensource/index.md +++ b/03-opensource/index.md @@ -6,7 +6,9 @@ A curated collection of my favorite open-source tools and privacy-respecting alt - [rmlint: Duplicate File Scanning](productivity/rmlint-duplicate-scanning.md) ## 🛠️ Development Tools -- *Coming soon* +- [tmux: Persistent Terminal Sessions](dev-tools/tmux.md) +- [screen: Simple Persistent Sessions](dev-tools/screen.md) +- [rsync: Fast, Resumable File Transfers](dev-tools/rsync.md) ## 🎨 Media & Creative - *Coming soon* diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index b70251f..d37ac1d 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -31,7 +31,7 @@ DNS record and Caddy entry have been removed. ## Content -- 23 articles across 5 domains +- 31 articles across 5 domains - Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/` - Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab) diff --git a/README.md b/README.md index 4ff8afd..6d776bc 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > **Last updated:** 2026-03-13 -**Article count:** 28 +**Article count:** 31 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 1 | +| 🔓 Open Source Tools | `03-opensource/` | 4 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -73,6 +73,11 @@ ### Productivity - [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation +### Development Tools +- [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) — detachable sessions for long-running jobs over SSH +- [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) — lightweight terminal multiplexer, universally available +- [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) — incremental file sync locally and over SSH, survives interruptions + --- ## 🎙️ Streaming & Podcasting @@ -99,15 +104,15 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | +| 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | +| 2026-03-13 | [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) | Open Source | | 2026-03-13 | [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) | Troubleshooting | | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | | 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | | 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | -| 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | -| 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | -| 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | --- @@ -121,4 +126,3 @@ | Pi-hole setup and local DNS | Self-Hosting | Medium | No | | OBS audio routing on Linux (PipeWire) | Streaming | Medium | No | | Nextcloud setup with Docker | Self-Hosting | Medium | No | -| tmux basics | Linux | Low | No | diff --git a/SUMMARY.md b/SUMMARY.md index c639add..0081f0d 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -5,6 +5,9 @@ * [Introduction](02-selfhosting/index.md) * [Open Source & Alternatives](03-opensource/index.md) * [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) + * [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) + * [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) + * [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) * [Streaming](04-streaming/index.md) * [Introduction](04-streaming/index.md) * [Troubleshooting](05-troubleshooting/index.md) diff --git a/index.md b/index.md index c115bc5..0b56f9a 100644 --- a/index.md +++ b/index.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 28 +> **Article count:** 31 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 1 | +| 🔓 Open Source Tools | `03-opensource/` | 4 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -73,6 +73,11 @@ ### Productivity - [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation +### Development Tools +- [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) — detachable sessions for long-running jobs over SSH +- [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) — lightweight terminal multiplexer, universally available +- [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) — incremental file sync locally and over SSH, survives interruptions + --- ## 🎙️ Streaming & Podcasting @@ -99,15 +104,15 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | +| 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | +| 2026-03-13 | [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) | Open Source | | 2026-03-13 | [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) | Troubleshooting | | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | | 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | | 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | -| 2026-03-11 | [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) | Troubleshooting | -| 2026-03-11 | [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) | Troubleshooting | -| 2026-03-11 | [yt-dlp JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) | Troubleshooting | --- @@ -120,4 +125,3 @@ | Troubleshooting NVIDIA on Linux | Troubleshooting | Medium | No | | Pi-hole setup and local DNS | Self-Hosting | Medium | No | | Nextcloud setup with Docker | Self-Hosting | Medium | No | -| tmux basics | Linux | Low | No | From 6e0ceb09725db0f231d4da0647580f2620226c76 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 13 Mar 2026 23:48:40 -0400 Subject: [PATCH 09/12] wiki: add Vaultwarden article to privacy-security section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add 03-opensource/privacy-security/vaultwarden.md covering deployment with Docker Compose, Caddy reverse proxy, client setup, access model via Tailscale, and SQLite backup. Remove KeePassXC from backlog. Article count: 31 → 32. Open source section: 4 → 5. Co-Authored-By: Claude Sonnet 4.6 --- 03-opensource/index.md | 2 +- 03-opensource/privacy-security/vaultwarden.md | 95 +++++++++++++++++++ MajorWiki-Deploy-Status.md | 2 +- README.md | 10 +- SUMMARY.md | 1 + index.md | 10 +- 6 files changed, 110 insertions(+), 10 deletions(-) create mode 100644 03-opensource/privacy-security/vaultwarden.md diff --git a/03-opensource/index.md b/03-opensource/index.md index da60713..1549781 100644 --- a/03-opensource/index.md +++ b/03-opensource/index.md @@ -14,4 +14,4 @@ A curated collection of my favorite open-source tools and privacy-respecting alt - *Coming soon* ## 🔐 Privacy & Security -- *Coming soon* +- [Vaultwarden: Self-Hosted Password Manager](privacy-security/vaultwarden.md) diff --git a/03-opensource/privacy-security/vaultwarden.md b/03-opensource/privacy-security/vaultwarden.md new file mode 100644 index 0000000..3c44427 --- /dev/null +++ b/03-opensource/privacy-security/vaultwarden.md @@ -0,0 +1,95 @@ +# Vaultwarden — Self-Hosted Password Manager + +## Problem + +Password managers are a necessity, but handing your credentials to a third-party cloud service is a trust problem. Bitwarden is open source and privacy-respecting, but if you're already running a homelab, there's no reason to depend on their servers. + +## Solution + +[Vaultwarden](https://github.com/dani-garcia/vaultwarden) is an unofficial, lightweight Bitwarden-compatible server written in Rust. It exposes the same API that all official Bitwarden clients speak — desktop apps, browser extensions, mobile apps — so you get the full Bitwarden UX pointed at your own hardware. + +Your passwords never leave your network. + +--- + +## Deployment (Docker + Caddy) + +### docker-compose.yml + +```yaml +services: + vaultwarden: + image: vaultwarden/server:latest + container_name: vaultwarden + restart: unless-stopped + environment: + - DOMAIN=https://vault.yourdomain.com + - SIGNUPS_ALLOWED=false # disable after creating your account + volumes: + - ./vw-data:/data + ports: + - "8080:80" +``` + +Start it: + +```bash +sudo docker compose up -d +``` + +### Caddy reverse proxy + +``` +vault.yourdomain.com { + reverse_proxy localhost:8080 +} +``` + +Caddy handles TLS automatically. No extra cert config needed. + +--- + +## Initial Setup + +1. Browse to `https://vault.yourdomain.com` and create your account +2. Set `SIGNUPS_ALLOWED=false` in the compose file and restart the container +3. Install any official Bitwarden client (browser extension, desktop, mobile) +4. In the client, set the **Server URL** to `https://vault.yourdomain.com` before logging in + +That's it. The client has no idea it's not talking to Bitwarden's servers. + +--- + +## Access Model + +On MajorInfrastructure, Vaultwarden runs on **majorlab** and is accessible: + +- **Internally** — via Caddy on the local network +- **Remotely** — via Tailscale; vault is reachable from any device on the tailnet without exposing it to the public internet + +This means the Caddy vhost does not need to be publicly routable. You can choose to expose it publicly (Let's Encrypt works fine) or keep it Tailscale-only. + +--- + +## Backup + +Vaultwarden stores everything in a single SQLite database at `./vw-data/db.sqlite3`. Back it up like any file: + +```bash +# Simple copy (stop container first for consistency, or use sqlite backup mode) +sqlite3 /path/to/vw-data/db.sqlite3 ".backup '/path/to/backup/vw-backup-$(date +%F).sqlite3'" +``` + +Or include the `vw-data/` directory in your regular rsync backup run. + +--- + +## Why Not Bitwarden (Official)? + +The official Bitwarden server is also open source but requires significantly more resources (multiple services, SQL Server). Vaultwarden runs in a single container on minimal RAM and handles everything a personal or family vault needs. + +--- + +## Tags + +#vaultwarden #bitwarden #passwords #privacy #self-hosting #docker #linux diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index d37ac1d..f8d93cb 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -31,7 +31,7 @@ DNS record and Caddy entry have been removed. ## Content -- 31 articles across 5 domains +- 32 articles across 5 domains - Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/` - Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab) diff --git a/README.md b/README.md index 6d776bc..427fc79 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > **Last updated:** 2026-03-13 -**Article count:** 31 +**Article count:** 32 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 4 | +| 🔓 Open Source Tools | `03-opensource/` | 5 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -78,6 +78,9 @@ - [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) — lightweight terminal multiplexer, universally available - [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) — incremental file sync locally and over SSH, survives interruptions +### Privacy & Security +- [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) — Bitwarden-compatible server in a single Docker container, passwords stay on your hardware + --- ## 🎙️ Streaming & Podcasting @@ -104,6 +107,7 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) | Open Source | | 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | | 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | | 2026-03-13 | [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) | Open Source | @@ -112,7 +116,6 @@ | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | | 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | -| 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | --- @@ -120,7 +123,6 @@ | Topic | Domain | Priority | From Gap? | |---|---|---|---| -| KeePassXC self-hosted password management | Open Source | High | No | | Docker Compose networking deep dive | Self-Hosting | High | No | | Troubleshooting NVIDIA on Linux | Troubleshooting | Medium | No | | Pi-hole setup and local DNS | Self-Hosting | Medium | No | diff --git a/SUMMARY.md b/SUMMARY.md index 0081f0d..726bd0a 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -8,6 +8,7 @@ * [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) * [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) * [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) + * [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) * [Streaming](04-streaming/index.md) * [Introduction](04-streaming/index.md) * [Troubleshooting](05-troubleshooting/index.md) diff --git a/index.md b/index.md index 0b56f9a..d3b49b0 100644 --- a/index.md +++ b/index.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 31 +> **Article count:** 32 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 4 | +| 🔓 Open Source Tools | `03-opensource/` | 5 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -78,6 +78,9 @@ - [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) — lightweight terminal multiplexer, universally available - [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) — incremental file sync locally and over SSH, survives interruptions +### Privacy & Security +- [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) — Bitwarden-compatible server in a single Docker container, passwords stay on your hardware + --- ## 🎙️ Streaming & Podcasting @@ -104,6 +107,7 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-13 | [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) | Open Source | | 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | | 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | | 2026-03-13 | [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) | Open Source | @@ -112,7 +116,6 @@ | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | | 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | -| 2026-03-12 | [Docker & Caddy Recovery After Reboot](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) | Troubleshooting | --- @@ -120,7 +123,6 @@ | Topic | Domain | Priority | From Gap? | |---|---|---|---| -| KeePassXC self-hosted password management | Open Source | High | No | | Docker Compose networking deep dive | Self-Hosting | High | No | | Troubleshooting NVIDIA on Linux | Troubleshooting | Medium | No | | Pi-hole setup and local DNS | Self-Hosting | Medium | No | From 31d0a9806d8e0c26a8793022631a622161ee1378 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Sat, 14 Mar 2026 00:33:58 -0400 Subject: [PATCH 10/12] wiki: add yt-dlp article to media-creative section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cover installation, Plex-optimized format selection, playlist downloading, config file, and background session usage. Cross-reference existing JS challenge troubleshooting article. Article count: 32 → 33. Open source section: 5 → 6. Co-Authored-By: Claude Sonnet 4.6 --- 03-opensource/index.md | 2 +- 03-opensource/media-creative/yt-dlp.md | 131 +++++++++++++++++++++++++ MajorWiki-Deploy-Status.md | 2 +- README.md | 9 +- SUMMARY.md | 1 + index.md | 9 +- 6 files changed, 146 insertions(+), 8 deletions(-) create mode 100644 03-opensource/media-creative/yt-dlp.md diff --git a/03-opensource/index.md b/03-opensource/index.md index 1549781..957eeed 100644 --- a/03-opensource/index.md +++ b/03-opensource/index.md @@ -11,7 +11,7 @@ A curated collection of my favorite open-source tools and privacy-respecting alt - [rsync: Fast, Resumable File Transfers](dev-tools/rsync.md) ## 🎨 Media & Creative -- *Coming soon* +- [yt-dlp: Video Downloading](media-creative/yt-dlp.md) ## 🔐 Privacy & Security - [Vaultwarden: Self-Hosted Password Manager](privacy-security/vaultwarden.md) diff --git a/03-opensource/media-creative/yt-dlp.md b/03-opensource/media-creative/yt-dlp.md new file mode 100644 index 0000000..c1a4d72 --- /dev/null +++ b/03-opensource/media-creative/yt-dlp.md @@ -0,0 +1,131 @@ +# yt-dlp — Video Downloading + +## What It Is + +`yt-dlp` is a feature-rich command-line video downloader, forked from youtube-dl with active maintenance and significantly better performance. It supports YouTube, Twitch, and hundreds of other sites. + +--- + +## Installation + +### Fedora +```bash +sudo dnf install yt-dlp +# or latest via pip: +sudo pip install yt-dlp --break-system-packages +``` + +### Update +```bash +sudo pip install -U yt-dlp --break-system-packages +# or if installed as standalone binary: +yt-dlp -U +``` + +Keep it current — YouTube pushes extractor changes frequently and old versions break. + +--- + +## Basic Usage + +```bash +# Download a single video (best quality) +yt-dlp https://www.youtube.com/watch?v=VIDEO_ID + +# Download to a specific directory with title as filename +yt-dlp -o "/path/to/output/%(title)s.%(ext)s" URL +``` + +--- + +## Plex-Optimized Download + +For Plex direct play, you want H.264 video in an MP4 container with embedded subtitles: + +```bash +yt-dlp -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio' \ + --merge-output-format mp4 \ + -o "/plex/plex/%(title)s.%(ext)s" \ + --write-auto-subs --embed-subs \ + URL +``` + +- `-f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/...'` — prefer MP4 video + M4A audio; fall back to best available +- `--merge-output-format mp4` — merge streams into MP4 container (requires ffmpeg) +- `--write-auto-subs --embed-subs` — download auto-generated subtitles and bake them in + +--- + +## Playlists and Channels + +```bash +# Download a full playlist +yt-dlp -o "%(playlist_index)s - %(title)s.%(ext)s" PLAYLIST_URL + +# Download only videos not already present +yt-dlp --download-archive archive.txt PLAYLIST_URL +``` + +`--download-archive` maintains a file of completed video IDs — re-running the command skips already-downloaded videos automatically. + +--- + +## Format Selection + +```bash +# List all available formats for a video +yt-dlp --list-formats URL + +# Download best video + best audio, merge to mp4 +yt-dlp -f 'bestvideo+bestaudio' --merge-output-format mp4 URL + +# Download audio only (MP3) +yt-dlp -x --audio-format mp3 URL +``` + +--- + +## Config File + +Persist your preferred flags so you don't repeat them every command: + +```bash +mkdir -p ~/.config/yt-dlp +cat > ~/.config/yt-dlp/config << 'EOF' +-f bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio +--merge-output-format mp4 +--write-auto-subs +--embed-subs +--remote-components ejs:github +EOF +``` + +After this, a bare `yt-dlp URL` uses all your preferred settings automatically. + +--- + +## Running Long Downloads in the Background + +For large downloads or playlists, run inside `screen` or `tmux` so they survive SSH disconnects: + +```bash +screen -dmS yt-download bash -c \ + "yt-dlp -o '/plex/plex/%(title)s.%(ext)s' PLAYLIST_URL 2>&1 | tee ~/yt-download.log" + +# Check progress +screen -r yt-download +# or +tail -f ~/yt-download.log +``` + +--- + +## Troubleshooting + +For YouTube JS challenge errors, missing formats, and n-challenge failures on Fedora — see [yt-dlp YouTube JS Challenge Fix](../../05-troubleshooting/yt-dlp-fedora-js-challenge.md). + +--- + +## Tags + +#yt-dlp #youtube #video #plex #linux #media #dev-tools diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index f8d93cb..6874199 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -31,7 +31,7 @@ DNS record and Caddy entry have been removed. ## Content -- 32 articles across 5 domains +- 33 articles across 5 domains - Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/` - Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab) diff --git a/README.md b/README.md index 427fc79..7aa9035 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > **Last updated:** 2026-03-13 -**Article count:** 32 +**Article count:** 33 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 5 | +| 🔓 Open Source Tools | `03-opensource/` | 6 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -81,6 +81,9 @@ ### Privacy & Security - [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) — Bitwarden-compatible server in a single Docker container, passwords stay on your hardware +### Media & Creative +- [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) — download from YouTube and hundreds of other sites, Plex-optimized format selection + --- ## 🎙️ Streaming & Podcasting @@ -107,6 +110,7 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-14 | [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) | Open Source | | 2026-03-13 | [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) | Open Source | | 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | | 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | @@ -115,7 +119,6 @@ | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | -| 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | --- diff --git a/SUMMARY.md b/SUMMARY.md index 726bd0a..8bd77ed 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -9,6 +9,7 @@ * [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) * [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) * [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) + * [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) * [Streaming](04-streaming/index.md) * [Introduction](04-streaming/index.md) * [Troubleshooting](05-troubleshooting/index.md) diff --git a/index.md b/index.md index d3b49b0..6b647fb 100644 --- a/index.md +++ b/index.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 32 +> **Article count:** 33 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 5 | +| 🔓 Open Source Tools | `03-opensource/` | 6 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -81,6 +81,9 @@ ### Privacy & Security - [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) — Bitwarden-compatible server in a single Docker container, passwords stay on your hardware +### Media & Creative +- [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) — download from YouTube and hundreds of other sites, Plex-optimized format selection + --- ## 🎙️ Streaming & Podcasting @@ -107,6 +110,7 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-14 | [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) | Open Source | | 2026-03-13 | [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) | Open Source | | 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | | 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | @@ -115,7 +119,6 @@ | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | | 2026-03-13 | [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) | Troubleshooting | -| 2026-03-13 | [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) | Troubleshooting | --- From b81c8feda0ab5800bd02e410109d82c5d58d6230 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Sat, 14 Mar 2026 00:37:42 -0400 Subject: [PATCH 11/12] wiki: add alternatives section with SearXNG, FreshRSS, and Gitea MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add three new articles to 03-opensource/alternatives/: - SearXNG: private metasearch, Open WebUI integration - FreshRSS: self-hosted RSS, mobile app sync, OPML portability - Gitea: lightweight GitHub alternative, webhook pipeline Article count: 33 → 36. Open source section: 6 → 9. Co-Authored-By: Claude Sonnet 4.6 --- 03-opensource/alternatives/freshrss.md | 89 ++++++++++++++++++++++++ 03-opensource/alternatives/gitea.md | 95 ++++++++++++++++++++++++++ 03-opensource/alternatives/searxng.md | 88 ++++++++++++++++++++++++ 03-opensource/index.md | 5 ++ MajorWiki-Deploy-Status.md | 2 +- README.md | 15 ++-- SUMMARY.md | 3 + index.md | 15 ++-- 8 files changed, 301 insertions(+), 11 deletions(-) create mode 100644 03-opensource/alternatives/freshrss.md create mode 100644 03-opensource/alternatives/gitea.md create mode 100644 03-opensource/alternatives/searxng.md diff --git a/03-opensource/alternatives/freshrss.md b/03-opensource/alternatives/freshrss.md new file mode 100644 index 0000000..3d3b59e --- /dev/null +++ b/03-opensource/alternatives/freshrss.md @@ -0,0 +1,89 @@ +# FreshRSS — Self-Hosted RSS Reader + +## Problem + +RSS is the best way to follow websites, blogs, and podcasts without algorithmic feeds, engagement bait, or data harvesting. But hosted RSS services like Feedly gate features behind subscriptions and still have access to your reading habits. Google killed Google Reader in 2013 and has been trying to kill RSS ever since. + +## Solution + +[FreshRSS](https://freshrss.org) is a self-hosted RSS aggregator. It fetches and stores your feeds on your own server, presents a clean reading interface, and syncs with mobile apps via standard APIs (Fever, Google Reader, Nextcloud News). No subscription, no tracking, no feed limits. + +--- + +## Deployment (Docker) + +```yaml +services: + freshrss: + image: freshrss/freshrss:latest + container_name: freshrss + restart: unless-stopped + ports: + - "8086:80" + volumes: + - ./freshrss/data:/var/www/FreshRSS/data + - ./freshrss/extensions:/var/www/FreshRSS/extensions + environment: + - TZ=America/New_York + - CRON_MIN=*/15 # fetch feeds every 15 minutes +``` + +### Caddy reverse proxy + +``` +rss.yourdomain.com { + reverse_proxy localhost:8086 +} +``` + +--- + +## Initial Setup + +1. Browse to your FreshRSS URL and run through the setup wizard +2. Create an admin account +3. Go to **Settings → Authentication** — enable API access if you want mobile app sync +4. Start adding feeds under **Subscriptions → Add a feed** + +--- + +## Mobile App Sync + +FreshRSS exposes a Google Reader-compatible API that most RSS apps support: + +| App | Platform | Protocol | +|---|---|---| +| NetNewsWire | iOS / macOS | Fever or GReader | +| Reeder | iOS / macOS | GReader | +| ReadYou | Android | GReader | +| FeedMe | Android | GReader / Fever | + +**API URL format:** `https://rss.yourdomain.com/api/greader.php` + +Enable the API in FreshRSS: **Settings → Authentication → Allow API access** + +--- + +## Feed Auto-Refresh + +The `CRON_MIN=*/15` environment variable runs feed fetching every 15 minutes inside the container. For more control, add a host-level cron job: + +```bash +# Fetch all feeds every 10 minutes +*/10 * * * * docker exec freshrss php /var/www/FreshRSS/app/actualize_script.php +``` + +--- + +## Why RSS Over Social Media + +- **You control the feed** — no algorithm decides what you see or in what order +- **No engagement optimization** — content ranked by publish date, not outrage potential +- **Portable** — OPML export lets you move your subscriptions to any reader +- **Works forever** — RSS has been around since 1999 and isn't going anywhere + +--- + +## Tags + +#freshrss #rss #self-hosting #docker #linux #alternatives #privacy diff --git a/03-opensource/alternatives/gitea.md b/03-opensource/alternatives/gitea.md new file mode 100644 index 0000000..47ad57c --- /dev/null +++ b/03-opensource/alternatives/gitea.md @@ -0,0 +1,95 @@ +# Gitea — Self-Hosted Git + +## Problem + +GitHub is the default home for code, but it's a Microsoft-owned centralized service. Your repositories, commit history, issues, and CI/CD pipelines are all under someone else's control. For personal projects and private infrastructure, there's no reason to depend on it. + +## Solution + +[Gitea](https://gitea.com) is a lightweight, self-hosted Git service. It provides the full GitHub-style workflow — repositories, branches, pull requests, webhooks, and a web UI — in a single binary or Docker container that runs comfortably on low-spec hardware. + +--- + +## Deployment (Docker) + +```yaml +services: + gitea: + image: docker.gitea.com/gitea:latest + container_name: gitea + restart: unless-stopped + ports: + - "3002:3000" + - "222:22" # SSH git access + volumes: + - ./gitea:/data + environment: + - USER_UID=1000 + - USER_GID=1000 + - GITEA__database__DB_TYPE=sqlite3 +``` + +SQLite is fine for personal use. For team use, swap in PostgreSQL or MySQL. + +### Caddy reverse proxy + +``` +git.yourdomain.com { + reverse_proxy localhost:3002 +} +``` + +--- + +## Initial Setup + +1. Browse to your Gitea URL — the first-run wizard handles configuration +2. Set the server URL to your public domain +3. Create an admin account +4. Configure SSH access if you want `git@git.yourdomain.com` cloning + +--- + +## Webhooks + +Gitea's webhook system is how automated pipelines get triggered on push. Example use case — auto-deploy a MkDocs wiki on every push: + +1. Go to repo → **Settings → Webhooks → Add Webhook** +2. Set the payload URL to your webhook endpoint (e.g. `https://notes.yourdomain.com/webhook`) +3. Set content type to `application/json` +4. Select **Push events** + +The webhook fires on every `git push`, allowing the receiving server to pull and rebuild automatically. See [MajorWiki Setup & Pipeline](../../05-troubleshooting/majwiki-setup-and-pipeline.md) for a complete example. + +--- + +## Migrating from GitHub + +Gitea can mirror GitHub repos and import them directly: + +```bash +# Clone from GitHub, push to Gitea +git clone --mirror https://github.com/user/repo.git +cd repo.git +git remote set-url origin https://git.yourdomain.com/user/repo.git +git push --mirror +``` + +Or use the Gitea web UI: **+ → New Migration → GitHub** + +--- + +## Why Not Just Use GitHub? + +For public open source — GitHub is fine, the network effects are real. For private infrastructure code, personal projects, and anything you'd rather not hand to Microsoft: + +- Full control over your data and access +- No rate limits, no storage quotas on your own hardware +- Webhooks and integrations without paying for GitHub Actions minutes +- Works entirely over Tailscale — no public exposure required + +--- + +## Tags + +#gitea #git #self-hosting #docker #linux #alternatives #vcs diff --git a/03-opensource/alternatives/searxng.md b/03-opensource/alternatives/searxng.md new file mode 100644 index 0000000..1bc67b5 --- /dev/null +++ b/03-opensource/alternatives/searxng.md @@ -0,0 +1,88 @@ +# SearXNG — Private Self-Hosted Search + +## Problem + +Every search query sent to Google, Bing, or DuckDuckGo is logged, profiled, and used to build an advertising model of you. Even "private" search engines are still third-party services with their own data retention policies. + +## Solution + +[SearXNG](https://github.com/searxng/searxng) is a self-hosted metasearch engine. It queries multiple search engines simultaneously on your behalf — without sending any identifying information — and aggregates the results. The search engines see a request from your server, not from you. + +Your queries stay on your infrastructure. + +--- + +## Deployment (Docker) + +```yaml +services: + searxng: + image: searxng/searxng:latest + container_name: searxng + restart: unless-stopped + ports: + - "8090:8080" + volumes: + - ./searxng:/etc/searxng + environment: + - SEARXNG_BASE_URL=https://search.yourdomain.com/ +``` + +SearXNG requires a `settings.yml` in the mounted config directory. Generate one from the default: + +```bash +docker run --rm searxng/searxng cat /etc/searxng/settings.yml > ./searxng/settings.yml +``` + +Key settings to configure in `settings.yml`: + +```yaml +server: + secret_key: "generate-a-random-string-here" + bind_address: "0.0.0.0" + +search: + safe_search: 0 + default_lang: "en" + +engines: + # Enable/disable specific engines here +``` + +### Caddy reverse proxy + +``` +search.yourdomain.com { + reverse_proxy localhost:8090 +} +``` + +--- + +## Using SearXNG as an AI Search Backend + +SearXNG integrates directly with Open WebUI as a web search provider, giving your local AI access to current web results without any third-party API keys: + +**Open WebUI → Settings → Web Search:** +- Enable web search +- Set provider to `searxng` +- Set URL to `http://searxng:8080` (internal Docker network) or your Tailscale/local address + +This is how MajorTwin gets current web context — queries go through SearXNG, not Google. + +--- + +## Why Not DuckDuckGo? + +DDG is better than Google for privacy, but it's still a centralized third-party service. SearXNG: + +- Runs on your own hardware +- Has no account, no cookies, no session tracking +- Lets you choose which upstream engines to use and weight +- Can be kept entirely off the public internet (Tailscale-only) + +--- + +## Tags + +#searxng #search #privacy #self-hosting #docker #linux #alternatives diff --git a/03-opensource/index.md b/03-opensource/index.md index 957eeed..6fcf4d9 100644 --- a/03-opensource/index.md +++ b/03-opensource/index.md @@ -2,6 +2,11 @@ A curated collection of my favorite open-source tools and privacy-respecting alternatives to mainstream software. +## 🔄 Alternatives +- [SearXNG: Private Self-Hosted Search](alternatives/searxng.md) +- [FreshRSS: Self-Hosted RSS Reader](alternatives/freshrss.md) +- [Gitea: Self-Hosted Git](alternatives/gitea.md) + ## 🚀 Productivity - [rmlint: Duplicate File Scanning](productivity/rmlint-duplicate-scanning.md) diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index 6874199..976602a 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -31,7 +31,7 @@ DNS record and Caddy entry have been removed. ## Content -- 33 articles across 5 domains +- 36 articles across 5 domains - Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/` - Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab) diff --git a/README.md b/README.md index 7aa9035..2d5456c 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > **Last updated:** 2026-03-13 -**Article count:** 33 +**Article count:** 36 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 6 | +| 🔓 Open Source Tools | `03-opensource/` | 9 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -70,6 +70,11 @@ ## 🔓 Open Source Tools +### Alternatives +- [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) — metasearch engine that queries multiple engines without exposing your identity +- [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) — algorithm-free feed aggregator with mobile app sync +- [Gitea: Self-Hosted Git](03-opensource/alternatives/gitea.md) — lightweight GitHub alternative, webhooks, single Docker container + ### Productivity - [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation @@ -110,11 +115,11 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-14 | [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) | Open Source | +| 2026-03-14 | [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) | Open Source | +| 2026-03-14 | [Gitea: Self-Hosted Git](03-opensource/alternatives/gitea.md) | Open Source | | 2026-03-14 | [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) | Open Source | | 2026-03-13 | [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) | Open Source | -| 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | -| 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | -| 2026-03-13 | [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) | Open Source | | 2026-03-13 | [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) | Troubleshooting | | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | diff --git a/SUMMARY.md b/SUMMARY.md index 8bd77ed..c24a943 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -4,6 +4,9 @@ * [Self-Hosting](02-selfhosting/index.md) * [Introduction](02-selfhosting/index.md) * [Open Source & Alternatives](03-opensource/index.md) + * [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) + * [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) + * [Gitea: Self-Hosted Git](03-opensource/alternatives/gitea.md) * [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) * [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) * [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) diff --git a/index.md b/index.md index 6b647fb..62454a5 100644 --- a/index.md +++ b/index.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-13 -> **Article count:** 33 +> **Article count:** 36 ## Domains @@ -11,7 +11,7 @@ |---|---|---| | 🐧 Linux & Sysadmin | `01-linux/` | 9 | | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | -| 🔓 Open Source Tools | `03-opensource/` | 6 | +| 🔓 Open Source Tools | `03-opensource/` | 9 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | | 🔧 General Troubleshooting | `05-troubleshooting/` | 9 | @@ -70,6 +70,11 @@ ## 🔓 Open Source Tools +### Alternatives +- [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) — metasearch engine that queries multiple engines without exposing your identity +- [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) — algorithm-free feed aggregator with mobile app sync +- [Gitea: Self-Hosted Git](03-opensource/alternatives/gitea.md) — lightweight GitHub alternative, webhooks, single Docker container + ### Productivity - [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) — extremely fast duplicate file finding and storage reclamation @@ -110,11 +115,11 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-14 | [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) | Open Source | +| 2026-03-14 | [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) | Open Source | +| 2026-03-14 | [Gitea: Self-Hosted Git](03-opensource/alternatives/gitea.md) | Open Source | | 2026-03-14 | [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) | Open Source | | 2026-03-13 | [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) | Open Source | -| 2026-03-13 | [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) | Open Source | -| 2026-03-13 | [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) | Open Source | -| 2026-03-13 | [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) | Open Source | | 2026-03-13 | [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) | Troubleshooting | | 2026-03-13 | [rmlint: Duplicate File Scanning](03-opensource/productivity/rmlint-duplicate-scanning.md) | Open Source | | 2026-03-13 | [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) | Linux | From deb32ce756056fa6711146da1507c0711fcc2e17 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Sat, 14 Mar 2026 11:01:21 -0400 Subject: [PATCH 12/12] wiki: expand SUMMARY.md to include all articles across all sections Co-Authored-By: Claude Sonnet 4.6 --- SUMMARY.md | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index c24a943..df5a2ae 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,8 +1,23 @@ * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) - * [Storage: SnapRAID & MergerFS Setup](01-linux/storage/snapraid-mergerfs-setup.md) -* [Self-Hosting](02-selfhosting/index.md) - * [Introduction](02-selfhosting/index.md) + * [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md) + * [Managing Linux Services with systemd](01-linux/process-management/managing-linux-services-systemd-ansible.md) + * [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) + * [Package Management Reference](01-linux/packages/package-management-reference.md) + * [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md) + * [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md) + * [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) + * [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) + * [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md) +* [Self-Hosting & Homelab](02-selfhosting/index.md) + * [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) + * [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md) + * [Debugging Broken Docker Containers](02-selfhosting/docker/debugging-broken-docker-containers.md) + * [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) + * [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) + * [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) + * [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md) + * [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) * [Open Source & Alternatives](03-opensource/index.md) * [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) * [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) @@ -13,14 +28,14 @@ * [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) * [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) * [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) -* [Streaming](04-streaming/index.md) - * [Introduction](04-streaming/index.md) +* [Streaming & Podcasting](04-streaming/index.md) + * [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) * [Troubleshooting](05-troubleshooting/index.md) - * [ISP SNI Filtering & Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) - * [yt-dlp YouTube JS Challenge Fix](05-troubleshooting/yt-dlp-fedora-js-challenge.md) - * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) * [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) + * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) + * [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) * [Obsidian Vault Recovery — Loading Cache Hang](05-troubleshooting/obsidian-cache-hang-recovery.md) - * [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) * [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) + * [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) + * [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) * [MajorWiki Setup & Publishing Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md)