diff --git a/01-linux/distro-specific/wsl2-backup-powershell.md b/01-linux/distro-specific/wsl2-backup-powershell.md new file mode 100644 index 0000000..d8c83a9 --- /dev/null +++ b/01-linux/distro-specific/wsl2-backup-powershell.md @@ -0,0 +1,86 @@ +--- +title: WSL2 Backup via PowerShell Scheduled Task +domain: linux +category: distro-specific +tags: + - wsl2 + - windows + - backup + - powershell + - majorrig +status: published +created: '2026-03-16' +updated: '2026-03-16' +--- + +# WSL2 Backup via PowerShell Scheduled Task + +WSL2 distributions are stored as a VHDX file on disk. Unlike traditional VMs, there's no built-in snapshot or backup mechanism. This article covers a simple weekly backup strategy using `wsl --export` and a PowerShell scheduled task. + +## The Short Answer + +Save this as `C:\Users\majli\Scripts\backup-wsl.ps1` and register it as a weekly scheduled task. + +## Backup Script + +```powershell +$BackupDir = "D:\WSL\Backups" +$Date = Get-Date -Format "yyyy-MM-dd" +$BackupFile = "$BackupDir\FedoraLinux-43-$Date.tar" +$MaxBackups = 3 + +New-Item -ItemType Directory -Force -Path $BackupDir | Out-Null + +# Must shut down WSL first — export fails if VHDX is locked +Write-Host "Shutting down WSL2..." +wsl --shutdown +Start-Sleep -Seconds 5 + +Write-Host "Backing up FedoraLinux-43 to $BackupFile..." +wsl --export FedoraLinux-43 $BackupFile + +if ($LASTEXITCODE -eq 0) { + Write-Host "Backup complete: $BackupFile" + Get-ChildItem "$BackupDir\FedoraLinux-43-*.tar" | + Sort-Object LastWriteTime -Descending | + Select-Object -Skip $MaxBackups | + Remove-Item -Force + Write-Host "Cleanup done. Keeping last $MaxBackups backups." +} else { + Write-Host "ERROR: Backup failed!" +} +``` + +## Register the Scheduled Task + +Run in PowerShell as Administrator: + +```powershell +$Action = New-ScheduledTaskAction -Execute "PowerShell.exe" ` + -Argument "-NonInteractive -File C:\Users\majli\Scripts\backup-wsl.ps1" +$Trigger = New-ScheduledTaskTrigger -Weekly -DaysOfWeek Sunday -At 2am +$Settings = New-ScheduledTaskSettingsSet -StartWhenAvailable -RunOnlyIfNetworkAvailable:$false +Register-ScheduledTask -TaskName "WSL2 Backup - FedoraLinux43" ` + -Action $Action -Trigger $Trigger -Settings $Settings ` + -RunLevel Highest -Force +``` + +## Restore from Backup + +```powershell +wsl --unregister FedoraLinux-43 +wsl --import FedoraLinux-43 D:\WSL\Fedora43 D:\WSL\Backups\FedoraLinux-43-YYYY-MM-DD.tar +``` + +Then fix the default user — after import WSL resets to root. See [[wsl2-instance-migration-fedora43|WSL2 Instance Migration]] for the `/etc/wsl.conf` fix. + +## Gotchas + +- **`wsl --export` fails with `ERROR_SHARING_VIOLATION` if WSL is running.** The script includes `wsl --shutdown` before export to handle this. Any active WSL sessions will be terminated — schedule the task for a time when WSL is idle (2am works well). +- **Backblaze picks up D:\WSL\Backups\ automatically** if D: drive is in scope — provides offsite backup without extra config. +- **Each backup tar is ~500MB–1GB** depending on what's installed. Keep MaxBackups at 3 to balance retention vs disk usage. + +## See Also + +- [[wsl2-instance-migration-fedora43|WSL2 Instance Migration]] +- [[wsl2-rebuild-fedora43-training-env|WSL2 Training Environment Rebuild]] diff --git a/01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md b/01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md new file mode 100644 index 0000000..601f840 --- /dev/null +++ b/01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md @@ -0,0 +1,203 @@ +--- +title: WSL2 Fedora 43 Training Environment Rebuild +domain: linux +category: distro-specific +tags: + - wsl2 + - fedora + - unsloth + - pytorch + - cuda + - majorrig + - majortwin +status: published +created: '2026-03-16' +updated: '2026-03-16' +--- + +# WSL2 Fedora 43 Training Environment Rebuild + +How to rebuild the MajorTwin training environment from scratch on MajorRig after a WSL2 loss. Covers Fedora 43 install, Python 3.11 via pyenv, PyTorch with CUDA, Unsloth, and llama.cpp for GGUF conversion. + +## The Short Answer + +```bash +# 1. Install Fedora 43 and move to D: +wsl --install -d FedoraLinux-43 --no-launch +wsl --export FedoraLinux-43 D:\WSL\fedora43.tar +wsl --unregister FedoraLinux-43 +wsl --import FedoraLinux-43 D:\WSL\Fedora43 D:\WSL\fedora43.tar + +# 2. Set default user +echo -e "[boot]\nsystemd=true\n[user]\ndefault=majorlinux" | sudo tee /etc/wsl.conf +useradd -m -G wheel majorlinux && passwd majorlinux +echo "%wheel ALL=(ALL) ALL" | sudo tee /etc/sudoers.d/wheel + +# 3. Install Python 3.11 via pyenv, PyTorch, Unsloth +# See full steps below +``` + +## Step 1 — System Packages + +```bash +sudo dnf update -y +sudo dnf install -y git curl wget tmux screen htop rsync unzip \ + python3 python3-pip python3-devel gcc gcc-c++ make cmake \ + ninja-build pkg-config openssl-devel libffi-devel \ + gawk patch readline-devel sqlite-devel +``` + +## Step 2 — Python 3.11 via pyenv + +Fedora 43 ships Python 3.13. Unsloth requires 3.11. Use pyenv: + +```bash +curl https://pyenv.run | bash + +# Add to ~/.bashrc +export PYENV_ROOT="$HOME/.pyenv" +[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH" +eval "$(pyenv init - bash)" + +source ~/.bashrc +pyenv install 3.11.9 +pyenv global 3.11.9 +``` + +The tkinter warning during install is harmless — it's not needed for training. + +## Step 3 — Training Virtualenv + PyTorch + +```bash +mkdir -p ~/majortwin/{staging,datasets,outputs,scripts} +python -m venv ~/majortwin/venv +source ~/majortwin/venv/bin/activate + +pip install --upgrade pip +pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128 + +# Verify GPU +python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" +``` + +Expected output: `True NVIDIA GeForce RTX 3080 Ti` + +## Step 4 — Unsloth + Training Stack + +```bash +source ~/majortwin/venv/bin/activate + +pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" +pip install transformers datasets accelerate peft trl bitsandbytes \ + sentencepiece protobuf scipy einops + +# Pin transformers for unsloth-zoo compatibility +pip install "transformers<=5.2.0" + +# Verify +python -c "import unsloth; print('Unsloth OK')" +``` + +> [!warning] Never run `pip install -r requirements.txt` from inside llama.cpp while the training venv is active. It installs CPU-only PyTorch and downgrades transformers, breaking the CUDA setup. + +## Step 5 — llama.cpp (CPU-only for GGUF conversion) + +CUDA 12.8 is incompatible with Fedora 43's glibc for compiling llama.cpp (math function conflicts in `/usr/include/bits/mathcalls.h`). Build CPU-only — it's sufficient for GGUF conversion, which doesn't need GPU: + +```bash +# Install GCC 14 (CUDA 12.8 doesn't support GCC 15 which Fedora 43 ships) +sudo dnf install -y gcc14 gcc14-c++ + +cd ~/majortwin +git clone https://github.com/ggerganov/llama.cpp.git +cd llama.cpp + +cmake -B build \ + -DGGML_CUDA=OFF \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_C_COMPILER=/usr/bin/gcc-14 \ + -DCMAKE_CXX_COMPILER=/usr/bin/g++-14 + +cmake --build build --config Release -j$(nproc) 2>&1 | tee /tmp/llama_build.log & +tail -f /tmp/llama_build.log +``` + +Verify: +```bash +ls ~/majortwin/llama.cpp/build/bin/llama-quantize && echo "OK" +ls ~/majortwin/llama.cpp/build/bin/llama-cli && echo "OK" +``` + +## Step 6 — Shell Environment + +```bash +cat >> ~/.bashrc << 'EOF' +# MajorInfrastructure Paths +export VAULT="/mnt/c/Users/majli/Documents/MajorVault" +export MAJORANSIBLE="/mnt/d/MajorAnsible" +export MAJORTWIN_D="/mnt/d/MajorTwin" +export MAJORTWIN_WSL="$HOME/majortwin" +export LLAMA_CPP="$HOME/majortwin/llama.cpp" + +# Venv +alias mtwin='source $MAJORTWIN_WSL/venv/bin/activate && cd $MAJORTWIN_WSL' +alias vault='cd $VAULT' +alias ll='ls -lah --color=auto' + +# SSH Fleet Aliases +alias majorhome='ssh majorlinux@100.120.209.106' +alias dca='ssh root@100.104.11.146' +alias majortoot='ssh root@100.110.197.17' +alias majorlinuxvm='ssh root@100.87.200.5' +alias majordiscord='ssh root@100.122.240.83' +alias majorlab='ssh root@100.86.14.126' +alias majormail='ssh root@100.84.165.52' +alias teelia='ssh root@100.120.32.69' +alias tttpod='ssh root@100.84.42.102' +alias majorrig='ssh -p 2222 majorlinux@100.98.47.29' + +# DNF5 +alias update='sudo dnf upgrade --refresh' +alias install='sudo dnf install' +alias clean='sudo dnf clean all' + +# MajorTwin helpers +stage_dataset() { + cp "$VAULT/20-Projects/MajorTwin/03-Datasets/$1" "$MAJORTWIN_WSL/datasets/" + echo "Staged: $1" +} +export_gguf() { + cp "$MAJORTWIN_WSL/outputs/$1" "$MAJORTWIN_D/models/" + echo "Exported: $1 → $MAJORTWIN_D/models/" +} +EOF + +source ~/.bashrc +``` + +## Key Rules + +- **Always activate venv before pip installs:** `source ~/majortwin/venv/bin/activate` +- **Never train from /mnt/c or /mnt/d** — stage files in `~/majortwin/staging/` first +- **Never put ML artifacts inside MajorVault** — models, venvs, artifacts go on D: drive +- **Max viable training model:** 7B at QLoRA 4-bit (RTX 3080 Ti, 12GB VRAM) +- **Current base model:** Qwen2.5-7B-Instruct (ChatML format — stop token: `<|im_end|>` only) +- **Transformers must be pinned:** `pip install "transformers<=5.2.0"` for unsloth-zoo compatibility + +## D: Drive Layout + +``` +D:\MajorTwin\ + models\ ← finished GGUFs + datasets\ ← dataset archives + artifacts\ ← training run artifacts + training-runs\ ← logs, checkpoints +D:\WSL\ + Fedora43\ ← WSL2 VHDX + Backups\ ← weekly WSL2 backup tars +``` + +## See Also + +- [[wsl2-instance-migration-fedora43|WSL2 Instance Migration]] +- [[wsl2-backup-powershell|WSL2 Backup via PowerShell]] diff --git a/02-selfhosting/security/ansible-unattended-upgrades-fleet.md b/02-selfhosting/security/ansible-unattended-upgrades-fleet.md new file mode 100644 index 0000000..1d40b9d --- /dev/null +++ b/02-selfhosting/security/ansible-unattended-upgrades-fleet.md @@ -0,0 +1,94 @@ +--- +title: Standardizing unattended-upgrades Across Ubuntu Fleet with Ansible +domain: selfhosting +category: security +tags: + - ansible + - ubuntu + - apt + - unattended-upgrades + - fleet-management +status: published +created: '2026-03-16' +updated: '2026-03-16' +--- + +# Standardizing unattended-upgrades Across Ubuntu Fleet with Ansible + +When some Ubuntu hosts in a fleet self-update via `unattended-upgrades` and others don't, they drift apart over time — different kernel versions, different reboot states, inconsistent behavior. This article covers how to diagnose the drift and enforce uniform auto-update config across all Ubuntu hosts using Ansible. + +## Diagnosing the Problem + +If only some Ubuntu hosts are flagging for reboot, check: + +```bash +# What triggered the reboot flag? +cat /var/run/reboot-required.pkgs + +# Is unattended-upgrades installed and active? +systemctl status unattended-upgrades +cat /etc/apt/apt.conf.d/20auto-upgrades + +# When did apt last run? +ls -lt /var/log/apt/history.log* +``` + +The reboot flag is written to `/var/run/reboot-required` by `update-notifier-common` when packages like the kernel, glibc, or systemd are updated. If some hosts have `unattended-upgrades` running and others don't, the ones that self-updated will flag for reboot while the others lag behind. + +## The Fix — Ansible Playbook + +Add these tasks to your update playbook **before** the apt cache update step: + +```yaml +- name: Ensure unattended-upgrades is installed on Ubuntu servers + ansible.builtin.apt: + name: + - unattended-upgrades + - update-notifier-common + state: present + update_cache: true + when: ansible_facts['os_family'] == "Debian" + +- name: Enforce uniform auto-update config on Ubuntu servers + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + content: | + APT::Periodic::Update-Package-Lists "1"; + APT::Periodic::Unattended-Upgrade "1"; + owner: root + group: root + mode: '0644' + when: ansible_facts['os_family'] == "Debian" + +- name: Ensure unattended-upgrades service is enabled and running + ansible.builtin.systemd: + name: unattended-upgrades + enabled: true + state: started + when: ansible_facts['os_family'] == "Debian" +``` + +Running this across the `ubuntu` group ensures every host has the same config on every Ansible run — idempotent and safe. + +## Rebooting Flagged Hosts + +Once identified, reboot specific hosts without touching the rest: + +```bash +# Reboot just the flagging hosts +ansible-playbook reboot.yml -l teelia,tttpod + +# Run full update on remaining hosts to bring them up to the same kernel +ansible-playbook update.yml -l dca,majorlinux,majortoot +``` + +## Notes + +- `unattended-upgrades` runs daily on its own schedule — hosts that haven't checked yet will lag behind but catch up within 24 hours +- The other hosts showing `ok` (not `changed`) on the config tasks means they were already correctly configured +- After a kernel update is pulled, only an actual reboot clears the `/var/run/reboot-required` flag — Ansible reporting the flag is informational only + +## See Also + +- [[ansible-getting-started|Ansible Getting Started]] +- [[linux-server-hardening-checklist|Linux Server Hardening Checklist]] diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index d1fe1b6..244d108 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -102,3 +102,18 @@ Every time a new article is added, the following **MUST** be updated to maintain - [[MajorRig|MajorRig]] — alternative git push host (WSL2 path documented) - [[03-11-2026|Status Update 2026-03-11]] — deployment date journal entry - [[03-13-2026|Status Update 2026-03-13]] — content expansion and SUMMARY.md sync + +--- + +## Session Update — 2026-03-16 + +**Article count:** 45 (was 42) + +**New articles added:** +- `01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md` — full MajorTwin training env rebuild guide +- `01-linux/distro-specific/wsl2-backup-powershell.md` — WSL2 backup via PowerShell scheduled task +- `02-selfhosting/security/ansible-unattended-upgrades-fleet.md` — standardizing unattended-upgrades across Ubuntu fleet + +**SUMMARY.md:** Updated to include all 3 new articles. Run SUMMARY.md dedup script if duplicate content appears (see board file cleanup pattern). + +**Updated:** `updated: 2026-03-16` diff --git a/SUMMARY.md b/SUMMARY.md index c4a38f2..e522a14 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -9,6 +9,8 @@ * [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) * [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) * [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md) + * [WSL2 Training Environment Rebuild](01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md) + * [WSL2 Backup via PowerShell](01-linux/distro-specific/wsl2-backup-powershell.md) * [Self-Hosting & Homelab](02-selfhosting/index.md) * [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) * [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md) @@ -18,6 +20,7 @@ * [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) * [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md) * [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) + * [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) * [Open Source & Alternatives](03-opensource/index.md) * [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) * [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md)