Files

MajorLinux fb2e3f6168 wiki: add SELinux AVC chart, enriched alerts, new server setup, and pending articles; update indexes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 03:34:33 -04:00

5.9 KiB

Raw Blame History

title, domain, category, tags, status, created, updated

title

domain

WSL2 Fedora 43 Training Environment Rebuild

How to rebuild the MajorTwin training environment from scratch on MajorRig after a WSL2 loss. Covers Fedora 43 install, Python 3.11 via pyenv, PyTorch with CUDA, Unsloth, and llama.cpp for GGUF conversion.

The Short Answer

# 1. Install Fedora 43 and move to D:
wsl --install -d FedoraLinux-43 --no-launch
wsl --export FedoraLinux-43 D:\WSL\fedora43.tar
wsl --unregister FedoraLinux-43
wsl --import FedoraLinux-43 D:\WSL\Fedora43 D:\WSL\fedora43.tar

# 2. Set default user
echo -e "[boot]\nsystemd=true\n[user]\ndefault=majorlinux" | sudo tee /etc/wsl.conf
useradd -m -G wheel majorlinux && passwd majorlinux
echo "%wheel ALL=(ALL) ALL" | sudo tee /etc/sudoers.d/wheel

# 3. Install Python 3.11 via pyenv, PyTorch, Unsloth
# See full steps below

Step 1 — System Packages

sudo dnf update -y
sudo dnf install -y git curl wget tmux screen htop rsync unzip \
    python3 python3-pip python3-devel gcc gcc-c++ make cmake \
    ninja-build pkg-config openssl-devel libffi-devel \
    gawk patch readline-devel sqlite-devel

Step 2 — Python 3.11 via pyenv

Fedora 43 ships Python 3.13. Unsloth requires 3.11. Use pyenv:

curl https://pyenv.run | bash

# Add to ~/.bashrc
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

source ~/.bashrc
pyenv install 3.11.9
pyenv global 3.11.9

The tkinter warning during install is harmless — it's not needed for training.

Step 3 — Training Virtualenv + PyTorch

mkdir -p ~/majortwin/{staging,datasets,outputs,scripts}
python -m venv ~/majortwin/venv
source ~/majortwin/venv/bin/activate

pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

# Verify GPU
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Expected output: True NVIDIA GeForce RTX 3080 Ti

Step 4 — Unsloth + Training Stack

source ~/majortwin/venv/bin/activate

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install transformers datasets accelerate peft trl bitsandbytes \
    sentencepiece protobuf scipy einops

# Pin transformers for unsloth-zoo compatibility
pip install "transformers<=5.2.0"

# Verify
python -c "import unsloth; print('Unsloth OK')"

[!warning] Never run pip install -r requirements.txt from inside llama.cpp while the training venv is active. It installs CPU-only PyTorch and downgrades transformers, breaking the CUDA setup.

Step 5 — llama.cpp (CPU-only for GGUF conversion)

CUDA 12.8 is incompatible with Fedora 43's glibc for compiling llama.cpp (math function conflicts in /usr/include/bits/mathcalls.h). Build CPU-only — it's sufficient for GGUF conversion, which doesn't need GPU:

# Install GCC 14 (CUDA 12.8 doesn't support GCC 15 which Fedora 43 ships)
sudo dnf install -y gcc14 gcc14-c++

cd ~/majortwin
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

cmake -B build \
    -DGGML_CUDA=OFF \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_C_COMPILER=/usr/bin/gcc-14 \
    -DCMAKE_CXX_COMPILER=/usr/bin/g++-14

cmake --build build --config Release -j$(nproc) 2>&1 | tee /tmp/llama_build.log &
tail -f /tmp/llama_build.log

Verify:

ls ~/majortwin/llama.cpp/build/bin/llama-quantize && echo "OK"
ls ~/majortwin/llama.cpp/build/bin/llama-cli && echo "OK"

Step 6 — Shell Environment

cat >> ~/.bashrc << 'EOF'
# MajorInfrastructure Paths
export VAULT="/mnt/c/Users/majli/Documents/MajorVault"
export MAJORANSIBLE="/mnt/d/MajorAnsible"
export MAJORTWIN_D="/mnt/d/MajorTwin"
export MAJORTWIN_WSL="$HOME/majortwin"
export LLAMA_CPP="$HOME/majortwin/llama.cpp"

# Venv
alias mtwin='source $MAJORTWIN_WSL/venv/bin/activate && cd $MAJORTWIN_WSL'
alias vault='cd $VAULT'
alias ll='ls -lah --color=auto'

# SSH Fleet Aliases
alias majorhome='ssh majorlinux@100.120.209.106'
alias dca='ssh root@100.104.11.146'
alias majortoot='ssh root@100.110.197.17'
alias majorlinuxvm='ssh root@100.87.200.5'
alias majordiscord='ssh root@100.122.240.83'
alias majorlab='ssh root@100.86.14.126'
alias majormail='ssh root@100.84.165.52'
alias teelia='ssh root@100.120.32.69'
alias tttpod='ssh root@100.84.42.102'
alias majorrig='ssh majorlinux@100.98.47.29'  # port 2222 retired 2026-03-25, fleet uses port 22

# DNF5
alias update='sudo dnf upgrade --refresh'
alias install='sudo dnf install'
alias clean='sudo dnf clean all'

# MajorTwin helpers
stage_dataset() {
    cp "$VAULT/20-Projects/MajorTwin/03-Datasets/$1" "$MAJORTWIN_WSL/datasets/"
    echo "Staged: $1"
}
export_gguf() {
    cp "$MAJORTWIN_WSL/outputs/$1" "$MAJORTWIN_D/models/"
    echo "Exported: $1 → $MAJORTWIN_D/models/"
}
EOF

source ~/.bashrc

Key Rules

Always activate venv before pip installs: source ~/majortwin/venv/bin/activate
Never train from /mnt/c or /mnt/d — stage files in ~/majortwin/staging/ first
Never put ML artifacts inside MajorVault — models, venvs, artifacts go on D: drive
Max viable training model: 7B at QLoRA 4-bit (RTX 3080 Ti, 12GB VRAM)
Current base model: Qwen2.5-7B-Instruct (ChatML format — stop token: <|im_end|> only)
Transformers must be pinned: pip install "transformers<=5.2.0" for unsloth-zoo compatibility

D: Drive Layout

D:\MajorTwin\
  models\          ← finished GGUFs
  datasets\        ← dataset archives
  artifacts\       ← training run artifacts
  training-runs\   ← logs, checkpoints
D:\WSL\
  Fedora43\        ← WSL2 VHDX
  Backups\         ← weekly WSL2 backup tars

5.9 KiB Raw Blame History