Initial: backup/restore scripts, README

- backup.sh: weekly cron script for PVE1+PVE2
- restore.sh: interactive disaster recovery wizard
- README.md: step-by-step recovery guide for both nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,9 @@
+# Never store private keys or secrets
+*.key
+priv/
+id_rsa
+*.pem
+*.pfx
+
+# Logs
+*.log

README.md

@@ -0,0 +1,176 @@
+# Proxmox Config Backup & Restore
+
+Automated weekly backup of PVE1 and PVE2 configuration to GitHub.  
+In a host failure, this gets you back up in 15–30 minutes.
+
+---
+
+## Repository Structure
+
+```
+proxmox-config/
+  backup.sh          ← runs on each node weekly, commits changes
+  restore.sh         ← interactive restore on a fresh Proxmox install
+  README.md          ← this file
+  shared/            ← cluster-wide configs (backed up from PVE1)
+    storage.cfg        — storage definitions (local-lvm, GoFlex, ProxBackups)
+    datacenter.cfg     — cluster settings
+    user.cfg           — users and API tokens
+    corosync.conf      — cluster membership
+    jobs.cfg           — backup job schedules
+    vzdump.cron        — vzdump cron config
+    firewall/          — cluster firewall rules
+    ha/                — HA group/resource configs
+    nodes/
+      pve/
+        qemu-server/   — PVE1 VM .conf files (100–120)
+        lxc/           — PVE1 LXC container configs (110)
+      pve2/
+        qemu-server/   — PVE2 VM .conf files (106, 302)
+  pve/               ← PVE1-specific (hostname: pve, 10.48.200.90)
+    network/           — /etc/network/interfaces
+    cron/              — root crontab
+    scripts/           — custom shell/python scripts
+    systemd/           — custom service units
+    jarvis-agent/      — JARVIS monitoring agent config
+    ssh/               — authorized_keys (no private keys)
+  pve2/              ← PVE2-specific (hostname: pve2, 10.48.200.91)
+    (same structure)
+```
+
+---
+
+## Backup Schedule
+
+Both nodes run `backup.sh` via cron every **Sunday at 3:00 AM**.
+
+```
+0 3 * * 0 /usr/local/bin/proxmox-backup >> /var/log/proxmox-backup.log 2>&1
+```
+
+The script:
+- Pulls latest from GitHub (handles both nodes committing)
+- Collects all config files listed above
+- Skips large binaries (ollama, filebrowser — see reinstall notes below)
+- Skips private keys (`/etc/pve/priv/`, SSH private keys)
+- Commits with a timestamp and pushes
+
+---
+
+## Disaster Recovery — Host Failure
+
+### Scenario A: PVE1 (primary) fails
+
+PVE1 hosts most VMs (100–113, 118, 120). Storage for those VMs is on `local-lvm` (PVE1's LVM thin pool) or `GoFlex` NAS. VMs on local-lvm **are at risk** if the disk fails — restore from PBS.
+
+**Steps:**
+
+1. **Install fresh Proxmox** on replacement hardware.
+   - Set IP to `10.48.200.90` during install, or set after.
+   - Set hostname to `pve`.
+
+2. **Clone this repo:**
+   ```bash
+   apt install -y git
+   git clone https://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@github.com/myronblair/proxmox-config.git /opt/proxmox-config
+   ```
+
+3. **Run the restore script:**
+   ```bash
+   bash /opt/proxmox-config/restore.sh pve1
+   ```
+   The script is interactive — confirm each section as it goes.
+
+4. **Reboot** to apply network config:
+   ```bash
+   reboot
+   ```
+
+5. **Restore VMs from PBS** (Proxmox Backup Server at `10.48.200.85`):
+   - Log in to PVE web UI: `https://10.48.200.90:8006`
+   - Datacenter → Storage → Add → Proxmox Backup Server
+     - Server: `10.48.200.85`
+     - Datastore: `PBSBackup`
+     - Username: `root@pam`
+     - Fingerprint: see `shared/pbs_fingerprint.txt`
+   - For each VM: select the backup → Restore
+
+6. **Reinstall large binaries** (not in git — too large):
+   ```bash
+   # Ollama (local LLM on VM 210 inside this host)
+   curl -fsSL https://ollama.ai/install.sh | sh
+   ollama pull llama3.2
+
+   # File Browser
+   curl -fsSL https://raw.githubusercontent.com/filebrowser/get/master/get.sh | bash
+   # Config is at /root/.filebrowser.json or wherever it was set
+   ```
+
+7. **Start services:**
+   ```bash
+   systemctl start jarvis-agent filebrowser
+   ```
+
+---
+
+### Scenario B: PVE2 (secondary) fails
+
+PVE2 hosts VM 106 (unknown) and VM 302 (NetworkBackup). PVE2 uses `local-lvm` for storage.
+
+**Steps:**
+
+1. **Install fresh Proxmox**, set IP `10.48.200.91`, hostname `pve2`.
+
+2. **Clone repo and run restore:**
+   ```bash
+   apt install -y git
+   git clone https://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@github.com/myronblair/proxmox-config.git /opt/proxmox-config
+   bash /opt/proxmox-config/restore.sh pve2
+   ```
+
+3. **Join the cluster from PVE1:**
+   ```bash
+   # On PVE1:
+   pvecm add 10.48.200.91
+   # This auto-syncs all cluster state to PVE2
+   ```
+
+4. **Restore VMs 106 and 302 from PBS.**
+
+5. **Reboot PVE2**, then verify cluster: `pvecm status`
+
+---
+
+## What Is NOT Backed Up Here
+
+| Item | Where to find it |
+|------|-----------------|
+| VM disk images | Proxmox Backup Server (10.48.200.85) — runs nightly |
+| /etc/pve/priv/ | Private CA key, auth keys — DO NOT store in git; regenerated by Proxmox on fresh install |
+| SSH private keys | /root/.ssh/id_rsa — regenerate or restore from secure storage |
+| ollama binary | Reinstall via install script (see above) |
+| filebrowser binary | Reinstall via get.sh (see above) |
+| blockalign, urbackupclientctl | Reinstall from their respective sources if needed |
+
+---
+
+## Nodes Quick Reference
+
+| Node | Hostname | IP | Port | Role |
+|------|----------|----|------|------|
+| PVE1 | pve | 10.48.200.90 | 8006 | Primary — most VMs |
+| PVE2 | pve2 | 10.48.200.91 | 8006 | Secondary — VM 106, 302 |
+| PBS  | — | 10.48.200.85 | 8007 | Backup server |
+
+Proxmox web login: `root` / `Joker1974!!!`  
+FortiGate DDNS (PVE1 from internet): `orbisne.fortiddns.com`
+
+---
+
+## Manual Backup Trigger
+
+To run a backup immediately on either node:
+```bash
+/usr/local/bin/proxmox-backup
+tail -f /var/log/proxmox-backup.log
+```

backup.sh

@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+# =============================================================================
+# Proxmox Config Backup — runs on PVE1 or PVE2
+# Backs up all critical configs to GitHub weekly
+# Install: /usr/local/bin/proxmox-backup
+# Cron:    0 3 * * 0 /usr/local/bin/proxmox-backup >> /var/log/proxmox-backup.log 2>&1
+# =============================================================================
+set -euo pipefail
+
+PAT="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"
+REPO_URL="https://${PAT}@github.com/myronblair/proxmox-config.git"
+REPO_DIR="/opt/proxmox-config"
+NODE=$(hostname)
+LOG_PREFIX="[$(date '+%Y-%m-%d %H:%M:%S')] [$NODE]"
+
+log()  { echo "$LOG_PREFIX $*"; }
+die()  { echo "$LOG_PREFIX ERROR: $*" >&2; exit 1; }
+
+# ---------------------------------------------------------------------------
+# 1. Clone or update local repo
+# ---------------------------------------------------------------------------
+if [[ -d "$REPO_DIR/.git" ]]; then
+    log "Pulling latest from GitHub"
+    cd "$REPO_DIR"
+    git config user.email "proxmox-backup@orbishosting.com"
+    git config user.name  "Proxmox Backup"
+    git pull --rebase origin main -q || { log "Pull failed — continuing with local state"; true; }
+else
+    log "Cloning repo to $REPO_DIR"
+    git clone "$REPO_URL" "$REPO_DIR"
+    cd "$REPO_DIR"
+    git config user.email "proxmox-backup@orbishosting.com"
+    git config user.name  "Proxmox Backup"
+fi
+
+cd "$REPO_DIR"
+
+# ---------------------------------------------------------------------------
+# 2. Node-specific directories
+# ---------------------------------------------------------------------------
+NODE_DIR="$REPO_DIR/$NODE"
+mkdir -p \
+    "$NODE_DIR/network" \
+    "$NODE_DIR/cron" \
+    "$NODE_DIR/scripts" \
+    "$NODE_DIR/systemd" \
+    "$NODE_DIR/jarvis-agent"
+
+# ---------------------------------------------------------------------------
+# 3. Shared PVE cluster configs (only back up from primary node to avoid
+#    conflicts — PVE1/pve is the primary; pve2 skips this section)
+# ---------------------------------------------------------------------------
+if [[ "$NODE" == "pve" ]]; then
+    log "Backing up shared PVE cluster configs"
+    SHARED_DIR="$REPO_DIR/shared"
+    mkdir -p \
+        "$SHARED_DIR/firewall" \
+        "$SHARED_DIR/ha" \
+        "$SHARED_DIR/sdn" \
+        "$SHARED_DIR/nodes/pve/qemu-server" \
+        "$SHARED_DIR/nodes/pve/lxc" \
+        "$SHARED_DIR/nodes/pve2/qemu-server" \
+        "$SHARED_DIR/nodes/pve2/lxc"
+
+    # Top-level cluster configs
+    for f in storage.cfg datacenter.cfg user.cfg corosync.conf vzdump.cron jobs.cfg ceph.conf replication.cfg; do
+        [[ -f "/etc/pve/$f" ]] && cp "/etc/pve/$f" "$SHARED_DIR/$f" || true
+    done
+
+    # Firewall
+    [[ -d /etc/pve/firewall ]] && rsync -a --delete /etc/pve/firewall/ "$SHARED_DIR/firewall/" 2>/dev/null || true
+
+    # HA
+    [[ -d /etc/pve/ha ]] && rsync -a --delete /etc/pve/ha/ "$SHARED_DIR/ha/" 2>/dev/null || true
+
+    # SDN
+    [[ -d /etc/pve/sdn ]] && rsync -a --delete /etc/pve/sdn/ "$SHARED_DIR/sdn/" 2>/dev/null || true
+
+    # VM/LXC configs (per-node)
+    for PVENODE in pve pve2; do
+        [[ -d "/etc/pve/nodes/$PVENODE/qemu-server" ]] && \
+            rsync -a --delete /etc/pve/nodes/$PVENODE/qemu-server/ "$SHARED_DIR/nodes/$PVENODE/qemu-server/" 2>/dev/null || true
+        [[ -d "/etc/pve/nodes/$PVENODE/lxc" ]] && \
+            rsync -a --delete /etc/pve/nodes/$PVENODE/lxc/ "$SHARED_DIR/nodes/$PVENODE/lxc/" 2>/dev/null || true
+    done
+
+    # PBS fingerprint (needed for restore)
+    grep -A5 "pbs: ProxBackups" /etc/pve/storage.cfg > "$SHARED_DIR/pbs_fingerprint.txt" 2>/dev/null || true
+fi
+
+# ---------------------------------------------------------------------------
+# 4. Network config
+# ---------------------------------------------------------------------------
+log "Backing up network config"
+cp /etc/network/interfaces "$NODE_DIR/network/interfaces"
+[[ -d /etc/network/interfaces.d ]] && \
+    rsync -a --delete /etc/network/interfaces.d/ "$NODE_DIR/network/interfaces.d/" 2>/dev/null || true
+cp /etc/hosts    "$NODE_DIR/network/hosts"
+cp /etc/hostname "$NODE_DIR/network/hostname"
+[[ -f /etc/resolv.conf ]] && cp /etc/resolv.conf "$NODE_DIR/network/resolv.conf" || true
+
+# ---------------------------------------------------------------------------
+# 5. Root crontab
+# ---------------------------------------------------------------------------
+log "Backing up crontab"
+crontab -l 2>/dev/null > "$NODE_DIR/cron/root" || echo "# no crontab" > "$NODE_DIR/cron/root"
+[[ -f /etc/vzdump.conf ]] && cp /etc/vzdump.conf "$NODE_DIR/vzdump.conf" || true
+
+# ---------------------------------------------------------------------------
+# 6. Custom scripts (text only — skip large binaries)
+# ---------------------------------------------------------------------------
+log "Backing up custom scripts"
+for f in /usr/local/bin/*.sh /usr/local/bin/*.py /usr/local/bin/*.pl; do
+    [[ -f "$f" ]] || continue
+    size=$(stat -c%s "$f" 2>/dev/null || echo 0)
+    if [[ $size -lt 524288 ]]; then  # skip files > 512KB (binaries)
+        cp "$f" "$NODE_DIR/scripts/"
+    fi
+done
+
+# ---------------------------------------------------------------------------
+# 7. Custom systemd units (only non-stock units)
+# ---------------------------------------------------------------------------
+log "Backing up custom systemd units"
+STOCK_UNITS="chronyd.service iscsi.service pve-manager.service smartd.service sshd.service zed.service"
+for f in /etc/systemd/system/*.service; do
+    [[ -f "$f" ]] || continue
+    bname=$(basename "$f")
+    if ! echo "$STOCK_UNITS" | grep -qw "$bname"; then
+        cp "$f" "$NODE_DIR/systemd/"
+    fi
+done
+
+# ---------------------------------------------------------------------------
+# 8. JARVIS agent config
+# ---------------------------------------------------------------------------
+log "Backing up JARVIS agent config"
+[[ -f /opt/jarvis-agent/config.json ]] && \
+    cp /opt/jarvis-agent/config.json "$NODE_DIR/jarvis-agent/config.json" || true
+
+# ---------------------------------------------------------------------------
+# 9. SSH authorized_keys (for access restoration — no private keys)
+# ---------------------------------------------------------------------------
+mkdir -p "$NODE_DIR/ssh"
+[[ -f /root/.ssh/authorized_keys ]] && cp /root/.ssh/authorized_keys "$NODE_DIR/ssh/authorized_keys" || true
+[[ -f /root/.ssh/id_rsa.pub ]] && cp /root/.ssh/id_rsa.pub "$NODE_DIR/ssh/id_rsa.pub" || true
+
+# ---------------------------------------------------------------------------
+# 10. Commit and push
+# ---------------------------------------------------------------------------
+log "Committing changes"
+git add -A
+if git diff --cached --quiet; then
+    log "No changes to commit"
+else
+    CHANGES=$(git diff --cached --stat | tail -1)
+    git commit -m "[$NODE] Weekly backup $(date '+%Y-%m-%d') — $CHANGES"
+    log "Pushing to GitHub"
+    git push origin main
+    log "Backup complete"
+fi

restore.sh

@@ -0,0 +1,303 @@
+#!/usr/bin/env bash
+# =============================================================================
+# Proxmox Config Restore — run on a freshly installed Proxmox node
+#
+# Usage:
+#   bash restore.sh pve1    — restore PVE1 (primary, hostname "pve")
+#   bash restore.sh pve2    — restore PVE2 (secondary, hostname "pve2")
+#
+# Prerequisites on fresh Proxmox install:
+#   apt install -y git
+#   git clone https://<PAT>@github.com/myronblair/proxmox-config.git /opt/proxmox-config
+#   bash /opt/proxmox-config/restore.sh pve1
+# =============================================================================
+
+TARGET="${1:-}"
+REPO_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+RED='\033[0;31m'; YELLOW='\033[1;33m'; GREEN='\033[0;32m'; CYAN='\033[0;36m'; NC='\033[0m'
+
+header()  { echo -e "\n${CYAN}══════════════════════════════════════${NC}"; echo -e "${CYAN}  $*${NC}"; echo -e "${CYAN}══════════════════════════════════════${NC}"; }
+success() { echo -e "${GREEN}  ✓ $*${NC}"; }
+warn()    { echo -e "${YELLOW}  ⚠ $*${NC}"; }
+info()    { echo -e "  → $*"; }
+die()     { echo -e "${RED}  ✗ $*${NC}" >&2; exit 1; }
+
+confirm() {
+    local msg="$1"
+    echo -e "\n${YELLOW}  $msg${NC}"
+    read -rp "  Apply this section? [Y/n] " ans
+    [[ "${ans:-Y}" =~ ^[Yy]$ ]]
+}
+
+# ---------------------------------------------------------------------------
+# Validate arguments
+# ---------------------------------------------------------------------------
+if [[ -z "$TARGET" ]]; then
+    echo "Usage: $0 pve1|pve2"
+    echo ""
+    echo "  pve1 — restore primary node (hostname: pve, IP: 10.48.200.90)"
+    echo "  pve2 — restore secondary node (hostname: pve2, IP: 10.48.200.91)"
+    exit 1
+fi
+
+if [[ "$TARGET" == "pve1" ]]; then
+    NODE_DIR="$REPO_DIR/pve"
+    EXPECTED_HOSTNAME="pve"
+    EXPECTED_IP="10.48.200.90"
+elif [[ "$TARGET" == "pve2" ]]; then
+    NODE_DIR="$REPO_DIR/pve2"
+    EXPECTED_HOSTNAME="pve2"
+    EXPECTED_IP="10.48.200.91"
+else
+    die "Unknown target: $TARGET. Use pve1 or pve2."
+fi
+
+SHARED_DIR="$REPO_DIR/shared"
+
+[[ -d "$NODE_DIR" ]] || die "Node directory not found: $NODE_DIR"
+[[ $(id -u) -eq 0 ]] || die "Must run as root"
+
+# ---------------------------------------------------------------------------
+# Welcome banner
+# ---------------------------------------------------------------------------
+clear
+echo -e "${CYAN}"
+cat << 'BANNER'
+  ╔═══════════════════════════════════════════════════════╗
+  ║         PROXMOX CONFIG RESTORE — OrbiShosting         ║
+  ╚═══════════════════════════════════════════════════════╝
+BANNER
+echo -e "${NC}"
+echo "  Target : $TARGET ($EXPECTED_HOSTNAME — $EXPECTED_IP)"
+echo "  Source : $REPO_DIR"
+echo "  Date   : $(date)"
+echo ""
+warn "This script restores configs to a FRESH Proxmox install."
+warn "Running on an existing node may overwrite running config."
+echo ""
+read -rp "  Type 'yes' to continue: " confirm_start
+[[ "$confirm_start" == "yes" ]] || { echo "Aborted."; exit 0; }
+
+APPLIED=()
+SKIPPED=()
+
+# ---------------------------------------------------------------------------
+# SECTION 1: Hostname
+# ---------------------------------------------------------------------------
+header "1 / 8 — Hostname"
+info "Will set hostname to: $EXPECTED_HOSTNAME"
+[[ -f "$NODE_DIR/network/hostname" ]] && info "Backed-up hostname: $(cat $NODE_DIR/network/hostname)"
+if confirm "Set hostname to $EXPECTED_HOSTNAME?"; then
+    echo "$EXPECTED_HOSTNAME" > /etc/hostname
+    hostname "$EXPECTED_HOSTNAME"
+    success "Hostname set to $EXPECTED_HOSTNAME"
+    APPLIED+=("hostname")
+else
+    SKIPPED+=("hostname")
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 2: Network interfaces
+# ---------------------------------------------------------------------------
+header "2 / 8 — Network Configuration"
+if [[ -f "$NODE_DIR/network/interfaces" ]]; then
+    info "Current /etc/network/interfaces:"
+    cat /etc/network/interfaces | sed 's/^/    /'
+    info ""
+    info "Backed-up interfaces:"
+    cat "$NODE_DIR/network/interfaces" | sed 's/^/    /'
+    warn "Applying network config requires a reboot to take effect."
+    if confirm "Restore network interfaces?"; then
+        cp /etc/network/interfaces "/etc/network/interfaces.bak.$(date +%s)"
+        cp "$NODE_DIR/network/interfaces" /etc/network/interfaces
+        [[ -d "$NODE_DIR/network/interfaces.d" ]] && \
+            rsync -a "$NODE_DIR/network/interfaces.d/" /etc/network/interfaces.d/
+        success "Network interfaces restored (reboot required)"
+        APPLIED+=("network")
+    else
+        SKIPPED+=("network")
+    fi
+else
+    warn "No network interfaces backup found — skipping"
+    SKIPPED+=("network")
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 3: /etc/hosts
+# ---------------------------------------------------------------------------
+header "3 / 8 — /etc/hosts"
+if [[ -f "$NODE_DIR/network/hosts" ]]; then
+    if confirm "Restore /etc/hosts?"; then
+        cp /etc/hosts "/etc/hosts.bak.$(date +%s)"
+        cp "$NODE_DIR/network/hosts" /etc/hosts
+        success "/etc/hosts restored"
+        APPLIED+=("hosts")
+    else
+        SKIPPED+=("hosts")
+    fi
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 4: Custom scripts
+# ---------------------------------------------------------------------------
+header "4 / 8 — Custom Scripts (/usr/local/bin)"
+if [[ -d "$NODE_DIR/scripts" ]] && [[ -n "$(ls -A $NODE_DIR/scripts 2>/dev/null)" ]]; then
+    info "Scripts to restore:"
+    ls "$NODE_DIR/scripts" | sed 's/^/    /'
+    if confirm "Restore custom scripts to /usr/local/bin/?"; then
+        cp "$NODE_DIR/scripts/"* /usr/local/bin/ 2>/dev/null || true
+        chmod +x /usr/local/bin/*.sh /usr/local/bin/*.py 2>/dev/null || true
+        success "Scripts restored"
+        APPLIED+=("scripts")
+        if [[ "$TARGET" == "pve1" ]]; then
+            warn "Large binaries NOT restored (ollama, filebrowser, etc.)"
+            warn "Reinstall manually:"
+            warn "  ollama     : curl -fsSL https://ollama.ai/install.sh | sh"
+            warn "  filebrowser: curl -fsSL https://raw.githubusercontent.com/filebrowser/get/master/get.sh | bash"
+        fi
+    else
+        SKIPPED+=("scripts")
+    fi
+else
+    info "No custom scripts backed up"
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 5: Systemd units
+# ---------------------------------------------------------------------------
+header "5 / 8 — Systemd Service Units"
+if [[ -d "$NODE_DIR/systemd" ]] && [[ -n "$(ls -A $NODE_DIR/systemd 2>/dev/null)" ]]; then
+    info "Units to restore:"
+    ls "$NODE_DIR/systemd" | sed 's/^/    /'
+    if confirm "Restore and enable systemd units?"; then
+        for unit in "$NODE_DIR/systemd/"*.service; do
+            [[ -f "$unit" ]] || continue
+            bname=$(basename "$unit")
+            cp "$unit" /etc/systemd/system/
+            systemctl enable "$bname" 2>/dev/null || true
+            info "  Enabled: $bname"
+        done
+        systemctl daemon-reload
+        success "Systemd units restored and enabled"
+        APPLIED+=("systemd")
+        warn "Units are ENABLED but not started — start manually after verifying prerequisites"
+        warn "  e.g.: systemctl start jarvis-agent filebrowser"
+    else
+        SKIPPED+=("systemd")
+    fi
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 6: JARVIS agent
+# ---------------------------------------------------------------------------
+header "6 / 8 — JARVIS Monitoring Agent"
+if [[ -f "$NODE_DIR/jarvis-agent/config.json" ]]; then
+    info "Agent config found:"
+    cat "$NODE_DIR/jarvis-agent/config.json" | sed 's/^/    /'
+    if confirm "Restore JARVIS agent config?"; then
+        mkdir -p /opt/jarvis-agent
+        cp "$NODE_DIR/jarvis-agent/config.json" /opt/jarvis-agent/config.json
+
+        # Install agent if not present
+        if [[ ! -f /opt/jarvis-agent/jarvis-agent.py ]]; then
+            info "Downloading JARVIS agent..."
+            curl -sk https://jarvis.orbishosting.com/install-agent.sh | bash -s "$(hostname)" proxmox || \
+                warn "Auto-install failed — manually install: curl -sk https://jarvis.orbishosting.com/install-agent.sh | bash -s $(hostname) proxmox"
+        fi
+        success "JARVIS agent restored"
+        APPLIED+=("jarvis-agent")
+    else
+        SKIPPED+=("jarvis-agent")
+    fi
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 7: Root crontab
+# ---------------------------------------------------------------------------
+header "7 / 8 — Root Crontab"
+if [[ -f "$NODE_DIR/cron/root" ]] && ! grep -q "^# no crontab" "$NODE_DIR/cron/root"; then
+    info "Cron entries to restore:"
+    cat "$NODE_DIR/cron/root" | sed 's/^/    /'
+    if confirm "Restore root crontab?"; then
+        crontab "$NODE_DIR/cron/root"
+        success "Crontab restored"
+        APPLIED+=("crontab")
+    else
+        SKIPPED+=("crontab")
+    fi
+else
+    info "No crontab entries backed up — skipping"
+fi
+
+# ---------------------------------------------------------------------------
+# SECTION 8: PVE cluster configs (pve1 only — pve2 joins cluster instead)
+# ---------------------------------------------------------------------------
+header "8 / 8 — PVE Cluster / VM Configs"
+if [[ "$TARGET" == "pve1" ]] && [[ -d "$SHARED_DIR" ]]; then
+    echo ""
+    warn "PVE cluster configs (storage, VM configs, users, firewall) are"
+    warn "managed by Proxmox's cluster filesystem (/etc/pve/)."
+    warn ""
+    warn "IMPORTANT: Do NOT copy /etc/pve/ files directly on a fresh install."
+    warn "Proxmox will auto-populate these when the node is online."
+    warn ""
+    info "Instead, after Proxmox is installed and running:"
+    info "  1. Restore VMs from Proxmox Backup Server (PBS) at 10.48.200.85"
+    info "     → In PVE web UI: Datacenter → Storage → ProxBackups"
+    info "     → PBS fingerprint: $(grep 'fingerprint' $SHARED_DIR/storage.cfg 2>/dev/null | head -1 | tr -d ' \t' || echo 'see shared/storage.cfg')"
+    info ""
+    info "  2. Re-add storage manually if PBS isn't auto-discovered:"
+    info "     → Datacenter → Storage → Add → Proxmox Backup Server"
+    info ""
+    info "  3. VM configs for reference (not to copy directly):"
+    [[ -d "$SHARED_DIR/nodes/pve/qemu-server" ]] && ls "$SHARED_DIR/nodes/pve/qemu-server/" | sed 's/^/     /'
+    echo ""
+    if confirm "Copy VM .conf files to /etc/pve/nodes/pve/qemu-server/ (only do this if cluster is NOT active)?"; then
+        mkdir -p /etc/pve/nodes/pve/qemu-server /etc/pve/nodes/pve/lxc
+        cp "$SHARED_DIR/nodes/pve/qemu-server/"*.conf /etc/pve/nodes/pve/qemu-server/ 2>/dev/null || true
+        cp "$SHARED_DIR/nodes/pve/lxc/"*.conf /etc/pve/nodes/pve/lxc/ 2>/dev/null || true
+        cp "$SHARED_DIR/storage.cfg" /etc/pve/ 2>/dev/null || true
+        cp "$SHARED_DIR/datacenter.cfg" /etc/pve/ 2>/dev/null || true
+        success "PVE configs copied — restart pve-cluster to apply"
+        systemctl restart pve-cluster 2>/dev/null || true
+        APPLIED+=("pve-configs")
+    else
+        SKIPPED+=("pve-configs")
+        info "Skipped — restore VMs from PBS backup instead (recommended)"
+    fi
+
+elif [[ "$TARGET" == "pve2" ]]; then
+    warn "PVE2 does not hold the primary cluster config."
+    info "To restore PVE2 as a cluster member:"
+    info "  1. Fresh Proxmox install with hostname 'pve2' and IP 10.48.200.91"
+    info "  2. On PVE1: pvecm add 10.48.200.91"
+    info "  3. PVE2 will auto-sync all VM configs and cluster state from PVE1"
+    info "  4. VM 106 and 302 will reappear automatically"
+    SKIPPED+=("pve-configs")
+fi
+
+# ---------------------------------------------------------------------------
+# Summary
+# ---------------------------------------------------------------------------
+header "Restore Complete"
+echo ""
+if [[ ${#APPLIED[@]} -gt 0 ]]; then
+    success "Applied sections: ${APPLIED[*]}"
+fi
+if [[ ${#SKIPPED[@]} -gt 0 ]]; then
+    warn "Skipped sections: ${SKIPPED[*]}"
+fi
+echo ""
+echo -e "${YELLOW}  Next steps:${NC}"
+if [[ " ${APPLIED[*]} " =~ " network " ]]; then
+    echo "  • REBOOT to apply network changes: reboot"
+fi
+echo "  • Verify Proxmox web UI at https://$EXPECTED_IP:8006"
+echo "  • Restore VMs from PBS backup server at https://10.48.200.85:8007"
+echo "  • Start JARVIS agent: systemctl start jarvis-agent"
+if [[ "$TARGET" == "pve1" ]]; then
+    echo "  • Reinstall ollama if needed: curl -fsSL https://ollama.ai/install.sh | sh"
+    echo "  • Reinstall filebrowser: curl -fsSL https://raw.githubusercontent.com/filebrowser/get/master/get.sh | bash"
+fi
+echo ""