1 VPS BACKUP SYSTEM: Pangolin Docker#

Target VPS: 72.11.147.220
VPS Provider: RackNerd (RN-VPS)
Backup LXC: Proxmox 102
Retention: 28 Files (14 Days)

The backup is a pull operation: Proxmox LXC 102 initiates SSH to the VPS. The home ISP public IP is therefore used as the source IP seen by the VPS firewall. On 2026-08-12, the home IP was 208.96.219.86; this address is dynamic and must be updated if the ISP changes it.

1.1 VPS Network Security and ISP IP Changes#

The VPS currently uses these relevant controls:

• UFW: default deny (incoming); SSH (22/tcp) is allowed only from the current home public IP.
• Fail2ban: the sshd jail uses ignoreip for trusted home addresses and can add REJECT rules through UFW when an address is banned.
• CrowdSec: the Pangolin container may enforce decisions through the host firewall’s INPUT and DOCKER-USER chains.

When the home ISP IP changes, update the VPS before troubleshooting the backup:

1. From a device on the home network, find the new address:

   curl -4 ifconfig.me

2. On the VPS, allow the new IP and remove the old UFW SSH rule:

   sudo ufw allow from NEW_HOME_IP to any port 22 proto tcp
   sudo ufw delete allow from OLD_HOME_IP to any port 22 proto tcp

3. Update Fail2ban immediately and persistently:

   sudo fail2ban-client set sshd addignoreip NEW_HOME_IP
   sudo fail2ban-client set sshd delignoreip OLD_HOME_IP

   Replace OLD_HOME_IP with NEW_HOME_IP in /etc/fail2ban/jail.local, then reload:

   sudo fail2ban-client reload

4. Verify before rerunning the backup:

   sudo ufw status verbose
   sudo fail2ban-client get sshd ignoreip
   docker exec crowdsec-pangolin cscli decisions list

curl -4 ifconfig.me run on the VPS returns the VPS IP, not the home ISP IP. Run it from the home network.

1.2 VPS Security Audit Script#

The nightly VPS security email is generated by this script on the RackNerd VPS:

/home/marc/docker-compose/pangolin/scripts/security-audit.sh

It is scheduled in the VPS root crontab:

0 0 * * * /home/marc/docker-compose/pangolin/scripts/security-audit.sh

1.2.1 Audit Functions#

The script performs these checks:

1. Confirms that UFW is installed and active, then includes the current UFW rules in the report.
2. Checks for LD_PRELOAD and /etc/ld.so.preload library-hijacking indicators.
3. Checks for known malware files, including kdevtmpfsi, kinsing, and libgcwrap.so.
4. Checks running processes for known miners or malware processes such as xmrig, monero, and kinsing.
5. Checks the most recent root login IP using last -i root.
6. Sends a critical email if any check fails; otherwise sends a clean daily-status email.

1.2.2 Trusted Home IP#

The active script currently contains:

MY_HOME_IP="208.96.219.86"

This value must be updated after an ISP IP change. The script is not DDNS-aware and does not automatically follow marcoue.ddns.net.

When the ISP address changes, update the UFW and Fail2ban configuration first, then update this script’s MY_HOME_IP value. Validate the script without sending an email:

bash -n /home/marc/docker-compose/pangolin/scripts/security-audit.sh

On 2026-08-13, the alert UNKNOWN LAST LOGIN IP: 208.96.219.86 was a false positive caused by the script still containing the previous IP 72.11.191.80. The active script was updated to the current IP and backed up as security-audit.sh.before-2026-08-13.

1.2.3 Script Copies#

The scheduled script is the authoritative VPS copy. An unscheduled duplicate also exists:

/home/marc/docker-compose/pangolin/scripts/security-audit copy.sh

The duplicate is not referenced by cron. If it is retained, keep its MY_HOME_IP value synchronized with the active script to prevent future confusion.

2 VPS Configuration#

Run as root on VPS

2.1 User & Group Setup#

adduser --system --group --home /home/vps-backup --shell /bin/bash vps-backup
usermod -aG marc vps-backup
usermod -aG docker vps-backup
newgrp docker

2.2 Directory Permissions (ACLs)#

2.2.1 Basic Ownership#

chown root:vps-backup /home/marc/docker-compose/pangolin
chmod 750 /home/marc/docker-compose/pangolin

2.2.2 Recursive + Default ACLs#

setfacl -R -m "u:vps-backup:rX" /home/marc/docker-compose/pangolin
setfacl -Rd -m "u:vps-backup:rX" /home/marc/docker-compose/pangolin

2.2.3 Ensure Existing Files Are Readable#

find /home/marc/docker-compose/pangolin -type f -exec chmod 644 {} +

2.2.4 Lock Down acme.json#

setfacl -x "u:vps-backup" /home/marc/docker-compose/pangolin/config/letsencrypt/acme.json
chmod 600 /home/marc/docker-compose/pangolin/config/letsencrypt/acme.json

acme.json is intentionally excluded from rsync for safety.

2.2.5 Verify Log Directory Access#

setfacl -R -m "u:vps-backup:rX" /home/marc/docker-compose/pangolin/config/traefik/logs

2.3 SSH Security#

mkdir -p /home/vps-backup/.ssh
chmod 700 /home/vps-backup/.ssh
nano /home/vps-backup/.ssh/authorized_keys
chmod 600 /home/vps-backup/.ssh/authorized_keys
chown -R vps-backup:vps-backup /home/vps-backup/.ssh

2.4 LXC Configuration#

Run on Proxmox 102

2.4.1 Key Generation#

sudo -u vps-backup ssh-keygen -t ed25519 -f /home/vps-backup/.ssh/id_ed25519 -N ""
cat /home/vps-backup/.ssh/id_ed25519.pub

Copy the public key to the VPS.

2.4.2 Backup Storage Permissions#

chown -R vps-backup:vps-backup /mnt/VPSBackups
chmod 755 /mnt/VPSBackups

2.4.3 Backup Script Setup#

touch /home/vps-backup/RN-VPSPangoling_pull_backups.sh
chmod +x /home/vps-backup/RN-VPSPangoling_pull_backups.sh
sudo -u vps-backup nano /home/vps-backup/RN-VPSPangoling_pull_backups.sh

2.4.4 Automation (Crontab)#

sudo -u vps-backup crontab -e

0 12,20 * * * /bin/bash /home/vps-backup/RN-VPSPangoling_pull_backups.sh

2.5 Integrity Monitoring (Weekly Health Check)#

2.5.1 Script Setup#

touch /home/vps-backup/RN-VPSPangoling_pull_backups_integrity_check.sh
chown vps-backup:vps-backup /home/vps-backup/RN-VPSPangoling_pull_backups_integrity_check.sh
chmod +x /home/vps-backup/RN-VPSPangoling_pull_backups_integrity_check.sh

2.5.2 Manual Run & Logs#

sudo -u vps-backup /bin/bash /home/vps-backup/RN-VPSPangoling_pull_backups_integrity_check.sh
cat /home/vps-backup/VPSBackup-Integrity.log

2.5.3 Automation#

Add to crontab:

0 1 * * 0 /bin/bash /home/vps-backup/RN-VPSPangoling_pull_backups_integrity_check.sh

2.6 Verification Commands#

2.6.1 Check Disk Space (10GB Limit)#

df -h /mnt/VPSBackups

2.6.2 View Latest Heartbeat#

tail -n 10 /home/vps-backup/VPSBackup-Pangolin.log

2.6.3 Peek Inside Archive#

tar -tvf /mnt/VPSBackups/FILENAME.tar.gz | head -n 20

2.7 Email Installation & Configuration#

2.7.1 Install Mail Packages#

apt update && apt install msmtp msmtp-mta mailutils -y

2.7.2 Create Configuration File#

nano /etc/msmtprc

Paste:

defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp.log

account        gmail
host           smtp.gmail.com
port           587
from           proxmox.app@gmail.com
user           proxmox.app@gmail.com
password       YOUR_16_DIGIT_PASSWORD
tls_starttls   on

account        default : gmail

2.7.3 Secure Permissions#

chmod 600 /etc/msmtprc
chown vps-backup:vps-backup /etc/msmtprc

2.7.4 Initialize Log File#

touch /var/log/msmtp.log
chown vps-backup:vps-backup /var/log/msmtp.log
chmod 664 /var/log/msmtp.log

2.7.5 Test Email#

echo "Hello Marc, this is a test." | mail -s "LXC Test Email" proxmox.app@gmail.com

2.8 Restoration Procedure (Emergency)#

2.8.1 Transfer to VPS#

scp /mnt/VPSBackups/2026-02-02_19h00_PangolinBackup.tar.gz root@72.11.147.220:/home/marc/docker-compose/

2.8.2 Unpack on VPS#

mkdir -p /home/marc/docker-compose/pangolin
tar -xzvf /home/marc/docker-compose/2026-02-02_19h00_PangolinBackup.tar.gz \
  -C /home/marc/docker-compose/pangolin --strip-components=1
chmod 600 /home/marc/docker-compose/pangolin/config/letsencrypt/acme.json

2.8.3 CrowdSec Repair (If Needed)#

docker exec crowdsec-pangolin cscli hub update
docker exec crowdsec-pangolin cscli hub upgrade --force
docker exec crowdsec-pangolin kill -HUP 1

2.8.4 Full Restore Script#

# Run restore script:
/home/marc/docker-compose/pangolin1/scripts/PangolinRestore.sh
# Target archive:
/home/marc/docker-compose/2026-03-06_19h00_PangolinBackup.tar.gz

If rebuilding a new VPS, install CrowdSec first:

apt install crowdsec

2.9 File Reference Map#

2.10 Manual Restore Examples#

scp /mnt/VPSBackups/2026-01-29_21h12_PangolinBackup.tar.gz \
root@72.11.147.220:/home/marc/docker-compose/

tar -xzf /home/marc/docker-compose/2026-01-29_10h21_PangolinBackup.tar.gz \
-C /home/marc/docker-compose/pangolin3