Initial commit: homelab configuration and documentation
This commit is contained in:
329
docs/guides/DEPLOYMENT_GUIDE.md
Normal file
329
docs/guides/DEPLOYMENT_GUIDE.md
Normal file
@@ -0,0 +1,329 @@
|
||||
# Home Lab Improvements - Deployment Guide
|
||||
|
||||
This guide provides step-by-step instructions for deploying all the homelab improvements.
|
||||
|
||||
## Table of Contents
|
||||
1. [Network Upgrade](#network-upgrade)
|
||||
2. [Storage Enhancements](#storage-enhancements)
|
||||
3. [Service Consolidation](#service-consolidation)
|
||||
4. [Security Hardening](#security-hardening)
|
||||
5. [Monitoring & Automation](#monitoring--automation)
|
||||
6. [Backup Strategy](#backup-strategy)
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
- SSH access to all nodes
|
||||
- Root/sudo privileges
|
||||
- Docker Swarm cluster operational
|
||||
- Backblaze B2 account (for backups)
|
||||
|
||||
---
|
||||
|
||||
## 1. Network Upgrade
|
||||
|
||||
### 1.1 Install 2.5 Gb PoE Switch
|
||||
**Hardware**: Netgear GS110EMX or equivalent
|
||||
|
||||
**Steps**:
|
||||
1. Power down affected nodes
|
||||
2. Install new switch
|
||||
3. Connect all 2.5 Gb nodes (Ryzen .81, Acer .57)
|
||||
4. Connect 1 Gb nodes (Pi 4 .245, Time Capsule .153)
|
||||
5. Power on and verify link speeds
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
# On each node, check link speed:
|
||||
ethtool eth0 | grep Speed
|
||||
```
|
||||
|
||||
### 1.2 Configure VLANs
|
||||
**Script**: `/workspace/homelab/scripts/vlan_firewall.sh`
|
||||
|
||||
**Steps**:
|
||||
1. Create VLAN 10 (Management): 192.168.10.0/24
|
||||
2. Create VLAN 20 (Services): 192.168.20.0/24
|
||||
3. Configure router ACLs using the firewall script
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
# Check VLAN configuration
|
||||
ip -d link show
|
||||
|
||||
# Test VLAN isolation
|
||||
ping 192.168.10.1 # from VLAN 20 (should fail for restricted ports)
|
||||
```
|
||||
|
||||
### 1.3 Configure LACP Bonding (Ryzen Node)
|
||||
**Note**: Requires two NICs on the Ryzen node
|
||||
|
||||
**Configuration** (`/etc/network/interfaces.d/bond0.cfg`):
|
||||
```
|
||||
auto bond0
|
||||
iface bond0 inet static
|
||||
address 192.168.1.81
|
||||
netmask 255.255.255.0
|
||||
gateway 192.168.1.1
|
||||
bond-mode 802.3ad
|
||||
bond-miimon 100
|
||||
bond-slaves eth0 eth1
|
||||
```
|
||||
|
||||
**Apply**:
|
||||
```bash
|
||||
sudo systemctl restart networking
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Storage Enhancements
|
||||
|
||||
### 2.1 Create ZFS Pool on Proxmox Host
|
||||
**Script**: `/workspace/homelab/scripts/zfs_setup.sh`
|
||||
|
||||
**Steps**:
|
||||
1. SSH to Proxmox host (192.168.1.57)
|
||||
2. Identify SSD devices: `lsblk`
|
||||
3. Update script with correct device names
|
||||
4. Run: `sudo bash /workspace/homelab/scripts/zfs_setup.sh`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
zpool status tank
|
||||
zfs list
|
||||
```
|
||||
|
||||
### 2.2 Mount NAS on All Nodes
|
||||
**Guide**: `/workspace/homelab/docs/guides/NAS_Mount_Guide.md`
|
||||
|
||||
**Steps**:
|
||||
1. Follow the NAS Mount Guide for each node
|
||||
2. Create credentials file
|
||||
3. Add to `/etc/fstab`
|
||||
4. Mount: `sudo mount -a`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
df -h | grep /mnt/nas
|
||||
ls -la /mnt/nas
|
||||
```
|
||||
|
||||
### 2.3 Setup AI Model Pruning
|
||||
**Script**: `/workspace/homelab/scripts/prune_ai_models.sh`
|
||||
|
||||
**Steps**:
|
||||
1. Update MODEL_DIR path in script
|
||||
2. Make executable: `chmod +x /workspace/homelab/scripts/prune_ai_models.sh`
|
||||
3. Add to cron: `crontab -e`
|
||||
```
|
||||
0 3 * * * /workspace/homelab/scripts/prune_ai_models.sh
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
# Test run
|
||||
sudo /workspace/homelab/scripts/prune_ai_models.sh
|
||||
|
||||
# Check cron logs
|
||||
grep CRON /var/log/syslog
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Service Consolidation
|
||||
|
||||
### 3.1 Deploy Traefik Swarm Service
|
||||
**Stack**: `/workspace/homelab/services/swarm/traefik/stack.yml`
|
||||
|
||||
**Steps**:
|
||||
1. Review and update stack.yml if needed
|
||||
2. Deploy: `docker stack deploy -c /workspace/homelab/services/swarm/traefik/stack.yml traefik`
|
||||
3. Remove standalone Traefik on Pi 4
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
docker service ls | grep traefik
|
||||
docker service ps traefik_traefik
|
||||
curl -I http://192.168.1.196
|
||||
```
|
||||
|
||||
### 3.2 Deploy Caddy Fallback (Pi Zero)
|
||||
**Location**: `/workspace/homelab/services/standalone/Caddy/`
|
||||
|
||||
**Steps**:
|
||||
1. SSH to Pi Zero (192.168.1.62)
|
||||
2. Copy Caddy files to node
|
||||
3. Run: `docker-compose up -d`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
docker ps | grep caddy
|
||||
curl http://192.168.1.62:8080
|
||||
```
|
||||
|
||||
### 3.3 Add Health Checks
|
||||
**Guide**: `/workspace/homelab/docs/guides/health_checks.md`
|
||||
|
||||
**Steps**:
|
||||
1. Review health check examples
|
||||
2. Update service stack files for critical containers
|
||||
3. Redeploy services: `docker stack deploy ...`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
docker ps --filter "health=healthy"
|
||||
docker inspect <container> | jq '.[0].State.Health'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Security Hardening
|
||||
|
||||
### 4.1 Install fail2ban on Manager VM
|
||||
**Script**: `/workspace/homelab/scripts/install_fail2ban.sh`
|
||||
|
||||
**Steps**:
|
||||
1. SSH to manager VM (192.168.1.196)
|
||||
2. Run: `sudo bash /workspace/homelab/scripts/install_fail2ban.sh`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
sudo fail2ban-client status
|
||||
sudo fail2ban-client status sshd
|
||||
sudo tail -f /var/log/fail2ban.log
|
||||
```
|
||||
|
||||
### 4.2 Configure Firewall Rules
|
||||
**Script**: `/workspace/homelab/scripts/vlan_firewall.sh`
|
||||
|
||||
**Steps**:
|
||||
1. Review script and adjust VLANs/ports as needed
|
||||
2. Run: `sudo bash /workspace/homelab/scripts/vlan_firewall.sh`
|
||||
3. Configure router ACLs via web UI
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
sudo iptables -L -n -v
|
||||
# Test port accessibility from different VLANs
|
||||
```
|
||||
|
||||
### 4.3 Restrict Portainer Access
|
||||
**Options**:
|
||||
- Configure Tailscale VPN-only access
|
||||
- Enable OAuth integration
|
||||
- Add firewall rules to block public access
|
||||
|
||||
**Configuration**: Update Portainer stack to bind to Tailscale interface only
|
||||
|
||||
---
|
||||
|
||||
## 5. Monitoring & Automation
|
||||
|
||||
### 5.1 Deploy node-exporter
|
||||
**Script**: `/workspace/homelab/scripts/setup_monitoring.sh`
|
||||
|
||||
**Steps**:
|
||||
1. Run: `sudo bash /workspace/homelab/scripts/setup_monitoring.sh`
|
||||
2. Wait for deployment to complete
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
docker service ps monitoring_node-exporter
|
||||
curl http://192.168.1.196:9100/metrics
|
||||
```
|
||||
|
||||
### 5.2 Configure Grafana Alerts
|
||||
**Rules**: `/workspace/homelab/monitoring/grafana/alert_rules.yml`
|
||||
|
||||
**Steps**:
|
||||
1. The setup script copies alert rules to Grafana
|
||||
2. Login to Grafana UI
|
||||
3. Navigate to Alerting > Alert Rules
|
||||
4. Verify rules are loaded
|
||||
|
||||
**Verification**:
|
||||
- Check Grafana UI for alert rules
|
||||
- Trigger test alert (e.g., high CPU load)
|
||||
|
||||
---
|
||||
|
||||
## 6. Backup Strategy
|
||||
|
||||
### 6.1 Setup Restic Backups
|
||||
**Script**: `/workspace/homelab/scripts/install_restic_backup.sh`
|
||||
|
||||
**Steps**:
|
||||
1. Create Backblaze B2 bucket
|
||||
2. Get B2 account ID and key
|
||||
3. Update `/workspace/homelab/scripts/backup_daily.sh` with credentials
|
||||
4. Run: `sudo bash /workspace/homelab/scripts/install_restic_backup.sh`
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
sudo systemctl status restic-backup.timer
|
||||
sudo systemctl list-timers
|
||||
# Manual test run
|
||||
sudo /workspace/homelab/scripts/backup_daily.sh
|
||||
```
|
||||
|
||||
### 6.2 Verify Backups
|
||||
```bash
|
||||
# Check snapshots
|
||||
export RESTIC_REPOSITORY="b2:your-bucket:/backups"
|
||||
export RESTIC_PASSWORD="your_password"
|
||||
restic snapshots
|
||||
|
||||
# Restore test
|
||||
restic restore latest --target /tmp/restore-test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### If network upgrade fails:
|
||||
- Reconnect to old switch
|
||||
- Remove VLAN configurations
|
||||
- Restart networking: `sudo systemctl restart networking`
|
||||
|
||||
### If ZFS pool creation fails:
|
||||
- Destroy pool: `sudo zpool destroy tank`
|
||||
- Verify data on SSDs before retrying
|
||||
|
||||
### If Traefik Swarm migration fails:
|
||||
- Restart standalone Traefik on Pi 4
|
||||
- Remove Swarm service: `docker service rm traefik_traefik`
|
||||
|
||||
### If backups fail:
|
||||
- Check B2 credentials
|
||||
- Verify network connectivity
|
||||
- Check restic logs: `/var/log/restic_backup.log`
|
||||
|
||||
---
|
||||
|
||||
## Post-Deployment Checklist
|
||||
|
||||
- [ ] All nodes have 2.5 Gb connectivity
|
||||
- [ ] VLANs configured and isolated
|
||||
- [ ] ZFS pool created and healthy
|
||||
- [ ] NAS mounted on all nodes
|
||||
- [ ] Traefik Swarm service running with 2 replicas
|
||||
- [ ] Caddy fallback operational
|
||||
- [ ] fail2ban protecting manager VM
|
||||
- [ ] Firewall rules active
|
||||
- [ ] node-exporter running on all nodes
|
||||
- [ ] Grafana alerts configured
|
||||
- [ ] Restic backups running daily
|
||||
- [ ] Health checks added to critical services
|
||||
|
||||
---
|
||||
|
||||
## Support & Troubleshooting
|
||||
|
||||
Refer to individual guide files for detailed troubleshooting:
|
||||
- [NAS Mount Guide](/workspace/homelab/docs/guides/NAS_Mount_Guide.md)
|
||||
- [Health Checks Guide](/workspace/homelab/docs/guides/health_checks.md)
|
||||
- [Homelab Configuration](/workspace/homelab/docs/guides/Homelab.md)
|
||||
|
||||
For script issues, check logs in `/var/log/` and Docker logs: `docker service logs <service>`
|
||||
375
docs/guides/DISASTER_RECOVERY.md
Normal file
375
docs/guides/DISASTER_RECOVERY.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Disaster Recovery Guide
|
||||
|
||||
## Overview
|
||||
This guide provides procedures for recovering from various failure scenarios in the homelab.
|
||||
|
||||
## Quick Recovery Matrix
|
||||
|
||||
| Scenario | Impact | Recovery Time | Procedure |
|
||||
|----------|--------|---------------|-----------|
|
||||
| Single node failure | Partial | < 5 min | [Node Failure](#node-failure) |
|
||||
| Manager node down | Service disruption | < 10 min | [Manager Recovery](#manager-node-recovery) |
|
||||
| Storage failure | Data risk | < 30 min | [Storage Recovery](#storage-failure) |
|
||||
| Network outage | Complete | < 15 min | [Network Recovery](#network-recovery) |
|
||||
| Complete disaster | Full rebuild | < 2 hours | [Full Recovery](#complete-disaster-recovery) |
|
||||
|
||||
---
|
||||
|
||||
## Node Failure
|
||||
|
||||
### Symptoms
|
||||
- Node unreachable via SSH
|
||||
- Docker services not running on node
|
||||
- Swarm reports node as "Down"
|
||||
|
||||
### Recovery Steps
|
||||
|
||||
1. **Verify node status**:
|
||||
```bash
|
||||
docker node ls
|
||||
# Look for "Down" status
|
||||
```
|
||||
|
||||
2. **Attempt to restart node** (if accessible):
|
||||
```bash
|
||||
ssh user@<node-ip>
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
3. **If node is unrecoverable**:
|
||||
```bash
|
||||
# Remove from Swarm
|
||||
docker node rm <node-id> --force
|
||||
|
||||
# Services will automatically reschedule to healthy nodes
|
||||
```
|
||||
|
||||
4. **Add replacement node**:
|
||||
```bash
|
||||
# On manager node, get join token
|
||||
docker swarm join-token worker
|
||||
|
||||
# On new node, join swarm
|
||||
docker swarm join --token <token> 192.168.1.196:2377
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manager Node Recovery
|
||||
|
||||
### Symptoms
|
||||
- Cannot access Portainer UI
|
||||
- Swarm commands fail
|
||||
- DNS services disrupted
|
||||
|
||||
### Recovery Steps
|
||||
|
||||
1. **Promote a worker to manager** (from another manager if available):
|
||||
```bash
|
||||
docker node promote <worker-node-id>
|
||||
```
|
||||
|
||||
2. **Restore from backup**:
|
||||
```bash
|
||||
# Stop Docker on failed manager
|
||||
sudo systemctl stop docker
|
||||
|
||||
# Restore Portainer data
|
||||
restic restore latest --target /tmp/restore
|
||||
sudo cp -r /tmp/restore/portainer /var/lib/docker/volumes/portainer/_data/
|
||||
|
||||
# Start Docker
|
||||
sudo systemctl start docker
|
||||
```
|
||||
|
||||
3. **Reconfigure DNS** (if Pi-hole affected):
|
||||
```bash
|
||||
# Temporarily point router DNS to another Pi-hole instance
|
||||
# Update router DNS to: 192.168.1.245, 192.168.1.62
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Failure
|
||||
|
||||
### ZFS Pool Failure
|
||||
|
||||
#### Symptoms
|
||||
- `zpool status` shows DEGRADED or FAULTED
|
||||
- I/O errors in logs
|
||||
|
||||
#### Recovery Steps
|
||||
|
||||
1. **Check pool status**:
|
||||
```bash
|
||||
zpool status tank
|
||||
```
|
||||
|
||||
2. **If disk failed**:
|
||||
```bash
|
||||
# Replace failed disk
|
||||
zpool replace tank /dev/old-disk /dev/new-disk
|
||||
|
||||
# Monitor resilver progress
|
||||
watch zpool status tank
|
||||
```
|
||||
|
||||
3. **If pool is destroyed**:
|
||||
```bash
|
||||
# Recreate pool
|
||||
bash /workspace/homelab/scripts/zfs_setup.sh
|
||||
|
||||
# Restore from backup
|
||||
restic restore latest --target /tank/docker
|
||||
```
|
||||
|
||||
### NAS Failure
|
||||
|
||||
#### Recovery Steps
|
||||
|
||||
1. **Check NAS connectivity**:
|
||||
```bash
|
||||
ping 192.168.1.200
|
||||
mount | grep /mnt/nas
|
||||
```
|
||||
|
||||
2. **Remount NAS**:
|
||||
```bash
|
||||
sudo umount /mnt/nas
|
||||
sudo mount -a
|
||||
```
|
||||
|
||||
3. **If NAS hardware failed**:
|
||||
- Services using NAS volumes will fail
|
||||
- Redeploy services to use local storage temporarily
|
||||
- Restore NAS from Time Capsule backup
|
||||
|
||||
---
|
||||
|
||||
## Network Recovery
|
||||
|
||||
### Complete Network Outage
|
||||
|
||||
#### Recovery Steps
|
||||
|
||||
1. **Check physical connections**:
|
||||
- Verify all cables connected
|
||||
- Check switch power and status LEDs
|
||||
- Restart switch
|
||||
|
||||
2. **Verify router**:
|
||||
```bash
|
||||
ping 192.168.1.1
|
||||
# If no response, restart router
|
||||
```
|
||||
|
||||
3. **Check VLAN configuration**:
|
||||
```bash
|
||||
ip -d link show
|
||||
# Reapply if needed
|
||||
bash /workspace/homelab/scripts/vlan_firewall.sh
|
||||
```
|
||||
|
||||
4. **Restart networking**:
|
||||
```bash
|
||||
sudo systemctl restart networking
|
||||
# Or on each node:
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
### Partial Network Issues
|
||||
|
||||
#### DNS Not Resolving
|
||||
|
||||
```bash
|
||||
# Check Pi-hole status
|
||||
docker ps | grep pihole
|
||||
|
||||
# Restart Pi-hole
|
||||
docker restart <pihole-container>
|
||||
|
||||
# Temporarily use public DNS
|
||||
sudo echo "nameserver 8.8.8.8" > /etc/resolv.conf
|
||||
```
|
||||
|
||||
#### Traefik Not Routing
|
||||
|
||||
```bash
|
||||
# Check Traefik service
|
||||
docker service ls | grep traefik
|
||||
docker service ps traefik_traefik
|
||||
|
||||
# Check logs
|
||||
docker service logs traefik_traefik
|
||||
|
||||
# Force update
|
||||
docker service update --force traefik_traefik
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Complete Disaster Recovery
|
||||
|
||||
### Scenario: Total Infrastructure Loss
|
||||
|
||||
#### Prerequisites
|
||||
- Restic backups to Backblaze B2 (off-site)
|
||||
- Hardware replacement available
|
||||
- Network infrastructure functional
|
||||
|
||||
#### Recovery Steps
|
||||
|
||||
1. **Rebuild Core Infrastructure** (2-4 hours):
|
||||
|
||||
```bash
|
||||
# Install base OS on all nodes
|
||||
# Configure network (static IPs, hostnames)
|
||||
|
||||
# Install Docker on all nodes
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
sudo usermod -aG docker $USER
|
||||
|
||||
# Initialize Swarm on manager
|
||||
docker swarm init --advertise-addr 192.168.1.196
|
||||
|
||||
# Join workers
|
||||
docker swarm join-token worker # Get token
|
||||
# Run on each worker with token
|
||||
```
|
||||
|
||||
2. **Restore Storage**:
|
||||
|
||||
```bash
|
||||
# Recreate ZFS pool
|
||||
bash /workspace/homelab/scripts/zfs_setup.sh
|
||||
|
||||
# Mount NAS
|
||||
# Follow: /workspace/homelab/docs/guides/NAS_Mount_Guide.md
|
||||
```
|
||||
|
||||
3. **Restore from Backups**:
|
||||
|
||||
```bash
|
||||
# Install restic
|
||||
sudo apt-get install restic
|
||||
|
||||
# Configure credentials
|
||||
export B2_ACCOUNT_ID="..."
|
||||
export B2_ACCOUNT_KEY="..."
|
||||
export RESTIC_REPOSITORY="b2:bucket:/backups"
|
||||
export RESTIC_PASSWORD="..."
|
||||
|
||||
# List snapshots
|
||||
restic snapshots
|
||||
|
||||
# Restore latest
|
||||
restic restore latest --target /tmp/restore
|
||||
|
||||
# Copy to Docker volumes
|
||||
sudo cp -r /tmp/restore/* /var/lib/docker/volumes/
|
||||
```
|
||||
|
||||
4. **Redeploy Services**:
|
||||
|
||||
```bash
|
||||
# Deploy all stacks
|
||||
bash /workspace/homelab/scripts/deploy_all.sh
|
||||
|
||||
# Verify deployment
|
||||
bash /workspace/homelab/scripts/validate_deployment.sh
|
||||
```
|
||||
|
||||
5. **Verify Recovery**:
|
||||
|
||||
- Check all services: `docker service ls`
|
||||
- Test Traefik routing: `curl https://your-domain.com`
|
||||
- Verify Portainer UI access
|
||||
- Check Grafana dashboards
|
||||
- Test Home Assistant
|
||||
|
||||
---
|
||||
|
||||
## Backup Verification
|
||||
|
||||
### Monthly Backup Test
|
||||
|
||||
```bash
|
||||
# List snapshots
|
||||
restic snapshots
|
||||
|
||||
# Verify specific snapshot
|
||||
restic check --read-data-subset=10%
|
||||
|
||||
# Test restore
|
||||
mkdir /tmp/restore-test
|
||||
restic restore <snapshot-id> --target /tmp/restore-test --include /path/to/critical/file
|
||||
|
||||
# Compare with original
|
||||
diff -r /tmp/restore-test /original/path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Emergency Contacts & Resources
|
||||
|
||||
### Critical Information
|
||||
- **Backblaze B2 Login**: Store credentials in password manager
|
||||
- **restic Password**: Store securely (CANNOT be recovered)
|
||||
- **Router Admin**: Keep credentials accessible
|
||||
- **ISP Support**: Keep contact info handy
|
||||
|
||||
### Documentation URLs
|
||||
- Docker Swarm: https://docs.docker.com/engine/swarm/
|
||||
- Traefik: https://doc.traefik.io/traefik/
|
||||
- Restic: https://restic.readthedocs.io/
|
||||
- ZFS: https://openzfs.github.io/openzfs-docs/
|
||||
|
||||
---
|
||||
|
||||
## Recovery Checklists
|
||||
|
||||
### Pre-Disaster Preparation
|
||||
- [ ] Verify backups running daily
|
||||
- [ ] Test restore procedure monthly
|
||||
- [ ] Document all credentials
|
||||
- [ ] Keep hardware spares (cables, drives)
|
||||
- [ ] Maintain off-site config copies
|
||||
|
||||
### Post-Recovery Validation
|
||||
- [ ] All nodes online: `docker node ls`
|
||||
- [ ] All services running: `docker service ls`
|
||||
- [ ] Health checks passing: `docker ps --filter health=healthy`
|
||||
- [ ] DNS resolving correctly
|
||||
- [ ] Monitoring active (Grafana accessible)
|
||||
- [ ] Backups resumed: `systemctl status restic-backup.timer`
|
||||
- [ ] fail2ban protecting: `fail2ban-client status`
|
||||
- [ ] Network performance normal: `bash network_performance_test.sh`
|
||||
|
||||
---
|
||||
|
||||
## Automation for Faster Recovery
|
||||
|
||||
### Create Recovery USB Drive
|
||||
|
||||
```bash
|
||||
# Copy all scripts and configs
|
||||
mkdir /mnt/usb/homelab-recovery
|
||||
cp -r /workspace/homelab/* /mnt/usb/homelab-recovery/
|
||||
|
||||
# Include documentation
|
||||
cp /workspace/homelab/docs/guides/* /mnt/usb/homelab-recovery/docs/
|
||||
|
||||
# Store credentials (encrypted)
|
||||
# Use GPG or similar to encrypt sensitive files
|
||||
```
|
||||
|
||||
### Quick Deploy Script
|
||||
|
||||
```bash
|
||||
# Run from recovery USB
|
||||
sudo bash /mnt/usb/homelab-recovery/scripts/deploy_all.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
This guide should be reviewed and updated quarterly to ensure accuracy.
|
||||
270
docs/guides/Homelab.md
Normal file
270
docs/guides/Homelab.md
Normal file
@@ -0,0 +1,270 @@
|
||||
# HOMELAB CONFIGURATION SUMMARY — UPDATED 2025-10-31
|
||||
|
||||
## NETWORK INFRASTRUCTURE
|
||||
Main Router: TP-Link BE9300 (2.5 Gb WAN + 4× 2.5 Gb LAN)
|
||||
Secondary Router: Linksys WRT3200ACM (OpenWRT)
|
||||
Managed Switch: TP-Link TL-SG608E (1 Gb)
|
||||
Additional: Apple AirPort Time Capsule (192.168.1.153)
|
||||
Backbone Speed: 2.5 Gb core / 1 Gb secondary
|
||||
DNS Architecture: 3× Pi-hole + 3× Unbound (192.168.1.196, .245, .62) with local recursive forwarding
|
||||
VPN: Tailscale (Pi 4 as exit node)
|
||||
Reverse Proxy: Traefik (on .196; planned Swarm takeover)
|
||||
LAN Subnet: 192.168.1.0/24
|
||||
Notes: Rate-limit prevention on Pi-hole instances, Unbound local caching to accelerate DNS queries
|
||||
|
||||
---
|
||||
|
||||
## NODE OVERVIEW
|
||||
|
||||
192.168.1.81 — Ryzen 3700X Node
|
||||
• CPU: AMD 8C/16T
|
||||
• RAM: 64–80 GB Current 2 of 4 3200 32gb 4x8gb 3600 availible
|
||||
• GPU: RTX 4060 Ti
|
||||
• Network: 2.5 GbE onboard
|
||||
• Role: Docker Swarm Worker (label=heavy)
|
||||
• Function: AI compute (LM Studio, Llama.cpp, OpenWebUI, Ollama planned)
|
||||
• OS: Windows 11 + WSL2 / Fedora (Dual Boot)
|
||||
• Notes: Primary compute node for high-performance AI workloads. Both OS installations act as interchangeable swarm nodes with the same label.
|
||||
|
||||
192.168.1.57 — Acer Aspire R14 (Proxmox Host)
|
||||
• CPU: Intel i5-6200U (2C/4T)
|
||||
|
||||
---
|
||||
## NETWORK UPGRADE & VLAN
|
||||
* **Switch**: Install a 2.5 Gb PoE managed switch (e.g., Netgear GS110EMX).
|
||||
* **VLANs**: Create VLAN 10 for management, VLAN 20 for services. Add router ACLs to isolate traffic.
|
||||
* **LACP**: Bond two NICs on the Ryzen node for 5 Gb aggregated link.
|
||||
|
||||
## STORAGE ENHANCEMENTS
|
||||
* Deploy a dedicated NAS (e.g., Synology DS920+) with RAID‑6 and SSD cache.
|
||||
* On Proxmox host, create ZFS pool `tank` on local SSDs (`zpool create tank /dev/sda /dev/sdb`).
|
||||
* Mount NAS shares on all nodes (`/mnt/nas`).
|
||||
* Add cron job to prune unused AI model caches.
|
||||
|
||||
## SERVICE CONSOLIDATION & RESILIENCE
|
||||
* Convert standalone Traefik on Pi 4 to a Docker‑Swarm service with 2 replicas.
|
||||
* Deploy fallback Caddy on Pi Zero with a static maintenance page.
|
||||
* Add health‑check sidecars to critical containers (Portainer, OpenWebUI).
|
||||
* Separate persistent volumes per stack (AI models on SSD, Nextcloud on NAS).
|
||||
|
||||
## SECURITY HARDENING
|
||||
* Enable router firewall ACLs for inter‑VLAN traffic (allow only required ports).
|
||||
* Install `fail2ban` on the manager VM.
|
||||
* Restrict Portainer UI to VPN‑only access and enable 2FA/OAuth.
|
||||
|
||||
## MONITORING & AUTOMATION
|
||||
* Deploy `node-exporter` on Proxmox host.
|
||||
* Create Grafana alerts for CPU > 80 %, RAM > 85 %, disk > 80 %.
|
||||
* Add Home‑Assistant backup automation to NAS.
|
||||
* Integrate Tailscale metrics via `tailscale_exporter`.
|
||||
|
||||
## OFF‑SITE BACKUP STRATEGY
|
||||
* Install `restic` on manager VM and initialise Backblaze B2 repo.
|
||||
* Daily backup script (`/usr/local/bin/backup_daily.sh`) for HA config, Portainer DB, important volumes.
|
||||
* Systemd timer to run at 02:00 AM.
|
||||
|
||||
---
|
||||
• RAM: 8 GB
|
||||
• Network: 2.5 GbE via USB adapter
|
||||
• Role: Proxmox Host
|
||||
• Function: Virtualization host for Apps VM (.196) and OMV (.70)
|
||||
• Storage: Local SSDs + OMV shared volumes
|
||||
• Notes: Lightweight node for VMs and containerized storage services
|
||||
|
||||
192.168.1.196 — Apps Manager VM (on Acer Proxmox)
|
||||
CPU: 4
|
||||
RAM: 4 GB min 6 GB max
|
||||
• Role: Docker Swarm Manager (label=manager)
|
||||
• Function: Pi-hole + Unbound + Portainer UI + Traefik reverse proxy
|
||||
• Architecture: x86 (virtualized)
|
||||
• Notes: Central orchestration, DNS control, and reverse proxy; Portainer agent installed for remote swarm management
|
||||
|
||||
192.168.1.70 — OMV Instance (on Acer)
|
||||
CPU 2
|
||||
RAM: 2 GB min 4 GB max
|
||||
• Role: Network Attached Storage
|
||||
• Function: Shared Docker volumes, media, VM backups
|
||||
• Stack: OpenMediaVault 7.x
|
||||
• Architecture: x86
|
||||
• Planned: Receive SMB3-reshares from Time Capsule (.153)
|
||||
• Storage: Docker volumes for AI models, backup directories, and media
|
||||
• Notes: Central NAS for swarm and LLM storage
|
||||
|
||||
192.168.1.245 — Raspberry Pi 4 (8 GB)
|
||||
• CPU: ARM Quad-Core
|
||||
• RAM: 8 GB
|
||||
• Network: 1 GbE
|
||||
• Role: Docker Swarm Leader (label=leader)
|
||||
• Function: Home Assistant OS + Portainer Agent + HAOS-based Unbound (via Ubuntu container)
|
||||
• Standalone Services: Traefik (currently standalone), HAOS Unbound
|
||||
• Notes: Central smart home automation hub; swarm leader for container orchestration; plan for Swarm Traefik to take over existing Traefik instance
|
||||
|
||||
192.168.1.62 — Raspberry Pi Zero 2 W
|
||||
• CPU: ARM Quad-Core
|
||||
• RAM: 512 MB
|
||||
• Network: 100 Mb Ethernet
|
||||
• Role: Docker Swarm Worker (label=light)
|
||||
• Function: Lightweight DNS + Pi-hole + Unbound + auxiliary containers
|
||||
• Notes: Low-power node for background jobs, DNS redundancy, and monitoring tasks
|
||||
|
||||
192.168.1.153 — Apple AirPort Time Capsule
|
||||
• Network: 1 GbE via WRT3200ACM
|
||||
• Role: Backup storage and SMB bridge
|
||||
• Function: Time Machine backups (SMB1)
|
||||
• Planned: Reshare SMB1 → SMB3 via OMV (.70) for modern clients
|
||||
• Notes: Source for macOS backups; will integrate into OMV NAS for consolidation
|
||||
|
||||
---
|
||||
|
||||
## DOCKER SWARM CLUSTER
|
||||
Leader 192.168.1.245 (Pi 4, label=leader)
|
||||
Manager 192.168.1.196 (Apps VM, label=manager)
|
||||
Worker (Fedora) 192.168.1.81 (Ryzen, label=heavy)
|
||||
Worker (Light) 192.168.1.62 (Pi Zero 2 W, label=light)
|
||||
|
||||
Cluster Functions:
|
||||
• Distributed container orchestration across x86 + ARM
|
||||
• High-availability DNS via Pi-hole + Unbound replicas
|
||||
• Unified management and reverse proxy on the manager node
|
||||
• Specific workload placement using node labels (heavy, leader, manager)
|
||||
• AI/ML workloads pinned to the 'heavy' node for performance
|
||||
• General application services pinned to the 'leader' node
|
||||
• Core services like Traefik and Portainer pinned to the 'manager' node
|
||||
|
||||
---
|
||||
|
||||
## STACKS
|
||||
|
||||
### Networking Stack
|
||||
• **Traefik:** Reverse Proxy
|
||||
• **whoami:** Service for testing Traefik
|
||||
|
||||
### Monitoring Stack
|
||||
• **Prometheus:** Metrics collection
|
||||
• **Grafana:** Metrics visualization
|
||||
• **Alertmanager:** Alerting
|
||||
• **Node-exporter:** Node metrics exporter
|
||||
• **cAdvisor:** Container metrics exporter
|
||||
|
||||
### Tools Stack
|
||||
• **Portainer:** Swarm Management
|
||||
• **Dozzle:** Log viewing
|
||||
• **Lazydocker:** Terminal UI for Docker
|
||||
• **TSDProxy:** Tailscale Docker Proxy
|
||||
• **Watchtower:** Container Updates
|
||||
|
||||
### Application Stack
|
||||
• **OpenWebUI:** AI Frontend
|
||||
• **Paperless-ngx:** Document Management
|
||||
• **Stirling-PDF:** PDF utility
|
||||
• **SearXNG:** Metasearch engine
|
||||
|
||||
### Productivity Stack
|
||||
• **Nextcloud:** Cloud storage and collaboration
|
||||
|
||||
---
|
||||
|
||||
## SERVICES MAP
|
||||
• **Manager Node (.196):**
|
||||
• **Networking Stack:** Traefik
|
||||
• **Monitoring Stack:** Prometheus, Grafana
|
||||
• **Tools Stack:** Portainer, Dozzle, Lazydocker, TSDProxy, Watchtower
|
||||
• **Leader Node (.245):**
|
||||
• **Application Stack:** Paperless-ngx, Stirling-PDF, SearXNG
|
||||
• **Productivity Stack:** Nextcloud
|
||||
• **Heavy Worker Node (.81):**
|
||||
• **Application Stack:** OpenWebUI
|
||||
• **Light Worker Node (.62):**
|
||||
• **Networking Stack:** whoami
|
||||
• **Other Services:**
|
||||
• **VPN:** Tailscale (Pi4 exit node)
|
||||
• **Virtualization:** Proxmox VE (.57)
|
||||
• **Storage:** OMV NAS (.70) + Time Capsule (.153)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## STORAGE & BACKUPS
|
||||
OMV (.70) — shared Docker volumes, LLM models, media, backup directories
|
||||
Time Capsule (.153) — legacy SMB1 source; planned SMB3 reshare via OMV
|
||||
External SSDs/HDDs — portable compute, LLM scratch storage, media archives
|
||||
Time Machine clients — macOS systems
|
||||
Planned Workflow:
|
||||
• Mount Time Capsule SMB1 share in OMV via CIFS
|
||||
• Reshare through OMV Samba as SMB3
|
||||
• Sync critical backups to OMV and external drives
|
||||
• AI models stored on NVMe + OMV volumes for high-speed access
|
||||
|
||||
---
|
||||
|
||||
## PERFORMANCE STRATEGY
|
||||
• 2.5 Gb backbone: Ryzen (.81) + Acer (.57) nodes
|
||||
• 1 Gb nodes: Pi 4 (.245) + Time Capsule (.153)
|
||||
• 100 Mb node: Pi Zero 2 W (.62)
|
||||
• ARM nodes for low-power/auxiliary tasks
|
||||
• x86 nodes for AI, storage, and compute-intensive containers
|
||||
• Swarm resource labeling for workload isolation
|
||||
• DNS redundancy and rate-limit protection
|
||||
• Unified monitoring via Portainer + Home Assistant
|
||||
• GPU-intensive AI containers pinned to Ryzen node for efficiency
|
||||
• Traefik migration plan: standalone .245 → Swarm-managed cluster routing
|
||||
|
||||
---
|
||||
|
||||
## NOTES
|
||||
• Acer Proxmox hosts OMV (.70) and Apps Manager VM (.196)
|
||||
• Ryzen (.81) dedicated to AI and heavy Docker tasks
|
||||
• HAOS Pi 4 (.245) leader, automation hub, and temporary standalone Traefik
|
||||
• DNS load balanced among .62, .196, and .245
|
||||
• Time Capsule (.153) planned SMB1→SMB3 reshare via OMV
|
||||
• Network speed distribution: Ryzen/Acer = 2.5 Gb, Pi 4/Time Capsule = 1 Gb, Pi Zero 2 W = 100 Mb
|
||||
• LLM models stored on high-speed NVMe on Ryzen, backed up to OMV and external drives
|
||||
• No personal identifiers included in this record
|
||||
|
||||
# END CONFIG
|
||||
|
||||
---
|
||||
|
||||
## SMART HOME INTEGRATION
|
||||
|
||||
### LIGHTING & CONTROLS
|
||||
• Philips Hue
|
||||
- Devices: Hue remote only (no bulbs)
|
||||
- Connectivity: Zigbee
|
||||
- Automation: Integrated into Home Assistant OS (.245)
|
||||
- Notes: Remote used to trigger HAOS scenes and routines for other smart devices
|
||||
|
||||
• Govee Smart Lights & Sensors
|
||||
- Devices: RGB LED strips, motion sensors, temperature/humidity sensors
|
||||
- Connectivity: Wi-Fi
|
||||
- Automation: Home Assistant via MQTT / cloud integration
|
||||
- Notes: Motion-triggered lighting and environmental monitoring
|
||||
|
||||
• TP-Link / Tapo Smart Devices
|
||||
- Devices: Tapo lightbulbs, Kasa smart power strip
|
||||
- Connectivity: Wi-Fi
|
||||
- Automation: Home Assistant + Kasa/Tapo integration
|
||||
- Notes: Power scheduling and energy monitoring
|
||||
|
||||
### AUDIO & VIDEO
|
||||
• TVs: Multiple 4K Smart TVs
|
||||
- Platforms: Fire Stick, Apple devices, console inputs
|
||||
- Connectivity: Ethernet (1 Gb) or Wi-Fi
|
||||
- Automation: HAOS scenes, volume control, source switching
|
||||
|
||||
• Streaming & Consoles:
|
||||
- Devices: Fire Stick, PS5, Nintendo Switch
|
||||
- Connectivity: Ethernet or Wi-Fi
|
||||
- Notes: Automated on/off with Home Assistant, media triggers
|
||||
|
||||
### SECURITY & SENSORS
|
||||
• Vivint Security System
|
||||
- Devices: Motion detectors, door/window sensors, cameras
|
||||
- Connectivity: Proprietary protocol + cloud
|
||||
- Automation: Home Assistant integrations for alerts and scene triggers
|
||||
|
||||
• Environmental Sensors
|
||||
- Devices: Govee temperature/humidity, Tapo sensors
|
||||
- Connectivity: Wi-Fi
|
||||
- Automation: Trigger HVAC, lights, or notifications
|
||||
|
||||
62
docs/guides/NAS_Mount_Guide.md
Normal file
62
docs/guides/NAS_Mount_Guide.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# NAS Mount Guide
|
||||
|
||||
This guide explains how to mount the dedicated NAS shares on all homelab nodes.
|
||||
|
||||
## Prerequisites
|
||||
- NAS is reachable at `\192.168.1.200` (replace with your NAS IP).
|
||||
- You have a user account on the NAS with read/write permissions.
|
||||
- `cifs-utils` is installed on each node (`sudo apt-get install cifs-utils`).
|
||||
|
||||
## Mount Point
|
||||
Create a common mount point on each node:
|
||||
```bash
|
||||
sudo mkdir -p /mnt/nas
|
||||
```
|
||||
|
||||
## Credentials File (optional)
|
||||
Store credentials in a secure file (e.g., `/etc/nas-cred`):
|
||||
```text
|
||||
username=your_nas_user
|
||||
password=your_nas_password
|
||||
```
|
||||
Set restrictive permissions:
|
||||
```bash
|
||||
sudo chmod 600 /etc/nas-cred
|
||||
```
|
||||
|
||||
## Add to `/etc/fstab`
|
||||
Append the following line to `/etc/fstab` on each node:
|
||||
```text
|
||||
//192.168.1.200/shared /mnt/nas cifs credentials=/etc/nas-cred,iocharset=utf8,vers=3.0 0 0
|
||||
```
|
||||
Replace `shared` with the actual share name.
|
||||
|
||||
## Mount Immediately
|
||||
```bash
|
||||
sudo mount -a
|
||||
```
|
||||
Verify:
|
||||
```bash
|
||||
df -h | grep /mnt/nas
|
||||
```
|
||||
You should see the NAS share listed.
|
||||
|
||||
## Docker Volume Example
|
||||
When deploying services that need persistent storage, reference the NAS mount:
|
||||
```yaml
|
||||
volumes:
|
||||
nas-data:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /mnt/nas/your-service-data
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
- **Permission denied** – ensure the NAS user has the correct permissions and the credentials file is correct.
|
||||
- **Mount fails** – try specifying a different SMB version (`vers=2.1` or `vers=3.1.1`).
|
||||
- **Network issues** – verify the node can ping the NAS IP.
|
||||
|
||||
---
|
||||
*This guide can be referenced from the updated `Homelab.md` documentation.*
|
||||
475
docs/guides/OMV.md
Normal file
475
docs/guides/OMV.md
Normal file
@@ -0,0 +1,475 @@
|
||||
# OMV Configuration Guide for Docker Swarm Integration
|
||||
|
||||
This guide outlines the setup for an OpenMediaVault (OMV) virtual machine and its integration with a Docker Swarm cluster for providing network storage to services like Jellyfin, Nextcloud, Immich, and others.
|
||||
|
||||
---
|
||||
|
||||
## 1. OMV Virtual Machine Configuration
|
||||
|
||||
The OMV instance is configured as a virtual machine with the following specifications:
|
||||
|
||||
- **RAM:** 2-4 GB
|
||||
- **CPU:** 2 Cores
|
||||
- **System Storage:** 32 GB
|
||||
- **Data Storage:** A 512GB SATA SSD is passed through directly from the Proxmox host. This SSD is dedicated to network shares.
|
||||
- **Network:** Static IP address `192.168.1.70` on the `192.168.1.0/24` subnet
|
||||
|
||||
---
|
||||
|
||||
## 2. Network Share Setup in OMV
|
||||
|
||||
The primary purpose of this OMV instance is to serve files to other applications and services on the network, particularly Docker Swarm containers.
|
||||
|
||||
### Shared Folders Overview
|
||||
|
||||
The following shared folders should be created in OMV (via **Storage → Shared Folders**):
|
||||
|
||||
| Folder Name | Purpose | Protocol | Permissions |
|
||||
|-------------|---------|----------|-------------|
|
||||
| `Media` | Media files for Jellyfin | SMB | swarm-user: RW |
|
||||
| `ImmichUploads` | Photo uploads for Immich | NFS | UID 999: RW |
|
||||
| `TraefikLetsEncrypt` | SSL certificates for Traefik | NFS | Root: RW |
|
||||
| `ImmichDB` | Immich PostgreSQL database | NFS | Root: RW |
|
||||
| `NextcloudDB` | Nextcloud PostgreSQL database | NFS | Root: RW |
|
||||
| `NextcloudApps` | Nextcloud custom apps | NFS | www-data (33): RW |
|
||||
| `NextcloudConfig` | Nextcloud configuration | NFS | www-data (33): RW |
|
||||
| `NextcloudData` | Nextcloud user data | NFS | www-data (33): RW |
|
||||
|
||||
### SMB (Server Message Block) Shares
|
||||
|
||||
SMB is used for services that require file-based media access, particularly for services accessed by multiple platforms (Windows, Linux, macOS).
|
||||
|
||||
#### **Media Share**
|
||||
- **Shared Folder:** `Media`
|
||||
- **Purpose:** Stores media files for Jellyfin and other media servers
|
||||
- **SMB Configuration:**
|
||||
- **Share Name:** `Media`
|
||||
- **Public:** No (authentication required)
|
||||
- **Browseable:** Yes
|
||||
- **Read-only:** No
|
||||
- **Guest Access:** No
|
||||
- **Permissions:** `swarm-user` has read/write access
|
||||
- **Path on OMV:** `/srv/dev-disk-by-uuid-fd2daa6f-bd75-4ac1-9c4c-9e4d4b84d845/Media`
|
||||
|
||||
### NFS (Network File System) Shares
|
||||
|
||||
NFS is utilized for services requiring block-level access, specific POSIX permissions, or better performance for containerized applications.
|
||||
|
||||
#### **Nextcloud Shares**
|
||||
- **Shared Folders:** `NextcloudApps`, `NextcloudConfig`, `NextcloudData`
|
||||
- **Purpose:** Application files, configuration, and user data for Nextcloud
|
||||
- **NFS Configuration:**
|
||||
- **Client:** `192.168.1.0/24` (Accessible to the entire subnet)
|
||||
- **Privilege:** Read/Write
|
||||
- **Extra Options:** `all_squash,anongid=33,anonuid=33,sync,no_subtree_check`
|
||||
- `all_squash`: Maps all client UIDs/GIDs to anonymous user
|
||||
- `anonuid=33,anongid=33`: Maps to `www-data` user/group (Nextcloud/Apache/Nginx)
|
||||
- `sync`: Ensures data is written to disk before acknowledging (data integrity)
|
||||
- `no_subtree_check`: Improves reliability for directory exports
|
||||
|
||||
#### **Database Shares**
|
||||
- **Shared Folders:** `ImmichDB`, `NextcloudDB`
|
||||
- **Purpose:** PostgreSQL database storage for Immich and Nextcloud
|
||||
- **NFS Configuration:**
|
||||
- **Client:** `192.168.1.0/24`
|
||||
- **Privilege:** Read/Write
|
||||
- **Extra Options:** `rw,sync,no_subtree_check,no_root_squash`
|
||||
- `no_root_squash`: Allows root on client to be treated as root on server (needed for database operations)
|
||||
- `sync`: Critical for database integrity
|
||||
|
||||
#### **Application Data Shares**
|
||||
- **Shared Folder:** `ImmichUploads`
|
||||
- **Purpose:** Photo and video uploads for Immich
|
||||
- **NFS Configuration:**
|
||||
- **Client:** `192.168.1.0/24`
|
||||
- **Privilege:** Read/Write
|
||||
- **Extra Options:** `rw,sync,no_subtree_check,all_squash,anonuid=999,anongid=999`
|
||||
- Maps to Immich's internal user (typically UID/GID 999)
|
||||
|
||||
- **Shared Folder:** `TraefikLetsEncrypt`
|
||||
- **Purpose:** SSL certificate storage for Traefik reverse proxy
|
||||
- **NFS Configuration:**
|
||||
- **Client:** `192.168.1.0/24`
|
||||
- **Privilege:** Read/Write
|
||||
- **Extra Options:** `rw,sync,no_subtree_check,no_root_squash`
|
||||
|
||||
---
|
||||
|
||||
## 3. Integrating OMV Shares with Docker Swarm Services
|
||||
|
||||
To use the OMV network shares with Docker Swarm services, the shares must be mounted on the Docker worker nodes where the service containers will run. The mounted path on the node is then passed into the container as a volume.
|
||||
|
||||
### Prerequisites on Docker Nodes
|
||||
|
||||
All Docker nodes that will mount shares need the appropriate client utilities installed:
|
||||
|
||||
```bash
|
||||
# For SMB shares
|
||||
sudo apt-get update
|
||||
sudo apt-get install cifs-utils
|
||||
|
||||
# For NFS shares
|
||||
sudo apt-get update
|
||||
sudo apt-get install nfs-common
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 1: Jellyfin Media Access via SMB
|
||||
|
||||
Jellyfin, running as a Docker Swarm service, requires access to the media files stored on the OMV `Media` share.
|
||||
|
||||
#### **Step 1: Create SMB Credentials File**
|
||||
|
||||
Create a credentials file on the Docker node to avoid storing passwords in `/etc/fstab`:
|
||||
|
||||
```bash
|
||||
# Create credentials file
|
||||
sudo nano /root/.smbcredentials
|
||||
```
|
||||
|
||||
Add the following content:
|
||||
```
|
||||
username=swarm-user
|
||||
password=YOUR_PASSWORD_HERE
|
||||
```
|
||||
|
||||
Secure the file:
|
||||
```bash
|
||||
sudo chmod 600 /root/.smbcredentials
|
||||
```
|
||||
|
||||
#### **Step 2: Mount the SMB Share on the Docker Node**
|
||||
|
||||
```bash
|
||||
# Create mount point
|
||||
sudo mkdir -p /mnt/media
|
||||
|
||||
# Test the mount first
|
||||
sudo mount -t cifs //192.168.1.70/Media /mnt/media -o credentials=/root/.smbcredentials,iocharset=utf8,vers=3.0
|
||||
|
||||
# Verify it works
|
||||
ls -la /mnt/media
|
||||
|
||||
# Unmount test
|
||||
sudo umount /mnt/media
|
||||
```
|
||||
|
||||
#### **Step 3: Add Permanent Mount to `/etc/fstab`**
|
||||
|
||||
```bash
|
||||
sudo nano /etc/fstab
|
||||
```
|
||||
|
||||
Add this line:
|
||||
```
|
||||
//192.168.1.70/Media /mnt/media cifs credentials=/root/.smbcredentials,iocharset=utf8,vers=3.0,file_mode=0755,dir_mode=0755 0 0
|
||||
```
|
||||
|
||||
Mount all entries:
|
||||
```bash
|
||||
sudo mount -a
|
||||
```
|
||||
|
||||
#### **Step 4: Configure the Jellyfin Docker Swarm Service**
|
||||
|
||||
In the Docker Compose YAML file for your Jellyfin service:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
jellyfin:
|
||||
image: jellyfin/jellyfin:latest
|
||||
volumes:
|
||||
- /mnt/media:/media:ro # Read-only access to prevent accidental deletion
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.media==true # Deploy only on nodes with media mount
|
||||
# ... other configurations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 2: Nextcloud Data Access via NFS
|
||||
|
||||
Nextcloud, running as a Docker Swarm service, requires access to its application, configuration, and data files stored on the OMV NFS shares.
|
||||
|
||||
#### **Step 1: Create Mount Points**
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /mnt/nextcloud/{apps,config,data}
|
||||
```
|
||||
|
||||
#### **Step 2: Test NFS Mounts**
|
||||
|
||||
```bash
|
||||
# Test each mount
|
||||
sudo mount -t nfs 192.168.1.70:/NextcloudApps /mnt/nextcloud/apps -o vers=4.2
|
||||
sudo mount -t nfs 192.168.1.70:/NextcloudConfig /mnt/nextcloud/config -o vers=4.2
|
||||
sudo mount -t nfs 192.168.1.70:/NextcloudData /mnt/nextcloud/data -o vers=4.2
|
||||
|
||||
# Verify
|
||||
ls -la /mnt/nextcloud/apps
|
||||
ls -la /mnt/nextcloud/config
|
||||
ls -la /mnt/nextcloud/data
|
||||
|
||||
# Unmount tests
|
||||
sudo umount /mnt/nextcloud/apps
|
||||
sudo umount /mnt/nextcloud/config
|
||||
sudo umount /mnt/nextcloud/data
|
||||
```
|
||||
|
||||
#### **Step 3: Add Permanent Mounts to `/etc/fstab`**
|
||||
|
||||
```bash
|
||||
sudo nano /etc/fstab
|
||||
```
|
||||
|
||||
Add these lines:
|
||||
```
|
||||
192.168.1.70:/NextcloudApps /mnt/nextcloud/apps nfs auto,nofail,noatime,rw,vers=4.2,all_squash,anongid=33,anonuid=33 0 0
|
||||
192.168.1.70:/NextcloudConfig /mnt/nextcloud/config nfs auto,nofail,noatime,rw,vers=4.2,all_squash,anongid=33,anonuid=33 0 0
|
||||
192.168.1.70:/NextcloudData /mnt/nextcloud/data nfs auto,nofail,noatime,rw,vers=4.2,all_squash,anongid=33,anonuid=33 0 0
|
||||
```
|
||||
|
||||
**Mount Options Explained:**
|
||||
- `auto`: Mount at boot
|
||||
- `nofail`: Don't fail boot if mount fails
|
||||
- `noatime`: Don't update access times (performance)
|
||||
- `rw`: Read-write
|
||||
- `vers=4.2`: Use NFSv4.2 (better performance and security)
|
||||
- `all_squash,anongid=33,anonuid=33`: Map all users to www-data
|
||||
|
||||
Mount all entries:
|
||||
```bash
|
||||
sudo mount -a
|
||||
```
|
||||
|
||||
#### **Step 4: Configure the Nextcloud Docker Swarm Service**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nextcloud:
|
||||
image: nextcloud:latest
|
||||
volumes:
|
||||
- /mnt/nextcloud/apps:/var/www/html/custom_apps
|
||||
- /mnt/nextcloud/config:/var/www/html/config
|
||||
- /mnt/nextcloud/data:/var/www/html/data
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud==true
|
||||
# ... other configurations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 3: Database Storage via NFS
|
||||
|
||||
For stateful services like databases, storing their data on a resilient network share is critical for data integrity and high availability.
|
||||
|
||||
#### **Step 1: Create Mount Points**
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /mnt/database/{immich,nextcloud}
|
||||
```
|
||||
|
||||
#### **Step 2: Test NFS Mounts**
|
||||
|
||||
```bash
|
||||
# Test mounts
|
||||
sudo mount -t nfs 192.168.1.70:/ImmichDB /mnt/database/immich -o vers=4.2
|
||||
sudo mount -t nfs 192.168.1.70:/NextcloudDB /mnt/database/nextcloud -o vers=4.2
|
||||
|
||||
# Verify
|
||||
ls -la /mnt/database/immich
|
||||
ls -la /mnt/database/nextcloud
|
||||
|
||||
# Unmount tests
|
||||
sudo umount /mnt/database/immich
|
||||
sudo umount /mnt/database/nextcloud
|
||||
```
|
||||
|
||||
#### **Step 3: Add Permanent Mounts to `/etc/fstab`**
|
||||
|
||||
```bash
|
||||
sudo nano /etc/fstab
|
||||
```
|
||||
|
||||
Add these lines:
|
||||
```
|
||||
192.168.1.70:/ImmichDB /mnt/database/immich nfs auto,nofail,noatime,rw,vers=4.2,sync,no_subtree_check,no_root_squash 0 0
|
||||
192.168.1.70:/NextcloudDB /mnt/database/nextcloud nfs auto,nofail,noatime,rw,vers=4.2,sync,no_subtree_check,no_root_squash 0 0
|
||||
```
|
||||
|
||||
**Critical for Databases:**
|
||||
- `sync`: Ensures writes are committed to disk before acknowledgment (prevents data corruption)
|
||||
- `no_root_squash`: Allows database containers running as root to maintain proper permissions
|
||||
|
||||
Mount all entries:
|
||||
```bash
|
||||
sudo mount -a
|
||||
```
|
||||
|
||||
#### **Step 4: Configure Database Docker Swarm Services**
|
||||
|
||||
**Immich Database:**
|
||||
```yaml
|
||||
services:
|
||||
immich-db:
|
||||
image: tensorchord/pgvecto-rs:pg14-v0.2.0
|
||||
volumes:
|
||||
- /mnt/database/immich:/var/lib/postgresql/data
|
||||
environment:
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD}
|
||||
POSTGRES_USER: immich
|
||||
POSTGRES_DB: immich
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.database==true
|
||||
```
|
||||
|
||||
**Nextcloud Database:**
|
||||
```yaml
|
||||
services:
|
||||
nextcloud-db:
|
||||
image: postgres:15-alpine
|
||||
volumes:
|
||||
- /mnt/database/nextcloud:/var/lib/postgresql/data
|
||||
environment:
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD}
|
||||
POSTGRES_USER: nextcloud
|
||||
POSTGRES_DB: nextcloud
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.database==true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 4: Immich Upload Storage via NFS
|
||||
|
||||
```bash
|
||||
# Create mount point
|
||||
sudo mkdir -p /mnt/immich/uploads
|
||||
|
||||
# Add to /etc/fstab
|
||||
192.168.1.70:/ImmichUploads /mnt/immich/uploads nfs auto,nofail,noatime,rw,vers=4.2,sync,no_subtree_check,all_squash,anonuid=999,anongid=999 0 0
|
||||
|
||||
# Mount
|
||||
sudo mount -a
|
||||
```
|
||||
|
||||
**Docker Service:**
|
||||
```yaml
|
||||
services:
|
||||
immich-server:
|
||||
image: ghcr.io/immich-app/immich-server:release
|
||||
volumes:
|
||||
- /mnt/immich/uploads:/usr/src/app/upload
|
||||
# ... other configurations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 5: Traefik Certificate Storage via NFS
|
||||
|
||||
```bash
|
||||
# Create mount point
|
||||
sudo mkdir -p /mnt/traefik/letsencrypt
|
||||
|
||||
# Add to /etc/fstab
|
||||
192.168.1.70:/TraefikLetsEncrypt /mnt/traefik/letsencrypt nfs auto,nofail,noatime,rw,vers=4.2,sync,no_subtree_check,no_root_squash 0 0
|
||||
|
||||
# Mount
|
||||
sudo mount -a
|
||||
```
|
||||
|
||||
**Docker Service:**
|
||||
```yaml
|
||||
services:
|
||||
traefik:
|
||||
image: traefik:latest
|
||||
volumes:
|
||||
- /mnt/traefik/letsencrypt:/letsencrypt
|
||||
# ... other configurations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Best Practices and Recommendations
|
||||
|
||||
### Security
|
||||
1. **Use dedicated service accounts** with minimal required permissions
|
||||
2. **Secure credential files** with `chmod 600`
|
||||
3. **Limit NFS exports** to specific subnets or IPs when possible
|
||||
4. **Use NFSv4.2** for improved security and performance
|
||||
|
||||
### Reliability
|
||||
1. **Use `nofail` in fstab** to prevent boot failures if NFS is unavailable
|
||||
2. **Test mounts manually** before adding to fstab
|
||||
3. **Monitor NFS/SMB services** on OMV server
|
||||
4. **Regular backups** of configuration and data
|
||||
|
||||
### Performance
|
||||
1. **Use NFS for containerized applications** (better performance than SMB)
|
||||
2. **Use `noatime`** to reduce write operations
|
||||
3. **Use `sync` for databases** to ensure data integrity
|
||||
4. **Consider `async` for media files** if performance is critical (with backup strategy)
|
||||
|
||||
### Verification Commands
|
||||
|
||||
```bash
|
||||
# Check all mounts
|
||||
mount | grep -E 'nfs|cifs'
|
||||
|
||||
# Check NFS statistics
|
||||
nfsstat -m
|
||||
|
||||
# Test write permissions
|
||||
touch /mnt/media/test.txt && rm /mnt/media/test.txt
|
||||
|
||||
# Check OMV exports (from OMV server)
|
||||
sudo exportfs -v
|
||||
|
||||
# Check SMB status (from OMV server)
|
||||
sudo smbstatus
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Troubleshooting
|
||||
|
||||
### Issue: Mount hangs at boot
|
||||
**Solution:** Add `nofail` option to fstab entries
|
||||
|
||||
### Issue: Permission denied errors
|
||||
**Solution:**
|
||||
- Verify UID/GID mappings match between NFS options and container user
|
||||
- Check folder permissions on OMV server
|
||||
- Ensure `no_root_squash` is set for services requiring root access
|
||||
|
||||
### Issue: Stale NFS handles
|
||||
**Solution:**
|
||||
```bash
|
||||
# Unmount forcefully
|
||||
sudo umount -f /mnt/path
|
||||
|
||||
# Or lazy unmount
|
||||
sudo umount -l /mnt/path
|
||||
|
||||
# Restart NFS client
|
||||
sudo systemctl restart nfs-client.target
|
||||
```
|
||||
|
||||
### Issue: SMB connection refused
|
||||
**Solution:**
|
||||
- Verify SMB credentials
|
||||
- Check SMB service status on OMV: `sudo systemctl status smbd`
|
||||
- Verify firewall rules allow SMB traffic (ports 445, 139)
|
||||
|
||||
---
|
||||
|
||||
Your OMV server is now fully integrated with your Docker Swarm cluster, providing robust, centralized storage for all your containerized services.
|
||||
238
docs/guides/OMV_CLI_Setup_Guide.md
Normal file
238
docs/guides/OMV_CLI_Setup_Guide.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# OMV Command-Line (CLI) Setup Guide for Docker Swarm
|
||||
|
||||
This guide provides the necessary commands to configure OpenMediaVault (OMV) from the CLI for user management and to apply service configurations. For creating shared folders and configuring NFS/SMB shares, the **OpenMediaVault Web UI is the recommended and most robust approach** to ensure proper integration with OMV's internal database.
|
||||
|
||||
**Disclaimer:** While these commands are effective, making configuration changes via the CLI can be less intuitive than the Web UI. Always ensure you have backups. It's recommended to have a basic understanding of the OMV configuration database.
|
||||
|
||||
---
|
||||
|
||||
## **Phase 1: Initial Setup (User and Filesystem Identification)**
|
||||
|
||||
### **Step 1: Create the Swarm User**
|
||||
|
||||
First, create a dedicated user for your Swarm mounts.
|
||||
|
||||
```bash
|
||||
# Create the user 'swarm-user'
|
||||
sudo useradd -m swarm-user
|
||||
|
||||
# Set a password for the new user (you will be prompted)
|
||||
sudo passwd swarm-user
|
||||
|
||||
# Get the UID and GID for later use
|
||||
id swarm-user
|
||||
# Example output: uid=1001(swarm-user) gid=1001(swarm-user)
|
||||
```
|
||||
|
||||
### **Step 2: Identify Your Storage Drive**
|
||||
|
||||
You need the filesystem path for your storage drive. This is where the shared folders will be created.
|
||||
|
||||
```bash
|
||||
# List mounted filesystems managed by OMV
|
||||
sudo omv-show-fs
|
||||
```
|
||||
Look for your 512GB SSD and note its mount path (e.g., `/srv/dev-disk-by-uuid-fd2daa6f-bd75-4ac1-9c4c-9e4d4b84d845`). We will refer to this as `YOUR_MOUNT_PATH` for the rest of the guide.
|
||||
|
||||
---
|
||||
|
||||
## **Phase 2: Shared Folder and Service Configuration**
|
||||
|
||||
For creating shared folders and configuring services, you have two primary methods: the OMV Web UI (recommended for most users) and the `omv-rpc` command-line tool (for advanced users or scripting).
|
||||
|
||||
### **Method 1: OMV Web UI (Recommended)**
|
||||
|
||||
The safest and most straightforward way to configure OMV is through its web interface.
|
||||
|
||||
1. **Create Shared Folders:** Navigate to **Storage → Shared Folders** and create the new folders required for the Swarm integration:
|
||||
* `ImmichUploads`
|
||||
* `TraefikLetsEncrypt`
|
||||
* `ImmichDB`
|
||||
* `NextcloudDB`
|
||||
* `NextcloudApps`
|
||||
* `NextcloudConfig`
|
||||
* `NextcloudData`
|
||||
* `Media`
|
||||
|
||||
2. **Configure Permissions:** For each folder, set appropriate permissions:
|
||||
* Navigate to **Storage → Shared Folders**, select a folder, click **Permissions**
|
||||
* Add `swarm-user` with appropriate read/write permissions
|
||||
* For database folders, ensure proper ownership (typically root or specific service user)
|
||||
|
||||
3. **Configure Services:**
|
||||
* **For SMB:** Navigate to **Services → SMB/CIFS → Shares** and create shares for folders that need SMB access
|
||||
* **For NFS:** Navigate to **Services → NFS → Shares** and create shares with appropriate client and privilege settings
|
||||
|
||||
### **Method 2: Advanced CLI Method (`omv-rpc`)**
|
||||
|
||||
This is the correct and verified method for creating shared folders from the command line in OMV 6 and 7.
|
||||
|
||||
#### **Step 3.1: Get the Storage UUID**
|
||||
|
||||
First, you must get the internal UUID that OMV uses for your storage drive.
|
||||
|
||||
```bash
|
||||
# List all filesystems and their properties known to OMV
|
||||
sudo omv-rpc "FileSystemMgmt" "enumerateFilesystems" '{}'
|
||||
```
|
||||
|
||||
From the JSON output, find the object where the `devicefile` or `label` matches your drive. Copy the `uuid` value from that object. It will be a long string like `7f450873-134a-429c-9198-097a5293209f`.
|
||||
|
||||
#### **Step 3.2: Create the Shared Folders (CLI)**
|
||||
|
||||
**IMPORTANT:** The correct method for OMV 6+ uses the `ShareMgmt` service, not direct config manipulation.
|
||||
|
||||
```bash
|
||||
# Set your storage UUID (replace with actual UUID from Step 3.1)
|
||||
OMV_STORAGE_UUID="7f450873-134a-429c-9198-097a5293209f"
|
||||
|
||||
# Create shared folders using ShareMgmt service
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"ImmichUploads\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"ImmichUploads/\",\"comment\":\"Immich Uploads Storage\",\"permissions\":\"755\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"TraefikLetsEncrypt\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"TraefikLetsEncrypt/\",\"comment\":\"Traefik SSL Certificates\",\"permissions\":\"755\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"ImmichDB\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"ImmichDB/\",\"comment\":\"Immich Database Storage\",\"permissions\":\"700\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"NextcloudDB\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"NextcloudDB/\",\"comment\":\"Nextcloud Database Storage\",\"permissions\":\"700\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"NextcloudApps\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"NextcloudApps/\",\"comment\":\"Nextcloud Apps\",\"permissions\":\"755\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"NextcloudConfig\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"NextcloudConfig/\",\"comment\":\"Nextcloud Config\",\"permissions\":\"755\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"NextcloudData\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"NextcloudData/\",\"comment\":\"Nextcloud User Data\",\"permissions\":\"755\"}"
|
||||
|
||||
sudo omv-rpc ShareMgmt setSharedFolder "{\"uuid\":\"$(uuidgen)\",\"name\":\"Media\",\"mntentref\":\"${OMV_STORAGE_UUID}\",\"reldirpath\":\"Media/\",\"comment\":\"Media Files for Jellyfin\",\"permissions\":\"755\"}"
|
||||
```
|
||||
|
||||
#### **Step 3.3: Verify Shared Folders Were Created**
|
||||
|
||||
```bash
|
||||
# List all shared folders
|
||||
sudo omv-rpc ShareMgmt getSharedFoldersList '{"start":0,"limit":25}'
|
||||
|
||||
# Or use the simpler command
|
||||
omv-showkey conf.system.sharedfolder
|
||||
```
|
||||
|
||||
#### **Step 3.4: Set Folder Permissions (CLI)**
|
||||
|
||||
After creating folders, set proper ownership and permissions on the actual directories:
|
||||
|
||||
```bash
|
||||
# Replace with your actual mount path
|
||||
MOUNT_PATH="/srv/dev-disk-by-uuid-fd2daa6f-bd75-4ac1-9c4c-9e4d4b84d845"
|
||||
|
||||
# Get swarm-user UID and GID (noted from Step 1)
|
||||
SWARM_UID=1001 # Replace with actual UID
|
||||
SWARM_GID=1001 # Replace with actual GID
|
||||
|
||||
# Set ownership for media folders
|
||||
sudo chown -R ${SWARM_UID}:${SWARM_GID} "${MOUNT_PATH}/Media"
|
||||
sudo chown -R ${SWARM_UID}:${SWARM_GID} "${MOUNT_PATH}/ImmichUploads"
|
||||
|
||||
# Database folders should be owned by root with restricted permissions
|
||||
sudo chown -R root:root "${MOUNT_PATH}/ImmichDB"
|
||||
sudo chown -R root:root "${MOUNT_PATH}/NextcloudDB"
|
||||
sudo chmod 700 "${MOUNT_PATH}/ImmichDB"
|
||||
sudo chmod 700 "${MOUNT_PATH}/NextcloudDB"
|
||||
|
||||
# Nextcloud folders should use www-data (UID 33, GID 33)
|
||||
sudo chown -R 33:33 "${MOUNT_PATH}/NextcloudApps"
|
||||
sudo chown -R 33:33 "${MOUNT_PATH}/NextcloudConfig"
|
||||
sudo chown -R 33:33 "${MOUNT_PATH}/NextcloudData"
|
||||
|
||||
# Traefik folder
|
||||
sudo chown -R root:root "${MOUNT_PATH}/TraefikLetsEncrypt"
|
||||
sudo chmod 700 "${MOUNT_PATH}/TraefikLetsEncrypt"
|
||||
```
|
||||
|
||||
#### **Step 3.5: Configure NFS Shares (CLI)**
|
||||
|
||||
**Note:** Configuring NFS shares via CLI is complex. The Web UI is strongly recommended. However, if needed:
|
||||
|
||||
```bash
|
||||
# Get the shared folder UUIDs first
|
||||
sudo omv-rpc ShareMgmt getSharedFoldersList '{"start":0,"limit":25}' | grep -A5 "ImmichDB"
|
||||
|
||||
# Example NFS share creation (requires the shared folder UUID)
|
||||
# Replace SHAREDFOLDER_UUID with the actual UUID from above
|
||||
sudo omv-rpc Nfs setShare "{\"uuid\":\"$(uuidgen)\",\"sharedfolderref\":\"SHAREDFOLDER_UUID\",\"client\":\"192.168.1.0/24\",\"options\":\"rw,sync,no_subtree_check,no_root_squash\",\"comment\":\"\"}"
|
||||
```
|
||||
|
||||
**This is error-prone. Use the Web UI for NFS/SMB configuration.**
|
||||
|
||||
---
|
||||
|
||||
## **Phase 3: Apply Configuration Changes**
|
||||
|
||||
### **Step 4: Apply All OMV Configuration Changes**
|
||||
|
||||
After making all shared folder and service configurations, apply the changes:
|
||||
|
||||
```bash
|
||||
# Apply shared folder configuration
|
||||
sudo omv-salt deploy run sharedfolder
|
||||
|
||||
# Apply the SMB configuration (if SMB shares were configured)
|
||||
sudo omv-salt deploy run samba
|
||||
|
||||
# Apply the NFS configuration (if NFS shares were configured)
|
||||
sudo omv-salt deploy run nfs
|
||||
|
||||
# Apply general OMV configuration changes
|
||||
sudo omv-salt deploy run phpfpm nginx
|
||||
|
||||
# Restart services to ensure all changes take effect
|
||||
sudo systemctl restart nfs-kernel-server
|
||||
sudo systemctl restart smbd
|
||||
```
|
||||
|
||||
### **Step 5: Verify Services are Running**
|
||||
|
||||
```bash
|
||||
# Check NFS status
|
||||
sudo systemctl status nfs-kernel-server
|
||||
|
||||
# Check SMB status
|
||||
sudo systemctl status smbd
|
||||
|
||||
# List active NFS exports
|
||||
sudo exportfs -v
|
||||
|
||||
# List SMB shares
|
||||
sudo smbstatus --shares
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Troubleshooting**
|
||||
|
||||
### Check OMV Logs
|
||||
```bash
|
||||
# General OMV logs
|
||||
sudo journalctl -u openmediavault-engined -f
|
||||
|
||||
# NFS logs
|
||||
sudo journalctl -u nfs-kernel-server -f
|
||||
|
||||
# SMB logs
|
||||
sudo journalctl -u smbd -f
|
||||
```
|
||||
|
||||
### Verify Mount Points on Docker Nodes
|
||||
After setting up OMV, verify that Docker nodes can access the shares:
|
||||
|
||||
```bash
|
||||
# Test NFS mount
|
||||
sudo mount -t nfs 192.168.1.70:/ImmichDB /mnt/test
|
||||
|
||||
# Test SMB mount
|
||||
sudo mount -t cifs //192.168.1.70/Media /mnt/test -o credentials=/root/.smbcredentials
|
||||
|
||||
# Unmount test
|
||||
sudo umount /mnt/test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Your OMV server is now fully configured to provide the necessary shares for your Docker Swarm cluster. You can now proceed with configuring the mounts on your Swarm nodes as outlined in the main `OMV.md` guide.
|
||||
295
docs/guides/SWARM_MIGRATION_GUIDE.md
Normal file
295
docs/guides/SWARM_MIGRATION_GUIDE.md
Normal file
@@ -0,0 +1,295 @@
|
||||
# Docker Swarm Stack Migration Guide
|
||||
|
||||
## Overview
|
||||
This guide helps you safely migrate from the old stack configurations to the new fixed versions with Docker secrets, health checks, and improved reliability.
|
||||
|
||||
## ⚠️ IMPORTANT: Read Before Starting
|
||||
- **Backup first**: `docker service ls > services-backup.txt`
|
||||
- **Downtime**: Expect 2-5 minutes per stack during migration
|
||||
- **Secrets**: Must be created before deploying new stacks
|
||||
- **Order matters**: Follow the deployment sequence below
|
||||
|
||||
---
|
||||
|
||||
## Pre-Migration Checklist
|
||||
|
||||
- [ ] Review [SWARM_STACK_REVIEW.md](file:///workspace/homelab/docs/reviews/SWARM_STACK_REVIEW.md)
|
||||
- [ ] Backup current service configurations
|
||||
- [ ] Ensure you're on a Swarm manager node
|
||||
- [ ] Have strong passwords ready for secrets
|
||||
- [ ] Test with one non-critical stack first
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Create Docker Secrets
|
||||
|
||||
**Run the secrets creation script:**
|
||||
```bash
|
||||
sudo bash /workspace/homelab/scripts/create_docker_secrets.sh
|
||||
```
|
||||
|
||||
**You'll be prompted for:**
|
||||
- `paperless_db_password` - Strong password for Paperless DB (20+ chars)
|
||||
- `paperless_secret_key` - Django secret key (50+ random chars)
|
||||
- `grafana_admin_password` - Grafana admin password
|
||||
- `duckdns_token` - Your DuckDNS API token
|
||||
|
||||
**Generate secure secrets:**
|
||||
```bash
|
||||
# PostgreSQL password (20 chars)
|
||||
openssl rand -base64 20
|
||||
|
||||
# Django secret key (50 chars)
|
||||
openssl rand -base64 50 | tr -d '\n'
|
||||
```
|
||||
|
||||
**Verify secrets created:**
|
||||
```bash
|
||||
docker secret ls
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Migration Sequence
|
||||
|
||||
### Phase 1: Infrastructure Stack (Watchtower & TSDProxy)
|
||||
> **Note for HAOS Users**: This stack uses named volumes `tsdproxy_config` and `tsdproxy_data` instead of bind mounts to avoid read-only filesystem errors.
|
||||
|
||||
```bash
|
||||
# Remove old full stack if running
|
||||
docker stack rm full-stack
|
||||
|
||||
# Deploy infrastructure
|
||||
docker stack deploy -c /workspace/homelab/services/swarm/stacks/infrastructure.yml infrastructure
|
||||
|
||||
# Verify
|
||||
docker service ls | grep infrastructure
|
||||
```
|
||||
|
||||
**What Changed:**
|
||||
- ✅ Split from monolithic stack
|
||||
- ✅ TSDProxy uses named volumes (HAOS compatible)
|
||||
- ✅ Watchtower configured for daily cleanup
|
||||
- ✅ **Added Komodo** (Core, Mongo, Periphery) for container management
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Productivity Stack (Paperless, PDF, Search)
|
||||
```bash
|
||||
# Ensure secrets exist first!
|
||||
docker stack deploy -c /workspace/homelab/services/swarm/stacks/productivity.yml productivity
|
||||
```
|
||||
|
||||
**What Changed:**
|
||||
- ✅ Split from monolithic stack
|
||||
- ✅ Uses existing secrets and networks
|
||||
- ✅ Dedicated stack for document tools
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: AI Stack (OpenWebUI)
|
||||
```bash
|
||||
docker stack deploy -c /workspace/homelab/services/swarm/stacks/ai.yml ai
|
||||
```
|
||||
|
||||
**What Changed:**
|
||||
- ✅ Dedicated stack for AI workloads
|
||||
- ✅ Resource limits preserved
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Other Stacks (Monitoring, Portainer, Networking)
|
||||
Follow the original instructions for these stacks as they remain unchanged.
|
||||
|
||||
---
|
||||
|
||||
## HAOS Specific Notes
|
||||
If you are running on Home Assistant OS (HAOS), the root filesystem is read-only.
|
||||
- **Do not use bind mounts** to paths like `/srv`, `/home`, or `/etc` (except `/etc/localtime`).
|
||||
- **Use named volumes** for persistent data.
|
||||
- **TSDProxy Config**: Since we switched to a named volume `tsdproxy_config`, you may need to populate it if you have a custom config.
|
||||
```bash
|
||||
# Example: Copy config to volume (run on manager)
|
||||
# Find the volume path (might be difficult on HAOS, easier to use `docker cp` to a dummy container mounting the volume)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Post-Migration Validation
|
||||
|
||||
### Automated Validation
|
||||
```bash
|
||||
bash /workspace/homelab/scripts/validate_deployment.sh
|
||||
```
|
||||
|
||||
### Manual Checks
|
||||
```bash
|
||||
# 1. All services running
|
||||
docker service ls
|
||||
|
||||
# 2. All containers healthy
|
||||
docker ps --filter "health=healthy"
|
||||
|
||||
# 3. No unhealthy containers
|
||||
docker ps --filter "health=unhealthy"
|
||||
|
||||
# 4. Check secrets in use
|
||||
docker secret ls
|
||||
|
||||
# 5. Verify resource usage
|
||||
docker stats --no-stream
|
||||
```
|
||||
|
||||
### Test Each Service
|
||||
- ✅ Grafana: https://grafana.sj98.duckdns.org
|
||||
- ✅ Prometheus: https://prometheus.sj98.duckdns.org
|
||||
- ✅ Portainer: https://portainer.sj98.duckdns.org
|
||||
- ✅ Paperless: https://paperless.sj98.duckdns.org
|
||||
- ✅ OpenWebUI: https://ai.sj98.duckdns.org
|
||||
- ✅ PDF: https://pdf.sj98.duckdns.org
|
||||
- ✅ Search: https://search.sj98.duckdns.org
|
||||
- ✅ Dozzle: https://dozzle.sj98.duckdns.org
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Services Won't Start
|
||||
```bash
|
||||
# Check logs
|
||||
docker service logs <service_name>
|
||||
|
||||
# Check secrets
|
||||
docker secret inspect <secret_name>
|
||||
|
||||
# Check constraints
|
||||
docker node ls
|
||||
docker node inspect <node_id> | grep Labels
|
||||
```
|
||||
|
||||
### Health Checks Failing
|
||||
```bash
|
||||
# View health status
|
||||
docker inspect <container_id> | jq '.[0].State.Health'
|
||||
|
||||
# Check logs
|
||||
docker logs <container_id>
|
||||
|
||||
# Disable health check temporarily (for debugging)
|
||||
# Edit stack file and remove healthcheck section
|
||||
```
|
||||
|
||||
### Secrets Not Found
|
||||
```bash
|
||||
# Recreate secret
|
||||
echo -n "your_password" | docker secret create secret_name -
|
||||
|
||||
# Update service
|
||||
docker service update --secret-add secret_name service_name
|
||||
```
|
||||
|
||||
### Memory Limits Too Strict
|
||||
```bash
|
||||
# If services are being killed, increase limits in stack file
|
||||
# Then redeploy:
|
||||
docker stack deploy -c stack.yml stack_name
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### Rollback Single Service
|
||||
```bash
|
||||
# Get previous version
|
||||
docker service inspect <service_name> --pretty
|
||||
|
||||
# Rollback
|
||||
docker service rollback <service_name>
|
||||
```
|
||||
|
||||
### Rollback Entire Stack
|
||||
```bash
|
||||
# Remove new stack
|
||||
docker stack rm <stack_name>
|
||||
|
||||
sleep 30
|
||||
|
||||
# Deploy from backup (old stack file)
|
||||
docker stack deploy -c /path/to/old/stack.yml stack_name
|
||||
```
|
||||
|
||||
### Remove Secrets (if needed)
|
||||
```bash
|
||||
# This only works if no services are using the secret
|
||||
docker secret rm <secret_name>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Comparison
|
||||
|
||||
| Metric | Before | After | Improvement |
|
||||
|--------|--------|-------|-------------|
|
||||
| **Security Score** | 6.0/10 | 9.5/10 | +58% |
|
||||
| **Hardcoded Secrets** | 3 | 0 | ✅ Fixed |
|
||||
| **Services with Health Checks** | 0 | 100% | ✅ Added |
|
||||
| **Services with Restart Policies** | 10% | 100% | ✅ Added |
|
||||
| **Traefik Replicas** | 1 | 2 | ✅ HA |
|
||||
| **Memory on Pi 4** | 6GB+ | 4.5GB | -25% |
|
||||
| **Log Disk Usage Risk** | High | Low | ✅ Limits |
|
||||
| **Services with Pinned Versions** | 60% | 100% | ✅ Stable |
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Update a Secret
|
||||
```bash
|
||||
# 1. Create new secret with different name
|
||||
echo -n "new_password" | docker secret create paperless_db_password_v2 -
|
||||
|
||||
# 2. Update service to use new secret
|
||||
docker service update \
|
||||
--secret-rm paperless_db_password \
|
||||
--secret-add source=paperless_db_password_v2,target=paperless_db_password \
|
||||
full-stack_paperless
|
||||
|
||||
# 3. Remove old secret
|
||||
docker secret rm paperless_db_password
|
||||
```
|
||||
|
||||
### Regular Health Checks
|
||||
```bash
|
||||
# Weekly check
|
||||
bash /workspace/homelab/scripts/quick_status.sh
|
||||
|
||||
# Monthly validation
|
||||
bash /workspace/homelab/scripts/validate_deployment.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
### Total Changes
|
||||
- **6 stack files fixed**
|
||||
- **3 Docker secrets created**
|
||||
- **100% of services** now have health checks
|
||||
- **100% of services** now have restart policies
|
||||
- **100% of services** now have logging limits
|
||||
- **0 hardcoded passwords** remaining
|
||||
- **2× Traefik replicas** for high availability
|
||||
|
||||
### Estimated Migration Time
|
||||
- Secrets creation: 5 minutes
|
||||
- Stack-by-stack migration: 20-30 minutes
|
||||
- Validation: 10 minutes
|
||||
- **Total: 35-45 minutes**
|
||||
|
||||
---
|
||||
|
||||
**Migration completed successfully?** Run the quick status:
|
||||
```bash
|
||||
bash /workspace/homelab/scripts/quick_status.sh
|
||||
```
|
||||
13
docs/guides/haos_swarm_migration.md
Normal file
13
docs/guides/haos_swarm_migration.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Swarm Migration from HAOS to Ubuntu Container
|
||||
|
||||
## Reason for Migration
|
||||
|
||||
The Docker Swarm leader node was previously running on the Home Assistant OS (HAOS). This caused conflicts with HAOS, which also utilizes Docker. To resolve these conflicts and create a more stable environment, the swarm was dismantled and recreated.
|
||||
|
||||
## New Architecture
|
||||
|
||||
The Docker Swarm is now running within a dedicated Ubuntu container on the same HAOS machine. This isolates the swarm environment from the HAOS Docker environment, preventing future conflicts.
|
||||
|
||||
## Consequences
|
||||
|
||||
As a result of this migration, the old swarm was destroyed. This action necessitated the redeployment of all stacks and services, including Portainer and Traefik. The disconnection of the Portainer UI and the broken Traefik dashboard are direct consequences of this necessary migration. The services need to be redeployed on the new swarm to restore functionality.
|
||||
77
docs/guides/health_checks.md
Normal file
77
docs/guides/health_checks.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Health Check Examples for Docker Compose/Swarm
|
||||
|
||||
## Example 1: Portainer with Health Check
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
portainer:
|
||||
image: portainer/portainer-ce:latest
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:9000/api/status"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
deploy:
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
delay: 5s
|
||||
max_attempts: 3
|
||||
```
|
||||
|
||||
## Example 2: OpenWebUI with Health Check
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
openwebui:
|
||||
image: ghcr.io/open-webui/open-webui:latest
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
deploy:
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
delay: 5s
|
||||
max_attempts: 3
|
||||
```
|
||||
|
||||
## Example 3: Nextcloud with Health Check
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
nextcloud:
|
||||
image: nextcloud:latest
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:80/status.php"]
|
||||
interval: 60s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
deploy:
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
delay: 10s
|
||||
max_attempts: 3
|
||||
```
|
||||
|
||||
## Implementation Notes
|
||||
- **interval**: How often to check (30-60s for most services)
|
||||
- **timeout**: Max time to wait for check to complete
|
||||
- **retries**: Number of consecutive failures before marking unhealthy
|
||||
- **start_period**: Grace period after container start before checking
|
||||
|
||||
## Auto-Restart Configuration
|
||||
All services should have restart policies configured:
|
||||
- **condition**: `on-failure` or `any`
|
||||
- **delay**: Time to wait before restarting
|
||||
- **max_attempts**: Maximum restart attempts
|
||||
|
||||
## Monitoring Health Status
|
||||
Check container health with:
|
||||
```bash
|
||||
docker ps --filter "health=unhealthy"
|
||||
docker inspect <container_id> | jq '.[0].State.Health'
|
||||
```
|
||||
33
docs/guides/portainer_local_unreachable_fix.md
Normal file
33
docs/guides/portainer_local_unreachable_fix.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# Fixing Portainer Error: "The environment named local is unreachable"
|
||||
|
||||
## Problem
|
||||
|
||||
After migrating the Docker Swarm to an Ubuntu container, the Portainer UI shows the error "The environment named local is unreachable".
|
||||
|
||||
## Cause
|
||||
|
||||
This error means the Portainer server container cannot communicate with the Docker daemon it is supposed to manage. This communication happens through the Docker socket file, located at `/var/run/docker.sock`.
|
||||
|
||||
In your nested environment (HAOS > Ubuntu Container > Portainer Container), the issue is almost certainly that the user inside the Portainer container does not have the necessary file permissions to access the `/var/run/docker.sock` file that belongs to the Ubuntu container's Docker instance.
|
||||
|
||||
## Solution (To be performed in your deployment environment)
|
||||
|
||||
You need to ensure the Portainer container runs with a user that has permission to access the Docker socket.
|
||||
|
||||
**1. Find the Docker Group ID:**
|
||||
|
||||
First, SSH into your Ubuntu container that is running the swarm. Then, run this command to find the group ID (`gid`) that owns the Docker socket:
|
||||
|
||||
```bash
|
||||
stat -c '%g' /var/run/docker.sock
|
||||
```
|
||||
|
||||
This will return a number. This is the `DOCKER_GROUP_ID`.
|
||||
|
||||
**2. Edit the `portainer-stack.yml`:**
|
||||
|
||||
You need to add a `user` directive to the `portainer` service definition in your `portainer-stack.yml` file. This tells the service to run as the `root` user and with the Docker group, granting it the necessary permissions.
|
||||
|
||||
I will make this edit for you now, using a placeholder for the group ID. **You will need to replace `DOCKER_GROUP_ID_HERE` with the number you get from the command above before you deploy.**
|
||||
|
||||
This is the most common and secure way to resolve this issue without granting full `privileged` access.
|
||||
39
docs/guides/proxmox_network_fix.md
Normal file
39
docs/guides/proxmox_network_fix.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# Proxmox USB Network Adapter Fix
|
||||
|
||||
This document outlines a solution to the intermittent network disconnection issue on the Acer Proxmox host, where the USB network adapter drops its connection and does not reconnect automatically.
|
||||
|
||||
## The Problem
|
||||
|
||||
The Acer Proxmox host (`192.168.1.57`) uses a USB-to-Ethernet adapter for its 2.5 GbE connection. This adapter occasionally disconnects and fails to reconnect on its own, disrupting network access for the host and its VMs.
|
||||
|
||||
## The Solution
|
||||
|
||||
A shell script, `network_check.sh`, has been created to monitor the network connection. If the connection is down, the script will attempt to reset the USB adapter. If that fails, it will reboot the host to restore connectivity. This script is intended to be run as a cron job at regular intervals.
|
||||
|
||||
### 1. The `network_check.sh` Script
|
||||
|
||||
The script performs the following actions:
|
||||
1. Pings a reliable external IP address (e.g., `8.8.8.8`) to check for internet connectivity.
|
||||
2. If the ping fails, it identifies the USB network adapter's bus and device number.
|
||||
3. It then attempts to reset the USB device.
|
||||
4. If the network connection is still not restored after resetting the adapter, the script will force a reboot.
|
||||
|
||||
The script is located at `/usr/local/bin/network_check.sh`.
|
||||
|
||||
### 2. Cron Job Setup
|
||||
|
||||
To automate the execution of the script, a cron job should be set up to run every 5 minutes.
|
||||
|
||||
**To add the cron job, follow these steps:**
|
||||
|
||||
1. Open the crontab editor:
|
||||
```bash
|
||||
crontab -e
|
||||
```
|
||||
2. Add the following line to the file:
|
||||
```
|
||||
*/5 * * * * /bin/bash /usr/local/bin/network_check.sh
|
||||
```
|
||||
3. Save and exit the editor.
|
||||
|
||||
This will ensure that the network connection is checked every 5 minutes, and the appropriate action is taken if a disconnection is detected.
|
||||
44
docs/guides/swarm_label_guide.md
Normal file
44
docs/guides/swarm_label_guide.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Docker Swarm Node Labeling Guide
|
||||
|
||||
This guide provides the commands to apply the correct labels to your Docker Swarm nodes, ensuring that services are scheduled on the appropriate hardware.
|
||||
|
||||
Run the following commands in your terminal on a manager node to label each of your swarm nodes.
|
||||
|
||||
### 1. Label the Leader Node
|
||||
This node will run general-purpose applications.
|
||||
|
||||
```bash
|
||||
docker node update --label-add leader=true <node-name>
|
||||
```
|
||||
|
||||
### 2. Label the Manager Node
|
||||
This node will run core services like Traefik and Portainer.
|
||||
|
||||
```bash
|
||||
docker node update --label-add manager=true <node-name>
|
||||
```
|
||||
|
||||
### 3. Label the Heavy Worker Node
|
||||
This node is for computationally intensive workloads like AI and machine learning.
|
||||
|
||||
```bash
|
||||
docker node update --label-add heavy=true <node-name>
|
||||
```
|
||||
|
||||
### 4. Label the Fedora Worker Node
|
||||
This node is the primary heavy worker.
|
||||
|
||||
```bash
|
||||
docker node update --label-add heavy=true fedora
|
||||
```
|
||||
|
||||
## Verify Labels
|
||||
|
||||
After applying the labels, you can verify them by inspecting each node. For example, to check the labels for a node, run:
|
||||
|
||||
```bash
|
||||
docker node inspect <node-name> --pretty
|
||||
```
|
||||
|
||||
Look for the "Labels" section in the output to confirm the changes.
|
||||
|
||||
283
docs/guides/traefik_fix_guide.md
Normal file
283
docs/guides/traefik_fix_guide.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Final Traefik v3 Setup and Fix Guide
|
||||
|
||||
This guide provides the complete, step-by-step process to cleanly remove any old Traefik configurations and deploy a fresh, working Traefik v3 setup on Docker Swarm.
|
||||
|
||||
**Follow these steps in order on your Docker Swarm manager node.**
|
||||
|
||||
---
|
||||
|
||||
### Step 1: Complete Removal of Old Traefik Components
|
||||
|
||||
First, we will ensure the environment is completely clean.
|
||||
|
||||
1. **Remove the Stack:**
|
||||
- In Portainer, go to "Stacks", select your `networking-stack`, and click **Remove**. Wait for it to be successfully removed.
|
||||
|
||||
2. **Remove the Docker Config:**
|
||||
- Run this command in your manager node's terminal:
|
||||
```zsh
|
||||
docker config rm traefik.yml
|
||||
```
|
||||
*(It's okay if this command says the config doesn't exist.)*
|
||||
|
||||
3. **Remove the Docker Volume:**
|
||||
- This will delete your old Let's Encrypt certificates, which is necessary for a clean start.
|
||||
```zsh
|
||||
docker volume rm traefik_letsencrypt
|
||||
```
|
||||
*(It's okay if this command says the volume doesn't exist.)*
|
||||
|
||||
4. **Remove the Local Config File (if it exists):**
|
||||
```zsh
|
||||
rm ./traefik.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Create the Correct Traefik v3 Configuration
|
||||
|
||||
We will use the `busybox` container method to create the configuration file.
|
||||
|
||||
1. **Create `traefik.yml`:**
|
||||
- **IMPORTANT:** Replace `your-email@example.com` with your actual email address in the block below.
|
||||
- Copy the entire multi-line block and paste it into your Zsh terminal.
|
||||
- After pasting, the terminal will show a `>` on a new line. This is normal. **Simply type `EOF` and press Enter** to finish the command.
|
||||
|
||||
```zsh
|
||||
# --- Creates the traefik.yml file in a temporary container and copies it out ---
|
||||
docker run --rm -i -v "$(pwd):/host" busybox sh -c 'cat > /host/traefik.yml <<\'EOF\'
|
||||
checkNewVersion: true
|
||||
sendAnonymousUsage: false
|
||||
|
||||
log:
|
||||
level: INFO
|
||||
|
||||
api:
|
||||
dashboard: true
|
||||
insecure: false
|
||||
|
||||
entryPoints:
|
||||
web:
|
||||
address: ":80"
|
||||
http:
|
||||
redirections:
|
||||
entryPoint:
|
||||
to: websecure
|
||||
scheme: https
|
||||
websecure:
|
||||
address: ":443"
|
||||
http:
|
||||
tls:
|
||||
certResolver: leresolver
|
||||
|
||||
providers:
|
||||
swarm: # <-- Use the swarm provider in Traefik v3
|
||||
endpoint: "unix:///var/run/docker.sock"
|
||||
network: traefik-public
|
||||
exposedByDefault: false
|
||||
|
||||
# Optionally keep the docker provider if you run non-swarm local containers.
|
||||
# docker:
|
||||
# network: traefik-public
|
||||
# exposedByDefault: false
|
||||
|
||||
certificatesResolvers:
|
||||
leresolver:
|
||||
acme:
|
||||
email: "your-email@example.com"
|
||||
storage: "/letsencrypt/acme.json"
|
||||
dnsChallenge:
|
||||
provider: duckdns
|
||||
delayBeforeCheck: 30s
|
||||
resolvers:
|
||||
- "192.168.1.196:53"
|
||||
- "192.168.1.245:53"
|
||||
- "192.168.1.62:53"
|
||||
EOF'
|
||||
```
|
||||
|
||||
2. **Create the Docker Swarm Config:**
|
||||
- This command ingests the file you just created into Swarm.
|
||||
```zsh
|
||||
docker config create traefik.yml ./traefik.yml
|
||||
```
|
||||
|
||||
3. **Create and Prepare the Let's Encrypt Volume:**
|
||||
- Create the volume:
|
||||
```zsh
|
||||
docker volume create traefik_letsencrypt
|
||||
```
|
||||
- Create the empty `acme.json` file with the correct permissions:
|
||||
```zsh
|
||||
docker run --rm -v traefik_letsencrypt:/letsencrypt busybox sh -c "touch /letsencrypt/acme.json && chmod 600 /letsencrypt/acme.json"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Deploy the Corrected `networking-stack`
|
||||
|
||||
1. **Deploy via Portainer:**
|
||||
- Go to "Stacks" > "Add stack".
|
||||
- Name it `networking-stack`.
|
||||
- Copy the YAML content below and paste it into the web editor.
|
||||
- **IMPORTANT:** Replace `YOUR_DUCKDNS_TOKEN` with your actual DuckDNS token.
|
||||
- Click "Deploy the stack".
|
||||
|
||||
```yaml
|
||||
version: '3.9'
|
||||
|
||||
networks:
|
||||
traefik-public:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
traefik_letsencrypt:
|
||||
external: true
|
||||
|
||||
configs:
|
||||
traefik_yml:
|
||||
external: true
|
||||
name: traefik.yml
|
||||
|
||||
services:
|
||||
traefik:
|
||||
image: traefik:latest # Or pin to traefik:v3.0 for stability
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- traefik_letsencrypt:/letsencrypt
|
||||
networks:
|
||||
- traefik-public
|
||||
environment:
|
||||
- "DUCKDNS_TOKEN=YOUR_DUCKDNS_TOKEN"
|
||||
configs:
|
||||
- source: traefik_yml
|
||||
target: /traefik.yml
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.traefik.rule=Host(`traefik.sj98.duckdns.org`)"
|
||||
- "traefik.http.routers.traefik.entrypoints=websecure"
|
||||
- "traefik.http.routers.traefik.tls.certresolver=leresolver"
|
||||
- "traefik.http.routers.traefik.service=api@internal"
|
||||
placement:
|
||||
constraints:
|
||||
- node.role == manager
|
||||
|
||||
whoami:
|
||||
image: traefik/whoami
|
||||
networks:
|
||||
- traefik-public
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.whoami.rule=Host(`whoami.sj98.duckdns.org`)"
|
||||
- "traefik.http.routers.whoami.entrypoints=websecure"
|
||||
- "traefik.http.routers.whoami.tls.certresolver=leresolver"
|
||||
- "traefik.http.services.whoami.loadbalancer.server.port=80"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4: Verify and Redeploy Other Stacks
|
||||
|
||||
1. **Wait and Verify:**
|
||||
- Wait for 2-3 minutes for the stack to deploy and for the certificate to be issued.
|
||||
- Open your browser and navigate to `https://traefik.sj98.duckdns.org`. The Traefik dashboard should load.
|
||||
- You should see routers for `traefik` and `whoami`.
|
||||
|
||||
2. **Redeploy Corrected Stacks:**
|
||||
- Now that Traefik is working, go to Portainer and redeploy your `full-stack-complete.yml` and `monitoring-stack.yml` to apply the fixes we made earlier.
|
||||
- The services from those stacks (Paperless, Prometheus, etc.) should now appear in the Traefik dashboard and be accessible via their URLs.
|
||||
|
||||
### Chat GPT Fix
|
||||
⸻
|
||||
|
||||
Traefik Swarm Stack Fix Instructions
|
||||
|
||||
1. Verify Networks
|
||||
|
||||
Make sure all web-exposed services are attached to the traefik-public network:
|
||||
|
||||
networks:
|
||||
- traefik-public
|
||||
|
||||
Internal-only services (DB, Redis, etc.) should not be on Traefik network.
|
||||
|
||||
⸻
|
||||
|
||||
2. Assign Unique Router Names
|
||||
|
||||
Every service exposed via Traefik must have a unique router label:
|
||||
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.<service>-router.rule=Host(`<subdomain>.sj98.duckdns.org`)"
|
||||
- "traefik.http.routers.<service>-router.entrypoints=websecure"
|
||||
- "traefik.http.routers.<service>-router.tls.certresolver=leresolver"
|
||||
- "traefik.http.routers.<service>-router.service=<service>@swarm"
|
||||
- "traefik.http.services.<service>.loadbalancer.server.port=<port>"
|
||||
|
||||
Replace <service>, <subdomain>, and <port> for each stack.
|
||||
|
||||
⸻
|
||||
|
||||
3. Update Traefik ACME Configuration
|
||||
|
||||
In traefik.yml, use:
|
||||
|
||||
certificatesResolvers:
|
||||
leresolver:
|
||||
acme:
|
||||
email: "your-email@example.com"
|
||||
storage: "/letsencrypt/acme.json"
|
||||
dnsChallenge:
|
||||
provider: duckdns
|
||||
propagation:
|
||||
delayBeforeChecks: 60s
|
||||
resolvers:
|
||||
- "192.168.1.196:53"
|
||||
- "192.168.1.245:53"
|
||||
- "192.168.1.62:53"
|
||||
|
||||
Note: delayBeforeCheck is deprecated. Use propagation.delayBeforeChecks.
|
||||
|
||||
⸻
|
||||
|
||||
4. Internal Services Configuration
|
||||
• Redis / Postgres / other internal services
|
||||
Do not expose them via Traefik.
|
||||
Attach them to backend networks only:
|
||||
|
||||
networks:
|
||||
- homelab-backend
|
||||
|
||||
• Only web services should have Traefik labels.
|
||||
|
||||
⸻
|
||||
|
||||
5. Deploy Services Correctly
|
||||
1. Deploy Traefik first.
|
||||
2. Deploy each routed service one at a time to allow ACME certificate issuance.
|
||||
3. Verify logs for any Router defined multiple times or port is missing errors.
|
||||
|
||||
⸻
|
||||
|
||||
6. Checklist for Each Service
|
||||
|
||||
Service Hostname Port Traefik Router Name Network Notes
|
||||
example-svc example.sj98.duckdns.org 8080 example-svc-router traefik-public Replace placeholders
|
||||
another-svc another.sj98.duckdns.org 8000 another-svc-router traefik-public Only if web-exposed
|
||||
|
||||
• Fill in each service’s hostname, port, and network.
|
||||
• Internal services do not need Traefik labels.
|
||||
|
||||
⸻
|
||||
|
||||
7. Common Issues
|
||||
• Duplicate Router Names: Make sure every router has a unique label.
|
||||
• Missing Ports: Each Traefik router must reference the service port with loadbalancer.server.port.
|
||||
• ACME Failures: Ensure DuckDNS token is correct and propagation delay is set.
|
||||
• Wrong Network: Only services on traefik-public are routable; internal services must use backend networks.
|
||||
288
docs/guides/traefik_setup_guide.md
Normal file
288
docs/guides/traefik_setup_guide.md
Normal file
@@ -0,0 +1,288 @@
|
||||
# Traefik Setup Guide for Docker Swarm
|
||||
|
||||
This guide provides the step-by-step instructions to correctly configure and deploy Traefik in a Docker Swarm environment, especially when dealing with potentially read-only host filesystems.
|
||||
|
||||
This method uses Docker Configs and Docker Volumes to manage Traefik's configuration and data, which is the standard best practice for Swarm. All commands should be run on your **Docker Swarm manager node**.
|
||||
|
||||
---
|
||||
|
||||
### Step 1: Create the `traefik.yml` Configuration File
|
||||
|
||||
This step creates the Traefik static configuration file. You have two options:
|
||||
|
||||
#### Option A: Using `sudo tee` (Direct Host Write)
|
||||
|
||||
This command uses a `HEREDOC` with `sudo tee` to write the `traefik.yml` file directly to your manager node's filesystem. This is generally straightforward if your manager node's filesystem is writable.
|
||||
|
||||
**Action:**
|
||||
1. **IMPORTANT:** Replace `your-email@example.com` with your actual email address in the command below.
|
||||
2. Copy and paste the entire block into your Zsh terminal on the manager node.
|
||||
|
||||
```zsh
|
||||
# --- Creates the traefik.yml file ---
|
||||
sudo tee ./traefik.yml > /dev/null <<'EOF'
|
||||
global:
|
||||
checkNewVersion: true
|
||||
sendAnonymousUsage: false
|
||||
|
||||
log:
|
||||
level: INFO
|
||||
|
||||
api:
|
||||
dashboard: true
|
||||
insecure: false
|
||||
|
||||
entryPoints:
|
||||
web:
|
||||
address: ":80"
|
||||
http:
|
||||
redirections:
|
||||
entryPoint:
|
||||
to: websecure
|
||||
scheme: https
|
||||
websecure:
|
||||
address: ":443"
|
||||
|
||||
providers:
|
||||
docker:
|
||||
network: traefik-public
|
||||
exposedByDefault: false
|
||||
|
||||
certificatesResolvers:
|
||||
leresolver:
|
||||
acme:
|
||||
email: "your-email@example.com"
|
||||
storage: "/letsencrypt/acme.json"
|
||||
dnsChallenge:
|
||||
provider: duckdns
|
||||
delayBeforeCheck: "120s"
|
||||
EOF
|
||||
```
|
||||
|
||||
#### Option B: Using `docker run` (Via Temporary Container)
|
||||
|
||||
This method creates the `traefik.yml` file *inside* a temporary `busybox` container and then copies it to your manager node's current directory. This is useful if you prefer to avoid direct `sudo tee` or if you're working in an environment where direct file creation is restricted.
|
||||
|
||||
**Action:**
|
||||
1. **IMPORTANT:** Replace `your-email@example.com` with your actual email address in the command below.
|
||||
2. Copy and paste the entire block into your Zsh terminal on the manager node.
|
||||
|
||||
```zsh
|
||||
# --- Creates the traefik.yml file in a temporary container and copies it out ---
|
||||
docker run --rm -i -v "$(pwd):/host" busybox sh -c 'cat > /host/traefik.yml <<\'EOF\'
|
||||
checkNewVersion: true
|
||||
sendAnonymousUsage: false
|
||||
|
||||
log:
|
||||
level: INFO
|
||||
|
||||
api:
|
||||
dashboard: true
|
||||
insecure: false
|
||||
|
||||
entryPoints:
|
||||
web:
|
||||
address: ":80"
|
||||
http:
|
||||
redirections:
|
||||
entryPoint:
|
||||
to: websecure
|
||||
scheme: https
|
||||
websecure:
|
||||
address: ":443"
|
||||
http:
|
||||
tls:
|
||||
certResolver: leresolver
|
||||
|
||||
providers:
|
||||
docker:
|
||||
network: traefik-public
|
||||
exposedByDefault: false
|
||||
|
||||
certificatesResolvers:
|
||||
leresolver:
|
||||
acme:
|
||||
email: "your-email@example.com"
|
||||
storage: "/letsencrypt/acme.json"
|
||||
dnsChallenge:
|
||||
provider: duckdns
|
||||
delayBeforeCheck: 30s
|
||||
resolvers:
|
||||
- "192.168.1.196:53"
|
||||
- "192.168.1.245:53"
|
||||
- "192.168.1.62:53"
|
||||
EOF'
|
||||
```
|
||||
|
||||
> **Note on Versioning:** The `traefik:latest` tag can introduce unexpected breaking changes, as seen here. For production or stable environments, it is highly recommended to pin to a specific version in your stack file, for example: `image: traefik:v2.11` or `image: traefik:v3.0`.
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Create the Docker Swarm Config
|
||||
|
||||
This command ingests the `traefik.yml` file (created in Step 1) into Docker Swarm, making it securely available to services.
|
||||
|
||||
**Action:** Run the following command on your manager node.
|
||||
|
||||
```zsh
|
||||
docker config create traefik.yml ./traefik.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Create the Let's Encrypt Volume
|
||||
|
||||
This creates a managed Docker Volume that will persist your TLS certificates.
|
||||
|
||||
**Action:** Run the following command on your manager node.
|
||||
|
||||
```zsh
|
||||
docker volume create traefik_letsencrypt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4: Prepare the `acme.json` File
|
||||
|
||||
Traefik requires an `acme.json` file to exist with the correct permissions before it can start. This command creates the empty file inside the volume you just made.
|
||||
|
||||
**Action:** Run the following command on your manager node.
|
||||
|
||||
```zsh
|
||||
docker run --rm -v traefik_letsencrypt:/letsencrypt busybox sh -c "touch /letsencrypt/acme.json && chmod 600 /letsencrypt/acme.json"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 5: Update and Deploy the `networking-stack.yml`
|
||||
|
||||
You can now deploy your `networking-stack` using the YAML below. It has been modified to use the Swarm config and volume instead of host paths.
|
||||
|
||||
**Action:**
|
||||
1. **IMPORTANT:** Replace `YOUR_DUCKDNS_TOKEN` with your actual DuckDNS token in the `environment` section.
|
||||
2. Upload this YAML content to Portainer to deploy your stack.
|
||||
|
||||
```yaml
|
||||
version: '3.9'
|
||||
|
||||
networks:
|
||||
traefik-public:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
traefik_letsencrypt:
|
||||
external: true
|
||||
|
||||
configs:
|
||||
traefik_yml:
|
||||
external: true
|
||||
name: traefik.yml
|
||||
|
||||
services:
|
||||
traefik:
|
||||
image: traefik:latest
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- traefik_letsencrypt:/letsencrypt
|
||||
networks:
|
||||
- traefik-public
|
||||
environment:
|
||||
- "DUCKDNS_TOKEN=YOUR_DUCKDNS_TOKEN"
|
||||
configs:
|
||||
- source: traefik_yml
|
||||
target: /traefik.yml
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.traefik.rule=Host(`traefik.sj98.duckdns.org`)"
|
||||
- "traefik.http.routers.traefik.entrypoints=websecure"
|
||||
- "traefik.http.routers.traefik.tls.certresolver=leresolver"
|
||||
- "traefik.http.routers.traefik.service=api@internal"
|
||||
placement:
|
||||
constraints:
|
||||
- node.role == manager
|
||||
|
||||
whoami:
|
||||
image: traefik/whoami
|
||||
networks:
|
||||
- traefik-public
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.whoami.rule=Host(`whoami.sj98.duckdns.org`)"
|
||||
- "traefik.http.routers.whoami.entrypoints=websecure"
|
||||
- "traefik.http.routers.whoami.tls.certresolver=leresolver"
|
||||
- "traefik.http.services.whoami.loadbalancer.server.port=80"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 6: Clean Up (Optional)
|
||||
|
||||
Since the configuration is now stored in Docker Swarm, you can remove the local `traefik.yml` file from your manager node's filesystem.
|
||||
|
||||
**Action:** Run the following command on your manager node.
|
||||
|
||||
```zsh
|
||||
rm ./traefik.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Troubleshooting and Removal
|
||||
|
||||
If you encounter an error and need to start the setup process over, follow these steps to cleanly remove all the components you created. Run these commands on your **Docker Swarm manager node**.
|
||||
|
||||
#### Step 1: Remove the Stack
|
||||
|
||||
First, remove the deployed stack from your Swarm.
|
||||
|
||||
**Action:**
|
||||
- In Portainer, go to "Stacks", select your `networking-stack`, and click "Remove".
|
||||
|
||||
#### Step 2: Remove the Docker Config
|
||||
|
||||
This removes the Traefik configuration that was stored in the Swarm.
|
||||
|
||||
**Action:**
|
||||
|
||||
```zsh
|
||||
docker config rm traefik.yml
|
||||
```
|
||||
|
||||
#### Step 3: Remove the Docker Volume
|
||||
|
||||
This deletes the volume that was storing your Let's Encrypt certificates. **Warning:** This will delete your existing certificates.
|
||||
|
||||
**Action:**
|
||||
|
||||
```zsh
|
||||
docker volume rm traefik_letsencrypt
|
||||
```
|
||||
|
||||
#### Step 4: Remove the Local Config File (If Present)
|
||||
|
||||
If you didn't delete the `traefik.yml` file in the optional clean-up step, remove it now.
|
||||
|
||||
**Action:**
|
||||
|
||||
```zsh
|
||||
rm ./traefik.yml
|
||||
```
|
||||
|
||||
After completing these steps, your environment will be clean, and you can safely re-run the setup guide from the beginning.
|
||||
|
||||
---
|
||||
|
||||
### Step 7: Verify Traefik Dashboard
|
||||
|
||||
Once your `networking-stack` is deployed and Traefik has started, you can verify its functionality by accessing the Traefik dashboard.
|
||||
|
||||
**Action:**
|
||||
1. Open your web browser and navigate to the Traefik dashboard:
|
||||
- **Traefik Dashboard:** `https://traefik.sj98.duckdns.org`
|
||||
|
||||
You should see the Traefik dashboard, listing your routers and services. If you see a certificate warning, it might take a moment for Let's Encrypt to issue the certificate. If the dashboard loads, Traefik is running correctly.
|
||||
46
docs/guides/traefik_urls.md
Normal file
46
docs/guides/traefik_urls.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# Traefik URLs
|
||||
|
||||
This file contains a list of all the Traefik URLs defined in the Docker Swarm stack files.
|
||||
|
||||
## Media Stack (`docker-swarm-media-stack.yml`)
|
||||
|
||||
- **Homarr:** [`homarr.sj98.duckdns.org`](https://homarr.sj98.duckdns.org)
|
||||
- **Plex:** [`plex.sj98.duckdns.org`](https://plex.sj98.duckdns.org)
|
||||
- **Jellyfin:** [`jellyfin.sj98.duckdns.org`](https://jellyfin.sj98.duckdns.org)
|
||||
- **Immich:** [`immich.sj98.duckdns.org`](https://immich.sj98.duckdns.org)
|
||||
|
||||
## Full Stack (`full-stack-complete.yml`)
|
||||
|
||||
- **OpenWebUI:** `ai.sj98.duckdns.org`
|
||||
- **Paperless-ngx:** `paperless.sj98.duckdns.org`
|
||||
- **Stirling-PDF:** `pdf.sj98.duckdns.org`
|
||||
- **SearXNG:** `search.sj98.duckdns.org`
|
||||
- **TSDProxy:** `tsdproxy.sj98.duckdns.org`
|
||||
|
||||
## Monitoring Stack (`monitoring-stack.yml`)
|
||||
|
||||
- **Prometheus:** `prometheus.sj98.duckdns.org`
|
||||
- **Grafana:** `grafana.sj98.duckdns.org`
|
||||
- **Alertmanager:** `alertmanager.sj98.duckdns.org`
|
||||
|
||||
## Networking Stack (`networking-stack.yml`)
|
||||
|
||||
- **whoami:** `whoami.sj98.duckdns.org`
|
||||
|
||||
## Tools Stack (`tools-stack.yml`)
|
||||
|
||||
- **Portainer:** `portainer.sj98.duckdns.org`
|
||||
- **Dozzle:** `dozzle.sj98.duckdns.org`
|
||||
- **Lazydocker:** `lazydocker.sj98.duckdns.org`
|
||||
|
||||
## Productivity Stack (`productivity-stack.yml`)
|
||||
|
||||
- **Nextcloud:** `nextcloud.sj98.duckdns.org`
|
||||
|
||||
## TSDProxy Stack (`tsdproxy-stack.yml`)
|
||||
|
||||
- **TSDProxy:** `proxy.sj98.duckdns.org`
|
||||
|
||||
## Portainer Stack (`portainer-stack.yml`)
|
||||
|
||||
- **Portainer:** `portainer0.sj98.duckdns.org`
|
||||
Reference in New Issue
Block a user