117 lines
4.3 KiB
Markdown
117 lines
4.3 KiB
Markdown
# Multi-Device Grafana Dashboard Setup
|
|
|
|
This directory contains Grafana dashboards configured to monitor multiple NixOS devices: `mirai`, `tsuba`, and `ryu`.
|
|
|
|
## Dashboard Overview
|
|
|
|
### 1. Multi-Device System Metrics (`multi-device-system-dashboard.json`)
|
|
- **Purpose**: Compare system metrics across all devices on a single dashboard
|
|
- **Features**:
|
|
- Device filtering via template variable (can select one, multiple, or all devices)
|
|
- CPU usage comparison by device
|
|
- Memory usage comparison by device
|
|
- Disk usage comparison by device
|
|
- Network I/O comparison by device
|
|
- Service status overview
|
|
- System uptime tracking
|
|
- Load average trends
|
|
|
|
### 2. Device-Specific System Dashboard (`device-specific-system-dashboard.json`)
|
|
- **Purpose**: Detailed system monitoring for a single selected device
|
|
- **Features**:
|
|
- Device selection via template variable
|
|
- Detailed CPU, memory, and disk metrics
|
|
- Network and disk I/O operations
|
|
- System status indicators
|
|
- Load average trends (1m, 5m, 15m)
|
|
- Enhanced visualizations with thresholds and color coding
|
|
|
|
### 3. Multi-Device Process Monitoring (`multi-device-processes-dashboard.json`)
|
|
- **Purpose**: Monitor processes across all devices
|
|
- **Features**:
|
|
- Device filtering for process metrics
|
|
- Process resource usage table with device column
|
|
- CPU usage trends by process and device
|
|
- Memory usage (resident and virtual) by process
|
|
- Process I/O throughput monitoring
|
|
- Process count tracking per device
|
|
|
|
### 4. Legacy Dashboards
|
|
- `system-dashboard.json`: Original single-device system dashboard
|
|
- `processes-dashboard.json`: Original single-device process dashboard
|
|
|
|
## Device Labels
|
|
|
|
Each device is configured with specific labels for better organization:
|
|
|
|
- **mirai**: `device=mirai, type=server, arch=x86_64`
|
|
- **tsuba**: `device=tsuba, type=server, arch=aarch64`
|
|
- **ryu**: `device=ryu, type=desktop, arch=x86_64`
|
|
|
|
## Metrics Collection
|
|
|
|
### Node Exporter (Port 9100)
|
|
Collects system-level metrics:
|
|
- CPU usage and load average
|
|
- Memory utilization
|
|
- Disk usage and I/O
|
|
- Network interface statistics
|
|
- SystemD service status
|
|
- System uptime
|
|
|
|
### Process Exporter (Port 9256)
|
|
Collects process-level metrics:
|
|
- CPU usage per process
|
|
- Memory usage (resident and virtual)
|
|
- Process I/O operations
|
|
- Process count and lifecycle
|
|
|
|
## Template Variables
|
|
|
|
### Device Selection
|
|
All multi-device dashboards include a `$device` template variable that allows:
|
|
- **Single device selection**: Monitor one specific device
|
|
- **Multiple device selection**: Compare selected devices
|
|
- **All devices**: Monitor all available devices simultaneously
|
|
|
|
The variable automatically populates with available devices based on Prometheus metrics.
|
|
|
|
## Usage Tips
|
|
|
|
1. **Quick Overview**: Start with the multi-device system dashboard to get an overview of all devices
|
|
2. **Detailed Analysis**: Use device-specific dashboards for in-depth analysis of a particular device
|
|
3. **Process Monitoring**: Use the multi-device process dashboard to track resource-intensive processes across devices
|
|
4. **Filtering**: Use the device template variable to focus on specific devices of interest
|
|
5. **Time Ranges**: Adjust time ranges based on your monitoring needs (default: last 1 hour)
|
|
|
|
## Customization
|
|
|
|
### Adding New Devices
|
|
To add monitoring for new devices:
|
|
1. Configure Prometheus exporters on the new device
|
|
2. Add scrape targets to the Prometheus configuration in `grafana.nix`
|
|
3. Include appropriate device labels
|
|
4. The dashboards will automatically detect the new device
|
|
|
|
### Modifying Queries
|
|
All dashboard queries use the `device=~"$device"` filter to support multi-device functionality. When customizing:
|
|
- Ensure queries include the device filter
|
|
- Use `{{device}}` in legend formats to distinguish between devices
|
|
- Consider device-specific thresholds if needed
|
|
|
|
## Troubleshooting
|
|
|
|
### Device Not Appearing
|
|
1. Check if Prometheus can reach the device's exporters
|
|
2. Verify the device label is correctly configured
|
|
3. Check Prometheus targets at `prometheus.darksailor.dev/targets`
|
|
|
|
### Missing Metrics
|
|
1. Ensure the required exporters are running on the target device
|
|
2. Check firewall rules allow access to exporter ports (9100, 9256)
|
|
3. Verify network connectivity between devices
|
|
|
|
### Performance
|
|
- Limit the number of devices selected simultaneously for better performance
|
|
- Adjust refresh rates if dashboards become slow
|
|
- Consider shorter time ranges for detailed analysis |