dotfiles/nixos/mirai/services/grafana/README.md

# Multi-Device Grafana Dashboard Setup

This directory contains Grafana dashboards configured to monitor multiple NixOS devices: `mirai`, `tsuba`, and `ryu`.

## Homepage Dashboard

When you access Grafana at `grafana.darksailor.dev`, you'll be greeted with the **Monitoring Hub Homepage** that provides:

- **Device Status Overview**: Real-time status table showing CPU, memory, disk usage, and uptime for all devices
- **Dashboard Navigation**: Color-coded tiles linking to all monitoring dashboards
- **Quick Actions**: Direct links to Prometheus targets, dashboard management, and documentation
- **Organized Layout**: Dashboards are grouped by category for easy navigation

## Dashboard Organization

Dashboards are organized into folders for better management:

### **📊 Multi-Device Monitoring**
Located in `multi-device/` folder:

#### 1. Multi-Device System Metrics
- **Purpose**: Compare system metrics across all devices on a single dashboard
- **Features**:
  - Device filtering via template variable (can select one, multiple, or all devices)
  - CPU usage comparison by device
  - Memory usage comparison by device
  - Disk usage comparison by device
  - Network I/O comparison by device
  - Service status overview
  - System uptime tracking
  - Load average trends

#### 2. Multi-Device Process Monitoring
- **Purpose**: Monitor processes across all devices
- **Features**:
  - Device filtering for process metrics
  - Process resource usage table with device column
  - CPU usage trends by process and device
  - Memory usage (resident and virtual) by process
  - Process I/O throughput monitoring
  - Process count tracking per device

### **🔍 Device-Specific**
Located in `device-specific/` folder:

#### Device-Specific System Dashboard
- **Purpose**: Detailed system monitoring for a single selected device
- **Features**:
  - Device selection via template variable
  - Detailed CPU, memory, and disk metrics
  - Network and disk I/O operations
  - System status indicators
  - Load average trends (1m, 5m, 15m)
  - Enhanced visualizations with thresholds and color coding

### **📚 Legacy Dashboards**
Located in `legacy/` folder:
- `system-dashboard.json`: Original single-device system dashboard
- `processes-dashboard.json`: Original single-device process dashboard

## Device Labels

Each device is configured with specific labels for better organization:

- **mirai**: `device=mirai, type=server, arch=x86_64`
- **tsuba**: `device=tsuba, type=server, arch=aarch64`
- **ryu**: `device=ryu, type=desktop, arch=x86_64`

## Metrics Collection

### Node Exporter (Port 9100)
Collects system-level metrics:
- CPU usage and load average
- Memory utilization
- Disk usage and I/O
- Network interface statistics
- SystemD service status
- System uptime

### Process Exporter (Port 9256)
Collects process-level metrics:
- CPU usage per process
- Memory usage (resident and virtual)
- Process I/O operations
- Process count and lifecycle

## Template Variables

### Device Selection
All multi-device dashboards include a `$device` template variable that allows:
- **Single device selection**: Monitor one specific device
- **Multiple device selection**: Compare selected devices
- **All devices**: Monitor all available devices simultaneously

The variable automatically populates with available devices based on Prometheus metrics.

## Usage Guide

### Getting Started
1. **Homepage First**: Visit `grafana.darksailor.dev` to see the homepage with device overview
2. **Dashboard Navigation**: Click on the colored tiles to access different dashboard categories
3. **Quick Status Check**: Use the device status table on the homepage for immediate health overview

### Dashboard Navigation Tips
1. **Multi-Device Comparison**: Start with multi-device dashboards to compare all systems
2. **Deep Dive Analysis**: Use device-specific dashboards for detailed single-device monitoring
3. **Process Troubleshooting**: Use multi-device process dashboard to identify resource issues
4. **Template Variables**: Use the device dropdown to filter by specific devices
5. **Time Ranges**: Adjust time ranges based on your monitoring needs (default: last 1 hour)
6. **Folder Organization**: Browse dashboards by category using the folder structure

## Customization

### Adding New Devices
To add monitoring for new devices:
1. Configure Prometheus exporters on the new device
2. Add scrape targets to the Prometheus configuration in `grafana.nix`
3. Include appropriate device labels
4. The dashboards will automatically detect the new device

### Modifying Queries
All dashboard queries use the `device=~"$device"` filter to support multi-device functionality. When customizing:
- Ensure queries include the device filter
- Use `{{device}}` in legend formats to distinguish between devices
- Consider device-specific thresholds if needed

## Troubleshooting

### Device Not Appearing
1. Check if Prometheus can reach the device's exporters
2. Verify the device label is correctly configured
3. Check Prometheus targets at `prometheus.darksailor.dev/targets`

### Missing Metrics
1. Ensure the required exporters are running on the target device
2. Check firewall rules allow access to exporter ports (9100, 9256)
3. Verify network connectivity between devices

### Performance
- Limit the number of devices selected simultaneously for better performance
- Adjust refresh rates if dashboards become slow
- Consider shorter time ranges for detailed analysis

## Homepage Features

### Device Status Table
- Real-time overview of all monitored devices
- Color-coded status indicators (UP/DOWN)
- CPU, memory, and disk usage percentages
- System uptime tracking
- Automatically updates every 30 seconds

### Dashboard Categories
- **Blue**: Multi-Device System monitoring
- **Green**: Device-Specific detailed monitoring  
- **Purple**: Multi-Device Process monitoring
- **Orange**: Legacy single-device dashboards

### Quick Actions
- **Prometheus Targets**: Direct link to check scrape target health
- **Dashboard Management**: Access to Grafana's dashboard browser
- **Data Sources**: Manage Prometheus and other data sources
- **Documentation**: Link to setup and usage documentation