Agent doctor
sudo crashpilot doctor sudo crashpilot heartbeat sudo crashpilot snapshot sudo crashpilot update sudo crashpilot analyze --force
Recent alerts
Alert thresholds
- CPU load
- 90% or more
- Memory used
- 90% or more
- GPU load
- 95% or more
- Disk
- 90% used or 5 GB or less free (critical at 95% or 2 GB)
- Back online
- after more than 5 minutes without a heartbeat
- Kernel log
- any critical dmesg line
The dashboard colours load amber from 75% and red from 90%, and shows a system as offline after 5 minutes without a heartbeat. Alerts are not raised during a maintenance window. To get them in Slack or at a webhook, set that up on the Health page.
Live bandwidth
Speedtest capacity.
Offline history
2 incidents · 42m 0s offline · latest Oct 27, 12:36 PM
Load history
Live dmesg
[Mon Jun 8 10:14:02] pcieport 0000:00:01.0: AER: Corrected error received [Mon Jun 8 10:16:41] NVRM: Xid 13, pid=4218, Graphics Exception
Recent crash reports
OOM killer invoked PyTorch training job (PID 38291) consuming 62 GB of 64 GB RAM on Ubuntu, triggered by an unconstrained local training process.
NVIDIA RTX 4090 GPU fault (Xid 79 - GPU has fallen off the bus) caused by a defective PCIe riser cable on a mining-style GPU mount, resulting in intermittent PCIe link degradation.
CPU thermal emergency shutdown at 97°C triggered by dust-clogged heatsink after 18 months without cleaning, causing ACPI THERMTRIP on CPU Package thermal zone.