Incidents
Triage, assign, resolve, and export crash investigations across the fleet.
| Incident | System | State | Owner | Detected |
|---|---|---|---|---|
Highoom kill OOM killer invoked PyTorch training job (PID 38291) consuming 62 GB of 64 GB RAM on Ubuntu, triggered by an unconstrained local training process. | Ubuntu Workstation | investigating | SRE on-call | 5/20/2026 |
Criticalgpu fault NVIDIA RTX 4090 GPU fault (Xid 79 - GPU has fallen off the bus) caused by a defective PCIe riser cable on a mining-style GPU mount, resulting in intermittent PCIe link degradation. | Ubuntu Workstation | new | Unassigned | 5/18/2026 |
Highthermal shutdown CPU thermal emergency shutdown at 97°C triggered by dust-clogged heatsink after 18 months without cleaning, causing ACPI THERMTRIP on CPU Package thermal zone. | Ubuntu Workstation | new | Unassigned | 5/15/2026 |