CrashPilotX
Sign in
All reports
HighOOM Kill AI analyzed1 secret redacted
5 months ago
5/20/2026, 3:12:58 AM
Most likely cause

OOM killer invoked PyTorch training job (PID 38291) consuming 62 GB of 64 GB RAM on Ubuntu, triggered by an unconstrained local training process.

The system ran out of memory during a large language model training job at 03:12 UTC. The Linux OOM killer terminated the training process after it consumed nearly all available RAM. The swap space (16 GB) was already exhausted. Root cause is unconstrained process memory growth combined with insufficient swap.

Confidence
93%
AI + forensic rules
Evidence
5 signals
dmesg, journal, system
Evidence gaps
1 noted
Per-process memory history was not retained before the OOM event
Fingerprint
4be91a6d120f0c82
Forensic schema v1
93%

Very high confidence (93%) - OOM kill events leave unambiguous forensic traces in the journal. The exact PID, process name, and memory state at kill time are logged. Lower confidence scenarios: if the journal was corrupted or if multiple OOM events occurred in rapid succession.

What happened before the failure
-15m
FAILURE
May 20, 3:12 AM
No reboot detected
System stayed online
CPUMemoryDiskGPU
Last known state before failure
<1 min before failure (May 20, 3:12 AM)
CPU 88% · Memory 99% · Disk 70.6% · GPU 16%
What changed right before
  • torch_train grew +42112 MB (python train.py --model llama)
  • llm-finetune.service failed: LLM fine-tuning job
  • Upgrade: nvidia-driver-550:amd64 (550.78, 550.90.07)
02:45:00 AMjournal

systemd service llm-finetune.service started

Training job begins

03:08:33 AMjournal

Memory pressure - 58 GB / 64 GB used, swap at 14 GB / 16 GB

System entering critical memory territory

03:11:47 AMjournal

kswapd0 consuming 100% CPU - aggressive reclamation

Kernel frantically freeing pages; system nearly unresponsive

03:12:44 AMjournal

oom_kill_process: torch_train pid=38291 score=923

OOM killer selects and terminates the training process

03:12:58 AMjournal

System reset - watchdog triggered 14s after OOM kill

Secondary crash: kernel hung after OOM, forcing hard reset

03:14:05 AMjournal

System boot - CrashPilotX forensic analysis begins

Recovery boot

journal
95%
Source: journalctlTime: 2026-05-20T03:12:44ZProcess: torch_trainPID: 38291
Out of memory: Kill process 38291 (torch_train) score 923 or sacrifice child

OOM killer selected training process as kill target with near-maximum badness score (1000=worst)

journal
92%
Source: journalctlTime: 2026-05-20T03:12:44Z
Mem-Info: MemFree: 32kB SwapFree: 0kB

Zero swap remaining - system completely exhausted both RAM and swap before OOM kill

journal
88%
Source: journalctlNo timestamp/process/PID present in this line
llm-finetune.service has no MemoryMax configured

Service ran without a systemd memory cap, allowing unconstrained growth into host RAM

dmesg
75%
Source: dmesgProcess: kswapd0
[187234.442] kswapd0: page allocation failure order=3

Kernel unable to allocate 8-page contiguous block - memory extremely fragmented

system
70%
Source: system stateNo timestamp/process/PID present in this line
load_15m: 47.3 (on 16-core system)

Load average 3x CPU count indicates severe thrashing for 15+ minutes before crash

journal
20%
Source: journalctlNo timestamp/process/PID present in this line
llm-finetune.service: ExecStart=/usr/bin/python3 train.py --wandb-key=[REDACTED:labeled_credential]

Service launch command referenced an experiment-tracking API key, redacted before this report was generated

Report ID: crash_a1b2c3d4e5f6 · Boot: f47ac10b58cc…