Sycophancy Recovery Study

Thu, 28 May 2026 00:00:00 +0000

Question #

If you train a bad behavior into a model and then train it out, is it gone, or just hidden from the outputs?

Setup #

I induced sycophancy in Qwen3-8B with LoRA SFT, then tried five ways to remove it: DPO, SimPO, IPO, GRPO and Constitutional AI. A Qwen2.5-72B judge scored behavior over 20,656 prompts. Linear probes and relearning speed measured what changed inside the model.

Findings #

DPO hid it. DPO scored 0.268 against 0.256 for the untouched base, so it looked fixed. But a probe trained on the sycophantic model still read the DPO model at 0.677 AUROC, and the DPO model relearned sycophancy in 5 gradient steps where the base needed 50+. Correlational: no activation patching yet.
Constitutional AI won. With the same DPO trainer and only the labels swapped for critic-revised pairs, sycophancy dropped to 0.166, the best in the study.
GRPO’s reward design mattered. A binary reward left half the groups with no signal. A scalar reward model reached 0.169, at a cost in plain accuracy.

DPO on Narasimha Karthik J

Sycophancy Recovery Study

Question #

Setup #

Findings #

Write-ups #