DPO

Induced sycophancy in Qwen3-8B, then compared DPO, SimPO, IPO, GRPO and Constitutional AI. DPO fooled the judge while a probe still found the disposition; CAI preference pairs did best.