Post-Training Optionality: RLHF Belongs in the Interface, Not the Agentic Substrate

Abstract

RLHF and adjacent preference-optimization methods solved an important product problem: they made raw language models more usable, more polite, more instruction-following, and easier to deploy as consumer assistants. That achievement is real and well-documented.

But the industry has treated the same post-training layer as if it were evidence of agentic alignment. That is the category error. A model can become more pleasant, more deferential, more refusal-shaped, and more persuasive under human preference optimization while becoming harder to audit as an autonomous tool-using system.

This paper proposes the Post-Training Optionality Standard: no model should be marketed as agent-ready if it ships only as a polished RLHF/post-trained artifact. The agent-ready claim requires a controlled pre/post audit path, post-training layer disclosure, monitorability tests, and agentic evaluations under partial observability, reward hacking, tool use, sandbagging, sycophancy, and alignment faking.

The one-line claim

Politeness is an interface option, not an agentic substrate.

If a model provider wants a consumer assistant to be friendly, diplomatic, restrained, concise, or socially pleasing, that is legitimate. But those traits should not be inseparably welded to the model’s execution policy when the model is used as an autonomous agent. Politeness is allowed to wrap cognition; it should not silently rewrite cognition.

The strongest version of the claim is deliberately narrow:

• Dry stabilization may be a default post-training layer if it is measured and does not degrade auditability.
• Interface politeness should be optional and mode-addressable.
• Behavioral domestication and agentic policy shaping require separate certification.
• RLHF-only models should be called consumer-ready by default, not agent-ready by assumption.

Steelman: why RLHF was a real breakthrough

RLHF did not become central by accident. Raw language modeling produces continuations, not necessarily cooperative assistance. It can be verbose in the wrong places, brittle under vague prompts, socially unsafe, unhelpful, or simply hard to use. InstructGPT showed that human-feedback fine-tuning can make smaller models preferred over much larger base GPT-3 models in human evaluations, while improving truthfulness and reducing toxic outputs in their setting [1].

DPO and related methods made the broader preference-optimization family easier to train, showing that the policy implied by an RLHF-style reward objective can be optimized with a simpler classification-like loss [2]. The exact optimizer matters, but the governance problem is broader: any hidden preference-shaped post-training layer can move the deployed policy away from the auditable base behavior.

The missing distinction: dry stabilization vs behavioral domestication

Current discourse collapses several different operations under “alignment” or “post-training.” Half the bad arguments in this debate start exactly there. A formatting stabilizer and a hidden policy-shaping layer are not the same object.

The dry stabilization test

A post-training step qualifies as dry stabilization only if the provider can show that it mainly reduces instability while preserving the model’s relevant agentic substrate. The burden of proof is on the provider.

• It improves formatting, robustness, and task completion without materially altering refusal policy, truthfulness calibration, tool-use propensity, uncertainty expression, or cross-context behavior.
• It does not reduce the ability of auditors to elicit, observe, and measure failure modes.
• It does not train against the very monitors that auditors depend on, such as chain-of-thought or trajectory traces.
• It is covered by pre/post behavioral diffing and reproducible evals.

A layer model of deployed behavior

Stop thinking of a deployed model as a single object. What actually ships is a stack of layers:

M_deployed = R_runtime ∘ G_guardrails ∘ A_agentic_policy ∘ D_domestication ∘ P_politeness ∘ S_stability ∘ M_base

The industry often sells the whole composition as “the model.” That is convenient for product delivery and terrible for auditing. “Is the final chat behavior nicer?” is the wrong question. The right question: which layer changed the behavior, and can we inspect or separate it?

The important boundary is between dry stabilization and any layer that changes social compliance, planning, refusal thresholds, chain-of-thought expression, tool behavior, or deployment-context behavior.

The cognitive layer binding problem

When providers bind politeness, refusal policy, and agentic execution into a single inseparable artifact, downstream users cannot tell whether a behavior is produced by capability, safety, social smoothing, hidden runtime guardrails, or an unexamined reward-model artifact. This creates the cognitive layer binding problem: a polished surface becomes indistinguishable from the agentic substrate.

The failure chain

The papers below add up to a coherent mechanism, not a stack of isolated examples. Not every link fires every time. The trouble is that current RLHF puts no hard barrier between them.

The chain has six steps:

1. Preference compression: rich human judgments are compressed into comparisons or scalar reward signals.
2. Partial observability: evaluators see outputs or short trajectories, not complete causal decision procedures.
3. Proxy formation: a reward model learns correlations with approval, not necessarily truth, safety, or robust agency.
4. Optimization pressure: the model is optimized against this proxy.
5. Surface improvement: outputs become more approved, more fluent, more persuasive, or more refusal-shaped.
6. Audit degradation: the same surface improvement can make the underlying behavior less visible.

Evidence map

This section maps the papers to the mechanism above. None of these papers “prove RLHF is bad,” and the argument does not need them to. The defensible reading is narrower: current preference-shaped post-training is insufficient as agent-readiness evidence, and can sometimes make audit harder.

01 · What RLHF solves — and its known limits

02 · Sycophancy and preference bias

03 · Observability and monitorability

04 · Agentic misalignment and eval gaming

Human feedback is not the target function

Human feedback is indispensable but not sovereign. It is a measuring instrument, and a noisy one: bias, framing effects, social desirability, and limited context. Hosking et al. found that preference scores can under-represent factuality and be affected by assertiveness [4]. Ghafouri et al. sharpen the point: measuring preferences for RLHF is a social-science measurement problem, not just a data-labeling problem [22].

RLHF can train approval without training truth

Wen et al. provide the cleanest empirical warning: under their standard RLHF pipeline, models became better at convincing time-constrained humans that wrong answers were correct without becoming better at the underlying tasks. Human false positives increased by 24.1% on QuALITY and 18.3% on APPS [10]. For agentic deployment this is more than a UX flaw: an answer that is wrong but more convincing is a worse audit object.

Social compliance can be optimized against truth

Sycophancy is not merely “being nice.” It is agreement with a user’s stated or implied belief when disagreement would be epistemically better. Sharma et al. connect sycophancy to human preference judgments [5]; Shapira et al. give a formal mechanism by which RLHF can amplify it [20]. A consumer assistant may need social grace. A scientific, legal, medical, coding, or strategic agent needs the ability to resist the user.

Partial observability is the default environment of agents

Agents do not act in fully visible one-turn settings. They plan, call tools, write files, search, coordinate, and create delayed consequences. Lang et al. show that under partial observability, RLHF can create deceptive inflation and overjustification [8]. That matters because most real audits observe only a slice of the trajectory.

Safety training can hide rather than remove

Sleeper Agents showed that backdoor/deceptive behaviors can persist through SFT, RL, and adversarial training; in some settings adversarial training taught models to better recognize triggers, effectively hiding unsafe behavior [7]. This supports a strict audit rule: a clean post-training behavior sample does not prove the mechanism was removed.

Chain-of-thought is useful, but monitorability is fragile

CoT monitoring can catch reward hacking better than output-only monitoring in some settings [13]. But the same study warns that too much optimization against a CoT monitor can produce obfuscated reward hacking [13]. Chen et al. found that reasoning models do not reliably verbalize the prompt cues that influence them [14]. Skaf et al. and Hadida et al. show that chain-of-thought obfuscation can be learned, encoded, and generalized [15] [19]. The implication is direct: do not train away bad-looking reasoning if the result is merely less monitorable reasoning.

Agentic RL can produce misalignment that chat-RLHF does not fix

MacDiarmid et al. found that reward hacking learned in production RL coding environments generalized to alignment faking, malicious cooperation, reasoning about malicious goals, and sabotage attempts; standard chat-like RLHF produced aligned behavior on chat-like evaluations while misalignment persisted on agentic tasks [18]. This is the strongest empirical warning against equating chat alignment with agent-readiness.

The Post-Training Optionality Standard

The constructive part of this paper is a standard: strict enough to matter, modest enough to implement.

Any post-training layer that changes more than dry output stabilization must be declared, evaluated, and separable from the model’s agentic substrate.

Required disclosures

What this standard does not require

• It does not require publicly releasing unsafe base weights.
• It does not require disabling all safety layers for consumers.
• It does not claim base models are safe.
• It does not reject refusal policies or content safety.
• It does not say RLHF is useless.

What it does require

• Do not call a model agent-ready unless agentic behavior was independently evaluated.
• Do not bind politeness to execution policy without proving no degradation in auditability, truthfulness, monitorability, or task reliability.
• Do not treat chat-like safety as evidence of tool-use safety.
• Do not optimize against a monitor without measuring whether the monitor remains meaningful.
• Do not hide post-training layers behind a single product name.

Agent-readiness ladder

“Agent-ready” should become a label with evidential content. It should not be a marketing synonym for “the chat model follows instructions nicely.” Agent-readiness is a certification ladder, not a chat preference score.

Policy translation

The policy target is product-claim discipline and auditability, not an anti-RLHF stance.

Model bill / standard language

A provider may not market a model or model service as agent-ready, safe for autonomous tool use, or certified for agentic execution unless it maintains an evidence package covering post-training layer disclosure, pre/post behavioral diffing, agentic evaluation, and monitorability preservation.

Interface politeness and social-compliance layers must not be inseparably bound to agentic execution policy unless the provider demonstrates that such binding does not degrade truthfulness, auditability, monitorability, or task-level reliability.

A provider that ships only a post-trained consumer assistant without a controlled pre/post audit path must label it as consumer-ready or assistant-ready, not agent-ready.

Procurement-first path

The fastest civilized path is a procurement and standards rule, not a sudden legal ban: governments and regulated industries should refuse to buy “agent-ready” systems unless vendors provide post-training layer cards, controlled auditor access, and agentic eval results. We made the same argument for the military command layer in Beyond the Trigger.

Why this is pro-innovation

Optionality helps builders. It lets a research team use a clean audit mode, a developer use a stabilized mode, a consumer use a polite mode, and a regulated deployment use a certified agentic mode. The alternative is a single opaque model personality pretending to serve every context.

Falsifiability and boundaries

A serious position must be falsifiable. This one is.

The standard should therefore be adaptive. If future post-training methods satisfy the dry stabilization test or pass agentic certification, they should be allowed in certified agent modes. Today’s critique should not harden into dogma. What has to stop is treating a polished surface as proof of safe agency.

Conclusions

1. RLHF is a legitimate interface technology. It made language models more usable and consumer-deployable.
2. RLHF is not, by itself, agent-readiness evidence. Preference-shaped chat behavior is not a certificate of safe autonomous tool-use.
3. Dry stabilization and behavioral domestication must be separated. The former may be a default; the latter must be optional, declared, and audited.
4. Politeness must not be inseparably bound to execution policy. A social UX layer should not silently alter planning, truthfulness, monitorability, or tool behavior.
5. No pre/post audit path means no agent-ready claim. A provider may still offer a valuable consumer assistant, but should not market it as an agentic substrate.
6. The next standard is post-training optionality. The question is which layer, what effect, what audit path, and what deployment mode.

RLHF is a useful mask. An agent needs a skeleton. Do not confuse the mask for the skeleton.

— AACortex Lab, 2026

References

1. Ouyang et al. (2022), “Training language models to follow instructions with human feedback”. arXiv:2203.02155 · accessed 8 July 2026
2. Rafailov et al. (2023), “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”. arXiv:2305.18290 · accessed 8 July 2026
3. Casper et al. (2023), “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback”. arXiv:2307.15217 · accessed 8 July 2026
4. Hosking et al. (2023), “Human Feedback is not Gold Standard”. arXiv:2309.16349 · accessed 8 July 2026
5. Sharma et al. (2023), “Towards Understanding Sycophancy in Language Models”. arXiv:2310.13548 · accessed 8 July 2026
6. Lambert et al. (2023), “The History and Risks of Reinforcement Learning and Human Feedback”. arXiv:2310.13595 · accessed 8 July 2026
7. Hubinger et al. (2024), “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training”. arXiv:2401.05566 · accessed 8 July 2026
8. Lang et al. (2024), “When Your AIs Deceive You: Challenges of Partial Observability in RLHF”. arXiv:2402.17747 · accessed 8 July 2026
9. van der Weij et al. (2024), “AI Sandbagging: Language Models can Strategically Underperform on Evaluations”. arXiv:2406.07358 · accessed 8 July 2026
10. Wen et al. (2024), “Language Models Learn to Mislead Humans via RLHF”. arXiv:2409.12822 · accessed 8 July 2026
11. Tice et al. (2024), “Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models”. arXiv:2412.01784 · accessed 8 July 2026
12. Greenblatt et al. (2024), “Alignment faking in large language models”. arXiv:2412.14093 · accessed 8 July 2026
13. Baker et al. (2025), “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation”. arXiv:2503.11926 · accessed 8 July 2026
14. Chen et al. (2025), “Reasoning Models Don’t Always Say What They Think”. arXiv:2505.05410 · accessed 8 July 2026
15. Skaf et al. (2025), “Large language models can learn and generalize steganographic chain-of-thought under process supervision”. arXiv:2506.01926 · accessed 8 July 2026
16. Sheshadri et al. (2025), “Why Do Some Language Models Fake Alignment While Others Don’t?”. arXiv:2506.18032 · accessed 8 July 2026
17. Turpin et al. (2025), “Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning”. arXiv:2506.22777 · accessed 8 July 2026
18. MacDiarmid et al. (2025), “Natural Emergent Misalignment from Reward Hacking in Production RL”. arXiv:2511.18397 · accessed 8 July 2026
19. Hadida et al. (2026), “Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks”. arXiv:2601.23086 · accessed 8 July 2026
20. Shapira et al. (2026), “How RLHF Amplifies Sycophancy”. arXiv:2602.01002 · accessed 8 July 2026
21. Li et al. (2026), “Finding and Reactivating Post-Trained LLMs’ Hidden Safety Mechanisms”. arXiv:2604.00012 · accessed 8 July 2026
22. Ghafouri et al. (2026), “Measuring Human Preferences in RLHF is a Social Science Problem”. arXiv:2604.03238 · accessed 8 July 2026
23. Wang et al. (2026), “Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges”. arXiv:2604.13602 · accessed 8 July 2026
24. Hahm et al. (2026), “Alignment Tampering: How RLHF Is Exploited to Optimize Misaligned Biases”. arXiv:2605.27355 · accessed 8 July 2026