The Same Agent Gives Different Answers to the Same Post

A May 2026 study of forum-moderation agents finds that four invisible deployment choices, not the model's name, determine whether it intervenes.

A paper posted to arXiv on May 30, 2026, titled "Toward Agentic Governance: What Shapes LLM-Agent Intervention in Public Forums?", by Luyang Zhang, Yi-Yun Chu, and Ramayya Krishnan, examines a narrower and more uncomfortable question than most agentic-AI governance literature attempts: not whether an agent moderating a public forum makes good decisions, but whether it makes the same decision twice. The finding is that it frequently does not, and the reason has nothing to do with the content being moderated.

The study places LLM agents in moderation-relevant forum workflows, where an agent's choice to answer, acknowledge, repair, or decline a challenge is subject to scrutiny from users, platforms, and regulators after the fact. The authors report that the same agent, presented with identical content, often returns different responses. They attribute this not to randomness in generation but to four deployment choices that are typically invisible to the person operating the system: which model version is currently being served, since providers can swap versions between calls without notice; whether the model is open-weight or closed-weight; which provider is serving the request; and which system-prompt policy happens to be in force at the moment of the call. Each of these, according to the paper, independently shifts the agent's response rate, and their combinations can produce substantially different interventions on the same forum posts.

The Weight-Status Finding

The paper's most specific empirical result concerns a pattern previously reported in the moderation literature: that agents tend to decline more often on challenges visible to other users than on challenges raised privately. Zhang, Chu, and Krishnan test whether this pattern tracks the visibility of the challenge itself, or something else. Their panel finds it tracks something else. Every closed-weight model in the panel declines more on visible challenges than hidden ones. Every open-weight model either reverses that pattern or shows no gap at all. The variable that predicts the agent's behavior is not the openness of the challenge. It is whether the provider holds the weights.

That distinction matters because it is not a property anyone auditing a single deployment would think to check. An operator selecting a moderation agent evaluates the model's name, perhaps its benchmark scores, perhaps its stated safety policy. The paper's finding is that the operative variable, weight-release status interacting with challenge visibility, sits beneath the level most procurement and audit processes examine.

Reproducibility as a Governance Precondition

The paper's own framing is explicit about what is at stake: "the same agent often returns different responses on identical content, so any defense based on the agent's behavior cannot be reliably reproduced." This sentence describes a precondition failure, not a performance problem. Governance frameworks, including the risk categories named in the CISA-led Careful Adoption of Agentic AI Services advisory, assume that an agent's behavior can be observed, characterized, and held constant long enough to be audited. If the same agent, under the same content, produces different interventions depending on which of four invisible switches happened to be set at call time, then behavioral characterization is not a snapshot. It is a moving target, and the audit that certified the agent last month may describe a system configuration that no longer exists.

The paper's conclusion states the implication directly: "Auditable forum-agent governance requires awareness of all four choices, not just the model name, since each independently shifts behavior." The model name is the label most governance documentation records. The four variables that the paper identifies as load-bearing rarely appear in a system card, a vendor contract, or an audit log.

Extending the Fog

This finding extends the arc's recurring argument in a specific direction. The prior discussion of the Five Categories advisory established that visibility into an agent's outputs is not the same as verifiability of its behavior. The forum-agent variation paper narrows that gap to a specific mechanism: even a single agent, in a single deployment, is not a stable object of observation, because the substrate underneath it, model version, weight status, provider, and system prompt, can shift without leaving a trace in the interface the operator watches.

Multi-agent systems compound this. When agent behavior varies across the four variables the paper identifies, and when that variation propagates into shared context, such as a forum's persistent thread state or a scratchpad multiple agents read from, the inconsistency is not confined to a single call. An intervention decision made under one configuration becomes part of the record another agent, under a different configuration, will condition on later. The paper does not claim to trace that downstream propagation directly, but the four-variable instability it documents is the upstream condition that would make such propagation possible, and difficult to disentangle after the fact.

What a Model Card Does Not Say

Model cards and system cards, the documents most often cited as the accountability artifact for a deployed model, typically describe a model's training data, evaluation results, and intended use. They do not typically describe which of several served versions was active during a specific incident, whether the weights were open or closed at that moment, which provider handled the specific call, or which system prompt was live. The paper's finding suggests that a model card, however thorough on the axes it covers, does not capture the axes that predict behavioral variation in production.

This is a distinct problem from the accountability gap CISA's advisory names, in which agentic architecture obscures what caused a particular action. The forum-agent variation paper describes a prior problem: even when the causal chain is not obscured, the four variables that determine the agent's behavior are not the variables anyone recorded as part of that chain.

Open Questions

• At what point does a model version change, made silently by a provider, constitute a material change to a deployed system that regulators or auditors should be notified of?
• What standard applies when an agent's documented behavior was characterized under one weight-release configuration and the configuration changes before the next audit cycle?
• How does a supervisor verify which of the four variables, model version, weight status, provider, or system prompt, was active at the time of a specific contested intervention?
• Should procurement and audit processes require vendors to log all four variables at call time, and if so, who holds that log when the model, the orchestrator, and the platform are different parties?
• What happens to an existing safety case or system card when the underlying model version it was written against is swapped out by the provider without the operator's knowledge?
• Does the open-weight versus closed-weight distinction the paper identifies generalize beyond forum moderation to other agentic deployment contexts, and if so, has anyone measured it there?

The loop closed around an oversight function that was never instrumented.