Counterfactual Evidence Audits Predict LLM-Agent Susceptibility to Ranked Context
Quick summary
arXiv:2606.00914v2 Announce Type: replace Abstract: LLM agents increasingly decide from evidence assembled by upstream systems: retrievers choose documents, recommenders choose posts, and memory systems choose prior events. Existing evaluations usually hold this evidence fixed, missing failures in which individually ordinary items form a systematically one-sided context. We introduce a counterfactual evidence audit: expose an agent to two mirrored sets of five documents, measure the difference in six downstream decisions, and use that contrast to predict its response to disjoint 45-document co
Key takeaways
- arXiv:2606.00914v2 Announce Type: replace Abstract: LLM agents increasingly decide from evidence assembled by upstream systems: retrievers choose documents, recommenders choose posts, and memory systems choose prior events.
- Existing evaluations usually hold this evidence fixed, missing failures in which individually ordinary items form a systematically one-sided context.
- We introduce a counterfactual evidence audit: expose an agent to two mirrored sets of five documents, measure the difference in six downstream decisions, and use that contrast to predict its response to disjoint 45-document co
Why it matters
The importance of “Counterfactual Evidence Audits Predict LLM-Agent Susceptibility to Ranked Context” will be measured by what changes in practice. User behavior, access conditions, verifiable performance and responsible-use outcomes are the signals worth following.
Member comments