Lost in the middle

Liu et al. (TACL 2024) found that in multi-document question answering and key-value retrieval, language models' performance was often highest when the relevant information was at the beginning or end of the input context and degraded significantly when it was in the middle, including for models built for long contexts. For answer engines that put many passages into one prompt, this implies (not measured on production engines) that where a passage lands in the grounding set, not only whether it is retrieved, affects whether it is used.