Chunking: structure and context matter more than size

Retrievers score passages, not pages: DPR, for example, splits documents into disjoint 100-word blocks as its basic retrieval units. How the boundaries are drawn matters less than often assumed: across document retrieval, evidence retrieval and answer generation, Qu et al. (Findings of NAACL 2025) found semantic chunking's extra cost was not justified by consistent gains over fixed-size chunking, which often performed better on real (non-synthetic) documents, and they note that splitting documents into structured sections before fixed-size chunking avoids its main failure. Two preprints on specific corpora point the same way: chunking along document structure (titles, sections, tables) rather than fixed-size windows improved retrieval and RAG answer quality on financial reports (Jimeno Yepes et al., 2024) and gave higher top-K retrieval effectiveness on enterprise documents (Taiwo and Yusoff, 2026). What each chunk contains matters more: Anthropic reports that chunks lacking context (which company, which period) retrieve poorly. Google advises breaking long content into sections with headings, though it says heading order does not matter to Search and there is no ideal number of headings.