Page footers appear in every chunk and hurt retrieval. What should you do?
Why C is the answer
Filter them out before chunking
Text repeated on every page adds noise to every chunk. Filtering it out first means retrieval matches the real content.
In detail
- Footers repeat on each page and add noise to retrieval.
- Filtering out extraneous content stops it from degrading a RAG application.
- Look for repeated headers, page numbers and legal lines.
Use case
An illustrative scenario, not a real incident.
Each page of a report carries the same footer, and searches keep matching it. Removing footers before chunking leaves content that actually matters.
See it in different ways
flowchart LR A["Source documents"] --> B["Extract text"] B --> C["Filter extraneous content"] C --> D["Chunk"] D --> E["Delta table in Unity Catalog"] E --> F["Vector Search index"]
stateDiagram-v2 [*] --> Raw Raw --> Text: parse or OCR Text --> Clean: remove extraneous content Clean --> Chunked: split Chunked --> Governed: Delta table in Unity Catalog Governed --> Indexed: Vector Search Indexed --> [*]
flowchart LR S0["Extract text"] --> S1["Filter out footers"] --> S2["Chunk"] --> S3["Index"]
Key point: Remove noise before chunking.
Why the other options are wrong
- A. Increase the chunk overlap
- More overlap spreads the noise further.
- B. Raise the model's temperature
- Temperature has no effect on retrieval.
- D. Add them to the prompt, so the model can ignore them
- The footers would still be in the chunks and the index.