Improving LLM reading comprehension with compression

SQuAD 2.0 (Stanford Question Answering Dataset) is a reading comprehension benchmark created by Rajpurkar et al. at Stanford University. It tests whether models can read a passage of text and correctly answer questions about it — or recognize when no answer exists.

The dataset combines 100,000+ answerable questions from the original SQuAD 1.1 with over 50,000 adversarially crafted unanswerable questions. The unanswerable questions are designed to look plausible — they are written by crowdworkers who were shown the passage and asked to write questions that cannot be answered from it.

This makes SQuAD 2.0 significantly harder than its predecessor. Models must not only find correct answers in the text but also learn when to abstain — a critical capability for production systems where confidently wrong answers are worse than no answer at all.

We sampled 150 questions from the SQuAD 2.0 validation set (69 answerable, 81 unanswerable), all drawn from a single article about the Normans to keep passage context consistent. Each question was run through seven bear-1.2 compression configurations (aggressiveness from 0.05 to 0.7) and one uncompressed control.

The control sends the complete, uncompressed Wikipedia passage straight to gpt-5-mini. This isolates compression as the only variable — any accuracy difference is attributable to what bear-1.2 kept or removed. Responses are evaluated by gpt-5-mini acting as an LLM-judge, comparing each answer against the gold answer key from the dataset.

Light compression improved accuracy over the uncompressed baseline. Heavier compression reduced accuracy below baseline, showing a clear sensitivity to aggressiveness level on this benchmark.

The best result: 72.0% accuracy at aggressiveness 0.05 — a 4.0 percentage point improvement over the 68.0% baseline. On 150 questions, that translates to 6 additional correct answers while removing 17.3% of input tokens.

SQuAD 2.0 tests two distinct capabilities: finding correct answers in the text (answerable) and recognizing when no answer exists (unanswerable). The impact of compression differed sharply between them.

Answerable questions were highly robust to compression. Accuracy stayed at 95.7% through aggressiveness 0.3 and remained above 88% even at 0.7 — the key information needed to extract an answer was consistently preserved.

Unanswerable questions showed the most dramatic improvement. At aggressiveness 0.05, detection improved from 44.4% to 51.9% — a 7.5 percentage point gain. By removing noise, compression made it easier for the model to confidently determine that a passage does not contain the answer.

Unlike FinanceBench, where all configurations stayed above baseline, SQuAD 2.0 shows a clear inflection point. Light compression (0.05) is the sweet spot — accuracy goes up by 4.0 percentage points while removing 17% of tokens.

At aggressiveness 0.1, accuracy returns to near-baseline (68.7%). Beyond 0.3, accuracy drops below the uncompressed control. This makes reading comprehension more sensitive to compression than financial document QA, likely because SQuAD answers require extracting specific spans from the text — removing too many tokens risks discarding the exact phrase the model needs.