Zero accuracy loss on conversational QA with 14% fewer tokens

CoQA (Conversational Question Answering) is a benchmark created by Reddy et al. at Stanford University. It tests whether models can engage in multi-turn conversations about a passage of text — answering a series of interconnected questions where each answer may depend on the context of previous turns.

The dataset spans 7 diverse domains including children's stories, news articles, Wikipedia, and English exam passages. This diversity tests whether comprehension holds across different writing styles and complexity levels.

What makes CoQA challenging is the conversational nature: questions often use pronouns and references that only make sense given the prior dialogue. The model must track context across turns while also understanding the source passage — a dual attention requirement that stresses both memory and comprehension.

We sampled 150 questions from the CoQA validation set, drawn from conversations across 4 source domains (CNN, MCTest, RACE, Wikipedia). Each question was run through seven bear-1.2 compression configurations (aggressiveness from 0.05 to 0.7) and one uncompressed control.

The control sends the complete, uncompressed passage and conversation history straight to gpt-5-mini. This isolates compression as the only variable — any accuracy difference is attributable to what bear-1.2 kept or removed. Responses are evaluated by gpt-5-mini acting as an LLM-judge, comparing each answer against the gold answer key from the dataset.

At aggressiveness 0.05, bear-1.2 matched the uncompressed baseline exactly — 87.3% accuracy, 131 out of 150 correct — while removing 14.3% of input tokens. This is a free reduction in cost and latency with zero impact on answer quality.

Accuracy remained close to baseline at 0.1 (85.3%, −2.0pp) and degraded gradually with heavier compression. At the most aggressive setting (0.7), accuracy dropped to 63.3% — a 24 percentage point loss, indicating that conversational QA is sensitive to heavy token removal.

CoQA draws from multiple source domains, each with different writing styles and complexity. At aggressiveness 0.05, accuracy held steady across all four domains — and one domain actually improved.

MCTest (children's stories) improved from 85.4% to 90.2% — a 4.8 percentage point gain. These simpler narratives may contain more compressible noise, and removing it helps the model focus on the story elements needed for answers.

RACE and Wikipedia were unchanged at 81.1% and 92.3% respectively. CNN saw a small drop from 91.5% to 88.1%, though it remained the second-highest performing domain.

CoQA is a multi-turn dataset where each conversation contains up to 20 follow-up questions about a passage. Later turns often require resolving coreferences and building on prior answers, making them inherently harder.

At aggressiveness 0.05, bear-1.2 actually improved early-turn accuracy from 90.9% to 94.5% (+3.6pp) on turns 1–5, while mid-conversation turns (6–10) saw a modest dip from 88.9% to 83.3%. Late-conversation turns (11–15) improved from 78.6% to 85.7%, suggesting that light compression can help the model focus on the most relevant parts of longer passages.

Aggressiveness 0.05 is the clear sweet spot: identical accuracy to the uncompressed baseline with 14% token savings. At 0.1, accuracy drops by only 2 percentage points while saving nearly 20% of tokens.

Beyond 0.3, accuracy falls below 80% and the degradation accelerates. At 0.7, over half the tokens are removed but accuracy drops to 63.3%. Conversational QA appears more sensitive to heavy compression than single-turn tasks, likely because multi-turn context tracking requires preserving more of the passage and dialogue history.