Reducing LLM response times through compression

Time-to-first-token (TTFT) is the wall-clock time from sending a request to receiving the LLM's first streamed token. For chatbots, coding assistants, and agents, this number determines how fast your product feels to users.

When you compress context before sending it to an LLM, two things happen: you pay a small overhead for compression, but the LLM processes fewer tokens — which means faster prefill and faster time-to-first-token. The question is whether the latency savings from shorter context outweigh the compression cost.

This benchmark measures both paths end-to-end. The baseline sends full context directly to each model's API. The compressed path runs context through the TTC /v1/compress endpoint at a fixed compression_ratio of 0.5 — a static setting that removes half the tokens on any content — then sends the compressed output to the same model. Accuracy under bear-2 compression is benchmarked separately — see CoQA and SEC filings.

We measured streaming TTFT across four input sizes (10K to 200K estimated tokens) using 33 public-domain congressional hearing transcripts from govinfo.gov — long, real spoken-word documents representative of meeting, call, and support workloads. Each configuration ran 10 times from an EC2 instance in us-west-2; medians are reported.

The compressed path uses bear-2 at static compression ratio 0.5 (ratios 0.35 and 0.65 were also measured). Reasoning and thinking were set to their minimum on every model so results isolate prefill rather than thinking behavior. Nine current-generation models across OpenAI, Anthropic, Google, DeepSeek, and Alibaba were tested with identical text.

At 200K-token context with 50% token reduction, eight of nine models answered 29–49% faster. At the heavier 0.65 ratio the same models reached 40–63%. The slowest baselines gain the most: Qwen3-Next drops from 7.7s to 4.0s, Claude Opus 5 from 6.8s to 4.0s.

Compression also creates context headroom. A 200K-token document sits at the edge of Claude's window — denser documents of the same nominal size can tokenize past it and fail outright — while at ratio 0.5 the same content fits every model tested with half the window to spare for output and history.

Savings grow with context length. On Claude Haiku 4.5, compression saves 180ms at 10K tokens and 1.1 seconds (36%) at 200K — because prefill time grows with input length while compression overhead barely does.

GPT-5.6 Luna shows the same pattern with lower absolute latencies: savings of 17–28% at 10K–100K tokens, rising to 43% (1.0 second) at 200K.

The static-ratio compression calls used in this benchmark completed in 0.1–0.6 seconds median at every size from the same region — a small fraction of the seconds saved on the LLM side at large contexts.

The production fan-out path is faster still: our February 2026 measurement of bear-1.2 sustained up to 1.7M tokens per second with sub-120ms P50 latency at all sizes tested.