Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

Summary

arXiv:2607.01237v1 Announce Type: new Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in ex

SpinGraph analysis pending — check back after processing.

• Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
• Parameter Golf: What Really Works?
• From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
• Comparing Architectures for Supervised Political Scaling
• Grounded Optimization: A Layered Engineering Framework for Reducing LLM Hallucination in Automated Personal Document Rewriting
• FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning