Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries

Summary

Embedding model inference often struggles with efficiency when serving large volumes of short requests—a common pattern in search, retrieval, and recommendation systems. At Voyage AI by MongoDB, we call these short requests queries, and other requests are called documents. Queries typically must be served with very low latency (typically 100–300 ms). Queries are typically short, and their token-length distribution is highly skewed. As a result, query inference tends to be memory-bound rather tha

SpinGraph analysis pending — check back after processing.

• That’s a Wrap: MongoDB’s 2025 in Review & 2026 Predictions
• Vision RAG: Enabling Search on Any Documents
• MongoDB.local San Francisco 2026: Ship Production AI, Faster
• Edge AI Made Easy: MongoDB and ObjectBox Data Synchronization
• Building a Movie Recommendation Engine with Hugging Face and Voyage AI
• Innovating with MongoDB | Customer Successes, February 2026