Introducing
Sarvam Token Factory
Today, we are launching Sarvam Token Factory for everyone. Token Factory is our inference stack to serve our own models and leading open-weight models from GPU clusters located in India. On most of the models we serve, we match or beat the leading inference providers globally on both speed and cost.
Along with Sarvam 105B, we serve GLM 5.3, GLM 5.3 Flash, DeepSeek V4.1 Flash 1017, DeepSeek V4 Pro, Kimi K3, Kimi K3.1 Pro and Gemma 4 31B through one OpenAI-compatible API. See the full model catalogue.
Sarvam Token Factory at a glance
- Cost: We offer globally competitive pricing on every model we serve. DeepSeek V4.1 Flash launches with introductory pricing of ₹14 per million input tokens and ₹55 per million output tokens.
- Performance: Sarvam 105B brings together all our inference optimisations to deliver best-in-class performance for conversation at the lowest cost, returning its first token in 0.2 seconds with single-user throughput of 330 tokens per second.
- Latency: Gemma 4 31B returns its first token in 0.40 seconds on a 10K-token prompt, 5× faster than the only other provider serving it.
- Throughput: Kimi K3.1 Pro delivers up to 1.9× tokens per GPU at the same per-user speed as InferenceX’s published results, measured on coding tasks.
- Caching: We built a three-tier cache with cache-aware routing and admission for the newest attention architectures. On DeepSeek V4.1 Flash, we serve 450B+ tokens a day across diverse use cases, with 95.7% of prompt tokens served from cache.
- Scale: We run India’s largest B200 cluster and are expanding our fleet to serve more models at population scale.
- Deployment: Apart from Sarvam Token Factory, we also run this inference optimization stack on-premise for critical institutions.
Performance: Throughput & Latency
Gemma 4 31B is built for real-time conversation and chat in Indian languages. Measured from Mumbai on a 10K-token prompt, Gemma 4 31B returns its first token in 0.40 seconds and streams at 308 tokens per second.
Gemma 4 31B from Mumbai, 10K-token prompt
Time to first token, lower is better
Output speed, higher is better
Gemma 4 31B from Mumbai, 10K-token prompt. Time to first token: Sarvam Token Factory 0.40 s, Other provider 2.00 s. Output speed: Sarvam Token Factory 308 tokens per second, Other provider 193 tokens per second.
| Provider | Time to first token (s) | Output speed (tokens/s) |
|---|---|---|
| Sarvam Token Factory | 0.40 | 308 |
| Other provider | 2.00 | 193 |
DeepSeek V4.1 Flash 1017, one of the most used models on OpenRouter, is available through two endpoints. The Interactive at about 200 tokens per second per user and is built for coding agents. The Standard endpoint is built for background and batch work, with price and speed comparable to GPT-6 Luna and DeepSeek’s own API. Both endpoints serve the model at its native precision. Per million tokens, Interactive is priced at ₹25 for input and ₹105 for output, and Standard at ₹14 for input and ₹55 for output.
DeepSeek V4.1 Flash: price vs output speed
DeepSeek V4.1 Flash: price vs output speed. Sarvam Standard: blended price $0.092 per million tokens, output speed 130 tok/s. Sarvam Interactive: blended price $0.168 per million tokens, output speed 200 tok/s. DeepSeek API: blended price $0.092 per million tokens, output speed 60 tok/s. Together: blended price $0.184 per million tokens, output speed 278 tok/s. Baseten: blended price $0.185 per million tokens, output speed 165 tok/s. Fireworks: blended price $0.184 per million tokens, output speed 193 tok/s. Fireworks US: blended price $0.276 per million tokens, output speed 169 tok/s. Luna (OpenAI), GPT-6 Luna for reference: blended price $0.077 per million tokens, output speed 67 tok/s.
| Endpoint | Blended price (USD per M tokens) | Output tokens per second |
|---|---|---|
| Sarvam Standard | 0.092 | 130 |
| Sarvam Interactive | 0.168 | 200 |
| DeepSeek API | 0.092 | 60 |
| Together | 0.184 | 278 |
| Baseten | 0.185 | 165 |
| Fireworks | 0.184 | 193 |
| Fireworks US | 0.276 | 169 |
| Luna (OpenAI) | 0.077 | 67 |
Measured from India, Token Factory matches the fastest provider on time to first token for GLM 5.3 with 10K-token prompts. We measured every provider the same way, using a recreation of Artificial Analysis’s method. Each test sends one prompt at a time, at 1K and 10K tokens, from a client in AWS Mumbai. We report the median over 72 hours.
Time to first token from Mumbai, lower is better
GLM 5.3: time to first token
DeepSeek V4.1 Flash: time to first token
Time to first token from Mumbai, lower is better. GLM 5.3: 1K-token prompt, Sarvam Token Factory 0.68 s, Together 0.82 s, Fireworks 0.84 s, DeepInfra 2.17 s; 10K-token prompt, Sarvam Token Factory 1.19 s, Together 1.19 s, Fireworks 1.57 s, DeepInfra 2.18 s. DeepSeek V4.1 Flash: 1K-token prompt, Sarvam Token Factory 0.51 s, Together 0.61 s, Fireworks 1.01 s, DeepInfra 1.08 s; 10K-token prompt, Sarvam Token Factory 0.90 s, Together 0.71 s, Fireworks 1.55 s, DeepInfra 1.75 s.
| Model and prompt | Sarvam Token Factory (s) | Together (s) | Fireworks (s) | DeepInfra (s) |
|---|---|---|---|---|
| GLM 5.3, 1K-token prompt | 0.68 | 0.82 | 0.84 | 2.17 |
| GLM 5.3, 10K-token prompt | 1.19 | 1.19 | 1.57 | 2.18 |
| DeepSeek V4.1 Flash, 1K-token prompt | 0.51 | 0.61 | 1.01 | 1.08 |
| DeepSeek V4.1 Flash, 10K-token prompt | 0.90 | 0.71 | 1.55 | 1.75 |
We ran our stack against upstream vLLM and Dynamo, without our changes, on GLM 5.3 Flash, with both stacks disaggregated on the same B200 nodes, and against InferenceX’s published results on Kimi K3.1 Pro.
On GLM 5.3 Flash, running SemiAnalysis's AgentX coding-agent traces on the same two B200 nodes, our stack returns the first token up to 2.1× sooner than stock vLLM and Dynamo, while serving at the same throughput per GPU. At 64 concurrent sessions, the median first token takes 667 ms against 1,428 ms, and the slowest 1% wait 3.2 s against 14.1 s. Both stacks stream at about the same speed per user, because the gain comes from prefill, not decode.
At the same tokens per GPU, GLM 5.3 Flash answers up to 2.1× sooner
1,428 ms → 667 ms
Median first token at 64 sessions, 2.1× sooner
14.1 s → 3.2 s
Slowest 1% of first tokens (p99) at 64 sessions, 4.4× sooner
190 → 192 tok/s
Median output speed per user at 64 sessions
At the same tokens per GPU, GLM 5.3 Flash answers up to 2.1× sooner. Sarvam's optimised stack at 1 sessions: median first token 699 ms, p99 2.3 s, 1,097 tokens per second per GPU, 248 per user. Sarvam's optimised stack at 4 sessions: median first token 503 ms, p99 1.7 s, 1,131 tokens per second per GPU, 234 per user. Sarvam's optimised stack at 16 sessions: median first token 551 ms, p99 2.7 s, 1,725 tokens per second per GPU, 229 per user. Sarvam's optimised stack at 32 sessions: median first token 646 ms, p99 2.2 s, 6,455 tokens per second per GPU, 220 per user. Sarvam's optimised stack at 64 sessions: median first token 667 ms, p99 3.2 s, 10,768 tokens per second per GPU, 192 per user. Stock open-source stack at 1 sessions: median first token 1,043 ms, p99 3.8 s, 1,038 tokens per second per GPU, 241 per user. Stock open-source stack at 4 sessions: median first token 912 ms, p99 9.7 s, 1,087 tokens per second per GPU, 232 per user. Stock open-source stack at 16 sessions: median first token 959 ms, p99 4.2 s, 1,692 tokens per second per GPU, 227 per user. Stock open-source stack at 32 sessions: median first token 1,168 ms, p99 14.2 s, 6,370 tokens per second per GPU, 215 per user. Stock open-source stack at 64 sessions: median first token 1,428 ms, p99 14.1 s, 10,193 tokens per second per GPU, 190 per user.
| Stack | Sessions | First token p50 (ms) | First token p99 (s) | Tokens per GPU (tok/s) | Output per user (tok/s) |
|---|---|---|---|---|---|
| Sarvam's optimised stack | 1 | 699 | 2.3 | 1,097 | 248 |
| Sarvam's optimised stack | 4 | 503 | 1.7 | 1,131 | 234 |
| Sarvam's optimised stack | 16 | 551 | 2.7 | 1,725 | 229 |
| Sarvam's optimised stack | 32 | 646 | 2.2 | 6,455 | 220 |
| Sarvam's optimised stack | 64 | 667 | 3.2 | 10,768 | 192 |
| Stock open-source stack | 1 | 1,043 | 3.8 | 1,038 | 241 |
| Stock open-source stack | 4 | 912 | 9.7 | 1,087 | 232 |
| Stock open-source stack | 16 | 959 | 4.2 | 1,692 | 227 |
| Stock open-source stack | 32 | 1,168 | 14.2 | 6,370 | 215 |
| Stock open-source stack | 64 | 1,428 | 14.1 | 10,193 | 190 |
On SemiAnalysis’s agent traces, we serve up to 1.9× InferenceX’s tokens per GPU at the same per-user speed, from 14 to 48 concurrent sessions, against their published Kimi K3.1 Pro results.
At the same speed per user, Kimi K3.1 Pro serves 1.9× the tokens per GPU
1.9× throughput
Same speed per user: 70 tok/s per user (median), our c24 against the InferenceX curve
9.1K → 13.4K tok/s
Peak tokens per GPU, best point on each curve (1.5×)
At the same speed per user, Kimi K3.1 Pro serves 1.9× the tokens per GPU. Sarvam Inference Stack at 1 sessions: 667 tokens per second per GPU at 112.2 per user. Sarvam Inference Stack at 4 sessions: 1,282 tokens per second per GPU at 91.2 per user. Sarvam Inference Stack at 8 sessions: 2,189 tokens per second per GPU at 86.1 per user. Sarvam Inference Stack at 14 sessions: 3,411 tokens per second per GPU at 83.3 per user. Sarvam Inference Stack at 24 sessions: 6,177 tokens per second per GPU at 69.5 per user. Sarvam Inference Stack at 48 sessions: 10,909 tokens per second per GPU at 48.1 per user. Sarvam Inference Stack at 72 sessions: 12,807 tokens per second per GPU at 31.2 per user. Sarvam Inference Stack at 96 sessions: 13,387 tokens per second per GPU at 21.9 per user. InferenceX Stack at 1 sessions: 711 tokens per second per GPU at 164.5 per user. InferenceX Stack at 4 sessions: 1,437 tokens per second per GPU at 107.2 per user. InferenceX Stack at 8 sessions: 2,271 tokens per second per GPU at 84.0 per user. InferenceX Stack at 14 sessions: 3,472 tokens per second per GPU at 65.4 per user. InferenceX Stack at 24 sessions: 6,106 tokens per second per GPU at 51.5 per user. InferenceX Stack at 48 sessions: 9,066 tokens per second per GPU at 32.8 per user. InferenceX Stack at 72 sessions: 4,523 tokens per second per GPU at 7.1 per user. InferenceX Stack at 96 sessions: 6,244 tokens per second per GPU at 8.0 per user. At 70 tokens per second per user, ours serves 6,177 tokens per second per GPU against 3,213 on the InferenceX curve.
| Stack | Sessions | Median output per user (tok/s) | Tokens per GPU (tok/s) |
|---|---|---|---|
| Sarvam Inference Stack | 1 | 112.2 | 667 |
| Sarvam Inference Stack | 4 | 91.2 | 1,282 |
| Sarvam Inference Stack | 8 | 86.1 | 2,189 |
| Sarvam Inference Stack | 14 | 83.3 | 3,411 |
| Sarvam Inference Stack | 24 | 69.5 | 6,177 |
| Sarvam Inference Stack | 48 | 48.1 | 10,909 |
| Sarvam Inference Stack | 72 | 31.2 | 12,807 |
| Sarvam Inference Stack | 96 | 21.9 | 13,387 |
| InferenceX Stack | 1 | 164.5 | 711 |
| InferenceX Stack | 4 | 107.2 | 1,437 |
| InferenceX Stack | 8 | 84.0 | 2,271 |
| InferenceX Stack | 14 | 65.4 | 3,472 |
| InferenceX Stack | 24 | 51.5 | 6,106 |
| InferenceX Stack | 48 | 32.8 | 9,066 |
| InferenceX Stack | 72 | 7.1 | 4,523 |
| InferenceX Stack | 96 | 8.0 | 6,244 |
At 72 concurrent sessions, InferenceX’s KV cache reaches 89% and starts evicting prefixes. Those prefixes get re-prefilled, and p95 TTFT climbs to 77 seconds. We use Moonshot’s Mooncake (G3), a shared cache that keeps prefixes alive across prefill servers instead of evicting them. Combined with per-engine session caps, p95 TTFT stays at 3.1 seconds under the same load.
Under heavy agent load, the shared cache keeps first-token time steady
72 concurrent sessions
Median (p50)
Slowest 5% (p95)
96 concurrent sessions
Median (p50)
Slowest 5% (p95)
12.8K vs 4.5K
72 sessions, tok/s per GPU, Sarvam vs InferenceX
13.4K vs 6.2K
96 sessions, tok/s per GPU, Sarvam vs InferenceX
Under heavy agent load, the shared cache keeps first-token time steady. 72 sessions (spec decode on for both; their KV 89% full): median first token Sarvam 0.8 s, InferenceX 11.6 s; 95th percentile Sarvam 3.1 s, InferenceX 76.6 s; throughput 12.8K vs 4.5K tokens per second per GPU. 96 sessions (InferenceX turns spec decode off): median first token Sarvam 1.1 s, InferenceX 0.9 s; 95th percentile Sarvam 9.3 s, InferenceX 18.8 s; throughput 13.4K vs 6.2K tokens per second per GPU.
| Sessions | Condition | Sarvam p50 (s) | InferenceX p50 (s) | Sarvam p95 (s) | InferenceX p95 (s) | Tokens per GPU, Sarvam vs InferenceX |
|---|---|---|---|---|---|---|
| 72 | spec decode on for both; their KV 89% full | 0.79 | 11.58 | 3.13 | 76.62 | 12.8K vs 4.5K |
| 96 | InferenceX turns spec decode off | 1.08 | 0.87 | 9.29 | 18.83 | 13.4K vs 6.2K |
Technical Deep Dive
Disaggregation, speculative decoding, quantisation and prefix caching are established techniques. Each new model tests the assumptions behind their implementation, from kernel correctness and attention-state storage to scheduling under real traffic. Each issue affects speed, cost or correctness. These issues need to be found and fixed within hours of a model’s release.
Our work spans three layers:
- The inference engine decides how efficiently a GPU turns a request into tokens.
- The cache decides how much of that work is done only once.
- The control plane decides which GPU gets each request, when it gets it, and what is already in that GPU’s memory.
Token Factory: three layers around a shared cache
Token Factory: three layers around a shared cache. Requests enter the control plane: Cache-aware router: sends each turn to where its prompt sits; Admission: counts only uncached tokens; Planner: scales on rejections and backpressure. The router sends work to the inference engine's Prefill pool: reads the whole prompt in one pass; tuned all-to-all, MXFP8 and NVFP4, which passes KV and model state at aligned checkpoints to the Decode pool: generates one token at a time; fused clamp kernel, suffix-tree drafts. The planner scales the decode pool. The prefill pool reads two cache tiers: GPU memory: holds a prompt ~4 s; ~2 s under load; G3 shared store (Mooncake): read by every prefill server; holds prompts for hours or days, depending on traffic.
| Component | Layer | What it does |
|---|---|---|
| Cache-aware router | Control plane | sends each turn to where its prompt sits |
| Admission | Control plane | counts only uncached tokens |
| Planner | Control plane | scales on rejections and backpressure |
| Prefill pool | Inference engine | reads the whole prompt in one pass; tuned all-to-all, MXFP8 and NVFP4 |
| Decode pool | Inference engine | generates one token at a time; fused clamp kernel, suffix-tree drafts |
| GPU memory | Shared cache | holds a prompt ~4 s; ~2 s under load |
| G3 shared store (Mooncake) | Shared cache | read by every prefill server; holds prompts for hours or days, depending on traffic |
Long agent sessions needed prefill and decode on separate GPUs
Every request goes through two phases. Prefill reads the whole prompt in one compute-heavy pass, and decode then generates tokens one at a time, limited by memory bandwidth. When both share a GPU, each new 100K-token coding prompt stalls every session already decoding there. We run the two phases on separate pools, each configured for its own work, and added EFA support to the KV transfer layer alongside InfiniBand so the split works across all our clusters.
Splitting prefill and decode is now standard. Further gains depend on transferring the cache, attention state and draft model state correctly between GPUs. We keep real-time voice workloads on the same GPU instead of using a disaggregated architecture because they have shorter input tokens.
Kernel tuning across models helps fix correctness and performance both
GLM 5.3 Flash clamps expert activations to a fixed range. vLLM’s fused decode kernel skipped this step, causing deviations of up to 165% from the reference and stray characters in place of code indentation. We fused clamping, activation, multiplication and quantisation into one kernel. It matches the reference bit for bit on in-range inputs and runs about 40% faster. On AgentX, the missing clamp showed up as stray Chinese characters in 32 of 299 long code responses. With the clamp in place, there were none.
Two other bugs caused garbled output from the first token. The kernel that initialises the indexer cache lacked a position input, so request tails wrote to the same cache block and raced with each other. Rejected draft tokens could also overwrite committed keys. After we fixed both, our test bench returned without garbled output, and draft acceptance rose from 38% to 84%, improving speed alongside correctness.On GLM 5.3 Flash, we traced prefill delays to the all-to-all exchange between GPUs. vLLM allocates 20 GPU compute units to this exchange. Increasing that to 64 : 96 and tuning chunk sizes cut median time to first token from 28.7 s to 4.7 s at a 90% cache hit rate. With two additional kernel changes, uncached prefill throughput rose from about 14K to 22K tokens per second per GPU.
On DeepSeek V4.1 Flash 1017, DeepGEMM’s MegaMoE kernels increased decode throughput by 28% at 128 concurrent sessions. Upgrading to vLLM v0.32 raised uncached prefill throughput from about 19K to 41K tokens per second per GPU, measured at an 84% cache hit rate.
Moving to MXFP8
We measure every format end to end on our own traffic before it ships. GLM 5.3 moved from FP8 weights on SGLang to MXFP8 weights on vLLM. MXFP8 is an 8-bit format with a shared scale for every block of 32 values, and B200’s tensor cores run it natively. Uncached prefill per GPU doubled, from about 3.7K to 7.6K tokens per second, decode rose 25%, and p95 first-token time on real chat halved, with accuracy unchanged against the FP8 baselines. MXFP8 has served GLM 5.3 in production since mid-October.
Moving GLM 5.3 to MXFP8 on vLLM doubled uncached prefill per GPU
Uncached prefill
2.1×
Decode
+25%
Real chat, TTFT p95
2.1× lower
Moving GLM 5.3 to MXFP8 on vLLM doubled uncached prefill per GPU. Uncached prefill: SGLang FP8 3,700, vLLM MXFP8 7,600 input tokens per second per GPU. Decode: SGLang FP8 5,226, vLLM MXFP8 6,519 output tokens per second. Real chat p95 time to first token: SGLang FP8 19.6 s, vLLM MXFP8 9.2 s.
| Measure | SGLang FP8 | vLLM MXFP8 |
|---|---|---|
| Uncached prefill (input tokens per second per GPU) | 3,700 | 7,600 |
| Decode (output tokens per second) | 5,226 | 6,519 |
| Real chat, p95 time to first token (s) | 19.6 | 9.2 |
Multi-tier cache for context reuse across agent turns
Agent prompts on GLM 5.3 average 131K tokens. After a tool call, the agent returns tens of seconds or minutes later with almost the same prompt. GPU memory retained that context for only about 4 seconds, falling to 2s under load. Nearly every returning turn required a full prefill, repeating the compute and adding latency. We added G3 to every prefill server to retain context so agents can reuse it when they return.
Over a week in production On DeepSeek, the combined cache hit rate rose from 91.4% to 95.7%, halving the tokens that needed prefill. During peak traffic on GLM 5.3, G3 reduced the required prefill nodes from 58 to 34.
Where prompt tokens were served in production
Share of prompt tokens by where they were served
Where prompt tokens were served in production. DeepSeek V4.1 Flash, 872B prompt tokens: 91.4% from the GPU cache, 4.3% from G3, 4.3% recomputed. GLM 5.3, 164B prompt tokens: 88.6% from the GPU cache, 3.4% from G3, 8.1% recomputed.
| Model | Prompt tokens | GPU cache (%) | G3 (%) | Recomputed (%) |
|---|---|---|---|---|
| DeepSeek V4.1 Flash | 872B prompt tokens | 91.4 | 4.3 | 4.3 |
| GLM 5.3 | 164B prompt tokens | 88.6 | 3.4 | 8.1 |
Aligning the multi-tier cache with new attention architectures
On DeepSeek V4.1 Flash 1017, sliding-window cache lookups and writes used different positions, preventing G3 hits for 74% of prompt lengths. The context was cached but couldn’t be found, so the GPUs processed it again. We aligned the positions, raising the cold cache hit rate from 37.1% to 99.8% and cutting latency from 2.80 s to 0.64 s, with no additional storage. For other hybrid models, we save sliding-window state at every turn boundary in both GPU memory and G3. This reduced returning conversations with no reusable context from 78% to 8.5%.
DeepSeek V4.1 Flash: which prompt lengths hit the cache
37.1% to 99.8%
G3 cold hit rate
2.80 s to 0.64 s
Latency
1,878 B/token
Storage cost, same as the GPU cache, no extra blocks
Not fixed
Upstream: GPU-cache fix #57613 is still open and stores an extra block per boundary
DeepSeek V4.1 Flash: which prompt lengths hit the cache. Each cell is one value of prompt length modulo 128. GPU cache, before, misses at remainders 1 to 32: 25% of lengths miss. G3 store, before, misses at remainders 33 to 127: 74.2% of lengths miss. Both, after, every remainder hits: 0% miss. G3 cold hit rate rose from 37.1% to 99.8%. Latency fell from 2.80 s to 0.64 s. Storage cost is 1,878 bytes per token, the same as the GPU cache. Upstream: not fixed.
| Cache | Remainders that miss | Share of lengths that miss |
|---|---|---|
| GPU cache, before | 1 to 32 | 25% |
| G3 store, before | 33 to 127 | 74.2% |
| Both, after | none | 0% |
Making the control plane cache-aware
Dynamo’s router accounts for context cached on GPU, host memory and disk, but with vLLM and G3, it missed shared Mooncake cache hits and counted cached tokens as new prefill work. Admission limits counted those tokens too, separately per frontend, rejecting 1,381 of 1,400 requests in a cache-heavy test while GPUs sat mostly idle. The planner couldn’t see the rejections, so short queues masked unmet demand. We updated routing to account for G3 hits and congestion, balancing context reuse with capacity. Admission now counts only uncached tokens under one shared in-flight limit across frontends, and the same test completed without rejections.
Router cache-hit accounting and tool-call grammar, before and after
Requests sent to the wrong worker
17.6% to 3.5%
Time to first token, p95
24 s to 9 s
Tool calls leaked as raw text
7.7% to 0.08%
Router cache-hit accounting and tool-call grammar, before and after. At 128 concurrent sessions, requests sent to the wrong worker fell from 17.6% with stock accounting to 3.5% once fixed, and p95 time to first token fell from 24 s to 9 s. DeepSeek V4.1 Flash tool calls leaked as raw text fell from 7.7% with no grammar to 0.08% with a DSML grammar.
| Measure | Before | After |
|---|---|---|
| Requests sent to the wrong worker (stock accounting vs fixed) | 17.6% | 3.5% |
| Time to first token, p95 (stock accounting vs fixed) | 24 s | 9 s |
| Tool calls leaked as raw text (no grammar vs DSML grammar) | 7.7% | 0.08% |
Stock planner and our planner on the same replayed load
Prefill GPU-hours, 104 minutes from load start
47% fewer
Worst minute, p95 time to first token
Inside the SLA
1,050 vs 1,054
requests completed
78.5%
cache hit rate in both runs
0.3 → 0.5 → 1
requests per second over the 30-minute ramp
40 (about 4%)
briefly turned away by ours at the peak
Stock planner and our planner on the same replayed load. Prefill GPU-hours over 104 minutes: Stock planner 32.3, Our planner 17.1. Worst-minute p95 time to first token: Stock planner 19.7 s, Our planner 9.96 s, against a 10 s SLA. Both runs: 1,050 vs 1,054 requests completed, 78.5% cache hit rate in both runs, load ramped 0.3, 0.5, 1 requests per second over 30 minutes; 40 (about 4%) briefly turned away by ours at the peak. Timeline, Stock planner: about 1 min, 2nd pod at 2% busy; about 21 min, p95 19.7 s, 3rd pod at 9% busy; about 105 min, still 2 pods. Our planner: about 20 min, 40 requests get a 529; about 23 min, 2nd pod serving; about 47 min, back to 1 pod.
| Measure | Stock planner | Our planner |
|---|---|---|
| Prefill GPU-hours, 104 minutes | 32.3 | 17.1 |
| Worst-minute p95 time to first token (s) | 19.7 | 9.96 |
| Stock planner event: 2nd pod at 2% busy (min) | 1 | |
| Stock planner event: p95 19.7 s, 3rd pod at 9% busy (min) | 21 | |
| Stock planner event: still 2 pods (min) | 105 | |
| Our planner event: 40 requests get a 529 (min) | 20 | |
| Our planner event: 2nd pod serving (min) | 23 | |
| Our planner event: back to 1 pod (min) | 47 |
New architectures carry state that the handoff between GPUs was dropping
Disaggregated serving was dropping model-specific state in the prefill-to-decode handoff. The transfer code assumed a standard attention KV cache.
Kimi K3.1 Pro and GLM 5.3 Flash’s draft model both carry extra state, and it wasn’t reaching the decode GPU intact.
On Kimi K3.1 Pro, the prefill GPU, router and decode GPU each used different checkpoint positions. Aligning them raised multi-turn cache hits from 39% to 73%.
On GLM 5.3 Flash, only part of the draft model’s cache was transferred, so accepted draft tokens fell from about 5 per step to 1. Matching the cache layout on both GPUs brought acceptance back to 4.7.
Aligning state checkpoints kept the handoff from losing work
Before
Checkpoints at different positions; unmatched ones (hollow) lose state
After
All three place checkpoints at the same 7,680-token boundaries
Kimi K3.1 Pro, multi-turn conversations
39% to 73%
GLM 5.3 Flash drafter, accepted tokens per step
4.92 to 1.05 to 4.69
Aligning state checkpoints kept the handoff from losing work. Before: the prefill GPU, router and decode GPU place state checkpoints at different positions, so only the first and last line up and the rest lose state. After: all three place checkpoints at the same 7,680-token boundaries. Kimi K3.1 Pro multi-turn prefix cache hit rate: 39% misaligned, 73% aligned. GLM 5.3 Flash drafter accepted tokens per step: 4.92 before the split, 1.05 split and misaligned, 4.69 aligned.
| Measure | Before split | Misaligned | Aligned |
|---|---|---|---|
| Kimi K3.1 Pro multi-turn prefix cache hit rate (%) | 39 | 73 | |
| GLM 5.3 Flash drafter, accepted tokens per step | 4.92 | 1.05 | 4.69 |
Bringing it all together to serve Sarvam 105B more efficiently
With Sarvam 105B, we control both the weights and the inference engine, so we can optimise them together for the workloads we serve. Most traffic comes from agents with long system prompts, growing context from tool results and short replies. Moving from BF16 to MXFP8 weights with DSpark speculative decoding increased output throughput by about 2.2× while reducing each replica from two B200s to one. We also fixed a vLLM bug that skipped CUDA graphs after warm-cache restarts, leaving GPU utilisation at 38%. Four kernel and precision changes added another 30% output per GPU on agent workloads. These included 4-bit NVFP4 experts calibrated on real MoE inputs and a custom Triton kernel for 105B’s 64 attention heads, replacing a FlashInfer kernel that assumed 128. When agent sessions exceeded GPU cache capacity, G3 increased throughput by 2.21× and cut the share of prompt tokens needing recomputation from 56.8% to 17.6%. None of these changes reduced accuracy.
Sarvam 105B: cumulative speedup by optimisation step
Sarvam 105B: cumulative speedup by optimisation step. BF16 baseline (2× B200): 1.00× cumulative speedup against the BF16 baseline. Engine kernels (compiled MLA): 1.73× cumulative speedup against the BF16 baseline. torch.compile + CUDA graphs (models team): 2.20× cumulative speedup against the BF16 baseline. NVFP4 experts: 2.42× cumulative speedup, plus 9.8% over the previous step on agentic traffic. Triton K-concat (for MLA): 2.59× cumulative speedup, plus 7.2% over the previous step on agentic traffic. FP8 prefill attn (TRT-LLM ragged): 2.76× cumulative speedup, plus 6.7% over the previous step on agentic traffic. Fused FP8 K/V: 2.86× cumulative speedup, plus 3.6% over the previous step on agentic traffic. The last four steps are chained onto 2.20×.
| Step | Cumulative speedup | Gain over previous step |
|---|---|---|
| BF16 baseline (2× B200) | 1.00× | |
| Engine kernels (compiled MLA) | 1.73× | |
| torch.compile + CUDA graphs (models team) | 2.20× | |
| NVFP4 experts | 2.42× | +9.8% |
| Triton K-concat (for MLA) | 2.59× | +7.2% |
| FP8 prefill attn (TRT-LLM ragged) | 2.76× | +6.7% |
| Fused FP8 K/V | 2.86× | +3.6% |
Kimi K3.1 Pro shows how quickly we can optimise a model we didn’t train. At 1,454 GiB, Kimi K3.1 Pro runs across two nodes using pipeline parallelism. We started with the pre-v0.32 vLLM nightly used in InferenceX’s B200 recipe. On our EFA clusters, the two pipeline stages deadlocked during warm-up. We fixed this by connecting the stages before warm-up and giving both the same warm-up batch. Profiling showed GPUs were busy for 92–97% of each decode step. Expert GEMMs took the largest share; sparse attention accounted for less than 4%. Further gains required reducing the work per step.
Experts and communication take over half of a Kimi K3.1 Pro decode step
Experts and communication take over half of a Kimi K3.1 Pro decode step. One 31.7 ms decode step, share of GPU kernel time: Expert GEMMs (MXFP4 weights) 31.9%; Communication (tensor + pipeline parallel) 24.1%; Dense GEMMs (MXFP8) 17.3%; Expert routing 5.1%; Linear attention (KDA) 4.1%; Attention (gated sparse (DSA) + DCP) 3.8%; Everything else (norms, copies, sampling, other GEMMs) 13.7%.
| Kernel group | Detail | Share of GPU time (%) |
|---|---|---|
| Expert GEMMs | MXFP4 weights | 31.9 |
| Communication | tensor + pipeline parallel | 24.1 |
| Dense GEMMs | MXFP8 | 17.3 |
| Expert routing | 5.1 | |
| Linear attention | KDA | 4.1 |
| Attention | gated sparse (DSA) + DCP | 3.8 |
| Everything else | norms, copies, sampling, other GEMMs | 13.7 |
Every change was measured: we kept the six that paid off
Every change was measured: we kept the six that paid off. Kept: One linear-attention state copy during speculation +22%; CUDA-graph sizes 320, 384, 448 +16.7%; Skip CUDA-graph padding rows in MoE routing +12.7%; Uneven pipeline split across two nodes +9.8%; DSpark with 3 draft tokens instead of 4 +6.2%; MXFP8 for BF16 linears on decode +6%. Flat, not kept: Overlap last stage draft with next step flat; Expert load balancing (EPLB) flat.
| Change tested | What was measured | Measured change (%) | Outcome |
|---|---|---|---|
| One linear-attention state copy during speculation | KV use 40% instead of 99% | +22% | Kept |
| CUDA-graph sizes 320, 384, 448 | no padding a 352-request step to 512 | +16.7% | Kept |
| Skip CUDA-graph padding rows in MoE routing | per user at 256 sessions | +12.7% | Kept |
| Uneven pipeline split across two nodes | throughput | +9.8% | Kept |
| DSpark with 3 draft tokens instead of 4 | at 320 sessions | +6.2% | Kept |
| MXFP8 for BF16 linears on decode | per user on real code, 7.7 GiB freed | +6% | Kept |
| Overlap last stage draft with next step | flat once its NCCL deadlock was fixed | flat | Flat, not kept |
| Expert load balancing (EPLB) | load already 0.96 to 0.98 balanced | flat | Flat, not kept |
Agents explore the search space and engineers decide
Each new attention variant, architecture and workload adds more configurations to test. An orchestrator agent reads fleet telemetry and assigns experiments to sub-agents working on different layers of the stack. They use profilers, a simulator calibrated on our servers and our evaluation suite to test each change. Simulation filters out most candidates before live A/B testing. An engineer approves every change before deployment. This lets us run dozens of experiments per model in days, without engineers setting up each one manually. We build on the work of the teams behind vLLM, SGLang, Dynamo, Mooncake, NIXL, DeepEP, DeepGEMM and FlashInfer.
Availability
Sarvam Token Factory is available today to developers and enterprises. We’ll continue to expand the models available, supporting new open-weight models from day one and adding compute as demand grows. We’re also building more tools for developers to build and deploy agents on Token Factory.
We already run this stack on-premise for critical institutions and offer the same deployment option to enterprises. Teams can choose the models that fit their workloads and build an AI stack they own, without lock-in.
Read More
Curious what else we're building? Explore our APIs and start creating.
Curious what else we're building?
Explore our APIs and start creating.


