Sarvam AI

Introducing Sarvam-105B-1711

·18 min read

Today, we're releasing Sarvam-105B-1711, a new open-weight checkpoint of Sarvam 105B, trained on our upgraded B200 infrastructure. It is a Flash-Lite model built for high-volume work such as customer support, voice agents and agentic tasks. Compared with our March release, it follows instructions more reliably across long conversations, calls tools more accurately and handles Indian languages far better, and its average across eight conversational agent benchmarks rose from 42.6 to 73.9. Our inference optimisations have also cut its blended price by two-thirds, from $0.44 to $0.15 per million tokens. The weights are on Hugging Face in BF16 and a 65 GB NVFP4 version that runs on a single workstation GPU.

  • It matches Gemini 3.5 Flash-Lite and Gemma 4 31B at a fraction of their price. Across the eight conversational agent benchmarks, Sarvam-105B-1711 averages 73.9 at a blended $0.15 per million tokens. Gemini 3.5 Flash-Lite averages 74.0 at $0.85, more than five times the price, and Gemma 4 31B averages 73.0 at $0.54, more than three times the price.
  • The largest gains since March are on conversational agentic skills. Instruction following rose from 32.5 to 82.5 on IFBench, tool use in customer-service conversations from 46.2 to 85.5 on Tau2, and Indian-language performance from 30.1 to 79.1 on our Indic suite. It is also the most consistent model we tested on Tau2, completing 69.5% of customer-service tasks correctly in all four attempts, against 63.9% for Gemini 3.5 Flash-Lite and 62.1% for Gemma 4 31B.
  • This model performs well on real-world tasks inside enterprise applications. Our voice fine-tune powers Samvaad, scoring 68.7 on Samvaad bench (released on HF), up from 57.9 for the base checkpoint and ahead of Gemma 4 31B. In NVFP4 format, the model fits on a single RTX PRO 6000 Blackwell GPU reaching 99 tokens per second per user, so banks, government departments and strategic sectors can run it on their own premises.
  • Training and serving throughput roughly doubled on Blackwell. By training in 8-bit precision, we raised pretraining throughput from 5,500 to 10,900 tokens per second per GPU, ahead of NVIDIA's reference numbers for comparable models, so the same cluster now trains on twice the tokens. In reinforcement learning, faster generation from 8-bit precision and speculative decoding cut each training step from about 13 minutes to about 7, with no drop in reward. In serving, a 4-bit version of the model generates about 6,000 tokens per second per B200 across 64 concurrent users, twice our previous setup and twice Gemma 4 31B, which roughly halves our cost per token.
  • The stack is ready to train larger models. We ran long-context training on 512 GPUs, processing 330 billion tokens in a little over seven days to extend the model's context to 160K tokens. We have also learned to run reinforcement learning on realistic work tasks of the kind in OpenAI's GDPval benchmark, such as summarising a month of shipments from a spreadsheet of purchase orders, with the model using tools and running code in a sandbox. We continue to optimise how we build multi-tool environments for long-horizon tasks as we prepare to bring more training compute online for larger models.

Performance

Sarvam-105B-1711 is a Flash-Lite model. We compare it with other models in this class: two closed models (GPT-6 Luna and Gemini 3.5 Flash-Lite) and three open models (Gemma 4 31B, Nemotron 3 Super and Mistral Small 4), all run on our own evaluation setup. Sarvam-105B-1711 is priced at or below all of them. Flash-Lite models are used for customer support, voice agents, internal assistants, retrieval-grounded answers, and extraction and classification at high volume.

A more capable Sarvam 105B for conversational agents

Indian enterprises spend a large share of their tokens on chat: customer support, voice agents in Indian languages, internal assistants and question answering over company documents, run millions of times a day. Here “chat” covers both plain conversation and agentic chat, where the model calls tools over several turns. To do this well, a model must follow every instruction in a long system prompt across many turns, call the right tool with the right arguments, answer from the retrieved text, and do it all in Indian languages and code-mixed speech.

We measure these skills with eight benchmarks: IFBench, MultiChallenge and Advanced-IF for instruction following; WritingBench and Arena-Hard v2 for chat quality; Tau2 and BFCL v4 for tool calling; and the Sarvam Indic suite for Indian languages. Figure 1 plots their average against price. Sarvam-105B-1711 (NVFP4 checkpoint) averages 73.9 at $0.15 per million tokens, against 74.0 for Gemini 3.5 Flash-Lite at $0.85, 66.3 for GPT-6 Luna (high) at $0.20 and 73.0 for Gemma 4 31B at $0.54.

Conversational agent benchmark average against price

Sarvam-105B-1711Sarvam-105B (March)Other Flash-Lite models
Average of IFBench, MultiChallenge, Advanced-IF, WritingBench (×10), Arena-Hard v2, Tau2, BFCL v4 and the Sarvam Indic suite, against blended price per million tokens. Up and to the left is better. Sarvam-105B-1711 is shown in NVFP4.

Conversational agent benchmark average against price. Sarvam-105B-1711 (NVFP4): 73.9 at $0.15 per million tokens. Sarvam-105B (March): 42.6 at $0.44 per million tokens. Gemini 3.5 Flash-Lite (high): 74.0 at $0.85 per million tokens. GPT-6 Luna (high): 66.3 at $0.20 per million tokens. Gemma 4 31B: 73.0 at $0.54 per million tokens. Nemotron 3 Super: 62.2 at $0.17 per million tokens. Mistral Small 4: 49.7 at $0.26 per million tokens. Qwen3.8 27B (xhigh): 74.6 at $0.96 per million tokens.

Conversational agent benchmark average against price
ModelBenchmark averageBlended price (USD per million tokens)Price source
Sarvam-105B-1711 (NVFP4)73.9$0.15Token Factory ₹10 / ₹25
Sarvam-105B (March)42.6$0.44Blended $0.44
Gemini 3.5 Flash-Lite (high)74.0$0.85Google standard $0.30 / $2.50
GPT-6 Luna (high)66.3$0.20OpenRouter $0.10 / $0.50
Gemma 4 31B73.0$0.54Token Factory list price ₹35 / ₹90
Nemotron 3 Super62.2$0.17OpenRouter $0.08 / $0.45
Mistral Small 449.7$0.26OpenRouter $0.15 / $0.60
Qwen3.8 27B (xhigh)74.6$0.96Blended $0.96

Among the Flash-Lite models, Sarvam-105B-1711 scores highest on IFEval (96.5), IFBench (82.5), Arena-Hard v2 hard prompts (82.9), and is within 0.3 of the best Tau2 average (85.5, against 85.8 for Qwen3.8 27B). On the Indic suite it scores 79.1, within 2 points of the best (80.9, GPT-6 Luna) and well ahead of Qwen3.8 27B (64.8), Mistral Small 4 (44.3) and Nemotron 3 Super (39.5).

Consistency across repeated conversations. For a customer-facing agent, getting the same task right every time matters more than getting it right once. Sarvam-105B-1711 solves 69.5% of Tau2 tasks in all four of four tries, ahead of Gemini 3.5 Flash-Lite (63.9) and Gemma 4 31B (62.1) (Figure 2).

Consistency on repeated customer tasks (Tau2 pass^4)

pass^4 is the share of Tau2 tasks (airline, retail, telecom) a model solves in all four of four independent tries, as defined in the tau-bench paper. Higher is more consistent.

Consistency on repeated customer tasks (Tau2 pass^4). Sarvam-105B-1711: 69.5. Qwen3.8 27B (xhigh): 69.0. Gemini 3.5 Flash-Lite: 63.9. Gemma 4 31B: 62.1. GPT-6 Luna (high): 54.4. Nemotron 3 Super: 49.9. Claude Haiku 4.5: 38.7. Mistral Small 4: 25.5.

Consistency on repeated customer tasks (Tau2 pass^4)
ModelTau2 pass^4
Sarvam-105B-171169.5
Qwen3.8 27B (xhigh)69.0
Gemini 3.5 Flash-Lite63.9
Gemma 4 31B62.1
GPT-6 Luna (high)54.4
Nemotron 3 Super49.9
Claude Haiku 4.538.7
Mistral Small 425.5

Performance on search, knowledge and reasoning tasks

Beyond chat, we track agentic search, cowork tasks, knowledge and math. On agentic work, Sarvam-105B-1711 is second to Gemini 3.5 Flash-Lite on GDPval-S (66.9 against 73.0), OpenAI's GDPval tasks judged by GPT-6 Luna. It is also second to Gemini 3.5 Flash-Lite on our internal cowork tasks (57.1 against 61.4) and close to it on BrowseComp (41.4 against 42.9). On math, it scores 89.7 on AIME 26 without tools, ahead of Gemini 3.5 Flash-Lite, Gemma 4 31B, Nemotron 3 Super and Mistral Small 4, and 94.2 with tools. Table 1 has the full results.

Table 1 Sarvam-105B-1711 against closed and open Flash-Lite models

BenchmarkSarvam-105B-1711$0.15 / M tokensNemotron 3 Super$0.17 / M tokensGPT-6 Luna (high)$0.20 / M tokensMistral Small 4$0.26 / M tokensGemma 4 31B$0.54 / M tokensGemini 3.5 Flash-Lite (high)$0.85 / M tokensQwen3.8 27B (xhigh)$0.96 / M tokens
Instruction Following and Chat
IFEval96.590.888.482.394.795.694.0
IFBench82.570.957.341.077.873.579.6
MultiChallenge62.250.856.730.663.664.865.2
Advanced-IF60.942.457.931.460.264.962.1
WritingBench8.658.737.818.557.928.078.66
Arena-Hard v2, hard prompts82.973.459.748.465.768.979.8
Arena-Hard v2, creative writing65.473.766.365.689.390.282.5
Arena-Hard v2, overall76.673.562.254.675.077.280.8
Tool Calling
Tau2 average85.577.577.555.381.183.185.8
Tau2 airline78.078.076.041.582.582.583.0
Tau2 retail79.674.675.771.375.775.079.8
Tau2 telecom98.979.880.953.185.191.794.7
Tau3 banking, all tools19.816.237.43.618.623.750.3
BFCL v4 overall62.455.460.055.068.867.871.6
Indian Languages
Indic suite79.139.580.944.378.679.764.8
Search and Co-Work
BrowseComp41.427.966.214.533.242.9‡
GDPval-S66.956.8-49.261.173.0†
Internal co-work57.142.8-33.650.361.4†
Knowledge and Reasoning
MMLU-Pro85.285.187.080.688.188.186.9
GPQA Diamond80.478.986.771.084.385.288.7
HLE, no tools12.317.330.69.622.117.930.8*
HLE, with tools16.318.9-7.819.916.5‡
HMMT Feb 2671.876.485.974.477.364.881.6
HMMT Feb 26, with tools87.983.1-56.889.487.187.1
AIME 2689.787.693.787.189.083.293.6
AIME 26, with tools94.289.2-85.095.896.799.2
LiveCodeBench v675.375.585.268.481.973.990.3*
Scroll
Scroll

* Reported officially. † Qwen3.8 27B exhausted its output token budget in 52.6% of GDPval-S rollouts and 50.3% of Internal co-work rollouts. On the remaining rollouts, Qwen3.8 27B scores 82.5 and Sarvam-105B-1711 scores 71.5 on GDPval-S (n=396), and 78.2 vs 61.4 on Internal co-work partial score (n=179). ‡ HLE with tools: Qwen3.8 27B exhausted the 8k per-turn output budget in 68.6% of rollouts. BrowseComp: it returned no final answer in 48.6% of rollouts under the same 8k per-turn limit. Columns are sorted by blended price per million tokens (3 input : 1 output). Sarvam-105B-1711 and Gemma 4 31B: Token Factory list price (₹10 / ₹25 and ₹35 / ₹90, at ₹90 per $). Gemini 3.5 Flash-Lite: Google standard tier ($0.30 / $2.50). GPT-6 Luna ($0.10 / $0.50), Nemotron 3 Super ($0.08 / $0.45), Mistral Small 4 ($0.15 / $0.60) and Qwen3.8 27B (Alibaba Cloud, $0.425 / $2.55): OpenRouter.

Benchmark settings and harness details are in benchmark_config.yaml.

Sarvam-105B-1711 in production

Sarvam offers products across enterprise use cases: Indus Cowork for knowledge work, Samvaad for voice agents in Indian languages, Sarvam Dictate for dictation, and on-prem agents that run on a single workstation GPU. We tried Sarvam-105B-1711 in each of them.

Indus Cowork

On Indus, our co-work product, we found that the model can handle knowledge work across multiple disciplines, including tasks that require dozens of tool calls in a single turn. On harder tasks, it explores more approaches and checks its work more often.

Sarvam-105B-1711 on four real work tasks

Three tasks come from GDPval and one is a multi-turn lab operations conversation. Pick a profession to see the prompt, every command the model ran and the files it produced.

Sarvam-105B-1711 on four real work tasks. Lab operations coordinator: Tracks lab requests over three prompts and builds a review deck. Administrative services manager: Turns a PDF cleanup calendar into a staff memo and an Excel schedule. Medical secretary: Builds Excel request forms and email templates for three pathology labs. Concierge: Researches Napa wineries and writes a four-page Word guide with a photo.

Sarvam-105B-1711 on four real work tasks
ProfessionTask
Lab operations coordinatorTracks lab requests over three prompts and builds a review deck.
Administrative services managerTurns a PDF cleanup calendar into a staff memo and an Excel schedule.
Medical secretaryBuilds Excel request forms and email templates for three pathology labs.
ConciergeResearches Napa wineries and writes a four-page Word guide with a photo.

Samvaad voice agents

Conversational agents deployed in voice applications require low-latency responses with great Indian-language comprehension and tool-use capabilities in no-think mode. We fine-tune Sarvam-105B-1711 for voice applications and deploy it in Samvaad, our platform for enterprise voice agents across 12 Indian languages. The model handles the complexities of real conversations: callers switching languages mid-sentence, revising earlier details, or making ambiguous requests. It must maintain context, ask for missing information, follow client-specific instructions, and choose the appropriate tool calls, all while responding naturally. It also needs to interpret imperfect transcriptions and unexpected tool outputs.

We measure this on Samvaad bench, which is available on Hugging Face. It has 338 next-turn evaluations taken from real-world scenarios in 13 languages: English and 12 Indian languages, often code-mixed. Each example gives the model the system prompt (persona, conversation flow and policy), the conversation so far and the tools it can call, and asks for the next agent action: a spoken reply, a tool call, or both. About half the turns are tool calls with no spoken reply. Most examples come from loan collection and loan sales, with surveys and onboarding making up much of the rest. Each response is checked against yes/no rubrics by an LLM judge (3,340 in total) and, for tool calls, by code checks on the tool name and arguments; a turn passes only if every check that applies passes.

On Samvaad bench, the voice fine-tune scores 68.7 with thinking turned off, against 57.9 for Sarvam-105B-1711 and 68.0 for Gemma 4 31B (Table 2).

Table 2 Samvaad bench, thinking turned off

ModelSamvaad bench
Sarvam-105B-1711, voice fine-tune68.7
Gemma 4 31B68.0
Sarvam-105B-171157.9
Nemotron 3 Super43.3
Scroll
Scroll

The voice fine-tune is scored in NVFP4.

Sarvam-105B-1711 on six voice agent calls

Six voice agent calls in English, Hindi, Bengali and Telugu. Pick a call to hear the caller, then the replies from Sarvam-105B-1711 and GPT-6 Luna, with the transcript, its English translation and the part of the system prompt that decides the right reply.

Sarvam-105B-1711 on six voice agent calls. Hindi, Reading a short reply: Sarvam-105B-1711 reads the caller's short reply as "go ahead" and moves the call on. GPT-6 Luna mistakes it for a bad line. English, A long complaint, then "hold on": The caller ends a long complaint by asking the agent to hold. Sarvam-105B-1711 waits as asked. GPT-6 Luna ignores the request, calls a tool and says nothing. Hindi, Payment follow-through: Sarvam-105B-1711 follows the script and asks for a payment date. GPT-6 Luna re-asks for agreement the caller has already given. Bengali, Following the script: Sarvam-105B-1711 takes "hello" as the cue the script defines and starts the survey. GPT-6 Luna asks for consent a second time. Telugu, When to move on: After two unclear answers, Sarvam-105B-1711 moves on to age as the script says. GPT-6 Luna asks a third time. Hindi, A long, multi-part question: The caller asks three things in one long turn. Sarvam-105B-1711 answers all three from the case details. GPT-6 Luna only calls a tool and says nothing.

Sarvam-105B-1711 on six voice agent calls
LanguageSituationWhat happened
HindiReading a short replySarvam-105B-1711 reads the caller's short reply as "go ahead" and moves the call on. GPT-6 Luna mistakes it for a bad line.
EnglishA long complaint, then "hold on"The caller ends a long complaint by asking the agent to hold. Sarvam-105B-1711 waits as asked. GPT-6 Luna ignores the request, calls a tool and says nothing.
HindiPayment follow-throughSarvam-105B-1711 follows the script and asks for a payment date. GPT-6 Luna re-asks for agreement the caller has already given.
BengaliFollowing the scriptSarvam-105B-1711 takes "hello" as the cue the script defines and starts the survey. GPT-6 Luna asks for consent a second time.
TeluguWhen to move onAfter two unclear answers, Sarvam-105B-1711 moves on to age as the script says. GPT-6 Luna asks a third time.
HindiA long, multi-part questionThe caller asks three things in one long turn. Sarvam-105B-1711 answers all three from the case details. GPT-6 Luna only calls a tool and says nothing.

Sarvam Dictate

Sarvam Dictate is a dictation API: you send audio, and it returns the text the person meant to type, in English and 22 Indian languages. People do not speak the way they type. They say "um", start a sentence twice, correct themselves halfway through, and use names a speech model has never heard. Dictate pairs Saaras v4, our speech recognition model, with Sarvam 105B, which we trained with reinforcement learning to turn the raw transcript into finished text. An app can also send a vocabulary of up to 1,000 names and terms, and Dictate uses a term only where it was actually said. It is built on what we learned from Kivi, our dictation app, where people have spoken more than 20 million words.

On 2,959 real English dictations, Sarvam Dictate needs 5.0% of characters edited to reach what the person meant, against 8.1% for Gemini 3.5 Transcribe and 10.6% for AssemblyAI Dictation. On 692 real dictations in 15 Indian languages the gap is wider: 2.7%, against 12.2% and 11.8%. The finished text arrives about half a second after the audio ends, around 3.5 times faster than Gemini 3.5 Transcribe. It costs ₹0.5 per minute of audio, with speech recognition and the language model included.

Sarvam Dictate has the lowest error rate, at $0.32 an hour of audio

EnglishIndian languages
We measure error rate with AMEND, which counts the character edits needed to match what the speaker meant to write. Lower on the chart means fewer edits. Further left means a lower price. Each system receives the same audio, and two judge models count the edits. Of the languages tested, AssemblyAI supports only Hindi, Marathi and Urdu, so its score covers the 484 dictations in those languages. Prices are in US dollars and use list rates. For Gemini 3.5 Transcribe, which is billed per token, we use Google’s own estimate.

Sarvam Dictate has the lowest error rate, at $0.32 an hour of audio. Sarvam Dictate: 5.0% on English and 2.7% on Indian languages at $0.32 an hour. Gemini 3.5 Transcribe: 8.1% on English and 12.2% on Indian languages at $0.31 an hour. AssemblyAI Dictation: 10.6% on English and 11.8% on Indian languages at $0.62 an hour.

Sarvam Dictate has the lowest error rate, at $0.32 an hour of audio
SystemPrice per hour of audio (US$)Error rate, EnglishError rate, Indian languages
Sarvam Dictate$0.325.0%2.7%
Gemini 3.5 Transcribe$0.318.1%12.2%
AssemblyAI Dictation$0.6210.6%11.8%

Sarvam Dictate and Gemini 3.5 Transcribe on eight dictations

Eight dictations in English, Hindi mixed with English, and Telugu. Pick one to hear the recording, then compare what Saaras v4 heard with the finished text from Sarvam Dictate and from Gemini 3.5 Transcribe (smart mode) on the same audio. Outputs are shown exactly as returned.

Sarvam Dictate and Gemini 3.5 Transcribe on eight dictations. Hindi + English, The step that failed: AWS was the one sign-in that didn't go through, and that is the line Gemini left out. Sarvam Dictate kept it, and wrote the Hindi in Latin letters as the speaker had set. English, The actual question: The speaker is asking if the LLM step can be removed, with only the recording links sent. Gemini's version leaves that question out. Sarvam Dictate keeps it. English, A buried question: Midway, the speaker asks whether to not start with pricing. Gemini drops that, along with "But discuss pricing". Sarvam Dictate keeps the words, though it misses the question mark. English, The full instruction: Gemini wrote only the first five words, and heard "due level" as "D level". Sarvam Dictate wrote the whole instruction and dropped just the repeated "leave it". English, A status update: "Added the secret" is missing from Gemini's version, and it ends with a few words the speaker never said. Sarvam Dictate wrote it as spoken. Hinglish, A Slack message: This was a Slack message, so the speaker had Latin letters on. Gemini has no such setting and wrote Devanagari. Sarvam Dictate wrote it the way the person would type it. Hinglish, The opening line: Gemini skips the first sentence. Sarvam Dictate keeps all of it, in Latin letters, and breaks it into paragraphs. Telugu, A proper name: Sarvam Dictate writes Mandelbrot correctly in Telugu script. Gemini mixes Devanagari letters into the name. When "Mandelbrot" is given as a dictionary term, both get it right.

Sarvam Dictate and Gemini 3.5 Transcribe on eight dictations
LanguageSituationWhat happened
Hindi + EnglishThe step that failedAWS was the one sign-in that didn't go through, and that is the line Gemini left out. Sarvam Dictate kept it, and wrote the Hindi in Latin letters as the speaker had set.
EnglishThe actual questionThe speaker is asking if the LLM step can be removed, with only the recording links sent. Gemini's version leaves that question out. Sarvam Dictate keeps it.
EnglishA buried questionMidway, the speaker asks whether to not start with pricing. Gemini drops that, along with "But discuss pricing". Sarvam Dictate keeps the words, though it misses the question mark.
EnglishThe full instructionGemini wrote only the first five words, and heard "due level" as "D level". Sarvam Dictate wrote the whole instruction and dropped just the repeated "leave it".
EnglishA status update"Added the secret" is missing from Gemini's version, and it ends with a few words the speaker never said. Sarvam Dictate wrote it as spoken.
HinglishA Slack messageThis was a Slack message, so the speaker had Latin letters on. Gemini has no such setting and wrote Devanagari. Sarvam Dictate wrote it the way the person would type it.
HinglishThe opening lineGemini skips the first sentence. Sarvam Dictate keeps all of it, in Latin letters, and breaks it into paragraphs.
TeluguA proper nameSarvam Dictate writes Mandelbrot correctly in Telugu script. Gemini mixes Devanagari letters into the name. When "Mandelbrot" is given as a dictionary term, both get it right.

On-Prem Agents

Agents are most useful where the data is most sensitive, and that is exactly where they are hardest to deploy. A bank cannot send customer ledgers to a third-party API. A hospital cannot let patient records leave its network. Law firms, government and defence bodies, and manufacturers guarding proprietary designs all face the same constraint: the work that would benefit most from an agent is the work that cannot leave the building. Sarvam-105B-1711 removes that constraint. The model runs on a single NVIDIA RTX PRO 6000, which is enough to power Indus Cowork entirely on-premises.

Sarvam 105B on a single RTX PRO 6000 Blackwell with the network switched off, running Indus Cowork as a confidential document agent. It reviews three vendor contracts against a contracting playbook, then writes a risk memo and a presentation. Peak throughput was 197 tokens/s, with an average of 98.

At 65 GB, the NVFP4 checkpoint is the only one that fits on a single 96 GB RTX PRO 6000 Blackwell. On our test workstation we run one copy of the model per RTX PRO 6000 Blackwell Max-Q GPU, with no tensor parallelism, served with vLLM 0.33.0, the DSpark drafter and an FP8 KV cache.

Table 3 gives its speed with 500-token replies.

Table 3 Sarvam-105B-1711 NVFP4 on one RTX PRO 6000 Blackwell, 500-token replies

Prompt lengthConcurrent requestsPer-user decode speed (tokens/s)Total throughput (tokens/s)
10K tokens19994
10K tokens437140
50K tokens16231
50K tokens41444
Scroll
Scroll

Total throughput counts the wait for each request's first token, so with one request it is below the per-user decode speed.

FP8 KV cache. Storing the KV cache in FP8 instead of BF16 fits 1.9× as many tokens, with the same answers on short prompts and on passkey retrieval at about 110K tokens. At 10K tokens both caches have room to spare and perform within 3% of each other. At 50K tokens a BF16 cache holds only two or three requests, so a fourth user waits in a queue: 46 seconds to the first token against 6.5 with FP8, and 28 against 44 tokens/s in total.

A kernel fix for this GPU. vLLM's only MLA decode kernel for this GPU is written in Triton. With an FP8 KV cache it asked for 102,400 bytes of shared memory, above the card's 101,376-byte limit, so the server failed at start-up. Running the kernel with one pipeline stage instead of two cuts that to 83,968 bytes, and decode stays within 1 to 3% of a BF16 cache. The same change is proposed upstream in vllm-project/vllm#54013.

Training and serving Sarvam 105B on Blackwell

We moved Sarvam's training stack to NVIDIA B200 GPUs and their MXFP8 format. Pretraining throughput for Sarvam 105B doubled, from 19% to 38% MFU. Each RL step runs 1.79× faster at the same reward. We serve the model in NVFP4 with a DSpark drafter at up to 6,067 decode tokens/s per B200. Along the way we found that block-scaled attention can leak information from future tokens into past outputs, and built an MXFP8 attention kernel that prevents it.

Pretraining

Our MXFP8 recipe trains Sarvam 105B at 857 TFLOP/s (10.9K tokens/s) per GPU on 64 B200s, at 8K context with a global batch of 256 sequences, up from 428 TFLOP/s (5.5K tokens/s) at the start (Figure 8a). For comparison, NVIDIA's Megatron Bridge numbers for MoE models on B200 are 725 TFLOP/s for DeepSeek-V3 on 256 GPUs and 614 for Qwen3-30B-A3B on 8 GPUs, both in MXFP8 at 4K context.

Pretraining throughput and MFU on B200

a. Throughput on B200 against NVIDIA's published recipes

b. Sarvam 105B long-context CPT, BF16

(a) Pretraining throughput on B200; NVIDIA reference points are from the Megatron Bridge 26.02.01 tables. (b) MFU at each step of tuning the long-context recipe.

Pretraining throughput and MFU on B200. Sarvam MXFP8 recipe 857 TFLOP/s (38.1% MFU) per B200; long-context MFU rose from 7.6% to 24.3%.

Pretraining throughput and MFU on B200
PanelRunSetupTFLOP/s per GPUMFU (%)
aSarvam MXFP8 recipe8K context, 64 GPUs85738.1
aNVIDIA ref: DeepSeek-V3MXFP8, 4K, 256 GPUs72532.2
aNVIDIA ref: Qwen3-30B-A3BMXFP8, 4K, 8 GPUs61427.3
aSarvam 105B long-context CPTBF16, 160K, 512 GPUs54724.3
aSarvam MXFP8 starting recipe8K context, 64 GPUs42819.0
aNVIDIA ref: GPT-OSS-120BBF16, 4K, 64 GPUs37316.6
bH100 recipe32K context7.6
bETP = 132K context21.6
bLess recomp.32K context24.0
bLess logs32K context24.9
bHybridEP32K context26.7
b128KLonger context24.1
b160K, 16 nodesLonger context26.2
b160K, 64 nodesLonger context24.3

MXFP8 started out slower than BF16. MXFP8 speeds up matrix multiplies (GEMMs) and nothing else. Every GEMM input must first be quantised, in two layouts for the forward and backward passes, and activation and gradient copies are rebuilt for every microbatch. With small microbatches, launching these kernels cost as much as the GEMMs saved. Each change in Table 4 moves more of the step into GEMMs.

Table 4 Changes to the MXFP8 pretraining recipe

ChangeWhy it helps
CUDA graphs over the router and token preprocessingMany small kernels replay as one CUDA graph launch. Expert compute stays outside the graph because the number of tokens per expert changes every step.
Token permutation fused into HybridEP dispatch, and a fused CuTe DSL expert MLPRemoves memory-bound passes and intermediate tensors between GEMMs.
Freed memory spent on cutting tensor parallelism from 8 to 1Every GEMM becomes 8× larger, and two TP collectives per layer disappear.
Expert all-to-all overlapped with deferred weight-gradient GEMMsHides expert communication behind work that nothing downstream waits on.
BF16 gradient buffers and larger buckets, with FP32 reductionHalves data-parallel traffic and cuts the number of small collectives.
Scroll
Scroll

Sarvam 105B mixed-precision pretraining recipe on B200 at 8K context

428 → 857 TFLOP/s

Per GPU, from the starting recipe to the final one (+100%)

5.5K → 10.9K tokens/s

Per GPU on 64 B200s, 8K context

Pretraining throughput across successive recipe versions. Measurements exclude checkpointing and evaluation.

Sarvam 105B mixed-precision pretraining recipe on B200 at 8K context. BF16 baseline: 428. + Batching / memory: 475. + MXFP8: 572. + Execution tuning: 666. + BF16 gradient: 721. + EP comm overlap: 763. + TP1 / MBS1: 806. + CuTeDSL: 857.

Sarvam 105B mixed-precision pretraining recipe on B200 at 8K context
Recipe versionDetailTFLOP/s per GPU
BF16 baselineTP8 / MBS4428
+ Batching / memoryTP8 / MBS8475
+ MXFP8linear ops572
+ Execution tuningTP2 / MBS2666
+ BF16 gradienttransport + buckets721
+ EP comm overlap+ delayed wgrad763
+ TP1 / MBS1with overlap806
+ CuTeDSLMLP fusion857

We added the changes one at a time and kept each only if a run of about 1,000 steps showed higher throughput (Figure 9). Coding agents implemented and launched the variants.

Long context. We extended Sarvam 105B to 160K tokens with continued pretraining in BF16 on 330B tokens, using 512 B200s for a little over seven days. Tuning raised MFU from 7.6% to 24.3% (Figure 8b). Two changes mattered most. Keeping each expert whole on one GPU, instead of splitting it across 8 as our H100 recipe did, raised MFU to 21.6%; the split left 256-wide GEMMs too small to keep a B200 busy. In the backward pass, recomputing MLA's up-projections instead of core attention gave 23.6% MFU against 18.0%, since attention recompute costs O(L²) compute to save O(L) memory. We also add context parallelism before pipeline parallelism: its communication hides under attention's compute, while pipeline bubbles do not shrink as context grows.

Numerics. Selected linear-layer GEMMs run in MXFP8, master weights and gradient accumulation stay in FP32, and Adam moments are stored in BF16. We kept FP32 accumulation because BF16 accumulation error grew with the number of microbatches. We compared the recipe with a BF16 baseline on seven model sizes from 0.65B to 69.5B parameters, scaled from the Sarvam 105B architecture, at budgets up to 8B tokens. At the largest sizes validation loss stayed close to the baseline; smaller models were more sensitive to BF16 Adam moments.

Reinforcement learning

Our RL stack is a fork of NVIDIA's NeMo RL. It generates rollouts with vLLM and trains with Megatron, so faster vLLM inference means faster RL. We sped it up in three passes: inference in vLLM, then RL runs, then the trainer.

Faster inference in vLLM

Kernels. We enabled torch.compile for Sarvam's MLA attention and filled in default routing fields so vLLM uses FlashInfer's TRT-LLM MoE kernels. Decode became 1.26 to 1.73× faster and prefill 1.32 to 1.34× faster, at batch sizes 1 to 128 on two B200s in BF16.

Speculative decoding. We generate rollouts with the DSpark drafter described in the Inference section below.

MXFP8. We quantised the model and the drafter to MXFP8 so the pair fits on one B200, with an FP8 KV cache. Accuracy did not regress against BF16.

Bringing it into RL

We measured the gains on asynchronous GRPO for Sarvam 105B on an instruction-following task. The run used 32 B200s, half for generation and half for training, with 1,024 rollouts per step averaging 10.4K tokens; the trainer can learn from rollouts up to three policy versions old. In BF16 a step took 758 seconds, 437 of them waiting for rollouts.

MXFP8. The model fits on one B200 instead of two, so the same 16 GPUs run 16 generation engines instead of 8. Generation throughput rose 1.53× and the step fell to 561 seconds. Attention, the MoE router, the shared experts and the first layer's MLP stay in BF16.

Speculative decoding. With 3 draft tokens and 67% accepted, generation rose another 1.73×, to 33.4K tokens/s, and steps took 420 to 440 seconds. Generation is 2.65× faster overall but the step only 1.79×, because training time did not change. Our inference workers typically ran at 32 to 64 concurrent requests, where DSpark cut end-to-end latency sharply. With GRPO, a step can't finish until the slowest rollouts complete, so lower latency helped remove the stragglers that held up training.

Drafter acceptance held through training. We expected the drafter's acceptance length to drop sharply during RL, as the policy moves away from the model the drafter was trained on. It did not: the DSpark drafter kept its acceptance length over more than 1,000 RL steps.

Accuracy. MXFP8 raised the KL divergence between generator and trainer from 0.030 to 0.041. Speculative decoding added none, since verification keeps the full model's output distribution. Truncated importance sampling corrects the remaining drift. Mean reward was 0.759 in BF16, 0.755 with MXFP8 and 0.759 with both changes.

Removing idle time from the trainer

Each GRPO step waits for rollouts, computes log-probabilities, prepares the batch, trains, and writes a trace to disk. Only batch preparation and training need to block, so we moved the rest off the critical path:

  • Log-probabilities during the wait. Weights only change during training, so we compute log-probabilities for each rollout group as it arrives.
  • Trace writes in the background. Traces were JSON lines written on the training thread: 3.5 GB and an 86.6-second stall per step at 64K context. Compressed Parquet on a background thread is 16× smaller and blocks for about 0.3 seconds.
  • Validation without pausing. We lend 2 of 16 generation engines to validation, frozen on that step's weights. Over 31 steps this cut total time from 210.7 to 180.8 minutes, left the trainer no idle time, and ties each validation score to one policy version.

We upstreamed the vLLM changes: torch.compile for Sarvam MLA, routing defaults for FlashInfer's TRT-LLM MoE kernels, EAGLE3 support and speculative decoding across pipeline stages.

Agentic RL optimisations

Agentic RL adds a harness that runs tools and code in sandboxes. Preloading shared dependencies in persistent harness workers and forking a controller per rollout cut median startup at 1,024 concurrent rollouts from 139 to 11.5 seconds. Adaptive rate limiting smooths bursts of sandbox requests. Once inference was faster, long tool calls became the bottleneck, so we tuned rollout timeouts, off-policy staleness, batch size and the split of GPUs between generation and training together.

Inference

We quantised Sarvam-105B-1711 to NVFP4 and serve it with the DSpark drafter. Table 5 compares the BF16 and NVFP4 checkpoints on the Figure 1 benchmarks.

Speculative decoding. We trained a DSpark draft model for Sarvam 105B and added the vLLM hooks it needs. The drafter proposes several tokens and the full model checks them in one forward pass. This makes inference significantly faster than without DSpark, with no drop in accuracy. On one B200 with the NVFP4 checkpoint, DSpark makes each user's decode 2.0 to 2.6× faster at every load from 1 to 128 concurrent requests, and total throughput 1.9 to 2.4× higher (Figure 10).

Per-user decode speed on B200, with and without DSpark

Sarvam-105B-1711, NVFP4 with DSparkSarvam-105B-1711, NVFP4 without DSpark
Decode tokens/s per user on one B200 with vLLM 0.33.0: held-out prompts, thinking on, temperature 1.0, 128K context, DSpark proposing 7 tokens per step. Labels give the speed-up from DSpark at each concurrency. Because the sampling and draft length differ from Figure 11, the absolute numbers differ too.

Per-user decode speed on B200, with and without DSpark. 1 concurrent: 642 with, 245 without (2.6×). 8 concurrent: 310 with, 145 without (2.1×). 16 concurrent: 254 with, 114 without (2.2×). 32 concurrent: 182 with, 81 without (2.2×). 64 concurrent: 127 with, 60 without (2.1×). 128 concurrent: 85 with, 42 without (2.0×).

Per-user decode speed on B200, with and without DSpark
Concurrent requestsWith DSpark (tok/s per user)Without DSpark (tok/s per user)Speed-up
16422452.6×
83101452.1×
162541142.2×
32182812.2×
64127602.1×
12885422.0×

What is quantised. Sarvam 105B has 31 MoE layers with 128 routed experts each, and those experts hold most of the parameters. We store them in NVFP4: 4-bit values with one FP8 scale per 16 values and one FP32 scale per tensor. Their input activations are quantised the same way at runtime. Attention, the first layer's dense MLP and the shared experts are in MXFP8. The routers and the output head stay in BF16.

Calibration. The 4-bit activations need fixed scales per layer. We take them from conversations generated by the model itself, with thinking on and off. Every routed expert received calibration tokens.

Memory. The checkpoint is 65 GB, against 111 GB in MXFP8 and 198 GB in BF16. On one B200 that leaves room for 3.04M tokens of FP8 KV cache, enough for 23 requests at the full 128K context.

Serving stack. We produce the checkpoint with llm-compressor and serve it with vLLM 0.33.0, which reads the mixed MXFP8 and NVFP4 layout from the config and uses FlashInfer's TRT-LLM NVFP4 MoE kernels. The DSpark drafter is in MXFP8 and ships in the same repository. NVFP4 needs Blackwell's FP4 tensor cores.

Table 5 BF16 and NVFP4 checkpoints on the conversational agent benchmarks

BenchmarkSarvam-105B-1711 BF16Sarvam-105B-1711 NVFP4
IFBench82.581.8
MultiChallenge62.262.5
Advanced-IF60.959.5
WritingBench8.658.66
Arena-Hard v2, overall76.675.3
Tau2 average85.584.1
BFCL v4 overall62.462.7
Indic suite79.178.8
Scroll
Scroll

On held-out data with thinking on, the NVFP4 checkpoint with DSpark decodes 666 tokens/s for a single request. At 64 concurrent requests it reaches 6,067 decode tokens/s per GPU, about 2.1× our BF16 configuration, 1.2× MXFP8 and 2.0× Gemma 4 31B in MXFP8 with MTP (Figure 11). We quantise both weights and activations to 4 bits (W4A4) because vLLM runs that format on FlashInfer's TRT-LLM NVFP4 kernels. With 4-bit weights and 16-bit activations it falls back to the slower Marlin kernels, and throughput drops to 3,043 tokens/s per GPU, below MXFP8.

Serving throughput on B200

a. Throughput against decode speed per user

Sarvam-105B-1711, NVFP4 + DSparkSarvam-105B-1711, MXFP8 + DSparkSarvam-105B-1711, BF16 + DSparkGemma 4 31B, MXFP8 + MTPGemma 4 31B, BF16 + MTP

b. Throughput at 64 concurrent requests

Decode tokens/s per GPU on 1,100 held-out prompts, thinking on, greedy decoding; DSpark proposes 5 tokens per step and Gemma's MTP drafter 3. (a) Each line runs from 1 to 64 concurrent requests. (b) Throughput at 64 concurrent requests.

Serving throughput on B200. At 64 concurrent requests: Sarvam-105B-1711, NVFP4 + DSpark 6,067, Sarvam-105B-1711, MXFP8 + DSpark 5,073, Gemma 4 31B, MXFP8 + MTP 2,994, Sarvam-105B-1711, BF16 + DSpark 2,888, Gemma 4 31B, BF16 + MTP 1,809 tokens per second per GPU.

Serving throughput on B200
ConfigurationConcurrent requestsPer user (tok/s)Per GPU (tok/s)
Sarvam-105B-1711, NVFP4 + DSpark1666609
Sarvam-105B-1711, NVFP4 + DSpark83212,124
Sarvam-105B-1711, NVFP4 + DSpark162452,954
Sarvam-105B-1711, NVFP4 + DSpark321734,484
Sarvam-105B-1711, NVFP4 + DSpark641216,067
Sarvam-105B-1711, MXFP8 + DSpark1534515
Sarvam-105B-1711, MXFP8 + DSpark82431,668
Sarvam-105B-1711, MXFP8 + DSpark161952,479
Sarvam-105B-1711, MXFP8 + DSpark321473,576
Sarvam-105B-1711, MXFP8 + DSpark641135,073
Sarvam-105B-1711, BF16 + DSpark1577274
Sarvam-105B-1711, BF16 + DSpark8259902
Sarvam-105B-1711, BF16 + DSpark162141,301
Sarvam-105B-1711, BF16 + DSpark321692,207
Sarvam-105B-1711, BF16 + DSpark641282,888
Gemma 4 31B, MXFP8 + MTP1307288
Gemma 4 31B, MXFP8 + MTP82321,444
Gemma 4 31B, MXFP8 + MTP161842,138
Gemma 4 31B, MXFP8 + MTP321192,778
Gemma 4 31B, MXFP8 + MTP64672,994
Gemma 4 31B, BF16 + MTP1320150
Gemma 4 31B, BF16 + MTP8250689
Gemma 4 31B, BF16 + MTP161991,148
Gemma 4 31B, BF16 + MTP321311,477
Gemma 4 31B, BF16 + MTP64771,809

This lets us serve Sarvam-105B-1711 at much lower cost. It matches Gemini 3.5 Flash-Lite's conversational agent benchmark average at under a fifth of Google's list price.

Every part of this stack carries over to larger models: MXFP8 pretraining at 38% MFU, long-context training at 24.3% MFU on 512 B200s, RL steps 1.79× faster at the same reward, and NVFP4 serving at 6,067 decode tokens/s per B200. We are now ready to train much larger models.

Getting started with Sarvam-105B-1711

Use the API. Sarvam-105B-1711 is live on Sarvam Token Factory. Create an API key on the dashboard and you can start sending requests right away.

Run it yourself. We've open-sourced the NVFP4 weights at sarvamai/sarvam-105b-1711. This is the exact model we serve on Sarvam Token Factory. Support for efficient inference, including DSpark speculative decoding, is already merged into vLLM, and the model card lists the hosting settings we use in production.

Train it further. For anyone who wants to fine-tune or post-train the model, we've also released the BF16 weights at sarvamai/sarvam-105b-1711-bf16. Post-training support is already merged into NeMo RL, so you can start reinforcement learning runs today, including in FP8.

Read More

Curious what else we're building? Explore our APIs and start creating.