Voice AI is in vogue, and the clearest signal is where the industry’s giants are putting their model weights. The newest example is now official: Nvidia open transcription is live on Hugging Face, in the shape of Nemotron 3.5 ASR Streaming, a 600-million-parameter speech model that turns audio into text across 40 language-locales in real time, under an open license that allows commercial use.
It is the newest Nvidia open transcription release, and it landed in a loud season for voice. Microsoft moved its MAI-Transcribe model into MAI Playground, ElevenLabs topped the text-to-speech arena with Eleven v4, and Meta rebuilt its entire strategy around open-weight models. Transcription, the unglamorous layer between a spoken word and a usable token, has quietly become competitive infrastructure.
That makes this a release worth unpacking. This guide covers what the new Nvidia open transcription model actually is, how its cache-aware streaming architecture works, what the published numbers say about latency and cost per stream, how it stacks up against its rivals, and what voice teams should pilot first.
Table of contents
- Why Voice AI Is In Vogue Right Now
- Meet The Nvidia Open Transcription Model
- Inside The Nvidia Open Transcription Architecture
- Forty Language-Locales In Three Tiers
- Latency, Throughput And The Cost Per Stream
- Granary-v2: The Data Behind The Nvidia Open Transcription Push
- How The Nvidia Open Transcription Model Stacks Up
- What Voice Builders Should Do Now
- Risks And Open Questions
- Frequently Asked Questions About Nvidia Open Transcription
- References
Why Voice AI Is In Vogue Right Now
Voice Is Becoming The Default Interface
For most of a decade, voice was a novelty bolted onto products built for typing. That inversion is underway across the whole industry. Meeting assistants transcribe every call, contact centres route millions of conversations through automatic speech recognition, dictation has replaced keyboards for field workers, and voice agents now place and receive real phone calls for banks, clinics, and retailers.
The traffic follows the interface. Products such as the hidden voice tab previewed in Claude show that even text-first assistants now treat speaking as a first-class mode. When user behaviour shifts this visibly, model vendors reorganise their roadmaps around ears and mouths, and transcription becomes the metered raw material that a new Nvidia open transcription model addresses head-on.
Open Weights Raised The Stakes
Accuracy alone stopped being a moat once open models caught the leaders. A developer can download a research-grade transcription model today, fine-tune it on domain audio, and serve it on their own hardware, without ever signing a speech-API contract. That is why the big labs keep releasing stronger open checkpoints, and why the Nvidia open transcription release belongs in the same sentence as Meta’s and Microsoft’s: influence, ecosystem lock-in at the framework level, and a public scoreboard.
Nvidia sits in a peculiar and powerful position in that race, and the Nvidia open transcription line is how it presses the advantage. It sells the GPUs the industry runs on, and its open models are tuned to squeeze more throughput out of exactly those cards. The release of the Nemotron 3.5 Super VL family showed the same playbook in vision-language; now the Nvidia open transcription line gets the same treatment for speech.
Meet The Nvidia Open Transcription Model
What Nvidia Actually Shipped
The model behind the Nvidia open transcription headline is Nemotron 3.5 ASR Streaming 0.6B, published on Hugging Face and mirrored in the NeMo framework. It is a cache-aware FastConformer-RNNT system with 24 encoder layers, native punctuation and capitalisation, and configurable chunk sizes of 80, 160, 320, 560 and 1120 milliseconds. One deployment can therefore be a near-instant captioner or a high-throughput batch transcriber without retraining.
It is the multilingual extension of the English-only Nemotron speech streaming model Nvidia shipped earlier, rebuilt around language-ID prompt conditioning. The stated goal is blunt: one model that replaces forty per-language pipelines, transcribing cleanly from the highest-accuracy locales down to long-tail ones.
A Family Tree, Not A Lone Model
The release belongs to the Nemotron Speech line, which spans automatic speech recognition, text-to-speech, speaker diarisation and speech-to-speech. Its siblings include the Parakeet family, whose parakeet-tdt-0.6b-v2 and v3 checkpoints have become default picks for offline transcription, and the Canary models, which combine transcription and translation across 25 languages and were trained on Nvidia’s Granary dataset.
Read this way, the Nvidia open transcription model is less a single product than the newest branch of a full speech catalogue, refreshed continuously on Hugging Face. That catalogue approach is the same one Nvidia applies to its language models, and it is how the company hopes developers end up standardising on NeMo.
What “Open” Buys You In Practice
The weights ship under the OpenMDW-1.1 license, the permissive “do what you want with attribution” style terms Nvidia now uses for its model weights. That means free commercial use, fine-tuning, and redistribution of adapted checkpoints, with no usage-tier negotiation and no clause that flips once you find product-market fit.
Nvidia also published a fine-tuning walkthrough for this exact model, with before-and-after results for improving weaker locales. For buyers comparing vendors, that combination of open weights, training recipes and published evaluation is the honest version of the Nvidia open transcription race: you can verify every claim on your own audio before signing anything.
Inside The Nvidia Open Transcription Architecture
Cache-Aware Streaming Kills The Overlap
Traditional streaming speech recognition is wasteful. Buffered systems re-process overlapping windows of audio, so the same frames get computed again every time the buffer slides forward. The Nvidia open transcription model instead keeps caches for every self-attention and convolution layer in its encoder, reusing the hidden states from previous steps.
The consequence is that each processed frame is strictly non-overlapping: the model only ever computes on new audio while still carrying full context. Streaming accuracy stops being a tax on the architecture, and server capacity stops evaporating into recomputation.
FastConformer Up Top, RNNT Down The Bottom
The encoder is a 24-layer FastConformer that compresses audio into an acoustic embedding of 1024 dimensions per timestep. A separate language encoding expands a 128-dimensional one-hot vector across the time axis, and that vector is concatenated with the audio features and projected into an RNNT decoder, which emits text incrementally.
Because the decoder is a transducer, it can output tokens as the speaker talks instead of waiting for the end of an utterance. Punctuation and capitalisation arrive inside the same pass, so there is no post-processing model to host, patch, or pay for anywhere in the Nvidia open transcription pipeline.
One Model, Forty Languages, One Prompt
Language choice is a prompt: callers pass a locale such as es-ES or de-DE, and the same weights transcribe that language. Set target_lang to auto and the model detects the spoken language itself and appends the detected language tag after the terminal punctuation, which lets a single deployment swallow mixed-language call queues.
There is no separate language-identification component to train, host, or debug. For teams that previously ran a language detector in front of per-language transcription models, the entire front stage collapses into one call into the Nvidia open transcription stack.
Forty Language-Locales In Three Tiers
Coverage is organised into three tiers, from fully production-ready locales to ones that need a fine-tuning pass first. English arrives in both US and GB spellings, and the ready tier reaches across European, Middle Eastern, South Asian and East Asian languages, with Japanese and Korean evaluated in character error rate, as is standard for those scripts.
| Tier | Locales | What it means |
|---|---|---|
| Transcription-ready (19) | en-US, en-GB, es-US, es-ES, fr-FR, fr-CA, it-IT, pt-BR, pt-PT, nl-NL, de-DE, tr-TR, ru-RU, ar-AR, hi-IN, ja-JP, ko-KR, vi-VN, uk-UA | Highest-accuracy ASR, out of the box |
| Broad-coverage (13) | pl-PL, sv-SE, cs-CZ, nb-NO, da-DK, bg-BG, fi-FI, hr-HR, sk-SK, zh-CN, hu-HU, ro-RO, et-EE | Production ASR across an additional 13 locales |
| Adaptation-ready (8) | el-GR, lt-LT, lv-LV, mt-MT, sl-SI, he-IL, th-TH, nn-NO | Recognised by the tokenizer; fine-tune on in-domain data to unlock |
The Ready Tier Carries The Pitch
Nineteen locales at launch is a serious claim, and it reframes the model as a single replacement for a fleet of single-language deployments. The list also signals Nvidia’s priorities: strong European coverage, Arabic and Hindi, plus Japanese and Korean, the three language groups most enterprise voice roadmaps ask for first.
Broad Coverage And Adaptation Are Different Promises
The broad-coverage tier produces usable transcriptions immediately, with the caveat that accuracy will trail the ready tier on some accents and domains. The adaptation-ready tier is honest about its limits: Greek, Thai, Hebrew and five others exist in the tokenizer today, and Nvidia’s own documentation says to fine-tune before trusting them.
That third tier is exactly where open weights shine. Instead of waiting for the vendor’s next release, teams can train the long-tail locales they actually need on their own recorded calls, using the published recipes. The Nvidia open transcription model treats that self-serve path as a supported workflow, not a hack.
Language Detection As A Built-In Feature
With automatic detection enabled, every utterance comes back tagged with its language code after the final punctuation mark. Routing, analytics and compliance filtering can key off that tag directly, and code-switching speakers, a chronic failure case for per-language pipelines, get handled inside the same model.
Latency, Throughput And The Cost Per Stream
The model exposes its latency knob at runtime: chunk sizes from 80 to 1120 milliseconds trade response delay against accuracy and concurrency. Bigger chunks see more context, so accuracy climbs, while smaller chunks keep near-instant turnaround for live captions and conversational agents.
Nvidia’s own benchmarks make the throughput argument concrete. On a single H100, the cache-aware model sustains roughly seventeen times more concurrent live streams than its own buffered Parakeet RNNT 1.1B at the 80ms setting, as the chart below shows.
The Headline Numbers
| Chunk size | Nemotron 3.5 ASR (0.6B) | Parakeet RNNT 1.1B (buffered) | Advantage |
|---|---|---|---|
| 80ms (lowest latency) | 240 concurrent streams | 14 concurrent streams | ~17x |
| 1120ms (highest throughput) | 2,400 concurrent streams | 400 concurrent streams | 6x |
Why The Gap Matters More Than The Score
Word error rate charts get the headlines, but concurrency is what shows up on invoices. A service that sustains 240 live streams per GPU instead of 14 cuts the infrastructure behind a voice agent fleet by an order of magnitude, and it delays saturation: the Nemotron curve holds low final-token latency well past 1,000 parallel requests while the buffered model saturates after a few hundred.
For anyone pricing voice products, that arithmetic changes what a minute of conversation costs. Lower GPU time per stream means lower per-call margins demanded, which is precisely the lever Nvidia pulls when it tells enterprises that open models are cheaper than closed APIs, not merely freer.
Granary-v2: The Data Behind The Nvidia Open Transcription Push
463,062 Hours, 48 Languages
Model releases do not happen in isolation, and days before this coverage window the team refreshed the dataset that feeds them. Granary-v2, the second release of Nvidia’s generated-label corpus, now covers 48 languages across 16 source corpora, with labels for about 463,062 hours of speech, of which roughly 165,300 pass the pipeline’s own quality gate.
The translation targets are equally large: approximately 92,496 hours of X-to-English speech translation, and about 39,375 hours of English-to-X pairs fanned out to 31 target languages at once. Those are the numbers behind a single multilingual model covering forty locales.
Six Text Targets Per Utterance
Granary-v2 re-processes everything through a new labelling pipeline that produces six targets per utterance: a normalized transcript, inverse-text-normalized form, captioning labels, code-switched variants, speech question-answer pairs, and translations. The instruction-ready pairs mean one corpus trains not just transcription but the whole instruction-following speech stack.
Corpora on the include list read like a who’s who of open speech data: VoxPopuli re-labelled with MOSEL labels, LibriLight, People’s Speech, Multilingual LibriSpeech, Common Voice 26.0, and Singapore’s National Speech Corpus among them, and they are why the Nvidia open transcription line keeps improving quarter over quarter.
The Pipeline Ships Open Too
Unusually, Nvidia published the labelling pipeline itself inside NeMo Curator, with the end-to-end runs to follow. Competitors can rebuild the dataset recipe, auditors can trace where a label came from, and researchers can argue with the quality gate instead of guessing at it. It is the same transparency move as the open weights, extended upstream into the data.
How The Nvidia Open Transcription Model Stacks Up
The obvious comparisons are Nvidia’s own older checkpoints, and the open field outside. The table below contrasts the new release with the family it replaces and with the most-installed open transcriber in production today, OpenAI’s Whisper large-v3.
| Model | Params | Streaming | Languages | Weights license |
|---|---|---|---|---|
| Nemotron 3.5 ASR Streaming | 0.6B | Native cache-aware, 5 runtime chunk sizes | 40 locales in 3 tiers | OpenMDW-1.1 |
| Parakeet RNNT 1.1B | 1.1B | Buffered | Multilingual | Open (Nvidia terms) |
| Canary-1B v2 | 1B | Offline | 25, transcription plus translation | OpenMDW-1.1 |
| Whisper large-v3 (OpenAI) | 1.55B | Offline (VAD-segmented hacks) | ~100 | MIT |
Against Its Own Family
The cleanest benchmark Nvidia published is against itself: the new 0.6B model versus the 1.1B Parakeet RNNT it is designed to replace. Half the parameters, dramatically more concurrency, and FLEURS accuracy that improves with chunk size while staying competitive at the lowest-latency setting. For existing NeMo users, that is an easy upgrade path, and the model card even recommends which checkpoint to pick per use case.
Against The Wider Open Field
Outside the Nvidia open transcription family, the field is broad but mostly batch-first. Whisper remains the most-deployed open transcriber and covers far more languages, with an MIT license and a vast ecosystem of ports. But it was designed as an offline model; real-time deployments wrap it in voice-activity segmentation that adds latency and misses mid-sentence turns. The Nvidia open transcription model attacks exactly that seam: native streaming, native punctuation, native language tagging.
Against Microsoft’s MAI-Transcribe line, the difference is deployment philosophy. Microsoft pushes transcription into its managed playground and cloud; Nvidia ships weights you can run anywhere and hands you the training recipes. For regulated or cost-sensitive organisations, the Nvidia open transcription release is the stronger default in the current field.
Where Rivals Still Win
Honest ledger, other side. Teams running purely offline captioning pipelines already have superb open options, like the Parakeet checkpoints used in our own SRT auto-caption workflow, where the streaming machinery adds nothing. And the adaptation-ready tier means eight advertised languages are not truly production-ready until someone fine-tunes them, so the full forty is a promise, not a present.
What Voice Builders Should Do Now
Pilots Worth Running This Quarter
Three pilots expose the real value quickly. First, replay a week of recorded calls through the batch mode and compare word error rate against your current transcriber on your own accents. Second, wire the 80ms chunk mode into one live customer-facing flow and measure end-to-end delay humans actually perceive. Third, test the language-tagging output against your routing rules.
If your volumes are meaningful, add the arithmetic: at Nvidia’s concurrency figures, the same fleet of calls that saturated several buffered GPUs fits on one or two cards, and that delta lands directly in the gross margin of every voice feature you ship.
Fine-Tune The Languages You Actually Need
The published fine-tuning walkthrough gives before-and-after numbers for lifting weaker locales, and the OpenMDW license means those adapted checkpoints are yours to deploy or redistribute. Long-tail-language teams finally hold the whip hand: instead of waiting for vendors, you train on your own domain audio and keep the result.
Three Deployment Paths
Developers can go three ways: NeMo for training and research-grade pipelines, Speech NIM microservices for containerised production endpoints, and NeMo-Speech.cpp, a lightweight native C++ runtime with quantised GGUF checkpoints, for laptops, edge boxes and kiosk devices. The same Nvidia open transcription weights underpin all three, so prototypes never strand. You can track the rest of this release wave in our AI Models, Tools & Releases hub.
Risks And Open Questions
The Benchmark Trap
Word error rate on FLEURS is a reasonable yardstick and an easy one to overfit a roadmap against. Real contact-centre audio, with crosstalk, hold music, bad headsets and a dozen accents per call, punishes models that post beautifully on clean evaluation sets. Judge any Nvidia open transcription claim, or anyone else’s, on your own recordings before it becomes load-bearing.
License Fine Print And Provenance
OpenMDW-1.1 is permissive, but the provenance of training data still invites scrutiny, especially where corpora carry non-commercial or attribution clauses upstream. Nvidia documents source corpora and their licenses for Granary-v2, which is more transparency than most, yet enterprise legal teams should still read that table before shipping.
Consent, Recording And Voice Rights
Cheap, accurate, real-time transcription of everything raises the same old alarms louder. Wiretapping law, consent rules for recorded calls, and the emerging case law on voice itself, like the Japanese court ruling that human voices are protected, all sit downstream of the model. The technology keeps getting better; the governance around it has not.
Frequently Asked Questions About Nvidia Open Transcription
Is the new Nvidia transcription model free for commercial use?
Yes. Nemotron 3.5 ASR Streaming ships under the OpenMDW-1.1 license, which permits commercial use, modification and redistribution of the weights. The model card states plainly that it is ready for commercial use, and there is no usage-tier switch once you scale.
How many languages does the Nvidia open transcription model support?
Forty language-locales in three tiers: 19 transcription-ready locales including English, Spanish, French, German, Arabic, Hindi, Japanese and Korean; 13 broad-coverage locales such as Polish, Swedish and Mandarin; and 8 adaptation-ready locales including Greek, Hebrew and Thai that need fine-tuning first.
Does it work in real time, or only on recorded audio?
Both. It is built as a streaming model: cache-aware chunks from 80ms to 1120ms, incremental output from the RNNT decoder, punctuation and capitalisation in the same pass, and optional automatic language detection. Batch transcription of recorded corpora is simply the high-throughput end of the same model.
How is Nemotron 3.5 ASR different from Parakeet and Canary?
Parakeet checkpoints are Nvidia’s FastConformer ASR family, strongest offline, and Canary adds multilingual transcription with translation. Nemotron 3.5 ASR is the cache-aware streaming branch: smaller than Parakeet 1.1B, far more concurrent streams per GPU, and language prompting across 40 locales instead of one deployment per language.
References
Nemotron 3.5 ASR Streaming 0.6B model card (Hugging Face)
Nemotron Speech collection (Hugging Face)
Granary-v2 multilingual speech dataset (Hugging Face)
Granary-v2 technical report (NVIDIA Research)
Granary-v2 labelling pipeline in NeMo Curator (GitHub)
Fine-tuning Nemotron 3.5 ASR with before/after results (Hugging Face blog)
NVIDIA NeMo framework (GitHub)
NeMo-Speech.cpp native runtime (GitHub)
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.
