After bleeding cash on dense LLMs, I’ve moved our agents to Nemotron 3 Nano

The night our “smart” agents melted our GPU budget

A few months ago, I sat in a 10 p.m. incident review staring at a Grafana dashboard that looked like a cardiogram on fast‑forward. Our “autonomous” agents were supposed to be quietly triaging logs and drafting remediation plans. Instead, a minor spike in production errors had triggered a feedback loop of nested tool calls, 70B‑parameter completions, and 32K‑token RAG queries that slammed our H100 cluster to 100% utilization for nearly an hour.

The bill for that one episode cleared five figures. Latency for everything else went to hell. And the kicker? When we inspected the traces, half the work could have been done by a mid‑sized model with a long context window and decent tool‑use, not a dense behemoth tuned for benchmark glory.

That was the night I drew a line: no more blindly throwing dense 70B models at agentic workflows. If an “enterprise AI strategy” cannot survive a single noisy incident without burning through GPU capacity, it is not a strategy. It is a very expensive science project.

That decision is what pushed us to take NVIDIA’s Nemotron 3 Nano seriously. I went in skeptical. I came out having re‑architected our default stack around it.

My thesis: Nemotron 3 Nano is the first open agentic model that actually respects ops reality

I am not here to gush about another “SOTA” model. Benchmarks are cheap; operating something at scale, under SLA and budget constraints, is expensive. Nemotron 3 Nano is the first open‑weight model where the architecture, deployment story, and economics line up for agentic systems in 2025.

On paper, the headline is simple: Nemotron 3 Nano is a roughly 30B‑parameter hybrid Mamba-Transformer sparse MoE, but it only activates about 3B parameters per token. It delivers:

• Roughly 3.3x faster inference than Qwen3‑30B‑A3B on H200‑class GPUs,
• Up to 1,000,000‑token context windows without KV‑cache explosions, and
• Throughput in the ballpark of 250-300 tokens/second on a single H100, or 2,200-2,500 tok/s on an 8x H100 cluster.

Those are impressive numbers, but they are not why we reorganized our roadmap around it. The real reasons are more operational:

• It loads in roughly 15 GB of active VRAM while downloading about 60 GB of weights, making single‑GPU testing entirely realistic.
• The sparse MoE and FP8 / NVFP4 quantization story drive a 2–4x reduction in inference cost versus dense 30B models at similar or better reasoning quality.
• It comes with an actually usable toolkit: NeMo integration, function calling, reasoning‑budget controls, and documented mitigations for the usual “week one” failure modes like Mamba kernel mismatches and Flash Attention OOMs.

In other words: it is not just a clever paper. It is deployable. And in 2025, that is a rarer quality than most vendors are willing to admit.

What changed when we swapped our agents to Nemotron 3 Nano

We did not start with a clean slate. Before Nemotron 3 Nano, our production agent stack was a messy mix of:

• A dense 70B model for complex reasoning and long‑form plans, accessed via API.
• A 30B open‑weight model (Qwen‑class) running on our own H100s for mid‑tier tasks.
• A 7–8B class model for quick intent detection and routing.

The pattern was predictable: agents escalated to the dense model the moment tasks got even slightly hairy. The 30B model was a partial relief valve, but anything involving multi‑step tool use, large RAG contexts, or code synthesis ended up on the costly path.

We replaced that mid‑tier 30B with Nemotron 3 Nano and deliberately re‑routed as many agent workflows as possible to it first. For three of our heaviest pipelines, the results were immediate:

• Support & incident triage agents: 1M‑token context meant we could stuff entire multi‑day log windows and historical tickets into a single prompt instead of chunking and re‑querying. Average tokens per incident went up ~40%, but the cost per incident dropped by ~55% because we were doing it on Nano at sparse‑MoE prices instead of bouncing between multiple dense calls.
• Coding & refactoring agents: We ran a set of internal competitive programming and refactoring suites we had previously used to compare Llama‑3.1‑70B against our in‑house 30B. Nemotron 3 Nano either matched or edged out the 70B on structured reasoning tasks, while halving the GPU time required.
• Analytics & log exploration agents: The ability to hold hundreds of thousands of tokens of logs in context changed how we designed the system. Instead of iterative search‑and‑answer loops that hammered the vector store, we could do single, coherent analysis passes. Latency for complex investigations dropped from minutes to under a minute on an 8x H100 pod.

When we did the month‑over‑month numbers, the picture was blunt: for these agentic workloads, Nemotron 3 Nano became our default. The dense 70B stayed in the picture only for niche tasks where we could directly measure a quality lift that justified its much higher cost.

The architecture that finally uses sparsity like operators actually need

Most MoE marketing over the last few years has been a lie by omission. “256B parameters!” sounds fantastic until you realize you are activating almost all of them during inference, blowing through memory, and suffering weird tail latencies from imbalanced expert routing.

Nemotron 3 Nano’s hybrid Mamba–Transformer sparse MoE is the first time I have seen sparsity deployed in a way that feels designed for operators rather than for leaderboard screenshots. You get:

• ~3B active parameters per token, consistently. That is a scale you can plan around when you are sizing GPU memory and scheduling.
• Mamba layers that handle very long‑range dependencies without the quadratic blow‑up in compute and memory you get from pure attention on million‑token contexts.
• Transformer attention where it matters, paired with modern kernels like Flash Attention for the hardware that can support it.

Add FP8 and NVFP4 quantization, and suddenly the phrase “1M‑token context” is not a science demo but something you can actually leave turned on for an always‑on service. The MoE routing is surprisingly well‑behaved: we had some early imbalances, but the NeMo recipes include router tweaks that kept expert utilization within a tolerable spread.

And then there is Multi‑Token Prediction (MTP). For the Super/Ultra roadmap, NVIDIA is leaning into decoding multiple tokens per step without tanking coherence. In internal tests with an early Nano build using MTP‑style layers, we saw around 1.5x generation speedups at the same perceived quality. Remember when I mentioned the cost implications of agent loops hammering dense models? This is where that latency headroom starts compounding into real budget savings.

Cost and hardware: where the numbers stopped being theoretical

Let us talk about the part the board actually cares about: total cost of ownership.

On the infra side, the setup looks roughly like this for us:

• Single‑GPU testing: An 80 GB H100 or B200 handles Nemotron 3 Nano comfortably. Peak VRAM use sits in the mid‑teens of GB thanks to the 3B active parameters per token. Throughput for our typical 8K‑in / 16K‑out prompts sits around 250–300 tok/s.
• Production pods: 8x H100 in a DGX‑class or DGX Cloud‑style setup, configured with tensor parallelism across all eight GPUs. We see aggregate throughput in the 2,200–2,500 tok/s range with 1M‑token context enabled.

GPU time on major clouds for H100‑class hardware typically prices in around $2–4 per hour with committed usage. At those rates, and with the above throughputs, our internal math puts Nemotron 3 Nano’s cost per million tokens at roughly:

• On our own DGX‑style hardware: about $0.15–0.25 per million tokens end‑to‑end, including some overhead and inefficiency.
• Through a managed provider like Together AI: ballpark list pricing of $0.20 per million input tokens and $0.60 per million output tokens, still comfortably below what we pay for comparable dense 30B offerings.

Now here is the uncomfortable question: if you can get 3x the throughput and 2–4x better cost per token at similar or better reasoning quality, why are you still paying dense‑model tax for every agent call?

The honest answers I hear from peers are usually some mix of “integrations are easier with the big closed models,” “our team does not have GPU expertise,” or “benchmarks say the 70B is better.” All valid concerns. But Nemotron 3 Nano is the first open‑weight system where those excuses are starting to collapse:

• Native hosting on Together AI and NVIDIA DGX Cloud solves the “we do not run GPUs well” argument.
• NeMo and the usual Python orchestration ecosystem (LangGraph, CrewAI, function‑calling APIs) make integration as straightforward as any of the major APIs.
• On reasoning‑heavy benchmarks-coding, math, multi‑step planning-Nano is right there with or ahead of other 30B models and competitive with larger dense ones.

Once you put real latency and cost numbers next to your agent traces, the appeal of “just call the biggest model” starts to vanish.

The deployment landmines-and how we defused them

None of this means Nemotron 3 Nano is plug‑and‑play magic. We hit plenty of bumps moving from lab to production, but they were refreshingly mundane compared to some of the horror stories from bleeding‑edge models.

• Mamba layer mismatches: Our first attempt to run Nano on a slightly mismatched PyTorch stack ended in cryptic kernel crashes. Pinning the mamba-ssm version to a known‑good release and ensuring BF16/FP8 support on the GPU solved it. Lesson: treat the hybrid backbone like a first‑class dependency, not an afterthought.
• Flash Attention OOMs on smaller cards: On 40 GB‑class GPUs and some 4090s, enabling Flash Attention with very long contexts was a fast path to out‑of‑memory errors. Our fix was pragmatic: fall back to scaled‑dot‑product attention on those SKUs, and use FP8 quantization aggressively. We took a small throughput hit but kept the deployment viable.
• MoE routing imbalance: In early runs, a handful of experts were getting hammered while others idled. Following NVIDIA’s router balancing recipes—temperature tuning and load‑balancing loss terms—brought the spread down to acceptable levels and smoothed out our tail latencies.
• Context management for agents: Just because you can stuff 1M tokens into context does not mean you should let your agents do it blindly. We had to add explicit budgeting and summarization policies: for many workflows, 200–300K high‑signal tokens beat a raw million‑token dump.

The through‑line here is important: these are fixable, reasonably well‑documented problems. Contrast that with the regressions we have seen from chasing every new model release with opaque weights, shifting APIs, and no control over the inference stack. As an operator, I will happily trade a week of kernel and router tuning for a year of predictable costs and behavior.

Where Nemotron 3 Nano is the wrong tool

Let me be clear: I am not claiming Nemotron 3 Nano is the right answer for every use case. There are real scenarios where it is overkill or simply the wrong fit.

• Ultra‑low‑VRAM environments: If you are trying to serve models on 16–24 GB GPUs or edge hardware, a 30B‑class sparse MoE is still heavy. You are better off with a well‑tuned 8–14B model.
• Latency‑critical single‑shot inference: For sub‑50ms mobile UX or on‑device completion, Nano is not your friend. You want tiny distilled models, not a hybrid Mamba–MoE beast.
• Highly specialized creative domains: For some creative writing or niche domain tasks, bespoke fine‑tuned models still beat big generalists. Nemotron 3 Nano can be fine‑tuned, but if your workload is narrow enough, you might not need this much capacity.

But for the class of problems I care most about—agentic systems that orchestrate tools, reason over large corpora, and maintain long‑horizon state—Nano has become my default. The burden of proof has flipped: if someone wants to put a different model into that critical path, they now have to prove it beats Nemotron 3 Nano on throughput, cost, and quality, not just benchmarks.

Why I’m aligning our roadmap with NVIDIA’s Nemotron line

The other reason I am comfortable standardizing on Nemotron 3 Nano is that it is clearly not a one‑off drop. NVIDIA is treating Nemotron as a first‑class product line with a roadmap that actually matters to operators:

• Nemotron 3 Super & Ultra: Planned variants that retain the same basic architecture but add more sophisticated LatentMoE structures and more aggressive NVFP4 quantization. The claim is roughly +20% accuracy and about 2x throughput over Nano on next‑gen GPUs like B200 and H200.
• Multi‑Token Prediction everywhere: Making MTP a standard part of the decoding stack gives us a clearer latency roadmap. If we can shave another 30–50% off generation time without sacrificing coherence, entire classes of synchronous agent interactions become economically viable.
• Tight NeMo and DGX integration: Having a single vendor own the silicon, the kernels, the model architecture, and the deployment toolkit is not just a vertical integration play—it is an operational simplification. I get why people fear lock‑in, but as someone on the hook for uptime and costs, I will take a coherent stack over a Franken‑pipeline of half‑integrated components any day.

Will every prediction in that roadmap land exactly on time? Of course not. But the direction is obvious: more sparsity, more quantization, longer contexts, and more aggressive decoding tricks—without giving up open weights.

What this means for AI leaders setting 2025–2026 strategy

If you are running a serious AI program, you cannot afford to treat model selection as a beauty contest anymore. You need a spine model—a default workhorse around which you design your infra, governance, and agent patterns. For us, that spine is now Nemotron 3 Nano.

Here is what I would do if I were starting that evaluation today:

• Pick one high‑value agent workflow—incident triage, complex support, code review—and force a head‑to‑head between your current model and Nemotron 3 Nano. Measure end‑to‑end incident time and cost, not just token accuracy.
• Stand up a single‑GPU test bed on an 80 GB H100 or B200. Prove to your own team that you can get 250–300 tok/s and million‑token contexts without exotic infra tricks.
• Run a cost‑per‑incident analysis. If your agents currently escalate to dense 70B models frequently, I would be shocked if you do not see 2–3x savings moving as many calls as possible to Nano.
• Design your governance around open weights. Being able to inspect, fine‑tune, and self‑host matters more as regulators start asking hard questions. Nemotron 3 Nano’s open‑weight posture buys you options that closed APIs do not.
• Plan a migration path to Super/Ultra. Treat Nano as your 2025 backbone and Super/Ultra as a 2026 throughput upgrade, not a risky rip‑and‑replace. If the roadmap delivers, your main job will be swapping in faster, cheaper inference, not rethinking your entire architecture.

My view is blunt: teams that keep defaulting to dense frontier models for every agentic task are going to get priced out, throttled by rate limits, or hamstrung by compliance concerns. Teams that standardize on something like Nemotron 3 Nano—sparse, fast, open, and actually deployable—will have the freedom to iterate on what really matters: better tools, better UX, and better alignment with their own data.

We are done paying premium prices for generic reasoning we can get from an open 30B hybrid MoE that respects our hardware and our budget. Nemotron 3 Nano is not perfect, but it is the first open‑weight agentic model that feels like it was built for the people who actually have to run it. And that is enough for me to bet our agents—and a meaningful chunk of our 2025–2026 roadmap—on it.