Testing Nemotron 3 Super in MPP BI: a single-GPU 120B model in practice
We loaded NVIDIA's 120B Nemotron 3 Super on a single GPU and tested it inside MPP BI against GPT-OSS-120B, on speed, long context, and Armenian and Georgian reasoning.
↳ 96.3% vs 52.3% RULER score at 256K tokens, Nemotron vs GPT-OSS
Why it caught our attention
At MPP Insights, we build BI platforms where companies can analyze their data and create dashboards. A big part of our work is adding AI models that can understand data, read database structures, and help users work faster.
The problem is, most strong AI models need multiple GPUs to run well. That makes things expensive and harder to manage, especially for companies that want to run everything on their own setup. This is even more important in BI systems: these workflows usually include large schemas, documentation, and long interactions, so the model needs to handle a lot of information while still being fast enough to feel responsive.
When NVIDIA released Nemotron 3 Super, our technical team decided to test it. It is a 120B model, but because of how it is designed, only about 12B parameters are active per request. NVIDIA also released a version that can run on a single 96GB GPU. That is what caught our attention: if a model this big can run on one GPU, testing and deployment become much easier.
So our technical team loaded it on a single RTX PRO 6000 and tested it inside MPP BI, our business intelligence platform for data analytics and visualization. MPP BI is built on a data-centric architecture, which means that instead of copying data out to a separate engine for processing, it runs computations directly inside the database where the data already lives.
Mixture-of-experts architecture
NVIDIA designed the model as a Mixture-of-Experts (MoE) system. The total model size is 120 billion parameters, but only about 12 billion parameters are active during each request. Inside the model there are many expert blocks: small specialized parts trained to handle specific types of tasks. When a request arrives, a small component called a router selects only the experts that should handle it. The model stays large and capable while the computation required per request goes down.
Hybrid Mamba–MoE architecture
Mamba is a newer sequence-processing approach that helps models work with long text inputs more efficiently. It helps Nemotron process large amounts of context without losing track of earlier information, such as very large documents, long conversations, or complex database schemas — especially useful for analytical work, where the model has to handle a lot of structured data.
NVFP4 precision format
Many models are trained at high precision and then compressed later to reduce memory usage, which often costs some accuracy. Nemotron 3 Super was trained directly in NVFP4 from the start, a very compact 4-bit format, so it needs much less memory without losing much accuracy — which is what allows the checkpoint to fit on a single 96GB GPU while still leaving room for the KV cache during generation.
Comparison setup
To keep the comparison fair, we focused on models that run in a similar setup. The main comparison is GPT-OSS-120B: both models can run on a single high-end GPU, so they are realistic for production, including MPP BI deployments. We also included Qwen3.5-122B as a reference — it performs well, but it usually needs 4–8 GPUs, so it is less practical for our setup.
Our goal was to test these models in real conditions inside MPP BI, and we focused on three things:
- Can the model run on a single high-end GPU?
- Does it stay stable with a long context?
- Does it work across different languages we see in practice?
What the benchmarks say
Before testing the model inside MPP BI, we looked at public benchmarks to understand where Nemotron 3 Super stands compared to other models. These numbers come from public evaluations used across the AI community; we focused only on what matters for our use case.
Long-context performance
Long context means how much text a model can handle at once — long conversations, large documents, or complex database schemas. In BI systems like MPP BI this matters because the model often works with large and growing context. We looked at RULER, a benchmark that checks how well a model can understand and reason over very long inputs.
| Context length | Nemotron 3 Super | GPT-OSS-120B |
|---|---|---|
| 256K tokens | 96.3% | 52.3% |
| 512K tokens | 95.7% | 46.7% |
| 1M tokens | 91.8% | 22.3% |
Result: Nemotron stays stable even when the input gets very large. GPT-OSS starts to lose track as the context gets longer.
Coding and reasoning benchmarks
| Benchmark | Nemotron 3 Super | GPT-OSS-120B |
|---|---|---|
| SWE-Bench Verified | 60.5 | 41.9 |
| LiveCodeBench | 81.2 | 88.0 |
| AIME | 90.2 | 92.5 |
Result: at this stage there was no clear winner. Each model was strong in different areas: Nemotron on real software engineering tasks (SWE-Bench Verified), GPT-OSS on code generation (LiveCodeBench) and math (AIME).
What we tested ourselves: languages beyond English
MPP BI is used in multilingual environments, and our R&D office is in Yerevan, Armenia. We also work with clients and data in multiple languages, so we wanted to understand how these models behave when English is not the main language. We ran our own tests on Nemotron and GPT-OSS-120B using the same hardware, the same software stack, and the same evaluation process, focusing on Armenian and Georgian — low-resource languages where training data is limited, which makes them a good stress test for multilingual ability.
| Language (INCLUDE benchmark) | Nemotron 3 Super | GPT-OSS |
|---|---|---|
| Armenian | 51.20 | 29.39 |
| Georgian | 29.55 | 28.51 |
On both languages, both models perform at a relatively low level and stay close to each other, with Nemotron showing a small advantage but not a meaningful gap. On ArmBench, 1,000 Armenian reasoning questions across subjects, GPT-OSS scores slightly higher overall, by around 3 points, but the breakdown is mixed: GPT-OSS does better in math, business, and physics, while Nemotron does better in law, health, and general knowledge, and is more consistent in following the required answer format.
What this means for our work in MPP BI: we cannot assume strong performance just because a model performs well in English benchmarks. We still need to validate behavior per language and per workflow, especially in analytical systems where correctness and structure matter more than general language ability.
A week inside MPP BI: what we saw
After the benchmarks, we moved to real testing inside MPP BI, because this is where model behavior actually matters. The model works best with structured data — economics, medicine, social science — where the input is clear and the output needs to follow a fixed format. It follows instructions well through multi-step analysis flows and stays reliable without breaking the logic or losing track of the task.
In real usage we noticed a consistent pattern: the model doesn’t jump to answers quickly. It spends more time checking and rechecking before it responds — validating one aggregation, adjusting it, checking another dimension, repeating the cycle a few times before producing a final result. The output is correct, but the process takes more steps than expected. Context handling inside MPP BI was strong in practice: we worked with long sessions of up to 256K tokens and didn’t see instability, matching what we observed in the benchmarks.
Inference speed inside MPP BI
We ran both models on a single RTX PRO 6000 using 2K input and 2K output tokens, under different levels of concurrent users. At low load, both models feel responsive — time to first token stays under 200ms. The difference appears during generation: GPT-OSS is consistently faster, delivering about 2.2× to 2.5× higher throughput than Nemotron depending on load. In a typical MPP BI flow — query a dataset, check the schema, build a result step by step — the full process takes about 20 seconds with GPT-OSS and around 40 seconds with Nemotron under moderate load.
We also saw a pattern in behavior: when context becomes very large, GPT-OSS starts to lose stability, while Nemotron stays consistent. So the choice isn’t only about speed. It depends on the type of workload — smaller and fast workflows favor GPT-OSS, while large-context analytical workflows favor Nemotron.
Conclusion: what this means for MPP BI
We see strong potential in Nemotron 3 Super, especially for analytical workflows inside MPP BI. But based on our testing, we are not switching to it for all production use cases, and the main reason is speed — on our baseline single-GPU setup, GPT-OSS-120B is about 2x faster, and that difference is visible in real MPP BI workflows.
For multilingual use, the improvement was limited: on Armenian and Georgian, both models performed at a similar level, so multilingual performance alone isn’t enough to justify slower speed in production. Where Nemotron clearly stands out is context handling — it stays stable even up to 200K tokens and beyond, which matters for analytical workflows with large schemas and long reasoning chains. A single-GPU setup also makes things more practical to test, deploy, and iterate, and keeps costs predictable — but one GPU is still a limit as the number of users grows, so model efficiency matters just as much as model quality.
When Nemotron makes sense
- You need long context, around 64K to 256K tokens
- The workflow involves complex analytical reasoning
- You are testing new ideas or building agent-based flows
- Speed is not the main concern
When GPT-OSS is a better fit
- You have many concurrent users
- Speed and response time matter more
- Context size stays relatively small
- You need higher throughput on the same hardware