NVIDIA Releases Bolt E4B: A Small Model for Local Assistants and Robots, with Less Chatter and Better Tool Use

NVIDIA has released Nemotron-3.5-Bolt-E4B 4GB, a compact model tailored for local assistants, interactive characters, and robotics. Its primary focus is natural conversation turn-taking and calling external tools to complete tasks. Built on Google's Gemma-4-E4B-it and further trained through reinforcement learning, its runtime VRAM footprint under target workloads is approximately 4GB when quantized. Key highlights include:

• Multi-turn tool calling and action decisions: Supports invoking application APIs across continuous dialogues, reading execution results, and deciding subsequent actions.
• One-toggle conversational mode: A dedicated flag in the chat template automatically adds instructions for concise replies and lighter formatting, outputting colloquial short sentences ideal for text-to-speech playback.
• Configurable reasoning mechanism: Allows runtime enabling or disabling of explicit intermediate reasoning, along with setting reasoning token budget caps to control response latency.
• 64K context window: Accommodates system prompts, multi-turn dialogue history, tool schema definitions, environment state, and retrieved knowledge.
• Cross-device edge deployment: Targets GeForce RTX graphics cards, Jetson embedded compute boards, RTX Spark, and edge appliances.

Billed as a 4GB Model: How Much VRAM and System RAM Is Really Needed?

In edge deployment discussions, "8B," "4B," "4GB," and "6GB" are frequently conflated. Understanding the resource requirements of Nemotron-3.5-Bolt-E4B requires distinguishing parameter count, weight file sizes, and runtime memory.

The model has a total parameter count of 8B and an effective parameter count of 4B. This design inherits the Per-Layer Embedding (PLE) architecture of the underlying Gemma-4-E4B base: the model lets each layer look up its own dedicated token embedding table. Although these tables take up substantial storage, they function primarily as rapid table lookups rather than dense computation layers executed at every step. Consequently, while its physical parameter footprint is 8B, its effective compute scale is classified as ~4B. Crucially, the lookup tables must still reside in storage, so the required weight footprint cannot be halved based on this distinction. (B denotes billion: 8B is 8 billion parameters; 4B is 4 billion. Parameters measure model scale, whereas GB measures storage or memory capacity.)

Quantization lowers numerical precision in weight storage to reduce disk footprint and memory demands; different schemes make different trade-offs between size and capabilities. In actual storage and runtime environments, weight sizes, VRAM usage, and total system RAM differ considerably:

• Weight file size: The full-precision BF16 weights take roughly 15GB. The official Q4_K_M 4-bit quantized GGUF file calibrated via importance matrix (imatrix) is approximately 6.18GB, whereas Unsloth's UD-Q4_K_XL quantized file is roughly 4.76GB. Verify the exact quantization variant before downloading.
• Runtime VRAM footprint: Under typical single-session operational workloads (such as 3,000 input tokens and 50 output tokens), VRAM consumption on an RTX 5070 is around 3.5GB; in a multi-turn extended session with a 32K cache, it reaches roughly 4.3GB; and on a Jetson Orin Nano with 8GB unified memory, total inference runtime memory occupies 4.2GB to 4.9GB. This accounts for the "4GB" moniker—holding the VRAM footprint under typical target workloads to roughly 4GB. Here, 6.18GB reflects the full disk weight size, whereas 3.5GB counts solely GPU VRAM usage, which must still run alongside system memory. The 4GB label is a workload VRAM target, not an assertion that an entire system requires only 4GB of RAM.
• Total system memory (RAM): Unsloth provides an operational reference of roughly 6GB RAM. Actual usage will fluctuate depending on the quantization version, context length, and serving software; upon seeing "6GB," overhead must still be reserved for the operating system and background applications.

A smaller model footprint leaves more room for speech recognition, speech synthesis, vision models, and application logic to coexist on a single machine. However, the Jetson's 8GB is unified shared memory; beyond Bolt's footprint, OS and application overhead must be factored into the budget.

Why Can It Cut the Chatter and Stay Focused on Execution?

When smaller models serve as local assistants, verbose responses cause two primary friction points: first, getting trapped in lengthy internal monologues that churn out extensive self-analysis rather than executing tools; second, generating long-winded written explanations with complex formatting, which introduces severe latency for text-to-speech systems and sounds unnatural.

The base Gemma-4-E4B-it model already includes an instruction-tuned (IT) foundation. For Nemotron-3.5-Bolt-E4B, NVIDIA bypassed additional supervised fine-tuning (SFT) in post-training, opting exclusively for reinforcement learning. The research team initially experimented with SFT using diverse teacher-model datasets; however, while SFT boosted tool-calling performance, it degraded conversational quality and inflated answer length. The final release instead focuses on evaluating whether actions are executed correctly (such as verifying tool call arguments and execution outcomes via deterministic rule checks) and whether dialogue sounds natural. It trains on these signals by blending Reinforcement Learning with Verifiable Rewards (RLVR) with Reinforcement Learning from Human Feedback (RLHF). (Training was conducted within the NeMo-RL and NeMo-Gym frameworks.)

To make the compact model nimbler during interactions, three mechanisms were implemented:

1. A 2:1 ratio of non-reasoning to reasoning samples: Every training sample was generated and scored under both reasoning enabled (enable_thinking=true) and reasoning disabled (enable_thinking=false). Training batches maintained a ratio of two non-reasoning samples to one reasoning sample, training the model to parse instructions and output accurate actions without always requiring an explicit intermediate chain of thought.
2. Dynamic length penalty adjustment: During reinforcement learning, models frequently lengthen their reasoning chains to avoid mistakes. The team penalized excessively long reasoning and responses within a Generative Reward Model (GenRM) environment, cyclically tightening and relaxing this penalty so that accuracy and conciseness improved together. In official length comparisons across 21 BF16 benchmarks (with reasoning enabled for most tasks, but disabled for EQ-Bench, RoleBench, and RULER), Bolt averaged a score of 63.9 with an average output of 672 tokens; the base Gemma-4-E4B-it scored 58.1 with 782 tokens; while the similarly sized Qwen3.5-4B scored 57.8 but output an average of 3,604 tokens. Bolt's average output is roughly 81% lower than Qwen's and roughly 14% lower than Gemma's in this test. Thus, claims of "up to ~80% fewer" depend heavily on the comparison baseline; length penalization was a training mechanism, and this comparison reflects end-to-end model outputs.
3. Speak-before-action template ordering: When an assistant's turn includes both natural language and tool calls, the chat template positions explanatory text ahead of the tool payload. This allows the agent to inform the user verbally (e.g., "Checking today's weather for you now...") before triggering a time-consuming tool, bridging conversational confirmation with practical execution.

The official evaluation also tested a blacksmith merchant NPC in a game environment: when a player requests a golden axe, the NPC asks for two diamonds; after receiving just one, the NPC explicitly demands the second diamond, and only after both are collected does it autonomously call tools to look up the crafting recipe, fetch materials, craft the golden axe, and hand it to the player. In this pipeline, the model generates the tool call request, while the external host application executes the real local API and returns the result and updated state to the model. The model then uses this updated evidence to drive the next action and dialogue turn, maintaining a closed execution loop.

How Fast Does It Run Locally? TTFT and Generation Throughput

Evaluating local assistant latency requires separating Time to First Token (TTFT) from generation throughput in Tokens Per Second (TPS). TTFT is primarily governed by the input-processing "prefill" phase, whereas TPS measures the incremental "decoding" phase.

TTFT spans the moment a request reaches the model to the generation of its first token, dictated by prefill computation. However, TTFT is not equivalent to the full end-to-end latency in voice interactions (which also includes speech recognition and audio transmission). Generation throughput represents how many tokens the model outputs per second once decoding begins. In hardware tests, 32K cache indicates pre-cached context, 2K represents fresh turn input, and 256 denotes output length; because multiple conditions change between workloads, differences cannot be attributed solely to cache size.

Looking at official benchmarks under Q4_K_M quantization:

• On a desktop discrete GPU, the RTX 5070 (12GB), handling a standard 3,000-token prompt yielded a TTFT of 548 ms and a TPS of 133.4 tok/s.
• On a compute-constrained embedded edge device, the Jetson Orin Nano (8GB), processing that same 3,000-token prompt saw TTFT jump to 5,785 ms (nearly 5.8 seconds), with decoding settling at 15.5 tok/s.

This side-by-side comparison under identical workloads demonstrates that on low-compute edge hardware, prefill latency from extensive system prompts and historical context will noticeably lengthen user wait times, making prompt length control essential for developers.

To accelerate decode throughput during generation, the model was released alongside a speculative sampling draft model based on the DFlash architecture (DFlash Parallel Drafter). Speculative sampling uses a lightweight draft model to quickly guess an upcoming token sequence; the primary model verifies the block in a single pass, accepting candidates that pass verification and cutting down on sequential decode cycles. This draft model employs a 5-layer Transformer that predicts 7 candidate tokens simultaneously in one forward pass, which the primary model verifies in parallel while strictly preserving its original probability distribution.

The acceleration gains correspond to the following baselines:

• On an RTX PRO 6000 running BF16 NIM with reasoning enabled at concurrency 1, pairing with DFlash raised single-user decode throughput to 2.74 times the baseline without speculative sampling (from 124.7 tok/s to 341.2 tok/s);
• On an RTX 5070 tested at batch size 1 on the SPEED-Bench coding subset, with both target models running Q4_K_M quantization, Bolt paired with a BF16 DFlash draft model achieved 272.6 tok/s—a 21% boost (1.21 times the baseline) over Gemma-4-E4B-it's native MTP implementation (225.9 tok/s). This setup differs from the earlier 133.4 tok/s measurement and must not be conflated with it.

DFlash accelerates subsequent generation. In the NIM setup above, the change in TTFT remained under 100 ms; when input prompts are extensive, prefill delay still requires separate mitigation.

Can It Process Images and Audio? Multimodal and Cross-Lingual Evaluation

Bolt inherits Gemma 4's image and audio understanding, but supported modalities depend on deployment: the official GGUF model card lists I/O as Text / text, but the official NIM BF16 microservice natively supports text, image, and audio inputs. The Unsloth repository also supplies a companion mmproj projector file for multimodal runtimes. Loading a text GGUF does not automatically configure vision capabilities; proper vision requires matching runtime files and tooling.

The inherited modalities also exhibit divergent shifts:

1. Vision comprehension has regressed: While the Gemma-4-E4B base possesses multimodal capabilities, text-only reinforcement learning post-training caused alignment drift between the visual encoder and the language model. Evaluations show degraded vision metrics relative to Gemma-4-E4B-it: with reasoning enabled, the document recognition benchmark OmniDocBench 1.5 score (lower is better) rose from 0.21 to 0.42; multidisciplinary vision understanding on MMMU-Pro Vision dropped from 47.63 to 42.02; and MathVision dropped from 48.29 to 46.25. Dedicated fine-tuning runs on the vision encoder failed to yield notable improvements.
2. Audio comprehension has improved: With reasoning enabled, Bolt-E4B scored 80.30 on the single-turn VoiceBench benchmark (surpassing the base's 78.20) and reached 25.22 on the multi-turn Audio MultiChallenge (above the base's 19.03). Both models cap voice inputs at 30 seconds per turn; because over 50% of dialogues in AudioMC contain turns exceeding 30 seconds, benchmark scores reflect this per-turn ceiling rather than full, untruncated audio. These scores gauge input comprehension; speaking answers aloud still requires external speech synthesis.
3. Perception and action division in the robot demo: In the official Reachy Mini desktop robot demonstration, real-time visual perception of the environment was handled by NVIDIA Cosmos 3 Edge. Bolt-E4B ingested Cosmos's perception output, managed dialogue logic, and triggered existing motion scripts from the robot's animation library to drive physical movement. Both models ran locally in tandem on a Jetson Orin Nano 8GB.
4. Cross-lingual trade-offs: Post-training was conducted on English data. While WMT24++ benchmarks show machine translation quality was broadly preserved, multilingual knowledge and reasoning (MMLU-ProX) saw minor but consistent drops. With reasoning enabled, its Chinese MMLU-ProX score was 65.17—a 1.61-point drop compared to Gemma-4-E4B-it (66.78). In non-reasoning mode, it scored 63.32, trailing the base by 1.12 points.

Benchmark Performance: Where It Excels and Where It Falls Short

NVIDIA's reported "~40% average improvement over Nano 4B" spans 21 benchmarks; the 70.2-point score highlighted in promotional tweet graphics represents an average across 6 tasks under Q4_K_M quantization. In that graphic, TTFT was clocked on an RTX 5090 using ~3,000 input tokens and ~50 output tokens. That 70.2 score cannot be treated as mobile device throughput or a general intelligence aggregate.

All scores were measured by NVIDIA under a unified NeMo-Gym evaluation harness, providing an apples-to-apples basis for comparison; Brevity Benchmark and When2Call-MT have not been publicly released.

Reasoning modes across the six benchmarks below are not identical: explicit reasoning was disabled for RULER under both BF16 and Q4_K_M, while the other five benchmarks had reasoning enabled. Disabling explicit reasoning lets it answer directly, but involves accuracy trade-offs on specific tasks—for example, in BF16, disabling reasoning drops When2Call-MT from 63.36 to 51.25 and Tau2-Bench from 55.10 to 34.34. Across all 21 benchmarks, Bolt's non-reasoning mode still captured the top or tied-top score in 14 tasks against its peers; whether to disable reasoning should be decided based on accuracy requirements.

1. Strengths: Rule Adherence and Tool Judgments

• System instruction conflict adherence (IHEval BF16 84.79 / Q4_K_M 82.40): In tests where user inputs conflict with system instructions or attempt malicious jailbreaks, Bolt-E4B adheres to system rules significantly better than base Gemma-4-E4B-it (BF16 67.04 / Q4 66.61) and Qwen3.5-4B (BF16 60.57 / Q4 62.87). While this shows stronger resistance to rule overrides, failures can still occur.
• Multi-turn constraints and role consistency: Maintaining persona and following rules across extended interactions is critical for local interactive agents. With reasoning enabled, Bolt-E4B achieved 53.10 on MultiChallenge (BF16), which tests instruction and constraint consistency across multiple turns (compared to 42.08 for base Gemma-4-E4B-it). On RoleMRC (BF16), evaluating persona comprehension and adherence, Bolt-E4B scored 80.63 (versus 74.62 for the base). These results show it is better at remembering multi-turn constraints and staying in character, which is distinct from emotional empathy.
• Tool invocation judgment (When2Call-MT BF16 63.36 / Q4_K_M 60.50): Compared to Qwen3.5-4B (BF16 37.85 / Q4 42.88), Bolt-E4B more reliably decides when to answer directly, when to invoke external tools, and when to ask clarifying questions, performing on par with base Gemma-4-E4B-it (BF16 62.90 / Q4 61.12).
• Response conciseness (Brevity Benchmark BF16 82.52 / Q4_K_M 82.84): While retaining key informational content, its responses are more succinct than base Gemma-4-E4B-it (BF16 79.99 / Q4 78.49) and Qwen3.5-4B (BF16 75.35 / Q4 75.17).
• Resistance to bad tool output (RelyToolBench BF16 63.16 / Q4_K_M 69.29): When external APIs return misleading or erroneous fields, Bolt's ability to resist hallucinations and correct execution substantially outclasses Qwen3.5-4B (BF16 43.35 / Q4 38.34).
• Grounding in retrieved knowledge: Grounding answers in retrieved material differs from handling tool errors. When a local assistant reads a device manual or local knowledge base, it must ground its answers strictly in provided text rather than fabricating information. With reasoning enabled, Bolt-E4B scored 42.61 on GaRAGe (BF16), which measures grounded generation from retrieved context, surpassing Gemma-4-E4B-it's 34.73. On RAGTruth (BF16), which detects ungrounded claims, Bolt-E4B showed a marginal gain (77.59 vs. 76.97 for the base); unsupported assertions remain possible.

2. Weaknesses: Complex Multi-Step Tasks and Long-Context Retrieval

• Complex multi-step customer service (Tau2-Bench BF16 55.10 / Q4_K_M 52.32): On intricate customer support workflows involving state transitions, ticket cancellations, and rebooking, Bolt-E4B outperforms base Gemma-4-E4B-it (BF16 38.50 / Q4 38.72) but falls well short of Qwen3.5-4B (BF16 75.06 / Q4 61.74), showing strain on multi-step planning tasks.
• Ultra-long document retrieval (RULER 64K BF16 86.99 / Q4_K_M 85.76): In precise needle-in-a-haystack retrieval across a full 64K context window, Bolt's performance mirrors base Gemma-4-E4B-it (BF16 86.55 / Q4 86.14), but trails Qwen3.5-4B (BF16 95.15 / Q4 95.37) and falls behind previous-generation Nano 4B (BF16 91.51 / Q4 91.31). Dense long-context retrieval is not its strength.
• Emotional comprehension and empathy (EQ-Bench v3 BF16 39.40 / Q4_K_M 38.15): Evaluated with reasoning disabled, Bolt scored slightly lower than base Gemma-4-E4B-it (BF16 41.17 / Q4 39.77), though it remained ahead of Qwen3.5-4B (BF16 32.92 / Q4 32.07).

How to Get Started and Configure the Model

When downloading or configuring weights, ensure you select the E4B variant to avoid accidentally pulling the much larger 74B models in the family (such as 74B-A9B, whose weights exceed 20GB and require vastly different hardware setups).

1. Graphical Interface (Unsloth Desktop)

Ideal for users seeking local chat on Mac, Windows, or Linux:

• Open Unsloth Desktop and navigate to the Model Hub.
• Search for Nemotron 3.5 Bolt, and explicitly pick the E4B variant from the list (such as NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q4_K_XL.gguf).
• For initial testing, set context length to 4,096 or 8,192 tokens as a baseline; increase gradually based on available VRAM, rather than assuming hardware can handle a full long-context workload out of the box.

2. Developer Deployment Reference

Linux developers with NVIDIA GPUs can refer to the official Hugging Face model card (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB) and NIM documentation for container deployment guidelines. Context window allocations should be tailored to available memory when launching the service to avoid out-of-memory errors from blindly setting a 64K window.

3. Key Invocation Parameters and Mode Controls

When invoking the model via API clients, note official parameter recommendations:

• Sampling parameters: Official guidelines recommend keeping temperature=1.0 and top_p=0.95 across both reasoning and non-reasoning modes.
• Conversational mode (conversational_mode): Passing chat_template_kwargs={"conversational_mode": True} prompts the chat template to append instructions encouraging brief, colloquial responses with minimal formatting; it does not aggressively strip all formatting. This switch operates independently of reasoning mode and can be combined freely.
• Reasoning mode control (enable_thinking): Setting chat_template_kwargs={"enable_thinking": False} disables explicit intermediate reasoning output. This only suppresses the output reasoning chain and does not mean the model performs no reasoning at all. For reasoning token budgets, --reasoning-budget is exclusive to llama.cpp's llama-server, whereas NIM provides the request-level thinking_token_budget parameter. Hard budget limits truncate reasoning chains directly, requiring validation on target tasks; pipeline delays from speech recognition or tool execution must still be handled separately.
• Licensing notice: Model weights are released under the OpenMDW-1.1 license, permitting commercial and non-commercial use according to the official model card. Although the Gemma-4 base uses Apache 2.0, these post-trained weights are governed by OpenMDW-1.1 terms and must be used in accordance with those terms.

Nemotron-3.5-Bolt-E4B 4GB concentrates on local assistant and controlled interactive scenarios where edge compute is constrained. It excels at keeping outputs brief, orchestrating multi-turn tool workflows, and adhering to system prompt hierarchies, making it well-suited for responsive desktop companions, in-vehicle voice agents, or robot action dispatching. However, it exhibits clear limitations in dense long-context retrieval and complex multi-step task planning. Developers should benchmark their specific task profiles against hardware latency limits when choosing a deployment.