NVIDIA Dynamo Enhances Streaming for Agentic Workflows

NVIDIA has unveiled significant updates to its Dynamo platform, aimed at optimizing agentic workflows with enhanced streaming, parsing, and tool-call handling. These updates focus on improving responsiveness and accuracy for applications that rely on multi-turn interactions, such as coding assistants and other AI-driven tools.

One of the key highlights is the introduction of streaming tool-call dispatch. This new feature enables tool calls to execute as soon as they are decoded, sidestepping the need to wait for the full response turn to complete. This adjustment not only speeds up the time-to-first-token (TTFT) for users but also removes inefficiencies in agent workflows where reasoning and tool responses are interleaved.

Performance Gains Through Prompt Stability

A core improvement centers on prompt stability and KV-cache reuse. By eliminating session-specific preambles, such as Anthropic billing headers, Dynamo ensures consistent token prefixes across sessions. This change reduced TTFT by nearly fivefold in NVIDIA’s tests, from 912ms to 169ms, on a system using a 52K-token prompt.

For developers, maintaining stable prefixes is crucial when handling large, complex prompts across multiple user sessions. These optimizations are particularly valuable for agentic models like Claude Code and Codex, which require precise and repeatable interactions to function effectively.

Enhanced Parsing for Complex Interactions

Dynamo has also overhauled its reasoning and tool-call parsers, extracting them into reusable modules. This allows developers to achieve better alignment between parsed outputs and harness requirements. The updates address a long-standing issue where prior reasoning was either dropped or malformed during multi-turn interactions. In agentic workflows where reasoning explains tool-call sequences, retaining structured reasoning is critical.

For example, NVIDIA demonstrated how its Nemotron-3-Super-120B model can now process interleaved reasoning and tool calls more effectively, ensuring that each reasoning segment remains correctly attached to its corresponding tool action. This prevents issues where reasoning was previously grouped incorrectly, leading to lost context.

Streaming Behavior and Tool Dispatch

Another major improvement is the ability to stream tokenized responses while dispatching tool calls via a side channel. Previously, tool calls were buffered until the end of a response, delaying execution. With the new inline streaming and dispatch capabilities, tool calls become actionable as soon as they are parsed, significantly improving responsiveness for real-time applications.

NVIDIA illustrated this with a timeline comparison showing how Dynamo now parses and streams tool calls mid-response, enabling immediate execution. This redesign minimizes harness-side complexity and ensures seamless integration with custom systems.

Improved API Compliance

The updates also enhance Dynamo’s compatibility with the Anthropic Messages API, a critical interface for tools like Claude Code and OpenClaw. Fixes include proper token counting at the start of streams and the ability to serve model metadata endpoints, both of which bring Dynamo closer to native backend parity.

For Codex users, compatibility with OpenAI’s Responses API has also been improved. NVIDIA has addressed field preservation issues that occurred during internal request processing, ensuring that Codex-specific features like reasoning summaries and tool-call truncation are supported without degrading performance.

What’s Next

Looking forward, NVIDIA is making parts of Dynamo’s serving stack available as modular components, including protocol, parser, and tokenizer crates. This modularity allows developers to build custom harnesses or extend existing ones without duplicating Dynamo’s core functionality.

These updates position Dynamo as a leading solution for agentic workloads, enabling more efficient and accurate multi-turn interactions across a range of applications. For developers and enterprises relying on AI-driven tools, these enhancements offer a more reliable and high-performance infrastructure for tasks such as coding, data analysis, and beyond.

Image source: Shutterstock