NVIDIA has posted a new edge-agentic result: TensorRT Edge-LLM running Qwen3.6-27B on one Jetson AGX Thor Developer Kit completed the MLPerf Inference v6.1 Edge Agentic benchmark in 24 minutes 36 seconds, producing 52.33 tokens per second. That is 6.4x faster than the llama.cpp reference using Q4_K_M quantization, which needed 2 hours 37 minutes on the same hardware.
A benchmark built for agentic trajectories
MLPerf's Edge Agentic benchmark doesn't look like a typical chatbot test. It replays recorded software-engineering agent trajectories: 20 conversations and 1,007 generated turns, with input length growing to roughly 23.5K tokens. Each turn requires a tool call, then observes the result and continues the same conversation. The performance phase measures that long-horizon behavior. An accuracy phase uses Berkeley Function Calling Leaderboard (BFCL) v4 prompts with single-turn only and reasoning disabled, which is a deliberately lightweight check for edge devices.
It is worth being precise about the comparison: the 6.4x gap contrasts a chip-specific, optimized stack against a general-purpose reference implementation, not against a best-case alternative for Jetson AGX Thor. Still, the result shows how much headroom exists at the runtime layer.
The optimization stack behind the speedup
TensorRT Edge-LLM deploys four interlocking techniques to cut down on the two biggest edge bottlenecks: memory bandwidth and repetitive prefill.
- NVFP4 quantization for weights and activations, including the language-model head, plus an FP8 KV cache, shrinks DRAM traffic during decoding.
- KV cache and recurrent-state reuse restores cached attention pages and Qwen3.6's hybrid recurrent state. On this run, about 96% of prompt tokens came from hot cache, with only ~0.5M of 13.6M tokens actually prefilled.
- Tree-based multi-token prediction uses an 8-step, top-2, 16-node verification tree. The target model verifies multiple high-probability branches in one pass. NVIDIA reports roughly 40% additional decoding performance over linear MTP on function-calling work.
- The MAXN power mode and the board's 128 GB unified memory give the model and speculative decoding state room to operate.
Try the stack
The MLPerf submission code lives in the TensorRT Edge-LLM release/0.9.1-mlpinf branch. Developers can download a published, calibrated Qwen3.6-27B NVFP4 checkpoint, then follow the mlperf readme to export the checkpoint with the tree-MTP interface, build TensorRT engines, and launch the OpenAI-compatible server. The MLCommons endpoint harness uses temperature 0, seed 42, reasoning disabled, and concurrency 1.