Enterprise video analytics has long been stuck in a reactive mode—systems that answer “What happened?” but leave the “What now?” untouched. A new technical walkthrough from NVIDIA changes that by weaving together two powerful blueprints into a single agentic workflow that not only analyzes footage but automatically dispatches tickets, escalates anomalies, and feeds findings into operational tools.
Moving Beyond Simple Video Analysis
The challenge isn’t new: video feeds, document stores, and ticketing systems live in separate silos. Bridging them requires capturing user intent, retrieving relevant organizational context, and generating structured, actionable reports. NVIDIA tackles this with a composable stack built on NVIDIA NemoClaw, an open collection of autonomous agent blueprints, and two specialized reference workflows:
- Metropolis Blueprint for Video Search and Summarization (VSS) – ingests streaming or archival video, generates captions and metadata, and supports semantic search and event summarization.
- AI Blueprint for Retrieval-Augmented Generation (RAG) – indexes enterprise documents like manuals, policies, and SOPs into a GPU-accelerated vector store for fast semantic retrieval.
Together, they let an agent move from generic video queries to context-rich, goal-driven analysis. The key is a mandatory human-in-the-loop (HITL) step that scopes the entire task before any heavy processing begins. Users specify the scenario, events of interest, objects to track, and an optional knowledge-retrieval query—tying the video directly to organizational knowledge.
The Agentic Toolchain: Three Tools that Make It Work
Inside the orchestration layer, three specialized tools work in sequence:
- Long video summary (LVS) tool: performs hierarchical summarization with HITL parameter collection. It asks what the user cares about and passes that intent along with the video and any retrieved context.
- Knowledge retrieval (frag) tool: calls the RAG blueprint to pull in domain-specific documents, handling embedding, reranking, and vector search internally.
- Report generation tool: fuses the video analysis with the retrieved context into a structured, timestamped assessment complete with citations and recommended actions.
NemoClaw coordinates these tools by reading a skill definition file that declares the required parameters, then walking the user through prompts or accepting programmatic input for batch jobs. This modular design means the agent can be customized for virtually any video-heavy workflow—from safety compliance to customer behavior analytics.
A Practical Walkthrough: The Healthy Eating Coach
The blog illustrates the entire pipeline with a “healthy eating coach” demo. A user uploads a meal preparation video, and the agent asks a few focused questions: What are we analyzing? What nutritional guidelines should apply? With that context, the system retrieves relevant reference documents via RAG, processes the video, and delivers a timestamped report that explains why a meal is healthy and what adjustments are recommended.
But the real jump happens next. NemoClaw doesn’t stop at report generation—it automatically creates a Jira ticket summarizing the findings, assigning priority and ownership so follow-ups are tracked. This downstream action pattern generalizes: the agent can also:
- Compare patterns across multiple video analysis runs.
- Bundle supporting evidence for review or compliance.
- Escalate anomalies to the appropriate team.
- Route identified gaps into corrective workflows.
The architecture layers orchestration, the VSS agent, the RAG blueprint, and LLM fusion cleanly, so developers can swap in different video sources or knowledge bases without rewriting the agent logic.