Tao Generate Referring Expressions

This is NVIDIA's four-stage pipeline for turning images with KITTI bounding boxes into rich referring expressions. A VLM (Gemini, NIM, or any OpenAI-compatible endpoint) generates per-object descriptions, scene captions, grouped grounding phrases tied to bboxes, and an optional verification pass that catches mismatches. It's built for auto-labeling traffic and scene datasets where you already have basic bbox annotations but need natural language references. The workflow is resumable, skips already-processed images, and writes everything to a unified JSONL. Start with 5 to 10 images, check if the groupings and spatial relationships make sense, swap to a stronger model if needed, then scale up. Runs in the TAO Toolkit container with decent parallelism controls.

Install to Claude Code

npx -y skills add nvidia/skills --skill tao-generate-referring-expressions --agent claude-code