SketchVLM: Vision language models can annotate images to explain thoughts and guide users

About this episode

The episode discusses SketchVLM, a framework that enhances vision-language models by allowing them to annotate images for better user understanding.

🤗 Upvotes: 22 | cs.CV, cs.AI Authors: Brandon Collins, Logan Bolton, Hung Huy Nguyen, Mohammad Reza Taesiri, Trung Bui, Anh Totti Nguyen Title: SketchVLM: Vision language models can annotate images to explain thoughts and guide users Arxiv: http://arxiv.org/abs/2604.22875v2 Abstract: When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated…

People in this episode

Hosts: Jingwen Liang, Gengyu Wang

Topics covered

• vision language models
• image annotation
• human-AI collaboration
• visual reasoning
• SVG overlays

Keywords

• SketchVLM
• vision language models
• image annotation
• visual reasoning
• SVG overlays
• human-AI collaboration
• interactive demo

Mentioned in this episode

Organizations: SketchVLM, Gemini-3-Pro, GPT-5

More episodes of Daily Paper Cast

• EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments · June 13, 2026 · 24 min
• MiniMax Sparse Attention · June 13, 2026 · 26 min
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning · June 13, 2026 · 23 min
• InterleaveThinker: Reinforcing Agentic Interleaved Generation · June 13, 2026 · 21 min
• FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents · June 13, 2026 · 23 min
• Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding? · June 13, 2026 · 20 min