Using OCR Heads to Verbalize Image Semantics
Quick summary
arXiv:2609.18823v1 Announce Type: cross Abstract: How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens. For example, pointing these heads at an image token containing the word "bike" causes Qwen3-VL-8B to output "bike," but pointing them at a bird wing causes the model to output the
Key takeaways
- arXiv:2609.18823v1 Announce Type: cross Abstract: How do VLMs map from pixels to semantics?
- To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR).
- Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens.
Why it matters
This model development creates a new option for users and a new testing obligation for developers. A fixed evaluation set comparing quality, cost and failure behavior is more useful than launch claims.
Member comments