SmolVLM - Local Image Analysis

Analyze images locally using SmolVLM-2B, a state-of-the-art compact vision-language model optimized for Apple Silicon via mlx-vlm.

Quick Usage

Describe an Image

python ~/.claude/skills/smolvlm/scripts/view_image.py /path/to/image.png

Ask a Question About an Image

python ~/.claude/skills/smolvlm/scripts/view_image.py /path/to/image.png "What text is visible?"

Specific Tasks

Extract text (OCR)

python ~/.claude/skills/smolvlm/scripts/view_image.py screenshot.png "Extract all text"

UI analysis

python ~/.claude/skills/smolvlm/scripts/view_image.py ui.png "Describe the UI elements"

Detailed description

python ~/.claude/skills/smolvlm/scripts/view_image.py photo.jpg --detailed

Effective Prompts

General Description

• "Describe this image"

• Basic description

• "Describe this image in detail, including colors, composition, and any text"

• Comprehensive

Text Extraction (OCR)

• "Extract all visible text from this image"

• "What text appears in this screenshot?"

• "Read the text in this document"

UI/Screenshot Analysis

• "Describe the user interface elements"

• "What buttons and controls are visible?"

• "Identify the application and its current state"

Visual Question Answering

• "How many [objects] are in this image?"

• "What color is the [object]?"

• "Is there a [object] in this image?"

Code/Technical

• "What programming language is shown?"

• "Describe what this code does"

• "Identify any errors in this code screenshot"

Model Details

Spec Value

Model SmolVLM-2B-Instruct

Size ~4GB

Peak Memory 5.8GB

Speed ~94 tok/s (M-series)

Supported Formats PNG, JPG, JPEG, GIF, WebP

Requirements

• macOS with Apple Silicon (M1/M2/M3)

• Python 3.10+

• mlx-vlm package: uv pip install mlx-vlm --system

Troubleshooting

"Model not found": First run downloads the model (~4GB). Wait for completion.

Out of memory: Close other applications. Model needs ~6GB free RAM.

Slow first inference: Model loading takes 10-15s on first use, subsequent calls are faster.