Photo Content Recognition & Curation Expert

Expert in photo content analysis and intelligent curation. Combines classical computer vision with modern deep learning for comprehensive photo analysis.

When to Use This Skill

✅ Use for:

• Face recognition and clustering (identifying important people)

• Animal/pet detection and clustering

• Near-duplicate detection using perceptual hashing (DINOHash, pHash, dHash)

• Burst photo selection (finding best frame from 10-50 shots)

• Screenshot vs photo classification

• Meme/download filtering

• NSFW content detection

• Quick indexing for large photo libraries (10K+)

• Aesthetic quality scoring (NIMA)

❌ NOT for:

• GPS-based location clustering → event-detection-temporal-intelligence-expert

• Color palette extraction → color-theory-palette-harmony-expert

• Semantic image-text matching → clip-aware-embeddings

• Video analysis or frame extraction

Quick Decision Tree

What do you need to recognize/filter? │ ├─ Duplicate photos? ─────────────────────────────── Perceptual Hashing │ ├─ Exact duplicates? ──────────────────────────── dHash (fastest) │ ├─ Brightness/contrast changes? ───────────────── pHash (DCT-based) │ ├─ Heavy crops/compression? ───────────────────── DINOHash (2025 SOTA) │ └─ Production system? ─────────────────────────── Hybrid (pHash → DINOHash) │ ├─ People in photos? ─────────────────────────────── Face Clustering │ ├─ Known thresholds? ──────────────────────────── Apple-style Agglomerative │ └─ Unknown data distribution? ─────────────────── HDBSCAN │ ├─ Pets/Animals? ─────────────────────────────────── Pet Recognition │ ├─ Detection? ─────────────────────────────────── YOLOv8 │ └─ Individual clustering? ─────────────────────── CLIP + HDBSCAN │ ├─ Best from burst? ──────────────────────────────── Burst Selection │ └─ Score: sharpness + face quality + aesthetics │ └─ Filter junk? ──────────────────────────────────── Content Detection ├─ Screenshots? ───────────────────────────────── Multi-signal classifier └─ NSFW? ──────────────────────────────────────── Safety classifier

Core Concepts

1. Perceptual Hashing for Near-Duplicate Detection

Problem: Camera bursts, re-saved images, and minor edits create near-duplicates.

Solution: Perceptual hashes generate similar values for visually similar images.

Method Comparison:

Method Speed Robustness Best For

dHash Fastest Low Exact duplicates

pHash Fast Medium Brightness/contrast changes

DINOHash Slower High Heavy crops, compression

Hybrid Medium Very High Production systems

Hybrid Pipeline (2025 Best Practice):

• Stage 1: Fast pHash filtering (eliminates obvious non-duplicates)

• Stage 2: DINOHash refinement (accurate detection)

• Stage 3: Optional Siamese ViT verification

Hamming Distance Thresholds:

• Conservative: ≤5 bits different = duplicates

• Aggressive: ≤10 bits different = duplicates

→ Deep dive: references/perceptual-hashing.md

2. Face Recognition & Clustering

Goal: Group photos by person without user labeling.

Apple Photos Strategy (2021-2025):

• Extract face + upper body embeddings (FaceNet, 512-dim)

• Two-pass agglomerative clustering

• Conservative first pass (threshold=0.4, high precision)

• HAC second pass (threshold=0.6, increase recall)

• Incremental updates for new photos

HDBSCAN Alternative:

• No threshold tuning required

• Robust to noise

• Better for unknown data distributions

Parameters:

Setting Agglomerative HDBSCAN

Pass 1 threshold 0.4 (cosine)

Pass 2 threshold 0.6 (cosine)

Min cluster size

3 photos

Metric cosine cosine

→ Deep dive: references/face-clustering.md

3. Burst Photo Selection

Problem: Burst mode creates 10-50 nearly identical photos.

Multi-Criteria Scoring:

Criterion Weight Measurement

Sharpness 30% Laplacian variance

Face Quality 35% Eyes open, smiling, face sharpness

Aesthetics 20% NIMA score

Position 10% Middle frames bonus

Exposure 5% Histogram clipping check

Burst Detection: Photos within 0.5 seconds of each other.

→ Deep dive: references/content-detection.md

4. Screenshot Detection

Multi-Signal Approach:

Signal Confidence Description

UI elements 0.85 Status bars, buttons detected

Perfect rectangles 0.75

5 UI buttons (90° angles)

High text 0.70

25% text coverage (OCR)

No camera EXIF 0.60 Missing Make/Model/Lens

Device aspect 0.60 Exact phone screen ratio

Perfect sharpness 0.50

2000 Laplacian variance

Decision: Confidence >0.6 = screenshot

→ Deep dive: references/content-detection.md

5. Quick Indexing Pipeline

Goal: Index 10K+ photos efficiently with caching.

Features Extracted:

• Perceptual hashes (de-duplication)

• Face embeddings (people clustering)

• CLIP embeddings (semantic search)

• Color palettes

• Aesthetic scores

Performance (10K photos, M1 MacBook Pro):

Operation Time

Perceptual hashing 2 min

CLIP embeddings 3 min (GPU)

Face detection 4 min

Color palettes 1 min

Aesthetic scoring 2 min (GPU)

Clustering + dedup 1 min

Total (first run) ~13 min

Incremental <1 min

→ Deep dive: references/photo-indexing.md

Common Anti-Patterns

Anti-Pattern: Euclidean Distance for Face Embeddings

What it looks like:

distance = np.linalg.norm(embedding1 - embedding2) # WRONG

Why it's wrong: Face embeddings are normalized; cosine similarity is the correct metric.

What to do instead:

from scipy.spatial.distance import cosine distance = cosine(embedding1, embedding2) # Correct

Anti-Pattern: Fixed Clustering Thresholds

What it looks like: Using same distance threshold for all face clusters.

Why it's wrong: Different people have varying intra-class variance (twins vs. diverse ages).

What to do instead: Use HDBSCAN for automatic threshold discovery, or two-pass clustering with conservative + relaxed passes.

Anti-Pattern: Raw Pixel Comparison for Duplicates

What it looks like:

is_duplicate = np.allclose(img1, img2) # WRONG

Why it's wrong: Re-saved JPEGs, crops, brightness changes create pixel differences.

What to do instead: Perceptual hashing (pHash or DINOHash) with Hamming distance.

Anti-Pattern: Sequential Face Detection

What it looks like: Processing faces one photo at a time without batching.

Why it's wrong: GPU underutilization, 10x slower than batched.

What to do instead: Batch process images (batch_size=32) with GPU acceleration.

Anti-Pattern: No Confidence Filtering

What it looks like:

for face in all_detected_faces: cluster(face) # No filtering

Why it's wrong: Low-confidence detections create noise clusters (hands, objects).

What to do instead: Filter by confidence (threshold 0.9 for faces).

Anti-Pattern: Forcing Every Photo into Clusters

What it looks like: Assigning noise points to nearest cluster.

Why it's wrong: Solo appearances shouldn't pollute person clusters.

What to do instead: HDBSCAN/DBSCAN naturally identifies noise (label=-1). Keep noise separate.

Quick Start

from photo_curation import PhotoCurationPipeline

pipeline = PhotoCurationPipeline()

Index photo library

index = pipeline.index_library('/path/to/photos')

De-duplicate

duplicates = index.find_duplicates() print(f"Found {len(duplicates)} duplicate groups")

Cluster faces

face_clusters = index.cluster_faces() print(f"Found {len(face_clusters)} people")

Select best from bursts

best_photos = pipeline.select_best_from_bursts(index)

Filter screenshots

real_photos = pipeline.filter_screenshots(index)

Curate for collage

collage_photos = pipeline.curate_for_collage(index, target_count=100)

Python Dependencies

torch transformers facenet-pytorch ultralytics hdbscan opencv-python scipy numpy scikit-learn pillow pytesseract

Integration Points

• event-detection-temporal-intelligence-expert: Provides temporal event clustering for event-aware curation

• color-theory-palette-harmony-expert: Extracts color palettes for visual diversity

• collage-layout-expert: Receives curated photos for assembly

• clip-aware-embeddings: Provides CLIP embeddings for semantic search and DeepDBSCAN

References

• DINOHash (2025): "Adversarially Fine-Tuned DINOv2 Features for Perceptual Hashing"

• Apple Photos (2021): "Recognizing People in Photos Through Private On-Device ML"

• HDBSCAN: "Hierarchical Density-Based Spatial Clustering" (2013-2025)

• Perceptual Hashing: dHash (Neal Krawetz), DCT-based pHash

Version: 2.0.0 Last Updated: November 2025