CLIP-Aware Image Embeddings

Smart image-text matching that knows when CLIP works and when to use alternatives.

MCP Integrations

MCP	Purpose
Firecrawl	Research latest CLIP alternatives and benchmarks
Hugging Face (if configured)	Access model cards and documentation

Quick Decision Tree

Your task:
├─ Semantic search ("find beach images") → CLIP ✓
├─ Zero-shot classification (broad categories) → CLIP ✓
├─ Counting objects → DETR, Faster R-CNN ✗
├─ Fine-grained ID (celebrities, car models) → Specialized model ✗
├─ Spatial relations ("cat left of dog") → GQA, SWIG ✗
└─ Compositional ("red car AND blue truck") → DCSMs, PC-CLIP ✗

When to Use This Skill

✅ Use for:

• Semantic image search
• Broad category classification
• Image similarity matching
• Zero-shot tasks on new categories

❌ Do NOT use for:

• Counting objects in images
• Fine-grained classification
• Spatial understanding
• Attribute binding
• Negation handling

Installation

pip install transformers pillow torch sentence-transformers --break-system-packages

Validation: Run python scripts/validate_setup.py

Basic Usage

Image Search

from transformers import CLIPProcessor, CLIPModel
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

# Embed images
images = [Image.open(f"img{i}.jpg") for i in range(10)]
inputs = processor(images=images, return_tensors="pt")
image_features = model.get_image_features(**inputs)

# Search with text
text_inputs = processor(text=["a beach at sunset"], return_tensors="pt")
text_features = model.get_text_features(**text_inputs)

# Compute similarity
similarity = (image_features @ text_features.T).softmax(dim=0)

Common Anti-Patterns

Anti-Pattern 1: "CLIP for Everything"

❌ Wrong:

# Using CLIP to count cars in an image
prompt = "How many cars are in this image?"
# CLIP cannot count - it will give nonsense results

Why wrong: CLIP's architecture collapses spatial information into a single vector. It literally cannot count.

✓ Right:

from transformers import DetrImageProcessor, DetrForObjectDetection

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")

# Detect objects
results = model(**processor(images=image, return_tensors="pt"))
# Filter for cars and count
car_detections = [d for d in results if d['label'] == 'car']
count = len(car_detections)

How to detect: If query contains "how many", "count", or numeric questions → Use object detection

---

Anti-Pattern 2: Fine-Grained Classification

❌ Wrong:

# Trying to identify specific celebrities with CLIP
prompts = ["Tom Hanks", "Brad Pitt", "Morgan Freeman"]
# CLIP will perform poorly - not trained for fine-grained face ID

Why wrong: CLIP trained on coarse categories. Fine-grained faces, car models, flower species require specialized models.

✓ Right:

# Use a fine-tuned face recognition model
from transformers import AutoFeatureExtractor, AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(
    "microsoft/resnet-50"  # Then fine-tune on celebrity dataset
)
# Or use dedicated face recognition: ArcFace, CosFace

How to detect: If query asks to distinguish between similar items in same category → Use specialized model

---

Anti-Pattern 3: Spatial Understanding

❌ Wrong:

# CLIP cannot understand spatial relationships
prompts = [
    "cat to the left of dog",
    "cat to the right of dog"
]
# Will give nearly identical scores

Why wrong: CLIP embeddings lose spatial topology. "Left" and "right" are treated as bag-of-words.

✓ Right:

# Use a spatial reasoning model
# Examples: GQA models, Visual Genome models, SWIG
from swig_model import SpatialRelationModel

model = SpatialRelationModel()
result = model.predict_relation(image, "cat", "dog")
# Returns: "left", "right", "above", "below", etc.

How to detect: If query contains directional words (left, right, above, under, next to) → Use spatial model

---

Anti-Pattern 4: Attribute Binding

❌ Wrong:

prompts = [
    "red car and blue truck",
    "blue car and red truck"
]
# CLIP often gives similar scores for both

Why wrong: CLIP cannot bind attributes to objects. It sees "red, blue, car, truck" as a bag of concepts.

✓ Right - Use PC-CLIP or DCSMs:

# PC-CLIP: Fine-tuned for pairwise comparisons
from pc_clip import PCCLIPModel

model = PCCLIPModel.from_pretrained("pc-clip-vit-l")
# Or use DCSMs (Dense Cosine Similarity Maps)

How to detect: If query has multiple objects with different attributes → Use compositional model

---

Evolution Timeline

2021: CLIP Released

• Revolutionary: zero-shot, 400M image-text pairs
• Widely adopted for everything
• Limitations not yet understood

2022-2023: Limitations Discovered

• Cannot count objects
• Poor at fine-grained classification
• Fails spatial reasoning
• Can't bind attributes

2024: Alternatives Emerge

• DCSMs: Preserve patch/token topology
• PC-CLIP: Trained on pairwise comparisons
• SpLiCE: Sparse interpretable embeddings

2025: Current Best Practices

• Use CLIP for what it's good at
• Task-specific models for limitations
• Compositional models for complex queries

LLM Mistake: LLMs trained on 2021-2023 data will suggest CLIP for everything because limitations weren't widely known. This skill corrects that.

---

Validation Script

Before using CLIP, check if it's appropriate:

python scripts/validate_clip_usage.py \
    --query "your query here" \
    --check-all

Returns:

• ✅ CLIP is appropriate
• ❌ Use alternative (with suggestion)

Task-Specific Guidance

Image Search (CLIP ✓)

# Good use of CLIP
queries = ["beach", "mountain", "city skyline"]
# Works well for broad semantic concepts

Zero-Shot Classification (CLIP ✓)

# Good: Broad categories
categories = ["indoor", "outdoor", "nature", "urban"]
# CLIP excels at this

Object Counting (CLIP ✗)

# Use object detection instead
from transformers import DetrImageProcessor, DetrForObjectDetection
# See /references/object_detection.md

Fine-Grained Classification (CLIP ✗)

# Use specialized models
# See /references/fine_grained_models.md

Spatial Reasoning (CLIP ✗)

# Use spatial relation models
# See /references/spatial_models.md

---

Troubleshooting

Issue: CLIP gives unexpected results

Check:

1. Is this a counting task? → Use object detection
2. Fine-grained classification? → Use specialized model
3. Spatial query? → Use spatial model
4. Multiple objects with attributes? → Use compositional model

Validation:

python scripts/diagnose_clip_issue.py --image path/to/image --query "your query"

Issue: Low similarity scores

Possible causes:

1. Query too specific (CLIP works better with broad concepts)
2. Fine-grained task (not CLIP's strength)
3. Need to adjust threshold

Solution: Try broader query or use alternative model

---

Model Selection Guide

Model	Best For	Avoid For
CLIP ViT-L/14	Semantic search, broad categories	Counting, fine-grained, spatial
DETR	Object detection, counting	Semantic similarity
DINOv2	Fine-grained features	Text-image matching
PC-CLIP	Attribute binding, comparisons	General embedding
DCSMs	Compositional reasoning	Simple similarity

Performance Notes

CLIP models:

• ViT-B/32: Fast, lower quality
• ViT-L/14: Balanced (recommended)
• ViT-g-14: Highest quality, slower

Inference time (single image, CPU):

• ViT-B/32: ~100ms
• ViT-L/14: ~300ms
• ViT-g-14: ~1000ms

Further Reading

• /references/clip_limitations.md - Detailed analysis of CLIP's failures
• /references/alternatives.md - When to use what model
• /references/compositional_reasoning.md - DCSMs and PC-CLIP deep dive
• /scripts/validate_clip_usage.py - Pre-flight validation tool
• /scripts/diagnose_clip_issue.py - Debug unexpected results

---

See CHANGELOG.md for version history.