name	fiftyone-find-duplicates
description	Find duplicate or near-duplicate images in FiftyOne datasets using brain similarity computation. Use when users want to deduplicate datasets, find similar images, cluster visually similar content, or remove redundant samples. Requires FiftyOne MCP server with @voxel51/brain plugin installed.

Find Duplicates in FiftyOne Datasets

Overview

Find and remove duplicate or near-duplicate images using FiftyOne's brain similarity operators. Uses deep learning embeddings to identify visually similar images.

Use this skill when:

Removing duplicate images from datasets
Finding near-duplicate images (similar but not identical)
Clustering visually similar images
Cleaning datasets before training

Prerequisites

FiftyOne MCP server installed and running
@voxel51/brain plugin installed and enabled
Dataset with image samples loaded in FiftyOne

Key Directives

ALWAYS follow these rules:

1. Set context first

set_context(dataset_name="my-dataset")

2. Launch FiftyOne App

Brain operators are delegated and require the app:

launch_app()

Wait 5-10 seconds for initialization.

3. Discover operators dynamically

# List all brain operators
list_operators(builtin_only=False)

# Get schema for specific operator
get_operator_schema(operator_uri="@voxel51/brain/compute_similarity")

4. Compute embeddings before finding duplicates

execute_operator(
    operator_uri="@voxel51/brain/compute_similarity",
    params={"brain_key": "img_sim", "model": "mobilenet-v2-imagenet-torch"}
)

5. Close app when done

close_app()

Complete Workflow

Step 1: Setup

# Set context
set_context(dataset_name="my-dataset")

# Launch app (required for brain operators)
launch_app()

Step 2: Verify Brain Plugin

# Check if brain plugin is available
list_plugins(enabled=True)

# If not installed:
download_plugin(
    url_or_repo="voxel51/fiftyone-plugins",
    plugin_names=["@voxel51/brain"]
)
enable_plugin(plugin_name="@voxel51/brain")

Step 3: Discover Brain Operators

# List all available operators
list_operators(builtin_only=False)

# Get schema for compute_similarity
get_operator_schema(operator_uri="@voxel51/brain/compute_similarity")

# Get schema for find_duplicates
get_operator_schema(operator_uri="@voxel51/brain/find_duplicates")

Step 4: Compute Similarity

# Execute operator to compute embeddings
execute_operator(
    operator_uri="@voxel51/brain/compute_similarity",
    params={
        "brain_key": "img_duplicates",
        "model": "mobilenet-v2-imagenet-torch"
    }
)

Step 5: Find Near Duplicates

execute_operator(
    operator_uri="@voxel51/brain/find_near_duplicates",
    params={
        "similarity_index": "img_duplicates",
        "threshold": 0.3
    }
)

Threshold guidelines (distance-based, lower = more similar):

0.1 = Very similar (near-exact duplicates)
0.3 = Near duplicates (recommended default)
0.5 = Similar images
0.7 = Loosely similar

This operator creates two saved views automatically:

near duplicates: all samples that are near duplicates
representatives of near duplicates: one representative from each group

Step 6: View Duplicates in App

After finding duplicates, use set_view to display them in the FiftyOne App:

Option A: Filter by near_dup_id field

# Show all samples that have a near_dup_id (all duplicates)
set_view(exists=["near_dup_id"])

Option B: Show specific duplicate group

# Show samples with a specific duplicate group ID
set_view(filters={"near_dup_id": 1})

Option C: Load saved view (if available)

# Load the automatically created saved view
set_view(view_name="near duplicates")

Option D: Clear filter to show all samples

clear_view()

The find_near_duplicates operator adds a near_dup_id field to samples. Samples with the same ID are duplicates of each other.

Step 7: Delete Duplicates

Option A: Use deduplicate operator (keeps one representative per group)

execute_operator(
    operator_uri="@voxel51/brain/deduplicate_near_duplicates",
    params={}
)

Option B: Manual deletion from App UI

Use set_view(exists=["near_dup_id"]) to show duplicates
Review samples in the App at http://localhost:5151/
Select samples to delete
Use the delete action in the App

Step 8: Clean Up

close_app()

Available Tools

Session View Tools

Tool	Description
`set_view(exists=[...])`	Filter samples where field(s) have non-None values
`set_view(filters={...})`	Filter samples by exact field values
`set_view(tags=[...])`	Filter samples by tags
`set_view(sample_ids=[...])`	Select specific sample IDs
`set_view(view_name="...")`	Load a saved view by name
`clear_view()`	Clear filters, show all samples

Brain Operators for Duplicates

Use list_operators() to discover and get_operator_schema() to see parameters:

Operator	Description
`@voxel51/brain/compute_similarity`	Compute embeddings and similarity index
`@voxel51/brain/find_near_duplicates`	Find near-duplicate samples
`@voxel51/brain/deduplicate_near_duplicates`	Delete duplicates, keep representatives
`@voxel51/brain/find_exact_duplicates`	Find exact duplicate media files
`@voxel51/brain/deduplicate_exact_duplicates`	Delete exact duplicates
`@voxel51/brain/compute_uniqueness`	Compute uniqueness scores

Common Use Cases

Use Case 1: Remove Exact Duplicates

For accidentally duplicated files (identical bytes):

set_context(dataset_name="my-dataset")
launch_app()

execute_operator(
    operator_uri="@voxel51/brain/find_exact_duplicates",
    params={}
)

execute_operator(
    operator_uri="@voxel51/brain/deduplicate_exact_duplicates",
    params={}
)

close_app()

Use Case 2: Find and Review Near Duplicates

For visually similar but not identical images:

set_context(dataset_name="my-dataset")
launch_app()

# Compute embeddings
execute_operator(
    operator_uri="@voxel51/brain/compute_similarity",
    params={"brain_key": "near_dups", "model": "mobilenet-v2-imagenet-torch"}
)

# Find duplicates
execute_operator(
    operator_uri="@voxel51/brain/find_near_duplicates",
    params={"similarity_index": "near_dups", "threshold": 0.3}
)

# View duplicates in the App
set_view(exists=["near_dup_id"])

# After review, deduplicate
execute_operator(
    operator_uri="@voxel51/brain/deduplicate_near_duplicates",
    params={}
)

# Clear view and close
clear_view()
close_app()

Use Case 3: Sort by Similarity

Find images similar to a specific sample:

set_context(dataset_name="my-dataset")
launch_app()

execute_operator(
    operator_uri="@voxel51/brain/compute_similarity",
    params={"brain_key": "search"}
)

execute_operator(
    operator_uri="@voxel51/brain/sort_by_similarity",
    params={
        "brain_key": "search",
        "query_id": "sample_id_here",
        "k": 20
    }
)

close_app()

Troubleshooting

Error: "No executor available"

Cause: Delegated operators require the App executor for UI triggers
Solution: Direct user to App UI to view results and complete deletion manually
Affected operators: find_near_duplicates, deduplicate_near_duplicates

Error: "Brain key not found"

Cause: Embeddings not computed
Solution: Run compute_similarity first with a brain_key

Error: "Operator not found"

Cause: Brain plugin not installed
Solution: Install with download_plugin() and enable_plugin()

Error: "Missing dependency" (e.g., torch, tensorflow)

The MCP server detects missing dependencies automatically
Response includes missing_package and install_command

Example response:

{
  "error_type": "missing_dependency",
  "missing_package": "torch",
  "install_command": "pip install torch"
}

Offer to run the install command for the user
After installation, restart MCP server and retry

Similarity computation is slow

Use faster model: mobilenet-v2-imagenet-torch
Use GPU if available
Process large datasets in batches

Best Practices

Discover dynamically - Use list_operators() and get_operator_schema() to get current operator names and parameters
Start with default threshold (0.3) and adjust as needed
Review before deleting - Direct user to App to inspect duplicates
Store embeddings - Reuse for multiple operations via brain_key
Handle executor errors gracefully - Guide user to App UI when needed

Performance Notes

Embedding computation time:

1,000 images: ~1-2 minutes
10,000 images: ~10-15 minutes
100,000 images: ~1-2 hours

Memory requirements:

~2KB per image for embeddings
~4-8KB per image for similarity index

fiftyone-find-duplicates

Install Skill

SKILL.md