Prepare Dataset

Load, preprocess, and validate datasets for machine learning model training including normalization and augmentation.

When to Use

• Setting up data pipelines for training
• Normalizing and cleaning raw data
• Splitting into train/validation/test sets
• Applying data augmentation

Quick Reference

# Dataset preparation pipeline
class DatasetLoader:
    def load(self, path: str) -> Tuple[ndarray, ndarray]:
        # Load raw data
        pass

    def normalize(self, data: ndarray) -> ndarray:
        # Normalize to [0, 1] or standardize
        pass

    def split(self, data: ndarray, ratios: Tuple[float, float, float]):
        # Split into train/val/test
        pass

    def augment(self, data: ndarray) -> ndarray:
        # Apply transformations if needed
        pass

Workflow

1. Load raw data: Read dataset from file (CSV, HDF5, NumPy)
2. Validate data: Check shape, dtype, missing values
3. Preprocess: Normalize, standardize, encode categorical features
4. Split sets: Create train/validation/test splits
5. Augment data: Apply transformations if needed (rotation, flip, etc.)

Output Format

Dataset preparation report:

• Raw data shape and statistics
• Data validation results (missing values, outliers)
• Preprocessing applied (normalization, encoding)
• Train/val/test split sizes
• Final dataset shape and statistics
• Augmentation transformations applied

References

• See extract-hyperparameters skill for data preprocessing config
• See evaluate-model skill for test set evaluation
• See /notes/review/mojo-ml-patterns.md for Mojo data loading