torchaudio

Overview

TorchAudio provides signal processing tools for PyTorch, enabling users to treat audio processing as part of the neural network graph. This allow transforms to be run on GPUs and handled via nn.Sequential pipelines.

When to Use

Use TorchAudio for converting raw audio waveforms into features like Mel Spectrograms, performing data augmentation (SpecAugment), or when high-performance resampling is required.

Decision Tree

1. Do you need to transform many audio files quickly?
   • MOVE: The transform module to GPU using .to('cuda').
2. Are you training an Automatic Speech Recognition (ASR) model?
   • USE: SpecAugment (TimeMasking, FrequencyMasking) on the spectrogram.
3. Do you need to align text to audio?
   • USE: The forced_align functional API with a Wav2Vec2 model.

Workflows

1. Audio Feature Extraction Pipeline

   1. Load the waveform as a PyTorch tensor.
   2. Apply Resample to match the target frequency (e.g., 16000Hz).
   3. Use Spectrogram to convert to a power/amplitude scale.
   4. Convert the spectrogram to mel-scale using MelScale.
   5. Optionally apply AmplitudeToDB to get decibel values.

2. GPU-Accelerated Data Augmentation

   1. Define an augmentation pipeline using nn.Sequential (e.g., TimeStretch, FrequencyMasking).
   2. Move the pipeline to the GPU (.to('cuda')).
   3. Process batches of spectrograms directly on the device to avoid CPU bottlenecks.

3. Pitch and Speed Perturbation

   1. Use torchaudio.transforms.PitchShift for pitch changes without affecting duration.
   2. Apply torchaudio.transforms.Speed for speed changes (which also affects pitch).
   3. Iterate through a range of factors to create a diverse augmented dataset for speech recognition.

Non-Obvious Insights

• GPU Efficiency: Moving audio transformations like MelScale to the GPU allows the entire feature extraction process to happen in parallel with model training, eliminating CPU data-loading bottlenecks.
• ASR Robustness: SpecAugment techniques (Time/Frequency Masking) should be applied to the spectrogram, not the raw waveform, as they are designed to simulate missing acoustic information in the frequency domain.
• Library Evolution: Starting with version 2.8, TorchAudio has entered maintenance, with some low-level decoding capabilities consolidated into the TorchCodec project.

Evidence

• "Transforms are implemented using torch.nn.Module... common ways to build a processing pipeline are to chain Modules together using torch.nn.Sequential." (https://pytorch.org/audio/stable/transforms.html)
• "PitchShift: Shift the pitch of a waveform by n_steps steps." (https://pytorch.org/audio/stable/transforms.html)

Scripts

• scripts/torchaudio_tool.py: Utility for building an audio feature pipeline.
• scripts/torchaudio_tool.js: Node.js wrapper for processing audio files via TorchAudio.

Dependencies

• torchaudio
• torch

References

• TorchAudio Reference