Large Scale Text Editing

Overview

This skill provides guidance for efficiently transforming large text files using text editors (primarily Vim). It covers pattern recognition from input/output samples, macro-based transformations, and verification strategies for bulk operations on files with thousands to millions of rows.

When to Use This Skill

• Transforming CSV or delimited text files with many rows
• Bulk find-and-replace operations with regex patterns
• Column reordering, reformatting, or restructuring text data
• Tasks with keystroke or efficiency constraints
• Any text transformation where a sample input/expected output is provided

Approach

1. Understand the Transformation Requirements

Before writing any transformation code or macros:

1. Sample the data appropriately - Never attempt to read entire large files directly. Use head, tail, or sed to extract samples:

   head -n 20 input.csv > sample_input.csv
   head -n 20 expected.csv > sample_expected.csv
   

2. Compare input and expected output - Identify all differences:

   • Whitespace changes (trimming, removal)
   • Case changes (uppercase, lowercase)
   • Column/field reordering
   • Delimiter changes
   • Content transformations

3. Spot-check throughout the file - Do not assume all rows follow the same pattern:

   sed -n '1p;500000p;999999p' input.csv  # Check beginning, middle, end
   

4. Check for edge cases in the data:

   • Empty fields
   • Fields with more or fewer delimiters than expected
   • Special characters that may interfere with regex
   • Quoted fields containing delimiters

2. Plan the Transformation

Decompose the transformation into discrete, testable operations:

1. List each transformation step (e.g., remove whitespace, uppercase, reorder columns)
2. Order steps logically - Some transformations may depend on others
3. Consider efficiency - If keystroke constraints exist, plan to minimize redundant operations

3. Test on a Small Sample First

Always verify the transformation works correctly before applying to the full file:

1. Create a backup of the original file before any modifications:

   cp input.csv input.csv.backup
   

2. Test on a small subset:

   head -n 100 input.csv > test_input.csv
   # Apply transformation to test_input.csv
   # Compare with expected output
   

3. Validate results match expected output exactly:

   diff test_output.csv expected_subset.csv
   

4. Apply to Full File

Only after successful small-scale testing:

1. Ensure backup exists
2. Apply the transformation
3. Verify immediately (see verification strategies below)

Vim-Specific Guidance

Macro-Based Transformations

For repetitive operations across many lines:

1. Record macros for each distinct operation:

   • qa to start recording into register a
   • Perform the operation on one line
   • q to stop recording
   • @a to replay, 100@a to replay 100 times

2. Use :normal for applying normal-mode commands:

   :%normal @a    " Apply macro a to all lines
   

3. Use substitution for global changes:

   :%s/\s\+//g           " Remove all whitespace
   :%s/.*/\U&/           " Uppercase entire line
   :%s/\([^,]*\),\([^,]*\),\([^,]*\)/\3,\2,\1/  " Reorder 3 columns
   

Counting Keystrokes

If there are keystroke constraints:

1. Count actual keystrokes, not escape sequences - In Vimscript strings, \\s represents \s (one escape + one character)
2. Document each macro's keystroke count separately
3. Consider that : commands count each character including the colon and Enter

Vim Script Files

For reproducible transformations, create a Vim script:

" transform.vim
" Load macros
let @a = 'macro_content_here'
let @b = 'another_macro_here'

" Apply transformations
%s/pattern/replacement/g
%normal @a

" Save and exit
wq

Run with: vim -s transform.vim input.csv

Verification Strategies

Mandatory Checks

1. Byte-for-byte comparison with expected output:

   diff output.csv expected.csv
   echo $?  # Should be 0 for identical files
   

2. Line count verification:

   wc -l output.csv expected.csv  # Must match
   

3. Spot-check random rows:

   diff <(sed -n '12345p' output.csv) <(sed -n '12345p' expected.csv)
   

Additional Validation

• Check file size - Transformed file should be in expected size range
• Verify no data loss - Row counts should match (unless filtering)
• Check encoding - Ensure no corruption of special characters
• Validate structure - Correct number of columns/delimiters per row

Common Pitfalls

Data Handling

Vim-Specific

Verification

Decision Tree

Task: Transform large text file
│
├─ Is the file > 1000 lines?
│  ├─ YES → Sample data first (head/tail/sed -n)
│  └─ NO → May read directly, but sampling still recommended
│
├─ Is there an expected output file?
│  ├─ YES → Compare samples to understand transformation
│  └─ NO → Clarify expected output format before proceeding
│
├─ Are there keystroke constraints?
│  ├─ YES → Plan macros carefully, count keystrokes accurately
│  └─ NO → Optimize for correctness over keystroke efficiency
│
├─ Ready to transform?
│  ├─ Create backup first
│  ├─ Test on small sample
│  ├─ Verify sample output matches expected
│  └─ Only then apply to full file
│
└─ After transformation:
   ├─ diff against expected output
   ├─ Verify line counts match
   └─ Spot-check random rows throughout file

Resources

For detailed Vim commands and regex patterns, refer to references/vim_editing_guide.md.