Git Security Checks

Expert guidance for pre-commit security validation and secret detection using detect-secrets and pre-commit hooks.

Core Expertise

• detect-secrets: Scan for hardcoded secrets and credentials
• Pre-commit Hooks: Automated security validation before commits
• Secrets Baseline: Manage false positives and legitimate secrets
• Security-First Workflow: Prevent credential leaks before they happen

detect-secrets Workflow

Initial Setup

# Install detect-secrets
pip install detect-secrets

# Create initial baseline
detect-secrets scan > .secrets.baseline

# Audit baseline for false positives
detect-secrets audit .secrets.baseline

Pre-commit Scan Workflow

Run detect-secrets before every commit:

# Scan for new secrets (using existing baseline)
detect-secrets scan --baseline .secrets.baseline

# If new secrets detected, audit them
detect-secrets audit .secrets.baseline

# Stage the updated baseline
git add .secrets.baseline

Audit Process

When new secrets are detected:

# Run audit to review flagged items
detect-secrets audit .secrets.baseline

# For each detected secret:
# - Press 'y' if it's a real secret (DON'T COMMIT)
# - Press 'n' if it's a false positive (safe to commit)
# - Press 's' to skip for now

# After audit, re-scan to update baseline
detect-secrets scan --baseline .secrets.baseline

Complete Pre-commit Security Flow

# 1. Scan for secrets with baseline
detect-secrets scan --baseline .secrets.baseline

# 2. If baseline updated, audit new findings
detect-secrets audit .secrets.baseline

# 3. Stage the updated baseline
git add .secrets.baseline

# 4. Run all pre-commit hooks
pre-commit run --all-files --show-diff-on-failure

# 5. Stage your actual changes
git add src/file.ts

# 6. Show what's staged
git status
git diff --cached --stat

# 7. Commit if everything passes
git commit -m "feat(auth): add authentication module"

Pre-commit Hook Integration

.pre-commit-config.yaml

Example configuration with detect-secrets:

repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']
        exclude: package-lock.json

Running Pre-commit Hooks

# Run all hooks on all files
pre-commit run --all-files

# Run all hooks on staged files only
pre-commit run

# Run specific hook
pre-commit run detect-secrets

# Show diff on failure for debugging
pre-commit run --all-files --show-diff-on-failure

# Install hooks to run automatically on commit
pre-commit install

Common Secret Patterns

detect-secrets scans for:

• API Keys: AWS, GitHub, Stripe, etc.
• Authentication Tokens: JWT, OAuth tokens, session tokens
• Passwords: Hardcoded passwords in config files
• Private Keys: RSA, SSH, PGP private keys
• Database Credentials: Connection strings with passwords
• Generic Secrets: High-entropy strings that look like secrets

Examples of What Gets Detected

# ❌ DETECTED: Hardcoded API key
API_KEY = "sk_live_abc123def456ghi789"

# ❌ DETECTED: AWS credentials
aws_access_key_id = AKIAIOSFODNN7EXAMPLE

# ❌ DETECTED: Database password
DB_URL = "postgresql://user:Pa$$w0rd@localhost/db"

# ❌ DETECTED: Private key
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA...

Managing False Positives

Excluding Files

In .secrets.baseline:

# Exclude specific files from scanning
detect-secrets scan --exclude-files 'package-lock\.json' > .secrets.baseline
detect-secrets scan --exclude-files '.*\.lock$' > .secrets.baseline
detect-secrets scan --exclude-files 'test/.*\.py' > .secrets.baseline

Inline Ignore Comments

# In code, mark false positives
api_key = "test-key-1234"  # pragma: allowlist secret

# Or use detect-secrets specific pragma
password = "fake-password"  # pragma: allowlist nextline secret

Baseline Management

# Update baseline to include current state
detect-secrets scan --baseline .secrets.baseline --update

# Re-audit all secrets in baseline
detect-secrets audit .secrets.baseline

# Show secrets in baseline
cat .secrets.baseline | jq '.results'

Security Best Practices

Never Commit Secrets

• Use environment variables: Store secrets in .env files (gitignored)
• Use secret managers: AWS Secrets Manager, HashiCorp Vault, etc.
• Use CI/CD secrets: GitHub Secrets, GitLab CI/CD variables
• Rotate leaked secrets: If accidentally committed, rotate immediately

Secrets File Management

# Example .gitignore for secrets
.env
.env.local
.env.*.local
*.pem
*.key
credentials.json
config/secrets.yml
.api_tokens

Handling Legitimate Secrets in Repo

For test fixtures or examples:

# 1. Use obviously fake values
API_KEY = "fake-key-for-testing-only"

# 2. Use placeholders
API_KEY = "<your-api-key-here>"

# 3. Mark in baseline as false positive
detect-secrets audit .secrets.baseline  # mark as 'n'

Emergency: Secret Leaked to Git History

If a secret is committed and pushed:

Immediate Actions

# 1. ROTATE THE SECRET IMMEDIATELY
# - Change passwords, revoke API keys, regenerate tokens
# - Do this BEFORE cleaning git history

# 2. Remove from current commit (if just committed)
git reset --soft HEAD~1
# Remove secret from files
git add .
git commit -m "fix(security): remove leaked credentials"

# 3. Force push (if not shared widely)
git push --force-with-lease origin branch-name

Full History Cleanup

# Use git-filter-repo to remove from all history
pip install git-filter-repo

# Remove specific file from all history
git filter-repo --path path/to/secret/file --invert-paths

# Remove specific string from all files
git filter-repo --replace-text <(echo "SECRET_KEY=abc123==>SECRET_KEY=REDACTED")

Prevention

# Always run security checks before committing
pre-commit run detect-secrets

# Check what's being committed
git diff --cached

# Use .gitignore for sensitive files
echo ".env" >> .gitignore
echo ".api_tokens" >> .gitignore

Workflow Integration

Daily Development Flow

# Before staging any files
detect-secrets scan --baseline .secrets.baseline
pre-commit run --all-files

# If secrets detected
detect-secrets audit .secrets.baseline
# Review and mark false positives

# Stage changes
git add .secrets.baseline  # If updated
git add src/feature.ts

# Final check before commit
git diff --cached  # Review changes
detect-secrets scan --baseline .secrets.baseline  # One more scan

# Commit
git commit -m "feat(feature): add new capability"

CI/CD Integration

# Example GitHub Actions workflow
name: Security Checks

on: [push, pull_request]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install detect-secrets
        run: pip install detect-secrets
      - name: Scan for secrets
        run: detect-secrets scan --baseline .secrets.baseline --fail-on-unaudited

Troubleshooting

Baseline Out of Sync

# Re-generate baseline from scratch
detect-secrets scan > .secrets.baseline.new
detect-secrets audit .secrets.baseline.new
mv .secrets.baseline.new .secrets.baseline

Too Many False Positives

# Exclude file patterns
detect-secrets scan --exclude-files 'test/.*' > .secrets.baseline

# Reduce sensitivity (use cautiously)
detect-secrets scan --base64-limit 4.5 > .secrets.baseline

Pre-commit Hook Failing

# Run pre-commit in verbose mode
pre-commit run detect-secrets --verbose

# Check baseline file exists
ls -la .secrets.baseline

# Update pre-commit hooks
pre-commit autoupdate

Secret Detected But File Not Changed

# Baseline may be stale
detect-secrets scan --baseline .secrets.baseline --update

# Audit to clear false positives
detect-secrets audit .secrets.baseline

Tools Reference

detect-secrets Commands

# Scan for secrets
detect-secrets scan

# Scan with baseline
detect-secrets scan --baseline .secrets.baseline

# Audit baseline
detect-secrets audit .secrets.baseline

# Update baseline
detect-secrets scan --baseline .secrets.baseline --update

# Exclude files
detect-secrets scan --exclude-files 'pattern'

# Custom plugins
detect-secrets scan --list-all-plugins

pre-commit Commands

# Install hooks
pre-commit install

# Run all hooks
pre-commit run --all-files

# Run specific hook
pre-commit run detect-secrets

# Update hook versions
pre-commit autoupdate

# Uninstall hooks
pre-commit uninstall