GitHub Copilot PR Review Workflow

Experiment Overview

Context

GitHub Copilot provides code review suggestions on pull requests. These suggestions need to be reviewed programmatically using the gh CLI since WebFetch returns 404 for private repos. This workflow covers setting up access and efficiently reviewing/applying suggestions.

Prerequisites: GitHub CLI Setup

Install gh CLI

Windows (manual download):

# Download from https://github.com/cli/cli/releases
# Extract to a permanent location
# Add to PATH

Verify installation:

gh --version
# gh version 2.63.2 (2024-12-18)

Authenticate for Private Repos

# Interactive login - opens browser
gh auth login

# Select: GitHub.com > HTTPS > Login with a web browser
# Follow the prompts to authenticate

Verify authentication:

gh auth status
# github.com
#   Logged in to github.com account username (keyring)
#   Git operations for github.com configured to use https protocol.

Verified Workflow

1. View PR Details

# Get PR metadata
gh pr view 36 --repo owner/repo --json title,body,state,additions,deletions,changedFiles,commits,mergeable

# Check if PR is mergeable
gh pr view 36 --repo owner/repo --json mergeable

2. List Copilot Suggestions

# Get all review comments (includes Copilot suggestions)
gh api repos/owner/repo/pulls/36/comments

# Filter for Copilot suggestions (look for user.login = "github-copilot[bot]")
gh api repos/owner/repo/pulls/36/comments --jq '.[] | select(.user.login == "github-copilot[bot]") | {path, body, line}'

Example Copilot suggestion format:

{
  "path": "alpaca_trading/gpu/vae_ood_detector.py",
  "body": "**Suggestion:** Use `torch.quantile` instead of `kthvalue` for percentile...",
  "line": 245,
  "diff_hunk": "@@ -240,6 +240,10 @@ def _compute_threshold(self, errors)..."
}

3. Categorize Suggestions

Create a table to track suggestions:

4. Apply Fixes

For each valid suggestion:

1. Read the file to understand context
2. Make the fix using Edit tool
3. Verify syntax with python -m py_compile file.py
4. Commit incrementally for traceability

# After all fixes
git add -A
git commit -m "fix: Apply Copilot suggestions

- Use torch.quantile for percentile calculations
- Fix LayerNorm dimension for Conv1d outputs
- Handle buffer edge cases for last transitions
- Add arXiv citations to docstrings
- Remove unused imports"

git push origin branch-name

5. Handle Multiple PRs with Overlapping Suggestions

When PR #35 and PR #36 have overlapping Copilot suggestions:

1. Identify the target PR (the one you'll merge)
2. Review suggestions from both PRs
3. Apply all valid suggestions to target PR only
4. Close superseded PR with explanation

# Close superseded PR
gh pr close 35 --repo owner/repo --comment "Superseded by #36 which includes all valid Copilot suggestions from both PRs."

6. Verify All Suggestions Applied

# Re-check Copilot suggestions after push
gh api repos/owner/repo/pulls/36/comments --jq 'length'

# Run tests
python -m pytest tests/test_affected_module.py -v

API Reference

Get PR Comments (All Types)

gh api repos/{owner}/{repo}/pulls/{pr_number}/comments

Get Review Threads

gh api repos/{owner}/{repo}/pulls/{pr_number}/reviews

Create PR

gh pr create --base stable --title "Title" --body "Description"

Close PR

gh pr close {pr_number} --repo owner/repo --comment "Reason"

View PR Files Changed

gh pr view {pr_number} --repo owner/repo --json files --jq '.files[].path'

Failed Attempts

Key Insights

• gh CLI is essential for private repos: WebFetch returns 404
• Copilot suggestions are in PR review comments on code lines: These are GitHub "review comments" (from pulls/{pr_number}/comments), not general PR discussion comments; filter by user.login.
• Categorize before fixing: Create a table to track all suggestions
• One target PR: When multiple PRs overlap, pick one and close others
• Commit messages matter: Reference the specific fix for traceability
• Verify after each fix: Syntax check prevents build failures

Common Copilot Suggestion Categories

References

• gh CLI documentation: https://cli.github.com/manual/
• GitHub REST API for PRs: https://docs.github.com/en/rest/pulls
• Example session: Reviewed 48 Copilot suggestions across PR #35 and #36