AI Tools Compared

An AI code review bot runs on every pull request, posts inline comments on specific lines, and enforces the rules your team cares about — without a human having to review every diff. This guide builds a working GitHub Actions bot that reviews PRs using Claude, posts comments as a GitHub App, and runs in under 60 seconds.

Architecture

The bot runs as a GitHub Actions workflow triggered on pull_request events. It:

  1. Fetches the PR diff via the GitHub API
  2. Splits the diff into reviewable hunks
  3. Sends each hunk to Claude with your custom review rules
  4. Posts inline comments using the GitHub Review API
  5. Submits a summary review (APPROVE, REQUEST_CHANGES, or COMMENT)

Step 1: Create the GitHub App

  1. Go to Settings > Developer settings > GitHub Apps > New
  2. Permissions: Pull requests (Read & write), Contents (Read)
  3. Install on your target repositories
  4. Generate a private key, download the PEM file

Add to repository secrets: APP_ID and APP_PRIVATE_KEY.

Step 2: GitHub Actions Workflow

# .github/workflows/ai-review.yml
name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    if: |
      github.event.pull_request.draft == false &&
      github.actor != 'dependabot[bot]'
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm install @anthropic-ai/sdk @octokit/app @octokit/rest
      - name: Run AI Review
        env:
          APP_ID: ${{ secrets.APP_ID }}
          APP_PRIVATE_KEY: ${{ secrets.APP_PRIVATE_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO_OWNER: ${{ github.repository_owner }}
          REPO_NAME: ${{ github.event.repository.name }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
        run: node .github/scripts/ai-review.js

Step 3: The Review Script

// .github/scripts/ai-review.js
import { App } from '@octokit/app';
import Anthropic from '@anthropic-ai/sdk';

const REVIEW_RULES = `You are a senior software engineer doing a code review.

Review for:
1. Security vulnerabilities (injection, auth issues, exposed secrets)
2. Logic errors and off-by-one bugs
3. Missing error handling (unhandled promises, missing null checks)
4. Performance issues (N+1 queries, unnecessary re-computation)

Do NOT comment on: code style, formatting, naming, or missing tests.

Respond with JSON:
{
  "comments": [
    {
      "path": "src/api/users.ts",
      "line": 42,
      "severity": "error" | "warning",
      "body": "Explanation and how to fix"
    }
  ],
  "summary": "Brief assessment",
  "decision": "APPROVE" | "REQUEST_CHANGES" | "COMMENT"
}`;

async function main() {
  const app = new App({
    appId: process.env.APP_ID,
    privateKey: process.env.APP_PRIVATE_KEY,
  });

  const octokit = await app.getInstallationOctokit(await getInstallationId(app));
  const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

  const { data: diff } = await octokit.rest.pulls.get({
    owner: process.env.REPO_OWNER,
    repo: process.env.REPO_NAME,
    pull_number: parseInt(process.env.PR_NUMBER),
    mediaType: { format: 'diff' }
  });

  const files = parseDiff(diff);
  if (files.length > 50) { console.log('Skipping: too many files'); return; }

  const reviewableFiles = files.filter(f =>
    !f.path.match(/\.(lock|snap|min\.(js|css)|pb\.go)/)
  );

  const allComments = [];
  let overallDecision = 'APPROVE';
  const batchSize = 10;

  for (let i = 0; i < reviewableFiles.length; i += batchSize) {
    const batch = reviewableFiles.slice(i, i + batchSize);
    const diffText = batch.map(f => `--- ${f.path}\n${f.diff}`).join('\n\n');

    const response = await anthropic.messages.create({
      model: 'claude-haiku-4-5',
      max_tokens: 2048,
      system: REVIEW_RULES,
      messages: [{ role: 'user', content: `Review this PR diff:\n\n${diffText}` }]
    });

    const result = JSON.parse(extractJson(response.content[0].text));
    allComments.push(...result.comments);
    if (result.decision === 'REQUEST_CHANGES') overallDecision = 'REQUEST_CHANGES';
  }

  await octokit.rest.pulls.createReview({
    owner: process.env.REPO_OWNER,
    repo: process.env.REPO_NAME,
    pull_number: parseInt(process.env.PR_NUMBER),
    commit_id: process.env.HEAD_SHA,
    event: overallDecision,
    body: `## AI Code Review\n\nFound ${allComments.length} issue(s).\n\n*Powered by Claude*`,
    comments: allComments.map(c => ({
      path: c.path, line: c.line,
      body: `**${c.severity.toUpperCase()}**: ${c.body}`
    }))
  });
}

function extractJson(text) {
  const match = text.match(/\{[\s\S]*\}/);
  if (!match) throw new Error('No JSON in response');
  return match[0];
}

async function getInstallationId(app) {
  for await (const { installation } of app.eachInstallation.iterator()) {
    return installation.id;
  }
  throw new Error('No installation found');
}

function parseDiff(diffText) {
  const files = [];
  let currentFile = null;
  let currentLine = 0;

  for (const line of diffText.split('\n')) {
    if (line.startsWith('+++ b/')) {
      if (currentFile) files.push(currentFile);
      currentFile = { path: line.slice(6), diff: '', changedLines: [] };
      currentLine = 0;
    } else if (line.startsWith('@@ ')) {
      const match = line.match(/@@ -\d+(?:,\d+)? \+(\d+)/);
      if (match) currentLine = parseInt(match[1]) - 1;
    } else if (currentFile) {
      if (line.startsWith('+')) { currentLine++; currentFile.changedLines.push(currentLine); }
      else if (!line.startsWith('-')) currentLine++;
      currentFile.diff += line + '\n';
    }
  }

  if (currentFile) files.push(currentFile);
  return files;
}

main().catch(console.error);

Customizing Review Rules

Add team-specific rules to REVIEW_RULES:

const CUSTOM_RULES = `
Additional rules for this codebase:
- All database queries must use parameterized statements (no string interpolation with user input)
- API endpoints must validate request body with Zod schemas before processing
- React components must not call hooks conditionally
`;

Cost at Scale

With Claude Haiku:

Upgrade to Sonnet for security-critical repositories where false negatives are expensive.

Built by theluckystrike — More at zovo.one