A generic AI reviewer approved the PR. You merged it. Hours later, production broke — the bot missed a breaking contract change buried in the diff because it didn’t understand your codebase conventions. A linter couldn’t catch it either; the code was syntactically fine.
Off-the-shelf code assistants work on anyone’s code. Custom tools built with LLM APIs work on yours — diff-aware reviews that flag contract changes, autocomplete constrained to your latency budget, migration assistants that know your framework’s sharp edges. Same models. Radically different outcomes.
This guide builds three such tools — with model selection tailored to each task. Recommendations are informed by independent coding benchmarks, including the SWE-bench leaderboard.
Code Completion with Fill-in-the-Middle
Standard chat completion: “given this prompt, generate text.” FIM completion: “given the code BEFORE the cursor and the code AFTER the cursor, fill in what goes BETWEEN.”
The model sees: <PRE> function calculateTotal(items) { const subtotal = items.reduce((sum, i) => sum + i.price, 0); <SUF> return subtotal * TAX_RATE; } <MID>
The model generates: // Apply discount for premium customers — inserted exactly where the cursor sits.
DeepSeek V4, GPT-5.5, and Claude Sonnet 4 all support FIM. The latency requirement is brutal — <500ms from keystroke to suggestion, or the developer types past where the suggestion would have been useful.
async def fim_complete(prefix: str, suffix: str, model: str = "deepseek-v4-flash") -> str:
"""Fill-in-the-middle completion for code."""
response = await client.chat.completions.create(
model=model,
messages=[{
"role": "user",
"content": f"<PRE>{prefix}<SUF>{suffix}<MID>"
}],
max_tokens=50, # Keep completions short — latency is everything
temperature=0.2 # Low temp for consistent, correct code
)
return response.choices[0].message.contentModel selection: DeepSeek V4 Flash for autocomplete — cheapest per token, fastest TTFT, and coding performance within 5 points of frontier models on single-line completion. Only escalate to GPT-5.5 or Claude Sonnet 4 for multi-line completions that require understanding broader context. Verify current rates before committing — per-token pricing shifts frequently across providers.
AI Code Review: Diff-Aware Prompting
PR review isn’t “read this code and find bugs.” It’s “read this diff — the specific lines that changed — and determine whether the change introduces issues given the surrounding context.”
The prompt structure that works:
<git_diff>
@@ -45,7 +45,9 @@ def process_payment(amount, user_id):
if amount <= 0:
raise ValueError("Amount must be positive")
- user = db.query(User).filter_by(id=user_id).first()
+ user = await db.query(User).filter_by(id=user_id).first()
+ if not user:
+ return None
return payment_gateway.charge(user, amount)
</git_diff>
Review this diff. Check for:
1. Security vulnerabilities
2. Performance regressions
3. Breaking changes to the function's contract (it used to raise on missing user, now returns None)
4. Missing error handling
Classify each finding as CRITICAL, WARNING, or INFO.The model sees exactly what changed — not the entire file. This keeps context small (cheaper) and focuses the model’s attention on the change (more accurate).
Critical finding #1: breaking contract change. The function previously raised an exception for missing users. Now it returns None. Every caller that doesn’t check for None will break. The model should flag this as CRITICAL — it caught a semantic change that a linter would miss because the code is syntactically valid. For securing your API keys when integrating code review bots into CI pipelines, see the security best practices.
Code Migration Assistant
Framework migration — Django to FastAPI, JavaScript to TypeScript, REST to GraphQL — is tedious, mechanical, and error-prone at scale. It’s also a perfect LLM task: pattern matching with structured transformation rules.
async def migrate_file(source: str, from_framework: str, to_framework: str) -> str:
"""Migrate a single file between frameworks with human-in-the-loop review."""
response = await client.chat.completions.create(
model="claude-sonnet-4-20250514",
messages=[{
"role": "system",
"content": MIGRATION_SYSTEM_PROMPTS[(from_framework, to_framework)]
}, {
"role": "user",
"content": f"Migrate this file:\n\n```python\n{source}\n```\n\nOutput the migrated file with comments explaining each change."
}],
temperature=0.1
)
return response.choices[0].message.contentThe workflow: LLM migrates → human reviews → human accepts/rejects per-change → accepted changes are committed. The LLM does the mechanical transformation. The human verifies correctness. Neither does the other’s job.
Model selection for migration: Claude Sonnet 4 for the actual migration — strongest code understanding at its price tier. GPT-4o Mini for change explanation generation — cheap, fast, good enough for explaining what changed and why.
Model Selection for Coding Tasks
| Task | Best Model | Why |
|---|---|---|
| Autocomplete (single-line) | DeepSeek V4 Flash | Cheapest, fastest TTFT, sufficient quality |
| Autocomplete (multi-line) | GPT-5.5 / Claude Sonnet 4 | Context understanding matters more for longer completions |
| Code review (bug finding) | Claude Opus 4 | Frontier reasoning for finding subtle bugs |
| Code review (style/format) | GPT-4o Mini | Style checks don’t need frontier capability |
| Migration (mechanical) | Claude Sonnet 4 | Best code understanding per dollar |
| Migration (complex refactor) | Claude Opus 4 | Complex transformations need frontier reasoning |
The cost difference: running all tasks through Opus costs ~$15/M input tokens. Tiering requests by complexity — autocomplete on DeepSeek Flash at $0.14/M, review on Opus at $15/M — brings the blended rate to ~$3/M. Same quality where it matters. 80% lower cost.
When AI Dev Tools Go Wrong: Two Postmortems
AI developer tools accelerate the developer — in both directions. When they are right, they save hours. When they are confidently wrong, they create bugs that pass review because “the AI approved it.”
The Code Review Bot That Approved a Critical Bug
A mid-stage SaaS company deployed an AI code review bot on every PR. The bot checked for security vulnerabilities, performance regressions, and breaking contract changes. It ran automatically, posting inline comments, and the team’s policy was: “if the bot approves and CI is green, merge.” The bot had a 94% developer acceptance rate. Leadership cited it as a productivity win.
A junior developer submitted a PR that changed an authentication middleware function from synchronous to async — but forgot to await the verify_token() call. The function returned a coroutine object instead of a boolean. In Python, a coroutine object is truthy. Every authentication check silently passed. No unit test caught it because the test stubbed verify_token() to return True synchronously. The bot reviewed the diff, saw the async keyword added, recognized the pattern as “migrating to async auth flow,” and approved the PR with a comment: “Nice async migration, LGTM.”
The change deployed to production on a Friday. By Monday morning, the auth bypass had been exploited — 3,400 user accounts accessed without valid tokens. The root cause was not the developer’s mistake (developers make mistakes; that is why review processes exist). The root cause was a review process that delegated judgment to an LLM without verifying the LLM’s reasoning. The bot’s comment — “Nice async migration, LGTM” — looked authoritative. It used the right pattern name. It sounded like a senior engineer. And it was completely wrong.
The fix was procedural, not technical. AI code review suggestions now display with a mandatory disclaimer: “AI-generated suggestion. Verify before accepting.” The team added a human sign-off requirement for any PR touching authentication, authorization, or payment logic. The AI review bot is now one input to the review process — not the review process.
The Migration Tool That Introduced Subtle Syntax Errors
A team migrated a 12,000-line Django codebase to FastAPI using Claude Sonnet 4. The migration was “mostly successful” — 94% of files passed syntax validation on first pass, 97% after manual fixes. The remaining 3% were manually rewritten. The team celebrated. The migration was declared complete.
Three weeks later, production started returning 500 errors on a specific endpoint — one that had handled 200+ requests per day without issue since the migration. The root cause: the LLM had migrated a Django queryset filter chain to a SQLAlchemy query, and the ordering of .filter() and .join() calls was reversed. In Django, the order didn’t affect the result — the ORM optimized the query plan. In SQLAlchemy, the order of joins changes which rows are included. The query was returning wrong results, and a downstream validation check that had worked in Django was throwing an unhandled exception in FastAPI.
The code had passed syntax validation. It had passed existing test cases (which used mock data, not real database queries). It had passed human review (the reviewer saw .filter().join() and it looked right — same method names, same arguments). The behavior change was invisible until real data hit it at scale.
The lesson: test AI-migrated code against production-sampled data before declaring migration complete. Run the old and new systems in parallel with a shadow traffic comparator. Any semantic difference between their outputs is a migration bug — track every difference, not just the ones that cause 500 errors. The 54 files that “passed” but contained latent behavioral changes were worse than the 3% that failed loudly. For automating semantic regression tests as part of your deployment pipeline, see our CI/CD testing guide.
FAQ
What’s the latency budget for code completion?
Under 500ms from keystroke to suggestion. Beyond 500ms, the developer has already typed past where the suggestion would have been inserted. This is the hardest constraint in AI developer tools — and the reason autocomplete uses the fastest, cheapest models available. Every millisecond of model latency is a millisecond the developer is waiting. Choosing the right model for each coding task — speed for autocomplete, depth for review — makes the latency budget work.
How do I verify AI-generated code is correct?
Three layers: (1) Syntax validation — does it parse? (2) Test execution — do existing tests still pass? (3) Human review — does the logic make sense? Never auto-commit AI-generated code without human review. The AI is a tool for the developer, not a replacement for the developer.
FIM vs. chat completion for code — what’s the difference in practice?
FIM: the model knows what comes before AND after the cursor. It generates code that fits syntactically and logically between them. Chat completion: the model only knows what comes before. It’s guessing what should follow — without knowing what actually follows. For in-editor completion, FIM is always better. For “write a function that does X,” chat completion is fine — there’s no surrounding code to constrain the output.
How do I prevent AI code review from giving bad advice?
The AI suggests. The human decides. Never auto-apply AI review suggestions. Track two metrics: (1) acceptance rate — what percentage of AI suggestions do developers accept? (2) reversal rate — of accepted suggestions, how many are later reverted? High reversal rate means the AI is confidently wrong. Adjust the review prompt or downgrade the model.
When should I NOT use AI for code generation?
AI code generation fails reliably in three scenarios. First: code with security implications — authentication flows, cryptographic operations, input sanitization. The AI does not understand threat models. It generates code that looks correct but contains subtle vulnerabilities (timing attacks, incorrect nonce handling, SQL injection through concatenation). Second: code constrained by external specifications — regulatory compliance, financial calculations with rounding rules, medical device software. The AI does not know your local tax code, your jurisdiction’s data handling requirements, or your industry’s precision specifications. Third: code that must be provably correct — consensus algorithms, distributed locking, anything where “it works most of the time” is a failure mode. Use AI for exploration, boilerplate, and mechanical transformation. Write security-critical code, compliance-bound logic, and correctness-critical algorithms yourself — with AI as a reviewer, not an author.
AI developer tools work best when they handle the mechanical and surface the suspicious. Autocomplete for boilerplate. Code review for catching contract changes linters miss. Migration for mechanical framework translation. The human stays in the loop — reviewing, accepting, rejecting. The AI accelerates. The human verifies.
Autocomplete at $0.14 per million tokens, code review at frontier quality — tiered model routing makes both possible from the same codebase. Route your dev tools through TokSpan — every coding task hits the right model through one endpoint. $5 in credits free.