Quantifying AI Coding Impact: Metrics Beyond Lines of Code

Lines of code and suggestion acceptance rates tell you almost nothing about whether AI tools help your team. Measure downstream outcomes instead: PR cycle time, change failure rate, rework within two weeks of merge, review load, and time-to-first-commit for new hires. Pair those with periodic developer surveys, then compare cohorts rather than chasing a single vanity number.

Why the Obvious Metrics Mislead You

The metrics vendors surface by default are the ones easiest to instrument, not the ones that matter. Acceptance rate is the worst offender. Developers accept a completion, then delete or rewrite half of it in the next thirty seconds. The tool logs a win. Your codebase logs churn.

Lines of code generated is worse. AI assistants are extremely good at producing verbose code: exhaustive error handling nobody asked for, redundant test scaffolding, boilerplate that a well-designed abstraction would eliminate. If your dashboard rewards volume, you are paying (in review time and maintenance burden) for the exact behavior you should discourage. I have watched a team celebrate a 40 percent jump in "AI-authored lines" while their median PR review time climbed from four hours to more than a day, because reviewers were wading through generated tests that asserted nothing meaningful.

"Percentage of code written by AI" has a subtler problem: attribution is genuinely ambiguous. If Copilot suggests a line, you accept it, rename two variables, and restructure the conditional, who wrote it? Tools answer this question differently and none of the answers are stable across versions, so the number is not comparable over time or across tools.

What Should You Actually Measure?

Anchor on outcomes your business already cares about, then work backward. The DORA research program gives you a defensible starting frame: deployment frequency, lead time for changes, change failure rate, and time to restore. These are lagging indicators and they move slowly, but they are hard to game and they connect directly to delivery.

Below DORA, add a layer of engineering-specific signals that respond faster to tooling changes:

PR cycle time, decomposed. Total cycle time hides where AI helps or hurts. Split it into time-to-first-review, review duration, and time-from-approval-to-merge. In my experience AI assistance compresses authoring time noticeably but often lengthens review duration, because reviewers face more code per PR and less familiar code. If review duration grows faster than authoring time shrinks, your net throughput can go negative even while every individual developer feels faster.

Rework rate. Track the fraction of AI-assisted lines modified or deleted within 14 days of merge. This is the single best proxy I have found for "the suggestion looked right but wasn't." A rework rate under roughly 10 percent on hot paths is healthy; above 25 percent, the tool is generating plausible-looking drafts that humans are quietly repairing.

Change failure rate and revert frequency, segmented by AI involvement. If your commit tooling can tag AI-assisted changes (a trailer like Co-authored-by: assistant or a custom commit annotation works), you can compare failure rates between cohorts. The comparison is noisy and confounded (developers reach for AI on unfamiliar code, which fails more often regardless), so treat it as a smoke alarm, not a verdict.

Review load per reviewer. Count lines under review per reviewer per week. AI shifts effort from writing to reviewing, and review capacity becomes the bottleneck long before anyone notices. When this number climbs 50 percent without headcount changes, quality erosion follows within a quarter.

Onboarding velocity. Time from first day to first merged nontrivial PR. This is where I have seen the clearest, least ambiguous AI wins: assistants that can answer "how does auth work in this repo" cut ramp time meaningfully, and the metric is nearly impossible to game.

How Do You Instrument This Without Building a Surveillance System?

This question matters more than the technical one. Metrics that identify individual developers get gamed immediately and poison trust. Aggregate at the team level, publish the methodology openly, and never route these numbers into performance reviews. The SPACE framework paper makes the case well: productivity is multidimensional, and any single metric used as a target stops measuring what you think it measures.

Practically, most of the data already exists in your Git host and CI system. GitHub exposes org-level Copilot telemetry through the Copilot metrics API, which reports daily aggregates for completions, chat, and per-language activity. It is aggregate-only by design, which is the right default. A minimal pull looks like this:

1gh api \
2 -H "Accept: application/vnd.github+json" \
3 /orgs/YOUR_ORG/copilot/metrics \
4 --jq '.[] | {date, active: .copilot_ide_code_completions.total_engaged_users}'

Note the version detail: this API replaced the older Copilot usage API, which GitHub deprecated. If you built dashboards against the usage endpoints, migrate; the schemas are not compatible.

For rework rate, a scheduled job against your Git history is enough. The core query is "of the lines added in commit X, how many survive N days later," which git blame answers if you tag AI-assisted commits at authoring time:

1-- warehouse-side, assuming commit metadata is ingested
2SELECT
3 team,
4 SUM(CASE WHEN modified_within_14d THEN lines_added ELSE 0 END)::float
5 / NULLIF(SUM(lines_added), 0) AS rework_rate
6FROM commit_line_survival
7WHERE ai_assisted = true
8GROUP BY team;

Expect the instrumentation to take one engineer two to three weeks for a first version, mostly spent on attribution edge cases (squash merges, rebases, and cherry-picks all mangle naive line tracking).

The Survey Layer You Cannot Skip

Telemetry misses the largest effect AI tools have: cognitive load. A developer who ships the same PR count but ends the week less exhausted is a real productivity gain that no Git metric captures, and it shows up later as retention and sustained quality.

Run a short quarterly survey, five questions maximum, with at least these two:

• "In the past month, how often did AI suggestions save you meaningful time?" (never / weekly / daily / multiple times daily)
• "How often did you spend more time fixing or verifying an AI suggestion than writing it yourself would have taken?"

The gap between those two answers is your real signal. Teams with high answer-one and high answer-two are in the danger zone: they feel productive but are shipping unverified code. GitHub's own research on Copilot leans on exactly this mixed-methods approach, combining telemetry with self-reported outcomes, and their published methodology on the GitHub blog is worth reading before you design your own study, if only to see how carefully they scope their claims.

Cost Accounting: The Denominator Everyone Forgets

Impact is a ratio, and the denominator moves. Per-seat plans (Copilot Business at 19 dollars per user per month, Cursor's team tiers at a comparable price point) are trivial next to salaries, so seat cost alone almost never decides anything. The real costs are elsewhere:

Agentic workflows burn tokens unpredictably. An autonomous agent iterating on a failing test suite can consume millions of tokens in a single session. On frontier models, a heavy agent user can generate hundreds of dollars in monthly API spend, an order of magnitude above a completion-only user. If you are measuring impact, measure spend per team alongside it, or your ROI number is fiction.

Latency shapes usage more than quality does. Inline completions need to land in under about 400 milliseconds or developers type through them and the tool effectively does not exist. Chat and agent responses tolerate seconds. This means your metrics will show completion features dominating usage on fast connections and cratering on VPN-heavy or remote setups, and that is a network problem, not an adoption problem. Check latency distributions before concluding a team "doesn't like the tool."

Review capacity is the hidden cost. Every AI-authored line still needs a human reader. Budget for it explicitly.

When Measuring Is the Wrong Move

Do not build this program if your team is under ten engineers. The statistical noise in cohort comparisons at that scale swamps any real effect, and the instrumentation effort exceeds the value of the answer. Just talk to your developers.

Do not measure during the first 60 days of a tool rollout. Early numbers reflect novelty and learning curves in both directions, and decisions made on them are usually wrong. I have seen a team kill a tool at week three based on rework rates that would have halved by week ten as prompting habits matured.

And do not measure at all if leadership intends to use the numbers for stack ranking. You will get beautiful dashboards and worthless data within one quarter, because every metric here is trivially gameable by an individual who knows it is watched.

A Rollout That Actually Works

Here is the sequence I use when standing this up for a platform or DX team:

Weeks 1 to 2: Baseline everything before changing anything. Pull six months of PR cycle time, review duration, revert rate, and deployment frequency from your existing Git host. No new tooling yet.

Weeks 3 to 4: Add commit-level AI attribution (trailer convention plus a pre-commit hook or IDE extension setting), and stand up the rework-rate job. Accept that attribution will be roughly 80 percent accurate; that is enough for cohort comparison.

Week 5: Ship the first survey. Keep it anonymous, publish the aggregate results to the whole org, and commit to a cadence.

Quarterly: Review team-level dashboards in a forum where engineers can contest the interpretation. The contestation is the point; a metric nobody argues about is a metric nobody trusts.

Ongoing: Retire any metric that has been flat for three consecutive quarters. A stable number has told you what it knows. The goal is not a permanent scoreboard; it is a feedback loop that answers a specific question and then makes room for the next one.

If you take one thing from this: pick two outcome metrics and one survey question before you look at a single vendor dashboard. The dashboards answer the vendor's question, which is "is this tool being used." Yours is "is our software getting better, faster, and cheaper to change." Those are different questions, and only one of them justifies the invoice.