Adoption and Productivity

Capability is potential. Productivity is a measured change in accepted work. A model can draft faster while making a reviewer slower, raise one person's output while congesting a shared queue, or improve a benchmark without changing anything an organization ships. Adoption therefore needs a counterfactual: what would the same eligible work have cost and produced under the policy it replaces?

Chapter 77 asked whether a buyer has credible alternatives. This chapter asks whether using one of them changes work enough to matter. A defensible claim names the accepted unit of work, task population, worker population, workflow, tool version, review policy, and time window. Change any one of those fields and the old productivity estimate may no longer apply.

Freeze the unit before measuring the tool

Start with a completed workflow unit, not a job title or a model call. A support unit might be a ticket accepted as resolved after a fixed return window. A software unit might be a merged change that passes review, tests, and a delayed defect window. A policy unit might be an adjudicated memo whose claims satisfy a locked rubric. Attempts that are abandoned, escalated, or rejected stay in the denominator.

This task-level boundary matters. Occupational exposure studies estimate where language models could reduce task time at unchanged quality; they do not measure adoption, achieved productivity, job displacement, or return on investment (Eloundou et al. 2024). Jobs combine many tasks with coordination, accountability, and work that remains outside the tool. Aggregate only after the workflow units are measured.

Record the evaluation contract before the pilot begins:

Productivity is not one proxy (Organisation for Economic Co-operation and Development 2001). Keep a decision vector containing accepted throughput, active labor time, end-to-end cycle time, first-pass quality, review time, rework, escalation, severity-weighted defects, incidents, customer outcomes, and worker experience. For assignment arm $z$, three useful summaries are

$$P_z = \frac{A_z}{H_z}, \qquad F_z = \frac{A_z^{(1)}}{N_z}, \qquad D_z = \frac{\sum_{s=1}^{S} w_s n_{zs}}{N_z}.$$

Here, $z=0$ denotes the baseline arm and $z=1$ the assisted arm. $P_z$ is accepted throughput; $A_z$ is the number of accepted units; and $H_z$ is total paid labor hours, including generation, checking, correction, escalation, and coordination. $F_z$ is first-pass yield, where $A_z^{(1)}$ counts units accepted without correction and $N_z$ counts every assigned eligible unit. $D_z$ is the severity-weighted defect rate; $s$ indexes one of $S$ predeclared severity classes, $w_s$ is its fixed weight, and $n_{zs}$ is the number of class-$s$ defects in arm $z$. Report raw defect counts beside $D_z$ so a chosen weight cannot hide harm.

Prompt count, tokens generated, logins, and satisfaction can explain a mechanism. They are not productivity. Neither is faster completion if acceptance quality, rework, or delayed defects deteriorate.

Estimate the effect of access, not enthusiasm

When feasible, randomly assign access within prespecified strata such as task family, baseline output, and worker experience. Here, $i$ indexes the unit of randomization, $Z_i=1$ indicate assignment to access, $Z_i=0$ indicate assignment to the baseline, and $Y_i$ be the locked primary outcome for that unit. With $n_1$ assisted assignments and $n_0$ baseline assignments, the sample intent-to-treat effect is

$$\widehat{\tau}_{\mathrm{ITT}} = \frac{1}{n_1}\sum_{i:Z_i=1}Y_i - \frac{1}{n_0}\sum_{i:Z_i=0}Y_i.$$

The intent-to-treat estimate measures the policy effect of offering this tool under these conditions. It retains people who were assigned access but did not use it. Actual use is a post-assignment choice. Comparing voluntary users with nonusers confounds the tool with motivation, task selection, and expected benefit; calling that difference causal creates selection bias. Noncompliance can also be reported through take-up and, with the required instrumental-variable assumptions, a complier effect, but that is not the average effect for every user (Angrist et al. 1996).

The unit of randomization must match the interference boundary. If colleagues share a queue, templates, reviewers, or meeting load, individual treatment can spill over into the control arm. Cluster-randomize the team or workflow when those interactions are material, and analyze at that level. A network or saturation design is needed when direct and spillover effects are separate decisions of interest (Hudgens and Halloran 2008).

Predeclare a small number of useful strata, such as task family, experience, error severity, and workflow maturity. Report effect sizes, uncertainty intervals, and subgroup counts. “Significant here but not there” is not proof that two subgroup effects differ. Keep the original random assignment through onboarding and learning periods, and version every change to the model, interface, prompt library, staffing, and task mix.

Put value and cost in the same ledger

Operational metrics should remain separate until the organization has a defensible conversion rule. When a financial decision is required, compare both arms over one accounting horizon $H$:

$$NB_H = (R_1-R_0) + (C_0-C_1) - C_{\mathrm{fixed}} - \mathbb{E}[L_1-L_0].$$

$NB_H$ is incremental net benefit over horizon $H$. $R_z$ is realized revenue or contribution value in arm $z$; $C_z$ is observed variable operating cost, including labor, model, review, rework, escalation, and coordination; and $C_{\mathrm{fixed}}$ is incremental integration, security, legal, procurement, and training cost. $L_z$ is loss from errors and incidents, so $\mathbb{E}[L_1-L_0]$ is the change in expected loss. Every term uses the same currency and accounting horizon. Arm 0 is the baseline arm and arm 1 the assisted arm.

Avoid two common accounting errors. First, an hour saved is capacity, not cash, unless the organization can redeploy it, avoid hiring, increase accepted output, or reduce paid hours. State that conversion rule. Second, do not count a quality improvement once as revenue and again as avoided rework. If quality or a rare catastrophic loss cannot be credibly priced, keep it as a separate guardrail and stress scenario.

For a steady-state unit-cost decision, $C_{\mathrm{fixed}}/N_H$ is the amortized fixed cost per accepted unit, where $N_H$ is planned accepted volume over $H$. Keep observed pilot cost separate from forecast volume. Report an uncertainty interval for $NB_H$ and sensitivity to labor value, volume, incident probability, and the defect window rather than presenting one precise return.

The explorer below uses illustrative normalized value units, not percentages, prices, evidence, or a forecast. Its purpose is to expose which assumptions can reverse the decision. A real pilot must populate the ledger from the same accounting period in both arms.

Why a useful tool can look unproductive at first

The gap between capability and measured output has a history. The productivity J-curve describes how a general-purpose technology can require complementary investment before its benefits appear in measured output. Organizations first build intangible capital: new processes, data, training, management routines, and software. During that period, the investment is costly while its future services are poorly measured (Brynjolfsson et al. 2021).

Generative AI does not remove that mechanism. Business-process redesign may change who drafts, who reviews, which queue owns an exception, and how accepted work reaches a customer. A separate chat window can improve individual drafting without changing a shared approval path. The resulting measurement lag is not a reason to assume that benefits will arrive later; it is a reason to measure complementary investment, adoption time, and the steady-state workflow instead of extrapolating from a demonstration.

Read every empirical result with its boundary

Early studies answer different questions and do not combine into one universal effect size.

Noy and Zhang randomly assigned 453 college-educated professionals to complete incentivized, midlevel writing tasks with or without ChatGPT. Access reduced completion time by 40 percent and raised output quality by 18 percent on those short tasks (Noy and Zhang 2023). The experiment establishes a bounded task effect, not a change in team output, employment, or firm profit.

Dell'Acqua and colleagues ran a preregistered experiment with 758 consultants at one firm. The study used two task bundles: 18 product, analysis, and writing tasks selected to be inside GPT-4's June 2023 capability boundary, and a business case containing one deliberately selected task outside it. In the inside bundle, participants with AI completed 12.2 percent more tasks and worked 25.1 percent faster, with higher rated quality. On the selected outside task, the combined AI groups were 19 percentage points less likely to be correct (Dell'Acqua et al. 2026). This contrast motivates a jagged frontier, but it does not define a permanent frontier for other models, workers, or workflows. The legacy synthetic curve has therefore been removed.

Brynjolfsson, Li, and Raymond observed the staggered managerial rollout of a real-time assistant across 5,172 customer-support agents, three million chats, and a single firm with contractors. Their main difference-in-differences estimate was a 15 percent increase in resolved issues per hour, with larger gains for less experienced and lower-skill agents. The highest-skill group saw little throughput gain and small quality declines on some measures (Brynjolfsson et al. 2025). This is strong field evidence for that assistant and setting, not a large randomized estimate across occupations.

Dillon and colleagues randomized integrated tool access among 7,137 workers at 66 opt-in firms. During months four through six, assignment reduced email-session time by 1.4 fewer hours per week; the instrumental-variable estimate for compliers was 2.0 fewer hours. The study detected no significant average change in meeting time, Word time, completed documents, or several coordination measures (Dillon et al. 2026). Telemetry showed a time-allocation effect, not output quality or organizational return. It also illustrates why individual drafting speed need not move a coordinated routine.

Becker and colleagues randomized whether 16 developers could use early-2025 tools on 246 tasks drawn from issues in mature open-source projects they knew well. AI allowance increased completion time by 19 percent, with a 95 percent interval from 2 to 39 percent slower, even though participants believed the tools had helped (Becker et al. 2025). The result concerns short brownfield issues, codebase experts, and February-to-June 2025 tools. It does not estimate current tools, greenfield development, long-run code quality, or team throughput. METR later changed its follow-up design because growing AI dependence made a representative no-AI control group difficult to recruit, a selection problem rather than a clean reversal of the first estimate (METR 2026).

Broader outcomes can move less than task behavior. A 2026 revision linking surveys in Denmark with administrative records found rapid chatbot use and task reorganization, including new oversight and integration work, but no detectable average effect on earnings or recorded hours. Its estimates ruled out effects larger than 2 percent at worker and workplace levels two years after ChatGPT's launch (Humlum and Vestergaard 2025). That result does not deny local time savings. It shows that saved task time, changed work, hours, earnings, and aggregate productivity are different outcomes.

Separate adoption, usage, capability, and realized value

Usage logs locate requests. They do not show whether an output was correct, accepted, or useful. Anthropic's initial Economic Index mapped more than four million Claude conversations in total; its central occupational analysis used self-selected Free and Pro conversations and excluded enterprise and API traffic. Software-development and writing requests dominated that product sample, but the study did not observe the user's job, final output, or productivity (Handa et al. 2025).

GDPval asks a different question. It gives a model an isolated deliverable, supplied context, and a one-shot task from one of 44 knowledge-work occupations. Expert graders found the strongest 2025 models approaching expert deliverable quality on the public 220-task gold set (Patwardhan et al. 2025). The benchmark excludes iterative client feedback, organizational context, integration, and most human oversight. Neither establishes causal productivity or an organizational return.

Adoption surveys supply another layer. A nationally representative U.S. Census Bureau supplement covering November 2025 through January 2026 found that 18 percent of U.S. firms reported AI use in a business function. Among adopting firms, 57 percent used it in no more than three functions (Bonney et al. 2026). Those self-reports measure adoption and breadth, not causal output gains; the paper's performance relationships are correlations.

Run an adoption pilot as a decision process

A pilot should finish with evidence for a bounded policy, not a sentiment score:

1. Freeze the unit. Declare eligible work, the acceptance rule, defect window, task strata, worker population, and current baseline.
2. Register the design. Lock the unit of randomization, primary outcome, minimum worthwhile effect, sample and duration, subgroup plan, missing-data rule, interim looks, harm boundaries, and rollback trigger.
3. Instrument both arms. Capture assignment, actual use, active labor, review, rework, escalation, coordination, model cost, failures, and versioned workflow events for every eligible unit.
4. Apply the acceptance rule. Use a condition-blind review where feasible, preserve failures in the denominator, and wait through the delayed-defect window.
5. Estimate effects. Report arm totals, take-up, intent-to-treat effects, uncertainty intervals, predeclared heterogeneity, spillovers, and learning by event time. Chapter 48 supplies the interval and stopping discipline.
6. Price the ledger. Convert only defensible benefits and costs, state the capacity-to-cash assumption, retain unpriced quality and safety guardrails, and stress rare losses.
7. Make a bounded decision. Go only when the lower uncertainty bound clears the hurdle and every guardrail passes; iterate when uptake or workflow friction prevents a conclusion; stop and rollback at a predeclared harm boundary.

The evidence record should retain the protocol and its amendments, assignment table, task and worker strata, treatment versions, raw arm totals, missingness, acceptance decisions, defect observations, cost rules, estimates and intervals, guardrail outcomes, and final decision. NIST's generative-AI risk profile likewise connects predeployment testing to documented go/no-go decisions, incident response, and postdeployment monitoring (National Institute of Standards and Technology 2024).

Capability is potential. Productivity is a measured change in accepted work.

Further reading