Field experimentT1T4T6CultureGovernanceInnerPeople
The Jagged Frontier: Better Inside, Confidently Wrong Outside
Navigating the Jagged Technological Frontier.pdf
Source
"Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality." Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, Karim R. Lakhani. Harvard Business School Working Paper 24-013, September 22, 2023 (SSRN 4573321). A pre-registered academic field experiment conducted inside Boston Consulting Group.
Evidence tier
field experiment. N=758 BCG individual-contributor strategy consultants, about 7% of that global cohort (p. 4); two pre-registered randomized experiments (Spring 2023, GPT-4), a no-AI baseline task then three arms (no AI / GPT-4 / GPT-4 plus prompt-engineering overview); human-graded outcomes (p. 9–13).
Thesis
AI capability forms a "jagged frontier": tasks of apparently similar difficulty sit inside or outside what the model can do, and the boundary is invisible from within the work. Inside the frontier, GPT-4 made elite consultants substantially better, faster, and more alike; just outside it, the same tool made them significantly more likely to be wrong — while sounding more persuasive. Value depends less on adoption than on how humans navigate the frontier.
Key findings
- Inside the frontier (18 realistic consulting tasks spanning creativity, analysis, writing, persuasion), AI users completed 12.2% more tasks, worked 25.1% faster, and produced over 40% higher quality than control (p. 4); arm-level quality gains of +42.5% and +38%, significant in all 108 pre-registered regressions (p. 14).
- Skill compression: bottom-half performers improved 43% against their own baseline vs 17% for the top half — everyone gained, the less skilled most (all elite in absolute terms) (p. 15).
- Outside the frontier (a business case built so GPT-4 would reach the wrong conclusion, p. 17): control was correct 84.5% of the time, AI groups 60% and 70% — a 19-percentage-point average drop, worst in the prompt-trained arm (−24 pp vs −13 pp) (p. 18).
- Persuasively wrong: on that task, AI users' recommendation quality scored 25.1% and 17.9% above control regardless of whether the answer was correct (p. 19) — and they finished up to 30% faster (p. 18).
- Copy-paste was the norm: modal average "retainment" of GPT text in submitted answers was ~0.87, near verbatim; training increased it, and higher retainment predicted higher grades inside the frontier (coefficient ~1.21) (p. 50–51).
- Homogenization: AI users' ideas showed markedly reduced variability, a simulated GPT-only condition being most homogeneous of all — better on average but "at risk of being more redundant" (p. 16, 53).
- Practice typology from 244 consultants' logs: "Centaurs" divide sub-tasks between human and AI by relative strength; "Cyborgs" integrate AI at sub-task level — validating, exposing contradictions, demanding explanations, pushing back (p. 20, 55–57).
Verbatim quotes
- "Within this growing frontier, AI can complement or even displace human work; outside of the frontier, AI output is inaccurate, less useful, and degrades human performance." (p. 5)
- "On tasks within the frontier, AI significantly improved human performance. Outside of it, humans relied too much on the AI and were more likely to make mistakes." (p. 19)
- "Professionals who had a negative performance when using AI tended to blindly adopt its output and interrogate it less" (p. 21)
- "In our study, since AI proved surprisingly capable, it was difficult to design a task in this experiment outside the AI’s frontier where humans with high human capital doing their job would consistently outperform AI." (p. 23)
Limits & contested points
One elite firm, one model: GPT-4 of April 2023, default settings (p. 12); the specific inside/outside placement of tasks is dated, and the paper says the frontier expands "often exponentially" (p. 23). The outside-frontier task was deliberately built to fool GPT-4 (p. 12, 17): the 19 pp penalty proves the mechanism, not the real-world frequency of such traps. Tasks were individual, time-boxed, simulated — no clients, teams, or consequences; "quality" is rubric-graded polish, and the wrong-but-higher-quality result shows graders reward exactly what AI supplies. Participants were volunteers with bonus-relevant incentives (p. 9–10). Centaur/Cyborg is an emergent qualitative typology, not yet causally linked to performance (p. 20, 55–56) — widely over-quoted as a validated playbook. The "40%" headline is human-graded quality on creative/writing tasks at one firm, routinely over-generalized to all knowledge work.
Second-order implications for leadership
- paper The unit of decision is not "adopt AI or not" but the workflow: evaluate "different configurations and combinations of humans and AI" task by task (p. 22) — an operating-model question sitting above tooling.
- paper Expertise formation is at risk: if inside-frontier work stops flowing to juniors, firms build "long-term training deficits"; frontier-navigation skill must be deliberately built (p. 22).
- paper Leaders must choose what their production function prizes — consistent high average output vs exploration — and may need multiple AI models or protected human-only work to keep idea diversity (p. 23).
- inference Wrong answers now arrive faster and better-argued; governance must stop letting speed and polish proxy for reliability, designing verification in where plausibility sufficed.
- inference The frontier moves faster than rules can; the durable control is developed judgment — people who interrogate rather than adopt (training made outside-frontier errors worse, p. 18).
HEI resonance & tensions
- People (inner drivers): failure outside the frontier was psychological — blind adoption, ceasing to interrogate (p. 21); holding one's own judgment against a confident machine is an inner-development capacity, not a tool skill.
- Governance: task-level human/AI configuration and verification norms (p. 22) are decisions about how decisions get made — governance design, not IT rollout.
- Culture: near-verbatim retainment (p. 50) and shrinking idea diversity (p. 53) show a default drift toward polished conformity; keeping variance alive is cultural work.
- Tension: the paper's remedy is granular — evaluate task by task — complicating "complementary things must change together" framing; and its training arm improved inside-frontier output while deepening outside-frontier failure (p. 14, 18), so enablement alone can worsen what matters most.
Use
- Landing-page proof point: "In a Harvard/BCG field experiment with 758 consultants, GPT-4 lifted work quality by more than 40% on tasks inside its capability range — and made consultants 19 percentage points more likely to get the answer wrong just outside it (Dell'Acqua et al., 2023)." Frame as strong field evidence from one firm and one 2023 model, not a universal law.
- CEO opener: "Your people's work is now split by an invisible line: on one side AI makes them roughly 40% better, on the other, faster, more persuasive — and wrong. Who owns knowing where that line runs as it moves?"
- Objection counter: against "this is a tech project — my CTO has it covered": tool and training were identical across both tasks; what flipped +40% quality into −19 points correctness was human judgment and work design — variables no deployment plan governs and only leadership can.