The Pragmatic Engineer — selected conversations
Measuring the impact of AI on software engineering – with Laura Tacho
Eighteen curated conversations on agent workflows, design, harnesses, and software quality
A manually reviewed selection of Pragmatic Engineer conversations with Maggie Appleton, Matt Pocock, Tibo Sottiaux, Martin Fowler, Dex Horthy, Boris Cherny, Mario Zechner and Armin Ronacher, Laura Tacho, Hillel Wayne, Mitchell Hashimoto, Kent Beck, Peter Steinberger, Grady Booch, Steve Yegge, DHH, Dax Raad, Charity Majors, and Addy Osmani. Topics include design prototyping, reusable skills, agent workflows, context and harness design, TDD and verification, architecture, measurement, accountability, and open-source governance. Practitioner observations, product-team claims, and forecasts are not controlled findings.
In this collection Browse 18 summaries 8 of 18
Gergely Orosz, host of The Pragmatic Engineer podcast, interviews Laura Tacho, CTO at developer-productivity company DX, about measuring AI-assisted software engineering without reducing impact to generated code or accepted suggestions 00:00–00:50(opens in a new tab), 08:54–16:56(opens in a new tab). Tacho's central argument is that organizations should establish a baseline, evaluate AI across utilization, impact, and cost, and run controlled experiments that include developer experience and delivery outcomes—not assume that faster code generation creates proportional business value 11:03–14:10(opens in a new tab), 1:07:20–1:08:18(opens in a new tab).
Key Points Covered
- Measure utilization, impact, and cost together: Tacho treats acceptance rate as a limited fit-for-purpose signal rather than proof of production use, velocity, innovation, or business impact 12:05–16:56(opens in a new tab).
- The most useful cases may not be code generation: In a DX study covering more than 180 companies, developers reporting substantial time savings ranked stack-trace analysis and refactoring above mid-loop code generation. Tacho argues for training around these less obvious uses 23:32–25:33(opens in a new tab).
- Generation can transfer effort into review: Tacho contrasts generated code, which still needs assessment and integration, with stack-trace analysis that can remove diagnostic toil; Orosz emphasizes the expert judgment required to evaluate output 26:36–28:43(opens in a new tab).
- Time savings can reduce job satisfaction: Tacho cites DORA research in which some developers reported that AI accelerated coding they enjoyed while leaving meetings, administration, and other toil. She proposes combining experience data with workflow and system metrics 29:46–35:52(opens in a new tab).
- Adoption needs enablement and segmentation: In the Booking.com case study, Tacho reports that training and office hours accompanied 65% weekly-or-daily adoption, while non-use could reflect license access or poor fit for specialized work rather than resistance 17:47–23:32(opens in a new tab).
- Customer gains do not establish causation: For Workhuman, Tacho reports an 11% increase in DX's developer-experience index and 15% higher velocity among frequent AI users than non-users. These are segmented vendor-reported findings, not evidence that AI causes the same gains elsewhere 42:30–46:33(opens in a new tab).
- Larger diffs may trade speed for stability: Tacho says DX data associates AI use with more-complex diffs, while the DORA study she presents forecasts reduced stability as adoption rises. Her larger-batch explanation is a hypothesis, so speed should remain paired with quality and reliability measures 51:42–54:49(opens in a new tab).
- Treat rollout choices as experiments: Tacho describes Indeed comparing tools across cohorts and use cases, then testing interventions such as preliminary AI review or assisted migrations. These are reported company experiments, not universal outcomes 58:58–1:05:12(opens in a new tab).
Full video: https://www.youtube.com/watch?v=xHHlhoRC8W4(opens in a new tab)