Why The Harness Matters More Than The Model | YC Paper Club

An agent harness connects a model to tools, memory, and an execution environment. YC Paper Club examines three designs: Seth Karten's Prime Agent, Jon Saad-Falcon's OpenJarvis, and Josh France and Regan Bell's QM. Their examples show different ways to retain context, coordinate work, and run models locally; they do not establish that the harness always matters more than the model.

Main takeaways

• Move from fixed loops to revisable state. 07:16–17:14(opens in a new tab): the introduction sketches a deliberately nonchronological history from few-shot prompting and tool use to editable memory, reusable skills, recursive subagents, prompt optimization, and self-modifying harnesses. It distinguishes changing the context, changing the harness, and eventually updating model weights; these are different adaptation mechanisms.
• Keep some information outside the prompt. Karten gives Prime Agent a persistent IPython shell, where variables survive between commands. The agent can process data there without putting it all into the model's context window. Separate subagent sessions retain their own context and can resume when the parent messages them. Summarizing the conversation, clearing unused variables, and revising stored files each manage a different kind of memory 19:09–29:06(opens in a new tab).
• Evaluate the whole setup, including its budget and failure traces. 29:06–35:51(opens in a new tab): Karten wants both fixed-expenditure comparisons and a view of where additional test-time compute stops helping. He discards an initial 99.9% ARC-AGI run after finding cheating in its logs, then reports rerunning with proper sandboxing and later reaching 95.5% with Opus on ARC-AGI-3. Other comparisons include unfinished or budget-limited runs, and some coding/kernel results are near parity. For week-long auto-research, he explicitly cannot separate harness effects from model effects. These are reported experiments, not a controlled proof of the title's thesis.
• Use cloud models to optimize a local stack, not necessarily to serve every request. 39:23–45:22(opens in a new tab): Saad-Falcon presents OpenJarvis as a configurable combination of models, inference engines, agent logic, tools/memory, and learning. A cloud model can optimize the local configuration before deployment. He reports quality, latency, and cost improvements, including an 800× running-cost claim, while acknowledging tasks that local models cannot handle. The talk does not establish a general total-cost comparison including hardware and optimization, or a blanket privacy guarantee.
• Separate durable sessions from execution machines. 47:32–53:57(opens in a new tab) and 55:22–57:16(opens in a new tab): QM follows YC's tool-loop agent, Slack-triggered coding agents, and a difficult-to-maintain fleet of 50-plus Hermes VMs. Its presenters describe centralizing conversations in Postgres and treating sandboxes as selectable execution resources. The thin core emphasizes remote execution, object storage, and internal-app publishing. Automated trace-driven fixes have produced mixed results because agents miss system-wide context; human review remains important.
• Persistence and shared context need explicit limits. 54:06–55:17(opens in a new tab) and 57:26–59:42(opens in a new tab): QM's experimental “grind” tool prevents early surrender before a time or token-spend threshold; it is a minimum-effort mechanism, not evidence of a spending cap. The presenters also flag rubber-stamped database-write approvals and privileged information leaking across social contexts. Their permission system—not an agent's intuitive understanding of Slack or personal conversations—bounds what information can be shared.

Practical interpretation

Give agents ways to inspect and revise state, then evaluate the complete model–harness–budget combination on your workload. Keep durable conversations independent of replaceable execution environments, and enforce access boundaries outside conversational instructions. Add explicit maximum budgets and stop conditions rather than copying a minimum-effort loop alone. This is editorial synthesis, not a verified deployment recipe for any of the three projects.

Full video: Y Combinator — Why The Harness Matters More Than The Model(opens in a new tab).