Local-First Agents

An AI agent that runs as a process on the user's own machine, rather than being hosted in a cloud service. The architectural difference determines the capability ceiling: a cloud agent can do what the user shares with it, while a local agent can do everything the machine can do.

Explanation

Cloud AI agents receive only the data a user explicitly pastes or uploads, and operate in a sandboxed context. A local agent has full system access: it can read any file on disk, including ones the user forgot existed, control connected hardware, use locally stored credentials, and observe ongoing machine state.

Peter Steinberger's framing: a local agent "can do every effing thing."¹ The implication is categorical rather than incremental. The difference between a cloud agent and a local agent is not speed or accuracy, it is the set of possible actions.

What local access enables. Discovery, in that the agent can surprise the user by surfacing data they forgot about, such as old recordings or files buried on disk and weaving them into a coherent narrative. Device control, since anything connected to the machine through standard interfaces becomes accessible: an oven, a Tesla, a Sonos system, a bed temperature controller.¹ Credential use, since existing API keys and login sessions are available without re-authentication or sharing. And continuous context, since the agent observes the machine over time, building an ambient model of the user's behavior without explicit logging.

Memory ownership as a corollary. Local-first architecture also resolves the memory silo problem. Cloud services lock memories inside their own databases, typically with no export path and no way for a competitor to access them. A local agent stores memory as files on the user's own machine, so the user owns their memories, which carries both privacy and portability implications.

Privacy implication. Agent memory is already more sensitive than search history, since people increasingly use their agents for personal problem-solving, relationship decisions, health concerns, financial choices, faster than they used to use search. A cloud provider holding this data represents a larger privacy surface than email or search history ever did.¹ Local-first means the data never leaves the machine.

Why it matters

The transition from cloud to local agents is not just a product architecture choice, it is the unlock that makes broad app disintermediation plausible. A cloud agent cannot replace a habit-tracking app because it lacks persistent, ambient access to what a user does and when.¹ A local agent already has that access by virtue of where it runs. It also simplifies the practical engineering problem: instead of configuring API integrations for every data source an agent needs, a local agent inherits access to all of them simply by running on the machine that has them.

Tensions and open questions

The security model at scale is not yet settled: full machine access means a mistaken or adversarially prompted local agent can do significant damage, and permissions and trust models for local agents remain immature.¹ Portability is also unresolved: if an agent runs on a laptop, what happens on a phone or tablet, since multi-device local-first is still an open architecture problem. Cloud and local are not strictly binary either, since hybrid models with local execution and cloud model inference blur the line; the capability unlock comes specifically from local execution, not from where inference happens. And while open-source local-first software is more auditable than a cloud black box in principle, most users will never actually audit the code, so the practical privacy guarantee is weaker than the theoretical one.