AI could be an incredibly positive force for humanity, compressing decades of research into months, putting services once unattainable to most within everyone’s reach, and so much more . It could also be used as the ultimate information control technology, increasing surveillance, facilitating power concentration, and foreclosing the pluralism that lets societies thrive.

Much of this information control is exercised through intermediaries: the individuals, institutions and systems that sit between people and what they want to do. This post maps where new intermediaries are emerging in the age of AI, why their power could be systemically dangerous even in careful hands, and how, by keeping that power checkable and contestable, we might reap AI’s benefits without surrendering pluralism, privacy or safety.

This post serves as a companion piece to the Scaling Trust programme thesis , to motivate and explain what we are funding and why we care about it.

Information control

Information control includes both surveillance (observation, tracking, collection) and influence (manipulation, shaping).¹

Intermediaries are a main point of information control. They often provide a valuable service, and exist due to economic, legal or safety realities of the environments they exist in. For example: payment networks leverage economies of scale to give better prices; notaries and registries certify who owns what; and brokers and pharmacists screen what reaches people before it can hurt them.

This post is particularly concerned with intermediaries whose operations or integrity cannot be easily verified by users in real time. Instead, users must rely on reputation, credentials, or legal recourse to address any misconduct ex-post.

Artificial intelligence stands to make intermediaries more powerful due to its information dynamics: (1) AI is more useful the more context it has, creating strong incentives for people to share increasingly sensitive information; (2) AI has unprecedented capabilities to interpret that information and act on it. This is accelerated by market forces: competitive pressures simultaneously push companies to collect more data in order to stay relevant as a business (e.g. to facilitate automation), and push users to share more with their AI tools to avoid falling behind those who do.

Powerful intermediaries

Intermediaries have repeatedly abused positions of trust for their own benefit. For example, regulators have found mobile carriers selling access to customers’ location data,² and banks manipulating the benchmark rate (LIBOR) they were trusted to report.³ However, beyond abuses of power, the trend of increasing information control within intermediaries causes systemic risks:

Power concentration faster than it can be checked. Information is power.⁴ Controlling information used to be slower, and harder; AI speeds it up. One might say this is fine as long as the ‘good guys’ are in power, using their control to protect us, and with their actions checked by the people.⁵ But power changes hands: the tools built by and for the benevolent intermediary could soon be in the hands of reckless ones. Our constitutional mechanics to keep them in check may not be robust or rapid enough to keep pace.

Monoculture. Surveillance and influence both have a chilling effect on society. People who know they’re being watched behave differently. People whose information comes from the same few sources slowly come to think the same way. A society under permanent visibility, or whose preferences are shaped by a few parties, slowly stops producing dissidents.⁶⁷ This might seem fine at first, but it is often through dissidents that we discover new things. Dissent is one of society’s error-correction mechanisms. Many ideas we now hold as obvious – that the Earth revolves around the Sun, that women should vote – each began as a deviant one.

Path of least resistance. Control is often convenient for whoever holds it, and centralised data collection is technically easier than decentralised alternatives. History has plenty of examples of the ‘easier’ route being taken, from the backdoored Clipper chip of the 90s crypto wars to the blanket data-retention laws of the 2000s.⁸ Surveillance and influence tend to arrive by default and short-term pragmatism, rather than by design.

Together, these dynamics risk eroding the values behind liberal democratic constitutions: individual liberty, pluralism, credible constraints on power.⁹

Information must flow: five layers

One way to zero in on the problem is to examine how information flows within the AI stack, and where new intermediaries emerge. Consider these five layers:

• Applications & agents – the interfaces, harnesses, and agents through which users share prompts, files, preferences and actions.
• Inference & serving – the infrastructure that serves models and processes queries, responses and associated metadata.
• Models – the weights in which patterns learned from training data are encoded.
• Training – the processes that select and transform web data, licensed material, synthetic data, and user interactions into model capabilities.
• Hardware – the chips and datacentres on which the rest of the stack depends.

Application and agent providers sit between users and their digital lives: they can observe our intentions, files and actions, and influence which options are presented or pursued. Model and inference providers sit between those applications and intelligence: they can retain interactions, determine how models behave, and decide which capabilities are available to whom. Cloud and hardware providers sit further upstream, between model developers and compute. They may see little user information directly, but they can determine who is able to build or operate powerful systems, and on what terms. These layers allow for different forms of control – surveillance, influence and gatekeeping.

Today these seeing and decision-making roles increasingly sit within connected corporate ecosystems, creating chains of intermediaries whose power compounds across the stack. The Mythos export control episode¹⁰ showed how such a control point can be exercised: a government directive to one model provider caused access to a general-purpose capability to disappear worldwide. These intermediaries already shape not only what people see, but what they can do. As agents are entrusted with more of our economic and social activity, that control will extend further into the world.

Two responses: verifiability and plurality

At each layer, there are two complementary responses. The first is to make power more transparent and verifiable, so that institutions and citizens can check it. The second is to create technologically and economically credible alternatives; this response enables plurality across the stack, and enables people to exit when those checks fail. One constrains power; the other distributes it.

The good news is that alternatives are already being built, often for practical business reasons as much as ideological ones. In July, Nvidia and a consortium of others signed Open Weights and American AI Leadership , making the case that open weights reduce costs, prevent lock-in, strengthen cybersecurity and let organisations control their own data and infrastructure. Thinking Machines has made a related case for AI that can be shaped by the people it serves, rather than having its values determined in a handful of places, and followed it by releasing Inkling with open weights. The motivations differ (competition, scientific transparency, sovereignty, privacy, customisation) but they point in the same direction: making AI more inspectable and giving people viable alternatives to centralised providers.¹¹

“a single locus of value alignment, however well run, becomes a locus of power to be captured”

– Thinking Machines’ manifesto

Across the stack, both responses are already taking shape. Figure 5 maps some of the projects doing the work: at every layer, at least one way to check the incumbents and at least one credible alternative.

The sixth layer

As AI agents begin to be deployed in the wild and start communicating with one another, a sixth layer is emerging: the coordination layer. The infrastructure that enables agents to interact with one another, including communication channels, protocols, negotiation practices, sensors, identity systems. Without it, an agent is essentially confined to single-player mode.

This layer could enable productive networks of agents and unlock tremendous value for humanity. But it could also negate efforts to distribute power elsewhere in the stack. Even an open model running on personal hardware does not protect us if every agentic interaction must clear through a central platform.

Consider the following example: your agent is negotiating a job offer with a company’s agent. You don’t want to reveal your minimum salary expectations, but they don’t want to reveal their ceiling. The easy architecture would be that both agents are hosted by the same provider which, as the trusted information escrow, matches them and lets each agent query the other’s context without “seeing” it. It’s convenient, it’s doable today and it’s easily policed in case of agent misbehavior.

If this becomes a default architecture, it could also mean that a large share of negotiation in society – salaries, rents, settlements, acquisitions – routes through a handful of intermediaries with a complete view of both sides.

Scaling Trust

Trusted information escrows have historically helped us navigate hard tradeoffs: security against utility, convenience against control, oversight against privacy. What’s exciting is that over the last few years, this tradeoff space itself has begun to move. ¹² Emerging technologies can relocate trust away from an intermediary with unrestricted access to everyone’s information and into cryptography, hardware, and other verifiable, scalable roots of trust. Increasingly that includes the physical world too – sensors that can prove what they measured, chips that can prove what they ran – because agents are heading there as well.

Now our two negotiating agents have another option: run a two-party secure computation that answers “do our preferences match?” and nothing else, or meet inside a trusted hardware enclave, and work it out while each side’s limit stays its own. The previous trusted intermediary gets replaced by mathematics and silicon.¹³ The new tools dissolve the old tradeoff. You can have privacy and utility,¹⁴ security and efficiency.

Components of this technology already support large-scale applications,¹⁵ but they are not yet flexible, efficient, or usable enough for open-ended agentic coordination.¹⁶ AI could accelerate their development both by hiding complexity from users (e.g. enabling agents to generate bespoke security protocols on-demand) and by speeding up the research process (e.g. running research loops on cryptography research problems). This is a core pillar of Scaling Trust , and why we’re excited to fund these technologies and the underlying research behind them. We want to scale trust, without scaling trusted intermediaries.

What about safety?

Readers who have come this far may empathise with the problems stated and still see (sometimes, painfully so) the other side: intermediaries are often where regulation is enforced, and they enable oversight and safety. A world with fewer intermediaries can indeed be a less governable world. And while I believe broadening direct access to powerful AI technologies is a net positive for humanity, it also comes with severe asymmetries in some domains. For instance, biosecurity is currently offence-dominant: a single malicious actor (or ‘dissident’), sufficiently empowered, can cause damage no defence yet can match.

So wat do? Are we stuck between a free but dangerous world, or a surveilled, controlled, ‘safer’ world?

Part of this post’s job is to show why the second option is not the safe harbour it appears to be, and to widen the safety conversation that often defaults to centralised control and alignment as the only available levers. A perfectly aligned model running on surveillance infrastructure still delivers the panopticon. Alignment binds the model to its principal; it does not bind the principal to us. And as laid out above, infrastructure outlives its operators, whatever is built under careful stewardship is inherited by whoever comes next.

Practically though, I have two answers for you:

1. In cases where ’trusted’ intermediaries remain because it is the structural and/or safety best option (e.g. screening gene-synthesis orders), allowing the public to check their power, building technologies to constrain it, and minimising duopolies/monopolies to enable contestability goes a long way towards creating a safe, yet less surveilled society.
2. In cases where trusted intermediaries can be entirely removed, trust doesn’t disappear, it shifts substrate to cryptography (trust in maths) and trusted hardware (trust in silicon). The same machinery that enables trustworthy interactions can enable distributed safety: proof that the safety-evaluated model was the one that ran, attestation that an agent stays within its declared constraints, audit trails that open under due process rather than on demand. The same goes for every layer of the stack, proving what a model was trained on, what a chip is running, what an inference provider retained. Overseers get proofs instead of feeds, dissolving the old tradeoff between oversight and privacy.

Technology is not enough

We’ve been somewhere like this before. In 1993, Eric Hughes wrote A Cypherpunk’s Manifesto :

“Privacy is necessary for an open society in the electronic age… Privacy is the power to selectively reveal oneself to the world.”

At the time, the US government treated encryption as a munition, put it on the same export-control list as missiles, and spent years advocating for a phone chip with a built-in government backdoor. Cryptographers and privacy advocates like Bernstein and Zimmermann, together with organisations such as the Electronic Frontier Foundation, fought hard against the ’easy’ defaults, arguing that a backdoored system is weaker for everyone and that strong cryptography is a matter of privacy and free expression.¹⁷ Congress’s own commissioned review agreed: the National Research Council’s 1996 CRISIS report concluded that the use of cryptography should not be restricted and export controls should be loosened. By 2000 the export controls had collapsed, and today, encryption has become core digital infrastructure, securing nearly all web traffic.¹⁸

I am not claiming today’s situation is the same. The stakes are higher given how powerful the technology is; control arguably sits more with corporations than with governments, and I have yet to see a situation where a clearly sub-par technological option is being pushed against a better alternative for humanity. What I am claiming, however, is that the defaults for artificial intelligence are now being set at every layer of the stack, and that they will shape our future. Having technological alternatives won’t be enough. We will need coordinated action across society: developers, policymakers, frontier labs, privacy and safety advocates, standards bodies, and citizens.

Which new intermediaries and points of information control are emerging? Which should remain, and how do we make those that do checkable and contestable by the people who depend on them? What kind of local maxima defaults do we risk falling into? How will we steer towards global maxima and which voices – whether policymakers, citizens, or organisations – will shape them? These questions deserve people, and institutions. If this is you, please reach out. We would like to support you!

By Alex Obadia , assisted by Fable 5. Thank you Nicola Greco, Nora Ammann, James Fox, Seb Krier, Lukas Petersson, Mona Wang, Allison Duettmann, Christian Catalini, Lewis Hammond, Louise Ellaway and Melissa Bradshaw for conversations and reviews that shaped this post.

Have comments or feedback on the post? Please send it over or comment directly on here using Hypothesis!