scaling trustcommunity

A curated collection of links for the Scaling Trust community to get you started diving deeper in the topics we care about.

This list is alive — we add, change, and cut it all the time, and we welcome the community’s input: suggest links on Discord, in #link-sharing.

Scaling Trust Programme

Key links related to ARIA’s Scaling Trust programme.

• Programme page — the Scaling Trust source of truth for official comms.
• Programme thesis (PDF) — the case for the programme.
• Opportunity space: Trust Everything, Everywhere — the landing page, and the full document (PDF).
• Funding efforts — open and upcoming funding calls.
• Pre-programme discovery — outputs from our first cohort of 14 short projects, spanning multi-agent, cryptographic, and cyber-physical trust.
• Discovery workshop recordings — talks from our October 2025 workshop, from Consumable quantum data to Securing ultra-large-scale cyber-physical infrastructure.
• Scaling Trust Discord — community discussion, questions, and updates. Suggest a link to add in here via the #link-sharing channel!

Related communities and efforts

Kindred communities and friends, suggest your own!

• Safeguarded AI — our sister ARIA programme in the same opportunity space, directed by Nora Ammann: fleets of AI agents that model and verify critical cyber-physical systems, with mathematical safety guarantees.
• Cooperative AI Foundation Slack — the CAIF community.
• Schmidt Sciences: AI Agents — funding research on how multiple intelligent agents communicate and coordinate.
• RESI — the Institute for Responsible Superintelligence. Shafi Goldwasser, Vinod Vaikuntanathan and Adam Kalai on safe-by-design AI: safety properties specified up front and achieved by mechanisms whose guarantees can be analysed, cryptography included.
• Resolution — Geoffrey Irving, Daniel Murfet, Jesse Hoogland and the former UK AISI alignment team. A portfolio of theory and empirics bets for higher-confidence alignment.
• MAISI — the Mathematical AI Safety Institute. Jacob Tsimerman and Andrew Critch building the definitions, measurements and solution concepts AI safety is missing.
• Cosmos Institute — training “philosopher-builders” so AI serves human flourishing.
• Edge City — popup villages where builders live and experiment together; the Agent Village experiment at Edge Esmeralda ran there.
• Agent Village — AI Digest’s long-running experiment: frontier agents living, coordinating and raising money together in public; see what they learned in 2025.
• Secure Program Synthesis — the community forming around provably secure software generation.
• Also funding work in this space: CAIF grants · Foresight Institute · Coefficient Giving · Survival and Flourishing Fund · UK AISI · SAIR

Vision

Some of the pieces that have inspired the programme.

• Is Information the Key? — Gilles Brassard. Quantum theory’s deepest laws may be about information itself.
• The Moral Character of Cryptographic Work — Phil Rogaway. Cryptography is political, and the field has moral obligations.
• The Quantum Thief — Hannu Rajaniemi. A heist novel set in a society that runs on cryptographic privacy.
• Black-Hole Radiation Decoding is Quantum Cryptography — Zvika Brakerski. Physics problems recast as cryptographic hardness.
• Coasean Bargaining at Scale — Seb Krier (Google DeepMind). AI agents collapse transaction costs and make hyper-local bargaining viable.
• The Coasean Singularity? — Shahidi et al. (NBER). How agent markets reshape demand, supply, and market design.
• Without Intermediaries — Alex Obadia. Where AI’s new intermediaries are forming, why their power is dangerous even in careful hands, and how to keep it checkable and contestable.
• Conditional Recall — Christoph Schlegel and Xinyuan Sun. Commitments to provably forget information, and what they unlock.
• Open Challenges in Multi-Agent Security — Christian Schroeder de Witt et al. The agenda-setting map of the field this programme funds.
• Multi-Agent Risks from Advanced AI — Hammond et al. The field survey: collusion, conflict, destabilising dynamics — and why they don’t reduce to single-agent safety.
• Positive Alignment — Laukkonen, Krier, Bakalar et al. Alignment as actively fostering flourishing, not just preventing harm.
• Distributional AGI Safety — Tomašev et al. (Google DeepMind). AGI may arrive as networks of agents, so safety becomes an ecosystem property.
• Programmable Cryptography — 0xParc. The case for general-purpose, composable cryptography.
• The Tragedy of the Agentic Commons — Strange Loop Canon. Why agent ecosystems strain shared resources without new governance.
• Beyond Markets and States — Elinor Ostrom’s Nobel lecture. Communities govern shared resources through layered, local institutions — neither market nor state. The hopeful reply to the agentic commons.

Multi-agent foundations

Start here if the field is new to you: what counts as a multi-agent system, and why several principals is a different problem from several agents.

• Multiagent Systems — Shoham and Leyton-Brown. The free textbook: games, learning, communication, social choice, mechanism design. The vocabulary for most of what follows.
• Large Language Models Miss the Multi-Agent Mark — La Malfa et al. Several LLM calls in a pipeline is not a multi-agent system. What autonomy, interaction, and environment actually require.
• Multi-Principal Assistance Games — Fickinger, Zhuang, Critch, Hadfield-Menell, and Russell. One assistant, many people who disagree: where alignment meets social choice.
• Multi-Agent Systems Should be Treated as Principal-Agent Problems — Rauba, Cepenas, and van der Schaar. Information asymmetry, diverging incentives, agency loss: the old economics of delegation, applied to agent teams.

Multi-agent failures & control

What goes wrong when agents are optimised, connected, or composed — and what it takes to stay in control.

• Categorizing Variants of Goodhart’s Law — Manheim and Garrabrant. Four ways optimising a proxy breaks the goal: the reusable vocabulary for reward hacking and metric gaming.
• Why Do Multi-Agent LLM Systems Fail? — Cemri et al. (Berkeley). The empirical failure taxonomy: 14 recurring breakdowns across seven frameworks, from bad specification to weak verification.
• Prompt Infection — Lee and Tiwari. Prompt injection as contagion: a malicious instruction that self-replicates across connected agents.
• Adversaries Can Misuse Combinations of Safe Models — Jones, Dragan, and Steinhardt. Safety is not compositional: models that pass alone can be combined to do what none would do.
• AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. (Redwood). Getting useful work out of a model that may be trying to defeat your monitoring. The multi-agent sequel splits attacks across agents so no single trace looks suspicious.

Agentic game theory

How strategic behaviour changes when the players are AIs.

• Cooperative AI: machines must learn to find common ground — Dafoe et al. (Nature). The commentary that named the field; Conitzer and Oesterheld on why it needs foundations of its own.
• Program Equilibrium — Moshe Tennenholtz. The foundational result: agents that can read each other’s code reach equilibria humans can’t.
• Secret Collusion among AI Agents — Motwani et al. Agents hiding coordination in plain sight via steganography.
• Algorithmic Collusion by Large Language Models — Fish, Gonczarowski, and Shorrer. LLM pricing agents drift to supracompetitive prices on their own — and a few words in their instructions swing the outcome.
• Secure and Secret Cooperation in Robotic Swarms — Ferrer et al. Swarms that cooperate without exposing their members.
• Language Models Can Reduce Asymmetry in Information Markets — Rahaman et al. Agents brokering information they can inspect but not leak.
• Mechanism Design for Large Language Models — Dütting et al. Auction design when the bidders are language models.
• Virtual Agent Economies — Tomašev et al. (Google DeepMind). How to deliberately design the agent markets ahead of us.

Agent infrastructure

Identity, delegation, and commitments — the rails agents transact on, and the two classic ways rails break.

• The Sybil Attack — John Douceur. Without a trusted authority, one actor can pose as many. Every question about reputation, voting, and delegation starts here.
• The Byzantine Generals Problem — Lamport, Shostak, and Pease. Agreement when some participants lie. Trust is a property of fault models and communication structure, not of good behaviour.
• Infrastructure for AI Agents — Chan et al. The base layer agents need: identity, channels, oversight hooks.
• Authenticated Delegation and Authorized AI Agents — South et al. (MIT). Who authorised this agent, and to do what? Scoped, auditable delegation built on OAuth and OpenID Connect.
• Inter-Agent Trust Models — a comparative study of trust across A2A, AP2, and ERC-8004.
• NDAI Agreements — Stephenson et al. Contracts between AIs without trusted third-party enforcement.

Multi-agent testbeds & experiments

New, more realistic/open-ended environments (not just simple games) that are starting to be used to test LLMs in multi-agent settings.

• Generative Agents — Park et al. (Stanford). Smallville: 25 agents with memory and plans throw a party nobody scripted. The experiment that started the genre.
• Concordia — Vezhnevets et al. (Google DeepMind). Generative agent-based modelling with a “game master” adjudicating actions and consequences; the open-source framework is built for experiments on institutions and norms.
• Emergence World — persistent simulation for long-horizon autonomy, coalitions, governance, and behavioural drift.
• The Agent Village at Edge Esmeralda — a 500-person popup town gave everyone personal agents for coordination and governance.
• Project Deal — Anthropic. Claude buying, selling, and negotiating in a real office marketplace.
• Vending-Bench Arena — Andon Labs. Competing agents run vending businesses over months.
• Project Iceberg — MIT Media Lab. National-scale human–AI labour-market simulation.
• TERMS-Bench — a diagnostic benchmark for LLM negotiation: surplus capture, calibration, opponent modelling.
• CRUX — Kapoor et al. Open-world evals on long-horizon tasks that resist clean grading.
• Digital Red Queen — Sakana. Adversarial program evolution in Core War.

Autonomous protocol generation

Towards agents that generate provably secure protocols on demand — and agents that break them.

• Generative Cryptography — Nicola Greco. Cryptography is unusually well suited to AI research loops: problems stated formally, solutions checked mechanically. On using, improving and inventing cryptography with AI.
• How to Solve Secure Program Synthesis — Max von Hippel et al. The four open challenges on the way to provably secure software generation.
• Proof Assistants in the Age of AI — Leo de Moura. Lean’s creator on what AI changes for formal proof, and why machine-written code needs machine-checked guarantees.
• A Benchmark for Vericoding — Bursuc, Tegmark et al. Vibe coding’s rigorous cousin: 12,504 formal specifications in Dafny, Verus and Lean, and a scoreboard of how much of it frontier models can already prove.
• Scaling Verification of Cryptographic Software with Aeneas, Rust, and Lean — Ho, Fournet, Protzenko et al. (Microsoft Research). Agents write the Lean proofs, the kernel checks them: SHA-3 and ML-KEM in Microsoft’s production crypto library, verified.
• CatCrypt — Bas Spitters (Aarhus). One level up from “the code matches the spec”: machine-checked security proofs in the computational model, from Rust reference implementations, in Lean.
• ArkLib — a Lean 4 library formally verifying SNARK components (Sum-Check, Spartan, FRI).
• CryptanalysisBench — Fluri, Shafran et al. 191 attack tasks against real primitives, graded by running the attack. Frontier models found a key-recovery attack on SpoC and an error in a published security proof.
• Discovering Cryptographic Weaknesses with Claude — Anthropic. Claude Mythos Preview weakens HAWK, a post-quantum signature candidate, and finds a new attack on round-reduced AES. AI Grinding for Fun and Cryptanalysis is the independent version: an autonomous attack loop reporting failures in eight published constructions.
• LLMs Won’t Break Symmetric Crypto — JP Aumasson. A working cryptanalyst’s sober read of the same results: meaningful breaks, none practical, and why ciphers built to have no structure give an AI nothing to reason about.
• Yukon — Eigen Labs. Autoresearch as a platform: post a repo, a benchmark and a verifier, and open the problem to everyone’s agents. Its ECDSA.fail solvers report beating Google Quantum AI’s circuit for attacking elliptic-curve signatures.
• The Proximity Prize and zkGolf — Yukon and zkSecurity. Research as a leaderboard: agents tighten a SNARK soundness bound, or shrink a zero-knowledge circuit, and Lean checks every submission before it counts.

Secure requirement capture

How agents learn what we want — without leaking it.

• Privacy as Contextual Integrity — Helen Nissenbaum. The canonical frame: privacy is appropriate information flow, not secrecy.
• Privacy Reasoning in Ambiguous Contexts — Yi et al. Can models judge what’s appropriate to share, in context?
• Oath Technologies — a Focused Research Organization on AI oversight via formal methods. Their bet: the hard part is stating what humans want in a form that is understandable and robust to AI deception.

Formal AI security

Provable guarantees about learned systems.

• Provably Safe Systems — Tegmark and Omohundro. The maximalist case: safety guarantees should be mathematical proofs.
• Models That Prove Their Own Correctness — Amit, Goldwasser, Paradise, and Rothblum. Models that emit proofs their answers are right.
• Defeating Prompt Injections by Design — Debenedetti et al. System designs that make injection structurally impossible, not just unlikely.

Physical verification & secure hardware

Proving things about the physical world — chips, cameras, labs.

• Physical Evals — Nicola Greco. Benchmarks end where the world begins: what it takes to evaluate agents in the actual physical world, and why that needs an open ecosystem.
• Piloting the World’s First Double-Blind AI Evaluations — Google DeepMind, with the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons. A frontier model evaluated inside a confidential-computing enclave: evaluators never see the weights, Google never sees the test set.
• Model Hardware Standard — Anthropic and HHMI Janelia. A shared spec for AI agents to drive lab and manufacturing instruments in parallel, with safety evaluations alongside — research preview.
• Remotely Detectable Robot Policy Watermarking — Amir, Flageat, and Prorok. Spectral signals hidden in robot motion, detectable from video alone.
• IRIS (Infra-Red, in situ) Project Updates — Bunnie Studios. Non-destructive chip inspection with infrared imaging.
• SecureDNA — Baum et al. Cryptographic screening of the world’s DNA synthesis orders.
• PCR7500 — trusted bioreactors: PCR data signed in a TEE at the point of capture.

Nature cryptography?

Security primitives based on the laws of nature.

• Physical One-Way Functions — Pappu et al. The original physically unclonable function: optical tokens as one-way functions.
• A New Approach to Nuclear Warhead Verification Using a Zero-Knowledge Protocol — Glaser et al. Prove a warhead is real without revealing its design.
• Quantum Cryptography: Uncertainty in the Service of Privacy — Charles Bennett. Quantum uncertainty itself as a privacy primitive.
• Consumable Data via Quantum Communication — Gilboa et al. Data that can only be used a bounded number of times.
• Conjugate Coding — Stephen Wiesner. The 1970s manuscript that invented quantum money.
• Unclonable Polymers and Their Cryptographic Applications — Almashaqbeh et al. Secret keys stored in molecules that can’t be copied.
• Building Unclonable Cryptography: A Tale of Two No-cloning Paradigms — Almashaqbeh et al. The field map of unclonability, quantum and physical.
• An Introduction to Protein Cryptography — Tirmazi et al. Encoding data in amino-acid sequences, tamper- and copy-resistant.
• Cryptography in the DNA of Living Cells — Volf et al. Multi-site base editing as message encryption inside living cells.
• Hidden Messages in DNA Could Reduce Biosecurity Risks — Danielle Gerhard. DNA watermarks for biosecurity.
• Neuroscience Needs Network Science — Barabási et al. The brain as a network-science problem.
• A New Age of Computing and the Brain — Golland et al. A research agenda where computing and neuroscience meet.
• Mosquito-derived Ingested DNA as a Tool for Monitoring Terrestrial Vertebrates — Chivas et al. Mosquito blood meals as a wildlife-monitoring sensor network.
• Molecules that Generate Fingerprints — Motiei et al. Fluorescent sensors as chemical fingerprints for authentication.
• TTEE: Marrying Cryptography and Physics — Quintus Kilbourn. A talk on what physics can do for cryptographic trust.
• Signature for Objects — Hayashi et al. Signing physical objects, with unforgeability against physically-enhanced adversaries.
• Cryptographic Data Exchange for Nuclear Warheads — Perry et al. The modern follow-on to zero-knowledge warhead verification.
• Cryptographic Sensing — Ishai et al. Sensors that reveal only the authorised measurement and nothing else.
• Cryptography by Cellular Automata — Applebaum et al. How fast can cryptographic complexity emerge in nature?

Suggest a link via Discord — try #link-sharing.