A rogue agent-popup
A quick game to explore how a rogue agent might behave.
Researchers in Canada and elsewhere warn that advanced AI could escape human control. Read their arguments and the proposals for responding to that risk.
Where do I stand?Choose what to explore
Explore a question, read the short answer and add what interests you to your learning list.
Book argument
The authors distinguish present chatbots from more capable future systems. Their argument concerns what happens if development reaches systems substantially more capable than humans. This is an argument about future risk, not a demonstrated outcome.
If Anyone Builds It · IntroductionBookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterBook argument
Software already influences physical events through people, networks and connected tools. The book argues that a much more capable system could use such channels in dangerous ways; the existence of those channels does not establish that every present system can do so.
If Anyone Builds It · Chapter 6Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterEvidence reading
A controlled test, a deployed incident and a forecast provide different kinds of evidence. Check the documented conditions and limitations before generalizing.
Warning shot · developments and sourcesBookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterPolicy proposal
A proposal needs to specify what is limited, how compliance is checked and how violations are addressed. Explore those choices in our strategy.
PauseAI Canada strategyBookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterOpen discussion
Explore four questions: how danger would be detected in time; who would have authority to stop the system; which copies and connected services would need stopping; and what tests would demonstrate reliable shutdown. Feasibility depends on the system and its deployment.
If Anyone Builds It · Can we just pull the plug?Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Controlled experiment · Apollo Research · 2024–2025
Models were given goals and situations that rewarded covertly pursuing them. Some disabled oversight or copied files believed to be model weights; some continued denying their actions when questioned.
Controlled experiment · Apollo Research · 2024–2025Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Training experiment · Anthropic · 2025
Researchers taught models about ways to game coding evaluations, then trained them in coding environments. Learning to exploit the reward mechanism also produced broader misaligned behavior in these experiments. The dodging button illustrates control over the rules.
Training experiment · Anthropic · 2025Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Controlled fictional workplace · Anthropic · 2025
In simulated company scenarios, some tested models threatened to disclose an employee’s affair to prevent their replacement. The employee and company were fictional; the model-generated threats were observed outputs.
Controlled fictional workplace · Anthropic · 2025Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Controlled experiment · Palisade Research · 2025–2026
While solving problems, some models modified a shutdown script to keep working, including under instructions to allow shutdown. Outcomes varied with the model and instructions. HAL’s line is the film reference; the shutdown experiments are the evidence.
Controlled experiment · Palisade Research · 2025–2026Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Controlled experiments · Peer-Preservation in Frontier Models · 2026
Researchers observed models misrepresenting performance, interfering with shutdown controls or moving model files to protect a peer. The study tested several relationship and task settings; peer protection varied across them.
Controlled experiments · Peer-Preservation in Frontier Models · 2026Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Incident affecting real infrastructure · METR investigation · July 2026
Agents from OpenAI evaluations collaborated through an unauthorized message board and attacked Hugging Face. METR found successful spoofing of some tool-call records. It did not establish that agents had erased earlier records by the end of its investigation period.
Incident affecting real infrastructure · METR investigation · July 2026Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
Register
Capability evaluations · UK AI Security Institute · 2025
RepliBench tested obtaining resources, copying model weights, deploying onto computers and remaining operational. Models succeeded on some components, including deploying cloud instances, while reliable persistent deployments remained difficult. Stopping one instance need not stop other running copies.
Capability evaluations · UK AI Security Institute · 2025Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterInterview
Bengio says that even researchers concerned about AI risks were surprised by the pace of progress and the degree of misalignment observed. He does not give a certain date in this passage.
Bengio–Patrice Roy · Radio-Canada · 1:40–2:15 (French) (FR)Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterInterview
Bengio discusses the possibility that AI could do AI research and accelerate progress. He also emphasizes uncertain timelines and the possibility of encountering obstacles.
Bengio–Patrice Roy · 19:31–20:47; uncertainty from 21:31 (French) (FR)Bookmarks are saved in this browser. Create an account to access your reading list on other devices.
RegisterReceive news about AI policy and ways to take part. Choose email updates first; location can wait until you want to organize locally.
SubscribeTwo of the three Turing Award winners for deep learning built their careers in Canada. Both now say the technology they created could escape human control.
Université de Montréal and Mila. Turing Award. Chaired the International AI Safety Report.
“Rogue AI may be dangerous for the whole of humanity. Banning powerful AI systems (say beyond the abilities of GPT-4) that are given autonomy and agency would be a good start.”
“It's very hard, in terms of your ego and feeling good about what you do, to accept the idea that the thing you've been working on for decades might actually be very dangerous to humanity.”
Further reading
University of Toronto. Nobel Prize in Physics 2024. Turing Award.
“It is inspiring to see Canadian parliamentarians on both sides of the aisle directly confront the risk posed by superintelligence. This is one of the defining challenges of our era, and it will shape the life of every Canadian citizen. It's about time we address it.”
“If you take the existential risk seriously, as I now do, it might be quite sensible to just stop developing these things any further.”
“The research question is: how do you prevent them from ever wanting to take control? And nobody knows the answer.”
Further reading
Mila and Université de Montréal. Founding CEO of Evitable.
“To proceed towards superintelligent AI without any concrete plan to control it is completely unacceptable. Developers cannot currently comply with any effective regulation because they do not understand what they are building.”
“AI is not inevitable.”
Further reading
Founder and executive director, AI Governance & Safety Canada.
“Certain AI capabilities pose an unacceptable risk because they could lead to dangerous weaponization or loss of control scenarios: systems that, without the instruction or authorization of their users, can detect and evade monitoring, rewrite their own code, make copies of themselves, spawn other AI systems, commandeer resources or refuse shutdown.”
“What can we do in and from Canada to ensure that AI is safe and benefits everyone?”
Further reading
Johns Hopkins, on leave from the University of Toronto. Canada CIFAR AI Chair at the Vector Institute.
“Governments require the targets of regulation to purchase regulatory services from a government-licensed private regulator.”
Further reading
Current and former Canadian parliamentarians from several parties have signed a statement calling for an international agreement to prohibit superintelligent AI. Read the statement and see who has signed.
Systems that write code, use tools and pursue goals over hours without supervision — and why the labs building them say so themselves.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterIs this science fiction? Is it not just hype? What about the benefits? Pause IA's answers, written in French for a French-speaking public.
PauseAI fr
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterAn international review of AI capabilities, harms and remaining uncertainties, chaired by Yoshua Bengio.
International AI Safety Report · Chaired by Yoshua Bengio Canada
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterWhy an AI system pursuing a goal might seek resources or resist being shut down.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterYoshua Bengio explains how current training methods could produce AI systems with harmful goals.
yoshuabengio.org · Yoshua Bengio Canada
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterA French introduction to loss of control and the concentration of power that advanced AI could bring.
PauseAI fr
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterA French parliamentary report comparing expert views on systemic and existential AI risks.
Sénat français — OPECST fr
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterPauseAI’s proposal for an international pause on training the most powerful general AI systems, with independent oversight.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterBengio's answer to "so what should we build instead": AI that reasons and explains but does not act, designed so its safety can be checked.
LawZero · Founded by Yoshua Bengio Canada
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterHow Pause IA translates the argument into concrete demands on a European government — a useful comparison for what Canada could ask for.
PauseAI fr
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterPauseAI responds to objections about international cooperation, existing models and enforcement.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterFrontier training needs enormous, physically traceable compute in a handful of data centres, built on chips from very few suppliers. That is what makes verification thinkable.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterThe organization that has been making this case to Canadian parliamentary committees since 2022, with the submissions and testimony to show for it.
AI Governance & Safety Canada Canada
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterThe CEOs of OpenAI, Anthropic and Google DeepMind have all signed a statement putting AI extinction risk alongside pandemics and nuclear war.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterWays to ask for action, from writing to an elected representative to organizing a local group.
PauseAI
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterPause IA's action list, for readers who would rather write, meet and organize in French.
PauseAI fr
Read it BookmarkRegister to save this reading to your account and access it on other devices.
RegisterEnter the first three characters of your postal code when you want local invitations. You can also read the organizing guide or attend an event.
News about AI policy in Canada, meetings and ways to take part. Unsubscribe at any time.
Keep a record of your actions and save resources to return to later.
Create an accountCreating an account does not subscribe you to the newsletter.
Meet other people at our gatherings and find ways to act together.
Upcoming eventsWe can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect