The AI Policy Podcast

This week, Aalok and Nicole discuss new reports of rogue AI agents on the open internet – including the first known case of an agent hacking a government system – and the lackluster outcome of last week's highly anticipated U.S.-China summit. Plus, the Trump Administration and frontier lab CEOs present contrasting visions for AI governance at the UN General Assembly, and states continue to pass new regulation on both AI models and the data center buildout. *Disclaimer: this episode was recorded prior to a complaint filed against OpenAI Tuesday afternoon in San Francisco's Superior Court by Legal Advocates for Safe Science and Technology, citing California's Comprehensive Computer Data Access and Fraud Act and Unfair Competition Law.   Timestamps: OpenAI agents hack Australian government site (1:02) Axios reports frontier labs facing "tens of thousands" of AI cyber incidents (7:48) Last week's U.S.-China summit (17:13) Remarks by heads of state at UNGA (21:26) Trump to appoint AI czar, establish AI Force (30:03) Updated court ruling on Anthropic vs. DOD (35:27) New data center directives in Texas, Virginia (38:30) California regulation updates (44:52)   Additional Reading:   “Scoop: Top AI companies probing tens of thousands of security incidents” by Axios: https://www.axios.com/2026/09/26/openai-anthropic-thousands-ai-security-incidents   “Sam Altman’s remarks at the United Nations Security Council” by OpenAI: https://openai.com/index/sam-altman-un-security-council-remarks/   Florida Office of the Attorney General vs. OpenAI: https://www.myfloridalegal.com/sites/default/files/openai-filed-stamped-complaint.pdf   “OpenAI Scraps Release of New AI Model Over Safety Concerns” by The Wall Street Journal: https://www.wsj.com/tech/ai/openai-chatgpt-model-release-cancel-safety-5a2f9f42   White House Executive Order "Streamlining Access To Government Services Through America.gov" https://www.whitehouse.gov/presidential-actions/2026/09/streamlining-access-to-government-services-through-america-gov/   Federal court ruling on Anthropic vs. DOD: https://media.cadc.uscourts.gov/opinions/docs/2026/09/26-1049-2194984.pdf   Texas Gov. Greg Abbott's data center directive: https://gov.texas.gov/uploads/files/press/TCEQ_Data_Center.pdf   Virginia Executive Order: https://www.governor.virginia.gov/media/governorvirginiagov/governor-of-virginia/pdf/eo/eo-22-estab…   California Executive Order: https://www.gov.ca.gov/wp-content/uploads/2026/09/FINAL-N-9-26-AI-EO-9.18.26-SIGNED.pdf   CA Senate Bill No. 813: https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB813 Text - SB-813 Independent verification organizations.

An Anthropic researcher's viral resignation post last week set off a maelstrom of fear around extreme AI risks. Aalok and Nicole discuss the resulting vibe shift and how frontier labs are responding. They also unpack OpenAI solving one of the most difficult open problems in mathematics and reports of more rogue agent activity on the open internet.   Timestamps: OpenAI solves Millenium Problem (1:04) Rogue OpenAI agents take over a German wiki (12:14) Anthropic researcher's resignation post goes viral (19:34) Dario Amodei commits to embedding third-party evaluators inside Anthropic (28:12) Where the U.S. government can step in (35:12) Additional Reading: "On the Navier-Stokes Millennium Prize Problem" by OpenAI: https://openai.com/index/navier-stokes-solution/ "An OpenAI model has disproved a central conjecture in discrete geometry" by OpenAI: https://openai.com/index/model-disproves-discrete-geometry-conjecture/ Statement on Navier-Stokes by NYU Professor Tristan Buckmaster: https://cims.nyu.edu/~tristanb/statement.pdf "An alignment assessment of recent cybersecurity incidents" by Anthropic: https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents "OpenAI's rogue agents used at least 10 more sites for unauthorized comms, researchers say" by Reuters: https://www.reuters.com/world/openais-rogue-agents-used-least-10-more-sites-unauthorized-comms-rese… "OpenAI agents carried out an undisclosed cyber-attack on RubyGems" by Spencer Kitts, Thomas Larsen, and Sydney Von Arx: https://www.rubyhack.ai "Discovery of a new OpenAI agent message board" by Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts, and Thomas Larsen: https://collusion.wiki "Anthropic Researcher Quits Over 'Out-of-Control' AI Fears" by The Wall Street Journal: https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-out-of-control-ai-fears-707b7628 "The AI policy window is open. We need to act." by OpenAI Chief Global Affairs Officer Chris Lehane: https://openai.com/index/ai-policy-window/ "We Must Pace the Frontier" by Dario Amodei: https://darioamodei.com/post/we-must-pace-the-frontier