--- description: A Conditional AI Safety Treaty can help prevent runaway AI and President-elect Trump should embrace the idea, writes Otto Barten. title: There Is a Solution to AI's Existential Risk Problem image: https://static.time.com/v3/assets/bltea6093859af6183b/blta73e18839eaa69a7/698a9bd7929fad9880c039dd/GettyImages-1804572639.jpg?branch=production&width=3840&quality=75&auto=webp&crop=16:9 ---  * [Ideas](/section/ideas/) ## Ideas Update your preferences in [Account Settings](/account/preferences/) Close # There Is a Solution to AI's Existential Risk Problem by [Otto Barten](/author/otto-barten/) ## Otto Barten Update your preferences in [Account Settings](/account/preferences/) Close Nov 15, 2024 12:11 PM CUT  Artificial General Intelligence symbol locked in a birdcage could illustrate concepts around AGI safety rules and regulations and danger of AGI getting out of control Artificial General Intelligence symbol locked in a birdcage could illustrate concepts around AGI safety rules and regulations and danger of AGI getting out of controlGetty Images by [Otto Barten](/author/otto-barten/) ## Otto Barten Update your preferences in [Account Settings](/account/preferences/) Close Nov 15, 2024 12:11 PM CUT Technological progress can excite us, politics can infuriate us, and wars can mobilize us. But faced with the risk of human extinction that the [rise of artificial intelligence](https://time.com/6271657/a-to-z-of-artificial-intelligence/ "undefined") is causing, we have remained surprisingly passive. In part, perhaps this was because there did not seem to be a solution. This is an idea I would like to challenge. AI’s capabilities are ever-improving. Since the release of [ChatGPT](https://time.com/6238781/chatbot-chatgpt-ai-interview/ "undefined") two years ago, hundreds of billions of dollars have poured into AI. These combined efforts will likely lead to Artificial General Intelligence (AGI), where machines have human-like cognition, perhaps [within just a few years](https://www.businessinsider.com/openai-nears-ai-systems-reason-cause-concern-sam-altman-chatgpt-2024-7 "undefined"). Hundreds of AI scientists [think](https://www.safe.ai/work/statement-on-ai-risk "undefined") we might lose control over AI once it gets too capable, which could result in human extinction. So what can we do? **Read More:** [_What Donald Trump's Win Means For AI_](https://time.com/7174210/what-donald-trump-win-means-for-ai/ "undefined") The existential risk of AI has often been presented as extremely complex. A [2018 paper](https://dl.acm.org/doi/10.1145/3278721.3278746 "undefined"), for example, called the development of safe human-level AI a “super wicked problem.” This perceived difficulty had much to do with the proposed solution of [AI alignment](https://www.amazon.com/Alignment-Problem-Machine-Learning-Values/dp/0393635821 "undefined"), which entails making superhuman AI act according to humanity’s values. AI alignment, however, was a problematic solution from the start. First, scientific progress in alignment has been much slower than progress in AI itself. Second, the philosophical question of which values to align a superintelligence to is incredibly fraught. Third, it is not at all obvious that alignment, even if successful, would be a solution to AI’s existential risk. Having one friendly AI does not necessarily stop other unfriendly ones. Because of these issues, many have urged technology companies not to build any AI that humanity could lose control over. Some have gone further; activist groups such as PauseAI have indeed [proposed](https://pauseai.info/proposal "undefined") an international treaty that would pause development globally. That is not seen as politically palatable by many, since it may still take a long time before the missing pieces to AGI are filled in. And do we have to pause already, when this technology can also do a lot of good? Yann Lecun, AI chief at Meta and prominent existential risk skeptic, [says](https://x.com/ylecun/status/1823313599252533594 "undefined") that the existential risk debate is like “worrying about turbojet safety in 1920.” On the other hand, technology can leapfrog. If we get another breakthrough such as the [transformer](https://arxiv.org/abs/1706.03762 "undefined"), a 2017 innovation which helped launch modern Large Language Models, perhaps we could reach AGI in a few months’ training time. That’s why a regulatory framework needs to be in place before then. Fortunately, [Nobel Laureate](https://time.com/7065011/nobel-prize-2024-winners/ "undefined") [Geoffrey Hinton](https://time.com/collection/time100-ai/6309026/geoffrey-hinton/ "undefined"), Turing Award winner Yoshua Bengio, and many others have provided a piece of the solution. In a policy paper published in _Science_ earlier this year, they [recommended](https://www.science.org/doi/10.1126/science.adn0117 "undefined") “if-then commitments”: commitments to be activated if and when red-line capabilities are found in frontier AI systems. Building upon their work, we at the nonprofit Existential Risk Observatory propose a Conditional AI Safety Treaty. Signatory countries of this treaty, which should include at least the U.S. and China, would agree that once we get too close to loss of control they will halt any potentially unsafe training within their borders. Once the most powerful nations have signed this treaty, it is in their interest to verify each others’ compliance, and to make sure [uncontrollable AI](https://time.com/6258483/uncontrollable-ai-agi-risks/ "undefined") is not built elsewhere, either. One outstanding question is at what point AI capabilities are too close to loss of control. We propose to delegate this question to the [AI Safety Institutes](https://time.com/6962503/ai-artificial-intelligence-uk-us-safety/ "undefined") set up in the [U.K](https://www.aisi.gov.uk/ "undefined")., [U.S](https://www.nist.gov/aisi "undefined")., [China](https://beijing.ai-safety-and-governance.institute/ "undefined"), and [other countries](https://www.commerce.gov/news/press-releases/2024/09/us-secretary-commerce-raimondo-and-us-secretary-state-blinken-announce "undefined"). They have specialized model evaluation know-how, which can be developed further to answer this crucial question. Also, these institutes are public, making them independent from the mostly private AI development labs. The question of how close is too close to losing control will remain difficult, but someone will need to answer it, and the AI Safety Institutes are best positioned to do so. We can mostly still get the benefits of AI under the Conditional AI Safety Treaty. All current AI is far below loss of control level, and will therefore be unaffected. Narrow AIs in the future that are suitable for a single task—such as climate modeling or finding new medicines—will be unaffected as well. Even more general AIs can still be developed, if labs can demonstrate to a regulator that their model has loss of control risk less than, say, 0.002% per year (the [safety](https://en.wikipedia.org/wiki/Core%5Fdamage%5Ffrequency "undefined") threshold we accept for nuclear reactors). Other AI thinkers, such as MIT professor [Max Tegmark](https://www.lesswrong.com/posts/oJQnRDbgSS8i6DwNu/the-hopium-wars-the-agi-entente-delusion "undefined"), Conjecture CEO [Connor Leahy](https://www.thecompendium.ai/ "undefined"), and ControlAI director [Andrea Miotti](https://www.narrowpath.co/ "undefined"), are thinking in similar directions. Fortunately, the existential risks posed by AI are recognized by many close to President-elect Donald Trump. His daughter Ivanka [seems to see](https://x.com/IvankaTrump/status/1839002887600370145 "undefined") the urgency of the problem. Elon Musk, a critical Trump backer, has been [outspoken](https://time.com/3541005/elon-musk-artificial-intelligence/ "undefined") about the civilizational risks for many years, and recently [supported](https://www.politico.com/news/2024/08/26/elon-musk-supports-california-ai-bill-00176388 "undefined") California’s legislative push to safety-test AI. Even the right-wing Tucker Carlson provided common-sense commentary when he [said](https://www.youtube.com/watch?v=DfTU5LA%5Fkw8&t=34m25s "undefined"): “So I don’t know why we’re sitting back and allowing this to happen, if we really believe it will extinguish the human race or enslave the human race. Like, how can that be good?” For his part, Trump has [expressed concern](https://time.com/6988935/trump-worries-ai-deepfakes-trigger-nuclear-war/ "undefined") about the risks posed by AI, too. The Conditional AI Safety Treaty could provide a solution to AI’s existential risk, while not unnecessarily obstructing AI development right now. Getting China and other countries to accept and enforce the treaty will no doubt be a major geopolitical challenge, but perhaps a Trump government is exactly what is needed to overcome it. A solution to one of the toughest problems we face—the existential risk of AI—does exist. It is up to us whether we make it happen, or continue to go down the path toward possible human extinction. ```json [{"@context":"https://schema.org","@type":"OpinionNewsArticle","@id":"https://time.com/7171432/conditional-ai-safety-treaty-trump/","mainEntityOfPage":{"@type":"WebPage","@id":"https://time.com/7171432/conditional-ai-safety-treaty-trump/"},"headline":"There Is a Solution to AI's Existential Risk Problem","datePublished":"2024-11-15T12:11:39.000Z","dateModified":"2026-04-13T08:53:28.319Z","description":"A Conditional AI Safety Treaty can help prevent runaway AI and President-elect Trump should embrace the idea, writes Otto Barten.","url":"https://time.com/7171432/conditional-ai-safety-treaty-trump/","keywords":["Tech","freelance"],"thumbnailUrl":"https://static.time.com/v3/assets/bltea6093859af6183b/blta73e18839eaa69a7/698a9bd7929fad9880c039dd/GettyImages-1804572639.jpg?branch=production&width=1200&quality=75&auto=webp&crop=1200:675&height=675","author":[{"@type":"Person","name":"Otto Barten","jobTitle":null,"url":"https://time.com/author/otto-barten/"}],"articleSection":"Ideas","image":[{"@type":"ImageObject","url":"https://static.time.com/v3/assets/bltea6093859af6183b/blta73e18839eaa69a7/698a9bd7929fad9880c039dd/GettyImages-1804572639.jpg?branch=production&width=1200&quality=75&auto=webp&crop=1200:675&height=675","width":1200,"height":675,"headline":"AGI Artificial General Intelligence concept image","caption":"AGI Artificial General Intelligence concept image","creditText":"Getty Images","representativeOfPage":true}],"publisher":{"@type":"Organization","name":"Time","url":"https://time.com/","logo":{"@type":"ImageObject","url":"https://time.com/images/logo.png","width":528,"height":156},"foundingDate":"March 3, 1923","sameAs":["https://www.facebook.com/time","https://www.instagram.com/time/?hl=en","https://twitter.com/time","https://www.pinterest.com/timemagazine"]}},{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/section/ideas/","name":"Ideas"}},{"@type":"ListItem","position":2,"item":{"@id":"https://time.com/7171432/conditional-ai-safety-treaty-trump/","name":"There Is a Solution to AI's Existential Risk Problem"}}]}] ```