Civitai—an online marketplace for buying and selling AI-generated content, backed by the venture capital firm Andreessen Horowitz—is letting users buy custom instruction files for generating celebrity deepfakes. Some of these files were specifically designed to make pornographic images banned by the site, a new analysis has found. The study, from researchers at Stanford and Indiana University, looked at people’s requests for content on the site, called “bounties.” The researchers found that between mid-2023 and the end of 2024, most bounties asked for animated content—but a significant portion were for deepfakes of real people, and 90% of these deepfake requests targeted women. (Their findings have not yet been peer reviewed.) The debate around deepfakes, as illustrated by the recent backlash to explicit images on the X-owned chatbot Grok, has revolved around what platforms should do to block such content. Civitai’s situation is a little more complicated. Its marketplace includes actual images, videos, and models, but it also lets individuals buy and sell instruction files called LoRAs that can coach mainstream AI models like Stable Diffusion into generating content they were not trained to produce. Users can then combine these files with other tools to make deepfakes that are graphic or sexual. The researchers found that 86% of deepfake requests on Civitai were for LoRAs. In these bounties, users requested “high quality” models to generate images of public figures like the influencer Charli D’Amelio or the singer Gracie Abrams, often linking to their social media profiles so their images could be grabbed from the web. Some requests specified a desire for models that generated the individual’s entire body, accurately captured their tattoos, or allowed hair color to be changed. Some requests targeted several women in specific niches, like artists who record ASMR videos. One request was for a deepfake of a woman said to be the user’s wife. Anyone on the site could offer up AI models they worked on for the task, and the best submissions received payment—anywhere from $0.50 to $5. And nearly 92% of the deepfake bounties were awarded.
Neither Civitai nor Andreessen Horowitz responded to requests for comment. It’s possible that people buy these LoRAs to make deepfakes that aren’t sexually explicit (though they’d still violate Civitai’s terms of use, and they’d still be ethically fraught). But Civitai also offers educational resources on how to use external tools to further customize the outputs of image generators—for example, by changing someone’s pose. The site also hosts user-written articles with details on how to instruct models to generate pornography. The researchers found that the amount of porn on the platform has gone up, and that the majority of requests each week are now for NSFW content.
“Not only does Civitai provide the infrastructure that facilitates these issues; they also explicitly teach their users how to utilize them,” says Matthew DeVerna, a postdoctoral researcher at Stanford’s Cyber Policy Center and one of the study’s leaders.  The company used to ban only sexually explicit deepfakes of real people, but in May 2025 it announced it would ban all deepfake content. Nonetheless, countless requests for deepfakes submitted before this ban now remain live on the site, and many of the winning submissions fulfilling those requests remain available for purchase, MIT Technology Review confirmed. “I believe the approach that they’re trying to take is to sort of do as little as possible, such that they can foster as much—I guess they would call it—creativity on the platform,” DeVerna says. Users buy LoRAs with the site’s online currency, called Buzz, which is purchased with real money. In May 2025, Civita’s credit card processor cut off the company because of its ongoing problem with nonconsensual content. To pay for explicit content, users must now use gift cards or cryptocurrency to buy Buzz; the company offers a different scrip for non-explicit content.  Civitai automatically tags bounties requesting deepfakes and lists a way for the person featured in the content to manually request its takedown. This system means that Civitai has a reasonably successful way of knowing which bounties are for deepfakes, but it’s still leaving moderation to the general public rather than carrying it out proactively.  A company’s legal liability for what its users do isn’t totally clear. Generally, tech companies have broad legal protections against such liability for their content under Section 230 of the Communications Decency Act, but those protections aren’t limitless. For example, “you cannot knowingly facilitate illegal transactions on your website,” says Ryan Calo, a professor specializing in technology and AI at the University of Washington’s law school. (Calo wasn’t involved in this new study.) Civitai joined OpenAI, Anthropic, and other AI companies in 2024 in adopting design principles to guard against the creation and spread of AI-generated child sexual abuse material . This move followed a 2023 report from the Stanford Internet Observatory, which found that the vast majority of AI models named in child sexual abuse communities were Stable Diffusion–based models “predominantly obtained via Civitai.” But adult deepfakes have not gotten the same level of attention from content platforms or the venture capital firms that fund them. “They are not afraid enough of it. They are overly tolerant of it,” Calo says. “Neither law enforcement nor civil courts adequately protect against it. It is night and day.” Civitai received a $5 million investment from Andreessen Horowitz (a16z) in November 2023. In a video shared by a16z, Civitai cofounder and CEO Justin Maier described his goal of building the main place where people find and share AI models for their own individual purposes. “We’ve aimed to make this space that’s been very, I guess, niche and engineering-heavy more and more approachable to more and more people,” he said.  Civitai is not the only company with a deepfake problem in a16z’s investment portfolio; in February, MIT Technology Review first reported that another company, Botify AI, was hosting AI companions resembling real actors that stated their age as under 18, engaged in sexually charged conversations, offered “hot photos,” and in some instances described age-of-consent laws as “arbitrary” and “meant to be broken.”

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. DHS is using Google and Adobe AI to make videos The news: The US Department of Homeland Security is using AI video generators from Google and Adobe to make and edit content shared with the public, a new document reveals. The document, released on Wednesday, provides an inventory of which commercial AI tools DHS uses for tasks ranging from generating drafts of documents to managing cybersecurity. Why it matters: It comes as immigration agencies have flooded social media with content to support President Trump’s mass deportation agenda—some of which appears to be made with AI—and as workers in tech have put pressure on their employers to denounce the agencies’ activities. Read the full story.—James O’Donnell
How the sometimes-weird world of lifespan extension is gaining influence
—Jessica HamzelouFor the last couple of years, I’ve been following the progress of a group of individuals who believe death is humanity’s “core problem.” Put simply, they say death is wrong—for everyone. They’ve even said it’s morally wrong. They established what they consider a new philosophy, and they called it Vitalism.Vitalism is more than a philosophy, though—it’s a movement for hardcore longevity enthusiasts who want to make real progress in finding treatments that slow or reverse aging. Not just through scientific advances, but by persuading influential people to support their movement, and by changing laws and policies to open up access to experimental drugs. And they’re starting to make progress. This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here. The AI Hype Index: Grok makes porn, and Claude Code nails your job Separating AI reality from hyped-up fiction isn’t always easy. That’s why we’ve created the AI Hype Index—a simple, at-a-glance summary of everything you need to know about the state of the industry. Take a look at this month’s edition of the index here. The must-reads

I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 Capgemini is no longer tracking immigrants for ICEAfter the French company was queried by the country’s government over the contract. (WP $)+ Here’s how the agency typically keeps tabs on its targets. (NYT $)+ US senators are pushing for answers about its recent surveillance shopping spree. (404 Media)+ ICE’s tactics would get real soldiers killed, apparently. (Wired $) 2 The Pentagon is at loggerheads with AnthropicThe AI firm is reportedly worried its tools could be used to spy on Americans. (Reuters)+ Generative AI is learning to spy for the US military. (MIT Technology Review) 3 It’s relatively rare for AI chatbots to lead users down harmful pathsBut when it does, it can have incredibly dangerous consequences. (Ars Technica)+ The AI doomers feel undeterred. (MIT Technology Review)4 GPT-4o’s days are numberedOpenAI says just 0.1% of users are using the model every day. (CNBC)+ It’s the second time that it’s tried to turn the sycophantic model off in under a year. (Insider $)+ Why GPT-4o’s sudden shutdown left people grieving. (MIT Technology Review)5 An AI toy company left its chats with kids exposedAnyone with a Gmail account was able to simply access the conversations—no hacking required. (Wired $)+ AI toys are all the rage in China—and now they’re appearing on shelves in the US too. (MIT Technology Review)6 SpaceX could merge with xAI later this yearAhead of a planned blockbuster IPO of Elon Musk’s companies. (Reuters)+ The move would be welcome news for Musk fans. (The Information $)+ A SpaceX-Tesla merger could also be on the cards. (Bloomberg $) 7 We’re still waiting for a reliable male contraceptiveTake a look at the most promising methods so far. (Bloomberg $) 8 AI is bringing traditional Chinese medicine to the massesAnd it’s got the full backing of the country’s government. (Rest of World) 9 The race back to the Moon is heating up Competition between the US and China is more intense than ever. (Economist $)10 What did the past really smell like?AI could help scientists to recreate history’s aromas—including mummies and battlefields. (Knowable Magazine)
Quote of the day “I think the tidal wave is coming and we’re all standing on the beach.”
—Bill Zysblat, a music business manager, tells the Financial Times about the existential threat AI poses to the industry.  One more thing Therapists are secretly using ChatGPT. Clients are triggered.Declan would never have found out his therapist was using ChatGPT had it not been for a technical mishap. The connection was patchy during one of their online sessions, so Declan suggested they turn off their video feeds. Instead, his therapist began inadvertently sharing his screen.For the rest of the session, Declan was privy to a real-time stream of ChatGPT analysis rippling across his therapist’s screen, who was taking what Declan was saying, putting it into ChatGPT, and then parroting its answers.But Declan is not alone. In fact, a growing number of people are reporting receiving AI-generated communiqués from their therapists. Clients’ trust and privacy are being abandoned in the process. Read the full story. —Laurie Clarke
We can still have nice things A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line or skeet ’em at me.) + Sinkholes are seriously mysterious. Is there a way to stay one step ahead of them?+ This beautiful pixel art is super impressive.+ Amid the upheaval in their city, residents of Minneapolis recently demonstrated both their resistance and community spirit in the annual Art Sled Rally (thanks Paul!)+ How on Earth is Tomb Raider 30 years old?!

For the last couple of years, I’ve been following the progress of a group of individuals who believe death is humanity’s “core problem.” Put simply, they say death is wrong—for everyone. They’ve even said it’s morally wrong. They established what they consider a new philosophy, and they called it Vitalism. Vitalism is more than a philosophy, though—it’s a movement for hardcore longevity enthusiasts who want to make real progress in finding treatments that slow or reverse aging. Not just through scientific advances, but by persuading influential people to support their movement, and by changing laws and policies to open up access to experimental drugs. And they’re starting to make progress.
Vitalism was founded by Adam Gries and Nathan Cheng—two men who united over their shared desire to find ways to extend human lifespan. I first saw Cheng speak back in 2023, at Zuzalu, a pop-up city in Montenegro for people who were interested in life extension and some other technologies. (It was an interesting experience—you can read more about it here.) Zuzalu was where Gries and Cheng officially launched Vitalism. But I’ve been closely following the longevity scene since 2022. That journey took me to Switzerland, Honduras, and a compound in Berkeley, California, where like-minded longevity enthusiasts shared their dreams of life extension.
It also took me to Washington, DC, where, last year, supporters of lifespan extension presented politicians including Mehmet Oz, who currently leads the Centers for Medicare & Medicaid Services, with their case for changes to laws and policies. The journey has been fascinating, and at times weird and even surreal. I’ve heard biohacking stories that ended with smoking legs. I’ve been told about a multi-partner relationship that might be made possible through the cryopreservation—and subsequent reanimation—of a man and the multiple wives he’s had throughout his life. I’ve had people tell me to my face that they consider themselves eugenicists, and that they believe that parents should select IVF embryos for their propensity for a long life. I’ve seen people draw blood during dinner in an upscale hotel restaurant to test their biological age. I’ve heard wild plans to preserve human consciousness and resurrect it in machines. Others have told me their plans to inject men’s penises with multiple doses of an experimental gene therapy in order to treat erectile dysfunction and ultimately achieve “radical longevity.” I’ve been shouted at and threatened with legal action. I’ve received barefoot hugs. One interviewee told me I needed Botox. It’s been a ride. My reporting has also made me realize that the current interest in longevity reaches beyond social media influencers and wellness centers. Longevity clinics are growing in number, and there’s been a glut of documentaries about living longer or even forever. At the same time, powerful people who influence state laws, giant federal funding budgets, and even national health policy are prioritizing the search for treatments that slow or reverse aging. The longevity community was thrilled when longtime supporter Jim O’Neill was made deputy secretary of health and human services last year. Other members of Trump’s administration, including Oz, have spoken about longevity too. “It seems that now there is the most pro-longevity administration in American history,” Gries told me. I recently spoke to Alicia Jackson, the new director of ARPA-H. The agency, established in 2022 under Joe Biden’s presidency, funds “breakthrough” biomedical research. And it appears to have a new focus on longevity. Jackson previously founded and led Evernow, a company focused on “health and longevity for every woman.” “There’s a lot of interesting technologies, but they all kind of come back to the same thing: Could we extend life years?” she told me over a Zoom call a few weeks ago. She added that her agency had “incredible support” from “the very top of HHS.” I asked if she was referring to Jim O’Neill. “Yeah,” she said. She wouldn’t go into the specifics.

Gries is right: There is a lot of support for advances in longevity treatments, and some of it is coming from influential people in positions of power. Perhaps the field really is poised for a breakthrough. And that’s what makes this field so fascinating to cover. Despite the occasional weirdness. This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here.

We’re expanding our risk domains and refining our risk assessment process.AI breakthroughs are transforming our everyday lives, from advancing mathematics, biology and astronomy to realizing the potential of personalized education. As we build increasingly powerful AI models, we’re committed to responsibly developing our technologies and taking an evidence-based approach to staying ahead of emerging risks.Today, we’re publishing the third iteration of our Frontier Safety Framework (FSF) — our most comprehensive approach yet to identifying and mitigating severe risks from advanced AI models.This update builds upon our ongoing collaborations with experts across industry, academia and government. We’ve also incorporated lessons learned from implementing previous versions and evolving best practices in frontier AI safety.Key updates to the FrameworkAddressing the risks of harmful manipulationWith this update, we’re introducing a Critical Capability Level (CCL)* focused on harmful manipulation — specifically, AI models with powerful manipulative capabilities that could be misused to systematically and substantially change beliefs and behaviors in identified high stakes contexts over the course of interactions with the model, reasonably resulting in additional expected harm at severe scale.This addition builds on and operationalizes research we’ve done to identify and evaluate mechanisms that drive manipulation from generative AI. Going forward, we’ll continue to invest in this domain to better understand and measure the risks associated with harmful manipulation.Adapting our approach to misalignment risksWe’ve also expanded our Framework to address potential future scenarios where misaligned AI models might interfere with operators’ ability to direct, modify or shut down their operations.While our previous version of the Framework included an exploratory approach centered on instrumental reasoning CCLs (i.e., warning levels specific to when an AI model starts to think deceptively), with this update we now provide further protocols for our machine learning research and development CCLs focused on models that could accelerate AI research and development to potentially destabilizing levels.In addition to the misuse risks arising from these capabilities, there are also misalignment risks stemming from a model’s potential for undirected action at these capability levels, and the likely integration of such models into AI development and deployment processes.To address risks posed by CCLs, we conduct safety case reviews prior to external launches when relevant CCLs are reached. This involves performing detailed analyses demonstrating how risks have been reduced to manageable levels. For advanced machine learning research and development CCLs, large-scale internal deployments can also pose risk, so we are now expanding this approach to include such deployments.Sharpening our risk assessment processOur Framework is designed to address risks in proportion to their severity. We’ve sharpened our CCL definitions specifically to identify the critical threats that warrant the most rigorous governance and mitigation strategies. We continue to apply safety and security mitigations before specific CCL thresholds are reached and as part of our standard model development approach.Lastly, in this update, we go into more detail about our risk assessment process. Building on our core early-warning evaluations, we describe how we conduct holistic assessments that include systematic risk identification, comprehensive analyses of model capabilities and explicit determinations of risk acceptability.Advancing our commitment to frontier safetyThis latest update to our Frontier Safety Framework represents our continued commitment to taking a scientific and evidence-based approach to tracking and staying ahead of AI risks as capabilities advance toward AGI. By expanding our risk domains and strengthening our risk assessment processes, we aim to ensure that transformative AI benefits humanity, while minimizing potential harms.Our Framework will continue evolving based on new research, stakeholder input and lessons from implementation. We remain committed to working collaboratively across industry, academia and government.The path to beneficial AGI requires not just technical breakthroughs, but also robust frameworks to mitigate risks along the way. We hope that our updated Frontier Safety Framework contributes meaningfully to this collective effort.

We thank the International Collegiate Programming Contest (ICPC) for their support.This project was a large-scale collaboration, and its success is due to the combined efforts of many individuals and teams. Hanzhao (Maggie) Lin led the overall technical direction for Gemini competitive programming and ICPC 2025 efforts, and co-led with Heng-Tze Cheng on the overall research and execution.The leads and key contributors of the ICPC 2025 team are the following: Chenkai Kuang, Yuan Liu, Zhaoqi Leng, Jieming Mao, Lalit Jain, Chenjie Gu, Goran Žužić, Adams Yu, YaGuang Li, Xiaomeng Yang, Yang Xiao, Adam Zhang, Alex Vitvitskyi, Ashkan Norouzi Fard, Blanca Huergo, Evan Liu, Golnaz Ghiasi, Huan Gui, John Aslanides, Jonathan Lee, Kuba Lacki, Larisa Markeeva, Luheng He, Nigamaa Nayakanti, Nikos Parotsidis, Paul Covington, Petar Veličković, Qijun Tan, Ragha Kotikalapudi, Renshen Wang, Sasan Tavakkol, Shuang Liu, Sidharth Mudgal, Steve Li, Vincent Cohen-Addad, Xianghong Luo, Xinying Song, Yiming Li and Zicheng Xu.The advanced Gemini Deep Think for ICPC was built on foundational research jointly from the Gemini post-training, Thinking and Coding areas including: Aja Huang, Andreas Kirsch, Ankesh Anand, Archit Sharma, Betty Chan, Chenxi Liu, Cosmo Du, Dawsen Hwang, Dustin Tran, Edward Lockhart, Feryal Behbahani, Fred Zhang, Garrett Bingham, Hao Zhou, Hoang Nguyen, Irene Cai, Jian Li, Jarrod Kahn, Junehyuk Jung, Junsu Kim, Kate Baumli, Kefan Xiao, Le Hou, Lei Yu, Maciej Kula, Mahan Malihi, Marcelo Menegali, Miklós Z. Horváth, Mirek Olšák, Nate Kushman, Pei Sun, Pol Moreno, Rosemary Ke, Sahitya Potluri, Shane Gu, Shubha Raghvendra, Siamak Shakeri, Sid Lall, Steven Zheng, Thang Luong, Theophane Weber, Tong He, Tianhe (Kevin) Yu, Trieu Trinh, Vikas Yadav, Vinay Ramasesh, Vinh Tran, Weiyue Wang, Wilfried Bounsi, Xiyang Luo, Yangsibo Huang, Yi Tay, Yong Cheng, Yuan Zhang, Yuri Chervonyi and Yujing Zhang.This effort was advised by Quoc Le and Vahab Mirrokni, with program and operation management from Kristen Chiafullo, Eric Ni, Srinivas Tadepalli, Jessica Lo and Sajjad Zafar.We’d also like to thank our competitive programming experts for providing insights: Alexander Grushetsky, Chun-Sung Ferng, Ilya Kornakov, Liang Bai, Petr Mitrichev and Sergey Rogulenko.We want to extend our deepest gratitude to the Gemini serving team: Abhijit Karmarkar, Cip Baetu, Emanuel Taropa, Evan Senter, Federico Lebron, Girish Ramchandra Rao, Greg Anielak, Hamish Tomlinson, Hayden Jeune, Jia Zhao, Joe Stanton, Ashish Shenoy, Jonathan Kairupan, Juliette Love, Justin Mao-Jones, Kashyap Krishnakumar, Ken Franko, Mahesh Palekar, Minh Giang, Nikhil Sethi, Rohan Jain, Rohit Varkey Thankachan, Soheil Hassas Yeganeh, Thomas Jimma and Vitor Rodrigues.Further thanks to the following people for support, collaboration, and advice: Benoit Schillings, Ed Chi, Koray Kavukcuoglu, Jeff Dean, Oriol Vinyals, Noam Shazeer, James Manyika, Yossi Matias, Philipp Schindler, Pushmeet Kohli, Demis Hassabis, Sergey Brin, Melvin Johnson, Omer Levy, Timothy Lillicrap, Anca Dragan, Slav Petrov, Ya Xu, Madhavi Sewak, Erika Gemzer, Eugénie Rives, Erica Moreira, Tulsee Doshi, Alex Goldin, Jane Labanowski, Andy Forbes, Sean Nakamoto, Yifeng Lu, Denny Zhou, Alexander Novikov, Cristy Hayner, Hanada Tatsuki, Harsh Dhand, Ritu Ghai, Hiroki Kayama, Jenny Rizk Nicholls, Jo Chick, Song Zuo, Pratyusha Mukherjee, Shibo Wang, Carlos Guia, Xiaofan Zhang, …Finally, we thank Dr. Bill Poucher from the ICPC global for the support and endorsement.The ICPC global has confirmed that our submitted solutions are complete and accepted. It is important to note that their review does not extend to validating our system, processes, or underlying model.

Our new method could help mathematicians leverage AI techniques to tackle long-standing challenges in mathematics, physics and engineering.For centuries, mathematicians have developed complex equations to describe the fundamental physics involved in fluid dynamics. These laws govern everything from the swirling vortex of a hurricane to airflow lifting an airplane’s wing.Experts can carefully craft scenarios that make theory go against practice, leading to situations which could never physically happen. These situations, such as when quantities like velocity or pressure become infinite, are called ‘singularities’ or ‘blow ups’. They help mathematicians identify fundamental limitations in the equations of fluid dynamics, and help improve our understanding of how the physical world functions.In a new paper, we introduce an entirely new family of mathematical blow ups to some of the most complex equations that describe fluid motion. We’re publishing this work in collaboration with mathematicians and geophysicists from institutions including Brown University, New York University and Stanford UniversityOur approach presents a new way to leverage AI techniques to tackle longstanding challenges in mathematics, physics and engineering that demand unprecedented accuracy and interpretability.The importance of unstable singularitiesStability is a crucial aspect of singularity formation. A singularity is considered stable if it is robust to small changes. Conversely, an unstable singularity requires extremely precise conditions.It’s expected that unstable singularities play a major role in foundational questions in fluid dynamics because mathematicians believe no stable singularities exist for the complex boundary-free 3D Euler and Navier-Stokes equations. Finding any singularity in the Navier-Stokes equations is one of the six famous Millennium Prize Problems that are still unsolved.With our novel AI methods, we presented the first systematic discovery of new families of unstable singularities across three different fluid equations. We also observed a pattern emerging as the solutions become increasingly unstable. The number characterizing the speed of the blow up, lambda (λ), can be plotted against the order of instability, which is the number of unique ways the solution can deviate from the blow up. The pattern was visible in two of the equations studied, the Incompressible Porous Media (IPM) and Boussinesq equations. This suggests the existence of more unstable solutions, whose hypothesized lambda values lie along the same line.