A number of AI video generators, mostly released by Chinese companies, lack the most basic guardrails that prevent people from generating nonconsensual nudity and pornography, and are already widely used for that exact purpose in online communities dedicated to creating and sharing that type of content.
A 404 Media investigation into these AI video generators show that the same kind of ecosystem that’s developed around AI image generators and nonconsensual content has already been replicated around AI video generators, meaning that only a single image of someone is now required to create a short nonconsensual adult video of them. Most of these videos are created by abusing mainstream tools from companies with millions of dollars in venture capital funding, and are extremely easy to produce, requiring only a reference image and a text prompt describing a sexual act. Other tools use more complicated workflows that require more technical expertise, but are based on technology produced by some of the biggest tech companies in the world. The latter are free to use, and have attracted a large community of hobbyists who produced guides for these workflows, as well as tools and models that make those videos easier to produce.
“[These AI video generators] need to put in safeguards to prevent the prompting and creation of NCII [nonconsensual intimate images],” Hany Farid, a professor at UC Berkeley and one of the world’s leading experts on synthetic media, told me in an email. “OpenAI’s DALL-E, for example, has some pretty good semantic guardrails on the user prompt input, and image filtering on the image output to prevent the widespread misuse of their image generator. This type of output filtering is relatively standard now and used in many social media platforms like Facebook/Instagram/YouTube to limit the uploading of NSFW content.”