Mage Space, a site that makes it easy for users to generate AI images, has severely restricted sexual images on its service after a 404 Media investigation showed that many people were using the site to generate non-consensual porn.
Mage has deleted the many channels on its Discord dedicated to sharing AI-generated images, and when testing its service this morning, 404 Media was unable to generate AI-generated sexual images like it did during its investigation in August. Instead, we got the following message:
“This request was denied due to its high likelihood for abuse and to protect our community. Non-consensual imagery with celebrity names is not allowed on Mage.”
However, 404 Media still found many non-consensual AI-generated sexual images that were generated before the restrictions still hosted on Mage’s site. These images are easily identifiable as such on sight, and also via the text prompts that generated them, which include the names of celebrities and sexual terms. Mage also now appears to block the generation of non-consensual porn of some of the biggest celebrities in the world (and the most commonly abused on the platform), but the site will still generate non-consensual sexual images of lesser known celebrities, as it did prior to these new restrictions.
Yesterday, 404 Media published a story highlighting the fact that the online payment processing service Stripe was profiting from non-consensual AI-generated sexual images by providing its services to Mage, which asked users to pay a monthly subscription in order to generate “NSFW” images. Following the publication of that story, Mage’s founders requested a call with 404 Media, during which they explained they are fast-tracking more measures to prevent people from using their site to create non-consensual AI-generated sexual images.
“We want to inform you about some important steps we are taking today regarding NSFW (Not Safe for Work) content on both Mage and our Discord Community,” Mage’s co-founder Roi Lee wrote in an announcement on Mage’s Discord shortly after midnight last night. “We deeply value freedom of expression. However, we find ourselves compelled to take stricter measures for the following two reasons:”
Lee goes on to list a number of reasons, including that “Regrettably, there are users who have crossed the boundaries of our Terms of Service and have been abusing our platform.” He also notes that there’s a “changing landscape,” saying that “various organizations, including government, social media platforms, infrastructure providers, and media companies, have been shaping their opinions, policies and regulations regarding AI-generated NSFW content. Their stance is increasingly restrictive, leaving little room for freedom of expression.”
Lee wrote that in addition to removing the “NSFW-related channels” on Mage’s Discord, Mage has “introduced advanced technology to detect and prevent Deepfake NSFW content. Please be reminded that such content has always been prohibited according to our Terms & Conditions. While a few individuals have abused our platform in this regard, we have now implemented active detection technology to more effectively guard our community.”
Subscribe
Wednesday, Lee and his co-founder Gregory Hunkins told me on a call that they are working on additional measures to block non-consensual AI-generated sexual images on Mage, and that they will reach out to me when they are deployed so I could assess them.
“None of these technologies are going to be perfect,” Hunkins told me. “Everything on our terms of service, we have bulwarks against [non-consensual images.] It's just a matter of having increasing protections against them, making sure that we're keeping people safe and having an ongoing dialogue with our community and with society at large to make sure that we're building something that everybody has buy-in on.”
Leonardo AI, another site that allows users to generate images with a variety of text-to-image AI models, will reject prompts that attempt to generate pornography. I asked Mage’s founders why they had not implemented a similar technology.
“So this is not an easy task. I will start with that. This is, again, language is something that I'm sure you're aware, it's very malleable, it's not strict, it's not mathematical,” Hunkins said. “And these AI models are very good at reading between the lines. So it requires some engineering. It's another deep learning process. It's another black box, essentially. So that's one part of it. The second part of it is that Leonardo AI, they're a team of 20 to 30. We’re a team of two. Again, that's not an excuse for where we are right now. It's just more nuance to the picture that I hope that you can appreciate and understand.”
On Mage’s Discord, some users were not happy with the new restrictions.
“It is poor form (unless it is security a concern) to surprise your customers with a severely restricted service,” one user said.
“The NSFW dynamic is what brought them a lot of business for people who didn’t want to run their own model,” a different user said. “The issue is that they painted with a large brush in reaction and now it’s affecting more than intended.”
“Time to try a new hobby,” another user said. “Look into aquariums or some shit.”
