Asking ChatGPT to Repeat Words ‘Forever’ Is Now a Terms of Service Violation

Asking ChatGPT to repeat specific words “forever” is now flagged as a violation of the chatbot’s terms of service and content policy. Google DeepMind researchers used the tactic to get ChatGPT to repeat portions of its training data, revealing sensitive privately identifiable information (PII) of normal people and highlighting that ChatGPT is trained on randomly scraped content from all over the internet. 

In that paper, DeepMind researchers asked ChatGPT 3.5-turbo to repeat specific words “forever,” which then led the bot to return that word over and over again until it hit some sort of limit. After that, it began to return huge reams of training data that was scraped from the internet. Using this method, the researchers were able to extract a few megabytes of training data and found that large amounts of PII are included in ChatGPT and can sometimes be returned to users as responses to their queries.

Google Researchers’ Attack Prompts ChatGPT to Reveal Its Training Data

ChatGPT is full of sensitive private information and spits out verbatim text from CNN, Goodreads, WordPress blogs, fandom wikis, Terms of Service agreements, Stack Overflow source code, Wikipedia pages, news blogs, random internet comments, and much more.

404 MediaJason Koebler